Best(fastest) way to save and load 150MB of data
-
Currently, my app downloads data from the internet on each run (about 100-200 MB). The data is a list of timestamp - value pairs (imagine QList<QMap<double,double>>). In order to speed up the initialization of the app, I decided to save the downloaded data locally on the first run, and on subsequent runs load the saved data, and download only new data from the internet.
Obviously, I only care about sequentially reading the saved data on each startup of the app, and then sequentially appending a bit of new data.The easiest option to save the data is plain text files (using QFile). Should I consider another method of storing the data ? Maybe JSON, some form of archiving, or even SQL ? What is the fastest method of loading the data (considering its size and the sequential access) ?
-
It really depends on your usecase what you want to do with your data.
-
Based on the description I'd say the best solution is to use a database. With a DB you won't need to worry about details of implementation, SQL will take care of it. It will also be easy to search through the records if needed, sort them etc.
-
Currently, my app downloads data from the internet on each run (about 100-200 MB). The data is a list of timestamp - value pairs (imagine QList<QMap<double,double>>). In order to speed up the initialization of the app, I decided to save the downloaded data locally on the first run, and on subsequent runs load the saved data, and download only new data from the internet.
Obviously, I only care about sequentially reading the saved data on each startup of the app, and then sequentially appending a bit of new data.The easiest option to save the data is plain text files (using QFile). Should I consider another method of storing the data ? Maybe JSON, some form of archiving, or even SQL ? What is the fastest method of loading the data (considering its size and the sequential access) ?
@cpper
I do not share @sierdzio's thought that storing what was a bunch of thousands(?) of consecutive records which you say you wanted in memory or for sequential access is best stored in a database/file which then has to be searched. IMHO in-memory manipulation is a lot faster than any file-based. But it does depend totally on your use case.From the way you described, my initial thought was to use
QDataStreamof the binary representation (e.g.QList<QMap<double,double>>)), which I assume to be "fast" on read/write, and no need to convert to/from text. But I don't know whether aQDataStreamfile allows any kind of "append new items" --- it may not. I'd be interested if anyone knows about this?Otherwise the "fastest" is surely just a sequential layout of the records, which is just (approximately?) a sequence of
double,doubles, easy to save, load, and reconstruct the maps. -
@sierdzio said in Best(fastest) way to save and load 150MB of data:
SQLite supports in-memory storage
Point taken. You probably know more than I do [which the OP should bear in mind!]. And I admit I haven't used SQLite and have a natural antipathy toward any database solution unless I see a database need. Which may be wrong of me. But my thought is that, even in-memory, there must be some huge overhead in storing/processing as database records what is apparently supposed to be just 150MB-worth of sequential
doubles? -
I don't know, to be honest - it would be interesting to benchmark it out. Databases tend to be pretty well optimized. But I suggested using a DB not necessarily for speed, but rather for convenience - it should be easier to synchronize local and remote data with a database which does a lot of heavy lifting (checking uniqueness, searching etc.) automatically.
(Z(:^
-
I don't know, to be honest - it would be interesting to benchmark it out. Databases tend to be pretty well optimized. But I suggested using a DB not necessarily for speed, but rather for convenience - it should be easier to synchronize local and remote data with a database which does a lot of heavy lifting (checking uniqueness, searching etc.) automatically.
-
Thanks for the answers. Here more detail about the data:
The downloaded and saved data is historic cryptocurrency price data, for multiple pairs (like BTC-USD, ETH-USD ...). I do some basic math with this data (like calculating % change) and plot it in QCharts. I'm more interested in the fastest way to load the data, since saving the biggest chunk takes place only once, at the first run of the app. On subsequent runs, only a small amount of new price data will be saved to the local data. Also, I don't care about things like searching or sorting the data.
-
Then use a simple plain QFile
-
Thanks for the answers. Here more detail about the data:
The downloaded and saved data is historic cryptocurrency price data, for multiple pairs (like BTC-USD, ETH-USD ...). I do some basic math with this data (like calculating % change) and plot it in QCharts. I'm more interested in the fastest way to load the data, since saving the biggest chunk takes place only once, at the first run of the app. On subsequent runs, only a small amount of new price data will be saved to the local data. Also, I don't care about things like searching or sorting the data.
@cpper said in Best(fastest) way to save and load 150MB of data:
and plot it in QCharts
Others may well know about QCharts but I'm afraid I don't. But if you are looking at the "fastest" way of accessing this data, my first question would be how do you present this data to QtCharts for the plotting? Is there a model? An in-memory or database one? Does it have to be a
QList<QMap<double,double>>)or something? One would assume the fastest would be however one can most quickly stuff the data from disk into the structure directly required for the plotting.
