Dealing with UDP streamed data - saving

RobbieP

I'm receiving data streamed in over UDP and I'd like to collate it, display it, and most importantly, save it. I was looking at the localstorage example, but it uses QML...is there a way to do a sqlite localstorage type data recording using regular QMainWindow style QT Apps? I'm still learning QT and there's so many options...I'm trying to decide the quickest turnaround option. I'd even be happy with buffering it to memory and periodically writing to a csv or json file...Any thoughts?

SGaist

Hi,

Qt's SQL module supports SQLite out of the box so you can use for your widget application (QtQuick as well but you have to create some interface in C++).

What kind of data are you receiving ?

RobbieP

Thanks! It's very simple really...multiple (up to 200) records in a single packet that contain a name and a value...the value is always a double. Think flight simulator position data streamed at about 30-60hz. Can you recommend a good example for something like that?

SGaist

I don't really have any reference application at hand. How are you going to save the data ? One row per record or one row per block of received data ?

RobbieP

one row per block of data probably...so if it were csv an example would be:
timestamp,elapsed,airspeed,120.345,altitude,6700.304,pitch,15.13,roll,1.223,yaw,0.121

the data comes in like this:

#define SP_PACKET_SIZE 200
#define NAME_SIZE 64
    struct SP_PacketStruct
    {
        int Size;                               ///< Number of items in the data packet
        char Name[SP_PACKET_SIZE][NAME_SIZE];   ///< Data labels
        double Value[SP_PACKET_SIZE];           ///< Data packet values
    };

SGaist

Then I would benchmark the insertion speed and maybe offload the storage to a secondary thread.

RobbieP

@SGaist That makes sense. What I'm curious about is the storage method though...I've been looking for the best option...is SqlIte what you would recommend? When you say benchmark, are you referring to buffering? I'll be in control of the send rate...it varies for each project. The highest rate we've ever dealt with is 125hz.

SGaist

No, to benchmark is to measure performance.

Depending on the volume of data you expect to get a full database system like PostgreSQL might be better.

Note that a NoSQL database might also be of interest.

RobbieP

I have experience with mysql...do you think sqlite would be a bad choice?

RobbieP

I ask because I'm not really sure I even need a database...most of our previous data has all been just dumped to csv files that our statisticians can then import into matlab for analysis.

SGaist

Hence the benchmarking :-)

RobbieP

Gotcha...I'm honestly leaning more towards just buffering to memory with periodic text file writing...

SGaist

If the amount of data is reasonable and the file writing speed is as well, that is a valid option. Don't forget the final dump when you close the application otherwise you might lose data.

RobbieP

yup. Before I start diving into this then, do you know of any good examples of buffered text file writing? Thanks for the discussion!

SGaist

Using QTextStream with QFile would be a good start.
Start simple, check speed and if not enough, rework that part.