Dealing with UDP streamed data - saving

[RobbieP]

Thanks! It's very simple really...multiple (up to 200) records in a single packet that contain a name and a value...the value is always a double. Think flight simulator position data streamed at about 30-60hz. Can you recommend a good example for something like that?

[SGaist]

I don't really have any reference application at hand. How are you going to save the data ? One row per record or one row per block of received data ?

[RobbieP]

one row per block of data probably...so if it were csv an example would be:
timestamp,elapsed,airspeed,120.345,altitude,6700.304,pitch,15.13,roll,1.223,yaw,0.121

the data comes in like this:

#define SP_PACKET_SIZE 200
#define NAME_SIZE 64
    struct SP_PacketStruct
    {
        int Size;                               ///< Number of items in the data packet
        char Name[SP_PACKET_SIZE][NAME_SIZE];   ///< Data labels
        double Value[SP_PACKET_SIZE];           ///< Data packet values
    };

[SGaist]

Then I would benchmark the insertion speed and maybe offload the storage to a secondary thread.

[RobbieP]

@SGaist That makes sense. What I'm curious about is the storage method though...I've been looking for the best option...is SqlIte what you would recommend? When you say benchmark, are you referring to buffering? I'll be in control of the send rate...it varies for each project. The highest rate we've ever dealt with is 125hz.

[SGaist]

No, to benchmark is to measure performance.

Depending on the volume of data you expect to get a full database system like PostgreSQL might be better.

Note that a NoSQL database might also be of interest.

[RobbieP]

I have experience with mysql...do you think sqlite would be a bad choice?

[RobbieP]

I ask because I'm not really sure I even need a database...most of our previous data has all been just dumped to csv files that our statisticians can then import into matlab for analysis.

[SGaist]

Hence the benchmarking :-)

[RobbieP]

Gotcha...I'm honestly leaning more towards just buffering to memory with periodic text file writing...

[SGaist]

If the amount of data is reasonable and the file writing speed is as well, that is a valid option. Don't forget the final dump when you close the application otherwise you might lose data.

[RobbieP]

yup. Before I start diving into this then, do you know of any good examples of buffered text file writing? Thanks for the discussion!

[SGaist]

Using QTextStream with QFile would be a good start.
Start simple, check speed and if not enough, rework that part.