Dealing with UDP streamed data - saving
-
I'm receiving data streamed in over UDP and I'd like to collate it, display it, and most importantly, save it. I was looking at the localstorage example, but it uses QML...is there a way to do a sqlite localstorage type data recording using regular QMainWindow style QT Apps? I'm still learning QT and there's so many options...I'm trying to decide the quickest turnaround option. I'd even be happy with buffering it to memory and periodically writing to a csv or json file...Any thoughts?
-
Hi,
Qt's SQL module supports SQLite out of the box so you can use for your widget application (QtQuick as well but you have to create some interface in C++).
What kind of data are you receiving ?
-
Thanks! It's very simple really...multiple (up to 200) records in a single packet that contain a name and a value...the value is always a double. Think flight simulator position data streamed at about 30-60hz. Can you recommend a good example for something like that?
-
I don't really have any reference application at hand. How are you going to save the data ? One row per record or one row per block of received data ?
-
one row per block of data probably...so if it were csv an example would be:
timestamp,elapsed,airspeed,120.345,altitude,6700.304,pitch,15.13,roll,1.223,yaw,0.121the data comes in like this:
#define SP_PACKET_SIZE 200 #define NAME_SIZE 64 struct SP_PacketStruct { int Size; ///< Number of items in the data packet char Name[SP_PACKET_SIZE][NAME_SIZE]; ///< Data labels double Value[SP_PACKET_SIZE]; ///< Data packet values };
-
Then I would benchmark the insertion speed and maybe offload the storage to a secondary thread.
-
@SGaist That makes sense. What I'm curious about is the storage method though...I've been looking for the best option...is SqlIte what you would recommend? When you say benchmark, are you referring to buffering? I'll be in control of the send rate...it varies for each project. The highest rate we've ever dealt with is 125hz.
-
No, to benchmark is to measure performance.
Depending on the volume of data you expect to get a full database system like PostgreSQL might be better.
Note that a NoSQL database might also be of interest.
-
Hence the benchmarking :-)
-
If the amount of data is reasonable and the file writing speed is as well, that is a valid option. Don't forget the final dump when you close the application otherwise you might lose data.
-
Using QTextStream with QFile would be a good start.
Start simple, check speed and if not enough, rework that part.