Data acquisition storage: QDataStream vs. ofstream vs. DB vs. csv

[jars121]

@JKSH said in Data acquisition storage: QDataStream vs. ofstream vs. DB vs. csv:

@jars121 said in Data acquisition storage: QDataStream vs. ofstream vs. DB vs. csv:

1. I'm not sure, I plan on re-instating the QVector approach to measure CPU load today.
2. As per answer 1 above, I'll review the frequency accuracy today.

Yes, do your benchmarks first. The data will help you see what your next steps should be.

3. I'm acquiring all sorts. I have 32 analog channels coming in via SPI ADCs (temperature, voltage, pressure, etc.), and 32 digital channels interfacing directly with the embedded device (GPIO inputs via sysfs and/or memory mapping), which can either be simple ON/OFF measurements (state of a relay for example), or pulse frequency/width measurements (PWM signals for example).

What pulse widths and frequencies are you expecting to see? This will determine your sampling rate for the digital inputs.

4. At the moment it's a Raspberry Pi 2, but this will change in the very near future to a more commercial embedded platform (with heavily modified OS, PCB design, etc.).
5. This will eventually become a commercial product, using the commercial version of Qt.

Customizing the OS and PCB could cost you a lot of time in development, testing, troubleshooting, and refining. Have you considered off-the-shelf components?

If you still want to take the custom PCB route for a commercial product, remember to include protection circuitry. End-users are notorious for plugging things in the wrong way (for example, switching the + and - terminals, or applying an input voltage that's too high)

Ok so I've done some basic tests, saving the data to a QVector<QVector<QPair<QString, double>>>, and the results are pretty telling/disappointing. The average (over around 15,000 samples) was around 1400us (1.4ms), which isn't too bad, but the jitter was astronomical. The lowest execution time within the 15,000 samples was around 0.7ms, but there were numerous instances of executions running for over 5ms.

This largely confirms a suspicion that's been nagging at me for a little while now; I'm not going to achieve the sampling rate nor the accuracy I need from within a non-RTOS. At this point, I'm going to reconsider my design, and look at incorporating a dedicated data acquisition microcontroller, which can store readings in a buffer and routinely pass the data to my device.

@J.Hilk said in Data acquisition storage: QDataStream vs. ofstream vs. DB vs. csv:

@jars121 said in Data acquisition storage: QDataStream vs. ofstream vs. DB vs. csv:

I'm using threading quite extensively throughout the application, and am currently running the SQLite function on a separate thread. Similarly, the data acquisition functions are run on separate threads as well. Each of the 32 digital inputs has its own thread, as the pulse frequency and width measurements rely on sysfs interrupts, which are a glorified poll of the gpio's corresponding sysfs file status (1 or 0), so they have to be on separate threads unfortunately. All 32 analog signals are on a single thread.

I would highly recomment that you reconsider this approach.

A pi2 has a decent cpu, IIRC 900MHz quad-core, but ~40 threads alone from your program with a precision timing resolution of around 1ms, will be hard. The OS will struggle heavily with juggling the threads back and forth between the cores to be processed. You should easily get delays in the ms-area before the first thread is sheduled to be processed the next time.

As per my comment above, I agree completely. I've been pondering how to minimise the number of digital input threads, but with sysfs interrupts (polls) it's just not feasible.

[JKSH]

@jars121 said in Data acquisition storage: QDataStream vs. ofstream vs. DB vs. csv:

At this point, I'm going to reconsider my design, and look at incorporating a dedicated data acquisition microcontroller, which can store readings in a buffer and routinely pass the data to my device.

What form factor do you want your device to be? Again, have you looked at off-the-shelf components instead of custom PCBs and custom OS'es?

Again, what pulse widths and frequencies are you expecting to see?

[jars121]

@JKSH said in Data acquisition storage: QDataStream vs. ofstream vs. DB vs. csv:

@jars121 said in Data acquisition storage: QDataStream vs. ofstream vs. DB vs. csv:

At this point, I'm going to reconsider my design, and look at incorporating a dedicated data acquisition microcontroller, which can store readings in a buffer and routinely pass the data to my device.

What form factor do you want your device to be? Again, have you looked at off-the-shelf components instead of custom PCBs and custom OS'es?

Again, what pulse widths and frequencies are you expecting to see?

I've actually done further testing this morning, made a few slight changes to my SPI approach, and I'm now getting over 2000 samples per channel across 7 channels on the ADC, so a dedicated microcontroller might not be needed just yet. I've got more testing to do, but it's starting to look a little promising.

I've poured over the available options in the market, and nothing meets all the requirements for the device. I'm designing the PCB in conjunction with writing the software, so despite the extra time, effort, expense, etc. associated with a custom design, it's necessary and accounted for in this case.

In terms of input frequencies, the highest I'd expect to see is maybe a couple of kHz, hence the requirement to at least obtain 1ksps. In testing of individual pulse inputs I've been able to measure well beyond this amount and with much smaller widths than I'd expect to see with high accuracy (verified with oscilloscope). With that said, the fact that each digital input is currently on its own thread is an obvious issue, so I'm investigating a means to still use interrupts, but in a non-blocking fashion on the same thread.