QSerialPort parsing binary packets at a fast speed

aha_1980

@lukutis222 I don't know why you would need a timer.

Your receive packets, and whenever a packet is received you can process it.

But before going that far I'd first try if you can receive all packets at that speed without loss. E.g. by just queing all received bytes for some time (one minute?) in a buffer and check afterwards that there are no missing bytes. As you have a CRC, that should be easy to verify.

If you can receive without loss, then you can push received packets to another process to handle them.

lukutis222

First of all. I would like to thank everyone for their input. I will reply to some statements:

@VentureLee
Regarding using seperate threads, this is exactly what I am doing is that not right? The SerialWorker class that is responsible for receiving and parsing data is on a seperate thread called workerThread whereas the the rest of the logic (in this case logging data to file) happens in main thread (handleProcessedData)

Regarding using the 1ms QTimer I agree that this is not the most optimal solution. I will rewrite the logic without relying on QTimer and post back the results here.

As you have seen from my code, to capture the timestamp (relative_timestamp) I use QElapsedTimer timer_usec that is started at the beggining of a reception.

Then everytime new packet is detected, before writing it to the log I do:


qint64 elapsedNanoseconds = timer_usec.nsecsElapsed(); 
double relative_timestamp = elapsedNanoseconds / 1e9;

Is that a good method for capturing timestamps? Is that guaranteed to have incremented correctly (assuming I am able to read data packets at 1ms intervals). Perhaps I am able to receive and parse data every 1ms but this timer is not to be trusted hence irregular timestamps printed in the logs?

@aha_1980
Thats good suggestion. Il try to send 1000 packets from my measurement device and see if I manage to receive all 1000 packets (without checking the time intervals).

aha_1980

@lukutis222 I can answer your third question: QElapsedTimer has the highest resolution you can get to measure time differences.

I think it will work perfectly for your case.

lukutis222

I have simplified my serialworker greatly:

#include "serialworker.h"
#include <qDebug>
#include <QDateTime>



SerialWorker::SerialWorker(SettingsWindow *settings_ptr, QObject *parent)
    : QObject{parent}
{
    settings_local = settings_ptr;

}

SerialWorker::~SerialWorker() {
    if (serial->isOpen()) {
        serial->close();
    }
    delete serial;
}




void SerialWorker::initialize() {
    qDebug("initialize method \n");

    serial = new QSerialPort(this);
    connect(serial, &QSerialPort::readyRead, this, &SerialWorker::processData);
}




void SerialWorker::processData() {
    static QByteArray buffer; // Persistent buffer for incomplete packets
    buffer.append(serial->readAll()); // Append the new data

    while (buffer.size() >= 13)// Minimum size of a packet
    {
        if (static_cast<uint8_t>(buffer[0]) == 0xAA && static_cast<uint8_t>(buffer[1]) == 0x55)
        {
            if (static_cast<uint8_t>(buffer[2]) == 0x09) { // Validate payload length
                if (buffer.size() < 13)
                    return;           // Wait for more data

                char channel = buffer[3];
                if (channel == static_cast<char>(0x41))
                {
                    float voltage, current;
                    memcpy(&voltage, buffer.mid(4, 4).data(), 4);
                    memcpy(&current, buffer.mid(8, 4).data(), 4);

                    uint8_t calculatedChecksum = 0;
                    for (int i = 2; i < 12; ++i)
                    {
                        calculatedChecksum ^= static_cast<uint8_t>(buffer[i]);
                    }

                    if (calculatedChecksum == static_cast<uint8_t>(buffer[12]))
                    {
                        // Emit the processed data to the main thread
                        emit dataProcessed(channel, voltage, current);
                        buffer.remove(0, 13); // Remove processed packet
                        continue;
                    } else
                    {
                        qDebug() << "Invalid checksum. Skipping packet.";
                        buffer.remove(0, 13); // Remove invalid packet
                    }
                }
                else
                {
                    qDebug() << "Unknown channel. Skipping packet.";
                    buffer.remove(0, 13); // Remove unknown packet
                }
            }
            else
            {
                qDebug() << "Invalid payload length. Skipping packet.";
                buffer.remove(0, 13); // Remove invalid packet
            }
        }
        else
        {
            qDebug() << "Invalid header. Searching for next valid header.";
            int headerIndex = buffer.indexOf(QByteArray::fromHex("AA55"), 1);
            if (headerIndex == -1)
            {
                buffer.clear(); // No valid header found, clear the buffer
            }
            else
            {
                buffer.remove(0, headerIndex); // Move to the next valid header
            }
        }
    }
}




void SerialWorker::write_data(QByteArray data){
    int64_t bytes_written = serial->write(data);
}



bool SerialWorker::Serial_connect(){

    if(serial->open(QIODevice::ReadWrite)){
        qDebug("Connection is succesfull \n");
        return 1;
    }
    else
    {
        //error
        qDebug() << serial->errorString();
    }
    return 0;

}
bool SerialWorker::Serial_disconnect(){
    if(serial->isOpen()){
        serial->close();
        return 1;
    }
    else{
        return 0;
    }
}

bool SerialWorker::Check_if_open(){
    if(serial->isOpen()){
        return 1;
    }
    else{
        return 0;
    }

}

QString SerialWorker::Get_Connect_Port(){
    if(Check_if_open() == 1){
        return serial->portName();
    }
    return "null";
}

I have configured my power analyzer device to send some test packets. This is pseudo code of my power analyzer:

        	   static int counter = 1;
        	   if (counter <= 10000) {
        	       struct DataPacket packet;
        	       uint8_t channel = 0x41; // ASCII 'A'

        	       // Replace voltage and current data with counter value
        	       float dummyVoltage = 0.0f; // Placeholder for voltage (can be set to 0)
        	       float dummyCurrent = (float)(counter); // Use the counter as the 'current' data

        	       // Encode the packet with counter value as the 'current'
        	       encodeDataPacket(channel, dummyVoltage, dummyCurrent, &packet);

        	       // Transmit the packet
        	       HAL_UART_Transmit(&huart4, (uint8_t *)&packet, sizeof(packet), 0xFF);

        	       counter++;
        	   }

It simply sends the packets that contains counter value. For example:

(The counter values are converted to single point floating precision)

0xAA 0x55 0x09 0x41 0x00 0x00 0x00 0x00 0x00 0x00 0x80 0x3F CRC
0xAA 0x55 0x09 0x41 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x40 CRC
0xAA 0x55 0x09 0x41 0x00 0x00 0x00 0x00 0x00 0x00 0x40 0x40 CRC

and etc...

When the processData detects a valid 13 byte packet, it calls:

emit dataProcessed(channel, voltage, current);

Then in my handleProcessedData I do the following:


void MainWindow::handleProcessedData(char channel, float voltage, float current) {
    if (channel == 'A') {
        static int counter = 1;  // Static counter to persist across function calls
                // Measure elapsed time in nanoseconds and calculate relative timestamp
        qint64 elapsedNanoseconds = timer_usec.nsecsElapsed();
        double relative_timestamp = elapsedNanoseconds / 1e9;
                // Format the log string to include counter, timestamp, voltage, and current
        QString logString = QString("%1, %2, %3 V, %4 mA")
                                .arg(counter)                                  // Counter value
                                .arg(QString::number(relative_timestamp, 'f', 6))  // Relative timestamp
                                .arg(QString::number(voltage, 'f', 2))        // Voltage
                                .arg(QString::number(current, 'f', 2));       // Current
                // Write the log string to the file
        logging_local->Write_to_file(logString);
        counter++;  // Increment the counter
    }
}

The purpose of this test is to simply send 10000 packets from my device and confirm if I am able to receive all 10000 packets on my QT application. After running the test, I can open the log file or plot the points to see what happened.

The results:

The QT application managed to receive all 10000 packets and it took roughly 10 seconds to do so which is correct. However, the time intervals between each packet are very inconsistent. Sometimes, it parse multiple packets within a couple of uS (micro seconds) and then some other times it parses some packets within 10's of ms (miliseconds), but the end result is that it received all 10000 packets within the expected time interval (10 seconds).

However, it is still not clear to me why the timestamps are so inconsistent. See a snippet of a log below.

1685, 2.452991, 0.00 V, 1685.00 mA
1686, 2.453939, 0.00 V, 1686.00 mA
1687, 2.454988, 0.00 V, 1687.00 mA
1688, 2.455882, 0.00 V, 1688.00 mA
1689, 2.456991, 0.00 V, 1689.00 mA
1690, 2.457946, 0.00 V, 1690.00 mA
1691, 2.469666, 0.00 V, 1691.00 mA
1692, 2.469721, 0.00 V, 1692.00 mA
1693, 2.469732, 0.00 V, 1693.00 mA
1694, 2.469741, 0.00 V, 1694.00 mA
1695, 2.469749, 0.00 V, 1695.00 mA
1696, 2.469756, 0.00 V, 1696.00 mA
1697, 2.469763, 0.00 V, 1697.00 mA
1698, 2.469770, 0.00 V, 1698.00 mA
1699, 2.469777, 0.00 V, 1699.00 mA
1700, 2.469784, 0.00 V, 1700.00 mA
1701, 2.472088, 0.00 V, 1701.00 mA
1702, 2.472121, 0.00 V, 1702.00 mA
1703, 2.472133, 0.00 V, 1703.00 mA
1704, 2.472141, 0.00 V, 1704.00 mA
1705, 2.507428, 0.00 V, 1705.00 mA
1706, 2.507476, 0.00 V, 1706.00 mA
1707, 2.507486, 0.00 V, 1707.00 mA
1708, 2.507494, 0.00 V, 1708.00 mA
1709, 2.507502, 0.00 V, 1709.00 mA
1710, 2.507509, 0.00 V, 1710.00 mA
1711, 2.507516, 0.00 V, 1711.00 mA
1712, 2.507523, 0.00 V, 1712.00 mA
1713, 2.507530, 0.00 V, 1713.00 mA
1714, 2.507537, 0.00 V, 1714.00 mA
1715, 2.507543, 0.00 V, 1715.00 mA
1716, 2.507551, 0.00 V, 1716.00 mA
1717, 2.507559, 0.00 V, 1717.00 mA
1718, 2.507566, 0.00 V, 1718.00 mA
1719, 2.507573, 0.00 V, 1719.00 mA
1720, 2.507580, 0.00 V, 1720.00 mA
1721, 2.507587, 0.00 V, 1721.00 mA
1722, 2.507594, 0.00 V, 1722.00 mA
1723, 2.507600, 0.00 V, 1723.00 mA
1724, 2.507607, 0.00 V, 1724.00 mA
1725, 2.507614, 0.00 V, 1725.00 mA
1726, 2.507621, 0.00 V, 1726.00 mA
1727, 2.507627, 0.00 V, 1727.00 mA
1728, 2.507634, 0.00 V, 1728.00 mA
1729, 2.507641, 0.00 V, 1729.00 mA
1730, 2.507648, 0.00 V, 1730.00 mA
1731, 2.507654, 0.00 V, 1731.00 mA
1732, 2.507661, 0.00 V, 1732.00 mA
1733, 2.507668, 0.00 V, 1733.00 mA
1734, 2.507674, 0.00 V, 1734.00 mA
1735, 2.507681, 0.00 V, 1735.00 mA
1736, 2.507690, 0.00 V, 1736.00 mA
1737, 2.507696, 0.00 V, 1737.00 mA
1738, 2.507703, 0.00 V, 1738.00 mA
1739, 2.507710, 0.00 V, 1739.00 mA
1740, 2.508267, 0.00 V, 1740.00 mA
1741, 2.509030, 0.00 V, 1741.00 mA
1742, 2.509948, 0.00 V, 1742.00 mA
1743, 2.510947, 0.00 V, 1743.00 mA
1744, 2.512067, 0.00 V, 1744.00 mA
1745, 2.512973, 0.00 V, 1745.00 mA
1746, 2.513943, 0.00 V, 1746.00 mA
1747, 2.516227, 0.00 V, 1747.00 mA
1748, 2.516279, 0.00 V, 1748.00 mA
1749, 2.517017, 0.00 V, 1749.00 mA
1750, 2.517944, 0.00 V, 1750.00 mA
1751, 2.518952, 0.00 V, 1751.00 mA
1752, 2.520017, 0.00 V, 1752.00 mA
1753, 2.520939, 0.00 V, 1753.00 mA
1754, 2.522184, 0.00 V, 1754.00 mA
1755, 2.522946, 0.00 V, 1755.00 mA
1756, 2.523933, 0.00 V, 1756.00 mA
1757, 2.524976, 0.00 V, 1757.00 mA
1758, 2.525922, 0.00 V, 1758.00 mA
1759, 2.526946, 0.00 V, 1759.00 mA
1760, 2.527952, 0.00 V, 1760.00 mA
1761, 2.529366, 0.00 V, 1761.00 mA
1762, 2.532158, 0.00 V, 1762.00 mA
1763, 2.533333, 0.00 V, 1763.00 mA
1764, 2.533362, 0.00 V, 1764.00 mA
1765, 2.533533, 0.00 V, 1765.00 mA
1766, 2.533974, 0.00 V, 1766.00 mA
1767, 2.534923, 0.00 V, 1767.00 mA
1768, 2.535962, 0.00 V, 1768.00 mA
1769, 2.536948, 0.00 V, 1769.00 mA
1770, 2.537951, 0.00 V, 1770.00 mA
1771, 2.539811, 0.00 V, 1771.00 mA
1772, 2.540188, 0.00 V, 1772.00 mA
1773, 2.543413, 0.00 V, 1773.00 mA
1774, 2.545811, 0.00 V, 1774.00 mA
1775, 2.545833, 0.00 V, 1775.00 mA
1776, 2.546210, 0.00 V, 1776.00 mA
1777, 2.546637, 0.00 V, 1777.00 mA
1778, 2.546659, 0.00 V, 1778.00 mA
1779, 2.547927, 0.00 V, 1779.00 mA
1780, 2.549926, 0.00 V, 1780.00 mA
1781, 2.550731, 0.00 V, 1781.00 mA
1782, 2.550756, 0.00 V, 1782.00 mA
1783, 2.551652, 0.00 V, 1783.00 mA
1784, 2.552000, 0.00 V, 1784.00 mA

For example
Between packet number 1685 and packet 1686, timestamp difference is:
2.453939 - 2.452991 = 0.000948 (0.9ms) which is almost correct.

However, if you look at some other packets such as 1704 and 1705:
2.507428 - 2.472141 = 0.035287 (35ms)

I have probed the RX/TX lines with logic analyzer to confirm whether irregularities are coming from the power analyzer side or not and I can confidently confirm that the analyzer works perfectly fine - it transmits packets at precisely every 1ms interavals.

If you want to check the whole log, you can download it via wetransfer link below:
https://we.tl/t-RCxUkiMwa6

At this point, I see only 2 possibilities:

logging_local->Write_to_file(logString); is causing my system to lag for a short amount of time. But that is not likely because logging happens in main thread whereas parsing the data happens in serialworker thread.
My processData method does not work reliably. Even though it succesfully parsed all 10000 packets as were expected, it did not manage to capture them "on time"

aha_1980

@lukutis222 How is the serial line connected to the computer? Via a hardware RS-232 or by an USB-Serial adapter?

Also, which operating system are you using?

lukutis222

@aha_1980
It is connected via USB->Serial connverter. But I am probing the RX TX lines and they seem perfectly fine on the Salae logic analyzer.

I am on Windows 11

JonB

@lukutis222
I am not an expert in hardware, please take this with a pinch of salt. But e.g. as per the https://forum.qt.io/topic/159883/timerevent-response-overtime we mentioned earlier, Windows is not a RTOS. How do you know when the OS might be off doing stuff? How do you know when your thread --- worker or main/UI --- gets scheduled? When the OS hands data to your program? Maybe 35ms is too much, I wouldn't know. But I don't think you can guarantee things to be consistently 1ms apart in real time. My guess is that if you rewrote the code from Qt to native Windows (using the same calls) you would see the same lumpiness? In which case it wouldn't be a Qt issue. But you may know more than I.

Christian Ehrlicher

As @JonB already wrote - if you want to process real time (for whatever reason) you have to use a proper os. There are so much buffers and schedulers involved that your numbers are in the exepcted range.

artwaw

I think I recall previous conversations in the forum, where it was hard to go below 15/16ms boundary with reliable results. You might want to dig a bit and find those posts.

Pl45m4

@JonB said in QSerialPort parsing binary packets at a fast speed:

But e.g. as per the https://forum.qt.io/topic/159883/timerevent-response-overtime we mentioned earlier

@artwaw Jon already "found" and referred to it :)

artwaw

Ah, great! Missed that entry...

Additionally, speaking of Windows configuration - default OS timer setting:

The default platform timer resolution is 15.6ms (15625000ns) and should be used whenever the system is idle. If the timer resolution is increased, processor power management technologies may not be effective. The timer resolution may be
increased due to multimedia playback or graphical animations.
Current Timer Resolution (100ns units) 156250

This probably can be tweaked but I'd rather not to.

kuzulis

No, it is impossible to rely on the packets reception time. You can't take a timestamps from that. As, I understand, do you need a timestamps, for the plotting voltage/current values, e.g. on the plot?

If, so, then you have a two options:

Use a proxy/route MCU which will receive a packages and set a timestamp for every received package, and then route a modified packages to your PC (e.g. with some other your protocol).
Or, as you know that a package received every one millisecond, then just increase a timestamp +1. So, your X-axis will contains some relative time, not absolute. You then can assign that time to everything.))

p. 2 is a more simple and preferable, IMHO. ))

lukutis222

Thanks again for all the responses.

@JonB and @Christian-Ehrlicher
So from what I understood, it is completely not doable to achieve what I want by using my current method.
Regards to using a proper OS, could you please point me in the right direction for some reading material about how to implement an OS within my QT application. I am familliar with FreeRTOS but that is for the embedded devices and not for QT desktop applications. Does the same principles apply here as well? Can I use FreeRTOS for this or this is completely different? When I look up on "google" about QT and OS, not much comes up apart from:
https://doc.qt.io/archives/QtForMCUs-2.2/qtul-using-with-freertos.html

In response to @kuzulis statement. Yes I need the timestamps to plot the current/voltage values. The workarround that you have mentioned (using relative time instead of an actual time when the packet was received) might work. I might have to use this method if I dont manage to find any other way to fix this "properly".