QSerialPort readyRead not able to receive data fast enough.

lukutis222 · 12 Aug 2022, 12:02

I have external MCU connected to my PC and I am making my own custom terminal. I have created a signal and socket connection:

    connect(&serial->serial_connection, &QSerialPort::readyRead, this, &Widget::readData);

Whenever I connect to the device, I start reading back the serial data that the device is transmitting. The readData function is being called:

void Widget::readData()
{
    QByteArray data = serial->serial_connection.readLine();
    QString DataAsString = QString(data);
    ui->Console_read->setTextColor(Qt::black); //by default the text will be black
    ui->Console_read->insertPlainText(data);
}

After the data is read, I insert this data to the console widget that I have created and it looks like this

My external device has a very simple program running. Every 1 second it prints a block of prints to the serial port. The external device code (ESP32 microcontroller):

static void HELLO_TASK(void* param);

void app_main(void)
{
    xTaskCreate(HELLO_TASK,"HELLO_TASK",5000,NULL,3,NULL); // receiving commands from main uart
}




static void HELLO_TASK(void* param){
    int counter = 0;
    for (;;)
    {    
        printf("***************BLOCK START*****************\n");
        printf("BLOCK COUNTER =%u \n",counter);
        ESP_LOGI("INFO","info log1");
        ESP_LOGW("WARNING","warning log1");
        ESP_LOGE("ERROR","error log1");
        ESP_LOGI("INFO","info log2");
        ESP_LOGW("WARNING","warning log2");
        ESP_LOGE("ERROR","error log2");
        ESP_LOGI("INFO","info log3");
        ESP_LOGW("WARNING","warning log3");
        ESP_LOGE("ERROR","error log3");
        printf("***************BLOCK END*****************\n");
        counter++;
        vTaskDelay(1000/portTICK_PERIOD_MS);
    }
}

I noticed very strange behaviour in my readData function. I think it is because of

QByteArray data = serial->serial_connection.readLine();

My QT console receive data in chunks. For example, instead of receiving a whole block of prints

***************BLOCK START*****************
BLOCK COUNTER =47 
[0;32mI (49657) INFO: info log1[0m
[0;33mW (49657) WARNING: warning log1[0m
[0;31mE (49657) ERROR: error log1[0m
[0;32mI (49657) INFO: info log2[0m
[0;33mW (49657) WARNING: warning log2[0m
[0;31mE (49667) ERROR: error log2[0m
[0;32mI (49667) INFO: info log3[0m
[0;33mW (49667) WARNING: warning log3[0m
[0;31mE (49677) ERROR: error log3[0m
***************BLOCK END*****************

I receive data that is split in multiple blocks for example:
I might receive block:

***************BLOCK START*****************
BLOCK COUNTER =47 
[0;32mI (49657) INFO: info log1[0m
[0;33mW (49657) WARNING: warning log1[0m
[0;31mE (49657) ERROR: error log1[0m
[0;32mI (49657) INFO: info log2[0m
[0;33mW (49657) WARNING: warning log2[0m
[0;31mE (49667) ERROR: error log2[0m

And then after 1 second I will receive remaining of block + beggining of a next block:

[0;32mI (49667) INFO: info log3[0m
[0;33mW (49667) WARNING: warning log3[0m
[0;31mE (49677) ERROR: error log3[0m
***************BLOCK END*****************
***************BLOCK START*****************
BLOCK COUNTER =48 
[0;32mI (50687) INFO: info log1[0m
[0;33mW (50687) WARNING: warning log1[0m

Also, I noticed that the serial data displayed on my console is lagging behind the data that is actually being transmitted. From the external MCU code, you can see that the new block of prints is being transmitted every 1 second. So after 60 seconds of external device running and sending data to my QT console, I should see the BLOCK COUNTER = 60 (ignoring the small FreeRTOS delay) but in reality, I see huge difference in block counter, my console would only show BLOCK COUNTER = 40 or something like that.

I would like to understand what is happening and why is it happening. Appreciate in advance.

JonB · wrote on 12 Aug 2022, 12:02

@lukutis222
I know nothing about "MCU" or possible speed issues. But your code in readData() is not robust. It expects to be able to read a full line. This is not how things work. All you know when readyRead signal is raised is that anywhere from the full data sent right down to a single byte has been received. Your readLine() could easily read "partial" lines. You must use some kind of buffering, and a line is only wholly received when the end-of-line character has been read into the buffer.

mpergand · wrote on 12 Aug 2022, 12:34

@lukutis222
From my understanding reading the doc,
you may try:

void Widget::readData()
{
    if( ! serial->serial_connection.canReadLine()) return;
    // go ahead
    QByteArray data = serial->serial_connection.readLine();
   ...
}

mrjj · 13 Aug 2022, 09:48

@lukutis222 said in QSerialPort readyRead not able to receive data fast enough.:

Make sure ESP_LOGE append \n to the output or readLine wont work as expected.

lukutis222 · wrote on 13 Aug 2022, 09:47

Thank you all for the tips. I will try what @mpergand suggested next week.

Just a quick update. I also, I tried to use readAll() method (From the QT Terminal example) and I can read all data without any issues but I do not prefer this method because I for my particular needs, I need to be able to read each individual line of data (Line of data I consider a string with a /n or /r termination at the end).

So if I have multiple lines printed as in my current example, I want to be able to get every single line seperately and perform different string operations for each line.

From what I understand about readAll that is not possible to do because when readyRead signal is raised, the readAll function will read whatever data has been filled on the serial buffer up to that particular moment (there can be multiple different seperate lines of data seperated by \n but readAll will merge them in one big buffer)

Maybe it is not correct to use readyRead signal here? Is there a possibility to use signal to detect a single line of data? From what I understand and learned about readyRead is that it triggers randomly ( does not matter if remote device sends 1 message, 2 messages or many more messages, there is no guarantee that the readyRead signal will be triggered once all data has been received (maybe idle detected on the receiver). It may trigger halfway through the receive and then trigger the readyRead signal once again to complete receiving remaining of the data.

lukutis222 · 13 Aug 2022, 09:58

@mrjj ESP_LOG automatically inserts \n hence why it is not shown in my code that I have uploaded. I will try @mpergand example next week and report my findings.

mrjj · wrote on 13 Aug 2022, 09:58

@lukutis222
Ok super.

One note
(Line of data I consider a string with a /n or /r termination at the end).

canReadLine only checks for \n

SGaist · wrote on 13 Aug 2022, 17:41

Hi,

Based on your requirements, using readAll and waiting to have a full frame will allow you to properly split each line and then apply the processing you want on them.

lukutis222 · 22 Aug 2022, 07:41

@mpergand said in QSerialPort readyRead not able to receive data fast enough.:

if( ! serial->serial_connection.canReadLine()) return;
// go ahead
QByteArray data = serial->serial_connection.readLine();

Sorry for the late feedback. I have been off sick last week :(

Now I am back and I have tried canReadLine method:

void Widget::readData()
{

    if( ! serial->serial_connection.canReadLine())
        return;
    // go ahead
    QByteArray data = serial->serial_connection.readLine();
    ui->Console_read->setTextColor(Qt::black); //by default the text will be black
    ui->Console_read->insertPlainText(data);
}

This method seems to work the same as my initial method. It is receiving data in chunks

How I expect it to receive data:

Receive a block:

***************BLOCK START*****************
BLOCK COUNTER =21 
[1B][0;32mI (22877) INFO: info log1[1B][0m
[1B][0;33mW (22877) WARNING: warning log1[1B][0m
[1B][0;31mE (22877) ERROR: error log1[1B][0m
[1B][0;32mI (22877) INFO: info log2[1B][0m
[1B][0;33mW (22877) WARNING: warning log2[1B][0m
[1B][0;31mE (22887) ERROR: error log2[1B][0m
[1B][0;32mI (22887) INFO: info log3[1B][0m
[1B][0;33mW (22887) WARNING: warning log3[1B][0m
[1B][0;31mE (22897) ERROR: error log3[1B][0m
***************BLOCK END*****************

1 second delay (because external device sends a new block of prints every 1 second)

Receive next block:

***************BLOCK START*****************
BLOCK COUNTER =22 
[1B][0;32mI (23907) INFO: info log1[1B][0m
[1B][0;33mW (23907) WARNING: warning log1[1B][0m
[1B][0;31mE (23907) ERROR: error log1[1B][0m
[1B][0;32mI (23907) INFO: info log2[1B][0m
[1B][0;33mW (23907) WARNING: warning log2[1B][0m
[1B][0;31mE (23917) ERROR: error log2[1B][0m
[1B][0;32mI (23917) INFO: info log3[1B][0m
[1B][0;33mW (23917) WARNING: warning log3[1B][0m
[1B][0;31mE (23927) ERROR: error log3[1B][0m
***************BLOCK END*****************

How it actually works:
Receive part of the 1st block :

***************BLOCK START*****************
BLOCK COUNTER =21 
[1B][0;32mI (22877) INFO: info log1[1B][0m
[1B][0;33mW (22877) WARNING: warning log1[1B][0m
[1B][0;31mE (22877) ERROR: error log1[1B][0m
[1B][0;32mI (22877) INFO: info log2[1B][0m
[1B][0;33mW (22877) WARNING: warning log2[1B][0m
[1B][0;31mE (22887) ERROR: error log2[1B][0m

Wait 1 second (I dont understand why it is waiting 1 second here even though the full block has not been received). After waiting 1 second, I receive the remaining of a first block and a beggining of a 2nd block:

[1B][0;32mI (22887) INFO: info log3[1B][0m
[1B][0;33mW (22887) WARNING: warning log3[1B][0m
[1B][0;31mE (22897) ERROR: error log3[1B][0m
***************BLOCK END*****************

***************BLOCK START*****************
BLOCK COUNTER =22 
[1B][0;32mI (23907) INFO: info log1[1B][0m
[1B][0;33mW (23907) WARNING: warning log1[1B][0m
[1B][0;31mE (23907) ERROR: error log1[1B][0m

and it keeps going on like that slowly building up delay.

However, with readAll(); method, I am able to receive an expected full block every 1 second without any issues. However with this method, I am unsure how can I split different each line and process them. In my particular example, a block of prints consists of 12 different lines :

        printf("***************BLOCK START*****************\n");
        printf("BLOCK COUNTER =%u \n",counter);
        ESP_LOGI("INFO","info log1");
        ESP_LOGW("WARNING","warning log1");
        ESP_LOGE("ERROR","error log1");
        ESP_LOGI("INFO","info log2");
        ESP_LOGW("WARNING","warning log2");
        ESP_LOGE("ERROR","error log2");
        ESP_LOGI("INFO","info log3");
        ESP_LOGW("WARNING","warning log3");
        ESP_LOGE("ERROR","error log3");
        printf("***************BLOCK END*****************\n");

When I use readAll, I receive all this data as one big chunk of data and not 12 seperate lines of data.
@SGaist Could you advise on how to properly split the incoming data into different lines? The reason why I need to process each line independantly because I want to apply different color text to different lines

JonB · 23 Aug 2022, 08:22

@lukutis222 said in QSerialPort readyRead not able to receive data fast enough.:

When I use readAll, I receive all this data as one big chunk of data and not 12 seperate lines of data.

Please use the documentation to find QStringList QString::split(QChar sep, Qt::SplitBehavior behavior = Qt::KeepEmptyParts, Qt::CaseSensitivity cs = Qt::CaseSensitive const for yourself.

lukutis222 · 23 Aug 2022, 08:30

@JonB Thanks for pointing in the right direction. I have read a little about Split method and found this old forum post from 2006:
https://www.qtcentre.org/threads/22-How-to-detect-new-line

I tried to implement it in my readData function:


void Widget::readData()
{

    QByteArray data = serial->serial_connection.readAll();
    QString DataAsString = QString(data);

    QStringList lines = DataAsString.split("\r", Qt::SkipEmptyParts );
    foreach( QString line, lines ) {
       line = line.trimmed();
       qDebug("Data split = %s \n",line.toStdString().c_str());
    }
    ui->Console_read->setTextColor(Qt::black); //by default the text will be black
    ui->Console_read->insertPlainText(data);

However, I am seeing very strange results when I try to qDebug print every line that has been split.
The application output from QT Creator:

Data split = ***************BLOCK START***************** 
Data split = BLOCK COUNTER =2 
Data split =  
Data split = [0;32mI (3377) INFO: info log1[0m 
Data split = [0;33mW (3377) WARNING: wa 
Data split = rning log1[0m 
Data split = [0;31mE (3377) ERROR: error log1[0m 
Data split = [0;32mI 
Data split = (3377) INFO: info log2[0m 
Data split = [0;33mW (3377) WARNING: warning log 
Data split = 2[0m 
Data split = [0;31mE (3387) ERROR: error log2[0m 
Data split = [0;32mI (3387) IN 
Data split = FO: info log3[0m 
Data split = [0;33mW (3387) WARNING: warning log3[0m 
Data split = [ 
Data split = 0;31mE (3397) ERROR: error log3[0m 
Data split = ***************BLOCK END*** 
Data split = ************** 
Data split =  
Data split =  
Data split = ***************BLOCK START***************** 
Data split = BLOCK COUNTER =3 
Data split =  
Data split = [0;32mI (4397) INFO: info log1[0m 
Data split = [0;33mW (4397) WARNING: wa 
Data split = rning log1[0m 
Data split = [0;31mE (4397) ERROR: error log1[0m 
Data split = [0;32mI 
Data split = (4397) INFO: info log2[0m 
Data split = [0;33mW (4397) WARNING: warning log 
Data split = 2[0m 
Data split = [0;31mE (4407) ERROR: error log2[0m 
Data split = [0;32mI (4407) IN 
Data split = FO: info log3[0m 
Data split = [0;33mW (4407) WARNING: warning log3[0m 
Data split = [ 
Data split = 0;31mE (4417) ERROR: error log3[0m 
Data split = ***************BLOCK END*** 
Data split = ************** 
Data split =  
Data split =

I can see many issues. For example, something strange happened here:

Data split = [0;33mW (3377) WARNING: wa 
Data split = rning log1[0m

And then:

Data split = [0;32mI 
Data split = (3377) INFO: info log2[0m

It does not seem to be splitting my string correctly. For some reason it thinks there there is a line termination in the middle of my message which definately is not correct.

JonB · 23 Aug 2022, 08:34

@lukutis222
I already explained why (the likely reason) in my first post:

But your code in readData() is not robust. It expects to be able to read a full line. This is not how things work. All you know when readyRead signal is raised is that anywhere from the full data sent right down to a single byte has been received. Your readLine() could easily read "partial" lines. You must use some kind of buffering, and a line is only wholly received when the end-of-line character has been read into the buffer.

Please stop trying things, read this and understand. Presumably your "extra line breaks" simply mark reaching the end of some arbitrary set of bytes which happen to have been received so far at that point, which can happen any time. You need to code/cope for this if you want to recognise where genuine line breaks occur in the input, so that your output is "nice".

lukutis222 · 23 Aug 2022, 08:53

@JonB
I am trying things out as this is the only way for me to learn faster. What you explained makes sense to me however I have no clue how to overcome this issue. This very simple thing ( Simply being able to access each line individually) seems to be much bigger problem that it actually is. For some reason it is very complicated to achieve what I want even though it sounds so simple.

Would you be able to suggest anything, show some pseudocode how can I achieve this ?

JonB · wrote on 23 Aug 2022, 08:53

@lukutis222
Think about the end of the stream of bytes received so far and returned by readAll(). Say This is a line.\r has been sent to you. There are two possible ways this might arrive:

This is a line.\r, i.e. the whole thing. That's great, your code will work. Since you have received a completed line you can afford to output it now as a complete line.
This is a li, i.e. chopped off somewhere in the middle. split() does not tell you about this, you don't know from that whether the line ended in \r or just stopped where it did at end of string. But you need to know that, because if it's the latter you cannot afford to output what you received so far as a whole line, you need to wait till next readyRead() for the remainder.

This is presumably what canReadLine() does for you, it checks there is a line terminator. I don't know why something like while (canReadLine()) qDebug() << readLine() did not work for you, to save you the effort of your own buffering. I don't know whether your serial lines actually end in \r or \r\n and whether that affects canReadLine().

If you can't use that and have to do your own readAll() followed by split() you need to do something like: search the received bytes backwards from the end to find the last \r/\n. Then only pass from the beginning to that point over to split() and output, since that's the totality of the complete lines. Put the remaining bytes (partial line) into a persistent buffer, next time you readAll() append those bytes to that buffer and repeat above process to find final completed line read so far.

lukutis222 · wrote on 24 Aug 2022, 09:25

Hello again. I have managed to achieve what I want using canReadLine method. Perhaps I was not using it correctly at first. I share my code and results now hopefully it will be helpful for someone else:

void Widget::readData()
{


    QRegularExpression regex("(\\033\\1330;32m)(.*)(\\033\\1330m)"); //INFO
    QRegularExpression regex2("(\\033\\1330;33m)(.*)(\\033\\1330m)"); //WARNING
    QRegularExpression regex3("(\\033\\1330;31m)(.*)(\\033\\1330m)"); //ERROR

    while (serial->serial_connection.canReadLine()) {
        QByteArray line = serial->serial_connection.readLine();
        qDebug("Data split = %s \n",line.toStdString().c_str());
        QRegularExpressionMatch match = regex.match(line);
        QRegularExpressionMatch match2 = regex2.match(line);
        QRegularExpressionMatch match3 = regex3.match(line);

        if (match.hasMatch()) {
            QString matched_text = match.captured(2);
            matched_text.insert(matched_text.length(), QString("\n"));
            ui->Console_read->setTextColor(QColor(0,128,0)); // 'custom' color
            ui->Console_read->insertPlainText(matched_text);
        }

        else if (match2.hasMatch()) {
            QString matched_text2 = match2.captured(2);
            matched_text2.insert(matched_text2.length(), QString("\n"));
            ui->Console_read->setTextColor(QColor(255,165,0)); // 'custom' color
            ui->Console_read->insertPlainText(matched_text2);
        }
        else if (match3.hasMatch()) {
            QString matched_text3 = match3.captured(2);
            matched_text3.insert(matched_text3.length(), QString("\n"));
            ui->Console_read->setTextColor(QColor(255,0,0)); // 'custom' color
            ui->Console_read->insertPlainText(matched_text3);
        }
    }
}

The code above is simply parsing 3 ANSI codes:
0;31m - should be Green color
0;32m - should be Orange color
0;33m - should be Red color.

If the data is received without any ANSI code, I will simply print it in black color.

Raw data (using Termite terminal):

My custom terminal:

I am aware that this is nowhere close to perfect and might not be able to parse different formats. But it works for my particular device and I can upgrade it later on

Thank you all for helping me out and pointing me in the right direction. I am very grateful for your help and patience :)

SGaist · 29 Aug 2022, 05:14

Since you are going to reuse these regular expression frequently, you should rather make them class members to avoid the creation and expression rebuild. Also, since you have three different search that are exclusive to each other, it's also wasteful to make all the three every time.

lukutis222 · 29 Aug 2022, 05:17

@SGaist Hello. Thanks for suggestion. Can you clarify what do you mean "it's also wasteful to make all the three every time.". How can I fix that?

jsulm · wrote on 29 Aug 2022, 05:17

@lukutis222 said in QSerialPort readyRead not able to receive data fast enough.:

How can I fix that?

Like @SGaist already suggested: make the regular expressions class members and initialise them once instead of on each readData() call...