How to implement reading serial data with ANSI color codes and printing out to textbox in color

lukutis222

@JonB When I decided to parse ANSI color codes I really did not expect this to be such complex topic.

As you have suggested, I started experimenting with different regex expressions. The one that I am currently working with is the following:

(0;33m)(.*)(0m)

My code:

    QRegularExpression regex("(0;33m)(.*)(0m)", QRegularExpression::MultilineOption);
    QRegularExpressionMatch match = regex.match(text);
    if (match.hasMatch()) {

        QString matched_text0 = match.captured(0);
        QString matched_text1 = match.captured(1);
        QString matched_text2 = match.captured(2);
        qDebug("matched text0 = %s \n",matched_text0.toStdString().c_str());
        qDebug("matched text1 = %s \n",matched_text1.toStdString().c_str());
        qDebug("matched text2 = %s \n",matched_text2.toStdString().c_str());
    }

Is now returning:

matched text0 = 0;33mW (00:30:02.590) THERMOSTAT: CURRENT TIME: Thu Jan  1 00:30:02 1970[0m 
matched text1 = 0;33m 
matched text2 = W (00:30:02.590) THERMOSTAT: CURRENT TIME: Thu Jan  1 00:30:02 1970[

Which I think is getting closer to what I want to achieve. matched text2 is a text that I want to highlight.

I am still not fully sure about how can I now ensure that the unmatched data does not get displayed on my console?

UPDATE

Do you know how can I format only the specific match that I choose instead of whole string? When I was using global match, I used capturedStart() but I dont think I can use that now.

For example, I want to format matched_text2 only. I tried the following:

    QRegularExpression regex("(0;33m)(.*)(0m)", QRegularExpression::MultilineOption);
    QRegularExpressionMatch match = regex.match(text);
    if (match.hasMatch()) {

        QString matched_text0 = match.captured(0);
        QString matched_text1 = match.captured(1);
        QString matched_text2 = match.captured(2);
        qDebug("matched text0 = %s \n",matched_text0.toStdString().c_str());
        qDebug("matched text1 = %s \n",matched_text1.toStdString().c_str());
        qDebug("matched text2 = %s \n",matched_text2.toStdString().c_str());
        setFormat(match.capturedStart(matched_text2), match.capturedLength(matched_text2), myClassFormat1);
    }

However, the format is not applied to the matched text

JonB

@lukutis222

match.capturedStart(matched_text2), match.capturedLength(matched_text2)

I believe you are doing totally the wrong thing here. If you want --- as you do --- the second (...) matched the whole thing is given in match.captured(2) (your matched_text2), and that is all you should be looking at.

lukutis222

@JonB

I really appreciate the help but I dont think we are on the same page and I am not sure if you fully understand what I am trying to achieve. I want to give a simpler example perhaps you can help me understand how the highlighter is meant to remove the unwanted text cause I am still not fully convinced this is possible. Imagine I have a text "12345HELLO12345" That I want to send to the console.

    const QByteArray data  = "12345HELLO12345";
    ui->Console->insertPlainText(data);
    syntaxhighlighter = new SyntaxHighlighter(ui->Console->document());

Since I have attached the syntaxhiglihhter to my console, as soon as the data is sent, the higlightblock function will be called.

Now 2 things need to happen inside higlightblock function:

Discard the "12345" before and after the word "HELLO"
Format the text and make it green color.

void SyntaxHighlighter::highlightBlock(const QString &text)
{
QTextCharFormat myClassFormat1;
myClassFormat1.setFontWeight(QFont::Bold);
myClassFormat1.setForeground(QColorConstants::Svg::darkorange);

qDebug(" text inside highlightblock = %s \n",text.toStdString().c_str());

QRegularExpression regex("(12345)(.*)(12345)", QRegularExpression::MultilineOption);
QRegularExpressionMatch match = regex.match(text);
if (match.hasMatch()) {

    QString matched_text0 = match.captured(0);
    QString matched_text1 = match.captured(1);
    QString matched_text2 = match.captured(2);
    qDebug("matched text0 = %s \n",matched_text0.toStdString().c_str());
    qDebug("matched text1 = %s \n",matched_text1.toStdString().c_str());
    qDebug("matched text2 = %s \n",matched_text2.toStdString().c_str());

}

}

The QT logs:

 text inside highlightblock = 12345HELLO12345 
matched text0 = 12345HELLO12345 
matched text1 = 12345 
matched text2 = HELLO

As you can see I have sucesfully mathced the text that I want in matched text2. How can I ensure the "12345" is discarded from the console and how can I format the text? My console shows:

I only want to see text "HELLO" in green color in the console

I hope this very simple example is clear and shows what I am trying to achieve.

SimonSchroeder

@lukutis222 said in How to implement reading serial data with ANSI color codes and printing out to textbox in color:

Now 2 things need to happen inside higlightblock function:

Discard the "12345" before and after the word "HELLO"

Format the text and make it green color.

The QSyntaxHighlight has one job and that is highlighting. Just as I have said before:

The syntax highlighter will help with the highlighting, but I don't think it will remove any characters from what is displayed. I think, your original approach seems valid.

The best suggestion I have is to drop the syntax highlighter because you have to first have to remove the unwanted part of the string before adding it to your ui->Console. But then the syntax highlighter has no information for highlighting. (The syntax highlighter highlights syntax; ANSII escape code are more like markup and cannot be handled by a syntax highlighter.)

I suggest that you first clean up the string (maybe replace the ANSII escape code by HTML) and add them to the text edit with the right format. Using HTML would help to just replace the escape codes and then use addHTML() instead of addPlainText(). The regular expression will still help to capture the right parts of the string and the escape codes.

JonB

@lukutis222 said in How to implement reading serial data with ANSI color codes and printing out to textbox in color:

I dont think we are on the same page and I am not sure if you fully understand what I am trying to achieve

:) I think/hope we are!

"(12345)(.*)(12345)"

You keep talking about removing the segment(s) you do not want. In this case, remove groups 1 & 3, to leave group 2.

I am turning that round. I am saying why can't you implement this by simply preserving what you do want? In this case, don't worry about "removing" anything, instead only output group 2. That is what we normally do with regular expression matches. In order to end up with HELLO why do you want to worry about how to remove the 12345 which comes before and/or after when all you want to end up with is the HELLO?

Certainly that is what I would do from the regular expression above. Isn't QString matched_text2 = match.captured(2); all you want to end up outputting from this input?

BTW, just in case we are not on the same page about one thing: you are aware that you cannot actually remove anything once output has gone to the console, aren't you? You have to not-send-output in the first place if you don't want something to appear.

I'll also say one further thing. Although it is useful to get these regular expressions working it's clearly taking some time for you to get what you want. If all you want to do is the one reg exp you have shown so far, and you will not be expanding that to more complex/varied additional ones, you could have written this in ten minutes by abandoning reg exps and just using a loop to go through the characters in the string omitting/removing/outputting. Recognising \033[ etc. for your couple of cases without reg exps is trivially easy.

lukutis222

@JonB , @SimonSchroeder

Thanks for confirming. When I say I want to remove the unwanted part, I literally meant that I want to discard this from being printed to the console. I was hoping that maybe regex had some hidden feature to discard the unwanted text or replace it with NULL characters or something like that. I kept asking how can I discard the unwanted text but maybe you misunderstood what I meant by discarding. Since you both just confirmed that it is not possible with my current approach, I must select different approach then. I can use highlighter to highlight the text but not replace or discard some characters.

@SimonSchroeder suggested replacing ANSI escape codes by HTML although I have no clue what that means at this point. I will try to research about that and see if that makes sense to me. It is still unclear to me how the data will be sent to the console, I assume the HTML will still show up as some strange symbols on the console?

JonB

@lukutis222
I'm still a little lost in your question! :)

You receive some data from a serial port, right? And you write code to output that to the console (e.g. your ui->Console->insertPlainText(data);), right? So you are choosing what to output. You do not have to copy everything you receive, and then later try to "remove" some bits of it, right? In any case, once something has gone to the console you can't get it back/change it.

Consequently just outputting what you do want to output from the string you showed, rather than trying to delete what you don't want, seems simpler.

I will leave you with a couple of thoughts:

The methods of QRegularExpression allow you to pick out captures. They do not offer to remove something from the input string.
QString does have methods to remove bits of a string and return a new one. QString &QString::replace(const QRegularExpression &re, const QString &after) can be used to find occurences of a reg exp and replace the matching part with something (including an empty string to remove it completely). And it can deal with captures too if you want that. Perhaps you would have been more comfortable starting from QString rather than from QRegularExpression?
Really to remove ANSI escape sequences you shouldn't be bothering to recognise that start-color-then-text-then-end-color pattern, like "(\\033[0;33m)(.*)(\\033[0m)". All you should do is recognise each ANSI escape sequence and remove it, without worrying about pairing it with a closing sequence or what comes in the middle. Now, for the two escape sequence you are at least currently interested they both match e.g.

\\033\[[^m]*m

i.e. escape, open-square, sequence of non-m characters, m character terminator. So you could just use that to remove all matching ANSI sequences you have, it covers both your begin and end ones. Then you don't have to worry about capture groups and stuff between a start and end marker.

lukutis222

@JonB

Yes I receive a bunch of data (alot of data actually) from a serial port. This data will come in 3 different colors (as far as I know).

If text has 0;33m ANSI prefix, I must display the text in orange color
if text has 0;32m ANSI prefix, I must display in green color
if text has 0;34m ANSI prefix, I must display in red color

I sort of understand what you mean, but one thing I am not sure about:

I can do some operation with QString before I send text to the console as you have suggested, I can even remove the ANSI color code and replace it with an empty string, however, how my higlighter is supposed to know how to format the text (apply different colors since I no longer have ASCI code to match and apply format.?

Are you suggesting that I can color different messages without using the higlihter. Some pseudo code:

void Widget::readData()
{
    const QByteArray data = serial->serial_connection.readAll();

// 1 .detect ANSI color code and replace it with empty characters 

// 2. Is it possible to apply different colors here instead of the highlighter?

    ui->Console->insertPlainText(data);



}

JonB

@lukutis222
I am getting a little lost in your questions, and also a touch exhausted :)

You seem to start with the ANSI escape codes, and you have shown output in the appropriate color from that. Then you want to remove those and replace them with something which shows the color, but you already had the color.... ???

I don't know what class ui->Console is so i don't know what you can do with it.

I don't know what you are or are not doing with a QSyntaxHighlighter.

I may be reaching my limit time on this topic... :)

lukutis222

@JonB
What I am trying to do is very simple. I am not sure if I am very bad at explaning but I have shown it with the 12345HELLO12345 example.

I have a string "12345HELLO12345" and I must do 2 things:

DISCARD "12345"
Color the remaining text in green color. So the output to the console should be "HELLO" in green color.

For my "real world" example, replace the "12345" with the ANSI color codes. The end user does not need to see the ANSI color codes in the console but he must see the text in different colors. The remote device might send me a string:

[0;33m HELLO FROM REMOTE DEVICE[0m

I need to do 2 things:

Replace the [0;33m and [0m with empty string
Color the remaining text ("HELLO FROM REMOVE DEVICE") to orange color.

The remote device can send a different string:

[0;32m THIS MESSAGE MUST BE GREEN[0m

I need to do 2 things:

Replace the [0;32m and [0m with empty string
Color the remaining text ("THIS MESSAGE MUST BE GREEN") to green color.

I was initially trying to do it all with the syntaxhighlighter because I thought that you said it is possible to discard (COMPLETELY REMOVE) the unwanted text. But I have just found out that this is not possible so my initial approach was not correct.

Another approach that you have suggested is to discard the unwanted data before it is sent to the console which makes sense, however, the syntax highlighter will have no way to distinguish whether that particular text must be displayed in green, yellow or red color since the ANSI color code has been already discarded before sending the data to the console

JonB

@JonB said in How to implement reading serial data with ANSI color codes and printing out to textbox in color:

I don't know what class ui->Console is so i don't know what you can do with it.

?

lukutis222

@JonB Sorry for the late response. The ui-> console class is QTextEdit

mrjj

@lukutis222
Hi
If you can already find and replace the ANSI codes then using a QTextEdit should be straightforward.
You simply note the code/color and remove it and then insert the "clean" text with the right color.

    auto output = ui->textEdit;
    output->setTextColor(Qt::red); // predefined color
    output->insertPlainText("hello");
    output->setTextColor(QColor(0,255,0)); // 'custom' color
    output->insertPlainText("device");

alt text

lukutis222

I am getting close to what I am trying to achieve and I have learnt a lot of things about QT while trying to achieve this. Currently, I am having issues of removing brackets("[" and "]") from the text that I receive. The messages that I receive from the remote device:

[1B][0;32mI (504) SEC_GATTS_DEMO: ESP_GATTS_REG_EVT[1B][0m

I have learnt how to replace characters before they are sent to the console for example removing the ANSI color code (0;32m)

const QByteArray data = serial->serial_connection.readAll();
QString DataAsString = QString(data);
DataAsString.replace(QRegularExpression("0;32m"), "");
ui->Console->insertPlainText(DataAsString);

I can also remove the ANSI color code using

DataAsString.replace(QRegularExpression("0;32m"), "");

However, there is this additional bracket that I am not certain where it comes from. Notice the [ and ] brackets highlighted with bold.
[1B] [ 0;32mI (504) SEC_GATTS_DEMO: ESP_GATTS_REG_EVT[1B] [ 0m

I would like to know how can I remove or replace it? This bracket octal code is \133 and I tried the following:

DataAsString.replace(QRegularExpression("\033\1330;32m"), "");
DataAsString.replace(QRegularExpression("\033[0;32m"), "");

But both does not seem to work for some reason. I am not sure why I cannot simple replace the brackets

JonB

@lukutis222
[ is a special character to mark the start of list of alternative characters to match till ]. So you need to do something to match a literal [ character.

Try either \\[ or [[] in your C string for that. [EDIT Forum showing \\[ wrong on this line, I typed two \s, it shows two in preview mode but only one in final mode for some reason, grrr, this forum is very inconsistent in showing some literals inside backticks :( ]
Personally I prefer \\033 (allow reg ex to see \033, which it interprets as octal character) over \033 (let C embed an actual ASCII 27 character in the string). Same for \133 -> \\133.

Perhaps play with these at regex101. Don't forget that if you are putting into a literal C string anything which works from there may need \ -> \\.

mpergand

@lukutis222

Unfortunatly what you're trying to do is not so trivial ...

Here a very basic attempt:

const char ESC=0x1b;            // escape char
const char* RESET="\033[0m";    // reset color

// basic 8 colors
const char *colors[]=
{
    "0;30","black",
    "0;31","red",
    "0;32","green",
    "0;33","yellow",
    "0;34","blue",
    "0;35","purple",
    "0;36","cyan",
    "0;37","white",
    nullptr,"black"
};


struct MsgAttr
{
        QString msg;
        QColor color;
};

bool parseMsg(const QByteArray& data, MsgAttr& attr)
{
    int index=0;
    // by default
    attr.color="black";
    attr.msg=data;
    const char* bytes=data.constData();     // use bytes data

    // search start/end of ansi color code
    if( (bytes[0] == ESC) && (bytes[1] =='[' ) )
        {
        index=2;

        while(bytes[index])   // search ending 'm'
            {
            if(bytes[index] == 'm') break;
            index++;
            }

        if(bytes[index] == 0) return false; // malformed msg
        }

    if(index>0)
        {
        QString colorStr=data.mid(2,index-2);
        int colorIndex=0;

        while(colors[colorIndex++])
            {
            if(colorStr == colors[colorIndex-1])
                attr.color=colors[colorIndex];
            }

        QString msg=data.mid(index+1);
        if(msg.endsWith(RESET))  // remove [0m
            msg.chop(strlen(RESET));

        attr.msg=msg;
        }

    return true;
}

// some text messages
   const char *msg[]=
   {
       "\033[0;30mBlack color\033[0m",
       "\033[0;31mRed color\033[0m",
       "\033[0;32mGreen color\033[0m",
       "\033[0;33mYellow color\033[0m",
       "\033[0;34mBlue color\033[0m",
       "\033[0;35mPurple color\033[0m",
       "\033[0;36mCyan color\033[0m",
       "\033[0;37mWhite color\033[0m",
       "no color",                     // no ansi color
       "\033[0;37mWhite color",        // no reset at the end
       "\033[0;37White color",         // malformed
       nullptr

   };

   MsgAttr attr;

   int m=0;
   while(msg[m])
       {
       bool ok=parseMsg(msg[m],attr);
       qDebug()<<"ok="<<ok<<"color="<<attr.color<<attr.msg;
       m++;
       }

console logs:

ok= true color= QColor(ARGB 1, 0, 0, 0) "Black color"
ok= true color= QColor(ARGB 1, 1, 0, 0) "Red color"
ok= true color= QColor(ARGB 1, 0, 0.501961, 0) "Green color"
ok= true color= QColor(ARGB 1, 1, 1, 0) "Yellow color"
ok= true color= QColor(ARGB 1, 0, 0, 1) "Blue color"
ok= true color= QColor(ARGB 1, 0.501961, 0, 0.501961) "Purple color"
ok= true color= QColor(ARGB 1, 0, 1, 1) "Cyan color"
ok= true color= QColor(ARGB 1, 1, 1, 1) "White color"
ok= true color= QColor(ARGB 1, 0, 0, 0) "no color"
ok= true color= QColor(ARGB 1, 1, 1, 1) "White color"
ok= false color= QColor(ARGB 1, 0, 0, 0) "\u001B[0;37White color"

This example supposes ansi colors are in the form 0;3x
but it could be 3x;0 as well.
see here:
https://gist.github.com/JBlond/2fea43a3049b38287e5e9cefc87b2124
http://jafrog.com/2013/11/23/colors-in-terminal.html

As I said, it's not trivial ;)

lukutis222

@JonB

Hello. It has been a while and I have learnt a couple of things since the last time. Since I have changed the logic of incoming data parsing ( using readAll instead of readLine ), the previous solution was no longer valid. I have come up with a simillar solution, It feels like I am very close to achieving what I want but not quite there yet.

My readData function:

    QString error_match = QString("\033\1330;31m");// error
    QString info_match = QString("\033\1330;32m"); // info
    QString warning_match = QString("\033\1330;33m"); //warning



void MainWindow::readData()
{

    // save the scrollbar position
    QScrollBar *scrollbar = ui->Console_read_2->verticalScrollBar();
    bool scrollbarAtBottom  = (scrollbar->value() >= (scrollbar->maximum() - 4));
    int scrollbarPrevValue = scrollbar->value();


    // this moves the cursor to the bottom to avoid writing data in the middle of the console
    QTextCursor cursor = ui->Console_read_2->textCursor();
    cursor.clearSelection();
    cursor.movePosition(QTextCursor::End);
    ui->Console_read_2->setTextCursor(cursor);
    
    QByteArray data = serial_local->serial_connection.readAll(); //read all data
    QString DataAsString = QString(data); // covert  qbytearray to string
    DataAsString.replace("\r", "\n"); // replay ce all \r with \n
    QStringList myStringList = DataAsString.split("\n"); // split all data to multiple lines

    for(int i =0;i < myStringList.length();i++)
    {
        myStringList[i] +=  "\n"; // append new line
        qDebug("myStringList[%i]=%s \n",i,myStringList.at(i).toStdString().c_str());

        //CAPTURE INFO LOGS
        if(myStringList.at(i).contains(info_match)){
            ui->Console_read_2->setTextColor(QColor(0,128,0)); // 'custom' color
            ui->Console_read_2->insertPlainText(myStringList.at(i));
            continue;

        }
         //CAPTURE WARNING LOGS
        else if(myStringList.at(i).contains(warning_match)){
            ui->Console_read_2->setTextColor(QColor(255,165,0)); // 'custom' color
            ui->Console_read_2->insertPlainText(myStringList.at(i));
            continue;

        }
        //CAPTURE ERROR LOGS
        else if(myStringList.at(i).contains(error_match)){
            ui->Console_read_2->setTextColor(QColor(255,0,0)); // 'custom' color
            ui->Console_read_2->insertPlainText(myStringList.at(i));
            continue;

        }
        //All other data with ANSI code is printed in black color
        else{
            ui->Console_read_2->setTextColor(Qt::white); //by default the text will be black
            ui->Console_read_2->insertPlainText(myStringList.at(i));
            continue;

        }

    }


    if (scrollbarAtBottom)
    {
        ui->Console_read_2->ensureCursorVisible();
    }
    else
    {
        ui->Console_read_2->verticalScrollBar()->setValue(scrollbarPrevValue);
    }


}

The external device that I am connected to is sending me bunch of data every 2 seconds

        ESP_LOGI("INFO","this is info log1"); 
        ESP_LOGW("WARNING","this is warning log1");
        ESP_LOGE("ERROR","this is error log1");
        ESP_LOGI("INFO","this is very very very very long info log2");
        ESP_LOGW("WARNING","this is very very very very long warning log2");
        ESP_LOGE("ERROR","this is very very very long error log2");
        ESP_LOGI("INFO","this is info log3");
        ESP_LOGW("WARNING","this is warning log3");
        ESP_LOGE("ERROR","this is error log3");
        printf("This is normal message without ANSI color code \n");
        delay(2000);

My current problems:

Since I use readAll, there is no guarantee that I will complete line. For example. You can see that in my for loop, I am printing each detected line that I have split with \n:

QStringList myStringList = DataAsString.split("\n"); // split all data to multiple lines

    for(int i =0;i < myStringList.length();i++)
    {
        myStringList[i] +=  "\n"; // append new line
        qDebug("myStringList[%i]=%s \n",i,myStringList.at(i).toStdString().c_str());

On the console application, I see the following:

myStringList[0]=[0;32mI (10872) INFO: this is info log1[0m
 
myStringList[1]=
 
myStringList[2]=[0;33mW (10872) W
 
myStringList[0]=ARNING: this is warning log1[0m
 
myStringList[1]=
 
myStringList[2]=[0;31mE (10872) ERROR: this i
 
myStringList[0]=s error log1[0m
 
myStringList[1]=
 
myStringList[2]=[0;32mI (10872) INFO: this is very very very 
 
myStringList[0]=very long info log2[0m
 
myStringList[1]=
 
myStringList[2]=[0;33mW (10872) WARNING: this is very 
 
myStringList[0]=very very very long warning log2[0m
 
myStringList[1]=
 
myStringList[2]=[0;31mE (10882) ERROR: th
 
myStringList[0]=is is very very very long error log2[0m
 
myStringList[1]=
 
myStringList[2]=[0;32mI (10892) INFO:
 
myStringList[0]= this is info log3[0m
 
myStringList[1]=
 
myStringList[2]=[0;33mW (10892) WARNING: this is warnin
 
myStringList[0]=g log3[0m
 
myStringList[1]=
 
myStringList[2]=[0;31mE (10892) ERROR: this is error log3[0m
 
myStringList[3]=
 
myStringList[4]=This
 
myStringList[0]= is normal message without ANSI color code 
 
myStringList[1]=
 
myStringList[2]=
 
myStringList[0]=[0;32mI (12902) INFO: this is info log1[0m
 
myStringList[1]=
 
myStringList[2]=[0;33mW (12902) W

Lets look at first 4 lines very closely:

myStringList[0]=[0;32mI (10872) INFO: this is info log1[0m
 
myStringList[1]=
 
myStringList[2]=[0;33mW (10872) W
 
myStringList[0]=ARNING: this is warning log1[0m

As you can see from above, I split my data into multiple lines but the line 3 (myStringList[2]) is incomplete. Hence the next line after this one will be also incomplete and will not match with the color code sequence due to the begining lost in the previous line.

Another issue is that when I split data using:

QStringList myStringList = DataAsString.split("\n"); // split all data to multiple lines,

I can see that almost every other line in my for loop is empty. Is that expected and this is how split works? I feel like it is inefficient and causes additional for loop cycles to go through all lines.

Is it correct to manually insert new line after every line?

    QStringList myStringList = DataAsString.split("\n"); // split all data to multiple lines
    for(int i =0;i < myStringList.length();i++)
    {
        myStringList[i] +=  "\n"; // append new line

Notice that after I split data into different lines with \n delimiter, I also append \n to the very end of it. If I do not do that, the data will be printed in console without any endlines

According to my understanding, in order to solve my problem I need to know where exactly does the new line starts and I am not sure how can I do that with readAll.

JonB

@lukutis222
It's a long post, so here are couple of observations.

I think you might be the same guy who has been asking about "reading lines" in another thread, since I recognise

QStringList myStringList = DataAsString.split("\n"); // split all data to multiple lines,

I did not know you are also the guy for this thread. Are you?? That person was saying incoming lines could end either in \n or in \r (which worries me). Is that you too, or quite separate?

Since I have changed the logic of incoming data parsing ( using readAll instead of readLine )
Since I use readAll, there is no guarantee that I will complete line.
According to my understanding, in order to solve my problem I need to know where exactly does the new line starts and I am not sure how can I do that with readAll.

Indeed, and you will want to code (correctly) for that. I do not know why you are no longer using readLine(), unless you are the other guy with the \r/\n issue. If you need to use readAll() you need to do a proper job to replace readLine/canReadLine(). And I suspect you are not.

Consider input of, say, "Line 1\nLine 2\n Line". The last line does not have a trailing \n, and hence is not complete. That means you must not process it yet as a complete line, you can process the first 2 lines there but must "buffer" the 3rd line to allow further characters to arrive at a future time.

You are using split("\n") to split your input into a list of lines. This is not good enough here. The problem is it will return 3 lines for the list, but you cannot tell from it that the third line did not end in a \n. If the input had been "Line 1\nLine 2\n Line\n" --- note now the trailing \n at the end making the last line complete --- you would get exactly the same result back from split(). So it's not good (precise) enough for this task.

readLine/canReadLine() would not have used split(). It would not divide all its input into lines, it would only have looked for the first newline in the string and dealt with that. Use e.g. QString::indexOf() or QByteArray::indexOf() to search for the first \n in the input. If there is not one then canReadLine() returns false, and you should not call readLine() yet, see how the docs state:

Note that on sequential devices, data may not be immediately available, which may result in a partial line being returned. By calling the canReadLine() function before reading, you can check whether a complete line (including the newline character) can be read.

If it does have a \n then readLine() can be called. Its job is to read up to and including the first \n and return that segment, while removing that from its currently buffered input, so that only the stuff after that \n remain to be read at a future time.

You need to implement similar. And if you are the person who wants to treat \rs like \ns deal with \r instead of \n as the first end-of-line marker appropriately.

I can see that almost every other line in my for loop is empty. Is that expected and this is how split works?

split() does not insert anything into your input/the resulting lines, and does not produce any empty/blank lines which are not in the input. If you are the person who claims that your input has "lines" ending in \r as well as in \n, and if you are presently replacing all \rs by \ns, I said I was never happy with that. If you actually have \r\n at the end of lines you will end up with \n\n and that will cause an unintended blank line. If you are not that person then:

Is it correct to manually insert new line after every line?
myStringList[i] += "\n"; // append new line

That depends on what you will do with the myStringList[] lines. If you append a \n here then you must not also output them with an explicit \n, or use some kind of "writeLine()", as that will lead to a double newline. But since you say

I also append \n to the very end of it. If I do not do that, the data will be printed in console without any endlines

I don't think you are doing that. So cannot answer where blank lines are coming from, analyze your input carefully.

Summary: Get rid of split("\n"), do it properly with e.g. indexOf() just getting the next line out of your buffer and leaving all further stuff to be read/dealt with later.

lukutis222

@JonB said in How to implement reading serial data with ANSI color codes and printing out to textbox in color:

readLine()

Yes I am the same guy who want to treat \r like \n and that is exactly the reason why I no longer can use readLine/canReadLine(). I thought posting in this existing thread made more sense than the other one for some reason.
From what I understand, the only way to handle incoming serial data with \r \n is to use readAll but that brings a bunch of new problems that I did not have when using readLine/canReadLine().

I will go through your suggestions very closely and see if I can come up with something clever.. Thank you very much !

UPDATE

Regarding the every other line in the loop emtpy

You were totally right regarding \n\n. Some data that I read comes with \r\n hence that results with \n\n..

I am now looking into your suggestions about indexOf()

JonB

@lukutis222 said in How to implement reading serial data with ANSI color codes and printing out to textbox in color:

Yes I am the same guy who want to treat \r like \n and that is exactly the reason why I no longer can use readLine/canReadLine(). I thought posting in this existing thread made more sense than the other one for some reason.

Ah ha! :) Agreed about continuing your question in this thread, just it would have helped if you had referenced that thread in this one now so that I/people would know! They are related because of your situation.

Now you have a real problem :( Per what you have said/"claimed" that your lines could be terminated by any of just \n, just \r or \r\n. Consider receiving "Line 1\r". Is this a complete line ready to be processed, or is it a partial line waiting for a \n to arrive later? The answer is you cannot tell, you do not know whether the next character will or won't be a \n. So you have a problem with this line, which would require careful coding, and would certainly defeat readLine().

Stop what you are coding now! Go back and look carefully at all the bytes you actually receive (without trying to break them into lines, for now). readLine() can deal with just \n at the end of a line. It can also deal with \r\n (we might take some further action on this, but it will be treated as a line). But it cannot deal with your claim that some lines end in only \r. Come back as & when you are sure whether you really have a \r and no following \n to be treated as a "complete line", or whether actually it's always \r\n....