How to implement reading serial data with ANSI color codes and printing out to textbox in color

lukutis222

Hello. For my first project, I have decided to create serial terminal that allows me to connect to serial devices and listen to the communication. I have looked at QT Creator serial terminal example and copied some code from there. I have a readData function that is connected via the signal and slot to &QSerialPort::readyRead.

connect(&serial->serial_connection, &QSerialPort::readyRead, this, &Widget::readData);

void Widget::readData()
{
    const QByteArray data = serial->serial_connection.readAll();
    
    if(strcmp(data,"[1B][0;32m")==0){ // this is not detecting :(
        qDebug("green color detected \n");
    }
    
    ui->Console->insertPlainText(data);
    if(ui->checkBox->isChecked()){
        QScrollBar *bar = ui->Console->verticalScrollBar();
        bar->setValue(bar->maximum());
    }
    else{

    }
   // QScrollBar *bar = ui->Console->verticalScrollBar();
    //bar->setValue(bar->maximum());
}

Once the serial data is received, I insert it to my console. It seems to work fine apart from one issue: I cannot find a way how to insert data to console in different colors. The serial device that I have connected to is printing out logs with ANSI color codes and I would like to represent different logging messages in different colors:

The screenshot above is a snippet from the serial data that I receive from a connected device. The messages have ANSI prefix such as 0;33m or [0;32m.

Could someone point me in the right direction here?

jsulm

@lukutis222 Sounds like https://doc.qt.io/qt-5/qsyntaxhighlighter.html is something you should consider

lukutis222

@jsulm Thank you for pointing me in the right direction. I will try to implement it to my code and let you know how it goes

lukutis222

@jsulm
I have managed to import the SyntaxHighlighter. I can now parse 2 different colors:

void SyntaxHighlighter::highlightBlock(const QString &text)
{
    QTextCharFormat myClassFormat1;
    myClassFormat1.setFontWeight(QFont::Bold);
    myClassFormat1.setForeground(QColorConstants::Svg::darkorange);


    QTextCharFormat myClassFormat2;
    myClassFormat2.setFontWeight(QFont::Bold);
    myClassFormat2.setForeground(QColorConstants::Svg::forestgreen);



    QRegularExpression regex("0;33m(.*)0m", QRegularExpression::MultilineOption);
    QRegularExpressionMatchIterator i = regex.globalMatch(text);

    while (i.hasNext())
    {
      QRegularExpressionMatch match = i.next();
      setFormat(match.capturedStart(), match.capturedLength(), myClassFormat1);
    }



    QRegularExpression regex2("0;32m(.*)0m", QRegularExpression::MultilineOption);
    QRegularExpressionMatchIterator j = regex2.globalMatch(text);

    while (j.hasNext())
    {
      QRegularExpressionMatch match = j.next();
      setFormat(match.capturedStart(), match.capturedLength(), myClassFormat2);
    }

}

I can either print in Orange or Green color. I am not sure if my implementation is correct but that seems to work.

There is one drawback:

[0;33mW (00:00:13.591) THERMOSTAT: CURRENT TIME: Thu Jan  1 00:00:13 1970[0m

The above is snippet from my terminal. As you can see it is printed in orange color.

I would like to exclude the color code [0;33m and termination [0m but I am not sure how can this be done. User does not need to see ANSI codes, just the text.

JonB

@lukutis222
You have to write the necessary regular expression to remove no more and no less than you wish to remove.

For example:

QRegularExpression regex2("0;32m(.*)0m", QRegularExpression::MultilineOption);

Won't the (.*) (at least potentially) match more than a single ANSI escape sequence?

That won't occur though on the example you show, I guess. Make sure you only outputting the match in the parentheses, not the whole thing?

lukutis222

@JonB Yes it is possible that this may match something else but I dont think there is any other way to handle this. At least not that I think of.

If there is a match for "0;33m" it is most likely an ANSI color code and not just a random serial message.

Can you clarify what do you mean Make sure you only outputting the match in the parentheses, not the whole thing?
That is what I am trying to figure out how to do right now. I am trying to exclude that color code from printing to the terminal.

UPDATE

I use the following Regex:

(?<=0;33m).*?(?=0m)

The expression above is looking for a match between ANSI prefix and postfix.

As you can see from the image above, it matched the string successfully, but it does not remove the unnecessary text. I am not sure how can this be done since the function that is sending the data and highlightblock functions are completely seperate.

void Widget::readData()
{
    const QByteArray data = serial->serial_connection.readAll();
    ui->Console->insertPlainText(data);

    if(ui->checkBox->isChecked()){
        QScrollBar *bar = ui->Console->verticalScrollBar();
        bar->setValue(bar->maximum());
    }
    else{

    }

}

I am inserting data to my console regardless of what it is and then it is up to the highlighter to format and remove unnecessary data.

Is it possible for the regex to completely remove the characters outside of match? At the moment, I am simply selecting all the characters between 0;33m and 0m. Additionally, I need to not only select the data, but completely remove the 0;33m and 0m

SimonSchroeder

Have a closer look at ANSI Escape codes: https://en.wikipedia.org/wiki/ANSI_escape_code

Colors are always introduces with an escape character. This character is, in your case, displayed as a left arrow. According to wikipedia this character has the value 0x1B. In your original code example you have written this as "[1B]" to match. However, "[1B]" is for characters and not a single character with the value 0x1B (usually unprintable).

The syntax highlighter will help with the highlighting, but I don't think it will remove any characters from what is displayed. I think, your original approach seems valid. You just have to create a string that starts with 0x1B instead of "[1B]".

JonB

@lukutis222 said in How to implement reading serial data with ANSI color codes and printing out to textbox in color:

Can you clarify what do you mean Make sure you only outputting the match in the parentheses, not the whole thing?
That is what I am trying to figure out how to do right now. I am trying to exclude that color code from printing to the terminal.

Indeed!

I haven't looked into exactly what you are doing/what you need to do, but: you have a pattern like abc(.*)def. What I think you are showing is that matches and what you output is the whole match, e.g. abcXYZdef. But what you want, and should be able to do, is only output the part which matched inside the parentheses each time. Am I right that you are not doing that?

I have not looked at how you accomplish that with globalMatch() and QRegularExpressionMatchIterator. But have a look at match() and QRegularExpressionMatch, see https://doc.qt.io/qt-6/qregularexpressionmatch.html#details. See how that uses QRegularExpressionMatch::captured(1) (not 0!) to pick out the first parentheses. That is what you are wanting to do. There must/ought be some way to achieve that within each global match?

While I think of it: there may be a better/more efficient way, but if all else fails: you get back a string of one "whole" match in the global iteration. If you now push just that through the same regular expression a second time but this time using match() you should be able to get just the parenthesized segment per the example above.

P.S.
Having said that. I see for globalMatch()/QRegularExpressionMatchIterator that https://doc.qt.io/qt-6/qregularexpressionmatchiterator.html#details says:

Each result is a QRegularExpressionMatch object holding all the information for that result (including captured substrings).

So I think it should already have accessible information about the sub-parenthesized capture groups. Ah, I think I see. In your code you use only match.capturedStart(), match.capturedLength(). That only accesses the whole match. You should be using match.captured(1) (or some number other than 0/omitted), then you will be good!

lukutis222

@SimonSchroeder
Thanks for the response. I am aware of that 1B code in front of the message I just was not sure what to do with it. I do not have control over what messages the connected serial device is sending. This is simply the format ESP32 (microcontroller) is writing serial data.
Are you suggesting that I should incorporate 1B in my regex?
At the moment I have:

(?<=0;33m).*?(?=0m)

lukutis222

@JonB
Hello. Thank you for this information. I have been reading and learning about this for a while but unfortunately I still cannot get this to work.

I do not understand how match() and match.captured work. I have put the example string and regex from the QT docs to the regex parser online:

QRegularExpression re("(\\d\\d) (?<name>\\w+)");
QRegularExpressionMatch match = re.match("23 Jordan");
if (match.hasMatch()) {
    QString number = match.captured(1); // first == "23"
    QString name = match.captured("name"); // name == "Jordan"
}

As you can see from above image, the regex does not seem to match anything. I am now studying more about match instead of global match and trying to find out how it works.

Just for the experimentation sake I have tried the following:

    QRegularExpression regex("(?<=0;33m).*?(?=0m)", QRegularExpression::MultilineOption);
    QRegularExpressionMatch match = regex.match(text);
    if (match.hasMatch()) {
        QString matched_text = match.captured(1); 
        qDebug("matched text = %s \n",matched_text.toStdString().c_str());
        
    }

The if condition is triggered so it finds a match but when trying to print it, it is printing "nothing"

UPDATE

I have tried the following:

    QRegularExpression regex("(?<=0;33m).*?(?=0m)", QRegularExpression::MultilineOption);
    QRegularExpressionMatch match = regex.match(text);
    if (match.hasMatch()) {

        QString matched_text0 = match.captured(0);
        QString matched_text1 = match.captured(1);
        QString matched_text2 = match.captured(2);
        qDebug("matched text0 = %s \n",matched_text0.toStdString().c_str());
        qDebug("matched text1 = %s \n",matched_text1.toStdString().c_str());
        qDebug("matched text2 = %s \n",matched_text2.toStdString().c_str());

    }

I can now succesfully print the matched text which is caputed using match.captured(0).

matched text0 = W (00:03:08.580) THERMOSTAT: CURRENT TIME: Thu Jan  1 00:03:08 1970[ 
matched text1 =  
matched text2 =

Is the above wrong since you suggested not using captured(0)?

I still cannot fully understand how can I get rid of the unmatched text. Since I write this data to my console using the following function

void Widget::readData()
{
    const QByteArray data = serial->serial_connection.readAll();
    ui->Console->insertPlainText(data);

    if(ui->checkBox->isChecked()){
        QScrollBar *bar = ui->Console->verticalScrollBar();
        bar->setValue(bar->maximum());
    }
    else{

    }

}

So regardless if the highlighter formats the data or not, the data is still there

SimonSchroeder

@lukutis222 said in How to implement reading serial data with ANSI color codes and printing out to textbox in color:

Are you suggesting that I should incorporate 1B in my regex?

If you want to remove that character as well (which I assume you want to), then yes! I am not entirely sure about the full syntax that QRegularExpression supports, but it seems like you could include this character by its octal value as "\033". This would mean that your regular expression should be "(?<=\033[0;33m).*?(?=\033[0m)".

Here is the explanation for your last successful try: Every match has the full match as captured(0) (which in general might be empty if there is no match). If you want to have additional matches you need to use parentheses "(\w) (\w)" would match two words where captured(0) has the whole captured string with two words and a space in between, captured(1) has the first word, and captured(2) has the second word. However, the question mark in (?...) tells the regular expression that the part inside the parentheses is not a capture. This is the reason why in your case there is only one captured string (have a look at captureCount() which should be 0 (0 means that there is only the implicitly captured group).

JonB

@lukutis222
I do not claim to follow the full ins and outs of your long post. But I believe @SimonSchroeder is saying what I would: it is your use of the "advanced" (?... stuff that is probably at issue.

Please start your testing and grabbing of (...) segments on a much simpler reg ex. Just for example:

"(\\033[0;33m)(.*)(\\033[0m)"

(I'm not 100% about \\033 vs \033, you may have to play, I believe either/both will work, for different reasons.) That should have captures 1--3. Get that principle working before you move onto a more complex expression!

lukutis222

@SimonSchroeder

Thanks for help. Maybe you know how can I simulate this on regex tester online. That would allow me to understand how your suggested expression is intended to work:

"(?<=\033[0;33m).*?(?=\033[0m)"

If I put it in my code:

    QRegularExpression regex("(?<=\033[0;33m).*?(?=\033[0m)", QRegularExpression::MultilineOption);
    QRegularExpressionMatchIterator i = regex.globalMatch(text);

    while (i.hasNext())
    {
      QRegularExpressionMatch match = i.next();
      setFormat(match.capturedStart(), match.capturedLength(), myClassFormat1);
    }

The error message is returned:

QRegularExpressionPrivate::doMatch(): called on an invalid QRegularExpression object

Trying this on https://regex101.com/ or https://www.regextester.com/ does not seem to work. Is that because regex testers online do not accept octal representations?
String under testing:

[1B][0;33mW (00:02:12.590) THERMOSTAT: CURRENT TIME: Thu Jan  1 00:02:12 1970[1B][0m

Regex used:

(?<=\033[0;33m).*?(?=\033[0m)

Result(no match):

JonB

@lukutis222
It has red-underlined (?<=. Did you hover that to see if it says anything? You are using Javascript reg exes, have you looked to see whether that supports this construct? regex101 (or whatever online) reg exps won't be identical to Qt's, that's also why it offers different "flavors" of reg ex to try.

I previously suggested you try a simpler example which does not use that construct. Why bother with it all when you can just use a plain (...) anyway, and write your code accordingly? I think you are making your first attempt harder than it need be. Up to you....

lukutis222

@JonB When I decided to parse ANSI color codes I really did not expect this to be such complex topic.

As you have suggested, I started experimenting with different regex expressions. The one that I am currently working with is the following:

(0;33m)(.*)(0m)

My code:

    QRegularExpression regex("(0;33m)(.*)(0m)", QRegularExpression::MultilineOption);
    QRegularExpressionMatch match = regex.match(text);
    if (match.hasMatch()) {

        QString matched_text0 = match.captured(0);
        QString matched_text1 = match.captured(1);
        QString matched_text2 = match.captured(2);
        qDebug("matched text0 = %s \n",matched_text0.toStdString().c_str());
        qDebug("matched text1 = %s \n",matched_text1.toStdString().c_str());
        qDebug("matched text2 = %s \n",matched_text2.toStdString().c_str());
    }

Is now returning:

matched text0 = 0;33mW (00:30:02.590) THERMOSTAT: CURRENT TIME: Thu Jan  1 00:30:02 1970[0m 
matched text1 = 0;33m 
matched text2 = W (00:30:02.590) THERMOSTAT: CURRENT TIME: Thu Jan  1 00:30:02 1970[

Which I think is getting closer to what I want to achieve. matched text2 is a text that I want to highlight.

I am still not fully sure about how can I now ensure that the unmatched data does not get displayed on my console?

UPDATE

Do you know how can I format only the specific match that I choose instead of whole string? When I was using global match, I used capturedStart() but I dont think I can use that now.

For example, I want to format matched_text2 only. I tried the following:

    QRegularExpression regex("(0;33m)(.*)(0m)", QRegularExpression::MultilineOption);
    QRegularExpressionMatch match = regex.match(text);
    if (match.hasMatch()) {

        QString matched_text0 = match.captured(0);
        QString matched_text1 = match.captured(1);
        QString matched_text2 = match.captured(2);
        qDebug("matched text0 = %s \n",matched_text0.toStdString().c_str());
        qDebug("matched text1 = %s \n",matched_text1.toStdString().c_str());
        qDebug("matched text2 = %s \n",matched_text2.toStdString().c_str());
        setFormat(match.capturedStart(matched_text2), match.capturedLength(matched_text2), myClassFormat1);
    }

However, the format is not applied to the matched text

JonB

@lukutis222

match.capturedStart(matched_text2), match.capturedLength(matched_text2)

I believe you are doing totally the wrong thing here. If you want --- as you do --- the second (...) matched the whole thing is given in match.captured(2) (your matched_text2), and that is all you should be looking at.

lukutis222

@JonB

I really appreciate the help but I dont think we are on the same page and I am not sure if you fully understand what I am trying to achieve. I want to give a simpler example perhaps you can help me understand how the highlighter is meant to remove the unwanted text cause I am still not fully convinced this is possible. Imagine I have a text "12345HELLO12345" That I want to send to the console.

    const QByteArray data  = "12345HELLO12345";
    ui->Console->insertPlainText(data);
    syntaxhighlighter = new SyntaxHighlighter(ui->Console->document());

Since I have attached the syntaxhiglihhter to my console, as soon as the data is sent, the higlightblock function will be called.

Now 2 things need to happen inside higlightblock function:

Discard the "12345" before and after the word "HELLO"
Format the text and make it green color.

void SyntaxHighlighter::highlightBlock(const QString &text)
{
QTextCharFormat myClassFormat1;
myClassFormat1.setFontWeight(QFont::Bold);
myClassFormat1.setForeground(QColorConstants::Svg::darkorange);

qDebug(" text inside highlightblock = %s \n",text.toStdString().c_str());

QRegularExpression regex("(12345)(.*)(12345)", QRegularExpression::MultilineOption);
QRegularExpressionMatch match = regex.match(text);
if (match.hasMatch()) {

    QString matched_text0 = match.captured(0);
    QString matched_text1 = match.captured(1);
    QString matched_text2 = match.captured(2);
    qDebug("matched text0 = %s \n",matched_text0.toStdString().c_str());
    qDebug("matched text1 = %s \n",matched_text1.toStdString().c_str());
    qDebug("matched text2 = %s \n",matched_text2.toStdString().c_str());

}

}

The QT logs:

 text inside highlightblock = 12345HELLO12345 
matched text0 = 12345HELLO12345 
matched text1 = 12345 
matched text2 = HELLO

As you can see I have sucesfully mathced the text that I want in matched text2. How can I ensure the "12345" is discarded from the console and how can I format the text? My console shows:

I only want to see text "HELLO" in green color in the console

I hope this very simple example is clear and shows what I am trying to achieve.

SimonSchroeder

@lukutis222 said in How to implement reading serial data with ANSI color codes and printing out to textbox in color:

Now 2 things need to happen inside higlightblock function:

Discard the "12345" before and after the word "HELLO"

Format the text and make it green color.

The QSyntaxHighlight has one job and that is highlighting. Just as I have said before:

The syntax highlighter will help with the highlighting, but I don't think it will remove any characters from what is displayed. I think, your original approach seems valid.

The best suggestion I have is to drop the syntax highlighter because you have to first have to remove the unwanted part of the string before adding it to your ui->Console. But then the syntax highlighter has no information for highlighting. (The syntax highlighter highlights syntax; ANSII escape code are more like markup and cannot be handled by a syntax highlighter.)

I suggest that you first clean up the string (maybe replace the ANSII escape code by HTML) and add them to the text edit with the right format. Using HTML would help to just replace the escape codes and then use addHTML() instead of addPlainText(). The regular expression will still help to capture the right parts of the string and the escape codes.

JonB

@lukutis222 said in How to implement reading serial data with ANSI color codes and printing out to textbox in color:

I dont think we are on the same page and I am not sure if you fully understand what I am trying to achieve

:) I think/hope we are!

"(12345)(.*)(12345)"

You keep talking about removing the segment(s) you do not want. In this case, remove groups 1 & 3, to leave group 2.

I am turning that round. I am saying why can't you implement this by simply preserving what you do want? In this case, don't worry about "removing" anything, instead only output group 2. That is what we normally do with regular expression matches. In order to end up with HELLO why do you want to worry about how to remove the 12345 which comes before and/or after when all you want to end up with is the HELLO?

Certainly that is what I would do from the regular expression above. Isn't QString matched_text2 = match.captured(2); all you want to end up outputting from this input?

BTW, just in case we are not on the same page about one thing: you are aware that you cannot actually remove anything once output has gone to the console, aren't you? You have to not-send-output in the first place if you don't want something to appear.

I'll also say one further thing. Although it is useful to get these regular expressions working it's clearly taking some time for you to get what you want. If all you want to do is the one reg exp you have shown so far, and you will not be expanding that to more complex/varied additional ones, you could have written this in ten minutes by abandoning reg exps and just using a loop to go through the characters in the string omitting/removing/outputting. Recognising \033[ etc. for your couple of cases without reg exps is trivially easy.

lukutis222

@JonB , @SimonSchroeder

Thanks for confirming. When I say I want to remove the unwanted part, I literally meant that I want to discard this from being printed to the console. I was hoping that maybe regex had some hidden feature to discard the unwanted text or replace it with NULL characters or something like that. I kept asking how can I discard the unwanted text but maybe you misunderstood what I meant by discarding. Since you both just confirmed that it is not possible with my current approach, I must select different approach then. I can use highlighter to highlight the text but not replace or discard some characters.

@SimonSchroeder suggested replacing ANSI escape codes by HTML although I have no clue what that means at this point. I will try to research about that and see if that makes sense to me. It is still unclear to me how the data will be sent to the console, I assume the HTML will still show up as some strange symbols on the console?