How to implement reading serial data with ANSI color codes and printing out to textbox in color

lukutis222

@JonB said in How to implement reading serial data with ANSI color codes and printing out to textbox in color:

readLine()

Yes I am the same guy who want to treat \r like \n and that is exactly the reason why I no longer can use readLine/canReadLine(). I thought posting in this existing thread made more sense than the other one for some reason.
From what I understand, the only way to handle incoming serial data with \r \n is to use readAll but that brings a bunch of new problems that I did not have when using readLine/canReadLine().

I will go through your suggestions very closely and see if I can come up with something clever.. Thank you very much !

UPDATE

Regarding the every other line in the loop emtpy

You were totally right regarding \n\n. Some data that I read comes with \r\n hence that results with \n\n..

I am now looking into your suggestions about indexOf()

JonB

@lukutis222 said in How to implement reading serial data with ANSI color codes and printing out to textbox in color:

Yes I am the same guy who want to treat \r like \n and that is exactly the reason why I no longer can use readLine/canReadLine(). I thought posting in this existing thread made more sense than the other one for some reason.

Ah ha! :) Agreed about continuing your question in this thread, just it would have helped if you had referenced that thread in this one now so that I/people would know! They are related because of your situation.

Now you have a real problem :( Per what you have said/"claimed" that your lines could be terminated by any of just \n, just \r or \r\n. Consider receiving "Line 1\r". Is this a complete line ready to be processed, or is it a partial line waiting for a \n to arrive later? The answer is you cannot tell, you do not know whether the next character will or won't be a \n. So you have a problem with this line, which would require careful coding, and would certainly defeat readLine().

Stop what you are coding now! Go back and look carefully at all the bytes you actually receive (without trying to break them into lines, for now). readLine() can deal with just \n at the end of a line. It can also deal with \r\n (we might take some further action on this, but it will be treated as a line). But it cannot deal with your claim that some lines end in only \r. Come back as & when you are sure whether you really have a \r and no following \n to be treated as a "complete line", or whether actually it's always \r\n....

lukutis222

@JonB

Hello. The reason why I want to be able to parse \n, \r and the combination of the two because I want to make my terminal as bulletproof and as much universal as possible.

I have used many terminals such as PuTTy, Termite, MobaXterm and every terminal had disatvantages, for example:

Termite - not supporting color codes. The controllers that I program always printing in color so its very convenient for me to see different logs in different colors.
PuTTy - supports color codes but is only for reading the data. It does not allow to write data to external device
MobaXterm - Supports color codes and seems to work very nicely, however, I didint find a way to write serial data to external device and only reading...

Hence for my first QT project I have decided to create this terminal because I thought it would be good C++ experience and I also I want this terminal to rule them all!

When I am programming my own device, I can sure use only \n. But lets assume some other guy connects its weird device with my terminal and his device only prints with \r.

JonB

@lukutis222
Then to cater for "just \n, just \r or \r\n" you will have to code appropriately, you can't use readLine() and you will have to do the necessary coding yourself (remember what I said about the "ambiguity" of the input read into the buffer ending in just \r). That is the answer.

lukutis222

@JonB

I was also not aware that readAll() automatically inserts \r in between different lines.

My remote device is sending the following data:

        printf("This is normal message1 without ANSI color code \n");
        printf("This is normal message2 without ANSI color code \n");
        printf("This is normal message3 without ANSI color code \n");
        printf("This is normal message4 without ANSI color code \n");
        printf("This is normal message5 without ANSI color code \n");
        printf("This is normal message6 without ANSI color code \n");
        printf("This is normal message7 without ANSI color code \n");
        printf("This is normal message8 without ANSI color code \n");
delay(2000);

As you can see from above, I use only \n termination.

In my readData() funciton I do the following(Notice that I do not replace \r with \n at the moment)

    QByteArray data = serial_local->serial_connection.readAll(); //read all data
    QString DataAsString = QString(data); // covert  qbytearray to string
    uint16_t received_data_length = DataAsString.length();
    //DataAsString.replace("\r", "\n"); // replay ce all \r with \n
    qDebug("****FULL DATA***** = %s \n",DataAsString.toStdString().c_str());
    qDebug("*****FULL DATA length***** = %u \n",received_data_length);
    //print all raw bytes
    for(int i = 0; i <received_data_length;i++){
        qDebug("%u\n",DataAsString[i]);
    }

The application output:

****FULL DATA***** = This is normal message1 without ANSI color code 
This is normal 
*****FULL DATA length***** = 64 
84
104
105
115
32
105
115
32
110
111
114
109
97
108
32
109
101
115
115
97
103
101
49
32
119
105
116
104
111
117
116
32
65
78
83
73
32
99
111
108
111
114
32
99
111
100
101
32
13
10
84
104
105
115
32
105
115
32
110
111
114
109
97
108

Notice that there is decimal 13 and 10 (carriage return and new line). I am trying to understand how did carriage return appear here if I do not send it.

This is confusing because I do not know if I should do indexOf('\n') or indexOf('\r'). If readAll automatically inserts \r between different lines it would make more sense to detect a complete line with \r

Christian Ehrlicher

@lukutis222 said in How to implement reading serial data with ANSI color codes and printing out to textbox in color:

I was also not aware that readAll() automatically inserts \r in between different lines.

readAll() does not modify any data read from the io device.

lukutis222

@Christian-Ehrlicher
I cannot wrap my head around how does \r appear in the serial data then... The remote device that I am connected to only sends \n

Christian Ehrlicher

@lukutis222 said in How to implement reading serial data with ANSI color codes and printing out to textbox in color:

I cannot wrap my head around how does \r appear in the serial data then...

Please output the QByteArray data, not the one converted to QString (for whatever reason binary data should be converted to a QString though).

lukutis222

@Christian-Ehrlicher


    QByteArray data = serial_local->serial_connection.readAll(); //read all data
    QString DataAsString = QString(data); // covert  qbytearray to string
    uint16_t received_data_length = DataAsString.length();

    for(int i = 0; i <received_data_length;i++){
        qDebug("%u\n",DataAsString[i]);
    }
    qDebug("****RAW DATA***** \n");
    for(int i = 0; i <data.length();i++){
        qDebug("%u\n",data[i]);
    }

Both identical.
Does that mean that my remote device "secretly" slips in another \r without me even wanting ?

Christian Ehrlicher

@lukutis222 said in How to implement reading serial data with ANSI color codes and printing out to textbox in color:

Does that mean that my remote device "secretly" slips in another \r without me even wanting ?

Yes

lukutis222

I think I managed to achieve what I want (didint test 100% but I believe I am getting very close). I want to share the current logic, perhaps you can advise on how to improve it.

void MainWindow::readData()
{
    // save the scrollbar position
    QScrollBar *scrollbar = ui->Console_read_2->verticalScrollBar();
    bool scrollbarAtBottom  = (scrollbar->value() >= (scrollbar->maximum() - 4));
    int scrollbarPrevValue = scrollbar->value();

    // this moves the cursor to the bottom to avoid writing data in the middle of the console
    QTextCursor cursor = ui->Console_read_2->textCursor();
    cursor.clearSelection();
    cursor.movePosition(QTextCursor::End);
    ui->Console_read_2->setTextCursor(cursor);

    QByteArray data = serial_local->serial_connection.readAll(); //read all data
    QString DataAsString = QString(data); // covert  qbytearray to string
    static QString incomplete_line = nullptr; // hold information about the last incomplete line
    uint16_t beggining_pointer = 0; // save the beggining pointer
    
    DataAsString.replace("\r", "\n"); // replace all \r with \n
    DataAsString.replace("\n\n", "\n"); // this ensures that all double \n\n (if exist) will be replaces with just a single \n

    int index = DataAsString.indexOf('\n');
    
    while (index != -1) {
        QString complete_line = DataAsString.mid(beggining_pointer, index-(beggining_pointer)); // index is where the \n was found in the string.
        complete_line = incomplete_line+complete_line; // append beggining of incomplete line to complete line (initially incomplete line will be empty)
        incomplete_line = nullptr; // reset the incomplete line after each append because its no longer relevant
        Format_and_insert(complete_line); // insert data to console
        beggining_pointer = index+1; // increment beggining pointer to the location where the previous \n was found because this will now be our beggining pointer
        index = DataAsString.indexOf('\n', index+1);

    }

     incomplete_line = DataAsString.mid(beggining_pointer, DataAsString.length()); // after going through every line, take the last beggining pointer and read till the end of string.

    if (scrollbarAtBottom)
    {
        ui->Console_read_2->ensureCursorVisible();
    }
    else
    {
        ui->Console_read_2->verticalScrollBar()->setValue(scrollbarPrevValue);
    }


}

I am reading line by line and when I find an uncomplete line, I save it in the incomplete_line buffer and use it on the next iteration.

Christian Ehrlicher

And now you're again simulating readLine()/canReadLine()...

JonB

@Christian-Ehrlicher
Which is what OP is wanting to do. His problem is that he wishes to treat \r, without requiring a following \n, as a line end. I have verified that Qt code for all of canReadLine/readLine/readLineData() have \n hard-coded in their logic. It is not possible to use these existing ones if you want to treat a bare \r as an acceptable line-ender. (In fact, he wishes to treat any/all of \n, \r or \r\n as line terminators, which complicates things.) While the proposed code above is "rough and ready", and has some flaws, and I personally was not prepared to write the full code of what he wants, if he is happy with this "approximation" it would seem that it would give him what he wants.

Of course, if he did not insist on accepting \r as a line terminator it would be a whole lot simpler....

Christian Ehrlicher

@JonB Overread this.

Still the code looks ugly and over-complicated

void parseBuffer()
{
  auto findNextSeparator = [](const QByteArray &ba, int startOfs) {
    for (int i = startOfs; i < ba.size(); ++i) {
      auto c = ba[i];
      if (c == '\n' || c == '\r') {
        return i;
      }
    }
    return -1;
  };
  auto fromIdx = 0;
  idx = findNextSeparator(m_buffer, fromIdx );
  while (idx >= 0) {
    auto line = m_buffer.mid(fromIdx , idx - fromIdx);
    if (!line.isEmpty())
      emit newLineReceived(line);
    fromIdx = idx + 1;
  }
  m_buffer = m_buffer.mid(fromIdx);
}

Did not check if '\r' or '\n' is in the 'line' buffer - maybe this needs a +/-1 adjustment.

JonB

@Christian-Ehrlicher
Please do not take this wrong, but your code is a touch simplistic. I don't blame you, as I was not prepared to write a comprehensive implementation!

Per the user's requirements, for example it fails to distinguish between incoming \r\n --- one line --- versus \n\n --- a line followed by a genuine blank line. It also does not deal with end-of-file at the end of a final line which has no \r or \n. It does not tell the OP when he needs to call readAll() to fetch more characters for the buffer, and address what to do if that returns eof. It does not deal well with the (unlikely but possible?) case where a readAll() returns a line ending in \r because that is all that has been received so far, when in fact the next character received in the future will be a \n after the \r. I am not saying his code deals with all of these cases either!

All of which is what I thought about when deciding whether I wished to offer the full, robust code, and decided not to... :)

Nonetheless, perhaps the OP will adapt your code to whatever situations he wants to cover.

Christian Ehrlicher

I did not wrote the readAll() stuff because it does not belong to the parsing - it's a separate thing which does basically

onReadyRead()
{
  m_buffer += m_socket->readAll();
  parseBuffer();
}

So the only thing left is the double empty line.

JonB

@Christian-Ehrlicher
Point taken. I was looking at the existing implementation of (woboq) readLineData(), which does a mixture of actual read()s and "parsing" (looking for \n), and how that interacts with readLine() & canReadLine(), and got a bit caught up in that.

@lukutis222
If it's all a bit complex probably best ignore my discussion with @Christian-Ehrlicher and think about whether you want to change your code over to his suggested approach.

Christian Ehrlicher

@JonB said in How to implement reading serial data with ANSI color codes and printing out to textbox in color:

I was looking at the existing implementation of (woboq) readLineData(), which does a mixture of actual read()s and "parsing" (looking for \n), and how that interacts with readLine() & canReadLine(), and got a bit caught up in that.

My approach needs some miore CPU time but is easier to understand :)

lukutis222

@JonB @Christian-Ehrlicher
Your discussion is very helpful :) Thank you very much