Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Get Qt Extensions
  • Unsolved
Collapse
Brand Logo
  1. Home
  2. Qt Development
  3. General and Desktop
  4. How to implement reading serial data with ANSI color codes and printing out to textbox in color
Forum Updated to NodeBB v4.3 + New Features

How to implement reading serial data with ANSI color codes and printing out to textbox in color

Scheduled Pinned Locked Moved Unsolved General and Desktop
51 Posts 7 Posters 7.8k Views 3 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • L Offline
    L Offline
    lukutis222
    wrote on last edited by lukutis222
    #28

    I am getting close to what I am trying to achieve and I have learnt a lot of things about QT while trying to achieve this. Currently, I am having issues of removing brackets("[" and "]") from the text that I receive. The messages that I receive from the remote device:

    [1B][0;32mI (504) SEC_GATTS_DEMO: ESP_GATTS_REG_EVT[1B][0m
    

    I have learnt how to replace characters before they are sent to the console for example removing the ANSI color code (0;32m)

    const QByteArray data = serial->serial_connection.readAll();
    QString DataAsString = QString(data);
    DataAsString.replace(QRegularExpression("0;32m"), "");
    ui->Console->insertPlainText(DataAsString);
    

    I can also remove the ANSI color code using

    DataAsString.replace(QRegularExpression("0;32m"), "");
    

    However, there is this additional bracket that I am not certain where it comes from. Notice the [ and ] brackets highlighted with bold.
    [1B] [ 0;32mI (504) SEC_GATTS_DEMO: ESP_GATTS_REG_EVT[1B] [ 0m

    I would like to know how can I remove or replace it? This bracket octal code is \133 and I tried the following:

    DataAsString.replace(QRegularExpression("\033\1330;32m"), "");
    DataAsString.replace(QRegularExpression("\033[0;32m"), "");
    

    But both does not seem to work for some reason. I am not sure why I cannot simple replace the brackets

    JonBJ M 2 Replies Last reply
    0
    • L lukutis222

      I am getting close to what I am trying to achieve and I have learnt a lot of things about QT while trying to achieve this. Currently, I am having issues of removing brackets("[" and "]") from the text that I receive. The messages that I receive from the remote device:

      [1B][0;32mI (504) SEC_GATTS_DEMO: ESP_GATTS_REG_EVT[1B][0m
      

      I have learnt how to replace characters before they are sent to the console for example removing the ANSI color code (0;32m)

      const QByteArray data = serial->serial_connection.readAll();
      QString DataAsString = QString(data);
      DataAsString.replace(QRegularExpression("0;32m"), "");
      ui->Console->insertPlainText(DataAsString);
      

      I can also remove the ANSI color code using

      DataAsString.replace(QRegularExpression("0;32m"), "");
      

      However, there is this additional bracket that I am not certain where it comes from. Notice the [ and ] brackets highlighted with bold.
      [1B] [ 0;32mI (504) SEC_GATTS_DEMO: ESP_GATTS_REG_EVT[1B] [ 0m

      I would like to know how can I remove or replace it? This bracket octal code is \133 and I tried the following:

      DataAsString.replace(QRegularExpression("\033\1330;32m"), "");
      DataAsString.replace(QRegularExpression("\033[0;32m"), "");
      

      But both does not seem to work for some reason. I am not sure why I cannot simple replace the brackets

      JonBJ Offline
      JonBJ Offline
      JonB
      wrote on last edited by JonB
      #29

      @lukutis222
      [ is a special character to mark the start of list of alternative characters to match till ]. So you need to do something to match a literal [ character.

      • Try either \\[ or [[] in your C string for that. [EDIT Forum showing \\[ wrong on this line, I typed two \s, it shows two in preview mode but only one in final mode for some reason, grrr, this forum is very inconsistent in showing some literals inside backticks :( ]
      • Personally I prefer \\033 (allow reg ex to see \033, which it interprets as octal character) over \033 (let C embed an actual ASCII 27 character in the string). Same for \133 -> \\133.

      Perhaps play with these at regex101. Don't forget that if you are putting into a literal C string anything which works from there may need \ -> \\.

      L 1 Reply Last reply
      1
      • L lukutis222

        I am getting close to what I am trying to achieve and I have learnt a lot of things about QT while trying to achieve this. Currently, I am having issues of removing brackets("[" and "]") from the text that I receive. The messages that I receive from the remote device:

        [1B][0;32mI (504) SEC_GATTS_DEMO: ESP_GATTS_REG_EVT[1B][0m
        

        I have learnt how to replace characters before they are sent to the console for example removing the ANSI color code (0;32m)

        const QByteArray data = serial->serial_connection.readAll();
        QString DataAsString = QString(data);
        DataAsString.replace(QRegularExpression("0;32m"), "");
        ui->Console->insertPlainText(DataAsString);
        

        I can also remove the ANSI color code using

        DataAsString.replace(QRegularExpression("0;32m"), "");
        

        However, there is this additional bracket that I am not certain where it comes from. Notice the [ and ] brackets highlighted with bold.
        [1B] [ 0;32mI (504) SEC_GATTS_DEMO: ESP_GATTS_REG_EVT[1B] [ 0m

        I would like to know how can I remove or replace it? This bracket octal code is \133 and I tried the following:

        DataAsString.replace(QRegularExpression("\033\1330;32m"), "");
        DataAsString.replace(QRegularExpression("\033[0;32m"), "");
        

        But both does not seem to work for some reason. I am not sure why I cannot simple replace the brackets

        M Offline
        M Offline
        mpergand
        wrote on last edited by mpergand
        #30

        @lukutis222

        Unfortunatly what you're trying to do is not so trivial ...

        Here a very basic attempt:

        const char ESC=0x1b;            // escape char
        const char* RESET="\033[0m";    // reset color
        
        // basic 8 colors
        const char *colors[]=
        {
            "0;30","black",
            "0;31","red",
            "0;32","green",
            "0;33","yellow",
            "0;34","blue",
            "0;35","purple",
            "0;36","cyan",
            "0;37","white",
            nullptr,"black"
        };
        
        
        struct MsgAttr
        {
                QString msg;
                QColor color;
        };
        
        bool parseMsg(const QByteArray& data, MsgAttr& attr)
        {
            int index=0;
            // by default
            attr.color="black";
            attr.msg=data;
            const char* bytes=data.constData();     // use bytes data
        
            // search start/end of ansi color code
            if( (bytes[0] == ESC) && (bytes[1] =='[' ) )
                {
                index=2;
        
                while(bytes[index])   // search ending 'm'
                    {
                    if(bytes[index] == 'm') break;
                    index++;
                    }
        
                if(bytes[index] == 0) return false; // malformed msg
                }
        
            if(index>0)
                {
                QString colorStr=data.mid(2,index-2);
                int colorIndex=0;
        
                while(colors[colorIndex++])
                    {
                    if(colorStr == colors[colorIndex-1])
                        attr.color=colors[colorIndex];
                    }
        
                QString msg=data.mid(index+1);
                if(msg.endsWith(RESET))  // remove [0m
                    msg.chop(strlen(RESET));
        
                attr.msg=msg;
                }
        
            return true;
        }
        
        // some text messages
           const char *msg[]=
           {
               "\033[0;30mBlack color\033[0m",
               "\033[0;31mRed color\033[0m",
               "\033[0;32mGreen color\033[0m",
               "\033[0;33mYellow color\033[0m",
               "\033[0;34mBlue color\033[0m",
               "\033[0;35mPurple color\033[0m",
               "\033[0;36mCyan color\033[0m",
               "\033[0;37mWhite color\033[0m",
               "no color",                     // no ansi color
               "\033[0;37mWhite color",        // no reset at the end
               "\033[0;37White color",         // malformed
               nullptr
        
           };
        
           MsgAttr attr;
        
           int m=0;
           while(msg[m])
               {
               bool ok=parseMsg(msg[m],attr);
               qDebug()<<"ok="<<ok<<"color="<<attr.color<<attr.msg;
               m++;
               }
        

        console logs:

        ok= true color= QColor(ARGB 1, 0, 0, 0) "Black color"
        ok= true color= QColor(ARGB 1, 1, 0, 0) "Red color"
        ok= true color= QColor(ARGB 1, 0, 0.501961, 0) "Green color"
        ok= true color= QColor(ARGB 1, 1, 1, 0) "Yellow color"
        ok= true color= QColor(ARGB 1, 0, 0, 1) "Blue color"
        ok= true color= QColor(ARGB 1, 0.501961, 0, 0.501961) "Purple color"
        ok= true color= QColor(ARGB 1, 0, 1, 1) "Cyan color"
        ok= true color= QColor(ARGB 1, 1, 1, 1) "White color"
        ok= true color= QColor(ARGB 1, 0, 0, 0) "no color"
        ok= true color= QColor(ARGB 1, 1, 1, 1) "White color"
        ok= false color= QColor(ARGB 1, 0, 0, 0) "\u001B[0;37White color"

        This example supposes ansi colors are in the form 0;3x
        but it could be 3x;0 as well.
        see here:
        https://gist.github.com/JBlond/2fea43a3049b38287e5e9cefc87b2124
        http://jafrog.com/2013/11/23/colors-in-terminal.html

        As I said, it's not trivial ;)

        1 Reply Last reply
        2
        • JonBJ JonB

          @lukutis222
          [ is a special character to mark the start of list of alternative characters to match till ]. So you need to do something to match a literal [ character.

          • Try either \\[ or [[] in your C string for that. [EDIT Forum showing \\[ wrong on this line, I typed two \s, it shows two in preview mode but only one in final mode for some reason, grrr, this forum is very inconsistent in showing some literals inside backticks :( ]
          • Personally I prefer \\033 (allow reg ex to see \033, which it interprets as octal character) over \033 (let C embed an actual ASCII 27 character in the string). Same for \133 -> \\133.

          Perhaps play with these at regex101. Don't forget that if you are putting into a literal C string anything which works from there may need \ -> \\.

          L Offline
          L Offline
          lukutis222
          wrote on last edited by lukutis222
          #31

          @JonB

          Hello. It has been a while and I have learnt a couple of things since the last time. Since I have changed the logic of incoming data parsing ( using readAll instead of readLine ), the previous solution was no longer valid. I have come up with a simillar solution, It feels like I am very close to achieving what I want but not quite there yet.

          My readData function:

              QString error_match = QString("\033\1330;31m");// error
              QString info_match = QString("\033\1330;32m"); // info
              QString warning_match = QString("\033\1330;33m"); //warning
          
          
          
          void MainWindow::readData()
          {
          
              // save the scrollbar position
              QScrollBar *scrollbar = ui->Console_read_2->verticalScrollBar();
              bool scrollbarAtBottom  = (scrollbar->value() >= (scrollbar->maximum() - 4));
              int scrollbarPrevValue = scrollbar->value();
          
          
              // this moves the cursor to the bottom to avoid writing data in the middle of the console
              QTextCursor cursor = ui->Console_read_2->textCursor();
              cursor.clearSelection();
              cursor.movePosition(QTextCursor::End);
              ui->Console_read_2->setTextCursor(cursor);
              
              QByteArray data = serial_local->serial_connection.readAll(); //read all data
              QString DataAsString = QString(data); // covert  qbytearray to string
              DataAsString.replace("\r", "\n"); // replay ce all \r with \n
              QStringList myStringList = DataAsString.split("\n"); // split all data to multiple lines
          
              for(int i =0;i < myStringList.length();i++)
              {
                  myStringList[i] +=  "\n"; // append new line
                  qDebug("myStringList[%i]=%s \n",i,myStringList.at(i).toStdString().c_str());
          
                  //CAPTURE INFO LOGS
                  if(myStringList.at(i).contains(info_match)){
                      ui->Console_read_2->setTextColor(QColor(0,128,0)); // 'custom' color
                      ui->Console_read_2->insertPlainText(myStringList.at(i));
                      continue;
          
                  }
                   //CAPTURE WARNING LOGS
                  else if(myStringList.at(i).contains(warning_match)){
                      ui->Console_read_2->setTextColor(QColor(255,165,0)); // 'custom' color
                      ui->Console_read_2->insertPlainText(myStringList.at(i));
                      continue;
          
                  }
                  //CAPTURE ERROR LOGS
                  else if(myStringList.at(i).contains(error_match)){
                      ui->Console_read_2->setTextColor(QColor(255,0,0)); // 'custom' color
                      ui->Console_read_2->insertPlainText(myStringList.at(i));
                      continue;
          
                  }
                  //All other data with ANSI code is printed in black color
                  else{
                      ui->Console_read_2->setTextColor(Qt::white); //by default the text will be black
                      ui->Console_read_2->insertPlainText(myStringList.at(i));
                      continue;
          
                  }
          
              }
          
          
              if (scrollbarAtBottom)
              {
                  ui->Console_read_2->ensureCursorVisible();
              }
              else
              {
                  ui->Console_read_2->verticalScrollBar()->setValue(scrollbarPrevValue);
              }
          
          
          }
          

          The external device that I am connected to is sending me bunch of data every 2 seconds

                  ESP_LOGI("INFO","this is info log1"); 
                  ESP_LOGW("WARNING","this is warning log1");
                  ESP_LOGE("ERROR","this is error log1");
                  ESP_LOGI("INFO","this is very very very very long info log2");
                  ESP_LOGW("WARNING","this is very very very very long warning log2");
                  ESP_LOGE("ERROR","this is very very very long error log2");
                  ESP_LOGI("INFO","this is info log3");
                  ESP_LOGW("WARNING","this is warning log3");
                  ESP_LOGE("ERROR","this is error log3");
                  printf("This is normal message without ANSI color code \n");
                  delay(2000);
          

          My current problems:

          • Since I use readAll, there is no guarantee that I will complete line. For example. You can see that in my for loop, I am printing each detected line that I have split with \n:
          QStringList myStringList = DataAsString.split("\n"); // split all data to multiple lines
          
              for(int i =0;i < myStringList.length();i++)
              {
                  myStringList[i] +=  "\n"; // append new line
                  qDebug("myStringList[%i]=%s \n",i,myStringList.at(i).toStdString().c_str());
          

          On the console application, I see the following:

          myStringList[0]=[0;32mI (10872) INFO: this is info log1[0m
           
          myStringList[1]=
           
          myStringList[2]=[0;33mW (10872) W
           
          myStringList[0]=ARNING: this is warning log1[0m
           
          myStringList[1]=
           
          myStringList[2]=[0;31mE (10872) ERROR: this i
           
          myStringList[0]=s error log1[0m
           
          myStringList[1]=
           
          myStringList[2]=[0;32mI (10872) INFO: this is very very very 
           
          myStringList[0]=very long info log2[0m
           
          myStringList[1]=
           
          myStringList[2]=[0;33mW (10872) WARNING: this is very 
           
          myStringList[0]=very very very long warning log2[0m
           
          myStringList[1]=
           
          myStringList[2]=[0;31mE (10882) ERROR: th
           
          myStringList[0]=is is very very very long error log2[0m
           
          myStringList[1]=
           
          myStringList[2]=[0;32mI (10892) INFO:
           
          myStringList[0]= this is info log3[0m
           
          myStringList[1]=
           
          myStringList[2]=[0;33mW (10892) WARNING: this is warnin
           
          myStringList[0]=g log3[0m
           
          myStringList[1]=
           
          myStringList[2]=[0;31mE (10892) ERROR: this is error log3[0m
           
          myStringList[3]=
           
          myStringList[4]=This
           
          myStringList[0]= is normal message without ANSI color code 
           
          myStringList[1]=
           
          myStringList[2]=
           
          myStringList[0]=[0;32mI (12902) INFO: this is info log1[0m
           
          myStringList[1]=
           
          myStringList[2]=[0;33mW (12902) W
          

          Lets look at first 4 lines very closely:

          myStringList[0]=[0;32mI (10872) INFO: this is info log1[0m
           
          myStringList[1]=
           
          myStringList[2]=[0;33mW (10872) W
           
          myStringList[0]=ARNING: this is warning log1[0m
          

          As you can see from above, I split my data into multiple lines but the line 3 (myStringList[2]) is incomplete. Hence the next line after this one will be also incomplete and will not match with the color code sequence due to the begining lost in the previous line.

          • Another issue is that when I split data using:
          QStringList myStringList = DataAsString.split("\n"); // split all data to multiple lines,
          

          I can see that almost every other line in my for loop is empty. Is that expected and this is how split works? I feel like it is inefficient and causes additional for loop cycles to go through all lines.

          • Is it correct to manually insert new line after every line?
              QStringList myStringList = DataAsString.split("\n"); // split all data to multiple lines
              for(int i =0;i < myStringList.length();i++)
              {
                  myStringList[i] +=  "\n"; // append new line
          

          Notice that after I split data into different lines with \n delimiter, I also append \n to the very end of it. If I do not do that, the data will be printed in console without any endlines

          According to my understanding, in order to solve my problem I need to know where exactly does the new line starts and I am not sure how can I do that with readAll.

          JonBJ 1 Reply Last reply
          0
          • L lukutis222

            @JonB

            Hello. It has been a while and I have learnt a couple of things since the last time. Since I have changed the logic of incoming data parsing ( using readAll instead of readLine ), the previous solution was no longer valid. I have come up with a simillar solution, It feels like I am very close to achieving what I want but not quite there yet.

            My readData function:

                QString error_match = QString("\033\1330;31m");// error
                QString info_match = QString("\033\1330;32m"); // info
                QString warning_match = QString("\033\1330;33m"); //warning
            
            
            
            void MainWindow::readData()
            {
            
                // save the scrollbar position
                QScrollBar *scrollbar = ui->Console_read_2->verticalScrollBar();
                bool scrollbarAtBottom  = (scrollbar->value() >= (scrollbar->maximum() - 4));
                int scrollbarPrevValue = scrollbar->value();
            
            
                // this moves the cursor to the bottom to avoid writing data in the middle of the console
                QTextCursor cursor = ui->Console_read_2->textCursor();
                cursor.clearSelection();
                cursor.movePosition(QTextCursor::End);
                ui->Console_read_2->setTextCursor(cursor);
                
                QByteArray data = serial_local->serial_connection.readAll(); //read all data
                QString DataAsString = QString(data); // covert  qbytearray to string
                DataAsString.replace("\r", "\n"); // replay ce all \r with \n
                QStringList myStringList = DataAsString.split("\n"); // split all data to multiple lines
            
                for(int i =0;i < myStringList.length();i++)
                {
                    myStringList[i] +=  "\n"; // append new line
                    qDebug("myStringList[%i]=%s \n",i,myStringList.at(i).toStdString().c_str());
            
                    //CAPTURE INFO LOGS
                    if(myStringList.at(i).contains(info_match)){
                        ui->Console_read_2->setTextColor(QColor(0,128,0)); // 'custom' color
                        ui->Console_read_2->insertPlainText(myStringList.at(i));
                        continue;
            
                    }
                     //CAPTURE WARNING LOGS
                    else if(myStringList.at(i).contains(warning_match)){
                        ui->Console_read_2->setTextColor(QColor(255,165,0)); // 'custom' color
                        ui->Console_read_2->insertPlainText(myStringList.at(i));
                        continue;
            
                    }
                    //CAPTURE ERROR LOGS
                    else if(myStringList.at(i).contains(error_match)){
                        ui->Console_read_2->setTextColor(QColor(255,0,0)); // 'custom' color
                        ui->Console_read_2->insertPlainText(myStringList.at(i));
                        continue;
            
                    }
                    //All other data with ANSI code is printed in black color
                    else{
                        ui->Console_read_2->setTextColor(Qt::white); //by default the text will be black
                        ui->Console_read_2->insertPlainText(myStringList.at(i));
                        continue;
            
                    }
            
                }
            
            
                if (scrollbarAtBottom)
                {
                    ui->Console_read_2->ensureCursorVisible();
                }
                else
                {
                    ui->Console_read_2->verticalScrollBar()->setValue(scrollbarPrevValue);
                }
            
            
            }
            

            The external device that I am connected to is sending me bunch of data every 2 seconds

                    ESP_LOGI("INFO","this is info log1"); 
                    ESP_LOGW("WARNING","this is warning log1");
                    ESP_LOGE("ERROR","this is error log1");
                    ESP_LOGI("INFO","this is very very very very long info log2");
                    ESP_LOGW("WARNING","this is very very very very long warning log2");
                    ESP_LOGE("ERROR","this is very very very long error log2");
                    ESP_LOGI("INFO","this is info log3");
                    ESP_LOGW("WARNING","this is warning log3");
                    ESP_LOGE("ERROR","this is error log3");
                    printf("This is normal message without ANSI color code \n");
                    delay(2000);
            

            My current problems:

            • Since I use readAll, there is no guarantee that I will complete line. For example. You can see that in my for loop, I am printing each detected line that I have split with \n:
            QStringList myStringList = DataAsString.split("\n"); // split all data to multiple lines
            
                for(int i =0;i < myStringList.length();i++)
                {
                    myStringList[i] +=  "\n"; // append new line
                    qDebug("myStringList[%i]=%s \n",i,myStringList.at(i).toStdString().c_str());
            

            On the console application, I see the following:

            myStringList[0]=[0;32mI (10872) INFO: this is info log1[0m
             
            myStringList[1]=
             
            myStringList[2]=[0;33mW (10872) W
             
            myStringList[0]=ARNING: this is warning log1[0m
             
            myStringList[1]=
             
            myStringList[2]=[0;31mE (10872) ERROR: this i
             
            myStringList[0]=s error log1[0m
             
            myStringList[1]=
             
            myStringList[2]=[0;32mI (10872) INFO: this is very very very 
             
            myStringList[0]=very long info log2[0m
             
            myStringList[1]=
             
            myStringList[2]=[0;33mW (10872) WARNING: this is very 
             
            myStringList[0]=very very very long warning log2[0m
             
            myStringList[1]=
             
            myStringList[2]=[0;31mE (10882) ERROR: th
             
            myStringList[0]=is is very very very long error log2[0m
             
            myStringList[1]=
             
            myStringList[2]=[0;32mI (10892) INFO:
             
            myStringList[0]= this is info log3[0m
             
            myStringList[1]=
             
            myStringList[2]=[0;33mW (10892) WARNING: this is warnin
             
            myStringList[0]=g log3[0m
             
            myStringList[1]=
             
            myStringList[2]=[0;31mE (10892) ERROR: this is error log3[0m
             
            myStringList[3]=
             
            myStringList[4]=This
             
            myStringList[0]= is normal message without ANSI color code 
             
            myStringList[1]=
             
            myStringList[2]=
             
            myStringList[0]=[0;32mI (12902) INFO: this is info log1[0m
             
            myStringList[1]=
             
            myStringList[2]=[0;33mW (12902) W
            

            Lets look at first 4 lines very closely:

            myStringList[0]=[0;32mI (10872) INFO: this is info log1[0m
             
            myStringList[1]=
             
            myStringList[2]=[0;33mW (10872) W
             
            myStringList[0]=ARNING: this is warning log1[0m
            

            As you can see from above, I split my data into multiple lines but the line 3 (myStringList[2]) is incomplete. Hence the next line after this one will be also incomplete and will not match with the color code sequence due to the begining lost in the previous line.

            • Another issue is that when I split data using:
            QStringList myStringList = DataAsString.split("\n"); // split all data to multiple lines,
            

            I can see that almost every other line in my for loop is empty. Is that expected and this is how split works? I feel like it is inefficient and causes additional for loop cycles to go through all lines.

            • Is it correct to manually insert new line after every line?
                QStringList myStringList = DataAsString.split("\n"); // split all data to multiple lines
                for(int i =0;i < myStringList.length();i++)
                {
                    myStringList[i] +=  "\n"; // append new line
            

            Notice that after I split data into different lines with \n delimiter, I also append \n to the very end of it. If I do not do that, the data will be printed in console without any endlines

            According to my understanding, in order to solve my problem I need to know where exactly does the new line starts and I am not sure how can I do that with readAll.

            JonBJ Offline
            JonBJ Offline
            JonB
            wrote on last edited by JonB
            #32

            @lukutis222
            It's a long post, so here are couple of observations.

            I think you might be the same guy who has been asking about "reading lines" in another thread, since I recognise

            QStringList myStringList = DataAsString.split("\n"); // split all data to multiple lines,

            I did not know you are also the guy for this thread. Are you?? That person was saying incoming lines could end either in \n or in \r (which worries me). Is that you too, or quite separate?

            Since I have changed the logic of incoming data parsing ( using readAll instead of readLine )
            Since I use readAll, there is no guarantee that I will complete line.
            According to my understanding, in order to solve my problem I need to know where exactly does the new line starts and I am not sure how can I do that with readAll.

            Indeed, and you will want to code (correctly) for that. I do not know why you are no longer using readLine(), unless you are the other guy with the \r/\n issue. If you need to use readAll() you need to do a proper job to replace readLine/canReadLine(). And I suspect you are not.

            Consider input of, say, "Line 1\nLine 2\n Line". The last line does not have a trailing \n, and hence is not complete. That means you must not process it yet as a complete line, you can process the first 2 lines there but must "buffer" the 3rd line to allow further characters to arrive at a future time.

            You are using split("\n") to split your input into a list of lines. This is not good enough here. The problem is it will return 3 lines for the list, but you cannot tell from it that the third line did not end in a \n. If the input had been "Line 1\nLine 2\n Line\n" --- note now the trailing \n at the end making the last line complete --- you would get exactly the same result back from split(). So it's not good (precise) enough for this task.

            readLine/canReadLine() would not have used split(). It would not divide all its input into lines, it would only have looked for the first newline in the string and dealt with that. Use e.g. QString::indexOf() or QByteArray::indexOf() to search for the first \n in the input. If there is not one then canReadLine() returns false, and you should not call readLine() yet, see how the docs state:

            Note that on sequential devices, data may not be immediately available, which may result in a partial line being returned. By calling the canReadLine() function before reading, you can check whether a complete line (including the newline character) can be read.

            If it does have a \n then readLine() can be called. Its job is to read up to and including the first \n and return that segment, while removing that from its currently buffered input, so that only the stuff after that \n remain to be read at a future time.

            You need to implement similar. And if you are the person who wants to treat \rs like \ns deal with \r instead of \n as the first end-of-line marker appropriately.

            I can see that almost every other line in my for loop is empty. Is that expected and this is how split works?

            split() does not insert anything into your input/the resulting lines, and does not produce any empty/blank lines which are not in the input. If you are the person who claims that your input has "lines" ending in \r as well as in \n, and if you are presently replacing all \rs by \ns, I said I was never happy with that. If you actually have \r\n at the end of lines you will end up with \n\n and that will cause an unintended blank line. If you are not that person then:

            Is it correct to manually insert new line after every line?
            myStringList[i] += "\n"; // append new line

            That depends on what you will do with the myStringList[] lines. If you append a \n here then you must not also output them with an explicit \n, or use some kind of "writeLine()", as that will lead to a double newline. But since you say

            I also append \n to the very end of it. If I do not do that, the data will be printed in console without any endlines

            I don't think you are doing that. So cannot answer where blank lines are coming from, analyze your input carefully.

            Summary: Get rid of split("\n"), do it properly with e.g. indexOf() just getting the next line out of your buffer and leaving all further stuff to be read/dealt with later.

            L 1 Reply Last reply
            0
            • JonBJ JonB

              @lukutis222
              It's a long post, so here are couple of observations.

              I think you might be the same guy who has been asking about "reading lines" in another thread, since I recognise

              QStringList myStringList = DataAsString.split("\n"); // split all data to multiple lines,

              I did not know you are also the guy for this thread. Are you?? That person was saying incoming lines could end either in \n or in \r (which worries me). Is that you too, or quite separate?

              Since I have changed the logic of incoming data parsing ( using readAll instead of readLine )
              Since I use readAll, there is no guarantee that I will complete line.
              According to my understanding, in order to solve my problem I need to know where exactly does the new line starts and I am not sure how can I do that with readAll.

              Indeed, and you will want to code (correctly) for that. I do not know why you are no longer using readLine(), unless you are the other guy with the \r/\n issue. If you need to use readAll() you need to do a proper job to replace readLine/canReadLine(). And I suspect you are not.

              Consider input of, say, "Line 1\nLine 2\n Line". The last line does not have a trailing \n, and hence is not complete. That means you must not process it yet as a complete line, you can process the first 2 lines there but must "buffer" the 3rd line to allow further characters to arrive at a future time.

              You are using split("\n") to split your input into a list of lines. This is not good enough here. The problem is it will return 3 lines for the list, but you cannot tell from it that the third line did not end in a \n. If the input had been "Line 1\nLine 2\n Line\n" --- note now the trailing \n at the end making the last line complete --- you would get exactly the same result back from split(). So it's not good (precise) enough for this task.

              readLine/canReadLine() would not have used split(). It would not divide all its input into lines, it would only have looked for the first newline in the string and dealt with that. Use e.g. QString::indexOf() or QByteArray::indexOf() to search for the first \n in the input. If there is not one then canReadLine() returns false, and you should not call readLine() yet, see how the docs state:

              Note that on sequential devices, data may not be immediately available, which may result in a partial line being returned. By calling the canReadLine() function before reading, you can check whether a complete line (including the newline character) can be read.

              If it does have a \n then readLine() can be called. Its job is to read up to and including the first \n and return that segment, while removing that from its currently buffered input, so that only the stuff after that \n remain to be read at a future time.

              You need to implement similar. And if you are the person who wants to treat \rs like \ns deal with \r instead of \n as the first end-of-line marker appropriately.

              I can see that almost every other line in my for loop is empty. Is that expected and this is how split works?

              split() does not insert anything into your input/the resulting lines, and does not produce any empty/blank lines which are not in the input. If you are the person who claims that your input has "lines" ending in \r as well as in \n, and if you are presently replacing all \rs by \ns, I said I was never happy with that. If you actually have \r\n at the end of lines you will end up with \n\n and that will cause an unintended blank line. If you are not that person then:

              Is it correct to manually insert new line after every line?
              myStringList[i] += "\n"; // append new line

              That depends on what you will do with the myStringList[] lines. If you append a \n here then you must not also output them with an explicit \n, or use some kind of "writeLine()", as that will lead to a double newline. But since you say

              I also append \n to the very end of it. If I do not do that, the data will be printed in console without any endlines

              I don't think you are doing that. So cannot answer where blank lines are coming from, analyze your input carefully.

              Summary: Get rid of split("\n"), do it properly with e.g. indexOf() just getting the next line out of your buffer and leaving all further stuff to be read/dealt with later.

              L Offline
              L Offline
              lukutis222
              wrote on last edited by lukutis222
              #33

              @JonB said in How to implement reading serial data with ANSI color codes and printing out to textbox in color:

              readLine()

              Yes I am the same guy who want to treat \r like \n and that is exactly the reason why I no longer can use readLine/canReadLine(). I thought posting in this existing thread made more sense than the other one for some reason.
              From what I understand, the only way to handle incoming serial data with \r \n is to use readAll but that brings a bunch of new problems that I did not have when using readLine/canReadLine().

              I will go through your suggestions very closely and see if I can come up with something clever.. Thank you very much !

              UPDATE

              • Regarding the every other line in the loop emtpy

              You were totally right regarding \n\n. Some data that I read comes with \r\n hence that results with \n\n..

              I am now looking into your suggestions about indexOf()

              JonBJ 1 Reply Last reply
              0
              • L lukutis222

                @JonB said in How to implement reading serial data with ANSI color codes and printing out to textbox in color:

                readLine()

                Yes I am the same guy who want to treat \r like \n and that is exactly the reason why I no longer can use readLine/canReadLine(). I thought posting in this existing thread made more sense than the other one for some reason.
                From what I understand, the only way to handle incoming serial data with \r \n is to use readAll but that brings a bunch of new problems that I did not have when using readLine/canReadLine().

                I will go through your suggestions very closely and see if I can come up with something clever.. Thank you very much !

                UPDATE

                • Regarding the every other line in the loop emtpy

                You were totally right regarding \n\n. Some data that I read comes with \r\n hence that results with \n\n..

                I am now looking into your suggestions about indexOf()

                JonBJ Offline
                JonBJ Offline
                JonB
                wrote on last edited by JonB
                #34

                @lukutis222 said in How to implement reading serial data with ANSI color codes and printing out to textbox in color:

                Yes I am the same guy who want to treat \r like \n and that is exactly the reason why I no longer can use readLine/canReadLine(). I thought posting in this existing thread made more sense than the other one for some reason.

                Ah ha! :) Agreed about continuing your question in this thread, just it would have helped if you had referenced that thread in this one now so that I/people would know! They are related because of your situation.

                Now you have a real problem :( Per what you have said/"claimed" that your lines could be terminated by any of just \n, just \r or \r\n. Consider receiving "Line 1\r". Is this a complete line ready to be processed, or is it a partial line waiting for a \n to arrive later? The answer is you cannot tell, you do not know whether the next character will or won't be a \n. So you have a problem with this line, which would require careful coding, and would certainly defeat readLine().

                Stop what you are coding now! Go back and look carefully at all the bytes you actually receive (without trying to break them into lines, for now). readLine() can deal with just \n at the end of a line. It can also deal with \r\n (we might take some further action on this, but it will be treated as a line). But it cannot deal with your claim that some lines end in only \r. Come back as & when you are sure whether you really have a \r and no following \n to be treated as a "complete line", or whether actually it's always \r\n....

                L 1 Reply Last reply
                0
                • JonBJ JonB

                  @lukutis222 said in How to implement reading serial data with ANSI color codes and printing out to textbox in color:

                  Yes I am the same guy who want to treat \r like \n and that is exactly the reason why I no longer can use readLine/canReadLine(). I thought posting in this existing thread made more sense than the other one for some reason.

                  Ah ha! :) Agreed about continuing your question in this thread, just it would have helped if you had referenced that thread in this one now so that I/people would know! They are related because of your situation.

                  Now you have a real problem :( Per what you have said/"claimed" that your lines could be terminated by any of just \n, just \r or \r\n. Consider receiving "Line 1\r". Is this a complete line ready to be processed, or is it a partial line waiting for a \n to arrive later? The answer is you cannot tell, you do not know whether the next character will or won't be a \n. So you have a problem with this line, which would require careful coding, and would certainly defeat readLine().

                  Stop what you are coding now! Go back and look carefully at all the bytes you actually receive (without trying to break them into lines, for now). readLine() can deal with just \n at the end of a line. It can also deal with \r\n (we might take some further action on this, but it will be treated as a line). But it cannot deal with your claim that some lines end in only \r. Come back as & when you are sure whether you really have a \r and no following \n to be treated as a "complete line", or whether actually it's always \r\n....

                  L Offline
                  L Offline
                  lukutis222
                  wrote on last edited by lukutis222
                  #35

                  @JonB

                  Hello. The reason why I want to be able to parse \n, \r and the combination of the two because I want to make my terminal as bulletproof and as much universal as possible.

                  I have used many terminals such as PuTTy, Termite, MobaXterm and every terminal had disatvantages, for example:

                  • Termite - not supporting color codes. The controllers that I program always printing in color so its very convenient for me to see different logs in different colors.

                  • PuTTy - supports color codes but is only for reading the data. It does not allow to write data to external device

                  • MobaXterm - Supports color codes and seems to work very nicely, however, I didint find a way to write serial data to external device and only reading...

                  Hence for my first QT project I have decided to create this terminal because I thought it would be good C++ experience and I also I want this terminal to rule them all!

                  When I am programming my own device, I can sure use only \n. But lets assume some other guy connects its weird device with my terminal and his device only prints with \r.

                  JonBJ 1 Reply Last reply
                  0
                  • L lukutis222

                    @JonB

                    Hello. The reason why I want to be able to parse \n, \r and the combination of the two because I want to make my terminal as bulletproof and as much universal as possible.

                    I have used many terminals such as PuTTy, Termite, MobaXterm and every terminal had disatvantages, for example:

                    • Termite - not supporting color codes. The controllers that I program always printing in color so its very convenient for me to see different logs in different colors.

                    • PuTTy - supports color codes but is only for reading the data. It does not allow to write data to external device

                    • MobaXterm - Supports color codes and seems to work very nicely, however, I didint find a way to write serial data to external device and only reading...

                    Hence for my first QT project I have decided to create this terminal because I thought it would be good C++ experience and I also I want this terminal to rule them all!

                    When I am programming my own device, I can sure use only \n. But lets assume some other guy connects its weird device with my terminal and his device only prints with \r.

                    JonBJ Offline
                    JonBJ Offline
                    JonB
                    wrote on last edited by
                    #36

                    @lukutis222
                    Then to cater for "just \n, just \r or \r\n" you will have to code appropriately, you can't use readLine() and you will have to do the necessary coding yourself (remember what I said about the "ambiguity" of the input read into the buffer ending in just \r). That is the answer.

                    L 1 Reply Last reply
                    0
                    • JonBJ JonB

                      @lukutis222
                      Then to cater for "just \n, just \r or \r\n" you will have to code appropriately, you can't use readLine() and you will have to do the necessary coding yourself (remember what I said about the "ambiguity" of the input read into the buffer ending in just \r). That is the answer.

                      L Offline
                      L Offline
                      lukutis222
                      wrote on last edited by lukutis222
                      #37

                      @JonB

                      I was also not aware that readAll() automatically inserts \r in between different lines.

                      My remote device is sending the following data:

                              printf("This is normal message1 without ANSI color code \n");
                              printf("This is normal message2 without ANSI color code \n");
                              printf("This is normal message3 without ANSI color code \n");
                              printf("This is normal message4 without ANSI color code \n");
                              printf("This is normal message5 without ANSI color code \n");
                              printf("This is normal message6 without ANSI color code \n");
                              printf("This is normal message7 without ANSI color code \n");
                              printf("This is normal message8 without ANSI color code \n");
                      delay(2000);
                      

                      As you can see from above, I use only \n termination.

                      In my readData() funciton I do the following(Notice that I do not replace \r with \n at the moment)

                          QByteArray data = serial_local->serial_connection.readAll(); //read all data
                          QString DataAsString = QString(data); // covert  qbytearray to string
                          uint16_t received_data_length = DataAsString.length();
                          //DataAsString.replace("\r", "\n"); // replay ce all \r with \n
                          qDebug("****FULL DATA***** = %s \n",DataAsString.toStdString().c_str());
                          qDebug("*****FULL DATA length***** = %u \n",received_data_length);
                          //print all raw bytes
                          for(int i = 0; i <received_data_length;i++){
                              qDebug("%u\n",DataAsString[i]);
                          }
                      
                      

                      The application output:

                      ****FULL DATA***** = This is normal message1 without ANSI color code 
                      This is normal 
                      *****FULL DATA length***** = 64 
                      84
                      104
                      105
                      115
                      32
                      105
                      115
                      32
                      110
                      111
                      114
                      109
                      97
                      108
                      32
                      109
                      101
                      115
                      115
                      97
                      103
                      101
                      49
                      32
                      119
                      105
                      116
                      104
                      111
                      117
                      116
                      32
                      65
                      78
                      83
                      73
                      32
                      99
                      111
                      108
                      111
                      114
                      32
                      99
                      111
                      100
                      101
                      32
                      13
                      10
                      84
                      104
                      105
                      115
                      32
                      105
                      115
                      32
                      110
                      111
                      114
                      109
                      97
                      108
                      

                      Notice that there is decimal 13 and 10 (carriage return and new line). I am trying to understand how did carriage return appear here if I do not send it.

                      This is confusing because I do not know if I should do indexOf('\n') or indexOf('\r'). If readAll automatically inserts \r between different lines it would make more sense to detect a complete line with \r

                      Christian EhrlicherC 1 Reply Last reply
                      0
                      • L lukutis222

                        @JonB

                        I was also not aware that readAll() automatically inserts \r in between different lines.

                        My remote device is sending the following data:

                                printf("This is normal message1 without ANSI color code \n");
                                printf("This is normal message2 without ANSI color code \n");
                                printf("This is normal message3 without ANSI color code \n");
                                printf("This is normal message4 without ANSI color code \n");
                                printf("This is normal message5 without ANSI color code \n");
                                printf("This is normal message6 without ANSI color code \n");
                                printf("This is normal message7 without ANSI color code \n");
                                printf("This is normal message8 without ANSI color code \n");
                        delay(2000);
                        

                        As you can see from above, I use only \n termination.

                        In my readData() funciton I do the following(Notice that I do not replace \r with \n at the moment)

                            QByteArray data = serial_local->serial_connection.readAll(); //read all data
                            QString DataAsString = QString(data); // covert  qbytearray to string
                            uint16_t received_data_length = DataAsString.length();
                            //DataAsString.replace("\r", "\n"); // replay ce all \r with \n
                            qDebug("****FULL DATA***** = %s \n",DataAsString.toStdString().c_str());
                            qDebug("*****FULL DATA length***** = %u \n",received_data_length);
                            //print all raw bytes
                            for(int i = 0; i <received_data_length;i++){
                                qDebug("%u\n",DataAsString[i]);
                            }
                        
                        

                        The application output:

                        ****FULL DATA***** = This is normal message1 without ANSI color code 
                        This is normal 
                        *****FULL DATA length***** = 64 
                        84
                        104
                        105
                        115
                        32
                        105
                        115
                        32
                        110
                        111
                        114
                        109
                        97
                        108
                        32
                        109
                        101
                        115
                        115
                        97
                        103
                        101
                        49
                        32
                        119
                        105
                        116
                        104
                        111
                        117
                        116
                        32
                        65
                        78
                        83
                        73
                        32
                        99
                        111
                        108
                        111
                        114
                        32
                        99
                        111
                        100
                        101
                        32
                        13
                        10
                        84
                        104
                        105
                        115
                        32
                        105
                        115
                        32
                        110
                        111
                        114
                        109
                        97
                        108
                        

                        Notice that there is decimal 13 and 10 (carriage return and new line). I am trying to understand how did carriage return appear here if I do not send it.

                        This is confusing because I do not know if I should do indexOf('\n') or indexOf('\r'). If readAll automatically inserts \r between different lines it would make more sense to detect a complete line with \r

                        Christian EhrlicherC Offline
                        Christian EhrlicherC Offline
                        Christian Ehrlicher
                        Lifetime Qt Champion
                        wrote on last edited by
                        #38

                        @lukutis222 said in How to implement reading serial data with ANSI color codes and printing out to textbox in color:

                        I was also not aware that readAll() automatically inserts \r in between different lines.

                        readAll() does not modify any data read from the io device.

                        Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
                        Visit the Qt Academy at https://academy.qt.io/catalog

                        L 1 Reply Last reply
                        3
                        • Christian EhrlicherC Christian Ehrlicher

                          @lukutis222 said in How to implement reading serial data with ANSI color codes and printing out to textbox in color:

                          I was also not aware that readAll() automatically inserts \r in between different lines.

                          readAll() does not modify any data read from the io device.

                          L Offline
                          L Offline
                          lukutis222
                          wrote on last edited by lukutis222
                          #39

                          @Christian-Ehrlicher
                          I cannot wrap my head around how does \r appear in the serial data then... The remote device that I am connected to only sends \n

                          Christian EhrlicherC 1 Reply Last reply
                          0
                          • L lukutis222

                            @Christian-Ehrlicher
                            I cannot wrap my head around how does \r appear in the serial data then... The remote device that I am connected to only sends \n

                            Christian EhrlicherC Offline
                            Christian EhrlicherC Offline
                            Christian Ehrlicher
                            Lifetime Qt Champion
                            wrote on last edited by Christian Ehrlicher
                            #40

                            @lukutis222 said in How to implement reading serial data with ANSI color codes and printing out to textbox in color:

                            I cannot wrap my head around how does \r appear in the serial data then...

                            Please output the QByteArray data, not the one converted to QString (for whatever reason binary data should be converted to a QString though).

                            Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
                            Visit the Qt Academy at https://academy.qt.io/catalog

                            L 1 Reply Last reply
                            3
                            • Christian EhrlicherC Christian Ehrlicher

                              @lukutis222 said in How to implement reading serial data with ANSI color codes and printing out to textbox in color:

                              I cannot wrap my head around how does \r appear in the serial data then...

                              Please output the QByteArray data, not the one converted to QString (for whatever reason binary data should be converted to a QString though).

                              L Offline
                              L Offline
                              lukutis222
                              wrote on last edited by lukutis222
                              #41

                              @Christian-Ehrlicher

                              
                                  QByteArray data = serial_local->serial_connection.readAll(); //read all data
                                  QString DataAsString = QString(data); // covert  qbytearray to string
                                  uint16_t received_data_length = DataAsString.length();
                              
                                  for(int i = 0; i <received_data_length;i++){
                                      qDebug("%u\n",DataAsString[i]);
                                  }
                                  qDebug("****RAW DATA***** \n");
                                  for(int i = 0; i <data.length();i++){
                                      qDebug("%u\n",data[i]);
                                  }
                              

                              Both identical.
                              Does that mean that my remote device "secretly" slips in another \r without me even wanting ?

                              Christian EhrlicherC 1 Reply Last reply
                              0
                              • L lukutis222

                                @Christian-Ehrlicher

                                
                                    QByteArray data = serial_local->serial_connection.readAll(); //read all data
                                    QString DataAsString = QString(data); // covert  qbytearray to string
                                    uint16_t received_data_length = DataAsString.length();
                                
                                    for(int i = 0; i <received_data_length;i++){
                                        qDebug("%u\n",DataAsString[i]);
                                    }
                                    qDebug("****RAW DATA***** \n");
                                    for(int i = 0; i <data.length();i++){
                                        qDebug("%u\n",data[i]);
                                    }
                                

                                Both identical.
                                Does that mean that my remote device "secretly" slips in another \r without me even wanting ?

                                Christian EhrlicherC Offline
                                Christian EhrlicherC Offline
                                Christian Ehrlicher
                                Lifetime Qt Champion
                                wrote on last edited by
                                #42

                                @lukutis222 said in How to implement reading serial data with ANSI color codes and printing out to textbox in color:

                                Does that mean that my remote device "secretly" slips in another \r without me even wanting ?

                                Yes

                                Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
                                Visit the Qt Academy at https://academy.qt.io/catalog

                                1 Reply Last reply
                                0
                                • L Offline
                                  L Offline
                                  lukutis222
                                  wrote on last edited by lukutis222
                                  #43

                                  I think I managed to achieve what I want (didint test 100% but I believe I am getting very close). I want to share the current logic, perhaps you can advise on how to improve it.

                                  void MainWindow::readData()
                                  {
                                      // save the scrollbar position
                                      QScrollBar *scrollbar = ui->Console_read_2->verticalScrollBar();
                                      bool scrollbarAtBottom  = (scrollbar->value() >= (scrollbar->maximum() - 4));
                                      int scrollbarPrevValue = scrollbar->value();
                                  
                                      // this moves the cursor to the bottom to avoid writing data in the middle of the console
                                      QTextCursor cursor = ui->Console_read_2->textCursor();
                                      cursor.clearSelection();
                                      cursor.movePosition(QTextCursor::End);
                                      ui->Console_read_2->setTextCursor(cursor);
                                  
                                      QByteArray data = serial_local->serial_connection.readAll(); //read all data
                                      QString DataAsString = QString(data); // covert  qbytearray to string
                                      static QString incomplete_line = nullptr; // hold information about the last incomplete line
                                      uint16_t beggining_pointer = 0; // save the beggining pointer
                                      
                                      DataAsString.replace("\r", "\n"); // replace all \r with \n
                                      DataAsString.replace("\n\n", "\n"); // this ensures that all double \n\n (if exist) will be replaces with just a single \n
                                  
                                      int index = DataAsString.indexOf('\n');
                                      
                                      while (index != -1) {
                                          QString complete_line = DataAsString.mid(beggining_pointer, index-(beggining_pointer)); // index is where the \n was found in the string.
                                          complete_line = incomplete_line+complete_line; // append beggining of incomplete line to complete line (initially incomplete line will be empty)
                                          incomplete_line = nullptr; // reset the incomplete line after each append because its no longer relevant
                                          Format_and_insert(complete_line); // insert data to console
                                          beggining_pointer = index+1; // increment beggining pointer to the location where the previous \n was found because this will now be our beggining pointer
                                          index = DataAsString.indexOf('\n', index+1);
                                  
                                      }
                                  
                                       incomplete_line = DataAsString.mid(beggining_pointer, DataAsString.length()); // after going through every line, take the last beggining pointer and read till the end of string.
                                  
                                      if (scrollbarAtBottom)
                                      {
                                          ui->Console_read_2->ensureCursorVisible();
                                      }
                                      else
                                      {
                                          ui->Console_read_2->verticalScrollBar()->setValue(scrollbarPrevValue);
                                      }
                                  
                                  
                                  }
                                  

                                  I am reading line by line and when I find an uncomplete line, I save it in the incomplete_line buffer and use it on the next iteration.

                                  1 Reply Last reply
                                  0
                                  • Christian EhrlicherC Offline
                                    Christian EhrlicherC Offline
                                    Christian Ehrlicher
                                    Lifetime Qt Champion
                                    wrote on last edited by
                                    #44

                                    And now you're again simulating readLine()/canReadLine()...

                                    Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
                                    Visit the Qt Academy at https://academy.qt.io/catalog

                                    JonBJ 1 Reply Last reply
                                    0
                                    • Christian EhrlicherC Christian Ehrlicher

                                      And now you're again simulating readLine()/canReadLine()...

                                      JonBJ Offline
                                      JonBJ Offline
                                      JonB
                                      wrote on last edited by JonB
                                      #45

                                      @Christian-Ehrlicher
                                      Which is what OP is wanting to do. His problem is that he wishes to treat \r, without requiring a following \n, as a line end. I have verified that Qt code for all of canReadLine/readLine/readLineData() have \n hard-coded in their logic. It is not possible to use these existing ones if you want to treat a bare \r as an acceptable line-ender. (In fact, he wishes to treat any/all of \n, \r or \r\n as line terminators, which complicates things.) While the proposed code above is "rough and ready", and has some flaws, and I personally was not prepared to write the full code of what he wants, if he is happy with this "approximation" it would seem that it would give him what he wants.

                                      Of course, if he did not insist on accepting \r as a line terminator it would be a whole lot simpler....

                                      Christian EhrlicherC 1 Reply Last reply
                                      0
                                      • JonBJ JonB

                                        @Christian-Ehrlicher
                                        Which is what OP is wanting to do. His problem is that he wishes to treat \r, without requiring a following \n, as a line end. I have verified that Qt code for all of canReadLine/readLine/readLineData() have \n hard-coded in their logic. It is not possible to use these existing ones if you want to treat a bare \r as an acceptable line-ender. (In fact, he wishes to treat any/all of \n, \r or \r\n as line terminators, which complicates things.) While the proposed code above is "rough and ready", and has some flaws, and I personally was not prepared to write the full code of what he wants, if he is happy with this "approximation" it would seem that it would give him what he wants.

                                        Of course, if he did not insist on accepting \r as a line terminator it would be a whole lot simpler....

                                        Christian EhrlicherC Offline
                                        Christian EhrlicherC Offline
                                        Christian Ehrlicher
                                        Lifetime Qt Champion
                                        wrote on last edited by Christian Ehrlicher
                                        #46

                                        @JonB Overread this.

                                        Still the code looks ugly and over-complicated

                                        void parseBuffer()
                                        {
                                          auto findNextSeparator = [](const QByteArray &ba, int startOfs) {
                                            for (int i = startOfs; i < ba.size(); ++i) {
                                              auto c = ba[i];
                                              if (c == '\n' || c == '\r') {
                                                return i;
                                              }
                                            }
                                            return -1;
                                          };
                                          auto fromIdx = 0;
                                          idx = findNextSeparator(m_buffer, fromIdx );
                                          while (idx >= 0) {
                                            auto line = m_buffer.mid(fromIdx , idx - fromIdx);
                                            if (!line.isEmpty())
                                              emit newLineReceived(line);
                                            fromIdx = idx + 1;
                                          }
                                          m_buffer = m_buffer.mid(fromIdx);
                                        }
                                        

                                        Did not check if '\r' or '\n' is in the 'line' buffer - maybe this needs a +/-1 adjustment.

                                        Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
                                        Visit the Qt Academy at https://academy.qt.io/catalog

                                        JonBJ 1 Reply Last reply
                                        0
                                        • Christian EhrlicherC Christian Ehrlicher

                                          @JonB Overread this.

                                          Still the code looks ugly and over-complicated

                                          void parseBuffer()
                                          {
                                            auto findNextSeparator = [](const QByteArray &ba, int startOfs) {
                                              for (int i = startOfs; i < ba.size(); ++i) {
                                                auto c = ba[i];
                                                if (c == '\n' || c == '\r') {
                                                  return i;
                                                }
                                              }
                                              return -1;
                                            };
                                            auto fromIdx = 0;
                                            idx = findNextSeparator(m_buffer, fromIdx );
                                            while (idx >= 0) {
                                              auto line = m_buffer.mid(fromIdx , idx - fromIdx);
                                              if (!line.isEmpty())
                                                emit newLineReceived(line);
                                              fromIdx = idx + 1;
                                            }
                                            m_buffer = m_buffer.mid(fromIdx);
                                          }
                                          

                                          Did not check if '\r' or '\n' is in the 'line' buffer - maybe this needs a +/-1 adjustment.

                                          JonBJ Offline
                                          JonBJ Offline
                                          JonB
                                          wrote on last edited by JonB
                                          #47

                                          @Christian-Ehrlicher
                                          Please do not take this wrong, but your code is a touch simplistic. I don't blame you, as I was not prepared to write a comprehensive implementation!

                                          Per the user's requirements, for example it fails to distinguish between incoming \r\n --- one line --- versus \n\n --- a line followed by a genuine blank line. It also does not deal with end-of-file at the end of a final line which has no \r or \n. It does not tell the OP when he needs to call readAll() to fetch more characters for the buffer, and address what to do if that returns eof. It does not deal well with the (unlikely but possible?) case where a readAll() returns a line ending in \r because that is all that has been received so far, when in fact the next character received in the future will be a \n after the \r. I am not saying his code deals with all of these cases either!

                                          All of which is what I thought about when deciding whether I wished to offer the full, robust code, and decided not to... :)

                                          Nonetheless, perhaps the OP will adapt your code to whatever situations he wants to cover.

                                          1 Reply Last reply
                                          0

                                          • Login

                                          • Login or register to search.
                                          • First post
                                            Last post
                                          0
                                          • Categories
                                          • Recent
                                          • Tags
                                          • Popular
                                          • Users
                                          • Groups
                                          • Search
                                          • Get Qt Extensions
                                          • Unsolved