Raspbbery Pi 3 Serial Port Performance Problem

aktay

Hi,

I am Abdullah, I am working on a project on the RP-3.

I am communicating with a remote device such as Battery Management System via UART.

The remote device is sending a data frame with a checksum at the end of the frame.

There are three different frames and each one includes voltage, current, and temperature information.

I can get the frames properly and I can show the data(voltage, current, temperature) on the screen by using Qwidget.

Everything is working properly but if I want to do one more process like logging data into the excel document with the qxlsx library, performance is decreasing during the process.

I noticed that the data from UART come as string substituted for char and I believe that they store into the frame buffer. So, it causes that my algorithm is broken and I could not get the data properly.

The code sequence which is read the data is shown below.

Read PORT

void MainWindow::portRead(){
       QByteArray portData;
          portData=serial->readAll();
             if(portData==QChar('[')){
             tempList.append(portData);
             arrayFirst=1;
             }else if(portData==QChar('*') && arrayFirst!=0){
               emit arraySplit(tempList);
                  tempList.clear();
                  arrayFirst=0;
             }else if(arrayFirst==1){
       tempList.append(portData);
             }
     
}

Split Data Send Class

void MainWindow::arraySplit(QByteArray signalArray){
    QByteArray checkSum;
    int checkSumFirst,checkSumEnd;
    for(int i=0;i<signalArray .length();i++){
        if(signalArray .at(i)==']'){
            checkSumFirst=i+1;
        }else if(signalArray .at(i)=='*'){
          checkSumEnd=i-1;
        }else if(i>= checkSumFirst && i<=checkSumEnd){
         checkSum.append(signalArray .at(i));
        }
    }
  signalArray.remove(checkSumFirst,checkSumEnd);
  int check=checkSum.toInt();
    if(checksumCalculate(signalArray,check)){
        QString value(signalArray);
        QStringList data=value.split(sep);
       data.removeLast();
       data.removeFirst();
       QString tempData=data.at(1);
       data.removeFirst();
      data.removeFirst();
      if(tempData=="21"){ //Voltage OP Code
          emit sendVolt(data); //send voltage class (thread signal)
          emit sendExcel(data,counter); //send excel class (thread signal)
          QString folderName = dateTime.toString("dd-MM-yyyy_hh-mm-ss");
          QString folder=QString(folderName+".xlsx");
          xlsx->saveAs(folder);
          counter++;
      }if(tempData=="23"){ //Temperature OP Code
        emit sendTemperature(data); //send temperature class (thread signal)
      }
      if(tempData=="25"){ //Current OP Code
        emit sendCurrent(data); //send current class (thread signal)
      }
    }

My solution is as follows. Add the incoming data into the loop and read it. But in this way the data is delayed. So this is not a good way.

void MainWindow::portRead(){
       QByteArray portData;
          portData=serial->readAll();
         for(int i=0;i<portData.size();i++){
             if(portData.at(i)==QChar('[')){
             tempList.append(portData.at(i));
             arrayFirst=1;
             }else if(portData.at(i)==QChar('*') && arrayFirst!=0){
               emit arraySplit(tempList);
                  tempList.clear();
                  arrayFirst=0;
             }else if(arrayFirst==1){
       tempList.append(portData.at(i));
             }
         }
     
}

Are there any idea that you can suggest me for getting data as more efficient?

Hearing you soon.

Thanks.

koahnig

Hi and welcome to devnet forum

If I understand correctly, you have no problem reading the serial data and display it somewhere until you try to export the data into excel sheets with qxlsx.

You are suspecting a performance problem. IMHO you are right that it is in either component serial plus display or qxlsx, but more likely qxlsx is causing the problem because of complexity of Excel formatting.
Therefore, I would start looking into details of cpu time consumption of components rather than pointing to somewhere and try to optimize there.

I have never worked RP, but I assume that you can check in parallel some cpu time checks. This might be through linux commands in RP OS in parallel or measures of time differences in your application. You probably need to research on your own what the best way is.

Another option would be storing the information into plain text files without any additional processing instead of your excel sheet. If this works sufficiently, you can check the performance of qxlsx by simply generating data with an internal time stamp and store with qxlsx in a similar fashion as you do with data coming from serial.

When the storing of plain files is already a problem, your board is too slow and you need to lower the data output by your device and/or looking for the optimization as you already looking for above.

My gut feeling tells me that probably you are facing already problems with your simulation of storing with qxlsx. This is probably slowing the whole process and casuing your problems.

aktay

I tried to add the data to MySQL. But the same thing happens again.Some data do not come as characters, they come as strings.

QT App CPU= %6.0, Memory usage: Regular increase. Avarege %14.5
Mysql CPU=1.0, Memory usage : Regular increase. Avarage % 4

I think that an arrangement should be made in software.
Are there any idea that you can suggest me for getting data as more efficient?

Thanks.

koahnig

@aktay said in Raspbbery Pi 3 Serial Port Performance Problem:

QT App CPU= %6.0, Memory usage: Regular increase. Avarege %14.5
Mysql CPU=1.0, Memory usage : Regular increase. Avarage % 4

That is good. There seem to be enough cpu available.

@aktay said in Raspbbery Pi 3 Serial Port Performance Problem:

I tried to add the data to MySQL. But the same thing happens again.Some data do not come as characters, they come as strings.
Where do you see the problem here?
A string is just a sequence of characters. Very often there are more just one byte/character in the buffer when you got the signal readyRead().
With

          QByteArray portData;
          portData=serial->readAll();

You are reading all bytes at once from the buffer associated with the serial into portData. Then you loop over the different bytes and seem to handle them correctly. It is not clear to me where you have a problem.

aktay

You may see correct data format below:

"["
"1"
"0"
"0"
"0"
","
"2"
"1"
","
"3"
"6"
"7"
"8"
","
"3"
"6"
"8"
//etc.

When I put on extra work (ex. write data to excel, write data to mysql etc.) data format changes.

"0"
"]"
"0"
"6"
"9"
"*"
"\r\n\x00\x00\x00"
**"[1000,25,+00"** // Normally I should get this as one charecter.
"0"
"0"
"0"
","
"2"
"]"
"0"
"7"
"7"
"*"
"\r"
"\n"
"["
"1"

Thanks.

koahnig

@aktay

Please show the code you are using to print this stuff.

aktay

I am read port after write console. I just using qDebug().

void MainWindow::portRead(){
       QByteArray portData;
          portData=serial->readAll();
qDebug()<<portData;  // I am writing the console here. Also I am sending data mysql and qwidget. (Via another Class and another function)
      
             if(portData==QChar('[')){
             tempList.append(portData);
             arrayFirst=1;
             }else if(portData==QChar('*') && arrayFirst!=0){
               emit arraySplit(tempList);
                  tempList.clear();
                  arrayFirst=0;
}else if(arrayFirst==1){
       tempList.append(portData);
             }
     
}

koahnig

@aktay

You are reading all bytes in the buffer with readAll() and you are outputting all those bytes with the next statement. Actually you are always reading a complete string and youare always outputting a complete string. Only the first case the loop trigger, you are probably using, is faster and readAll reads only one byte to the buffer.
Anything you add as processing will ultimately slow down you all other processing. In your case the loop reading the data is slowed by this additional code and suddenly your are reading more than one byte/character into your string. That's it.

However, what you need to check is that this code sample has a significant difference

@aktay said in Raspbbery Pi 3 Serial Port Performance Problem:

void MainWindow::portRead(){
       QByteArray portData;
          portData=serial->readAll();
qDebug()<<portData;  // I am writing the console here. Also I am sending data mysql and qwidget. (Via another Class and another function)
      
             if(portData==QChar('[')){
             tempList.append(portData);
             arrayFirst=1;
             }else if(portData==QChar('*') && arrayFirst!=0){
               emit arraySplit(tempList);
                  tempList.clear();
                  arrayFirst=0;
}else if(arrayFirst==1){
       tempList.append(portData);
             }
     
}

to the initial code sample which is probably causing your confusion:
@aktay said in Raspbbery Pi 3 Serial Port Performance Problem:

void MainWindow::portRead(){
       QByteArray portData;
          portData=serial->readAll();
         for(int i=0;i<portData.size();i++){
             if(portData.at(i)==QChar('[')){
             tempList.append(portData.at(i));
             arrayFirst=1;
             }else if(portData.at(i)==QChar('*') && arrayFirst!=0){
               emit arraySplit(tempList);
                  tempList.clear();
                  arrayFirst=0;
             }else if(arrayFirst==1){
       tempList.append(portData.at(i));
             }
         }
}

Sometime obviously you have changed the code.

In your initial code you have this construct:

        QByteArray portData;
        portData=serial->readAll();                            // you read all bytes into a "string"
        for(int i=0;i<portData.size();i++){                  // you loop over all individual bytes 
            if(portData.at(i)==QChar('[')){                     // you access byte by byte and compare with another byte a QChar
                tempList.append(portData.at(i));          // you are pushing byte by byte into a container

In the code of your last post:

      QByteArray portData;
      portData=serial->readAll();                            // you read all bytes  into a "string" as above
      qDebug()<<portData;  // I am writing the console here. Also I am sending data mysql and qwidget. (Via another Class and another function)
                                                                                     // you print all bytes to qDebug
      if(portData==QChar('[')){                                 // you compare all bytes in portData with one byte (usually the compiler should give a warning)
          tempList.append(portData);                      //  you push all bytes of portData into the container

aktay

As I said at the beginning, if I use the loop, the data comes out late. For this reason I do not use that code. Because I have to take the data instantaneously without delay.
I am using code below in my program.

void MainWindow::portRead(){
       QByteArray portData;
          portData=serial->readAll();
             if(portData==QChar('[')){
             tempList.append(portData);
             arrayFirst=1;
             }else if(portData==QChar('*') && arrayFirst!=0){
               emit arraySplit(tempList);
                  tempList.clear();
                  arrayFirst=0;
             }else if(arrayFirst==1){
       tempList.append(portData);

koahnig

@aktay
Restructuring your code to make more readable:

void MainWindow::portRead()
{
    QByteArray portData;
    portData=serial->readAll();                                 // this statements whatever is in the buffer of QSerialPort may be 0 to almost infinity

// you are compring a string with a character.  Will work only when character is casted autmatically to string
// since the comparison is for strings you may compare "[100" with "[" which gives false. 
// Only the when you have read one byte and it happens to be "[" you will have true.
    if(portData==QChar('['))                                        
    {
        tempList.append(portData);
        arrayFirst=1;
    }
// The same here as explained for "[" but for "*"
    else if(portData==QChar('*') && arrayFirst!=0)
    {
        emit arraySplit(tempList);
        tempList.clear();
        arrayFirst=0;
    }
    else if(arrayFirst==1)
   {
       tempList.append(portData);

As already written above readAll reads everything in the buffer into a string. If you do so, you need to loop for checking your chars, but you cannot as you do at the moment. You do not compare as you expect.
BTW there are many other means to scan QByteArray e.g. http://doc.qt.io/qt-5/qbytearray.html#indexOf-1 for locating a character.

Most likely you have another issue. How do you trigger portRead?

Typically this is done with the signal readyRead from QSerialPort. However, it is not triggered for every byte, but may be triggered when several bytes are in the buffer. This is dependent on comm speed.

aktay

Thank you for your replies. I understand what you mean. You can see my serial port configuration below.

QSerialPort *serial;

    serial = new QSerialPort(this);
    serial->setPortName("ttyS0");
    serial->setBaudRate(QSerialPort::Baud9600);
    serial->setDataBits(QSerialPort::Data8);
    serial->setParity(QSerialPort::NoParity);
    serial->setStopBits(QSerialPort::OneStop);
    serial->setFlowControl(QSerialPort::NoFlowControl);
    serial->open(QIODevice::ReadOnly);
    connect(serial,SIGNAL(readyRead()),this, SLOT(portRead()));

As you say, I use the typical method of triggering. As you know, the PortRead function is as follows.

void MainWindow::portRead(){
       QByteArray portData;
          portData=serial->readAll();
             if(portData==QChar('[')){
             tempList.append(portData);
             arrayFirst=1;
             }else if(portData==QChar('*') && arrayFirst!=0){
               emit arraySplit(tempList);
                  tempList.clear();
                  arrayFirst=0;
             }else if(arrayFirst==1){
       tempList.append(portData);

How should I trigger portRead for read the data from the serial port as char? (Without delay.)
2.How can I clean the buffer? I tried the Clear & flush methods but it did not work.

Thanks.

jsulm

@aktay said in Raspbbery Pi 3 Serial Port Performance Problem:

How should I trigger portRead for read the data from the serial port as char

I think you can't. Just use the loop in portRead().

koahnig

@aktay

OK, you are triggering portRead through readyRead signal and this is correct. The question was more that you are triggering with the right method.

Now some people assume that readyRead() is triggered in regularly e.g. on a byte basis or record basis. However, just to ensure that is not the case. Triggering on a byte basis would probably overflow the event loop for fast connections and recordwise is not possible either, since how should any device decide on a binary level when a record ends.
Without knowing the exact details of triggering a signal readyRead, you may assume that it is triggered when new data arrives and you are not in a slot and subsequent routines triggered by readyRead signal.

You started to see "problems" when you suddenly did not receive at each call of readyRead one byte. This is completely normal when your additional functionality pick up cpu resources.

In your case you are seeing more than one byte in the string returned by readAll after you have added functionality.

As already written before

@koahnig said in Raspbbery Pi 3 Serial Port Performance Problem:

You are reading all bytes in the buffer with readAll() and you are outputting all those bytes with the next statement. Actually you are always reading a complete string and youare always outputting a complete string. Only the first case the loop trigger, you are probably using, is faster and readAll reads only one byte to the buffer.

Now you have two choices:

emptying the QSerialPort buffer at once with readAll as you do. This implies that you are required to some looping afterwards and handling individual bytes as you had done in your initial post.
Looping around reading the QSerialPort buffer and use read and read byte by byte

Just neglecting a loop will not get you anywhere you like to be. Read also the comments I had added to your code here

@koahnig said in Raspbbery Pi 3 Serial Port Performance Problem:

void MainWindow::portRead()
{
QByteArray portData;
portData=serial->readAll(); // this statements whatever is in the buffer of QSerialPort may be 0 to almost infinity

// you are compring a string with a character. Will work only when character is casted autmatically to string
// since the comparison is for strings you may compare "[100" with "[" which gives false.
// Only the when you have read one byte and it happens to be "[" you will have true.
if(portData==QChar('['))
{
tempList.append(portData);
arrayFirst=1;
}
// The same here as explained for "[" but for ""
else if(portData==QChar('') && arrayFirst!=0)
{
emit arraySplit(tempList);
tempList.clear();
arrayFirst=0;
}
else if(arrayFirst==1)
{
tempList.append(portData);

One last thing you need to understand also is that when you use readAll at the beginning of the routine, the buffer is completely emptied at that point of time. Your processing in the routine and subsequent routines may take a while. Any byte stored into the buffer during your processing will most likely not create a new readyRead signal. Therefore, it is advisable to check the buffer's size with bytesAvailable at the end in order to make that you are not missing information which will pop up with the next signal readyRead.