How to manage incoming serial data to avoid data loss

Ketank16

Hi,

I have a program which is receiving some data continuously over serial port and I'm plotting data in real time. Since it has 8 channels, the part of filtering each line using the text at beginning and then emitting a signal to a slot, which plots the data taking long time in the later stages of the execution of program since the data array becomes huge. (Since, its a pollution measurement related project, normal execution ranges from 1 to 2 days continuously.)

So, the erroneous part is as follows:
As you can see below, as the execution of slot is delayed, the incoming data is saved in the buffer and when I read it, it becomes a big mess and my code cannot interpret that as it is written specifically to accept one channel data at any given time.

"23/10/19-07:43:03" Received line:  "Flow:0x001"
"23/10/19-07:43:04" Received line:  "CH1:0x2009CF02"
"23/10/19-07:43:04" Received line:  "Flow:0x000"
"23/10/19-07:43:04" Received line:  "CH2:0x2007D202"
"23/10/19-07:43:04" Received line:  "Flow:0x000"
"23/10/19-07:43:04" Received line:  "CH3:0x20B8B402"
"23/10/19-07:43:04" Received line:  "Flow:0x000"
"23/10/19-07:43:04" Received line:  "CH4:0x20BC270B"
"23/10/19-07:43:04" Received line:  "Flow:0x000"
"23/10/19-07:43:04" Received line:  "CH5:0x200A260D"
"23/10/19-07:43:05" Received line:  "\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n"

Ideally, multi threading could've been a solution for this, but I found out that slots in same class cannot be multi threaded.
Will queuing the signal-slots help in this case? Or is there any other solution possible?

Christian Ehrlicher

I don't see a problem here - your data is separated by \r\n so you can scan for it and split your lines. You have to do it either way since you can't rely on the fact that every readyAll() gives you back a complete line since it's a stream.

VRonin

@Ketank16 said in How to manage incoming serial data to avoid data loss:

as it is written specifically to accept one channel data at any given time

This is the problem. Just make it be able to deal with partial and buffered data. The fact that you receive exactly 1 channel of data at a time is purely a lucky cohincidence, you should handle the generic case rather than the exception

Ketank16

@VRonin By that line I mean I'm filtering the data using if(Character == "\r" || Character == "\n" || Character == "\r\n") where Character is conversion of incoming QByteArray into QString. Still I'm getting that big line because somehow readAll() is dumping all that data without parsing.

VRonin

Since the data looks like UTF-8 compatible text you can probably just use QTextStream::readLineInto as shown here to automatically split your data without using readAll()

Christian Ehrlicher

@Ketank16 : I don't understand what you're doing - on the one hand your telling us that you're looking for \r / \n and on the other that you're using readAll() - so where do you actually look for \r \n then when not in an intermediate buffer?

Ketank16

@Christian-Ehrlicher
So here is my code:

QByteArray byteArray = serial.readAll();
if(QString(byteArray) != NULL)
    {
        Character = QString(byteArray);
    }
    
    if(Character != "\r" && Character != "\n" && Character != "\r\n")
    {
        line.append(Character);
    }
    else if(Character == "\r" || Character == "\n" || Character == "\r\n")
    {
         //code
    }

I'm taking the characters first as long as it is not a \r or \n and the making lines out of it and processing it further. I've tried using serial.readLine() previously, but it also had same problem.

JonB

@Ketank16
I don't get this. I don't know if I'm speaking out of turn, but how do you know what serial.readAll() will return at any given call? It could return any number of bytes, regardless of where your "lines" are. But you compare the result as though it's either 1 or 2 characters.

Look at what your code does if the readAll() returns

\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:...

This is probably what @Christian-Ehrlicher was thinking of when he said:

so where do you actually look for \r \n then when not in an intermediate buffer?

You need to do some buffering here, the readAll()s go to buffer, that's what you search for \r or \ns. No? Look at @VRonin's suggestion of wrapping a QTextStream round your input.

VRonin

QTextStream stream(serial);
QString tempLine;
for(;;){
serial->startTransaction();
if(stream.readLineInto(&tempLine)){
line.append(tempLine);
serial->commitTransaction();
}
else{
serial->rollbackTransaction();
break;
}
}

Ketank16

Thank you @JonB .

So @VRonin I followed your solution. Thanks a ton its so clean and useful.
I just wanted one help: now that I'm using stream, I'm getting this character tempLine "\u0000" when I'm trying to use qDebug().

I searched on internet and its basically a NULL character but I'm not able to filter it by using if(tempLine != NULL) . Its still getting appended to line in the loop. Can you help with this small cosmetic part of code?

VRonin

what does templine.size() return?

JonB

@Ketank16
"\u0000" is a NUL [sic., not NULL] character. That has nothing to do with the NULL [sic.] pointer value. In other words, a character whose ASCII(?)/Unicode?/UTF-something(?) value is zero. It looks to me as though that character is actually in the input stream after Flow:, sometimes (and other times that is followed by character 0x001, a character with the value one, so presumably Flow: is followed by a byte of 0 or 1).

You should reply to @VRonin's latest question about templine.size() in case that is relevant.

Ketank16

Thanks @JonB for detailed explanation. As @VRonin said, I tried templine.size() and it is 1.

EDIT: I fixed it from the source of serial data. I'd like to ask if there's any way to mask the QIODevice errors flooding my output with Called while transaction already in progress. I'm using QSerialPort's readyRead() signal and using VRonin's code in my slot.

~~Though I noticed in the stream that the number after flow is received as two bytes always.~~

tempLine "F" 1
tempLine "l" 1
tempLine "o" 1
tempLine "w" 1
tempLine ":" 1
tempLine "0" 1
tempLine "x" 1
tempLine "00" 2
tempLine "0" 1
tempLine "\u0000" 1
tempLine "\u0000" 1

~~So VRonin's suggestion doesn't look relevant in my case. I'll see what can be done with the flow reading from source of this stream i.e. my microchip board and code running on it.~~

JonB

@Ketank16 said in How to manage incoming serial data to avoid data loss:

I'd like to ask if there's any way to mask the QIODevice errors flooding my output with Called while transaction already in progress. I'm using QSerialPort's readyRead() signal and using VRonin's code in my slot.

I would be wondering why you are getting this error rather than trying to remove the warnings?

Ketank16

I've added the readyRead() signal and connected it to the function: connect(&serialport, SIGNAL(readyRead()), this, SLOT(framesAvailable()));

Inside the slot is the function given by VRonin + few of my adjustments to filter out NUL characters.
And also I use the filter if(serialport.isTransactionStarted() == false) before the slot starts executing to make sure that I can avoid these warnings. But, I still get the warnings and right before the function I added qDebug() << "serialport.isTransactionStarted() = " << serialport.isTransactionStarted(); just to check whether it is detecting the current ongoing transaction, but it seems that isTransactionStarted() is always false in my case.

serialport.isTransactionStarted() =  false
QIODevice::startTransaction (QSerialPort): Called while transaction already in progress
QIODevice::startTransaction (QSerialPort): Called while transaction already in progress
QIODevice::startTransaction (QSerialPort): Called while transaction already in progress
QIODevice::startTransaction (QSerialPort): Called while transaction already in progress
serialport.isTransactionStarted() =  false
QIODevice::startTransaction (QSerialPort): Called while transaction already in progress
QIODevice::startTransaction (QSerialPort): Called while transaction already in progress
QIODevice::startTransaction (QSerialPort): Called while transaction already in progress
QIODevice::startTransaction (QSerialPort): Called while transaction already in progress
serialport.isTransactionStarted() =  false

Do you know any workaround?

VRonin

Can you show us the code of the slot you currently have?

Ketank16

@VRonin

void serial::framesAvailable()
{

    qDebug() << "serialport.isTransactionStarted() = " << serialport.isTransactionStarted();


    //QTextStream stream(serial);

    if(serialport.isTransactionStarted() == false)
    {
        for(;;)
        {
            serialport.startTransaction();
            if(stream.readLineInto(&tempLine))
            {
                if(tempLine != '\0' && tempLine.size() == 1)
                {
                    //qDebug() << "tempLine " << tempLine << tempLine.size();
                    line.append(tempLine);
                    serialport.commitTransaction();
                }
            }
            else
            {
                serialport.rollbackTransaction();
                break;
            }
        }

        //qDebug()<< "line " << line;
        if(line.size() == 192 && line.startsWith("CH"))
        {
            divided_line = line;
            line.clear();
            //qDebug()<< "line " << divided_line;
            emit linereceived(divided_line);
        }
    }
}