How to manage incoming serial data to avoid data loss

VRonin

Since the data looks like UTF-8 compatible text you can probably just use QTextStream::readLineInto as shown here to automatically split your data without using readAll()

Christian Ehrlicher

@Ketank16 : I don't understand what you're doing - on the one hand your telling us that you're looking for \r / \n and on the other that you're using readAll() - so where do you actually look for \r \n then when not in an intermediate buffer?

Ketank16

@Christian-Ehrlicher
So here is my code:

QByteArray byteArray = serial.readAll();
if(QString(byteArray) != NULL)
    {
        Character = QString(byteArray);
    }
    
    if(Character != "\r" && Character != "\n" && Character != "\r\n")
    {
        line.append(Character);
    }
    else if(Character == "\r" || Character == "\n" || Character == "\r\n")
    {
         //code
    }

I'm taking the characters first as long as it is not a \r or \n and the making lines out of it and processing it further. I've tried using serial.readLine() previously, but it also had same problem.

JonB

@Ketank16
I don't get this. I don't know if I'm speaking out of turn, but how do you know what serial.readAll() will return at any given call? It could return any number of bytes, regardless of where your "lines" are. But you compare the result as though it's either 1 or 2 characters.

Look at what your code does if the readAll() returns

\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:...

This is probably what @Christian-Ehrlicher was thinking of when he said:

so where do you actually look for \r \n then when not in an intermediate buffer?

You need to do some buffering here, the readAll()s go to buffer, that's what you search for \r or \ns. No? Look at @VRonin's suggestion of wrapping a QTextStream round your input.

VRonin

QTextStream stream(serial);
QString tempLine;
for(;;){
serial->startTransaction();
if(stream.readLineInto(&tempLine)){
line.append(tempLine);
serial->commitTransaction();
}
else{
serial->rollbackTransaction();
break;
}
}

Ketank16

Thank you @JonB .

So @VRonin I followed your solution. Thanks a ton its so clean and useful.
I just wanted one help: now that I'm using stream, I'm getting this character tempLine "\u0000" when I'm trying to use qDebug().

I searched on internet and its basically a NULL character but I'm not able to filter it by using if(tempLine != NULL) . Its still getting appended to line in the loop. Can you help with this small cosmetic part of code?

VRonin

what does templine.size() return?

JonB

@Ketank16
"\u0000" is a NUL [sic., not NULL] character. That has nothing to do with the NULL [sic.] pointer value. In other words, a character whose ASCII(?)/Unicode?/UTF-something(?) value is zero. It looks to me as though that character is actually in the input stream after Flow:, sometimes (and other times that is followed by character 0x001, a character with the value one, so presumably Flow: is followed by a byte of 0 or 1).

You should reply to @VRonin's latest question about templine.size() in case that is relevant.

Ketank16

Thanks @JonB for detailed explanation. As @VRonin said, I tried templine.size() and it is 1.

EDIT: I fixed it from the source of serial data. I'd like to ask if there's any way to mask the QIODevice errors flooding my output with Called while transaction already in progress. I'm using QSerialPort's readyRead() signal and using VRonin's code in my slot.

~~Though I noticed in the stream that the number after flow is received as two bytes always.~~

tempLine "F" 1
tempLine "l" 1
tempLine "o" 1
tempLine "w" 1
tempLine ":" 1
tempLine "0" 1
tempLine "x" 1
tempLine "00" 2
tempLine "0" 1
tempLine "\u0000" 1
tempLine "\u0000" 1

~~So VRonin's suggestion doesn't look relevant in my case. I'll see what can be done with the flow reading from source of this stream i.e. my microchip board and code running on it.~~

JonB

@Ketank16 said in How to manage incoming serial data to avoid data loss:

I'd like to ask if there's any way to mask the QIODevice errors flooding my output with Called while transaction already in progress. I'm using QSerialPort's readyRead() signal and using VRonin's code in my slot.

I would be wondering why you are getting this error rather than trying to remove the warnings?

Ketank16

I've added the readyRead() signal and connected it to the function: connect(&serialport, SIGNAL(readyRead()), this, SLOT(framesAvailable()));

Inside the slot is the function given by VRonin + few of my adjustments to filter out NUL characters.
And also I use the filter if(serialport.isTransactionStarted() == false) before the slot starts executing to make sure that I can avoid these warnings. But, I still get the warnings and right before the function I added qDebug() << "serialport.isTransactionStarted() = " << serialport.isTransactionStarted(); just to check whether it is detecting the current ongoing transaction, but it seems that isTransactionStarted() is always false in my case.

serialport.isTransactionStarted() =  false
QIODevice::startTransaction (QSerialPort): Called while transaction already in progress
QIODevice::startTransaction (QSerialPort): Called while transaction already in progress
QIODevice::startTransaction (QSerialPort): Called while transaction already in progress
QIODevice::startTransaction (QSerialPort): Called while transaction already in progress
serialport.isTransactionStarted() =  false
QIODevice::startTransaction (QSerialPort): Called while transaction already in progress
QIODevice::startTransaction (QSerialPort): Called while transaction already in progress
QIODevice::startTransaction (QSerialPort): Called while transaction already in progress
QIODevice::startTransaction (QSerialPort): Called while transaction already in progress
serialport.isTransactionStarted() =  false

Do you know any workaround?

VRonin

Can you show us the code of the slot you currently have?

Ketank16

@VRonin

void serial::framesAvailable()
{

    qDebug() << "serialport.isTransactionStarted() = " << serialport.isTransactionStarted();


    //QTextStream stream(serial);

    if(serialport.isTransactionStarted() == false)
    {
        for(;;)
        {
            serialport.startTransaction();
            if(stream.readLineInto(&tempLine))
            {
                if(tempLine != '\0' && tempLine.size() == 1)
                {
                    //qDebug() << "tempLine " << tempLine << tempLine.size();
                    line.append(tempLine);
                    serialport.commitTransaction();
                }
            }
            else
            {
                serialport.rollbackTransaction();
                break;
            }
        }

        //qDebug()<< "line " << line;
        if(line.size() == 192 && line.startsWith("CH"))
        {
            divided_line = line;
            line.clear();
            //qDebug()<< "line " << divided_line;
            emit linereceived(divided_line);
        }
    }
}