How to manage incoming serial data to avoid data loss

[VRonin]

@Ketank16 said in How to manage incoming serial data to avoid data loss:

as it is written specifically to accept one channel data at any given time

This is the problem. Just make it be able to deal with partial and buffered data. The fact that you receive exactly 1 channel of data at a time is purely a lucky cohincidence, you should handle the generic case rather than the exception

[Ketank16]

@VRonin By that line I mean I'm filtering the data using if(Character == "\r" || Character == "\n" || Character == "\r\n") where Character is conversion of incoming QByteArray into QString. Still I'm getting that big line because somehow readAll() is dumping all that data without parsing.

[VRonin]

Since the data looks like UTF-8 compatible text you can probably just use QTextStream::readLineInto as shown here to automatically split your data without using readAll()

[Christian Ehrlicher]

@Ketank16 : I don't understand what you're doing - on the one hand your telling us that you're looking for \r / \n and on the other that you're using readAll() - so where do you actually look for \r \n then when not in an intermediate buffer?

[Ketank16]

@Christian-Ehrlicher
So here is my code:

QByteArray byteArray = serial.readAll();
if(QString(byteArray) != NULL)
    {
        Character = QString(byteArray);
    }
    
    if(Character != "\r" && Character != "\n" && Character != "\r\n")
    {
        line.append(Character);
    }
    else if(Character == "\r" || Character == "\n" || Character == "\r\n")
    {
         //code
    }

I'm taking the characters first as long as it is not a \r or \n and the making lines out of it and processing it further. I've tried using serial.readLine() previously, but it also had same problem.

[JonB]

@Ketank16
I don't get this. I don't know if I'm speaking out of turn, but how do you know what serial.readAll() will return at any given call? It could return any number of bytes, regardless of where your "lines" are. But you compare the result as though it's either 1 or 2 characters.

Look at what your code does if the readAll() returns

\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:...

This is probably what @Christian-Ehrlicher was thinking of when he said:

so where do you actually look for \r \n then when not in an intermediate buffer?

You need to do some buffering here, the readAll()s go to buffer, that's what you search for \r or \ns. No? Look at @VRonin's suggestion of wrapping a QTextStream round your input.

[VRonin]

QTextStream stream(serial);
QString tempLine;
for(;;){
serial->startTransaction();
if(stream.readLineInto(&tempLine)){
line.append(tempLine);
serial->commitTransaction();
}
else{
serial->rollbackTransaction();
break;
}
}

[Ketank16]

Thank you @JonB .

So @VRonin I followed your solution. Thanks a ton its so clean and useful.
I just wanted one help: now that I'm using stream, I'm getting this character tempLine "\u0000" when I'm trying to use qDebug().

I searched on internet and its basically a NULL character but I'm not able to filter it by using if(tempLine != NULL) . Its still getting appended to line in the loop. Can you help with this small cosmetic part of code?

[VRonin]

what does templine.size() return?

[JonB]

@Ketank16
"\u0000" is a NUL [sic., not NULL] character. That has nothing to do with the NULL [sic.] pointer value. In other words, a character whose ASCII(?)/Unicode?/UTF-something(?) value is zero. It looks to me as though that character is actually in the input stream after Flow:, sometimes (and other times that is followed by character 0x001, a character with the value one, so presumably Flow: is followed by a byte of 0 or 1).

You should reply to @VRonin's latest question about templine.size() in case that is relevant.

[Ketank16]

Thanks @JonB for detailed explanation. As @VRonin said, I tried templine.size() and it is 1.

EDIT: I fixed it from the source of serial data. I'd like to ask if there's any way to mask the QIODevice errors flooding my output with Called while transaction already in progress. I'm using QSerialPort's readyRead() signal and using VRonin's code in my slot.

Though I noticed in the stream that the number after flow is received as two bytes always.

tempLine "F" 1
tempLine "l" 1
tempLine "o" 1
tempLine "w" 1
tempLine ":" 1
tempLine "0" 1
tempLine "x" 1
tempLine "00" 2
tempLine "0" 1
tempLine "\u0000" 1
tempLine "\u0000" 1

So VRonin's suggestion doesn't look relevant in my case. I'll see what can be done with the flow reading from source of this stream i.e. my microchip board and code running on it.

[JonB]

@Ketank16 said in How to manage incoming serial data to avoid data loss:

I'd like to ask if there's any way to mask the QIODevice errors flooding my output with Called while transaction already in progress. I'm using QSerialPort's readyRead() signal and using VRonin's code in my slot.

I would be wondering why you are getting this error rather than trying to remove the warnings?

[Ketank16]

I've added the readyRead() signal and connected it to the function: connect(&serialport, SIGNAL(readyRead()), this, SLOT(framesAvailable()));

Inside the slot is the function given by VRonin + few of my adjustments to filter out NUL characters.
And also I use the filter if(serialport.isTransactionStarted() == false) before the slot starts executing to make sure that I can avoid these warnings. But, I still get the warnings and right before the function I added qDebug() << "serialport.isTransactionStarted() = " << serialport.isTransactionStarted(); just to check whether it is detecting the current ongoing transaction, but it seems that isTransactionStarted() is always false in my case.

serialport.isTransactionStarted() =  false
QIODevice::startTransaction (QSerialPort): Called while transaction already in progress
QIODevice::startTransaction (QSerialPort): Called while transaction already in progress
QIODevice::startTransaction (QSerialPort): Called while transaction already in progress
QIODevice::startTransaction (QSerialPort): Called while transaction already in progress
serialport.isTransactionStarted() =  false
QIODevice::startTransaction (QSerialPort): Called while transaction already in progress
QIODevice::startTransaction (QSerialPort): Called while transaction already in progress
QIODevice::startTransaction (QSerialPort): Called while transaction already in progress
QIODevice::startTransaction (QSerialPort): Called while transaction already in progress
serialport.isTransactionStarted() =  false

Do you know any workaround?

[VRonin]

Can you show us the code of the slot you currently have?

[Ketank16]

@VRonin

void serial::framesAvailable()
{

    qDebug() << "serialport.isTransactionStarted() = " << serialport.isTransactionStarted();


    //QTextStream stream(serial);

    if(serialport.isTransactionStarted() == false)
    {
        for(;;)
        {
            serialport.startTransaction();
            if(stream.readLineInto(&tempLine))
            {
                if(tempLine != '\0' && tempLine.size() == 1)
                {
                    //qDebug() << "tempLine " << tempLine << tempLine.size();
                    line.append(tempLine);
                    serialport.commitTransaction();
                }
            }
            else
            {
                serialport.rollbackTransaction();
                break;
            }
        }

        //qDebug()<< "line " << line;
        if(line.size() == 192 && line.startsWith("CH"))
        {
            divided_line = line;
            line.clear();
            //qDebug()<< "line " << divided_line;
            emit linereceived(divided_line);
        }
    }
}