Unsolved How to manage incoming serial data to avoid data loss
-
Hi,
I have a program which is receiving some data continuously over serial port and I'm plotting data in real time. Since it has 8 channels, the part of filtering each line using the text at beginning and then emitting a signal to a slot, which plots the data taking long time in the later stages of the execution of program since the data array becomes huge. (Since, its a pollution measurement related project, normal execution ranges from 1 to 2 days continuously.)
So, the erroneous part is as follows:
As you can see below, as the execution of slot is delayed, the incoming data is saved in the buffer and when I read it, it becomes a big mess and my code cannot interpret that as it is written specifically to accept one channel data at any given time."23/10/19-07:43:03" Received line: "Flow:0x001" "23/10/19-07:43:04" Received line: "CH1:0x2009CF02" "23/10/19-07:43:04" Received line: "Flow:0x000" "23/10/19-07:43:04" Received line: "CH2:0x2007D202" "23/10/19-07:43:04" Received line: "Flow:0x000" "23/10/19-07:43:04" Received line: "CH3:0x20B8B402" "23/10/19-07:43:04" Received line: "Flow:0x000" "23/10/19-07:43:04" Received line: "CH4:0x20BC270B" "23/10/19-07:43:04" Received line: "Flow:0x000" "23/10/19-07:43:04" Received line: "CH5:0x200A260D" "23/10/19-07:43:05" Received line: "\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:0x000\r\nCH7:0x2009F502\r\nFlow:0x000\r\n"
Ideally, multi threading could've been a solution for this, but I found out that slots in same class cannot be multi threaded.
Will queuing the signal-slots help in this case? Or is there any other solution possible? -
I don't see a problem here - your data is separated by \r\n so you can scan for it and split your lines. You have to do it either way since you can't rely on the fact that every readyAll() gives you back a complete line since it's a stream.
-
@Ketank16 said in How to manage incoming serial data to avoid data loss:
as it is written specifically to accept one channel data at any given time
This is the problem. Just make it be able to deal with partial and buffered data. The fact that you receive exactly 1 channel of data at a time is purely a lucky cohincidence, you should handle the generic case rather than the exception
-
@VRonin By that line I mean I'm filtering the data using
if(Character == "\r" || Character == "\n" || Character == "\r\n")
where Character is conversion of incoming QByteArray into QString. Still I'm getting that big line because somehowreadAll()
is dumping all that data without parsing. -
Since the data looks like UTF-8 compatible text you can probably just use
QTextStream::readLineInto
as shown here to automatically split your data without usingreadAll()
-
@Ketank16 : I don't understand what you're doing - on the one hand your telling us that you're looking for \r / \n and on the other that you're using readAll() - so where do you actually look for \r \n then when not in an intermediate buffer?
-
@Christian-Ehrlicher
So here is my code:QByteArray byteArray = serial.readAll(); if(QString(byteArray) != NULL) { Character = QString(byteArray); } if(Character != "\r" && Character != "\n" && Character != "\r\n") { line.append(Character); } else if(Character == "\r" || Character == "\n" || Character == "\r\n") { //code }
I'm taking the characters first as long as it is not a \r or \n and the making lines out of it and processing it further. I've tried using
serial.readLine()
previously, but it also had same problem. -
@Ketank16
I don't get this. I don't know if I'm speaking out of turn, but how do you know whatserial.readAll()
will return at any given call? It could return any number of bytes, regardless of where your "lines" are. But you compare the result as though it's either 1 or 2 characters.Look at what your code does if the
readAll()
returns\nFlow:0x000\r\nCH6:0x21585908\r\nFlow:...
This is probably what @Christian-Ehrlicher was thinking of when he said:
so where do you actually look for \r \n then when not in an intermediate buffer?
You need to do some buffering here, the
readAll()
s go to buffer, that's what you search for\r
or\n
s. No? Look at @VRonin's suggestion of wrapping aQTextStream
round your input. -
QTextStream stream(serial); QString tempLine; for(;;){ serial->startTransaction(); if(stream.readLineInto(&tempLine)){ line.append(tempLine); serial->commitTransaction(); } else{ serial->rollbackTransaction(); break; } }
-
Thank you @JonB .
So @VRonin I followed your solution. Thanks a ton its so clean and useful.
I just wanted one help: now that I'm using stream, I'm getting this charactertempLine "\u0000"
when I'm trying to use qDebug().I searched on internet and its basically a NULL character but I'm not able to filter it by using
if(tempLine != NULL)
. Its still getting appended to line in the loop. Can you help with this small cosmetic part of code? -
what does
templine.size()
return? -
@Ketank16
"\u0000"
is a NUL [sic., not NULL] character. That has nothing to do with the NULL [sic.] pointer value. In other words, a character whose ASCII(?)/Unicode?/UTF-something(?) value is zero. It looks to me as though that character is actually in the input stream afterFlow:
, sometimes (and other times that is followed by character0x001
, a character with the value one, so presumablyFlow:
is followed by a byte of 0 or 1).You should reply to @VRonin's latest question about
templine.size()
in case that is relevant. -
Thanks @JonB for detailed explanation. As @VRonin said, I tried
templine.size()
and it is 1.EDIT: I fixed it from the source of serial data. I'd like to ask if there's any way to mask the QIODevice errors flooding my output with
Called while transaction already in progress
. I'm using QSerialPort'sreadyRead()
signal and using VRonin's code in my slot.Though I noticed in the stream that the number after flow is received as two bytes always.tempLine "F" 1
tempLine "l" 1
tempLine "o" 1
tempLine "w" 1
tempLine ":" 1
tempLine "0" 1
tempLine "x" 1
tempLine "00" 2
tempLine "0" 1
tempLine "\u0000" 1
tempLine "\u0000" 1So VRonin's suggestion doesn't look relevant in my case. I'll see what can be done with the flow reading from source of this stream i.e. my microchip board and code running on it. -
@Ketank16 said in How to manage incoming serial data to avoid data loss:
I'd like to ask if there's any way to mask the QIODevice errors flooding my output with Called while transaction already in progress. I'm using QSerialPort's readyRead() signal and using VRonin's code in my slot.
I would be wondering why you are getting this error rather than trying to remove the warnings?
-
I've added the readyRead() signal and connected it to the function:
connect(&serialport, SIGNAL(readyRead()), this, SLOT(framesAvailable()));
Inside the slot is the function given by VRonin + few of my adjustments to filter out NUL characters.
And also I use the filterif(serialport.isTransactionStarted() == false)
before the slot starts executing to make sure that I can avoid these warnings. But, I still get the warnings and right before the function I addedqDebug() << "serialport.isTransactionStarted() = " << serialport.isTransactionStarted();
just to check whether it is detecting the current ongoing transaction, but it seems thatisTransactionStarted()
is always false in my case.serialport.isTransactionStarted() = false QIODevice::startTransaction (QSerialPort): Called while transaction already in progress QIODevice::startTransaction (QSerialPort): Called while transaction already in progress QIODevice::startTransaction (QSerialPort): Called while transaction already in progress QIODevice::startTransaction (QSerialPort): Called while transaction already in progress serialport.isTransactionStarted() = false QIODevice::startTransaction (QSerialPort): Called while transaction already in progress QIODevice::startTransaction (QSerialPort): Called while transaction already in progress QIODevice::startTransaction (QSerialPort): Called while transaction already in progress QIODevice::startTransaction (QSerialPort): Called while transaction already in progress serialport.isTransactionStarted() = false
Do you know any workaround?
-
Can you show us the code of the slot you currently have?
-
void serial::framesAvailable() { qDebug() << "serialport.isTransactionStarted() = " << serialport.isTransactionStarted(); //QTextStream stream(serial); if(serialport.isTransactionStarted() == false) { for(;;) { serialport.startTransaction(); if(stream.readLineInto(&tempLine)) { if(tempLine != '\0' && tempLine.size() == 1) { //qDebug() << "tempLine " << tempLine << tempLine.size(); line.append(tempLine); serialport.commitTransaction(); } } else { serialport.rollbackTransaction(); break; } } //qDebug()<< "line " << line; if(line.size() == 192 && line.startsWith("CH")) { divided_line = line; line.clear(); //qDebug()<< "line " << divided_line; emit linereceived(divided_line); } } }