Terminal Example - read problem

[bzgec]

Hello.

My application is that when character is written to serial device, the device responds with received character and new LED state.
With Putty terminal i have no problem, here is an example of how it should work:

Here is also the Putty configuration:

The problem is with Qt terminal example. I downloaded the example code from GitHub repo, branch 5.15.1. Here is the output that i am getting:

The send character is always received but LED state string is corrupted most of the time.
Otherwise if Qt terminal is just receiving data there is no problem (connected device sends data automatically every 500ms).

What could be done to save this problem, also i didn't change any code from the terminal example?

Thank you for any answers.

[hskoglund]

Hi, interesting problem! Do you have Local echo enabled? if so, try turning it off. It might be that your keypresses and what's received collide somehow in the QPlainTextEdit window.

[bzgec]

Local echo is disabled, i also tried to output data with qDebug() but result is the same.

[hskoglund]

Hmm maybe the scrolling disturbs the reception of characters, what happens if you remove the linefeeds, say by changing the Console::putData() function (on line 66 in console.cpp) to this:

void Console::putData(const QByteArray &data)
{
    QString s = data;
    s = s.simplified();
    insertPlainText(s);
}

[mrdebug]

Maybe the problem is that simple terminal appends a new line each time On ReadyRead event is fired and sometime the event is fired two time on one line in, the first time for the first part of the row and the second time on the second part of the row.

Please try this
https://github.com/denisgottardello/QtComPort
in hexadecimal mode and using new line after 200 ms

[bzgec]

@hskoglund same problem but everything in one line.

[bzgec]

@mrdebug When i try to import this project in Qt creator it throws error:

No valid settings file could be found.

All settings files found in directory "x/QtComPort" where either too new or too old to be read

If i proceed and try to build and run the project it throws error:

This application failed to start because it could not find or load the Qt platform plugin "xcb"
in "".

[bzgec]

Ok so i put delay before sending LED status from device (Nucleo board) and it works fine (if delay is bigger than 2ms). So the problem is that Nucleo board is sending second message right after the first one with no delay between them.
But that is not a solution to my problem (it works in Putty why not in Terminal example).

EDIT:
I also tried to send messages from Nucleo board without '\n' but it makes no difference.

[hskoglund]

So the Terminal example program only works if the device pauses for > 2 ms (but Putty works anyway). It looks like when QSerialPort::readyRead() fires and the readAll() is issued, the program is oblivious of new characters for 2 ms :-(

One way is to simplify the technology, instead of relying on the readyRead() signal, you could try polling with a QTimer. As a start (just guessing!) if you change the MainWindow::readData() function to this:

void MainWindow::readData()
{
    for (;;)
    {
        const QByteArray data = m_serial->read(1);
        if (data.isEmpty())
            return;
        
        m_console->putData(data);
    }

does it change the behavior?

[bzgec]

@hskoglund Awsome, this works!

void MainWindow::readData()
{
    for (;;)
    {
        const QByteArray data = m_serial->read(1);
        if (data.isEmpty())
            return;

        m_console->putData(data);
    }
}

If i understand correctly this is still an asynchronous way which just checks if more data is available after reading one character instead of reading every characters it received in one go. And this helps because the QSerialPort::readyRead() doesn't fire fast enough (or corresponding slot doesn't execute fast enough) and by the time readAll() is called internal buffer is already overwritten with new data?

[kuzulis]

@bzgec ,

is already overwritten with new data?

No, that's not possible, the QSP never overwrites the previous data, it just appends.

Could you please do the following test (please add the qdebug output to see what happpens):

void MainWindow::readData()
{
    qDebug() << "Bytes available:" << m_serial->bytesAvailable();
    const QByteArray data = m_serial->readAll();
    qDebug() << "Data:" << data.toHex();
    ...
}

[hskoglund]

Hi, read the documentation for the readyRead() signal and it says: "....readyRead() is not emitted recursively..."
Doesn't that mean there's a theoretical "blind spot" in the readData() function:

oid MainWindow::readData()
{
    const QByteArray data = m_serial->readAll();
// begin "blind spot"
    m_console->putData(data);
// end of "blind spot"
}

I.e. if new characters arrive exactly at the same time as the ->putData(data) function runs, no readyRead() signal is fired (because we haven't got the hell out of Dodge/left readData()?
But the characters should not be lost anyway, just queued somewhere in a buffer, as @kuzulis says above *... never overwrites... , just appends." Maybe the Terminal example works better with another OS than your Linux, if you have a Windows PC around, you could try with that.

[JonB]

@hskoglund
Do you know what you have said about "blind spot" to be true/the case? It would be almost a show-stopper if it did! I don't think there will be any "blind spot", so far as Qt is concerned I don't think it will report any data as "arriving" while you are inside readData(), any new data which arrives after readAll() should cause a new readyRead signal to be raised after you exit this slot? Note that readyRead is not emitted a second time until after readAll() has been performed, so user code must read all already-arrived data before he will get a new notification for newly-arriving data.

[kuzulis]

An idea is that the QSP does work in a buffered mode (let's say how it does work, to be clear):

1. When the FIFO receives at leas one byte, the serial port file descriptor gets trigegred with the 'RX' event.

2. This event calls the 'data read' callback which reads all available data from the FIFO to the internal QSP buffer.

3. If a bytes read more than zero, then emits the readyRead() signal.

4. The QSP::read() or QSP::readAll() methods read the data from the internal buffer of QSP (but not from the device's FIFO).

5. The QSP::bytesAvailable() method returns a size of the internal QSP buffer (not the data count from the device's FIFO).

6. If a new data gets arrived into the device's FIFO after the QSP already did read from the FIFO, then the 'RX' event will be triggered in next time when the application runs the next Qt-event loop. The 'RX' event will be triggered in each Qt-event loop until the device's FIFO contains at least one byte.

So, please provide the debug output for the 'bytesAvailable()' and the 'readAll()' after each readyRead() signal to see what bytes are received.

[bzgec]

@kuzulis I tried your suggestion:

[hskoglund]

Well well well @kuzulis your idea with the hex dump really did the trick!
It seems that the problem isn't at all related to any serial communication but rather what happens when you stuff a QString with null bytes :-)

I.e. try tossing out the zero bytes at reception, something like thisL

void MainWindow::readData()
{
    QByteArray dataWithZeroes = m_serial->readAll();
    QByteArray data;
    for (auto b : dataWithZeroes)
         if (b)
            data += b;
             
    m_console->putData(data);
}

[JonB]

@hskoglund

 QByteArray data =  m_serial->readAll().replace('\0', "");

? Totally untested, but a lot shorter ;-)

[bzgec]

@kuzulis your idea to display hex data was quite good :)
Examples from @hskoglund and @JonB both work.
So the problem is that when "0\n" or "1\n" string is send and the new string is sent right behind them. Then readAll() reads whole buffer including string termination sign and other data behind it, so conversion to string (or when displaying to console) drops other characters behind terminating sign.

But i what about readLine(). If i try @hskoglund and @JonB example it works, but with original terminal example it completely drops everything right behind "0\n" or "1\n".

[JonB]

@bzgec said in Terminal Example - read problem:

so conversion to string (or when displaying to console) drops other characters behind terminating sign.

Then don't do so! Converting bytes containing \0s to strings is "problematic", as is displaying them. Do all your work on QByteArrays, and use the returned count not nul-string termination.

I haven't looked, but readLine() is liable to want to find \ns, and not treat \0s specially, or well in a string. If you have to do that, split your input on \0s as necessary. or remove them as we did.

[bzgec]

@JonB Yes i tried readLine() (and posted picture in previous answer) and the problem is the same as with readAll().

Ok so the problem was that the device (Nucleo board) was sending null-string termination in a message. And conversion to string (or when displaying to console) dropped everything behind it and that is why this @hskoglund's example worked (because it displayed one character at the time):

void MainWindow::readData()
{
    for (;;)
    {
        const QByteArray data = m_serial->read(1);
        if (data.isEmpty())
            return;

        m_console->putData(data);
    }
}

So my guess is that Putty uses one of the examples above. Which one would you recommend to use so that mistake like this doesn't show in console?

Thank you all for helping!

[kuzulis]

The QSP terminal example assumes that a received data contains in the ASCII table, that it is a visible symbols. The terminal example it is just a "simple example", and it does not handles the non-visual symbols yet. :)

[bzgec]

Ok so in my console application i am going to use code where you can select to filter \0 and \r. This option is checked in settings dialog (next to Local echo checkbox). Do you guys think that this code is OK or should i use some other option for filtering \0 and \r?

void mainWindow::readData() {
    QByteArray data;

    data = serialPort->readAll();
    if(settings->settings().filter_r == true) {
        data.replace('\r', "");
    }
    if(settings->settings().filter_0 == true) {
        data.replace('\0', "");
    }
    ui->console->putData(data);
}