How is Qt source code itself tested?

[J.Hilk]

@JonB said in How is Qt source code itself tested?:

So what happens about testing? Shove it out there and see if users notice anything? Serious question.

No idea, I'm asking that myself.
There are beta and prerelease phases for upcoming Qt Versions, but the number of people participating in those trials are minimal I assume.

I would at least expect an internal test of all Examples in the documentation.
That way the SerialBus bug would have been noticed, because the Simple Terminal example is broken because of it.

I noticed the bug from QModbus, that one isn't working via Serial Communication as well (in 5.13.1)

[aha_1980]

@JonB said in How is Qt source code itself tested?:

So (although it does not affect me) 5.13.1 seems to have completely broken anyone using serial ports. Pretty serious? Maybe not immediately noticeable to peoples' code which happens to use this somewhere?

Yeah that was a very bad example.

There are unit tests for core functions, which are run on every integration on several platforms, but obviously that's not enough.

Integration tests for hardware-near modules like QSerialPort or QSerialBus are difficult to have, as you would need hardware connected to the CI systems to fully test them.

And even if you test a lot, there might still be customers with a slightly other Windows version, an older driver or a newer firmware in the device. There are tons of possibilities that things go wrong.

As @J-Hilk said:

but the number of people participating in those trials are minimal I assume.

And because of that, most bugs appear on the final .0 release and are fixed on .1 or .2.

I guess that's the reason many people avoid .0 releases, which in turn gives them fewer testers and leaves the bugs undetected, and that leads to more buggy .1 releases.

I've no idea how to get out of this!

Regards

[J.Hilk]

@aha_1980
mmh,

I for once would test the prereleases more regularly if it would be more clear, when they are available and what modules exactly are changed (I would test explicitly those I use)

At the moment, the only information about a new version, I get from Twitter. Which does not inform you about prereleases.

[aha_1980]

Hi @J-Hilk,

see the following fresh blog post: https://www.qt.io/blog/qt-5.14.0-beta1-released

For new features there is a wiki page: https://wiki.qt.io/New_Features_in_Qt_5.14

The full change logs will be created in a few weeks, so unfortunately there is no full list of changes for every module yet.

Regards

[Pablo J. Rogina]

@aha_1980 said in How is Qt source code itself tested?:

And even if you test a lot, there might still be customers with a slightly other Windows version, an older driver or a newer firmware in the device.

Yes, you're right but it would be desirable to at least try testing a lot as you mention so when such customers with weird conditions appear having an issue, you at least have rule out lot of things.

There are tons of possibilities that things go wrong.

Yes, but if nothing is attempted / tested they'll be lying around. If you slowly but steady keep on adding tests you'll be narrowing your chances of regressions. Once you made an step in your journey, you don't want to step back...

[aha_1980]

Hi @Pablo-J-Rogina,

if you have a look at the QSerialPort auto tests you will see that there are already several tests, including write-read tests.

So it seems in that case it was rather a problem of actually running the tests on different machines.

Regards

[kuzulis]

@aha_1980 said in How is Qt source code itself tested?:

So it seems in that case it was rather a problem of actually running the tests on different machines.

No, a problem was in "trusting".

I'm sorry, but I'm did not tested that patch, because I was hoping (was trusted) for the person who submitted that patch (which broke the I / O in QSerialPort). That person is not a novice, it is a developer of that class for which that patch was directed. I assumed, that he already tested that patch before submitting to gerrit. It is my fault.

A main problem is that the Qt-guys discarted my suggestion about installing to the CI the virtual serial ports:

• On windows it is com0com
• On Linux it is tty0tty

it was a long time ago, you can found all in a mailing-list if you need.

So, installing that com0com && tty0tty drivers solves 99% of problems. QSerialPort already has an auto-tests (as said Andre before). But I am forced to do that auto-tests manually each time and each time, what bothers and takes a time.

[aha_1980]

@kuzulis said in How is Qt source code itself tested?:

A main problem is that the Qt-guys discarted my suggestion about installing to the CI the virtual serial ports:

Can you create a bugreport for that and assign the QA team? Then we can increase the pressure on them.

It is just a pity that the tests exist but cannot be run automatically.

[kuzulis]

https://bugreports.qt.io/browse/QTQAINFRA-3295

[fcarney]

com0com and com2tcp rock! Used those a lot in the past.