Running many qtest tests automatically (or not)
-
Hi there
I have several components in my projects (engine, app and separate qml subproject) and in the future may have even more (imagine plugins). All of these subprojects have own qtest-based test projects and whole project hierarchy looks as follows (I tried to clone the Qt's own structure):
@
proj\ app
\ engine qmlui tests
| |
| \ apptests
\ enginetests
qmluitests
@Everything works fine if I run each of the test binaries manually.
What is the way for automatically running all the tests in the whole tests subproject? Preferably with the common summary reporting in the end.
Or am I missing something basic and test structure is to be organized in the very different way?
-
P.S.
Ideally I would love to enable/disable some test projects conditionally. E.g. sometimes I'd want to run only fast unit tests, sometimes slow integration tests that connect to remote databases.But that's extra
-
AFAIK, QTestLib does not provide functionality for integration into test suites. QMake does not generate a "make test" target either. Some manual labor is required, for example a runner script that is executed by a custom "make test" target (see here: http://www.qtcentre.org/threads/24055-Unit-Test-Organisation-and-having-qmake-what-I-want).
One way is to use a common binary output folder like bin-tests/, name all your test binaries "test_<name>", and then execute all tests that have been built using bin-tests/test_*. In QMake, use conditionals to exclude test subdirs for fast test runs.
I know this sounds crazy. QMake should have proper support for generating test targets. After all, QTestLib is rather cool and part of Qt. Unfortunately, QMake does not have such a feature.
-
Thanks. I ended up creating a similar script. Starts all the executable tests for now, will probably modify it to launch only fast/slow tests when I need it.
It's a pity qtest doesn't provide and grouping/structuring for tests by default. Maybe because trolls use it for the whole Qt or the whole component only and they always run everything.
-
I know topic is old, but I just have to share a possible solution to that problem since no one mentioned it.
Qt provides an undocumented means to build and run tests without using scripts. Moreover, as most of Qt, it's a multiplatform solution.
Adding CONFIG += testcase to your .pro file will enablecheckbuild step in makefile.Checkwill run application after building. It also provides means to pass testrunner and arguments.In my case, all tests includes common.pri file, in which I set CONFIG += testcase. All test projects are build by using single
subdirsproject. Now, in QtCreator in build settings for mysubdirsproject I addcheckas an argument for Make in default Make step. Building that project will of course build all subprojects and pass check to them which will run each testcase.
I hope I described that clearly :).When invoking
make checkthere are 2 optional arguments:
TESTRUNNER and TESTARGS. If there are specified, TESTRUNNER is an executable, build target is first argument and TESTARGS are following arguments.
I don't remember where I found that, but it was in some kind of slideshow form conference maybe. -
I did set up everything like sidewinder wrote and noticed that there is a flaw: When one of the tests fails make just stops. No further tests are started and the compilation does not finish.
In the build settings I made a custom step which starts a shell script that invokes make check and manipulates the return status so that a following standard make step can be started in order to at least have a compiled program. The problem that not all tests are started in case of a error remains. Is there a solution for this ? Having something that behaves like Junit Tests would be nice.
E: Ok, I guess I figured it out: adding a -i arg to make ignores all errors in all rules and therefore all tests should run if there are no severe error during the comilation.