How to properly structure QtTest project?

Asperamanca

I have a small project with a few classes that I want to test. To me, it would make perfect sense to have one test class per production code class. However, the QtTest framework does not really support this:

The macro QTEST_MAIN does not support multiple classes
There is an option to execute single test classes using QTest::qExec(). However, the docs caution that "For stand-alone test applications, this function should not be called more than once, as command-line options for logging test output to files and executing individual test functions will not behave correctly."

Possible solutions I have considered:

I could make one test application per class. However, I'm not eager to flood my folder with project files. I tend to write lots of small classes. Also, I would need to switch applications a lot.
I could put everything in a single test class, but that feels like following the god class pattern.
I have found discussion how to solve this through code (e.g. on stackoverflow), but it still feels that this is an unintended way to use QTest

What is the intended way to structure QtTest projects in such a case?

SGaist

Hi,

One possible way would be to have one unit test per functionality rather than per class. Then you can test all the classes related to that functionality in one unit test.

Hope it helps

Paul Colby

Hi @Asperamanca,

There is an option to execute single test classes using QTest::qExec(). However, the docs caution that "For stand-alone test applications, this function should not be called more than once, as command-line options for logging test output to files and executing individual test functions will not behave correctly."

While that is technically correct, you can always manipulate the arguments before passing them to QTest::qExec() (which is exactly what I do in a few projects - more details below).

I could make one test application per class.

This is the recommended approach, and exactly what Qt does internally. See https://github.com/qt/qtbase/tree/5.11/tests/auto for example.

However, I'm not eager to flood my folder with project files. I tend to write lots of small classes. Also, I would need to switch applications a lot.

Understandable. It does depend on the size, type and style of the project. But I don't think its as ugly as it first seems - its still the same number of source files, encourages a logical directory layout, and the separate project files can become very useful later, for example making test application rebuilds much much faster (for very large projects) giving you quicker dev-test turn around cycles.

Also note, when using subdir projects with QMake, you will get a top-level check target that executes all test sub-projects (in addition to the check targets you'd expect on the individual test projects). Using separate test applications per class would be a real pain otherwise.

I could put everything in a single test class, but that feels like following the god class pattern.

That would definitely feel like an anti-pattern to me. Worth avoiding. (Would be appropriate for some functional tests though).

I have found discussion how to solve this through code (e.g. on stackoverflow), but it still feels that this is an unintended way to use QTest

Hmm... yeah, that approach doesn't handle the command line arguments at all. So, for example, if you wanted to run: myTests myTestFunction:myTestData, it would fail unless every test class implements myTestFunction, which is very rarely the case.

My personal recommendation is to use separate test classes, the way Qt itself does, most of the time - especially if this is some sort of shared library.

Otherwise, depending on the project style, I sometimes use a pretty neat little template test factory... You can see an example of how I do it here: https://github.com/pcolby/bipolar/blob/master/test/test.cpp

That solution recognises test class names on the command line, for example:

myTests # runs all tests
myTests TestClassA # runs all tests in TestClassA
myTests TestClassA testFunction # Runs just one test in TestClassA
myTests TestClassA testFunction::testData # Runs just one test in TestClassA, with just one data row.

It does this by recognising the test class argument, then executes just that test class (via QTest::qExec()), with the class name itself removed from the arguments list, so all other QTest built-in behaviours work the same.

(It also aggregates the test result counts at the end too).

Cheers.

Edit: PS You might find this related Qt feature request interesting reading too: QTest::qExec - better support for multiple test suites (QTBUG-23067)

Asperamanca

Thanks for the replies.

I'm very interested in the subdirs approach. Since I rarely work with subdirs, could you outline how the subdirs pro-file should look?

Also, if the subdirs executes multiple single project, do I get the time overhead from attaching the debugger (GDB in Creator is really slow) for every single project, or only once for the subdirs project?

Paul Colby

@Asperamanca said in How to properly structure QtTest project?:

Since I rarely work with subdirs, could you outline how the subdirs pro-file should look?

Have a read of http://doc.qt.io/qt-5/qmake-variable-reference.html#subdirs

At its most basic, its just TEMPLATE = subdirs and a list of sub-dirs like:

TEMPLATE = subdirs
SUBDIRS += test1 test2 test3

Have a look at Qt's auto unit tests for example: https://github.com/qt/qtbase/blob/5.11/tests/auto/auto.pro

Also, if the subdirs executes multiple single project, do I get the time overhead from attaching the debugger (GDB in Creator is really slow) for every single project, or only once for the subdirs project?

No sure. Never tried. But I don't think it makes to sense run all tests in the debugger. More likely, you should run all your tests normally, then only run the tests that fail in a debugger. In this sense, debugging is faster, because you're only debugging a much smaller binary. But if you want to run all tests in a debugger (not sure why) then I guess it probably would be a bit slower... maybe.

Cheers.