How to associate Qt unit testcase with an Id?

ur2cdanger

I am trying to make some changes to the existing Qt unit tests that we have. I am trying to associate each tests with an id so that it can be used for integrating with an external platform like Jira or any test tracking tool.

This is an example of what we have

TestCase {
    name: "Led"
    // Need a way to give id for this test. Something like an annotation like @testid(34) which will be given as output along with the test result
    function test_test1()
    {
        // some test
    }

    function test_test2()
    {

    }
 }

Is there any native Qt function/ any external way through which I can achieve this? There are two things I need. One is the id and the other is execution time for each tests. Id is the most important thing that I need. Any help would be grateful. Thanks.

Paul Colby

Hi @ur2cdanger,

If I understand you correctly, I would simply name the test class itself after the test case id / reference.

For example, if your test tracker has test case 123, with three tests - 1, 2 and 3, then I'd do something like:

class TestCase123 : public QObject {
    Q_OBJECT

private slots:
    void initTestCase();
    void test1();
    void test2();
    void test3();
};

Then your tests will output something like:

********* Start testing of TestCase123 *********
Config: Using QtTest library 5.5.1, Qt 5.5.1 (...)
PASS   : TestCase123::initTestCase()
PASS   : TestCase123::test1()
PASS   : TestCase123::test2()
PASS   : TestCase123::test3()
PASS   : TestCase123::cleanupTestCase()
Totals: 5 passed, 0 failed, 0 skipped, 0 blacklisted
********* Finished testing of TestCase123 *********

Or, for example, if using the -xunitxml outout:

<?xml version="1.0" encoding="UTF-8" ?>
<testsuite errors="0" failures="0" tests="5" name="TestCase123">
  <properties>
    <property value="5.5.1" name="QTestVersion"/>
    <property value="5.5.1" name="QtVersion"/>
  </properties>
  <testcase result="pass" name="initTestCase"/>
  <testcase result="pass" name="test1"/>
  <testcase result="pass" name="test2"/>
  <testcase result="pass" name="test3"/>
  <testcase result="pass" name="cleanupTestCase"/>
  <system-err/>
</testsuite>

and the other is execution time for each tests

The -xml output option gives you execution time for each of the test functions, eg:

<?xml version="1.0" encoding="UTF-8"?>
<TestCase name="TestCase123">
<Environment>
    <QtVersion>5.5.1</QtVersion>
    <QtBuild>Qt 5.5.1 (x86_64&#x002D;little_endian&#x002D;lp64 shared (dynamic) release build; by GCC 5.3.1 20160407)</QtBuild>
    <QTestVersion>5.5.1</QTestVersion>
</Environment>
<TestFunction name="initTestCase">
<Incident type="pass" file="" line="0" />
    <Duration msecs="0.002351"/>
</TestFunction>
<TestFunction name="test1">
<Incident type="pass" file="" line="0" />
    <Duration msecs="0.025596"/>
</TestFunction>
...
</TestCase>

Does that get you what you're after?

ur2cdanger

@PaulColby.

Thank you for your input. While the idea of naming the test case with the id seems tempting, it really dissolves the purpose of function name. Anyone who sees that function name would not understand anything unless they have the knowledge of the test tracker. Qt, with the framework that it is, should really have a way to associate an id with each tests. If no other way works out, I might have to use the way that you suggested but reluctantly.

Regarding the execution time, your solution seems very easy and that is what I exactly I want. I run with -xml option but I don't get the duration tag in the output. This is what I get.

<TestFunction name="test1">
<Incident type="pass" file="" line="0" />
</TestFunction>

Do I need to include anything in the function to get the duration? Thanks for your help.

Paul Colby

@ur2cdanger said:

While the idea of naming the test case with the id seems tempting, it really dissolves the purpose of function name. Anyone who sees that function name would not understand anything unless they have the knowledge of the test tracker.

Yeah, I can see how that would be a problem. There are a few ways you could annotate the test functions, but just looking at the source of the various test loggers, I can't see any way of including any such additional information in way they will actually be included in the output. Also, it doesn't appear that you can add your own custom logger, as both QTestResult and QTestLog (and derived classes) are all private.

It does sound like a good feature to add, so you might want to create a feature request for it.

Also, if you're using data functions, you might be able to get closer to what you want by having a consistent data tag naming convention. For example,

MyTestClass::myTestFunction_data() {
    #define TEST_REFERENCE "JIRA-123"

    QTest::addColumn<bool>("foo");
    QTest::newRow(TEST_REFERENCE ":true") << true;
    QTest::newRow(TEST_REFERENCE ":false") << false;
}

MyTestClass::myTestFunction() {
    ...
}

Will output something like:

PASS   : MyTestClass::myTestFunction(JIRA-123:true)
PASS   : MyTestClass::myTestFunction(JIRA-123:false)

or

<?xml version="1.0" encoding="UTF-8"?>
<TestCase name="MyTestClass">
<Environment>
    <QtVersion>5.5.1</QtVersion>
    <QtBuild>Qt 5.5.1 (x86_64&#x002D;little_endian&#x002D;lp64 shared (dynamic) release build; by GCC 5.3.1 20160407)</QtBuild>
    <QTestVersion>5.5.1</QTestVersion>
</Environment>
<TestFunction name="myTestFunction">
<Incident type="pass" file="" line="0">
    <DataTag><![CDATA[JIRA-123:true]]></DataTag>
</Incident>
<TestFunction name="myTestFunction">
<Incident type="pass" file="" line="0">
    <DataTag><![CDATA[JIRA-123:false]]></DataTag>
</Incident>
    <Duration msecs="0.041859"/>
</TestCase>

Do I need to include anything in the function to get the duration?

Not that I know of, and I see no conditions on the code. What version of Qt are you using?

Cheers.