QEXPECT_FAIL with Global Test Data

Publisher0663

Greetings,

I am having issues with my test application. The tests are completely data driven, such that I am using initTestCase_data to load the data files that I am using.

In some of the files, they do not contain data that they should, which is expected. However, when I use the macro QEXPECT_FAIL("file0", "...", Continue), the test still fails. I am accessing the data using QFETCH_GLOBAL and when I try to get the test tag (QTest::currentDataTag()), it just returns 0x0.

Is there any way to use QEXPECT_FAIL with global test data?

Qt Version: 5.15.3
OS: RockyLinux 8.7
Compiler: GCC 8.5.0

Paul Colby

Hi @Publisher0663,

QEXPECT_FAIL() will not work for you in this way, because it only covers QCOMPARE() and QVERIFY() (and their variations). From the docs:

The QEXPECT_FAIL() macro marks the next QCOMPARE() or QVERIFY() as an expected failure. Instead of adding a failure to the test log, an expected failure will be reported.

Put another way, QFETCH() and QFETCH_GLOBAL() believe it to be an error (they will QTEST_ASSERT_X()) for you to ever request data that isn't available.

Perhaps the easiest fix would be to swap QEXPECT_FAIL() for QSKIP intead? Maybe something like:

if (weExpectFetchGlobalToFailBecauseOfReasonX) {
    // instead of QEXPECT_FAIL()
    QSKIP("Data not available because of reason X");
}
QFETCH_GLOBAL(int, thing);

Or, it you actually want the rest of the test to execute (since you have Test::Continue in your example):

if (weExpectFetchGlobalToFailBecauseOfReasonX) {
    qInfo("Skipping part of the test because of reason x");
} else {
    QFETCH_GLOBAL(int, thing);
    // do tests
}
// do more tests (that don't need `thing`)

This also has the benefit of limiting the scope of the thing variable to just where you know it was actually loaded successfully. But personally I would break that last example into two separate tests, rather than do that conditional branching - just a stylistic thing.

Cheers.

Publisher0663

@Paul-Colby

Thanks for the reply.

I might not have been clear enough in my original post with how my QEXPECT_FAIL is implemented.

// initTestCase_data
addColumn<ObjectType>("object");
newRow("file0") << ObjectType("file0_path"); // Valid data for thing()
newRow("file1") << ObjectType("file1_path"); // Invalid data for thing()

// test_method
QFETCH_GLOBAL(ObjectType, object);
QEXPECT_FAIL("file1", "", Continue);
QVERIFY(object.thing() > 0);

I am implementing the expect fail before a QVERIFY, but it just ends up running the test anyways. I do not want to just skip the test, as I need to ensure that the data from other files do pass the test, just some specific files that I know do not have the data in them.
I also tried to implement a skip for specific files, but QTest::currentDataTag() returned 0x0 instead of "file1".

Paul Colby

Ah, thanks @Publisher0663, the sample code helps :)

The problem is that the dataIndex that gets passed to QEXPECT_FAIL() is only ever checked against the test-specific-data part of the current data tag, not the global part. Just to demonstrate the distinction, consider:

void MyTest::initTestCase_data() {
    QTest::addColumn<int>("globalData");
    QTest::addRow("file0") << 0; // Valid data for thing()
    QTest::addRow("file1") << 1; // Invalid data for thing()
}

void MyTest::myTest_data()
{
    QTest::addColumn<int>("testSpecificData");
    QTest::addRow("testCase0") << 0;
    QTest::addRow("testCase1") << 1;
    QTest::addRow("testCase2") << 2;
}

void MyTest::myTest()
{
    QFETCH_GLOBAL(int, globalData);
    QFETCH(int, testSpecificData);

    QEXPECT_FAIL("file1", "", Continue);     // This won't do anything.
    QEXPECT_FAIL("testCase2", "", Continue); // But this will.

    qDebug() << "currentDataTag:" << QTest::currentDataTag() << globalData << testSpecificData;
}

Output:

********* Start testing of MyTest *********
Config: Using QtTest library 6.5.1, Qt 6.5.1 (x86_64-little_endian-lp64 shared (dynamic) release build; by GCC 10.3.1 20210422 (Red Hat 10.3.1-1)), ubuntu 22.04
PASS   : MyTest::initTestCase()
QDEBUG : MyTest::myTest(file0:testCase0) currentDataTag: testCase0 0 0
PASS   : MyTest::myTest(file0:testCase0)
QDEBUG : MyTest::myTest(file0:testCase1) currentDataTag: testCase1 0 1
PASS   : MyTest::myTest(file0:testCase1)
QDEBUG : MyTest::myTest(file0:testCase2) currentDataTag: testCase2 0 2
FAIL!  : MyTest::myTest(file0:testCase2) QEXPECT_FAIL was called without any subsequent verification statements
QDEBUG : MyTest::myTest(file1:testCase0) currentDataTag: testCase0 1 0
PASS   : MyTest::myTest(file1:testCase0)
QDEBUG : MyTest::myTest(file1:testCase1) currentDataTag: testCase1 1 1
PASS   : MyTest::myTest(file1:testCase1)
QDEBUG : MyTest::myTest(file1:testCase2) currentDataTag: testCase2 1 2
FAIL!  : MyTest::myTest(file1:testCase2) QEXPECT_FAIL was called without any subsequent verification statements
PASS   : MyTest::cleanupTestCase()
Totals: 6 passed, 2 failed, 0 skipped, 0 blacklisted, 0ms
********* Finished testing of MyTest *********

You can see how the tests rows are matrixed or multiplied as each global row gets tested against each test-specific-data row, but QTest::currentDataTag() only reflects the latter, and not the global row name.

Now, if you had access to the global row name, then you could easily implement your own check like:

    QFETCH_GLOBAL(int, globalData);
    if (globalRowName == "file1") {
        QEXPECT_FAIL("", "", Continue); // Note the empty dataIndex param.
    }

But I can't see any (documented) way to get that global row name (internally, Qt Test stores it in private QTestResult object). So, I'd suggest three options, which I'll summarise first, then show some pseudo-code examples below.

Use test-specific-data - you can still easily share data between tests, much like you would with the global data, by simply putting the data into one data function, and calling it from the other/s. However, this means you don't get the matrix multiplication benefits of using both global and test-specific together; or
Add the global row name explicitly to each global data rows; or
Add a logical "is-valid" (or "expect-failure") flag to the global data.

Which of these makes most sense, depends on your specific test / dataset (they should all work, but some will be more meaningful / maintainable than others).

Option 1: Use Test-Specific Data Functions

// we have no MyTest::initTestCase_data() function.

void MyTest::myTest_data() {
    QTest::addColumn<int>("data");
    QTest::addRow("file0") << 0; // Valid data for thing()
    QTest::addRow("file1") << 1; // Invalid data for thing()
}

void MyTest::myTest()
{
    QFETCH(int, data);
    QEXPECT_FAIL("file1", "", Continue); // This works now.
    qDebug() << "currentDataTag:" << QTest::currentDataTag() << data;
}

void MyTest::anotherTest_data()
{
    myTest_data(); // Re-use myTest's test data.
    // We could add more rows here if wanted to.
}

void MyTest::anotherTest()
{
    QFETCH(int, data);
    QEXPECT_FAIL("file1", "", Continue); // This also works :)
    qDebug() << "currentDataTag:" << QTest::currentDataTag() << data;
}

Output:

********* Start testing of MyTest *********
Config: Using QtTest library 6.5.1, Qt 6.5.1 (x86_64-little_endian-lp64 shared (dynamic) release build; by GCC 10.3.1 20210422 (Red Hat 10.3.1-1)), ubuntu 22.04
PASS   : MyTest::initTestCase()
QDEBUG : MyTest::myTest(file0) currentDataTag: file0 0
PASS   : MyTest::myTest(file0)
QDEBUG : MyTest::myTest(file1) currentDataTag: file1 1
FAIL!  : MyTest::myTest(file1) QEXPECT_FAIL was called without any subsequent verification statements
QDEBUG : MyTest::anotherTest(file0) currentDataTag: file0 0
PASS   : MyTest::anotherTest(file0)
QDEBUG : MyTest::anotherTest(file1) currentDataTag: file1 1
FAIL!  : MyTest::anotherTest(file1) QEXPECT_FAIL was called without any subsequent verification statements
PASS   : MyTest::cleanupTestCase()
Totals: 4 passed, 2 failed, 0 skipped, 0 blacklisted, 0ms
********* Finished testing of MyTest *********

Option 2: Include Row Name in Global Rows

void MyTest::initTestCase_data() {
    QTest::addColumn<QString>("globalDataName");
    QTest::addColumn<int>("globalData");
    QTest::addRow("file0") << QStringLiteral("file0") << 0; // Valid data for thing()
    QTest::addRow("file1") << QStringLiteral("file1") << 1; // Invalid data for thing()
}

void MyTest::myTest()
{
    QFETCH_GLOBAL(QString, globalDataName);
    QFETCH_GLOBAL(int, globalData);
    if (globalDataName == QStringLiteral("file1")) {
        QEXPECT_FAIL("", "", Continue); // This works now.
    }
    qDebug() << "currentDataTag:" << QTest::currentDataTag() << globalDataName << globalData;
}

Output:

********* Start testing of MyTest *********
Config: Using QtTest library 6.5.1, Qt 6.5.1 (x86_64-little_endian-lp64 shared (dynamic) release build; by GCC 10.3.1 20210422 (Red Hat 10.3.1-1)), ubuntu 22.04
PASS   : MyTest::initTestCase()
QDEBUG : MyTest::myTest(file0) currentDataTag:  "file0" 0
PASS   : MyTest::myTest(file0)
QDEBUG : MyTest::myTest(file1) currentDataTag:  "file1" 1
FAIL!  : MyTest::myTest(file1) QEXPECT_FAIL was called without any subsequent verification statements
PASS   : MyTest::cleanupTestCase()
Totals: 3 passed, 1 failed, 0 skipped, 0 blacklisted, 1ms
********* Finished testing of MyTest *********

Of course, if the code duplication in initTestCase_data() offends, you could use a #define macro to reduce it, like:

    #define ADD_ROW(name) QTest::addRow(name) << QStringLiteral(name)
    ADD_ROW("file0") << 0;
    ADD_ROW("file1") << 1;

Side note, I actually do this sort of thing quite often, as name could be an enum class that I'm testing, eg:

    #define ADD_ROW(label) QTest::addRow(#label) << MyEnum::label
    ADD_ROW(Red) << ...;
    ADD_ROW(Blue) << ...;

That way I ensure the row name, and the enum label match (reduces cut-and-paste errors ;)

Option 3: Add Explicit Logic Flag to the Global Data

This very similar to option 2, but would make more sense if the "validness" is a property of the row itself, rather than a property of the test+row combination (ie is the row valid for all tests? or just for some and not others?)

void MyTest::initTestCase_data() {
    QTest::addColumn<int>("globalData");
    QTest::addColumn<bool>("isValid"); // or maybe the inverse as "expectFailure"
    QTest::addRow("file0") << 0 << true; // Valid data for thing()
    QTest::addRow("file1") << 1 << false; // Invalid data for thing()
}

void MyTest::myTest()
{
    QFETCH_GLOBAL(int, globalData);
    QFETCH_GLOBAL(bool, isValid);
    if (!isValid) {
        QEXPECT_FAIL("", "", Continue); // This works now.
    }
    qDebug() << "currentDataTag:" << QTest::currentDataTag() << globalData << isValid;
}

Output:

********* Start testing of MyTest *********
Config: Using QtTest library 6.5.1, Qt 6.5.1 (x86_64-little_endian-lp64 shared (dynamic) release build; by GCC 10.3.1 20210422 (Red Hat 10.3.1-1)), ubuntu 22.04
PASS   : MyTest::initTestCase()
QDEBUG : MyTest::myTest(file0) currentDataTag:  0 true
PASS   : MyTest::myTest(file0)
QDEBUG : MyTest::myTest(file1) currentDataTag:  1 false
FAIL!  : MyTest::myTest(file1) QEXPECT_FAIL was called without any subsequent verification statements
PASS   : MyTest::cleanupTestCase()
Totals: 3 passed, 1 failed, 0 skipped, 0 blacklisted, 0ms
********* Finished testing of MyTest *********

Personally, I would first consider whether the data truly is global, and if not, then switch to test-specific-data function/s instead.

But if it is indeed global, then I would add the data necessary for the expect-failure decision explicitly in the global data set (whether that's some sort of data-tag/name, or logical flag/categorisation) and thus not rely on the global data's row name (since that does not appear to be available inside the individual test functions today).

Cheers.

Publisher0663

@Paul-Colby

It is unfortunate that a global data index is not available, as that would have solved the issue that I was having.

I went ahead and implemented the third option. I may revisit how I am loading the files in the future to rewrite the test_data functions to only include the data that I need.