[Solved] Unit test failure notification



  • I’m using QMAKE_POST_LINK to run a unit test automatically for each build. My test script returns 0 on success and 1 on failure. The script is working and being automatically run OK.

    When I build in Qt Creator and the unit test fails, no notification comes up in the “Build Issues” pane. Is there a way to cause the post-link test failure to get flagged in “Build Issues”?



  • Further on this query.

    I find that if I write failure descriptive test to stderr rather than stdout, Qt Creator does take notice in that it displays the text in red rather than black in the "Compile Output" pane. But still nothing in the "Build Issues" pane.



  • deleted the duplicate thread that was created here http://developer.qt.nokia.com/forums/viewthread/1234/



  • As far as I understand it is up to QtTestLib to decide whether to print to stderr or stdout. Do you redirect output in your script?



  • Yes, I redirected the output text from the test script to stderr and Qt Creator then printed the text in red instead of black in the compile output pane but still nothing in Build Issues pane.



  • Also, I'm not using QtTestLib; perhaps that is the problem?


  • Moderators

    deanz: Qt Creator parses the compile output to generate the build issues. It does not expect unit test results when building, so it does not parse them and thus nothing ends up in the build issues.

    Sorry, there is no workaround available that I am aware of. We want to add proper unit test integration, but so far nobody found the time to actually do it:-/



  • Thanks, Tobias.


  • Moderators

    Hmmm... thinking about this: With a bit of hacking you could get your test results displayed in the build issues:-)

    I added a TaskList plugin a while back which grabs a file in CSV format and puts it into the build issues plane. So maybe you can wrap your unit test runner into a script that turns the output into such an CSV file... Creator does watch the file and updates the build issue pane whenever the file changes.

    Here is my "announcement mail":http://lists.trolltech.com/pipermail/qt-creator/2010-August/007574.html with some information on the syntax accepted. I think this information is actually already in the documentation, but I am not sure where;-)



  • Great!

    Thanks again, Tobias.

    I can easilly change my unit tester output to CSV format and your plug-in can take it from there. Looks like problem solved.



  • Tobias:

    What is the name of your new plugin? How do I download and install it?


  • Moderators

    deanz: It is called TaskList and is included in Qt Creator 2.1 beta. Sorry, I should have mentioned that:-)

    Oh, for now you need to open the .tasks file (once) via File->Open. When opening it any other way you get it displayed in an editor:-(



  • OK, thanks again, Tobias. I will wait for the 2.1 release but it sure sounds like a very useful addition to an already excellent product.


  • Moderators

    Yes, writing that plugin was 2h well spend;-)

    Actually I only did it since I wanted to get our count of krazy (some static code analysis tool) issues down and was too lazy to hop to each place manually. Good to hear that it works for other use-cases as well;-)



  • bq. Actually I only did it since I wanted to get our count of krazy (some static code analysis tool) issues down

    Ohh pooh :-(
    Just when I'm about to finish my own Static Code Quality plugin :-)


  • Moderators

    tbscope: Being able to hack up a script to import an arbitrary list of issues into Creator is not exactly a "static code quality plugin":-)

    I am looking forward to see your plugin! I am sure it will be way more user friendly than my hack.



  • [quote author="Tobias Hunger" date="1287579812"]Actually I only did it since I wanted to get our count of krazy (some static code analysis tool) issues down and was too lazy to hop to each place manually. Good to hear that it works for other use-cases as well;-)[/quote]

    Hi Tobias.
    we plan to use your plugin to display all the tasks generated by a static code analysis tool.
    It works great, there is just one thing left:
    For each task there exists documentation in the form of a html file. We want to be able to somehow open the html file via some handy action (right click on the task->open documentation or sth.)

    Do you think it would be possible to add such a functionality to your plugin?
    Well perhaps it is not only your plugin that would have to be extended but also the BuildIssues Implementation...

    Regards,
    Markus


  • Moderators

    Hi Markus!

    There is no such code in creator yet.

    We do have the concept of TaskHandlers though: Just create one that can accept your task and then displays the help if it is able to match the task to one of your HTML files.

    This should be very easy to implement in a plugin. There should be no need to meddle with existing Qt Creator code for this.

    Best Regards,
    Tobias



  • Thanks for your response.
    I just finished a TaskHandler that does what I want =) Now the next step is to put this into a plugin.

    I have to say, I really like the QtCreator code. Each time I need some extra functionality it is very easy to do that. The codebase is very friendly towards extending it!


  • Moderators

    Took you less than 11min to write that TaskHandler... I am impressed!

    Thanks for the compliments on our code. Positive feedback is always appreciated:-)



  • Well, I just used the CopyTaskHandler as a template and wrote 11 lines of code ;-)



  • Howdy folks,

    Great topic! Because I wanted to try out test driven development, I was looking for a convenient way to get UnitTest++ output into the QtCreator Build Issues pane. Luckily, the TaskHandler functionality has recently been added to QtCreator, so it could be done easier than I expected. Anyhow, I just wrote the QtTaskTestReporter class below. You can use it in a testinstantiator like this:

    @
    // --------------- testinstantiator.cpp ---------------
    #include "UnitTest++.h"
    #include "TestRunner.h"
    #include "QtTaskTestReporter.h"

    /* #include files containing your tests here */

    int main( int argc, const char* argv[])
    {
    UnitTest::QtTaskTestReporter reporter;
    UnitTest::TestRunner runner(reporter);
    runner.RunTestsIf(UnitTest::Test::GetTestList(), NULL, UnitTest::True(), 0);
    return 0;
    }
    @

    Just build the testinstantiator along wih your executable, and add running the testinstantiator executable as a Custom Process Step, after Make, under "Build Steps" in the "Projects -> Build Settings" pane. After Opening the generated .tasks file once with QtCreator, QtCreator will display UnitTest++ failures in the build issues pane during the entire QtCreator session. This enables swift, one-click navigation to the failing unit tests.

    I hope this is as useful to you as it is to me.

    Cheers,
    Daan

    @
    // --------------- QtTaskTestReporter.h ---------------
    /*

    • QtTaskTestReporter. Writes UnitTest++ output to a .task file.
    • Author: Daan Baas 2011 the.daanbaas@gmail.com
      */

    #ifndef QTTASKTESTREPORTER_H
    #define QTTASKTESTREPORTER_H

    #include <vector>
    #include <string>
    #include "TestReporter.h"
    #include "TestDetails.h"

    namespace UnitTest
    {
    class QtTaskTestReporter : public TestReporter
    {
    public:
    QtTaskTestReporter(const std::string& outputFileName = "unittest++.tasks");

        virtual void ReportTestStart(TestDetails const& test){ /* empty */ }
        virtual void ReportFailure(TestDetails const& test, char const* failure);
        virtual void ReportTestFinish(TestDetails const& test, float secondsElapsed){ /* empty */ }
        virtual void ReportSummary(int totalTestCount, int failedTestCount, int failureCount, float secondsElapsed);
    
    private:
        struct TaskData
        {
            //mimics a task from QtCreator's project explorer plugin
            //legal values for type are: "Unknown", "Error" and "Warning"
            std::string fileName;
            std::string lineNumber;
            std::string type;
            std::string description;
        };
    
        std::string taskToTabSeparatedString(const TaskData& iask) const;
        std::string backSlashesToForwardSlashes(const std::string& inputStr) const;
        std::string intToString(int number) const;
        void writeTasksToOutputFile&#40;&#41; const;
    
        typedef std::vector< TaskData > TaskList;
        TaskList m_tasks;
        const std::string m_outFileName;
    };
    

    }//end namespace UnitTest
    #endif // QTTASKTESTREPORTER_H

    //--------------- QtTaskTestReporter.cpp ---------------

    /*

    • QtTaskTestReporter. Writes UnitTest++ output to a .task file.
    • Author: Daan Baas 2011 the.daanbaas@gmail.com
      */

    #include "qttasktestreporter.h"
    #include <stdexcept>
    #include <iostream>
    #include <fstream>
    #include <sstream>
    #include <algorithm>

    namespace UnitTest
    {
    QtTaskTestReporter::QtTaskTestReporter(const std::string& outputFileName)
    :
    TestReporter(),
    m_outFileName(outputFileName)
    {
    //note: you can choose your own output filename, but make sure it has the
    //".tasks" extension so it can be picked up by the QtCreator task handler
    }

    void QtTaskTestReporter::ReportFailure(TestDetails const& test, char const* failure)
    {
        TaskData failTask;
    
        failTask.fileName = backSlashesToForwardSlashes(test.filename);
        failTask.lineNumber = intToString(test.lineNumber);
        failTask.type = "Error";
    
        std::stringstream failureMessage;
        failureMessage <<  "error: Failure in " << test.testName << ": " << failure;
        failTask.description = failureMessage.str() ;
    
        m_tasks.push_back(failTask);         
    }
    
    
    void QtTaskTestReporter::ReportSummary(int totalTestCount, int failedTestCount, int failureCount,
                                           float secondsElapsed)
    {
        writeTasksToOutputFile&#40;&#41;;
    }
    
    
    std::string QtTaskTestReporter::taskToTabSeparatedString(const TaskData& task) const
    {
        std::string result;
        result = task.fileName + '\t' + task.lineNumber + '\t' + task.type + '\t' + task.description;
        return result;
    }
    
    
    std::string QtTaskTestReporter::backSlashesToForwardSlashes(const std::string& inputStr) const
    {
        using namespace std;
        string result = inputStr;
        replace(result.begin(), result.end(), '\\', '/');
        return result;
    }
    
    
    std::string QtTaskTestReporter::intToString(int number) const
    {
        std::stringstream ss;
        ss << number;
        return ss.str();
    }
    
    void QtTaskTestReporter::writeTasksToOutputFile&#40;&#41; const
    {
        using namespace std;
        ofstream file&#40;m_outFileName.c_str(&#41;);
        if(!file)
        {
            cerr << "error: QtTaskTestReporter::ReportTestFinish() cannot open output file";
            return;
        }
    
        TaskList::const_iterator pos;
        for(pos = m_tasks.begin(); pos != m_tasks.end(); ++pos)
        {
            file << taskToTabSeparatedString(*pos) << std::endl;
        }
    
        file.close();
    }
    

    }//end namespace UnitTest
    @



  • I've started working on a test runner for Google Test kind of like dano's (thanks for the example!). The one thing I'm not happy about, however, is the need to manually open the .tasks file every time you restart the IDE. It doesn't seem to stick with the session or anything to automatically open the next time you run.

    Anybody have any suggestions on improving that situation?

    Thanks,
    Eric



  • Hi All,

    I have found an easier way to integrate the output from any unittest framework(or any other tool for that matter) into the issue pane of QtCreator, so that one click on an issue will directly take you to the corresponding testcase source file and line.

    Basically QtCreator parses the stderr stream and identifies issues. QtCreator is only aware of the format which compiler uses to report errors. So the trick is to format your error logs in the same format used by your compiler. This can be done either directly within unit test framework or by your own custom code to reformat the error strings from the framework and write to stderr.

    For eg, on linux-gcc setup the expected error format is as below,
    <source-file-path>:<line>:<column>: error: <error message>

    Unittest++, which is my choice of unit test framework reports errors almost in a similar format on linux. But those errors were written to stdout and the column number was missing from the error message.
    So I just changed the error printing line in Unittest++ source code to conform to the above format(with a dummy value for column), and that's it. Now the unit test failures will appear as issues, and mouse click will take you to the line of failure.

    My change for unittest++ is following,

    @
    diff --git a/UnitTest++/src/TestReporterStdout.cpp b/UnitTest++/src/TestReporterStdout.cpp
    index 563113c..96f725a 100644
    --- a/UnitTest++/src/TestReporterStdout.cpp
    +++ b/UnitTest++/src/TestReporterStdout.cpp
    @@ -13,13 +13,13 @@ namespace UnitTest {
    void TestReporterStdout::ReportFailure(TestDetails const& details, char const* failure)
    {
    #if defined(APPLE) || defined(GNUG)

    • char const* const errorFormat = "%s:%d: error: Failure in %s: %s\n";
    • char const* const errorFormat = "%s:%d:%d: error: Failure in %s: %s\n";
      #else
      char const* const errorFormat = "%s(%d): error: Failure in %s: %s\n";
      #endif

      using namespace std;

    • printf(errorFormat, details.filename, details.lineNumber, details.testName, failure);
    • fprintf(stderr, errorFormat, details.filename, details.lineNumber, 1, details.testName, failure);
      }
      @

    I am trying to get this change into unittest++, so that QtCreator will be supported out of the box.
    I think QTestLib can also follow a similar approach(and I was surprised to learn that they don't do this already).

    -lijo

    [Code formatting, please wrap in @-Tags, Volker]


  • Moderators

    lijo: There are a couple of limitations to your approach:

    • This requires tests to run as part of the build system (otherwise the compiler output parsers are not active)
    • This assumes a GCC based compiler, MSVC, Clang and others have very different output formats

    I'd recommend using the tasklist feature of Qt Creator instead: Just open a CSV-file ("documentation":http://doc.qt.nokia.com/qtcreator-2.5/creator-task-lists.html) via File->Open and its contents will be displayed in the build issue pane. The view will be updated whenever the file changes.

    I use this feature for a few different use-cases:

    Getting the output of the Krazy static code analysis tool into Qt Creator

    Run sanity checking scripts on my changed code (putting common errors and FIXME/TODO items into the build issues pane).

    Both is done via scripts available in Qt Creator sources (check the scripts directory). It should be straight forward to write a script that takes the output of your unit test s and turns that into the format required for .tasks files.



  • Thanks for the reply.

    bq. lijo: There are a couple of limitations to your approach:

    • This requires tests to run as part of the build system (otherwise the compiler output parsers are not active)

    For me running unit tests after each build is more important. If this happens smoothly, I would not have to run the unit tests manually at all.

    bq.

    • This assumes a GCC based compiler, MSVC, Clang and others have very different output formats

    I think any one who has access to the unit test framework source code can easily tweak its output to their compiler specified format.

    Event though limited, this approach works very well for my purpose as of now. If I get into any trouble with that, I will definitely check out tasklist :-)


Log in to reply
 

Looks like your connection to Qt Forum was lost, please wait while we try to reconnect.