Integrating Qt Test and Appveyor



  • Did anyone manage to integrate Qt test results into appveyor? If so, could you share how?


  • Lifetime Qt Champion

    Hi,

    Don't know if that helps but the PiwikTracker seems to use the text output of the unit tests.

    Hope that gives some clue to go further.



  • @SGaist I might be going blind, but I can't see any tests at all in that project: https://ci.appveyor.com/project/pbek/qt-piwik-tracker/build/1.0.24


  • Lifetime Qt Champion

    My bad, looks like they are run in travis.



  • Hi @VRonin,

    Did anyone manage to integrate Qt test results into appveyor? If so, could you share how?

    I haven't yet (it's been on my list of things to do for a while).

    There's two levels of possible integration - the most basic is simply making Qt output the XUnit format, and submitting that to AppVeyor.

    A quick GitHub search for yaml files that contain -xunitxml reveals a couple of projects that have tried this. But the only one that seems to have succeeded (and is publicly accessible on AppVeyor) is AureBeshTranslator:

    This "integration" looks to be very quick and easy, but only shows the final test outcomes (ie shows nothing at all until the tests are completed).

    The other integration option is to use the Build Worker API to report tests before, during, and after execution. For very big projects, this would be very nice. It would allow AppVeyor to show in their UI all tests that are going to be run, and check them off (turn green (or red)) as they execute. Of course, with just a small number of tests, there'd be no point.

    To do this integration, you'd have to use C++/Qt code to introspect the test classes, and hook into QtTest's init and cleanup functions, and submit via HTTP. Should be fairly easy, and this is something I've been toying with, but only in between lots of other side projects ;) Should be do-able. I'll definitely be interested if you find any projects that do this already :)

    Cheers.



  • @Paul-Colby said in Integrating Qt Test and Appveyor:

    This "integration" looks to be very quick and easy, but only shows the final test outcomes (ie shows nothing at all until the tests are completed).

    Thank you very much. I was looking at doing the same only using curl instead of powershell.

    It works!

    For reference my appveyor.yml looks like this:

    EDIT: the script below is wrong, scroll down

    test_script:
    - cd %TESTOUTPUTFOLDER%/bin
    - set PATH=../lib;%PATH%
    - call "../../../runtests.bat"
    - ps: (new-object net.webclient).UploadFile("https://ci.appveyor.com/api/testresults/junit/$($env:APPVEYOR_JOB_ID)", (Resolve-Path .\testsResults.xml))
    

    And runtests.bat is simply: for %%a in (*.exe) do %%a -xunitxml > testsResults.xml


  • Lifetime Qt Champion

    Shouldn't it be for %%a in (*.exe) do %%a -xunitxml >> testsResults.xml ?



  • It was not the only problem. Updated script in .appveyor.yml

    EDIT: the script below is wrong, scroll down

    test_script:
    - If NOT exist "%TESTOUTPUTFOLDER%/bin" (appveyor exit)
    - cd %TESTOUTPUTFOLDER%/bin
    - set PATH=../lib;%PATH%
    - for %%a in (*.exe) do (%%a -xunitxml >> testsResults.xml)
    
    on_finish:
    - ps: (new-object net.webclient).UploadFile("https://ci.appveyor.com/api/testresults/junit/$($env:APPVEYOR_JOB_ID)", (Resolve-Path .\testsResults.xml))
    

    This will probably still stop running tests as soon as one fails but good enough for rock and roll.

    The interesting detail is that you have to send the outputs to the junit reader of appveyor instead of the xunit one



  • This turned out to be infinitely more involved than I wanted.

    Merging different test results in a format that appveyor understands is not a walk in the park.

    This is my final script:

    - for %%a in (*.exe) do (%%a -xml > %%a_tstres.xml & set preventFail = 1)
    - ps: '$xmlOut = new-object System.Xml.XmlDocument;
        $xmlOut.AppendChild($xmlOut.CreateXmlDeclaration("1.0","UTF-8",$null));
        $rootXmlOut = $xmlOut.CreateElement("testsuites");
        $xmlOut.AppendChild($rootXmlOut);
        $totalPass = 0;
        $totalFail = 0;
        $totalError = 0;
        $totalSkip = 0;
        $totalTime = 0;
        $crashMessage = $null;
        Get-ChildItem ".\" -Filter "*_tstres.xml" | Foreach-Object {
            $XmlDocument = $null;
            $localPass = 0;
            $localFail = 0;
            $localError = 0;
            $localSkip = 0;
            $localTime = 0;
            $currentFilePath = $_.FullName;
            $testSuiteName = $_.BaseName.subString(0,$_.BaseName.Length - 11);
            if([bool]((Get-Content -Path $currentFilePath) -as [xml])){
                [xml]$XmlDocument = (Get-Content -Path $currentFilePath) -as [xml];
            }
            else{
                $localError = 1;
                $rawFilecontent = [IO.File]::ReadAllText($currentFilePath);
                if([string]::IsNullOrEmpty($rawFilecontent)){
                    $crashMessage = "Output file is empty: " + $currentFilePath;
                }
                else{
                    $crashMessage = $rawFilecontent;
                    $rawFileMatch = [regex]::match($rawFilecontent,"(?s)(.+<\/TestCase>)(.*)");
                    if($rawFileMatch.Success){
                        if([bool](($rawFileMatch.captures.groups[1].value) -as [xml])){
                            [xml]$XmlDocument = ($rawFileMatch.captures.groups[1].value) -as [xml];
                            $crashMessage = $rawFileMatch.captures.groups[2].value;
                        }
                    }
                }
            }
            $testSuiteXmlOut = $rootXmlOut.AppendChild($xmlOut.CreateElement("testsuite"));
            if($XmlDocument -ne $null){
                $testClassName = $XmlDocument.TestCase.name;
                $testSuiteXmlOut.SetAttribute("name",$testSuiteName);
                $testSuitePropertiesXmlOut = $testSuiteXmlOut.AppendChild($xmlOut.CreateElement("properties"));
                $testSuitePropertiesPropertyXmlOut = $testSuitePropertiesXmlOut.AppendChild($xmlOut.CreateElement("property"));
                $testSuitePropertiesPropertyXmlOut.SetAttribute("name","QtVersion");
                $testSuitePropertiesPropertyXmlOut.SetAttribute("value",($XmlDocument.TestCase.Environment.QtVersion));
                $testSuitePropertiesPropertyXmlOut = $testSuitePropertiesXmlOut.AppendChild($xmlOut.CreateElement("property"));
                $testSuitePropertiesPropertyXmlOut.SetAttribute("name","QtBuild");
                $testSuitePropertiesPropertyXmlOut.SetAttribute("value",($XmlDocument.TestCase.Environment.QtBuild));
                $testSuitePropertiesPropertyXmlOut = $testSuitePropertiesXmlOut.AppendChild($xmlOut.CreateElement("property"));
                $testSuitePropertiesPropertyXmlOut.SetAttribute("name","QTestVersion");
                $testSuitePropertiesPropertyXmlOut.SetAttribute("value",($XmlDocument.TestCase.Environment.QTestVersion));
                foreach($testFunction in $XmlDocument.SelectNodes("//TestFunction")){
                    $testFunctionName = $testFunction.name;
                    $countIncidents = $testFunction.ChildNodes.Count;
                    $testFunctionTime = [decimal]$testFunction.Duration.msecs;
                    $localTime = $localTime +$testFunctionTime;
                    foreach($incident in $testFunction.ChildNodes){
                        if($incident.Name -ne "Incident" -and $incident.Name -ne "Message"){
                            continue;
                        }
                        $incidentName = $testFunctionName;
                        if($incident.DataTag -ne $null){
                            $incidentName = $incidentName + " - " + $incident.DataTag.InnerText;
                        }
                        $incidentName = ($incidentName);
                        $testSuitetestcaseXmlOut = $testSuiteXmlOut.AppendChild($xmlOut.CreateElement("testcase"));
                        $testSuitetestcaseXmlOut.SetAttribute("name",$incidentName);
                        $testSuitetestcaseXmlOut.SetAttribute("classname",$testClassName);
                        $testSuitetestcaseXmlOut.SetAttribute("time",$testFunctionTime/(1000*$countIncidents));
                        if($incident.type -eq "skip"){
                            ++$localSkip;
                            $testSuitetestcaseSkipXmlOut = $testSuitetestcaseXmlOut.AppendChild($xmlOut.CreateElement("skipped"));
                            $testSuitetestcaseSkipXmlOut.SetAttribute("message","file: " + ($incident.file + " line: " + $incident.line + " " + $incident.Description.InnerText));
                        }
                        ElseIf ($incident.type -eq "fail"){
                            ++$localFail;
                            $testSuitetestcaseSkipXmlOut = $testSuitetestcaseXmlOut.AppendChild($xmlOut.CreateElement("failure"));
                            $testSuitetestcaseSkipXmlOut.SetAttribute("message",("file: " + $incident.file + " line: " + $incident.line + " " + $incident.Description.InnerText));
                        }
                        ElseIf ($incident.type -eq "qdebug" -or $incident.type -eq "qwarn" -or $incident.type -eq "system" -or $incident.type -eq "qfatal"){
                            $testSuitetestcaseCerrXmlOut = $testSuitetestcaseXmlOut.AppendChild($xmlOut.CreateElement("system-err"));
                            $testSuitetestcaseCerrXmlOut.AppendChild($xmlOut.CreateTextNode(($incident.Description.InnerText)));
                        }
                        else{
                            ++$localPass;
                        }
                    };
                };
            }
            if($localError -eq 1){
                $testSuitetestcaseXmlOut = $testSuiteXmlOut.AppendChild($xmlOut.CreateElement("testcase"));
                $testSuitetestcaseXmlOut.SetAttribute("name","SystemError");
                $testSuitetestcaseXmlOut.SetAttribute("classname",$testClassName);
                $testSuitetestcaseErrorXmlOut = $testSuitetestcaseXmlOut.AppendChild($xmlOut.CreateElement("error"));
                $testSuitetestcaseErrorXmlOut.SetAttribute("message",($crashMessage));
            }
            $testSuiteXmlOut.SetAttribute("time",$localTime/1000);
            $testSuiteXmlOut.SetAttribute("skipped",$localSkip);
            $testSuiteXmlOut.SetAttribute("tests",$localSkip+$localFail+$localError+$localPass);
            $testSuiteXmlOut.SetAttribute("failures",$localFail);
            $testSuiteXmlOut.SetAttribute("errors",$localError);
            $totalTime = $totalTime + $localTime;
            $totalSkip = $totalSkip + $localSkip;
            $totalError = $totalError + $localError;
            $totalFail = $totalFail + $localFail;
            $totalPass = $totalPass + $localPass;
        };
        $rootXmlOut.SetAttribute("time",$totalTime/1000);
        $rootXmlOut.SetAttribute("failures",$totalFail);
        $rootXmlOut.SetAttribute("errors",$totalError);
        $rootXmlOut.SetAttribute("tests",$totalPass);
        $xmlOut.save($pwd.Path + "\testResults.xml");
        if($totalError+$totalFail -gt 0){
            throw;
        }'
        
    on_finish:
    - ps: if (Test-Path ".\testResults.xml") {(new-object net.webclient).UploadFile("https://ci.appveyor.com/api/testresults/junit/$($env:APPVEYOR_JOB_ID)", (Resolve-Path .\testResults.xml));}
    

    Edit: it looks bad but it needs no customisation to apply to other projects at most you need to add at the beginning a line to cd in the folder where you have the tests


  • Qt Champions 2017

    @VRonin:

    This turned out to be infinitely more involved than I wanted.

    Yeah, as usual in programming ;)


Log in to reply
 

Looks like your connection to Qt Forum was lost, please wait while we try to reconnect.