Integrating Qt Test and Appveyor

VRonin

Did anyone manage to integrate Qt test results into appveyor? If so, could you share how?

Paul Colby

Did anyone manage to integrate Qt test results into appveyor? If so, could you share how?

I haven't yet (it's been on my list of things to do for a while).

There's two levels of possible integration - the most basic is simply making Qt output the XUnit format, and submitting that to AppVeyor.

A quick GitHub search for yaml files that contain -xunitxml reveals a couple of projects that have tried this. But the only one that seems to have succeeded (and is publicly accessible on AppVeyor) is AureBeshTranslator:

https://github.com/Ometeo/AureBeshTranslator/blob/master/appveyor.yml#L51 - the AppVeyor build configuration ib GitHub
https://ci.appveyor.com/project/Ometeo/AureBeshTranslator/build/tests - the test results on AppVeyor

This "integration" looks to be very quick and easy, but only shows the final test outcomes (ie shows nothing at all until the tests are completed).

The other integration option is to use the Build Worker API to report tests before, during, and after execution. For very big projects, this would be very nice. It would allow AppVeyor to show in their UI all tests that are going to be run, and check them off (turn green (or red)) as they execute. Of course, with just a small number of tests, there'd be no point.

To do this integration, you'd have to use C++/Qt code to introspect the test classes, and hook into QtTest's init and cleanup functions, and submit via HTTP. Should be fairly easy, and this is something I've been toying with, but only in between lots of other side projects ;) Should be do-able. I'll definitely be interested if you find any projects that do this already :)

Cheers.

SGaist

Hi,

Don't know if that helps but the PiwikTracker seems to use the text output of the unit tests.

Hope that gives some clue to go further.

VRonin

@SGaist I might be going blind, but I can't see any tests at all in that project: https://ci.appveyor.com/project/pbek/qt-piwik-tracker/build/1.0.24

SGaist

My bad, looks like they are run in travis.

Paul Colby

Hi @VRonin,

Did anyone manage to integrate Qt test results into appveyor? If so, could you share how?

I haven't yet (it's been on my list of things to do for a while).

There's two levels of possible integration - the most basic is simply making Qt output the XUnit format, and submitting that to AppVeyor.

A quick GitHub search for yaml files that contain -xunitxml reveals a couple of projects that have tried this. But the only one that seems to have succeeded (and is publicly accessible on AppVeyor) is AureBeshTranslator:

https://github.com/Ometeo/AureBeshTranslator/blob/master/appveyor.yml#L51 - the AppVeyor build configuration ib GitHub
https://ci.appveyor.com/project/Ometeo/AureBeshTranslator/build/tests - the test results on AppVeyor

This "integration" looks to be very quick and easy, but only shows the final test outcomes (ie shows nothing at all until the tests are completed).

The other integration option is to use the Build Worker API to report tests before, during, and after execution. For very big projects, this would be very nice. It would allow AppVeyor to show in their UI all tests that are going to be run, and check them off (turn green (or red)) as they execute. Of course, with just a small number of tests, there'd be no point.

To do this integration, you'd have to use C++/Qt code to introspect the test classes, and hook into QtTest's init and cleanup functions, and submit via HTTP. Should be fairly easy, and this is something I've been toying with, but only in between lots of other side projects ;) Should be do-able. I'll definitely be interested if you find any projects that do this already :)

Cheers.

VRonin

@Paul-Colby said in Integrating Qt Test and Appveyor:

This "integration" looks to be very quick and easy, but only shows the final test outcomes (ie shows nothing at all until the tests are completed).

Thank you very much. I was looking at doing the same only using curl instead of powershell.

It works!

For reference my appveyor.yml looks like this:

EDIT: the script below is wrong, scroll down

test_script:
- cd %TESTOUTPUTFOLDER%/bin
- set PATH=../lib;%PATH%
- call "../../../runtests.bat"
- ps: (new-object net.webclient).UploadFile("https://ci.appveyor.com/api/testresults/junit/$($env:APPVEYOR_JOB_ID)", (Resolve-Path .\testsResults.xml))

And runtests.bat is simply: for %%a in (*.exe) do %%a -xunitxml > testsResults.xml

SGaist

Shouldn't it be for %%a in (*.exe) do %%a -xunitxml >> testsResults.xml ?

VRonin

It was not the only problem. Updated script in .appveyor.yml

EDIT: the script below is wrong, scroll down

test_script:
- If NOT exist "%TESTOUTPUTFOLDER%/bin" (appveyor exit)
- cd %TESTOUTPUTFOLDER%/bin
- set PATH=../lib;%PATH%
- for %%a in (*.exe) do (%%a -xunitxml >> testsResults.xml)

on_finish:
- ps: (new-object net.webclient).UploadFile("https://ci.appveyor.com/api/testresults/junit/$($env:APPVEYOR_JOB_ID)", (Resolve-Path .\testsResults.xml))

This will probably still stop running tests as soon as one fails but good enough for rock and roll.

The interesting detail is that you have to send the outputs to the junit reader of appveyor instead of the xunit one

VRonin

This turned out to be infinitely more involved than I wanted.

Merging different test results in a format that appveyor understands is not a walk in the park.

This is my final script:

- for %%a in (*.exe) do (%%a -xml > %%a_tstres.xml & set preventFail = 1)
- ps: '$xmlOut = new-object System.Xml.XmlDocument;
    $xmlOut.AppendChild($xmlOut.CreateXmlDeclaration("1.0","UTF-8",$null));
    $rootXmlOut = $xmlOut.CreateElement("testsuites");
    $xmlOut.AppendChild($rootXmlOut);
    $totalPass = 0;
    $totalFail = 0;
    $totalError = 0;
    $totalSkip = 0;
    $totalTime = 0;
    $crashMessage = $null;
    Get-ChildItem ".\" -Filter "*_tstres.xml" | Foreach-Object {
        $XmlDocument = $null;
        $localPass = 0;
        $localFail = 0;
        $localError = 0;
        $localSkip = 0;
        $localTime = 0;
        $currentFilePath = $_.FullName;
        $testSuiteName = $_.BaseName.subString(0,$_.BaseName.Length - 11);
        if([bool]((Get-Content -Path $currentFilePath) -as [xml])){
            [xml]$XmlDocument = (Get-Content -Path $currentFilePath) -as [xml];
        }
        else{
            $localError = 1;
            $rawFilecontent = [IO.File]::ReadAllText($currentFilePath);
            if([string]::IsNullOrEmpty($rawFilecontent)){
                $crashMessage = "Output file is empty: " + $currentFilePath;
            }
            else{
                $crashMessage = $rawFilecontent;
                $rawFileMatch = [regex]::match($rawFilecontent,"(?s)(.+<\/TestCase>)(.*)");
                if($rawFileMatch.Success){
                    if([bool](($rawFileMatch.captures.groups[1].value) -as [xml])){
                        [xml]$XmlDocument = ($rawFileMatch.captures.groups[1].value) -as [xml];
                        $crashMessage = $rawFileMatch.captures.groups[2].value;
                    }
                }
            }
        }
        $testSuiteXmlOut = $rootXmlOut.AppendChild($xmlOut.CreateElement("testsuite"));
        if($XmlDocument -ne $null){
            $testClassName = $XmlDocument.TestCase.name;
            $testSuiteXmlOut.SetAttribute("name",$testSuiteName);
            $testSuitePropertiesXmlOut = $testSuiteXmlOut.AppendChild($xmlOut.CreateElement("properties"));
            $testSuitePropertiesPropertyXmlOut = $testSuitePropertiesXmlOut.AppendChild($xmlOut.CreateElement("property"));
            $testSuitePropertiesPropertyXmlOut.SetAttribute("name","QtVersion");
            $testSuitePropertiesPropertyXmlOut.SetAttribute("value",($XmlDocument.TestCase.Environment.QtVersion));
            $testSuitePropertiesPropertyXmlOut = $testSuitePropertiesXmlOut.AppendChild($xmlOut.CreateElement("property"));
            $testSuitePropertiesPropertyXmlOut.SetAttribute("name","QtBuild");
            $testSuitePropertiesPropertyXmlOut.SetAttribute("value",($XmlDocument.TestCase.Environment.QtBuild));
            $testSuitePropertiesPropertyXmlOut = $testSuitePropertiesXmlOut.AppendChild($xmlOut.CreateElement("property"));
            $testSuitePropertiesPropertyXmlOut.SetAttribute("name","QTestVersion");
            $testSuitePropertiesPropertyXmlOut.SetAttribute("value",($XmlDocument.TestCase.Environment.QTestVersion));
            foreach($testFunction in $XmlDocument.SelectNodes("//TestFunction")){
                $testFunctionName = $testFunction.name;
                $countIncidents = $testFunction.ChildNodes.Count;
                $testFunctionTime = [decimal]$testFunction.Duration.msecs;
                $localTime = $localTime +$testFunctionTime;
                foreach($incident in $testFunction.ChildNodes){
                    if($incident.Name -ne "Incident" -and $incident.Name -ne "Message"){
                        continue;
                    }
                    $incidentName = $testFunctionName;
                    if($incident.DataTag -ne $null){
                        $incidentName = $incidentName + " - " + $incident.DataTag.InnerText;
                    }
                    $incidentName = ($incidentName);
                    $testSuitetestcaseXmlOut = $testSuiteXmlOut.AppendChild($xmlOut.CreateElement("testcase"));
                    $testSuitetestcaseXmlOut.SetAttribute("name",$incidentName);
                    $testSuitetestcaseXmlOut.SetAttribute("classname",$testClassName);
                    $testSuitetestcaseXmlOut.SetAttribute("time",$testFunctionTime/(1000*$countIncidents));
                    if($incident.type -eq "skip"){
                        ++$localSkip;
                        $testSuitetestcaseSkipXmlOut = $testSuitetestcaseXmlOut.AppendChild($xmlOut.CreateElement("skipped"));
                        $testSuitetestcaseSkipXmlOut.SetAttribute("message","file: " + ($incident.file + " line: " + $incident.line + " " + $incident.Description.InnerText));
                    }
                    ElseIf ($incident.type -eq "fail"){
                        ++$localFail;
                        $testSuitetestcaseSkipXmlOut = $testSuitetestcaseXmlOut.AppendChild($xmlOut.CreateElement("failure"));
                        $testSuitetestcaseSkipXmlOut.SetAttribute("message",("file: " + $incident.file + " line: " + $incident.line + " " + $incident.Description.InnerText));
                    }
                    ElseIf ($incident.type -eq "qdebug" -or $incident.type -eq "qwarn" -or $incident.type -eq "system" -or $incident.type -eq "qfatal"){
                        $testSuitetestcaseCerrXmlOut = $testSuitetestcaseXmlOut.AppendChild($xmlOut.CreateElement("system-err"));
                        $testSuitetestcaseCerrXmlOut.AppendChild($xmlOut.CreateTextNode(($incident.Description.InnerText)));
                    }
                    else{
                        ++$localPass;
                    }
                };
            };
        }
        if($localError -eq 1){
            $testSuitetestcaseXmlOut = $testSuiteXmlOut.AppendChild($xmlOut.CreateElement("testcase"));
            $testSuitetestcaseXmlOut.SetAttribute("name","SystemError");
            $testSuitetestcaseXmlOut.SetAttribute("classname",$testClassName);
            $testSuitetestcaseErrorXmlOut = $testSuitetestcaseXmlOut.AppendChild($xmlOut.CreateElement("error"));
            $testSuitetestcaseErrorXmlOut.SetAttribute("message",($crashMessage));
        }
        $testSuiteXmlOut.SetAttribute("time",$localTime/1000);
        $testSuiteXmlOut.SetAttribute("skipped",$localSkip);
        $testSuiteXmlOut.SetAttribute("tests",$localSkip+$localFail+$localError+$localPass);
        $testSuiteXmlOut.SetAttribute("failures",$localFail);
        $testSuiteXmlOut.SetAttribute("errors",$localError);
        $totalTime = $totalTime + $localTime;
        $totalSkip = $totalSkip + $localSkip;
        $totalError = $totalError + $localError;
        $totalFail = $totalFail + $localFail;
        $totalPass = $totalPass + $localPass;
    };
    $rootXmlOut.SetAttribute("time",$totalTime/1000);
    $rootXmlOut.SetAttribute("failures",$totalFail);
    $rootXmlOut.SetAttribute("errors",$totalError);
    $rootXmlOut.SetAttribute("tests",$totalPass);
    $xmlOut.save($pwd.Path + "\testResults.xml");
    if($totalError+$totalFail -gt 0){
        throw;
    }'
    
on_finish:
- ps: if (Test-Path ".\testResults.xml") {(new-object net.webclient).UploadFile("https://ci.appveyor.com/api/testresults/junit/$($env:APPVEYOR_JOB_ID)", (Resolve-Path .\testResults.xml));}

Edit: it looks bad but it needs no customisation to apply to other projects at most you need to add at the beginning a line to cd in the folder where you have the tests

aha_1980

@VRonin:

This turned out to be infinitely more involved than I wanted.

Yeah, as usual in programming ;)

Violet Giraffe

@VRonin said in Integrating Qt Test and Appveyor:

This turned out to be infinitely more involved than I wanted.

Merging different test results in a format that appveyor understands is not a walk in the park.

Wow, hold on there. You don't need to merge anything, Appveyor correctly handles multiple uploaded results (thank goodness!)

test_script:
 - .\bin\release\%BUILD_ARCH%\fso_test.exe -xunitxml > tests_fso.xml
 - ps: (new-object net.webclient).UploadFile("https://ci.appveyor.com/api/testresults/junit/$($env:APPVEYOR_JOB_ID)", (Resolve-Path .\tests_fso.xml))

 - .\bin\release\%BUILD_ARCH%\fso_test_high_level.exe -xunitxml > tests_fso_high_level.xml
 - ps: (new-object net.webclient).UploadFile("https://ci.appveyor.com/api/testresults/junit/$($env:APPVEYOR_JOB_ID)", (Resolve-Path .\tests_fso_high_level.xml))

You can see the complete example and how it looks in Appveyor on my project's GitHub page: https://github.com/VioletGiraffe/file-commander

VRonin

@Violet-Giraffe Interesting, all the times I tried appveyor would refuse my second upload and fail the build. Since when do you have that project doing multiple uploads? It might be a new feature

Violet Giraffe

@VRonin said in Integrating Qt Test and Appveyor:

@Violet-Giraffe Interesting, all the times I tried appveyor would refuse my second upload and fail the build. Since when do you have that project doing multiple uploads? It might be a new feature

It could be new, I had that thought as well. I'm doing it since two days ago so I cannot testify that's how it always used to work.