Solved Integrating Qt Test and Appveyor
-
Hi,
Don't know if that helps but the PiwikTracker seems to use the text output of the unit tests.
Hope that gives some clue to go further.
-
@SGaist I might be going blind, but I can't see any tests at all in that project: https://ci.appveyor.com/project/pbek/qt-piwik-tracker/build/1.0.24
-
My bad, looks like they are run in travis.
-
Hi @VRonin,
Did anyone manage to integrate Qt test results into appveyor? If so, could you share how?
I haven't yet (it's been on my list of things to do for a while).
There's two levels of possible integration - the most basic is simply making Qt output the XUnit format, and submitting that to AppVeyor.
A quick GitHub search for yaml files that contain
-xunitxml
reveals a couple of projects that have tried this. But the only one that seems to have succeeded (and is publicly accessible on AppVeyor) is AureBeshTranslator:- https://github.com/Ometeo/AureBeshTranslator/blob/master/appveyor.yml#L51 - the AppVeyor build configuration ib GitHub
- https://ci.appveyor.com/project/Ometeo/AureBeshTranslator/build/tests - the test results on AppVeyor
This "integration" looks to be very quick and easy, but only shows the final test outcomes (ie shows nothing at all until the tests are completed).
The other integration option is to use the Build Worker API to report tests before, during, and after execution. For very big projects, this would be very nice. It would allow AppVeyor to show in their UI all tests that are going to be run, and check them off (turn green (or red)) as they execute. Of course, with just a small number of tests, there'd be no point.
To do this integration, you'd have to use C++/Qt code to introspect the test classes, and hook into QtTest's init and cleanup functions, and submit via HTTP. Should be fairly easy, and this is something I've been toying with, but only in between lots of other side projects ;) Should be do-able. I'll definitely be interested if you find any projects that do this already :)
Cheers.
-
@Paul-Colby said in Integrating Qt Test and Appveyor:
This "integration" looks to be very quick and easy, but only shows the final test outcomes (ie shows nothing at all until the tests are completed).
Thank you very much. I was looking at doing the same only using
curl
instead of powershell.It works!
For reference my appveyor.yml looks like this:
EDIT: the script below is wrong, scroll down
test_script: - cd %TESTOUTPUTFOLDER%/bin - set PATH=../lib;%PATH% - call "../../../runtests.bat" - ps: (new-object net.webclient).UploadFile("https://ci.appveyor.com/api/testresults/junit/$($env:APPVEYOR_JOB_ID)", (Resolve-Path .\testsResults.xml))
And
runtests.bat
is simply:for %%a in (*.exe) do %%a -xunitxml > testsResults.xml
-
Shouldn't it be
for %%a in (*.exe) do %%a -xunitxml >> testsResults.xml
? -
It was not the only problem. Updated script in
.appveyor.yml
EDIT: the script below is wrong, scroll down
test_script: - If NOT exist "%TESTOUTPUTFOLDER%/bin" (appveyor exit) - cd %TESTOUTPUTFOLDER%/bin - set PATH=../lib;%PATH% - for %%a in (*.exe) do (%%a -xunitxml >> testsResults.xml) on_finish: - ps: (new-object net.webclient).UploadFile("https://ci.appveyor.com/api/testresults/junit/$($env:APPVEYOR_JOB_ID)", (Resolve-Path .\testsResults.xml))
This will probably still stop running tests as soon as one fails but good enough for rock and roll.
The interesting detail is that you have to send the outputs to the
junit
reader of appveyor instead of the xunit one -
This turned out to be infinitely more involved than I wanted.
Merging different test results in a format that appveyor understands is not a walk in the park.
This is my final script:
- for %%a in (*.exe) do (%%a -xml > %%a_tstres.xml & set preventFail = 1) - ps: '$xmlOut = new-object System.Xml.XmlDocument; $xmlOut.AppendChild($xmlOut.CreateXmlDeclaration("1.0","UTF-8",$null)); $rootXmlOut = $xmlOut.CreateElement("testsuites"); $xmlOut.AppendChild($rootXmlOut); $totalPass = 0; $totalFail = 0; $totalError = 0; $totalSkip = 0; $totalTime = 0; $crashMessage = $null; Get-ChildItem ".\" -Filter "*_tstres.xml" | Foreach-Object { $XmlDocument = $null; $localPass = 0; $localFail = 0; $localError = 0; $localSkip = 0; $localTime = 0; $currentFilePath = $_.FullName; $testSuiteName = $_.BaseName.subString(0,$_.BaseName.Length - 11); if([bool]((Get-Content -Path $currentFilePath) -as [xml])){ [xml]$XmlDocument = (Get-Content -Path $currentFilePath) -as [xml]; } else{ $localError = 1; $rawFilecontent = [IO.File]::ReadAllText($currentFilePath); if([string]::IsNullOrEmpty($rawFilecontent)){ $crashMessage = "Output file is empty: " + $currentFilePath; } else{ $crashMessage = $rawFilecontent; $rawFileMatch = [regex]::match($rawFilecontent,"(?s)(.+<\/TestCase>)(.*)"); if($rawFileMatch.Success){ if([bool](($rawFileMatch.captures.groups[1].value) -as [xml])){ [xml]$XmlDocument = ($rawFileMatch.captures.groups[1].value) -as [xml]; $crashMessage = $rawFileMatch.captures.groups[2].value; } } } } $testSuiteXmlOut = $rootXmlOut.AppendChild($xmlOut.CreateElement("testsuite")); if($XmlDocument -ne $null){ $testClassName = $XmlDocument.TestCase.name; $testSuiteXmlOut.SetAttribute("name",$testSuiteName); $testSuitePropertiesXmlOut = $testSuiteXmlOut.AppendChild($xmlOut.CreateElement("properties")); $testSuitePropertiesPropertyXmlOut = $testSuitePropertiesXmlOut.AppendChild($xmlOut.CreateElement("property")); $testSuitePropertiesPropertyXmlOut.SetAttribute("name","QtVersion"); $testSuitePropertiesPropertyXmlOut.SetAttribute("value",($XmlDocument.TestCase.Environment.QtVersion)); $testSuitePropertiesPropertyXmlOut = $testSuitePropertiesXmlOut.AppendChild($xmlOut.CreateElement("property")); $testSuitePropertiesPropertyXmlOut.SetAttribute("name","QtBuild"); $testSuitePropertiesPropertyXmlOut.SetAttribute("value",($XmlDocument.TestCase.Environment.QtBuild)); $testSuitePropertiesPropertyXmlOut = $testSuitePropertiesXmlOut.AppendChild($xmlOut.CreateElement("property")); $testSuitePropertiesPropertyXmlOut.SetAttribute("name","QTestVersion"); $testSuitePropertiesPropertyXmlOut.SetAttribute("value",($XmlDocument.TestCase.Environment.QTestVersion)); foreach($testFunction in $XmlDocument.SelectNodes("//TestFunction")){ $testFunctionName = $testFunction.name; $countIncidents = $testFunction.ChildNodes.Count; $testFunctionTime = [decimal]$testFunction.Duration.msecs; $localTime = $localTime +$testFunctionTime; foreach($incident in $testFunction.ChildNodes){ if($incident.Name -ne "Incident" -and $incident.Name -ne "Message"){ continue; } $incidentName = $testFunctionName; if($incident.DataTag -ne $null){ $incidentName = $incidentName + " - " + $incident.DataTag.InnerText; } $incidentName = ($incidentName); $testSuitetestcaseXmlOut = $testSuiteXmlOut.AppendChild($xmlOut.CreateElement("testcase")); $testSuitetestcaseXmlOut.SetAttribute("name",$incidentName); $testSuitetestcaseXmlOut.SetAttribute("classname",$testClassName); $testSuitetestcaseXmlOut.SetAttribute("time",$testFunctionTime/(1000*$countIncidents)); if($incident.type -eq "skip"){ ++$localSkip; $testSuitetestcaseSkipXmlOut = $testSuitetestcaseXmlOut.AppendChild($xmlOut.CreateElement("skipped")); $testSuitetestcaseSkipXmlOut.SetAttribute("message","file: " + ($incident.file + " line: " + $incident.line + " " + $incident.Description.InnerText)); } ElseIf ($incident.type -eq "fail"){ ++$localFail; $testSuitetestcaseSkipXmlOut = $testSuitetestcaseXmlOut.AppendChild($xmlOut.CreateElement("failure")); $testSuitetestcaseSkipXmlOut.SetAttribute("message",("file: " + $incident.file + " line: " + $incident.line + " " + $incident.Description.InnerText)); } ElseIf ($incident.type -eq "qdebug" -or $incident.type -eq "qwarn" -or $incident.type -eq "system" -or $incident.type -eq "qfatal"){ $testSuitetestcaseCerrXmlOut = $testSuitetestcaseXmlOut.AppendChild($xmlOut.CreateElement("system-err")); $testSuitetestcaseCerrXmlOut.AppendChild($xmlOut.CreateTextNode(($incident.Description.InnerText))); } else{ ++$localPass; } }; }; } if($localError -eq 1){ $testSuitetestcaseXmlOut = $testSuiteXmlOut.AppendChild($xmlOut.CreateElement("testcase")); $testSuitetestcaseXmlOut.SetAttribute("name","SystemError"); $testSuitetestcaseXmlOut.SetAttribute("classname",$testClassName); $testSuitetestcaseErrorXmlOut = $testSuitetestcaseXmlOut.AppendChild($xmlOut.CreateElement("error")); $testSuitetestcaseErrorXmlOut.SetAttribute("message",($crashMessage)); } $testSuiteXmlOut.SetAttribute("time",$localTime/1000); $testSuiteXmlOut.SetAttribute("skipped",$localSkip); $testSuiteXmlOut.SetAttribute("tests",$localSkip+$localFail+$localError+$localPass); $testSuiteXmlOut.SetAttribute("failures",$localFail); $testSuiteXmlOut.SetAttribute("errors",$localError); $totalTime = $totalTime + $localTime; $totalSkip = $totalSkip + $localSkip; $totalError = $totalError + $localError; $totalFail = $totalFail + $localFail; $totalPass = $totalPass + $localPass; }; $rootXmlOut.SetAttribute("time",$totalTime/1000); $rootXmlOut.SetAttribute("failures",$totalFail); $rootXmlOut.SetAttribute("errors",$totalError); $rootXmlOut.SetAttribute("tests",$totalPass); $xmlOut.save($pwd.Path + "\testResults.xml"); if($totalError+$totalFail -gt 0){ throw; }' on_finish: - ps: if (Test-Path ".\testResults.xml") {(new-object net.webclient).UploadFile("https://ci.appveyor.com/api/testresults/junit/$($env:APPVEYOR_JOB_ID)", (Resolve-Path .\testResults.xml));}
Edit: it looks bad but it needs no customisation to apply to other projects at most you need to add at the beginning a line to
cd
in the folder where you have the tests -
This turned out to be infinitely more involved than I wanted.
Yeah, as usual in programming ;)
-
@VRonin said in Integrating Qt Test and Appveyor:
This turned out to be infinitely more involved than I wanted.
Merging different test results in a format that appveyor understands is not a walk in the park.
Wow, hold on there. You don't need to merge anything, Appveyor correctly handles multiple uploaded results (thank goodness!)
test_script: - .\bin\release\%BUILD_ARCH%\fso_test.exe -xunitxml > tests_fso.xml - ps: (new-object net.webclient).UploadFile("https://ci.appveyor.com/api/testresults/junit/$($env:APPVEYOR_JOB_ID)", (Resolve-Path .\tests_fso.xml)) - .\bin\release\%BUILD_ARCH%\fso_test_high_level.exe -xunitxml > tests_fso_high_level.xml - ps: (new-object net.webclient).UploadFile("https://ci.appveyor.com/api/testresults/junit/$($env:APPVEYOR_JOB_ID)", (Resolve-Path .\tests_fso_high_level.xml))
You can see the complete example and how it looks in Appveyor on my project's GitHub page: https://github.com/VioletGiraffe/file-commander
-
@Violet-Giraffe Interesting, all the times I tried appveyor would refuse my second upload and fail the build. Since when do you have that project doing multiple uploads? It might be a new feature
-
@VRonin said in Integrating Qt Test and Appveyor:
@Violet-Giraffe Interesting, all the times I tried appveyor would refuse my second upload and fail the build. Since when do you have that project doing multiple uploads? It might be a new feature
It could be new, I had that thought as well. I'm doing it since two days ago so I cannot testify that's how it always used to work.