(squish-python) squishrunner returns zero (0) exit code when using --retry 1 even if there are fatal errors

[Sawjan]

Steps to reproduce:

1. Install python urllib3 module: pip install urllib3==2.2.3
2. Use the following code in the squish test script (this will cause fatal error)

   import urllib
   
   urllib.request.urlretrieve(<some-url>)
   

3. Run the squish tests:

   squishrunner --testsuite <path-to-suite> --retry 1 --reportgen stdout --exitCodeOnFail 1
   

Fatal error:

START     	Start 'gui'                   	Test 'gui' started
START_TEST_CASE	Start 'tst_activity'          	Test 'tst_activity' started (tst_activity)
FATAL     	/drone/src/test/gui/shared/scripts/helpers/SyncHelper.py:22: Evaluation error	AttributeError: module 'urllib' has no attribute 'request'
END_TEST_CASE	End 'tst_activity'            	End of test 'tst_activity'
END       	End 'gui'                     	End of test 'gui'
*******************************************************
Summary:
Number of Test Cases:	   1
Number of Tests:	   0
Number of Errors:	   0
Number of Fatals:	   1
Number of Fails:	   0
Number of Passes:	   0
Number of Expected Fails:	   0
Number of Unexpected Passes:	   0
Number of Warnings:	   0
*******************************************************

Check exit code:

> echo $?
0

The exit code should be non-zero in terms of errors

Note: If I remove --retry 1 option and run the tests again then the exit code will be 1

Squish details:

• Squish version: 7.2.1
• Built Qt version: 6.6.x

[jsulm]

@Sawjan said in (squish-python) squishrunner returns zero (0) exit code when using --retry 1 even if there are fatal errors:

urllib.request.urlretrieve(<some-url>)

Pay attention to the error message you get.
Replace it with:

import urllib.request

[Sawjan]

Yes, import urllib.request is the correct one. But I used import urllib just to reproduce the fatal error while running the squish test. urllib is not the concern of this topic. I want to discuss the exit code of squishrunner with --retry 1 whenever there is a fatal error. I would expect the non-zero exit code but it returns 0.