CERN using Squish for Qt based Multi-Platform SCADA GUI Regression Testing
Ten years ago CERN developed a toolkit called the JCOP (Joint Controls Projects) Framework for the development of industrial control systems. This framework was designed to support the creation of systems for controlling experiments, particle accelerators, and CERN's technical infrastructure.
The JCOP Framework is a software layer designed to simplify and unify access to the underlying SCADA (Supervisory Control and Data Acquisition) technology built with WinCC-OA, itself built on Qt. The JCOP Framework consists of both code libraries and also a substantial amount of GUI software. Since the Framework is so widely used throughout CERN, it was necessary to create a formalized automated quality assurance process to provide confidence and ensure stability. This demanded a tool that would reliably support daily regression testing on both Windows and Linux.
After evaluating various GUI test tools, CERN test engineers, Paul Burkimsher, Manuel Gonzalez-Berges, and Stefan Klikovits chose to use Squish, and CERN acquired its first Squish license in 2006.
The CERN test engineers' initial approach was to make extensive use of Squish's test recording facility (in their case they chose to have the tests recorded in Python). Unfortunately, this wasn't always as robust as they had hoped because changes to their complex software stack (Qt, WinCC-OA, JCOP itself), or improvements in newer versions of Squish, could sometimes affect Squish's reliability of playback.
To meet this challenge the CERN test engineers analysed their recorded Squish tests and used this as the basis for creating their own high level object-oriented test library. Their library contains a class for every GUI widget they test. Having the library has allowed them to drastically cut down the duplicate code produced by simply recording tests, and to abstract GUI interactions into calls to their library. This has resulted in them being able to reduce their tests to library calls and verifications. And this in turn has made test development much faster and easier, since test engineers can build on all the predefined functionality. Of course, this work has also made all of their test cases far more robust, since any change to the underlying software stack that affects the tests can now be addressed by updating a single library, rather than having to check and change a large body of test cases.
Another important matter for the CERN test engineers is the ability to test on both Windows and Linux. They tried a number of different solutions and in the end adopted a CI (Continuous Integration) approach using the Hudson tool (and for which there is a Squish integration). Initially they ran all their tests on different physical machines, but they have now switched to using Virtual Machines hosted in the CERN Computer Centre—a change which made no difference to their testing or to Squish.
Recently, the CERN test engineers published a brief technical paper describing their use of Squish, the challenges that arose, and the solutions they found. This paper was presented at the 13th International Conference on Accelerator and Large Experimental Physics Control Systems, and is available from http://www.esrf.eu/icalepcs2011/papers/thbhmust01.pdf.
Squish itself, and all the tests created with it, are completely cross-platform, and work on Windows, Linux/Unix, Mac OS X, and embedded Linux.
froglogic GmbH is a software company based in Hamburg, Germany. Their flagship product is Squish, the market-leading automated testing tool for GUI applications. More about froglogic is at http://www.froglogic.com.
CERN, the European Organization for Nuclear Research, is one of the world's largest and most respected centres for scientific research. Its business is fundamental physics, finding out what the Universe is made of and how it works. At CERN, the world's largest and most complex scientific instruments are used to study the basic constituents of matter—the fundamental particles. By studying what happens when these particles collide, physicists learn about the laws of Nature.
[quote author="reggie" date="1321352470"]The CERN test engineers' initial approach was to make extensive use of Squish's test recording facility (in their case they chose to have the tests recorded in Python). Unfortunately, this wasn't always as robust as they had hoped because changes to their complex software stack (Qt, WinCC-OA, JCOP itself), or improvements in newer versions of Squish, could sometimes affect Squish's reliability of playback.[/quote]
Do I understand you correctly that Squish tests may break by simply updating Squish itself? Really? That does not sound so good to me. The chosen solution, while very vaguely described, sounds like a move away from Squish, not towards it. Is that correct? Did they basically revert to implementing their tests manually, because Squish could not deal with the complexity, or do I just misunderstand?
Did they really end up creating a test class for every class they wanted tested? That sounds like a big effort, and it reads like they are now basically using some form of unit testing instead of GUI testing as Squish provides?
No, Squish tests do not break because of updating Squish. I guess we'd have no customers if that would be the case :-)
The wording there may be unfortunate. What is meant is that you should not just record tests and leave it that way but rather see the recorder as an assistive tool to create tests and use methods such as refactoring to generate stable and easy to maintain scripts.
Something along the lines as you can find in our documentation at http://doc.froglogic.com/squish/4.1/all/tut-qt-creating-manual-tests.html
It is a common, unfortunately wrong expectation that GUI testing means that you just need to record and that's it. Short-term that may be ok and work but long-term this will be not what you want to maintain. So indeed, test automation is a mix of script development and using assistive tools such as recording, snippet recording, the spy, object map, etc.