- 29.1 Three Grains of Rice
- 29.2 Understanding Has to Grow
- 29.3 First Day Automated Testing
- 29.4 Attempting to Get Automation Started
- 29.5 Struggling with (against) Management
- 29.6 Exploratory Test Automation: Database Record Locking
- 29.7 Lessons Learned from Test Automation in an Embedded Hardware-Software Computer Environment
- 29.8 The Contagious Clock
- 29.9 Flexibility of the Automation System
- 29.10 A Tale of Too Many Tools (and Not Enough Cross-Department Support)
- 29.11 A Success with a Surprising End
- 29.12 Cooperation Can Overcome Resource Limitations
- 29.13 An Automation Process for Large-Scale Success
- 29.14 Test Automation Isn't Always What It Seems
29.9 Flexibility of the Automation System
Mike Bartley, United Kingdom
Lecturer and consultant
We developed our test automation system ourselves and devised ways to adapt our automated testing to be more efficient in ways that we have not seen elsewhere.
Because we had already developed an inhouse software version control and build system, it was fairly easy to integrate our automation tool with our build system. This made our testing more efficient because we could selectively test only those modules that had changed, as shown by our source code dependency tree. If nothing that a particular test depended on had changed, that test would not be executed. This dramatically reduced the build and cycle time and thus allowed us to put in place continuous integration of builds and tests. We did keep an option that forced all tests to rebuild and run if we wanted to run a full regression test.
We made it easy to remove tests from the test suite when a test needed updating because of changes in the software it was testing. Although we were then cutting down on the test coverage because the test(s) were not run, it meant that the maintenance of those tests didn’t have to be done immediately, thereby stopping the rest of the test suite from running.
We extended this to a way of “banning” specific tests for various reasons:
- The software has changed, but the tests have not yet been updated.
- A test is known to be giving a false failure (i.e., it fails but it should pass).
- A test is not restoring test machines to a known good state.
This idea proved to be a major benefit to our efficiency.