Four Classic Problems with Scripted Testing
- Problem 1: Almost None of the Test Scripts Were Correct
- Problem 2: Inattentional Blindness
- Problem 3: Inaccurate Perceptions
- Problem 4: Scripted Test Cases Gave the Illusion of Progress
- Additional Resources
Like this article? We recommend
Like this article? We recommend
Like this article? We recommend
On a recent project, we were instructed by the test manager to create hundreds of scripted test cases. Our intention was to execute these tests on the first good build we received from the development staff. We spent months developing the test scripts and planning for their execution. We created the test cases, entered them in our test management tool, traced them to requirements, and reviewed them with the business and other testers.
In spite of months of planning and hard work, I noticed some problems on our first day of test execution.
Problem 1: Almost None of the Test Scripts Were Correct
The expected results didn’t match the actual requirements once the day of testing came around. Given the number of test cases and the number of changing requirements, it was impossible to keep them up to date. I won’t say all that time was wasted, because it did get us looking at the requirements, and it did give us something close to resembling an expected result. But it didn’t do what it was intended to do: It didn’t accurately capture the steps needed to execute a test, or serve as an effective oracle for expected results. That is, it didn’t reduce the need for brain-engaged testing.