This chapter is from the book
This chapter is from the book
This chapter is from the book
Impact on Testing
The quality of testing is not so much determined by how much testing is performed or how fast testing is performed, but rather by what is tested. When there is a rush to get the product to the customer, the temptation (and often the practice) is to shortcut testing. When testing is rushed or arbitrarily reduced in scope, defects are missed by the testers but easily found by the customers and users of the software.
The key to reducing this negative impact is to design tests based on risk, without the intention of validating every possible combination of inputs and actions. In other words, pick your test cases carefully and design a test that is achievable. If we understand the complexity of systems, it is easy to see that there must be a better way to test software than the brute force method of trying all of the combinations. In Figure 8.1, you can see how the complexity of the test increases as the phases of testing become more encompassing. At the unit testing level, it is not impossible to completely test a single unit, but when you combine units and consider all of the possible interactions between them, the complexity starts to expand. At the system test level, the complexity is too large to test completely.
)
Figure 8.1 Complexity Increases as Testing Progresses.
Don’t be discouraged: The point is that to have enough time to perform an effective test, you must design a test based on an independent set of criteria, as opposed to a test of every possible combination of input. Possible criteria can include requirements, business processes, and business cases or scenarios.
When too much testing is attempted within the allotted time, testing exhibits the following characteristics:
- reduced test coverage
- increased risk of regression defects
- fatigue, burnout, and low morale
Let’s examine the nature of each of these impacts on testing.
Reduced Test Coverage
Test coverage gets reduced when test cases are skipped for the sake of time. One could make a convincing argument that since you can’t test everything anyway, why not toss out a few test cases? The key is in knowing the relative risk of the functions or attributes that the test cases are to test. The risk of the software functions is determined in a risk assessment. When you know the relative risk, you can prioritize the test cases, perform the high risk cases first, and work your way down to the lowest risk cases.
Reduced test coverage brings to light two factors that many people fail to consider when planning tests. First, they fail to allow for repeat tests (regression testing), and second, they do not know the risk of the functions to be tested. Performing a risk assessment should be a part of every testing process.
Increased Risk of Regression Defects
We have already discussed that you can only reasonably test a subset of all possible test cases, so what does this mean? Perhaps a better way to visualize this concept is with a graph of what typically happens during a rushed testing schedule. The diagram of manual testing in Chapter 5 (see Figure 5.1 on page 61) shows that testing takes place at the very end of the project. This “big bang” approach to testing can be relieved a great deal by continuous verification and validation throughout a project.
In Figure 5.1, a noble effort is made to completely test all cases each time a new release of the software is received from the developers. All goes well until the testers burn out by testing the software manually. As the deadline approaches (and passes), the work keeps coming, and the degree of testing is gradually compromised until only the changes are tested. This exposes the project to those nasty defects that get introduced as the result of fixing another defect: regression defects. In effect, the quality and the degree of testing are sacrificed to meet the deadline.
One way to deal with the scenario shown in Figure 5.1 is to automate testing. This solution is not a silver bullet, however. Groundwork must be laid in terms of process and people before the tool will be effective, as discussed in Chapter 5. Figure 5.2 (see page 61) shows that automated testing, like manual testing, requires a significant investment at the beginning to create test scripts and test cases, but the payback comes as those tests are repeated and verified by the automated test tool. Because the time required to repeat an automated test is much shorter than that for the original manual test, regression testing can be performed more times.
If you are currently using automated test tools effectively in your organization, you may be puzzled by our focus on manual testing. We have found that many people are still using manual methods and are struggling to maximize their test time. Automated testing helps you to leverage your time. However, the challenge remains to test the right conditions, as opposed to as many conditions as possible.
Fatigue, Burnout, and Low Morale
One of the most common results of trying to do too much testing in a given time is for people to simply give out or give up. Many of the testers who attend our seminars report that they work in shifts and on weekends just to keep up with the developers. Part of the problem might be management’s perception of testing. If the perception is that testing should be quick, easy, and performed by anyone with at least two fingers with which to type, then management will probably not be expecting burnout and low morale. Even if management has the attitude that long hours are part of the territory, the fact remains that tired, burned-out people are not very attentive to finding defects.