This chapter is from the book
This chapter is from the book
This chapter is from the book
1.5 The Limitations of Testing
In spite of its critical nature, testing has a number of pitfalls that make it far less effective and efficient than it should be. Testing is relatively ineffective in the sense that a significant number of residual defects remain in a completed system when it is placed into operation. Testing is also relatively inefficient when you consider the large amount of effort, funding, and time that is currently spent to find defects.
According to Capers Jones, most types of testing find only about 35% of the software defects [Jones 2013]. This is consistent with the following, more detailed analysis of defect detection rates as a function of test type and test capabilities, as shown in Table 1.3 [McConnell 2004].
Table 1.3 Defect Detection Rate
|
|
Defect Detection Rates |
||||
|
Test Type |
Lowest |
Mode |
Highest |
||
|
Unit Test |
15% |
30% |
50% |
||
|
Component Test |
20% |
30% |
35% |
||
|
Integration Test |
25% |
35% |
40% |
||
|
System Test |
25% |
40% |
55% |
||
|
Regression Test |
15% |
25% |
30% |
||
|
Low-volume Beta Test |
25% |
35% |
40% |
||
|
High-volume Beta Test |
60% |
75% |
85% |
||
|
Source: McConnell 2004 |
|||||
As Table 1.4 shows, no single type of testing is very effective at uncovering defects, regardless of defect type. Even when all of these testing methods are used on an average project, they still only identify four out of five of the code defects.
Table 1.4 Defect Detection Rate
|
Static Verification |
Project Defect Detection Rate |
||||
|
|
Worst |
Average |
Best |
||
|
Desk Checking |
23% |
25% |
27% |
||
|
Static Analysis |
0% |
55% |
55% |
||
|
Inspection |
0% |
0% |
93% |
||
|
Static Subtotal |
19% |
64% |
98% |
||
|
|
|
|
|
||
|
Testing |
Project Defect Detection Rate |
||||
|
|
Worst |
Average |
Best |
||
|
Unit Test |
28% |
30% |
32% |
||
|
Function Test |
31% |
33% |
35% |
||
|
Regression Test |
10% |
12% |
14% |
||
|
Component Test |
28% |
30% |
32% |
||
|
Performance Test |
6% |
10% |
14% |
||
|
System Test |
32% |
34% |
36% |
||
|
Acceptance Test |
13% |
15% |
17% |
||
|
Testing Subtotal |
72% |
81% |
87% |
||
|
|
|
|
|
||
|
Cumulative Total |
81.1% |
95.6% |
99.96% |
||
|
Source: Jones 2013b |
|||||