This chapter is from the book
This chapter is from the book
This chapter is from the book
Who Should Test Software?
In the early days of software engineering, projects were managed according to the “waterfall model” (see Figure 1.1).1 In this model, each part of the development process was performed as a separate “phase,” with the signed-off output of one phase being the input for the next. So the product managers or business analysts would create the product requirements, and after that was done the requirements would be handed to designers and architects to produce a software specification. Developers would be given the specification in order to produce code, and the code would be given to testers to do quality assurance. Finally the tested software could be released to customers (usually initially to a select few, known as beta testers).
)
Figure 1.1. The phases of development in the waterfall software project management process.
This approach to software project management imposes a separation between coders and testers, which turns out to have both benefits and drawbacks to the actual work of testing. The benefit is that by separating the duties of development and testing the code, there are more people who can find bugs. We developers can sometimes get attached to the code we’ve produced, and it can take a fresh pair of eyes to point out the flaws. Similarly, if any part of the requirements or specification is ambiguous, a chance exists that the tester and developer interpret the ambiguity in different ways, which increases the chance that it gets discovered.
The main drawback is cost. Table 1.1, reproduced from Code Complete, 2nd Edition, by Steve McConnell (Microsoft Press, 2004), shows the results of a survey that evaluated the cost of fixing a bug as a function of the time it lay “dormant” in the product. The table shows that fixing bugs at the end of a project is the most expensive way to work, which makes sense: A tester finds and reports a bug, which the developer must then interpret and attempt to locate in the source. If it’s been a while since the developer worked on that project, then the developer must review the specifications and the code. The bug-fix version of the code must then be resubmitted for testing to demonstrate that the issue has been resolved.
Table 1.1. Cost of Fixing Bugs Found at Different Stages of the Software Development Process
|
Cost of Bugs |
Time Detected |
|
|
|
|
|
Time Introduced |
Requirements |
Architecture |
Coding |
System Test |
Post-Release |
|
Requirements |
1 |
3 |
5–10 |
10 |
10–100 |
|
Architecture |
- |
1 |
10 |
15 |
25–100 |
|
Coding |
- |
- |
1 |
10 |
10–25 |
Where does this additional cost come from? A significant part is due to the communication between different teams: your developers and testers may use different terminology to describe the same concepts, or even have entirely different mental models for the same features in your app. Whenever this occurs, you’ll need to spend some time clearing up the ambiguities or problems this causes.
The table also demonstrates that the cost associated with fixing bugs at the end of the project depends on how early the bug was injected: A problem with the requirements can be patched up at the end only by rewriting a whole feature, which is a very costly undertaking. This motivates waterfall practitioners to take a very conservative approach to the early stages of a project, not signing off on requirements or specification until they believe that every “i” has been dotted and every “t” crossed. This state is known as analysis paralysis, and it increases the project cost.
Separating the developers and testers in this way also affects the type of testing that is done, even though there isn’t any restriction imposed. Because testers will not have the same level of understanding of the application’s internals and code as the developers do, they will tend to stick to “black box” testing that treats the product as an opaque unit that can be interacted with only externally. Third-party testers are less likely to adopt “white box” testing approaches, in which the internal operation of the code can be inspected and modified to help in verifying the code’s behavior.
The kind of test that is usually performed in a black box approach is a system test, or integration test. That’s a formal term meaning that the software product has been taken as a whole (that is, the system is integrated), and testing is performed on the result. These tests usually follow a predefined plan, which is the place where the testers earn their salary: They take the software specification and create a series of test cases, each of which describes the steps necessary to set up and perform the test, and the expected result of doing so. Such tests are often performed manually, especially where the result must be interpreted by the tester because of reliance on external state, such as a network service or the current date. Even where such tests can be automated, they often take a long time to run: The entire software product and its environment must be configured to a known baseline state before each test, and the individual steps may rely on time-consuming interactions with a database, file system, or network service.
Beta testing, which in some teams is called customer environment testing, is really a special version of a system test. What is special about it is that the person doing the testing probably isn’t a professional software tester. If any differences exist between the tester’s system configuration or environment and the customer’s, or use cases that users expect to use and the project team didn’t consider, this will be discovered in beta testing, and any problems associated with this difference can be reported. For small development teams, particularly those who cannot afford to hire testers, a beta test offers the first chance to try the software in a variety of usage patterns and environments.
Because the beta test comes just before the product should ship, dealing with beta feedback sometimes suffers as the project team senses that the end is in sight and can smell the pizza at the launch party. However, there’s little point in doing the testing if you’re not willing to fix the problems that occur.
Developers can also perform their own testing. If you have ever pressed Build & Debug in Xcode, you have done a type of white-box testing: You have inspected the internals of your code to try to find out more about whether its behavior is correct (or more likely, why it isn’t correct). Compiler warnings, the static analyzer, and Instruments are all applications that help developers do testing.
The advantages and disadvantages of developer testing almost exactly oppose those of independent testing: When developers find a problem, it’s usually easier (and cheaper) for them to fix it because they already have some understanding of the code and where the bug is likely to be hiding. In fact, developers can test as they go, so that bugs are found very soon after they are written. However, if the bug is that the developer doesn’t understand the specification or the problem domain, this bug will not be discovered without external help.