Testing Axioms
- Testing Axioms
- Software Testing Is a Risk-Based Exercise
- Testing Can't Show That Bugs Don't Exist
- The More Bugs You Find, the More Bugs There Are
- The Pesticide Paradox
- Not All the Bugs You Find Will Be Fixed
- When a Bug's a Bug Is Difficult to Say
- Product Specifications Are Never Final
- Software Testers Aren't the Most Popular Members of a Project Team
- Software Testing Is a Disciplined Technical Profession
- About This Article
This article lists the axioms, or truisms, related to software testing. Think of them as the "rules of the road" or the "facts of life" for software testing and software development. Each axiom is a little tidbit of knowledge that helps put some aspect of the overall software testing process into perspective.
It's Impossible to Test a Program Completely
As a new tester, you might believe that you can approach a piece of software, fully test it, find all the bugs, and assure that the software is perfect. Unfortunately, this isn't possible, even with the simplest programs, due to four key reasons:
-
The number of possible inputs is very large.
-
The number of possible outputs is very large.
-
The number of paths through the software is very large.
-
The software specification is subjective. You might say that a bug is in the eye of the beholder.
Multiply all these "very large" possibilities together, and you get a set of test conditions that's too large to attempt. If you don't believe it, consider the example shown in Figure 1, the Microsoft Windows Calculator.
){kind=link}
Even a simple program such as the Windows Calculator is too complex to completely test.
Assume that you are assigned to test the Windows Calculator. You decide to start with addition. You try 1+0=. You get an answer of 1. That's correct. Then you try 1+1=. You get 2. How far do you go? The calculator accepts a 32-digit number, so you must try all the possibilities up to
1+99999999999999999999999999999999=
Once you complete that series, you can move on to 2+0=, 2+1=, 2+2=, and so on. Eventually you'll get to
99999999999999999999999999999999+99999999999999999999999999999999=
Next you should try all the decimal values: 1.0+0.1, 1.0+0.2, and so on.
Once you verify that regular numbers sum properly, you need to attempt illegal inputs to assure that they're properly handled. Remember, you're not limited to clicking the numbers onscreen—you can press keys on your computer keyboard, too. Good values to try might be 1+a, z+1, 1a1+2b2, …. There are literally billions upon billions of these.
Edited inputs must also be tested. The Windows Calculator allows the Backspace and Delete keys, so you should try them. 1<backspace>2+2 should equal 4. Everything you've tested so far must be retested by pressing the Backspace key for each entry, for each two entries, and so on.
If you or your heirs manage to complete all these cases, you can then move on to adding three numbers, then four numbers, ….
There are so many possible entries that you could never complete them, even if you used a super computer to feed in the numbers. And that's only for addition. You still have subtraction, multiplication, division, square root, percentage, and inverse to cover.
The point of this example is to demonstrate that it's impossible to completely test a program, even software as simple as a calculator. If you decide to eliminate any of the test conditions because you feel they're redundant or unnecessary, or just to save time, you've decided not to test the program completely.