Like this article? We recommend
Like this article? We recommend
Like this article? We recommend
Impacting Software Quality
Although software quality is not the focus of this text, we do want to describe what we mean by software quality within the context of how AST impacts it.
Think of quality first based on the following two definitions and then apply them to the context of software.
ISO definition of quality: “The totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs”.[11]
Another definition of quality: The nature of a system that performs as expected (meeting all requirements), making available the required system features in a consistent fashion, demonstrating high availability under any type of constraint, such as stress, concurrency, security breach, and so on, thus consistently meeting the user’s expectations and satisfying the system’s user needs (a user can be a person or another system).
In both definitions, the essence of software quality is how well it meets expected functionality and performance.
There are also many metrics associated with quantifying software quality. Examples include
- Software defect density (number of defects per source lines of code or Function Points, or any other source code metric used on the program)
- Mean time to failure (MTTF)
- Mean time to critical failure (MTTCF)
- Number of high-priority defects that have not been found and removed
- Reliability (probability that no failure will occur in the next n time intervals)
One method for projecting software quality is the use of software reliability models. Software reliability models predict the results from continued testing and overall defect statistics associated with a software baseline under test. Therefore, one can use software reliability models to forecast the impact of additional testing on software quality.
Typical software reliability model results project the number of undiscovered software defects over time. One expects that as the test program continues, more testing will result in fewer defects yet undiscovered. A conceptual software model projection is shown in Figure Improved Quality with increased test time. The slope of the curve is uniquely projected for each project, based on defects discovered to date combined with the size and complexity of the software under test.
Figure "Improved Quality with increased test time"
The use of AST enables more testing to be completed. In particular, AST offers these benefits:
- A full set of regression tests can be run every time without the testers having to worry about which portions of the system should be retested (even though ideally an analysis takes place to identify the areas affected by a change).
- There is increased test coverage; AST tests can cover more combinations (more data variations and test scenarios) and more paths and branches than manual testing can, given the same amount of time.
- AST increases the amount of effective test time, because
- It can be run during off-shifts (lunch, nights, weekends).
- During automated test runs, testers can focus on more complex and new issues/areas.
- For some systems, AST may be able to run faster than clock-time rates (careful when applying this, sometimes there are system dependencies that need to be considered).
- Automated tests don’t get tired or inattentive.
- Automated tests are easier to replicate (avoiding nonreproducible defects).
- Some tests, such as performance tests, memory leak detection tests, concurrency, and so on, are nearly impossible to run manually.
- AST allows for distributed testing— spreading the workload across multiple machines, simulating the “live” or production environment. This is another very efficient way of using automation.
The impact of more effective test time on software quality can be illustrated as shown in Figure Improved Quality with increased test time (additional detail), using the software reliability model results from before. The more testing that can be completed, the greater will be the direct impact on reducing the number of software defects still undiscovered. The size of the impact is a function of the state of the software baseline relative to undiscovered defects (i.e., where you are on the curve).
Figure 2.2 Improved Quality with increased test time (additional detail)
Ultimately, the question everyone has to answer is, “How do we know when we are done testing?” Software reliability models provide a projection of how many defects are still undiscovered and can serve as a basis for answering the question of knowing when enough testing has been done. The reality is that often testing stops when the product needs to be shipped or when it was promised to a customer. The test team conducts as much testing as possible in the allotted time. AST provides a means of conducting more tests in ideally less time, but conducting even more tests in the same amount of time provides efficiencies that can have a direct impact on software quality. Software reliability models can be used as tools to estimate the impact.
Even without using software models, you can use results from past deliveries to start to analyze where problems were found after the product was delivered to see the impact AST could have on software quality. In assessing the results from previous releases or deliveries, were any problems reported regarding functionality that previously worked, but that the test team did not have time to fully regression-test? How many problems were identified as a result of sequence-of-operator actions not exercised because there was not enough time to test, or a combination of values that was never tried? Were there any problems due to boundary condition values that were not tested? How many issues were there with incompatibility with all configurations or after the application ran for a certain period of time? Were issues identified because there was not enough time to even test the functionality? Reviewing results from past deliveries can provide insight into the impact additional testing and AST would make on software quality.