- Why Automate?
- The Challenges of Testing Software Today
- Reducing the Time and Cost of Software Testing
- Impacting Software Quality
- Improvements to Your Software Test Program
- Summary
- Notes
Like this article? We recommend
Like this article? We recommend
Like this article? We recommend
The Challenges of Testing Software Today
The primary challenge today for testing software is that customers want more software functionality to be delivered faster and cheaper, while at the same time they expect the quality of the software to at least meet if not exceed their expectations.
The implications for the software test team are enormous. More functionality means each software baseline to be tested will be larger and more complex, which means more test cases must be run for each delivery. In other programs, faster deliveries mean less time to test each delivery, and more deliveries will be made each year, perhaps three or four product releases instead of one or two. More deliveries also mean more regression testing cycles. Reducing costs means the testing needs to be completed with fewer labor hours. Simply stated, there is more software to test, more often, with fewer people.
A report published by the NIST begins:
Reducing the cost of software development and improving software quality are important objectives of the U.S. software industry. However, the complexity of the underlying software needed to support the U.S.’s computerized economy is increasing at an alarming rate. The size of software products is no longer measured in terms of thousands of lines of code, but millions of lines of code. This increasing complexity along with a decreasing average market life expectancy for many software products has heightened concerns over software quality.[2]
This same report estimates that software errors cost the U.S. economy an estimated $59.5 billion annually, of which they estimate one-third could be eliminated by improved testing.
Commercial hardware and software technologies have made many significant advances throughout the past several decades in development practices, methodologies, and tools which have been enablers in fielding phenomenal new, innovative, affordable, and timely information-processing capabilities. In addition, through the use of standard interfaces and common architecture frameworks, libraries of common functionality are becoming available for reuse which will further reduce development time and cost.
With a focus on reducing development time and creating reusable capabilities, efficient and thorough testing of the software will be more critical than ever. Developer software tests will no longer be used only during initial development and then not again. With incremental software builds and highly reusable software capabilities, software tests will need to be executed and reused many times before a product is even delivered. Furthermore, complex processing systems historically have been dominated by hardware costs and, as a result, the testing methodologies that have evolved are traditionally focused around the hardware. The majority of the systems being developed today, however, are largely based on affordable commodity hardware with the principal schedule and cost drivers squarely centered on the development and testing of the software.
Systems today are largely software-based, growing in complexity, and becoming more and more dependent on the successful reuse of software developed from other programs. Despite the advances in development practices and tools, the goals of accelerating the rate at which systems can be delivered and reducing their costs cannot be met by simply writing software faster without comparable improvement in the practices and tools for testing it.
New factors impacting the cost, schedule, and manpower needed for testing programs include
- Redundant testing between development teams, test teams, Verification and Validation (V&V) teams, contractors, and even users, i.e. during user acceptance testing.
- Significantly increased levels of regression testing driven by technology insertion and new functionality
- Increasing complexity of computer software testing
- Interoperability, cross-platform, and certification testing requirements for each product release
In an article published by the Colorado State University Computer Science Department entitled “Automatic Test Software” [3] it was reported:
Much of testing (software) is still being done manually, and the process is intuitively guided. In this respect, software development lags behind hardware design and test, where use of tools is now regarded as mandatory . . . In the near future, reliability expectations in the market will particularly dictate all developers greatly rely on automation in testing. Today, no hardware engineer would think of doing a design without SPICE or VHDL level simulation. No one today thinks of doing manual test generation for hardware. The same will be true for software in only a few years..
The Colorado State University report is consistent with the findings of the IDC Software Research Group, which show that the market for automating software quality tools is growing substantially. The IDC [4]Software Research Group published a report entitled “Worldwide Distributed Automated Software Quality Tools 2005–2009 Forecast and 2004 Vendor Shares,” which begins by stating that the “automated software quality tools market was once again the growth leader across application life-cycle markets.” This report goes on to state, “The criticality of software to business, the increasing complexity of software applications and systems, and the relentless business pressures for quality, productivity, and faster time to market have all been positive drivers (resulting in growth in the market) and will continue to be in the foreseeable future.”[5]
In its most recent market research report, IDC states that it expects the market for automated software quality tools to almost double in the five years from 2005 ($947.6 million) to 2010 ($1842.1 million), citing increasing software complexity and the increased attention to corporate compliance as two of the key market drivers.
Organizations large and small are faced with increasingly large and sophisticated software projects while at the same time they are interested in delivering new and better products faster to market at the lowest possible cost. Unfortunately, most software development projects today are delivered late, are over budget, have less functionality, and have not been completely tested. When testing has become the bottleneck, AST can be a response to these challenges. How to approach AST and whether it can pay off are discussed throughout this book.