Automated Software Testing: Introduction, Management, and Performance: Introduction, Management, and Performance

Description

• Copyright 1999
• Dimensions: 7-3/8" x 9-1/4"
• Pages: 608
• Edition: 1st

• Book
• ISBN-10: 0-201-43287-0
• ISBN-13: 978-0-201-43287-9

With the urgent demand for rapid turnaround on new software releases--without compromising quality--the testing element of software development must keep pace, requiring a major shift from slow, labor-intensive testing methods to a faster and more thorough automated testing approach.

Automated Software Testing is a comprehensive, step-by-step guide to the most effective tools, techniques, and methods for automated testing. Using numerous case studies of successful industry implementations, this book presents everything you need to know to successfully incorporate automated testing into the development process.

In particular, this book focuses on the Automated Test Life Cycle Methodology (ATLM), a structured process for designing and executing testing that parallels the Rapid Application Development methodology commonly used today. Automated Software Testing is designed to lead you through each step of this structured program, from the initial decision to implement automated software testing through test planning, execution, and reporting. Included are test automation and test management guidance for:

• Acquiring management support
• Test tool evaluation and selection
• The automated testing introduction process
• Test effort and test team sizing
• Test team composition, recruiting, and management
• Test planning and preparation
• Test procedure development guidelines
• Automation reuse analysis and reuse library
• Best practices for test automation

0201432870B04062001



Extras

Related Articles

Introducing Automated Testing to a Project

Lessons in Test Automation: A Manager's Guide to Avoiding Pitfalls When Automating Testing

The Automated Testing Lifecycle Methodology (ATLM)



Sample Content

Table of Contents

Preface.

I. WHAT IS AUTOMATED TESTING?

1. The Birth and Evolution of Automated Testing.

Automated Testing.

Background on Software Testing.

The Automated Test Life-Cycle Methodology (ATLM).

Decision to Automate Test.

Test Tool Acquisition.

Automated Testing Introduction Phase.

Test Planning, Design, and Development.

Execution and Management of Tests.

Test Program Review and Assessment.

ATLM’s Role in the Software Testing Universe.

ATLM Relationship to System Development Life Cycle.

Test Maturity Model (TMM)—Augmented by Automated SoftwareTesting Maturity.

Test Automation Development.

Test Effort.

Software Testing Careers.

2. Decision to Automate Test.

Overcoming False Expectations for Automated Testing.

Automatic Test Plan Generation.

Test Tool Fits All.

Immediate Test Effort Reduction.

Immediate Schedule Reduction.

Tool Ease of Use.

Universal Application of Test Automation.

One Hundred Percent Test Coverage.

Benefits of Automated Testing.

Production of a Reliable System.

Improvement of the Quality of the Test Effort.

Reduction of Test Effort and Minimization of Schedule.

Acquiring Management Support.

Test Tool Proposal.

3. Automated Test Tool Selection and Evaluation.

Organization’s Systems Engineering Environment.

Third-Party Input from Management, Staff, and Customers andUsers.

Tool Criteria Reflecting the Systems Engineering Environment.

Level of Software Quality.

Help Desk Problem Reports.

Budget Constraints.

Types of Tests.

Long-Term Investment Considerations.

Test Tool Process.

Avoiding Shortcuts.

Tools That Support the Testing Life Cycle.

Business Analysis Phase Tools.

Requirements Definition Phase Tools.

Tools for the Analysis and Design Phase.

Programming Phase Tools.

Metrics Tools.

Other Testing Life-Cycle Support Tools.

Testing Phase Tools.

Test Tool Research.

Improvement Opportunities.

Evaluation Domain Definition.

Hands-On Tool Evaluation.

Evaluation Report.

License Agreement.

II. INTRODUCTION OF AUTOMATED TESTING TO A PROJECT.

4. Automated Testing Introduction Process.

Test Process Analysis.

Process Review.

Goals and Objectives of Testing.

Case Study: Test Objectives and Strategies.

Test Strategies.

Test Tool Consideration.

Review of Project-Specific System Requirements.

Application-Under-Test Overview.

Review of Project Schedule.

Test Tool Compatibility Check.

Demonstration of the Tool to the Project Team.

Test Tool Support Profile.

Review of Training Requirements

5. Test Team Management.

Organizational Structure of a Test Team.

Stovepipe Test Team.

Centralized Test Team.

IV&V Test Team.

Systems Methodology and Test Team.

Test Team Summary.

Test Program Tasks.

Test Effort Sizing.

Test Team Sizing Methods: An Overview.

Development Ratio Method.

Percentage Method.

Test Procedure Method.

Task Planning Method.

Test Effort Sizing Factors.

Test Engineer Recruiting.

Test Engineer Qualities.

Test Team Composition.

Job Requisition.

Recruiting Activities.

Locating Test Engineers.

Test Engineer Interviews.

Distinguishing the Best Candidate.

Roles and Responsibilities.

III. TEST PLANNING AND PREPARATION.

6. Test Planning: Smart Application of Testing.

Test Planning Activities.

Test Program Scope.

System Description.

Critical/High-Risk Functions.

Test Goals, Objectives, and Strategies.

Test Tools.

Test Program Parameters.

Verification Methods.

Test Requirements Definition.

Test Requirements Management.

Requirements Management Tools.

Assessing the Test Requirements Risk.

Prioritization of Tests.

Requirements Traceability Matrix.

Test Program Events, Activities, and Documentation.

Events.

Activities.

Documentation.

The Test Environment.

Test Environment Preparations.

Test Environment Integration and Setup.

The Test Plan.

Test Completion/Acceptance Criteria.

Sample Test Plan.

7. Test Analysis and Design.

Test Requirements Analysis.

Development-Level Test Analysis (Structural Approach).

System-Level Test Analysis (Behavioral Approach).

Test Program Design.

Test Program Design Models.

White-Box Techniques (Development-Level Tests).

Black-Box Techniques (System-Level Tests).

Test Design Documentation.

Test Procedure Design.

Test Procedure Definition.

Automated Versus Manual Test Analysis.

Automated Test Design Standards.

Case Study: Naming Conventions.

Manual Test Design Guidelines.

Detailed Test Design.

Test Data Requirements.

8. Test Development.

Test Development Architecture.

Technical Environment.

Environment Readiness Checks.

Automation Reuse Analysis.

Test Procedure Development/Execution Schedule.

Modularity-Relationship Analysis.

Explanation of the Sample Modularity- Relationship Matrix.

Calibration of the Test Tool.

Compatibility Work-Around Solutions.

Case Study: Incompatibility Work-Around Solution.

Manual Execution of Test Procedures.

Test Procedure Inspections—Peer Reviews.

Test Procedure Configuration Management.

Test Development Guidelines.

Design-to-Development Transiftion.

Reusable Test Procedures.

Case Study: Navigation Using Tabs or Mouse Clicks.

Case Study: Testing Bitmaps Using a Capture/Playback Tool.

Maintainable Test Procedures.

Case Study: Automating Documentation.

Case Study: Automated Random Testing.

Other Guidelines.

Automation Infrastructure.

Table-Driven Test Automation.

PC Environment Automated Setup Script.

Automated Recording Options.

Login Function.

Exit Function.

Navigation.

Verifying GUI Standards.

Smoke Test.

Case Study: Smoke Test Application.

Error-Logging Routine.

Help Function Verification Script.

Timed Message Boxes Function.

Advanced Math Functions.

IV. TEST EXECUTION AND REVIEW.

9. Test Execution.

Executing/Evaluating Test Phases.

Unit Test Execution and Evaluation.

Integration Test Execution and Evaluation.

System Test Execution and Evaluation.

Test Results Analysis of Regression Tests.

User Acceptance Test Execution and Evaluation.

Defect Tracking and New Build Process.

Defect Life-Cycle Model.

Test Program Status Tracking.

Earned Value Management System.

Case Study: System Test Status Tracking.

Test Metrics Collection and Analysis.

10. Test Program Review and Assessment.

Test Program Lessons Learned—Corrective Actions andImprovement Activity.

Test Program Return on Investment.

Case Study: TestProgram Return on Investment.

Case Study: Quantify Tool Return on Investment.

V. APPENDIXES.

A. How to Test Requirements.

Requirements Testing Approach.

Abstract.

The Quality Gateway.

Make the Requirement Measurable.

Quantifiable Requirements.

Nonquantifiable Requirements.

Keeping Track.

Coherency and Consistency.

Completeness.

Relevance.

Requirement or Solution?

Stakeholder Value.

Traceability.

Order in a Disorderly World.

Conclusions.

References.

B. Tools That Support the Automated Testing Life Cycle.

Introduction.

Business Analysis Phase.

Business Modeling Tools.

Configuration Management Tools.

Defect Tracking Tools.

Technical Review Management.

Documentation Generators.

Requirements Definition Phase.

Requirements Management Tools.

Requirements Verifiers.

Use Case Generators.

Analysis and Design Phase.

Visual Modeling Tools.

Structure Charts, Flowcharts, and Sequence Diagrams.

Test Procedure Generators.

Programming Phase.

Syntax Checkers/Debuggers.

Memory Leak and Runtime Error Detection Tools.

Code Checkers.

Static and Dynamic Analyzers.

Unit Testing Tools.

Metrics Tools.

Code (Test) Coverage Analyzers and Code Instrumentors.

Usability Measurement Tools.

Testing Support Tools.

Test Data Generators.

File Comparison Tools.

Simulation Tools.

Testing Phase.

Test Management Tools.

Network Testing Tools.

GUI Application Testing Tools.

Load/Performance Testing Tools.

Web Testing Tools.

Year 24710 Testing Tools.

Other Test Tool Vendors.

C. Test Engineer Development.

Technical Skills Stage.

Test Process Stage.

Team Effort Stage.

Technical Stewardship Stage.

Test/Project Management Stage.

Business/Product Management Stage.

D. Sample Test Plan.

Introduction.

Purpose.

Background.

System Overview.

Applicable Documents.

Master Schedule.

Roles and Responsibilities.

Project Organization.

Project Roles and Responsibilities.

Test Task Structure.

Test Team Resources.

Test Program.

Scope.

Test Approach.

Test Strategies.

Automated Tools.

Qualification Methods.

Test Requirements.

Test Design.

Test Development.

Test Environment.

Test Environment Configuration.

Test Data.

Test Execution.

Test Program Reporting.

Test Program Metrics.

Defect Tracking.

Configuration Management.

Detailed Test Schedule.

Appendixes.
D.A Test Procedure Development Guidelines.
D.B Test Verification Summary and Matrix.
D.C Test Procedures and Test Scripts
E. Best Practices.

Documented Process.

Managing Expectations.

Pilot Project.

Test Tool Compatibility Checks.

Test Tool Upgrades.

Baselined System Setup and Configuration.

Software Installations in the Test Environment Baseline.

Overall Test Program Objectives.

Keep Automation Simple.

Test Procedure Design and Development Standards.

Automated Versus Manual Test Analysis.

Reuse Analysis.

Test Team Communication with Other Teams.

Schedule Compatibility.

Customer Involvement.

Defect Documentation and Reporting.

Automated Test Advocates and Experts.

Test Team Assignments.

User Group Participation.

Test Tool Improvement Suggestions.

Become a Beta Testing Site.

Specialty Topic Experts. 0201432870T04062001

Preface

Preface

Automated Software Testing addresses the challenge for today’s software professionals who are faced with real schedule deadlines and need to introduce, manage, and perform automated testing on a project. The book addresses automated testing within a client-server or Web environment.

The focus of this book is the pragmatic concerns and information needed by the software test engineer/manager who faces the necessity of performing testing more thoroughly and quickly. By the same token, these same concerns may apply to the software developer who has the responsibility for development testing (that is, unit and integration testing) and, on some projects, system testing. The book also represents an informative guide that bolsters the ability of the quality assurance engineer to perform quality reviews on test design, test procedures, and the results of test activities.

The software project manager, who is responsible for the overall development effort, may also find this book useful. The text provides the project manager with guidelines concerning the goals and objectives of the testing effort and the decision about whether to automate testing. It also offers guidance on introducing automated testing on a project and outlines the processes for performing test planning, design, development, execution, and evaluation.

The authors have worked intimately with a number of automated testing professionals around the world, who were generous enough to share their problems and concerns. One primary concern voiced by these test engineers related to the fact that the test industry does not have the kind of structured methodologies that developers have traditionally enjoyed. Similarly, project managers, test managers, and test engineers may not be familiar with the kinds of approaches that are required to perform automated testing as opposed to the traditional test approach.

Clearly, the emphasis on automated testing represents a paradigm change for the software industry. This change does not simply involve the application of tools and the performance of test automation. Rather, it is pervasive across the entire test life cycle and the system development life cycle. The approach taken by project managers, test managers, software engineers, and test engineers is altered as a result. For software professionals to successfully make the leap to automated testing, structured approaches to testing must be embraced.

Automated Software Testing is revolutionary in that it promulgates a new structured, building-block approach to the entire test life cycle, while also providing relevant test automation and associated test management guidance needed by industry test professionals.


Automated Testing
Software project managers and software developers building today’s applications face the challenge of doing so within an ever-shrinking schedule and with minimal resources. As part of their attempt to do more with less, organizations want to test software adequately, but as quickly and thoroughly as possible. To accomplish this goal, organizations are turning to automated testing.

Faced with this reality and realizing that many tests cannot be executed manually, such as simulating 1,000 virtual users for volume testing, software professionals are introducing automated testing to their projects. While needing to introduce automated testing, software professionals may not know what’s involved in introducing an automated test tool to a software project, and they may be unfamiliar with the breadth of application that automated test tools have today. Automated Software Testing provides guidance in these areas.

The growth of automated test capability has stemmed in large part from the growing popularity of rapid application development (RAD), a software development methodology that focuses on minimizing the development schedule while providing frequent, incremental software builds. The objective of RAD is to engage the user early and throughout design and development of each build so as to refine the software, thereby ensuring that it more closely reflects the needs and preferences of the user. In this environment of continual changes and additions to the software through each software build, where requirements are encouraged to evolve, software testing takes on an iterative nature itself. Each new build is accompanied by a considerable number of new tests as well as rework to existing test scripts, just as there is rework on previously released software modules. Given the continual changes and additions to software applications, automated software testing becomes an important control mechanism to ensure accuracy and stability of the software through each build.

As noted above, a primary goal of RAD is to shorten the overall development schedule, by addressing the riskiest aspects of development in early builds. As a result, test activities are undertaken at the start of the initial RAD cycle and through each subsequent RAD cycle as well. As noted in Part III of this book, test design and development represent a complex undertaking. A test effort, in fact, may be as time-consuming as the effort required to develop the software application. When the project involves the integration of commercial off-the-shelf (COTS) products, for example, the test effort may even require more resources than software development. Instances where the test team does not participate in software specification or when test is not initiated soon enough pose a risk to the project. In these situations, potential outcomes include an incomplete software test effort, an insufficient test schedule, and an unplanned extension to the development schedule to accommodate testing.

Much of the test effort required on a project now needs to be supported by automated test tools. Manual testing is labor-intensive and error-prone, and it does not support the same kind of quality checks that are possible with an automated test tool. The introduction of an automated test tool can replace mundane manual test activities with a more efficient and repeatable automated test environment, which itself improves test engineer morale and retention.

Although some automated test tools began as capture and playback tools, the functionality and capabilities of automated test tool suites have been expanding. Automated test capabilities for software products include testing of the graphical user interface, requirements compliance, load performance, code coverage, Web interface, network communications, memory leakage, and more. New capabilities continue to be added to keep pace with the growing demand for test support.


ATLM
This book concentrates on the concerns of the software test professional within the framework of an Automated Test Life-cycle Methodology (ATLM). ATLM is a structured methodology geared toward ensuring successful implementation of automated testing. The ATLM approach mirrors the benefits of modern, rapid application development efforts, where such efforts engage the user early in the development cycle. The end user of the software product is actively involved throughout analysis, design, development, and test of each software build, which is augmented in an incremental fashion.

The ATLM incorporates a multistage process consisting of six components. It supports the detailed and interrelated activities that are required to decide whether to acquire an automated testing tool. The methodology takes into account the process of introducing and optimizing an automated test tool and addresses test planning, analysis, design, development, execution, and management. The scope of the test program is outlined within the test plan, as a top-level description of test approach and implementation. The scope is further refined through the definition of test goals, objectives and strategies, and test requirements. Similar to software application development, test requirements are specified before test design is constructed. Likewise, the test program must be mapped out and consciously designed to ensure that the most efficient and effective tests for the target application are performed. Test design is developed through graphical portrayals of the test effort, so as to give project and test personnel a mental framework on the boundary and scope of the test program.


Test Training
The evolution of automated testing has given birth to new career opportunities for software engineers. In fact, while the demand for automated software test professionals has exploded, community colleges and universities have not produced a reciprocal response to help support industry demand.

Some universities that are typically more proactive in responding to changes in the software industry have already implemented software test and quality assurance courses. For example, George Mason University (GMU) provides software test and quality assurance courses (see the GMU Web site at http://www.isse.gmu.edu/ for more information). Kansas State University (KSU) offers students several courses on the subject of software test and quality assurance and other courses that touch on the subject (see the KSU Web site at http://www.ksu.edu/).

Purdue University offers two undergraduate courses in software engineering that cover software testing and reliability. Purdue also has established a software engineering research center (http://serc.vuoregon.edu/serc/), in which faculty from eight universities and representatives from eleven companies participate. Among other areas this center supports research projects. For more information, see the Purdue University Web site at http://www.purdue.edu/.

The North Seattle Community College has established one of the most progressive testing curricula in the country. The college offers three levels of software testing courses (introduction, automation, and leadership) and one- and two-year software testing programs. For more information about these courses and programs, visit the their Web site at http://nsccux.sccd.ctc.edu/. Information concerning additional university and industry training resources is available on the author’s Web site at http://www.autotestco.com/.

The Automated Software Testing textbook is intended to help in classroom instruction on software testing that uses modern automated test tool capabilities. The book provides students with an introduction to the application and importance of software test, and it describes the different kinds of automated tests that can be performed. Instruction continues with the definition of the test approach, the roles and responsibilities of the test team, test planning, test design, test script development, test execution, defect tracking, and test progress reporting.


About the Authors
Automated Software Testing was developed by three software industry professionals.

Elfriede Dustin has performed as a computer systems analyst/programmer developing software applications and utilities, process and data modeling using CASE tools, and system design simulation models. She has experience supporting a variety of system application development projects, including health, financial, logistic, and enterprise information management systems. In addition, Elfriede has been responsible for implementing the entire development life cycle, from requirement analysis, to design, to development, to automated software testing. She has been a test manager and a lead consultant guiding the implementation of automated testing on many projects. Because of her automated test expertise, she has been sought out to help modify the capabilities of commercial test tool products, where her use and feedback on test products has proved invaluable.

Jeff Rashka has managed a multitude of significant information system and systems integration projects. System applications on which he has served as manager include worldwide transportation asset management, enterprise information management, financial management, bar-coded inventory management, and shipboard information systems. Jeff also has process improvement management experience in implementing the guidelines contained within the Software Engineering Institute’s Capability Maturity Model (CMM).

John Paul has worked as a senior programmer/analyst on financial and budgeting systems as well as a host of other information systems. His software development leadership responsibilities have included system analysis and design, application prototyping, and application development using a number of different methodologies and programming techniques. His software development responsibilities have included application testing using automated test tools. John has also assumed a lead role in the performance of year 2000 compliance testing.

The authors have applied their collective knowledge of software engineering, automated testing, and management to develop a book that addresses the pragmatic concerns and information needed by the software test engineer and manager. Automated Software Testing is designed to be a useful—and practical—guide for software engineers and software project managers who are responsible for software test activities.


Book Style and Organization
This book’s organization correlates with the phases, tasks, and steps of the ATLM. The sequence of the book is fashioned in a purposeful way. It addresses the reader as if he or she had just been handed a note giving that individual the responsibility for automated software testing on a project. A first question might be, "What exactly is automated testing, and why do I need it?" Part I of the book answers this question and provides other fundamental instruction allowing the would-be test to approach the new responsibility with confidence. The reader is then guided through the decision to automate testing as well as automated test tool selection and evaluation.

After receiving this fundamental instruction, the test engineer would have several more questions: "What is involved in setting up the tool?" "How do I get the test team in place?" "What early test planning is required?" Part II answers these questions. Specifically, the process for introducing an automated test tool is outlined and guidelines for structuring the test team are provided.

Part III focuses on automated test planning, analysis, design, and test automation (programming). This section of the book addresses test design techniques, which are comparable to the structured software design techniques introduced over the past 20 years. Specifically, it highlights the discipline required in the design of test automation. The goal is to give useful information on test design and test case development, so that the test engineer doesn’t have to discover (by trial and error) how to put together a good test design and set of test procedures.

Part IV helps the reader address several additional questions: "What’s involved in the performance of test?" "How do I manage my test schedule?" "How do I document and track defects?" This section provides guidelines pertaining to test execution, defect tracking, and test program status tracking. A set of best practices for the development and execution of automated test procedures is provided to assist test professionals in executing test activities in an efficient manner.

Overall, the book seeks to allay the concerns of the software test professional within the framework of the ATLM. ATLM is a structured methodology, and one that is geared toward ensuring successful implementation of automated testing. Readers with questions and comments may contact the authors via their home page at http://www.autotestco.com/. This Web site also provides more information on automated software testing and resources that are available to support automated software testing programs.


0201432870P04062001



Updates

Errata

It has come to our attention that the file ATLM.pdf on the CD-ROM that accompanies Automated Software Testing does not open correctly. A corrected PDF file for the ATLM graphic can be downloaded by clicking here [http://www.awl.com/cseng/titles/0-201-43287-0/ATLM.pdf]. Please note that the ATLM graphic is also available on the CD-ROM in three other formats: tiff, jpeg, eps. We apologize to our customers for any inconvenience this error may have caused.

Submit Errata

