Testing Embedded Software

About

Features

• Provides a practical and comprehensive overview of how to test embedded software.
• Includes classic as well as modern test design techniques.
• Demonstrates an approach that is useful in the real-life situations of 'limited time and resources.'
• In depth treatment of test design techniques that can be directly applied.
• Description of how to establish a professional test organision - with particular emphasis on roles, tasks, responsibilities, organisational hierarchies and communication structures.
• Coverage of how to organise the test process.

Description

The book provides a practical and comprehensive overview of how to test embedded software.

• The book describes how embedded systems can be tested in a structured, controlled way.
• The first complete description of all necessary ingredients of a testing process.
• It includes classic as well as modern test design techniques.
• The described approach is useful in real-life situations of 'limited time and resources.

More and more our society is pervaded by embedded software: cars, telecom, home entertainment devices are full of software. Embedded systems are becoming larger and more complex with an increasing amount of software, leading to a growing need for a structured testing method which helps to tackle the typical problems in embedded software testing.

Audience:

Managers or team leaders that are responsible for development and/or testing of embedded software and systems.  Also, people who actually perform the primary software testing activities.

User level:

Intermediate.

Bart Broekman has been a software test practitioner since 1990.  He participated in European embedded software research projects (ITEA) and is co-author of a book on test automation.

Edwin Notenboom has been a professional tester at Sogeti for six years.  Together with Bart Broekman, he participated in a european ITEA project on embedded systems since February 1999.

Sample Content

Table of Contents

Foreword
Preface
Acknowledgments
Recommendations
PART I INTRODUCTION
1. FUNDAMENTALS
1.1 Aims Of Testing
1.2 What Is An Embedded System?
1.3 Approach To The Testing Of Embedded Systems
2. THE TEMB METHOD
2.1 Overview
2.2 Temb Generic
2.3 Mechanism For Assembling The Dedicated Test Approach
PART II LIFECYCLE
3. MULTIPLE V-MODEL
3.1 Introduction
3.2 Test Activities In The Multiple Vs
3.3 The Nested Multiple V-Model
4. MASTER TEST PLANNING
4.1 Elements Of Master Test Planning
4.2 Activities
5. TESTING BY DEVELOPERS
5.1 Introduction
5.2 Integration Approach
5.3 Lifecycle
6. TESTING BY AN INDEPENDENT TEST TEAM
6.1 Introduction
6.2 Planning And Control Phase
6.3 Preparation Phase
6.4 Specification Phase
6.5 Execution Phase
6.6 Completion Phase
PART III TECHNIQUES
7. RISK-BASED TEST STRATEGY
7.1 Introduction
7.2 Risk Assessment
7.3 Strategy In Master Test Planning
7.4 Strategy For A Test Level
7.5 Strategy Changes During The Test Process
7.6 Strategy For Maintenance Testing
8. TESTABILITY REVIEW
8.1 Introduction
8.2 Procedure
9. INSPECTIONS
9.1 Introduction
9.2 Procedure
10. SAFETY ANALYSIS
10.1 Introduction
10.2 Safety Analysis Techniques
10.3 Safety Analysis Lifecycle
11. TEST DESIGN TECHNIQUES
11.1 Overview
11.2 State Transition Testing
11.3 Control Flow Test
11.4 Elementary Comparison Test
11.5 Classification-Tree Method
11.6 Evolutionary Algorithms
11.7 Statistical Usage Testing
11.8 Rare Event Testing
11.9 Mutation Analysis
12. CHECKLISTS
12.1 Introduction
12.2 Checklists For Quality Characteristics
12.3 General Checklist For High-Level Testing
12.4 General Checklist For Low-Level Testing
12.5 Test Design Techniques Checklist
12.6 Checklists Concerning The Test Process
PART IV INFRASTRUCTURE
13. EMBEDDED SOFTWARE TEST ENVIRONMENTS
13.1 Introduction
13.2 First Stage: Simulation
13.3 Second Stage: Prototyping
13.4 Third Stage: Pre-Production
13.5 Post-Development Stage
14. TOOLS
14.1 Introduction
14.2 Categorization Of Test Tools
15. TEST AUTOMATION
15.1 Introduction
15.2 The Technique Of Test Automation
15.3 Implementing Test Automation
16. MIXED SIGNALS
16.1 Introduction
16.2 Stimuli Description Techniques
16.3 Measurement And Analysis Techniques
PART V ORGANIZATION
17. TEST ROLES
17.1 General Skills
17.2 Specific Test Roles
18. HUMAN RESOURCE MANAGEMENT
18.1 Staff
18.2 Training
18.3 Career Perspectives
19. ORGANIZATION STRUCTURE
19.1 Test Organization
19.2 Communication Structures
20. TEST CONTROL
20.1 Control Of The Test Process
20.2 Control Of The Test Infrastructure
20.3 Control Of The Test Deliverables
PART VI APPENDICES
Appendix A: Risk Classification
Appendix B: Statecharts
B.1 States
B.2 Events
B.3 Transitions
B.4 Actions And Activities
B.5 Execution Order
B.6 Nested States
Appendix C: Blueprint Of An Automated Test Suite
C.1 Test Data
C.2 Start
C.3 Planner
C.4 Reader
C.5 Translator
C.6 Test Actions
C.7 Initialization
C.8 Synchronization
C.9 Error Recovery
C.10 Reporting
C.11 Checking
C.12 Framework
C.13 Communication
Appendix D: Pseudocode Evolutionary Algorithms
D.1 Main Process
D.2 Selection
D.3 Recombination
D.4 Mutation
D.5 Insertion
Appendix E: Example Test Plane.1 Assignment
E.2 Test Basis
E.3 Test Strategy
E.4 Planning
E.5 Threats, Risks, And Measures
E.6 Infrastructure
E.7 Test Organization
E.8 Test Deliverables
E.9 Configuration Management
Glossary
References

Preface

A growing industry requires structured testing

Embedded systems have to rely on high quality hardware as well as high quality software. Therefore, both hardware testing and software testing are essential parts of the test approach for an embedded system. However, this book concentrates more on the testing of software in embedded systems. Many hardware issues are included, but technical details of testing individual hardware components are not discussed — this is a profession in itself. Usually the technical people are competent in dealing with the technical intricacies involved in the testing of hardware on a detailed level. This book is mainly targeted at those who work with the software in embedded systems. It teaches them about the environment in which they work, the specific problems of testing their software, and techniques that are not normally taught in software education.

This book aims at providing answers and solutions to the growing problem of "getting the complex testing process under control." It focuses on the higher level of how to organize the overall testing process with its broad range of activities in both software and hardware environments. The authors have used concepts and material from the book Software Testing, a Guide to The TMap® Approach, and have adapted them to fit the embedded software world.

The book is not intended to have the academic quality of a thesis. It is strongly practically oriented and aims at providing overview, insight, and lots of practical guidelines, rather than detailed academic proof.

Testing is more than just exercising the system and checking if it behaves correctly. It also involves the planning of activities, designing test cases, managing the test infrastructure, setting up an organization, dealing with politics and much more. This book describes the TEmb method for structured testing of embedded software. It covers the broad range of issues that is essential in structured testing, answering the questions "what, when, how, by what and by whom?" TEmb uses the four cornerstones of structured testing as defined by the test management approach TMap® (Pol et al., 2002): lifecycle ("what, when") of the development and testing process; techniques ("how"); infrastructure ("by what"); organization ("by whom"). The structure of this book follows those four cornerstones.

The book is divided into six parts:

Part I describes some general principles of structured testing and embedded systems. It provides an overview of the TEmb method, showing how to assemble the suitable test approach for a particular embedded system.

Part II deals with lifecycle issues and thus with the process of developing and testing embedded software. The lifecycle cornerstone is the core of the testing process, providing the map of what has to be done and in what order. Various issues from the other three cornerstones apply at various points in that lifecycle.

Part III offers several techniques that are probably useful for most embedded software test projects. It contains techniques for executing a risk-based test strategy, a testability review, formal inspections, and safety analysis. It offers various techniques for designing test cases that can be used in different circumstances for different purposes.

Part IV deals with the infrastructure that testers need to do their job properly. It describes the different test environments required at different stages in the testing process. An overview is provided of the various tools that can be usefully applied for different test activities and purposes. Tools that assist in achieving automated test execution have always been very popular. This part explains the technical and organizational issues involved in this kind of test automation. Finally, specific issues are discussed that arise when the tester works in an environment dealing with analog as well as digital signals (so-called "mixed signals").

Part V describes various organizational aspects of testing. It is about those who have to perform the testing activities and who have to communicate with other personnel. It contains descriptions of the various test roles as well as the management and organizational structures. It also deals with how the testers should report on the progress of testing and on the quality of the system under test.

Part VI contains various appendices with background information on subjects such as risk classification, statechart models, a blueprint of an automated test suite and an example test plan.

Edwin Notenboom1

Foreword

The importance of software is increasing dramatically in nearly all sectors of industry. In the area of business administration, software has become an indispensable core technology. Furthermore, more and more products and innovations are also based on software in most technical fields. A typical example is provided by the automotive industry in which a rapidly increasing number of innovations are based on electronics and software to enhance the safety of the vehicles, and also to improve the comfort of the passengers and to reduce fuel consumption and emissions. In modern upper-class and luxury cars, 20 to 25 percent of the cost is on electronics and software, and this proportion is estimated to increase to up to 40 percent in the next ten years.

Software has a substantial influence on the quality of products as well as the productivity of a company. Practice, unfortunately, shows that it is impossible to develop a complex software-based system "first-time-right". Hence, comprehensive analytical measures have to be taken to check the results of the different development phases and to detect errors as early as possible. Testing constitutes the most important analysis technique, besides reviews and inspections. It is, however, a very sophisticated and time consuming task, particularly in the field of embedded systems. When testing embedded software, not only the software has to be considered but also the close connection to the hardware components, the frequently severe timing constraints and real-time requirements, and other performance-related aspects.

This book should be regarded as an important and substantial contribution to the significant improvement of the situation in the field of testing embedded systems. It provides a comprehensive description of the world of embedded software testing. It covers important aspects such as the testing lifecycle and testing techniques, as well as infrastructure and organization. The authors’ concentration on usability in industrial practice makes the book a valuable guide for any practitioner. Due to the wide range of application of embedded software, the book will be useful in many industrial businesses.

With its comprehensiveness and practical orientation this book provides a significant milestone on the long road to the effective and efficient testing of embedded software, and thus to the economic development of high-quality embedded systems. Several concepts from the book have already established a foothold in the DaimlerChrysler Group and I hope that it will find its way to many testers and software developers in various different industrial sectors in which the development of dependable embedded systems is part of the core business.

Dr. Klaus Grimm, Berlin, June 2002

Director of Software Technology Research at DaimlerChrysler AG

Updates

Submit Errata