Software Verification and Validation in High Assurance Agile Development: Ground Rules
This article originally appeared on Dean Leffingwell's blog, Scaling Software Agility.
Like this article? We recommend
Like this article? We recommend
Like this article? We recommend
Series Background
In this series of posts, I’ve been using Medical Device Development (as Regulated by U.S. FDA via CFR 820.30 and international standard IEC62304) as an exemplar for suggesting ways to develop high quality software in regulated (and other high assurance, high economic cost of failure) environments in an agile manner. This series is sponsored, in part, by Rally Software Development.
In an earlier post, I noted that there are set of immutable rules (driven by regulations, interpretations of regulations or established auditing practices) associated with software development, and thereby agile software development, for high assurance systems. Within our medical device examplar, for example, we noted that the folks at SoftwareCPR called out the set of things that teams must address in order to be in compliance with USA FDA CFR 820.30, and IEC62304. Not surprisingly, it’s a pretty long list as this graphic shows.
Many of these artifacts include the word “plan” so its clear that plans for activities such as verification and validation, problem resolution, risk management etc, must be developed, documented, and most importantly, followed. Since these plans form much of the body the enterprises “Quality Management System” (QMS-the body of internal governing documents and procedures that define how quality is achieved and regulations are met), many of these will have to be modified as the company adopts agile methods.
We’ll return to that activity later, but initially, the primary objective of this series to describe a suggested high assurance agile software engineering process that covers the definition , development and verification of the code itself. Without that, we won’t have accomplished much, and once we’ve done that, the necessary changes to a QMS will be more obvious.
In order to do THAT, we’ll need to address a subset of this list and we’ll need some specific artifacts and activities (our “ground rules”) to follow. We’ll highlight those in the balance of this post.
Software Requirements Specification
As “the software validation process cannot be completed without an established software requirements specification” (General Principles of Software Validation, FDA CDRH, this is the seminal document that will control most of our work. It must be correct, accurate, complete, consistent, version controlled and signed. This document must cover (paraphrasing from the above):
- all software system inputs, outputs, and functions.
- All performance requirements, including throughput, reliability, accuracy, response times (i.e., all nonfunctional requirements)
- Definition of external and user interfaces
- User interactions
- Operating environments (platforms, operating systems, other applications)
- All ranges limits, defaults and specific values the software will accept
- All safety related requirements specifications, features, or functions implemented in software
However, as we’ve noted before, the fact that we need such a document doesn’t mandate that we do it all Big and Up-Front. Indeed, to be agile, we can and will develop it incrementally. But prior to any release to users, we’ll have spiff it up, put a bow on it and make sure that, in aggregate, it absolutely correctly reflects the current state of the software system behavior as of that time. That’s real work, but it doesn’t mandate that we do it only once, and up-front.
Product Requirements Document/System Requirements Specification
While the centerpiece of our world is the SRS, it doesn’t stand alone. Indeed, for most systems, including systems of systems and devices which contain both hardware and software, the governing document which resides above the SRS is usually a Product Requirements Document (we’ll use the generic acronym PRD) or System Requirements Specification, which contains the complete specifications for the device as a whole.
The development and maintenance of this document is just as critical as the SRS, however its creation and content is outside the scope of our work here. Since it’s a fairly traditional document, one can refer to a variety of texts for descriptions of this document, including some of my own earlier works, (Managing Software Requirements: First (1999) and Second Editions (2003)). In a fashion similar to the SRS, it will be developed incrementally, and the system will be validated against these governing requirements when necessary.
Verification (with traceability) Processes
Verification is used to illustrate how each phase of the process, (PRS to SRS, SRS to code) etc, meets the requirements imposed by the prior phase. (see definitions post). As we develop the code, we’ll have to both 1) make sure that it works absolutely correctly, and 2) be able to prove that it does via documentation and audit trail. We’ll use agile methods to do it in real time, and just-in-time, but we’ll still have verify the code works as intended and be able to demonstrate traceability from:
- Product Requirements Document to Software Requirements Specification
- Software Requirements Specifications (we’ll use small, agile “user stories” as objects which express necessary system behaviors) to “story acceptance tests”
- Software Requirements Specification to code (via SCM change item)
- Software Requirements Specification to code unit tests (via SCM change item)
We’ll also need to be able to prove that we’ve done all this via change control mechanisms, tracking and traceability matrices, and regression test automation. Thankfully, we’ll be able to automate much of this with the new breed of agile tooling we can apply in support of agile requirements management, coding practices, unit testing, acceptance test automation, and SCM and build management, but we’ll have to do it, automated or not.
Validation Processes
The artifacts we described above and the code that delivers the end user value will evolve incrementally, one iteration at a time. However, before any software is released for use by the end user – be it alpha, beta or production code – the results of all this work will have to be validated. “Validation shall be performed under defined operating conditions … testing under actual or simulated usage scenarios…to ensure that the devices conform to define user needs and intended uses” (CFR 820.30). Validation is the final “end run” on quality, where we evaluate the system increment against its product (system) level functional and nonfunctional requirements to make sure it does what we intended.
From the standpoint of software, we’ll need to include at least the following activities:
- Aggregate increments of product requirements into a version controlled and definitive statement
- Aggregate increments of software requirements (user stories) into an SRS Document (traditional document, repository, data base, etc).
- Finalize traceability from PRD to SRS
- Run all unit testing and story level acceptance testing regression tests to assure that they all still pass
- Finalize and execute PRD (system) level feature acceptance tests
- Run all system qualities tests for nonfunctional requirements (reliability, accuracy, security)
- Run any exploratory, usability, and user acceptance tests
- Finalize and update traceability matrices to reflect current state
- Finalize/update risk analysis/hazard mitigation
- Conduct a design review
Design Review
Lastly, while it’s a little outside the scope of the agile development process per se, we’ll need to conduct a design review for each increment (“each manufacturer shall establish and maintain procedures to ensure that firm design reviews of the design results are planned and conducted at appropriate stages of the …. Development” (CFR820.30)), to make sure that the work is complete, the system has the necessary quality, and that we have followed the practices, agile and otherwise, that we have defined in our quality system.
Moving Forward with the Agile High Assurance Software Engineering Practices
Finally, with all this context behind us, we can move forward to full discussion of the model we posted earlier. For completeness of this post, here is the graphic of the iterative, incremental, agile model we have suggested:
We’ll start an in depth discussion of the high assurance agile software engineering practices we are suggesting in the next post in this series.