Software [In]security: Measuring Software Security
Like this article? We recommend
Like this article? We recommend
Like this article? We recommend
How a Large Financial Services Firm Builds Better Software at a Lower Cost
The biggest challenge software security faces is not a technical challenge, it is a business challenge — namely, how to describe the value proposition of building secure software in concrete, positive terms. Every major software security methodology, from Microsoft's SDL through the Cigital Touchpoints to OWASP CLASP, teaches the integration of security controls into the SDLC. Maturity models, including the BSIMM, extol the virtues of metrics and measurement. Yet clearly demonstrating business value of an ongoing software security initiative is a nontrivial undertaking. Doing so in the middle of a recession makes things even harder. In this short article we describe the business positioning of a successful software security initiative instituted at a large financial services firm (referred to for the remainder of the article simply as "the Firm").
Begin with the End in Mind
Barry Boehm's seminal work in SDLC measurement has taught two generations of programmers and software engineers a simple fact — finding and eradicating software defects early is cost-effective and economically sane. In fact, Barry's work shows that the cost of removing a software defect grows exponentially for each downstream phase of the development lifecycle in which it remains undiscovered. Boehm's TRW case study from the '70s is probably the most cited article on software measurement (especially by Marketing types). Figure 1 shows a typical slide created from the TRW data.
)
So why not rely on these data to justify software security controls? Any experienced business person will tell you that hackneyed old data may make for pretty PowerPoint, but (unless you are TRW) it is far too easy to object that any particular business is different in fundamental ways from TRW; different enough that defect cost measurements from TRW do not apply. Too many security controls might slow a software project down so much that the cure is worse than the disease.
To get past this classic show stopper of an argument, executives at the Firm set out to measure the impact of their own software security initiative using their own real numbers. By determining the actual costs of defect remediation at various SDLC stages, they can compute a real return on investment table. As an example of early lifecycle savings, the Firm's numbers show that fixing a defect post-production takes on average 32 hours of development time. By contrast, fixing a defect in development takes on average 3.2 hours of development time. This factor of ten adds up quickly. The resulting cost/benefit table (displayed as Figure 2) makes the value of finding and removing software security defects early crystal clear to any business person.
)
By relying on defect cost data computed from their actual code base, the Firm shows a productivity savings in year 4 of $15.6 million on a sustained investment of $200K. The resulting productivity gain is 12%.
Contrast this view of the impact of software security from a business perspective against more typical technical measurements such as defect count. Showing that a defect count is trending down (as in Figure 3) is an excellent technical result. But Figure 3 is much less useful to a business executive who wants to know how much it costs to accomplish this downward trend. Is it worth the effort? In business, the ends rarely justify the means on purely technical grounds; they must make financial sense too.
)
The Firm's software security initiative can state with confidence that according to the Figure 2 data, just over 12% of every dollar spent on software development is returned for a productivity gain and can thus be reinvested in high value activities. By spending wisely on software security, they save enough money to reinvest in activities other than costly rework.
Software Security Savings Require Investment
Business success with software security does not come easy. Integrating controls and best practices into an existing SDLC (or, more likely, a set of diverse SDLCs) involves instituting a cultural change in software development. The BSIMM model describes a number of software security activities divided into twelve practices, and includes data observed in the Firm's initiative.
Essential practices adopted by the Firm include: adopting and adapting a software security methodology (CLASP in this case), identifying and empowering business stakeholders, performing architectural risk analysis, leveraging standardized security architectures, carrying out code review with static analysis tools, and building and delivering an extensive training curriculum for developers.
By focusing as much attention on the business situation as on technical concerns, the Firm was able to achieve measurable business results in three years. One tactic involved identifying the most competent and highly respected developers and convincing their management team leaders to nominate them for a special program to teach them about security. Developers who participated and passed screening tests were then recognized as part of a newly established community of top developers (gaining them explicit and accepted responsibility to teach their peers about security).
Another tactic involved implementing a transparent and detailed reporting framework in which key performance metrics (like those shown in Figure 3) were shared with team leaders, their managers, and the CIO on a monthly basis. Over time, development leaders focused on issues that were negatively impacting their scores, and an internal competition developed. An arms race for the common good ensued, encouraged and closely monitored by the CIO.
In 2005, the Firm faced a dilemma now facing hundreds of companies. What is the best approach to securing software? By focusing on policy, process, training and automation, and by carefully measuring business impact while accounting for corporate culture, they built a mature, world-class software security initiative.