Playing the Game of Dungeons and Dragons

Process maturity: are the investments justified?

During an orientation briefing, the new vice president in charge of research and development asked why the firm was spending $2.4 million a year on software process improvement. Nobody in the room, including your champion, could provide a satisfactory answer (he hadn't seen your material, so he couldn't defend you). The productivity projections that you offered as justification were torn to shreds. The debate was hot and heavy. Most of the middle managers in the room grinned and vocally took sides against you. Finally, the vice president said:"If they (the process group) can't justify this expenditure, we should spend the money elsewhere—research?" Your boss called and asked you to provide an answer the next day. You are scheduled to brief this new vice president at noon tomorrow.

Most of the information you've seen in the literature about process improvement has harped on the benefits without translating them into numbers. While interesting, such discussions don't help you make a strong business case for software process improvement. Luckily, the consultant you hired has access to justification numbers. Most of them were generated internally as a product of your Level 3 metrics activities. He points to several papers when you query him about the other sources of numbers: [Butler, 1995], [Clark, 2000], and [McGib-bon, 1996]. He also identifies some internal benchmarking data that you can use to justify the projections that were originally used to justify your budgets. Your plan is to use this hard data to convince the new vice president that investment in process improvement pays off. However, because outsiders will be in the meeting, you must couch the numbers in such a way as to keep them secret. If you don't, the outsiders might leak proprietary information to your competition.

Accelerating Productivity Gains Through Process

You decide to use the published industry productivity benchmark of 100 source lines of code per staff-month (SLOC/SM) as a starting point to demonstrate the gain associated with process improvement. As stated in Chapter 4, productivity is defined as the ratio of outputs (SLOCs) to inputs used to generate them (SM of labor). The consultant supplies data that shows that the average productivity gain your firm experienced during the past five years was approximately 20 percent annually as it moved from Level 3 to Level 4. During the early years, the gain was just 10 percent annually. After considerable analysis, you believe that your firm was able to accelerate productivity gains by 10 percent per year (from 10 to 20 percent) by pursuing a process improvement strategy. These numbers correlate well with the initial business case that was developed to justify the original initiative. When the outsiders leave, you plan to show the new vice president a chart that confirms the trends. Of course, your firm pursued other improvements during this time span. For example, they made additional gains by increasing capital expenditures to acquire new equipment and tools.

For you, the acceleration results in a cost avoidance averaging $2 million annually over a five-year investment time span based on the analysis summarized in Table 5.3. When you put your real numbers in, the avoidance grows to $4 million annually. After digging some more, you find out that this is how the numbers were generated three years ago. You also learn that the trend lines used then to project benefits have held true throughout this time period. However, you will use the generic numbers for this computation to protect leakage of the numbers by the outsiders.

Table 5.3: Savings Attributed to Accelerating Productivity from 10 to 20 Percent Annually

It should be noted that you were very conservative in your calculations. For example, you assumed that there was no gain during the first year of the improvement strategy. As another example, you did not assume that your work-force grew as the workload did. Instead, you addressed the growth via cost avoidance in terms of SLOC that you did not have to generate. You defined productivity in terms of SLOC per staff-month because this was the metric that your firm historically captured and reported for software. You discarded newer productivity metrics based on function and applications points because they were too business system oriented [Jones, 1998]. When you asked the consultant, he stated that the SLOC/staff-month metric seemed appropriate because you could use the improvement trends to build a business case that justified your expenditures for the improvement initiative.

The $50 per SLOC assumption is also a best case.Your cost per SLOC actually ranges from $50 to $150.You took the bottom end of this range to be ultraconser-vative in your projections. If you had taken the $150 figure, middle management would not have believed your numbers because they would have seemed to be too good. For the internal revenue, you plan to use $100 per SLOC. This correlates well with industry benchmarks for productivity and represents a more realistic saving.

Early Defect Detection and Correction

While the productivity numbers look good, you want to show more benefits in your business case justification. When asked, the consultant points you to defect data that your process group has been collecting for several years. The defect introduction and removal rates have been tabulated, compared, and reported using data that you have captured throughout the software life cycle as a byproduct of the work product inspection process that you inserted as part of your Level 3 efforts. They demonstrate convincingly that the process improvements you've made lead to earlier defect removal (before release to the field).

You now want to compute the benefits early defect detection buys you. Like yours, several firms the consultant has worked with have been capturing such data to quantify these benefits from their projects. These firms have caught defects early and avoided the costly problems associated with their propagation after the systems have been fielded. They too have used work product inspections to capture the data. As these firms moved from Level 3 to Level 4, their data showed that the average number of errors that went to the field was reduced by a factor of between 20 and 25 percent. Again, this correlates well with your experience. In addition, the majority of these defects were caught in the requirements and design phases instead of during test and integration. According to your internal data, the cost of fixing a defect when it is found during these early stages saves you as much as 40 percent of your historical repair costs (saves $20 per defect based on 100 percent rework). For the 12 major programs that your firm has under development, these savings translate to $1.2 million annual cost avoidance, assuming

(12 projects/year)(10 defects/K SLOC entering test)
(500 K lines/project) = 60,000 defects
 
(60,000 defects)($20/defect cost avoidance) = $1.2 million

In addition, these firms found that their product quality was better than yours. The average defect density for their software products when released to the field was between 0.1 and 0.5 defects/K SLOC. Your defect densities are averaging between 5 and 8 errors/K SLOC. From your viewpoint, lower defect density translates into improved customer satisfaction. However, neither you nor your consultant knows how to quantify the impact of improved customer satisfaction and include it as part of a business case.

The consultant reported that being at Level 4 also had other advantages. He pointed to several firms that used statistical process-control U-chart reports to reallocate effort from processes that were under control to those that were more error-prone. You asked, "What is a U-chart?" The consultant explained that U-charts are statistical process-control tools that plot error trends between control limits. Using these charts, he showed you how to determine how well processes were behaving [Florence, 2001]. The results were extremely heartening because they showed that Level 4 firms were more concerned with controlling variability then writing waivers to get out of processes. You plan to use this information if needed to win arguments.

Exploitation of COTS

As you are tabulating your ROI using productivity improvement and earlier defect detection and correction figures, your consultant has a brainstorm. "Why not add the benefits we are deriving through COTS and product lines?" he suggests. As with many defense contractors, you have moved from custom hardware and software solutions to those offered off the shelf. Like others, you have found such solutions risky and error-prone. To counter these tendencies, you have adopted an inspection and licensing process aimed at improving your advantage with suppliers.

Based on the consultant's inputs, you can quantify the benefits associated with enterprise-wide software licensing. For example, Northrop saved over $2 million annually by adopting improved licensing processes [Reifer, 1999]. This saving was achieved at the project level by coordinating software license purchases to gain volume discounts via increased buying power. As part of its licensing initiative, Northrop was also able to negotiate license changes to transfer spare seats to projects other than those designated in the original license agreement. Finally, Northrop put a "try before you buy" practice into operation. Trial use enabled the company to identify defects that had to be corrected before purchasing the product. Defects often had side effects that biased their error data and were difficult to find and fix. Per these results, you believe that you can save at least $1 million annually through improved licensing.

Movement to Product Lines, Architecture, and Systematic Reuse

Now you are ready to quantify the gains associated with moving to product lines. You have limited experience here. Therefore, you must rely on the consultant's inputs relative to published data [Weiss, 1999]. Implementing architecture and systematic reuse is traditionally the hardest part of the strategy. The reason behind this is that the process guidelines used, such as the SW-CMM [Paulk, 1995] and SPICE [El Emam, 1997], offer little help in this area. Because your company is a defense contractor, other restrictions make it very difficult for you to share software across projects. For example, security provisions may force you to discard existing designs because their disclosure could provide insight into how to break into the system. Sharing is not something that your customers currently encourage or provide incentives to accomplish.

You ask the consultant for recommendations on how to bring product lines, architecture, and systematic reuse into your organization. Luckily, your firm has been pursuing architecture-based reuse for over a decade as part of your internal research and development efforts. The research team has completed a domain analysis and developed reference architectures, both hardware and software, to facilitate reuse at the system level [Bassett, 1996]. Their goal was to deploy this architecture using product line management concepts by making it part of the processes their engineers use to do their work [Reifer, 1997b]. Using their recommendations, you plan to incorporate reuse provisions into your processes as you develop them for Level 4.

The benefits the research team attributed to reuse are many and substantial. Systematic reuse saves money and time by making big jobs smaller. Table 5.4 illustrates this phenomenon using a sensor system software example. Using product line concepts effectively cuts the size of the job almost in half. Reuse in this context is planned, and the components themselves are designed for reuse. For example, alarms are instantiated from some existing alarm class by tailoring, not redesign or coding. Variability is controlled. The direct reuse of existing components previously developed to populate the selected product line architecture results in the savings shown in Table 5.4.

The benefits of cutting the size of the job in half can be quantified using a cost model like COCOMO II [Boehm, 2000]. Running the model for the example summarized in Table 5.4 results in the effort and duration estimates shown in Table 5.5. Both nominal and shortest development time options were estimated. The only cost driver varied was the process maturity (PMAT) factor that was set to reflect a Level 4 organization. Table 5.5 shows that you can cut the estimated duration in calendar months by about 20 percent and effort in staff-months by about one half in both cases (most likely or nominal and shortest duration estimate).

Table 5.4: Size of Application with and without Reuse

Table 5.5: Effort and Duration Estimates with and without Reuse

This example illustrates the benefits associated with systematic software reuse. It suggests that each of your projects can save as much as half its costs(about $5 million) and a year in schedule through reuse. In reality, your systems are much bigger and more complex than that shown in Table 5.4. As a result, your cost saving should exceed $5 million on each new system you generate using the product line architecture you have developed and the infrastructure you have introduced. Multiply these savings across all your product lines/families, and you estimate that you can realize savings of at least $10 million annually once the technology has been transferred into operations. However, the increased costs associated with maintaining your architecture and with designing assets for reuse need to be accounted for in your calculations. These costs are factored into the worksheet you have prepared to address the costs/benefits associated with systematic software reuse, which is shown in the accompanying box.

You believe that most of the nonrecurring investment costs as shown must be funded by projects as part of their budgets or covered by existing investments.

Your systematic software reuse initiative is just starting. It represents a major challenge because it will force you to change the way you do your business. Instead of developing products for projects, you will develop product line architectures for business units. In addition, major changes to the way you currently manage your business will be required to facilitate this changeover. Luckily, processes and tools that facilitate the change to systematic reuse exist and are starting to be used commercially [Jacobson, 1997]. But it would be unfair to assume that you will reap any benefit from this initiative during the next two years. Therefore, you have not included the incremental contribution of this reuse initiative in your ROI computation.