Economics of Software Quality: An Interview with Capers Jones, Part 1 of 2 (Podcast Transcript)
Like this article? We recommend
Like this article? We recommend
Like this article? We recommend
Welcome to OnSoftware, conversations with the industry's leading experts on a wide range of programming and development topics. You can access related resources, recommended reading, special offers, and more when you visit the OnSoftware resource center at InformIT.com/softwarecenter.
Rex Black: Hello, and welcome to a special podcast. I'm Rex Black, president of RBCS, a worldwide testing and quality assurance firm, providing training, consulting, and outsourcing services to clients ranging from small startups to Fortune 20 global enterprises. I'm also the author of eight books on the topic of software testing and quality, which is why I'm really pleased today to have the opportunity to talk to another author, Capers Jones, about his latest book.
If you don't know him, Capers Jones is currently the president of Capers Jones & Associates LLC. He is also the founder and former chairman of Software Productivity Research LLC (SPR). He holds the title of Chief Scientist Emeritus at SPR. Capers Jones founded SPR in 1984. Before founding SPR, Capers was Assistant Director of Programming Technology for the ITT Corporation, a programming technology center in Stratford, Connecticut. He was also a manager and software researcher at IBM in California, where he designed IBM's first software cost-estimating tool, in 1973.
Capers Jones is a well-known author and international public speaker. Some of his books have been translated into five languages. His two most recent books are Software Engineering Best Practices: Lessons from Successful Projects in the Top Companies [McGraw Hill, 2009] and The Economics of Software Quality [Addison-Wesley Professional, 2011]. Among his other book titles are Patterns of Software Systems Failure and Success [International Thomson Computer Press, 1995], Applied Software Management: Global Analysis of Productivity and Quality [McGraw Hill, 2008], Software Quality: Analysis and Guidelines for Success [International Thomson Computer Press, 2000], Estimating Software Costs: Bringing Realism to Estimating [McGraw Hill, 2007], and Software Assessments, Benchmarks, and Best Practices [Addison-Wesley Professional, 2000].
Today, Capers and I will talk about The Economics of Software Quality. In that book, Capers Jones and Olivier Bonsignour show how to systematically measure the economic impact of quality and how to use this information to deliver far more business value. For more information, and to purchase, visit InformIT.com/ESQ.
So, Capers, let's talk about your new book. Can you give our listeners a brief description of the book and its main themes?
Capers Jones: Well, thank you, Rex. The book is about the economics of software quality. And the reason that I had to write such a book, together with Olivier, was because many companies don't actually measure quality, and when they do try to measure it, they end up using metrics such as "cost per defect" or "lines of code," that actually distort the economic value of quality and make it difficult to see where the real advantages are. For example, "cost per defect" tends to penalize high quality and achieve its lowest values where the number of bugs is greatest. And "lines of code" metrics of course can't be used to measure requirements and design defects, which outnumber coding defects, and they also tend to penalize high-level languages. So the bottom line is that, for a long time, the industry has suffered under a misapprehension that high quality comes with high cost. But when you measure properly—when you use, for example, function-point metrics for normalizing the data—you discover that high quality ends up with lower costs, shorter schedules, and significant reductions in maintenance costs, all at the same time. But in order to see these economic advantages, you have to know how to measure quality and get the numbers right.
RB: Great. I really enjoyed reviewing the book. There are so many excellent points raised, such as the ones that you just mentioned, and they're all demonstrated with data. We see a lot of our clients, both large and small, making some really critical decisions about how to manage their software process and software quality without relying on data. So, do you have any thoughts on why we, as software engineers, don't rely more on data in our management decision-making?
CJ: Well, I wish the software engineering field did rely more on data, but unfortunately, the level of sophistication in software engineering is roughly equivalent to the level of sophistication of the medical profession in the days before sterile surgical processes were introduced—back in the days before surgeons sterilized their instruments prior to operating. What I mean by that is that a lot of companies do not pay attention to bugs before testing starts, which means that requirements and design bugs are still present, and end up in the software, and they're almost invisible to some kinds of testing. It means that they don't do pretest static analysis or pretest inspections. They often don't use trained and certified testing personnel, so that their test-case designs are somewhat amateurish. They try to let kind of casually trained developers do most of the testing, and the bottom line is that they end up delivering far too many bugs to clients, they spend far too much time during the test cycle because there are too many bugs when testing starts, and they have a very unbalanced combination of high cost and poor quality primarily because they don't measure well enough to know what the real advantages are.
RB: Yeah, we saw an example of that with a client not too long ago, where, based on the data that they had (which was kind of sketchy, but it was reliable enough), we were able to estimate that they had excess defect costs somewhere in the neighborhood of anywhere from $100 million dollars a year to $250 million dollars a year on a billion-dollar annual IT budget. And it's for exactly the reasons that you just outlined—letting this huge tsunami of bugs flood into the testing process and overwhelm it, and then be very difficult for them to recover once they got into that situation.
CJ: Yes, most projects that run late seem to be on time until testing starts. Then, because of the unexpected deluge of bugs, the testing schedule stretches out to two or three times what was anticipated, sometimes two or three shifts every day, and at that point it's too late to make an effective recovery because the bugs are already there. You have to keep them out before testing begins, in order to have a cost-effective development cycle.
RB: Yeah, what's really stunning sometimes in those situations is that companies, even when you point out that reality to them, refuse to accept it. We've had some clients where we explained, "Look, this is what's happening. You're not getting the bugs out soon enough, and this is why your testing processes are blowing up." And they say, "Well, yes, we understand that, but we don't have time to do design reviews and requirements reviews." [Chuckles.] Kind of difficult to explain, "No, the reason that you don't have time is that you don't have any time not to do those things," and it's sometimes hard for them to see that.
CJ: Yes, they get into a circular loop kind of situation. They think they don't have time, little realizing that if they did pretest inspections or static analysis, the testing cycle would be so much shorter that they would actually deliver early.
RB: [Laughs.] Yeah. I remember one of the things that really jumped out at me in your book was you had a chapter where you talked about accumulated technical debt, and just had some stunning figures on the accumulations of technical debt in some organizations. Can you maybe explain for our listeners a little bit more about that analysis that you did?
CJ: Yes. Technical debt is in a section that Olivier has written, too, so he'll talk more about it, I suspect, but the gist of the idea is this: Technical debt is the concept that if you skimp on defect removal and defect prevention while you're building software, when you finally deliver [the software], you're going to pay an ever-increasing amount of money for warranty repairs and fixing the bugs that you failed to eliminate before the software was delivered.
It's like paying interest on a loan; this technical debt stretches out for years and years and years, because in any given calendar year you're not likely to find more than, say, 25 or 30 percent of the bugs that were delivered. So you have at least a four- to six-year period, plus the fact that there's a situation called "bad fix injection": Something like 7 percent of your attempts to fix a bug add a new bug that wasn't there before. So you have kind of a compound interest that stretches out downstream. The basic idea to reduce technical debt is to prevent or remove as many bugs as possible before testing begins, and then to achieve very-high-efficiency test cases and test stages by using certified test personnel and scientific methods for designing test cases, which will give you higher coverage with reduced number of test cases, and that combination should raise your overall defect-removal efficiency from today's average of maybe 85 percent up to 99 percent, which means that your technical debt, which today is millions of dollars every year, would be cut down to almost nothing.
RB: Nice suggestions there. So, now, to pick up on something that you mentioned. You talk about, on average seven percent of defects are bad fixes, which is sometimes referred to in the testing world as regression. Now, we've noticed something interesting with some of our clients, where we do an analysis on what the percentage of bugs are that are regressions bugs or bad fixes. In some cases we've noticed significantly higher than seven percent, and I've generally attributed that to the presence of so-called defect clusters—highly unmaintainable code that also has a lot of bugs in it. Do you have any thoughts on that situation, and what companies that find themselves with excess bad-fix introduction can do to get out of it?
CJ: I do have some thoughts, and also some data—and my data confirms yours. It was discovered back in the late 1970s at IBM that bugs are not randomly distributed through the modules of big systems; they tend to clump in a small number of places IBM called error-prone modules. And in one extreme example, for a very buggy product that had 425 modules, 300 of them were zero-defect modules that never got any bugs, and 57% of the entire customer-reported bug rate came in from only 31 modules out of 425. Those things—a bad-fix injection rate for those can approach 50 percent, much higher than the seven percent average. They are often so complicated and so difficult to fix that they need to be surgically removed. In other words, you can't really repair them in place; you have to isolate them, and then develop better, newer modules, using better techniques to replace the ones that are error-prone.
RB: Right. So basically reengineering or refactoring or whatever the buzzword du jour is for that, but get in there and cut that bad stuff out, and replace it with good code.
CJ: Yes, and one caveat: If you let the people try to do it who built the error-prone module in the first place, you may end up with a new error-prone module that's just as bad as the first one; so you have to be sure that you either use better people or better technologies on the replacement version, because you don't want a second-generation error-prone module.
RB: Right. Right. It's difficult enough to convince management in many situations of the need to do that. You know, schedules are so tight, and budgets are so tight, [if] you say, "Well, we need to have two or three of our best programmers go off and spend two or three months, or maybe more, completely reengineering this one part of this system," many times, I would expect managers would react by saying, "Well, gee, you know, that would be nice, but we really can't afford to do that right now. Maybe we'll do it later." And of course that's one of those "laters" that never comes. Do you have any advice to listeners about, when you're in that situation, how to convince managers with data—how to make an economic argument that investing that time really is time well spent?
CJ: Well, I've seen situations where, when managers of commercial software or outsourced projects ignored error-prone modules, the clients sued them. And when it was discovered in court that they had not paid sufficient attention to those things, obviously there was a serious penalty for those who didn't take proactive steps to remove them. If you don't have the kind of software where your clients might sue you for ignoring these things, it's more difficult to make the case.
But, in general, the higher management in the company—the president, the board of directors, and the senior executives, who are on top of the software people—would welcome lowering the overall maintenance costs and the warranty repair costs, because they are enormous. As a matter of fact, one of the problems—the social problems—brought on by poor quality is the fact that company presidents and boards of directors as a class do not actually respect their software communities. They think that the software communities are less professional than the other parts of the company, primarily because they don't understand quality economics; they deliver software with far too many bugs; they're frequently late when they deliver software; and often, which is even worse, a significant percentage of software projects—the big ones—will be canceled and never get delivered at all, primarily because poor quality turned the return on investment from positive to negative. So the plug was pulled, and the projects were never finished.
RB: Yeah. Indeed. I've been involved in a couple post-mortems on projects like that, and it's just amazing—and somewhat depressing—how easily preventable the disasters were, in many cases.
How about for people who are on projects that are still in progress, and are still savable, if you will, and they see the signs of the kind of problems that you're talking about? Not taking time to do design reviews, not taking time to do code reviews, not taking time to do requirements reviews—a sort of, you know, "We'll let all this stuff sort itself out in the system testing or system integration-testing phase." Any tips to people who find themselves in these kinds of emerging-but-still-preventable disasters on how they should educate their managers about the economics of putting the project back on the right path?
CJ: Well, there's kind of a carrot-and-the-stick sort of suggestion. I worked as an expert witness in 15 lawsuits, and what's interesting is that, in all of the lawsuits where poor quality was part of the case, the technical employees—the software engineers and the test personnel, and Quality Assurance—knew about it, and wrote letters and gave suggestions to management about these problems and suggested that they be fixed. It was the project managers who did not seem to pass on the information to the clients and to higher management, because they naïvely hoped that those bugs and problems could be fixed in system test or at some later point, but they weren't. So when the situation finally exploded and ended up in court, it turns out that management resistance to passing on critical information about quality to clients and higher management is the chief source of the problem. The technical workers, the testers, and the QA people all knew about it, and wanted to fix the problems, but were actually prevented by management decisions.
Now, the "carrot" part (that was the "stick" part) is that, if you require as part of your monthly or weekly report for managers to higher management that the first section be called something like "Red Flag Items," and include quantifications of the number of bugs found compared to the number of bugs that were expected to be found, and the significance of those bugs on the status of the project; if you can change the reporting requirements for software projects so that bug counts and their significance are elevated to the very first thing that gets reported, then managers will begin to take quality seriously, and pretty soon the problems will go away. You won't have the same kind of problems, because managers are suddenly required to report information that they had been concealing.
RB: Right. Which is back to that old management aphorism of "what gets measured, gets done," basically, and its corollary that "what doesn't get measured, doesn't get done"; to the extent that organizations don't have good metrics about quality and defects, then that basically means that it won't get managed.
This concludes part 1 of Rex Black's interview with Capers Jones. Part 2 concludes their discussion about Capers Jones' new book The Economics of Software Quality.