Don't "Enron" Your Software Project
Date: Oct 10, 2009
In today's world of massive financial catastrophe (events such as the collapse of Lehman Brothers and Bear Stearns, or the Madoff pyramid scheme), it becomes almost quaint to think about comparatively small frauds such as the Enron scandal of 2001. These scandals make the news with seeming regularity, having profound effects on the financial markets.
Sadly, in the world of software development, we have our own versions of these scandals. Whereas financial scandals cause outrage and congressional hearings, the business of software delivery in IT has effectively normalized this condition, to the degree where we're surprised when it doesn't happen. The fact that most companies have to spend 80% of their software development budget maintaining code is not just an inconvenience for software developers who have to maintain bad code; it's literally a drag on economies that depend on productivity increases—usually from new computer technology—to grow. While it isn't a scandal that makes the news frequently, it's nonetheless an issue that acts like a tax on productivity.
Technical Debt: A Primer
Why do we spend 80% of our software development budgets on software maintenance? Simply put: Despite our better instincts, it's still too easy to do the wrong thing when developing software. One example is the practice of "cut-and-paste coding." Suppose you have a sales compensation system that computes commissions. A batch process runs during each payroll to perform the commission calculation. This process sits in the payroll system, but somewhere else, perhaps in some commission dashboard that your salespeople use to see interim progress, you have the same commission calculation. As the business sponsor of the system, you signed off on this system after it was built initially. And why not? Technically, the system met its requirements—the commission calculation was correct (in both places).
The problem occurs when your commission structure changes. When someone comes along to change the system to accommodate the new commission structure, the code in at least two (and possibly more) unrelated areas needs to be changed. Worse, you have no reliable way of knowing whether all of those changes are right, without a full regression test of the system—if such a test is even possible!
In such a case, you have technical debt, defined as "the sum of future maintenance that could have been avoided if the software had been developed according to reasonable software-design practices." Cut-and-paste coding is just one way that technical debt is introduced. Designs that lack testability, inappropriate coupling, and dozens of other such practices contribute to this problem.
Why We Get Technical Debt
Like credit card debt, technical debt might "feel" good when we're in the process of accumulating it. We do things that optimize for the short term, but have a long-term expense that's far higher than the short-term gain we get—most of the time, as least. During the initial build of a system, it's easier to cut-and-paste logic from one module to another, rather than finding a way to reuse the logic. Technical debt usually comes from short-term optimizations of time without regard to the long-term effects of the change.
Interest Rate on Technical Debt
If the short-term optimizations that occur most of the time had a minor effect on maintenance (something like a 6% mortgage, say), such decisions to take on technical debt would be just fine. However, we frequently allow software development organizations to take out technical loans that, if they were transparent, would throw the corporate treasury department into a spiral of panic.
Let's look at our case study of the cut-and-paste code. The investment to do it right the first time is low (two days of work). In this case, you might write a shared module that does a commission calculation, making it available to any other module that needs such a calculation. Say we allow the development team to take out the "loan," and do cut-and-paste rather than spend the two days. In exchange for the two days of work, we'll incur future maintenance costs that are far higher. Making a conservative assumption, we'll say that we only did the cut-and-paste operation twice, resulting in the same calculation being in three places. Every time we change the commission structure in the future, we'll need to do the following:
- Find each instance where the commission calculation is performed—hoping that we got all of them (1 day)
- Make the change at each location—possibly customizing each change due to stylistic differences introduced by different developers (1 day each, 3 days total)
- Test each change, including performing regression tests to make sure that the change doesn't cause problems in seemingly unrelated places (1 day each, 3 days total)
- Put each change through the standard change-management process in multiple systems (0.5 days each, 1.5 days total)
- Discover that we missed one or two places, and repeat the process for each one we didn't find originally, upsetting the salespeople whose commissions were paid incorrectly (2.5 days minimum, plus an immeasurable amount of lost confidence)
At this point, we have 11 days of additional work invested every time we change the commission structure. Guessing conservatively, perhaps we only change the commission structure three times over 10 years (some places change it annually), we have net-interest payments of 33 days of effort, for which we were paid two days of effort up front.
The approach in this example results in a debt with an interest rate of about 32.4% (starting with two days of effort gained and paying it off with 33 days of effort lost over a 10-year period). I'm not a corporate policy expert, but my guess is that most accounting departments would balk at IT regularly taking out loans with interest rates like 32.4% that make loan sharks envious.
The Enron Model of Technical Finance
Let's spend a few seconds looking at a little history. One of the big things that got Enron in trouble was that it routinely took risky bets. Depending on the outcome, Enron would count the wins on the main balance sheet, and count the losses in something it called an "off balance-sheet entity"—which basically was a structure, hidden from view, that held the debt related to the losses. Of course, the "off balance-sheet entity" wasn't reported in the financial statements that Wall Street used to judge the finances of the company, resulting in the ensuing scandal.
Sadly, common practice for return on investment (ROI) analysis in software development projects results in the creation of comparable technical "off balance-sheet entities." The occasional ROI analysis will include a line item for cost of ownership; in turn, where this line item has a very optimistic projection for ongoing maintenance costs, rare is the case where such a number is questioned. Rarer still is a case where an organization demands that a Statement of Work require meeting code-quality metrics that help manage down the amount of technical debt with which a solution is loaded on delivery. Put bluntly, most software is delivered with hidden liabilities—"off balance-sheet entities," if you will—that result in today's familiar high maintenance costs.
Costs of the Enron Software Finance Model
It would be bad enough if the costs only involved software maintenance. The damage of this practice is only the tip of the iceberg. The real costs occur when this sense of "custom software is hideously expensive long term" is normalized. Because this is considered normal, companies often do everything they can to avoid writing custom software. As a result, vast opportunities for additional productivity—not just in individual companies, but for the economy as a whole—go unrealized.
I believe that we've already started to see the effects. Time after time, I've seen companies in trouble due to bad software. Companies that could increase their margins by 10% simply by adding a second shift, but they couldn't run such a shift because of limitations in their legacy software—mostly related to maintenance. Companies where technology literally controls how often the company is willing to adjust the commission structure of the sales force. I've seen mergers and acquisitions become unprofitable as a result of high-tech integration costs. (Mergers often force early pay-down of technical debt.) Badly written software, laden with technical debt, extracts a terrible cost on our economy as a whole.
Technical Fraud
As a reaction to seeing this condition and its deleterious effects, I coined the term technical fraud to refer to the practice of incurring unmanaged and hidden technical debt. Many U.S. states have "lemon laws" that make it illegal to knowingly sell someone a car that has undisclosed maintenance problems. Selling a "lemon" is a fraudulent practice in the world of cars, and it should be considered as such in the world of software.
Are We All Technical Fraudsters?
Are most people who deliver software purposefully engaging in fraud? Of course not. We work in a business where this fraud has become normalized due to various forces that are mostly beyond our control; in particular, market forces that seek to minimize the initial delivery cost of software over the long-term maintenance costs.
The good news is that the situation isn't hopeless. In fact, practices, tools, and disciplines are already available that can help us to deliver software without engaging in technical fraud.
In future articles in this series on the topic of technical finance, I'll cover various practices that we can leverage to root out technical fraud, restoring the promise of software as a force that adds positive net economic value to companies.