This chapter is from the book
This chapter is from the book
This chapter is from the book
Why Agile?
But why the need for a concept such as “agile” to describe software development? Aren’t current software development processes good enough?
No, not really.
A process, in this context, can be defined as “a planned set of work tasks performed by workers in specific roles resulting in changes of attributes, state, or other characteristics of one or more work products.” The underlying assumptions are the following:
- The results of using the process are repeatable, resulting in a product with expected properties (e.g., functionality and quality).
- The production of the goal state of the work products is highly predictable when executing the process in terms of the project (e.g., cost, effort, calendar time) and product (e.g., functionality, timeliness, and robustness) properties.
- People can be treated as anonymous, largely interchangeable resources.
- The problems of software development are infinitely scalable—that is, doubling the resources will always result in halving the calendar time.
As it turns out, software is hard to develop. Most existing development processes are most certainly not repeatable or predictable in the sense above. There are many reasons proposed for why that is. For myself, I think software is fundamentally complex—that is, it embodies the “stuff” of complexity. That’s what software is best at—capturing how algorithms and state machines manipulate multitudes of data within vast ranges to achieve a set of computational results. It’s “thought stuff,” and that’s hard.
The best story I’ve heard about software predictability is from a blog on the SlickEdit Web site by Scott Westfall called the “The Parable of the Cave” (see sidebar).5 Estimating software projects turns out to be remarkably similar to estimating how long it will take to explore an unknown cave, yet managers often insist on being given highly precise estimates.
In addition, the scope of software is increasing rapidly. Compared to the scope of the software functionality in decades past, software these days does orders of magnitude more. Back in the day,6 my first IBM PC had 64kB of memory and ran a basic disk operating system called DOS. DOS fit on a single 360kB floppy disk. Windows XP weighs in at well over 30 million lines of code; drives hundreds of different printers, disks, displays, and other peripherals; and needs a gigabyte of memory to run comfortably. These software-intensive systems deliver far more functionality than the electronic-only devices they replace. Compare, for example, a traditional phone handset with a modern cell phone. Or compare a traditional electrocardiogram (ECG) that drove a paper recorder like the one I used in medical school with a modern ECG machine—the difference is remarkable. The modern machine can do everything the old machine did, plus detect a wide range of arrhythmias, track patient data, produce reports, and measure noninvasive blood pressure, blood oxygen concentration, a variety of temperatures, and even cardiac output.
Last, software development is really invention, and invention is not a highly predictable thing. In electronic and mechanical engineering, a great deal of the work is conceptually simply putting pieces together to achieve a desired goal, but in software those pieces are most often invented (or reinvented) for every project. This is not to oversimplify the problems of electronic or mechanical design but merely to point out that the underlying physics of those disciplines is far more mature and well understood than that of software.
But it doesn’t really matter if you believe my explanations; the empirical results of decades of software development are available. Most products are late.7 Most products are delivered with numerous and often significant defects. Most products don’t deliver all the planned functionality. We have become used to rebooting our devices, but 30 years ago it would have been unthinkable that we would have to turn our phones off, remove the batteries, count to 30, reinsert the batteries, and reboot our phones.8 Unfortunately, that is the “state of the art” today.
To this end, many bright people have proposed processes as a means of combating the problem, reasoning that if people engineered software rather than hacked away at it, the results would be better. And they have, to a large degree, been better. Nevertheless, these approaches have been based on the premise that software development can be treated the same as an industrial manufacturing process and achieve the same results. Industrial automation problems are highly predictable, and so this approach makes a great deal of sense when the underlying mechanisms driving the process are very well understood and are inherently linear (i.e., a small change in input results in an equally small change in output). It makes less sense when the underlying mechanisms are not fully understood or the process is highly nonlinear. Unfortunately, software development is neither fully understood nor even remotely linear.
It is like the difference in the application of fuzzy logic and neural networks to nonlinear control systems. Fuzzy logic systems work by applying the concept of partial membership and using a centroid computation to determine outputs. The partial membership of different sets (mapping to different equations) is defined by set membership rules, so fuzzy logic systems are best applied when the rules are known and understood, such as in speed control systems.
Neural networks, on the other hand, don’t know or care about rules. They work by training clusters of simple but deeply interconnected processing units (neurons). The training involves applying known inputs (“exemplars”) and adjusting the weights of the connections until you get the expected outputs. Once trained, the neural network can produce results from previously unseen data input sets and produce control outputs. The neural network learns the effects of the underlying mechanisms from actual data, but it doesn’t in any significant way “understand” those mechanisms. Neural networks are best used when the underlying mechanisms are not well understood because they can learn the data transformations inherent in the mechanisms.
Rigorously planned processes are akin to fuzzy logic—they make a priori assumptions about the underlying mechanisms. When they are right, a highly predictable scheme results. However, if those a priori assumptions are either wrong or missing, then they yield less successful results. In this case, the approach must be tuned with empirical data. To this end, most traditional processes do “extra” work and produce “extra” products to help manage the process. These typically include
- Schedules
- Management plans
- Metrics (e.g., source lines of code [SLOC] or defect density)
- Peer and management reviews and walk-throughs
- Progress reports
And so on.
The idea is that the execution of these tasks and the production of the work products correlate closely with project timeliness and product functionality and quality. However, many of the tasks and measures used don’t correlate very well, even if they are easy to measure. Even when they do correlate well, they incur extra cost and time.
Agile methods are a reaction in the developer community to the high cost and effort of these industrial approaches to software development. The mechanisms by which we invent software are not so well understood as to be highly predictable. Further, small changes in requirements or architecture can result in huge differences in development approach and effort. Because of this, empiricism, discipline, quality focus, and stakeholder focus must all be present in our development processes. To this end, agile methods are not about hacking code but instead are about focusing effort on the things that demonstrably add value and defocusing on efforts that do not.