- History
- Structured Development
- Lessons Hard Won
- Technical Innovation
This chapter is from the book
This chapter is from the book
This chapter is from the book
Structured Development
This was unquestionably the single most important advance prior to the '80s. It provided the first truly systematic approach to software development. When combined with the 3GLs of the '60s it enabled huge improvements in productivity.
SD had an interesting side effect that was not really noticed at the time. Applications were more reliable. It wasn't noticed because software was being used much more widely, so it had much higher visibility to non-software people. It still had a lot of defects, and those users still regarded software as unreliable. In fact, though, reliability improved from 150 defects/KLOC in the early '60s to about 15 defects/KLOC by 1980.6
SD was actually an umbrella term that covered a wide variety of software construction approaches. Nonetheless, they usually shared certain characteristics:
- Graphical representation. Each of these fledgling methodologies had some form of graphical notation. The underlying principle was simply that a picture is worth a thousand words.
- Functional isolation. The basic idea was that programs were composed of large numbers of algorithms of varying complexities that played together to solve a given problem. The notion of interacting algorithms was actually a pretty seminal one that arrived just as programs started to become too large for one person to handle in a reasonable time. Functional isolation formalized this idea in things like reusable function libraries, subsystems, and application layers.
- Application programming interfaces (API). When isolating functionality, it still has to be accessed somehow. This led to the notion of an invariant interface to the functionality that enabled all clients to access it in the same way while enabling the implementation of that functionality to be modified without the clients knowing about the changes.
- Programming by contract. This was a logical extension of APIs. The API itself became a contract between a service and its clients. The problem with earlier forays into this idea is that the contract is really about the semantics of the service, but the API only defined the syntax for accessing that semantics. The notion only started to become a serious contract when languages began to incorporate things such as assertions about behavior as part of the program unit. Still, it was a reasonable start for a very good idea.
- Top-down development. The original idea here was to start with high-level, abstract user requirements and gradually refine them into more specific requirements that became more detailed and more specifically related to the computing environment. Top-down development also happened to map very nicely into functional decomposition, which we'll get to in a moment.
- Emergence of analysis and design. SD identified development activities other than just writing 3GL code. Analysis was a sort of hybrid between requirements elicitation, analysis, and specification in the customer's domain and high-level software design in the developer's domain. Design introduced a formal step where the developer provided a graphical description of the detailed software structure before hitting the keyboard to write 3GL code.
SD enabled the construction of programs that were far more maintainable than those done previously. In fact, in very expert and disciplined hands these methods enabled programs to be developed that were just as maintainable as modern OO programs. The problem was that to do that required a lot more discipline and expertise than most software developers had. So another silver bullet missed the scoring rings, but at least it was on the paper. Nonetheless, it is worth noting that every one of these characteristics can be found in modern OO development (though some, like top-down design, have a very limited role).
Functional Decomposition
This was the core design technique that was employed in every SD approach; the Structured in SD refers to this. Functional decomposition deals with the solution exclusively as an algorithm. This is a view that is much closer to scientific programming than, say, management information systems programming. (The name of the first 3GL was FORTRAN, an acronym for FORmula TRANslator.) It is also very close to the hardware computational models that we will discuss shortly.
The basic principle of functional decomposition is divide and conquer. Basically, subdivide large, complex functionality into smaller, more manageable component algorithms in a classic top-down manner. This leads to an inverted tree structure where higher-level functions at the top simply invoke a set of lower-level functions containing the subdivided functionality. The leaves at the base of the tree are atomic functions (on the scale of arithmetic operators) that are so fundamental they cannot be further subdivided. An example is illustrated in Figure 1-1.
)
Figure 1-1 Example of functional decomposition of a task to compute employee stock benefits into more manageable pieces
Functional decomposition was both powerful and appealing. It was powerful because it was ideally suited to managing complexity in the Turing world of an algorithmic calculating machine, especially when the 3GLs provided procedures as basic language constructs. In a world full of complex algorithms, functional decomposition was very intuitive, so the scientific community jumped on functional decomposition like tickets for a tractor pull on Hard Liquor and Handgun Night.
It was appealing because it combined notions of functional isolation (e.g., the limbs of the tree and the details of the subdivided functions), programming by contract in that the subdivided functions provided services to higher-level clients, a road map for top-down development once the decomposition tree was defined, APIs in the form of procedure signatures, very basic depth-first navigation for flow of control, and reuse by invoking the same limb from different program contexts. Overall it was a very clever way of dealing with a number of disparate issues.
Alas, by the late '70s it was becoming clear that SD had ushered in a new set of problems that no one had anticipated. Those problems were related to two orthogonal realities:
- In the scientific arena algorithms didn't change; typically, they either stood the test of time or were replaced in their entirety by a new, superior algorithms. But in arenas like business programming, the rules were constantly changing and products were constantly evolving. So applications needed to be modified throughout their useful lives, sometimes even during the initial development.
- Hierarchical structures are difficult to modify.7
The problem was that functional decomposition was inherently a depth-first paradigm since functions can't complete until all subdivided child functions complete. This resulted in a rather rigid up-and-down hierarchical structure for flow of control that was difficult to modify when the requirements changed. Changing the flow of control often meant completely reorganizing groups of limbs.
Another problem was redundancy. The suite of atomic functions was typically quite limited, so different limbs tended to use many of the same atomic operations that other limbs needed. Quite often the same sequence of atomic leaf operations was repeated in the same order in different limbs. It was tedious to construct such redundant limbs, but it wasn't a serious flaw until maintenance was done. If the same change needed to be made in multiple limbs, one had to duplicate the same change multiple times. Such duplication increased the opportunities for inserting errors. By the late 1970s redundant code was widely recognized as one of the major causes of poor reliability resulting from maintenance.
To cure that problem, higher-level services were defined that captured particular sequences of operations once, and they were reused by invoking them from different limbs of the tree. That cured the redundancy but created an even worse problem. In a pure functional decomposition tree there is exactly one client for every procedure, so the tree is highly directed. The difficulty with reuse across limbs was that the tree became a lattice where services had both multiple descending child functions and multiple parent clients that invoked them, as shown in Figure 1-2.
)
Figure 1-2 Functional decomposition tree becoming a lattice. Shaded tasks are fundamental elements of different benefits. Dashed lines indicate crossovers from one decomposition limb to another to eliminate redundancy.
Figure 1-1 has been expanded to include the computation of multiple employee benefits that all require basically the same tasks to be done.8 Some tasks must be implemented in a unique way so the basic tree has the same structure for each benefit. But some tasks are exactly the same. To avoid redundancy, those nodes in the tree are reused by clients in different benefit limbs of the tree. This results in a lattice-like structure where tasks like Get base salary from DB have multiple clients that are in quite different parts of the application.
That lattice structure was a major flaw with respect to maintainability. When the requirements changed for some higher-level function (e.g., Insurance Benefit in Figure 1-2) in a particular limb, the change might need to be implemented in a descendant sub-function much lower in the tree. However, the sub-function might have multiple clients from other limbs due to reuse (e.g., Get base salary from DB). The requirements change might not apply to some of those clients that were in different limbs (e.g., Stock Benefit) because each limb where the sub-function was reused represented a different context. Then the low level change for one client might break other clients in other contexts.
The biggest difficulty for functional decomposition, though, was the implicit knowledge of context that came with depth-first processing. The fundamental paradigm was Do This. Higher-level functions were essentially just a collection of instructions to lower-level functions to do things. To issue an instruction to do something, the higher level function must (a) know who will do it, (b) know that they can do it, and (c) know that it is the next thing to do in the overall solution. All of these break functional isolation to some extent, but the last is most insidious because it requires that the calling function understand the much higher-level context of the whole problem solution. The higher-level function hard-wires that knowledge in its implementation, so when requirements changed for the overall context, that implementation had to be changed. This hierarchical functional dependence, combined with all the other problems just mentioned, resulted in the legendary spaghetti code.
Another way to look at this is through the idea of specification and DbC contracts. When a function is invoked, there is a Do This contract with the client invoking the function. That DbC contract represents an expectation of what service the function will provide. It does not matter whether the function provides the entire service itself (i.e., it is not subdivided) or delegates some or all of its functionality to lower-level functions. From the client's perspective, the contract with the function in hand is for the entire service. If the function in hand is a higher-level function in the tree, the specification of what that function does is the specification of all of the descending limbs. The lower-level functions descending from the higher-level functions are extensions of it, and their individual specifications are subsets of the higher-level function's specification.
This means that all of the limbs descending from a given higher-level function in the lattice form a complex dependency relationship originating with the higher-level function. That is, to fulfill its responsibilities in the DbC contract with its client, a higher-level function depends on every descendant lower-level function doing the right thing with respect to its client's expectations. It is that dependency chain that is the real root cause of spaghetti code. A lower-level function's specification cannot be changed without affecting a potentially long chain of parent (client) specifications.
A similar dependency problem existed for sequences of operations. To modify the sequence in which the leaf functions were invoked, the implementations of the higher-level functions had to be touched. That is, the sequence was determined by moving up and down the limbs in a systematic, depth-first manner. That navigation of the tree was hard-coded in the higher-level functions' implementations. So to change the sequence, the implementations of multiple higher-level functions had to be changed. In effect, the overall solution flow of control was hard-wired into the tree structure itself, which sometimes required reorganizing the entire tree to accommodate small sequencing changes.
One way this is manifested is in doing unit tests of higher-level functions. Since the specification of the higher-level function includes the specification of every descending lower-level function, it is not possible to unit test a higher-level function from the client's perspective without working implementations of every descending lower-level function.9
- A strong case can be made that the primary goal of the OO paradigm is to completely eliminate the hierarchical dependencies resulting from functional decomposition.
It is not terribly surprising that no one anticipated these problems. They only showed up when SD's productivity gains were realized. As applications became larger, there were more requirements changes. Meanwhile, the functional decomposition trees were growing in size and complexity. Only when the benefits were realized did the problems show up.