Improving the Planning Process

This chapter is from the book 

3.7 Getting Planning Just Right

Goldilocks would have been a great project planner. She had the rare ability to sense when the porridge was just right. She would have known that too little planning (the cold porridge) would leave her unsatisfied, while too much planning (the hot porridge) would burn her mouth. Recognizing when to stop and eat is the trickiest part of any software planning.

It is well-known that many projects suffer problems as a result of underplanning. On the other hand, zealous efforts to avoid underplanning frequently result in overplanning. Both extremes arise from the lack of a planning process and the lack of a specification blueprint that directs efforts into a plan that is just right.

Underplanning

Momma bear's porridge was too cold for Goldilocks. Momma bear didn't take enough time to make sure it was thoroughly cooked and flavorful. Maybe she was in too much of a hurry because Poppa bear was impatient to leave on schedule.

Virtually the entire industry is in consensus regarding the hazards of underplanning. Underplanning is the most commonly cited cause of project failure. Eliminating underplanning is the focus of most project planning efforts.

Responding to an omission uncovered during development can cost many, many times the amount of money it would have required to implement correctly the first time had it been specified in the plan. In evolutionary development, the greatest risk is that major functionality is overlooked until late in the process. At that time, major rewrites may be required. It is also quite risky to leave any detail unplanned any time programming practices do not result in highly maintainable code, and when the project size increases.

Scope creep (also commonly called requirements creep) is the most common nuisance effect of underplanning. Scope creep is basically a catchall term for any requirement changes that surface during development. There are three general types of scope creep:

Overlooked Requirements

These are requirements that were simply missed during planning. They may have been missed because the planner failed to recognize them or because the client failed to anticipate them. These are the most serious since they can have dramatic impact on how the solution is fundamentally structured. While every effort must be made to anticipate and plan for major requirements, it is likely that some will be missed.

Functionality Enhancements

These scope creep issues are all those thousand and one little tweaks that are “no big deal.” These are the kind of straws that can eventually break the camel's back. Many of these are things that the client realizes would be essential or nice to have only after seeing the application take shape. It is extremely difficult to anticipate all of these in advance.

Changing Requirements

These are changes that result from external changes, either initiated by the client or by outside forces. If the client changes their mind or revises their requirements, there is not much that can be done about it. If changes in third-party products, regulations, or supporting systems occur, there is likewise little that can be done to prevent it. This is always a risk that cannot be avoided.

The waterfall approaches attempt to minimize scope creep through exhaustive planning. When changing requirements occur, they are often quite disruptive to the process. Waterfall managers usually try to limit scope creep by creating substantial political and logistic barriers to change.

The evolutionary approaches start with the realistic assumption that the client will always overlook requirements and that implementing functionality enhancements as they are discovered is more efficient then planning minutia up front. They anticipate that unanticipated external forces will always force requirements to change and they prepare for that.

The adaptive perspective on scope creep is similar to the evolutionary viewpoint, but it still attempts to make quantum leaps in evolution through reasonable formal planning.

Perhaps the biggest risk of underplanning is that problems linked to scope creep will drive an already restrictive management further toward the extreme of overplanning and overly rigid specifications. Overplanning creates a different, and in many ways more severe, set of problems.

Overplanning

Poppa bear's porridge was too hot. He apparently was so concerned that it be sufficiently heated that it became impossible to eat. Maybe he got botulism from a previous bowl that was undercooked.

Fear of overlooking a major requirement and previous experience with “death by a thousand tweaks” are the major reasons that planners go overboard in an effort to specify every possible detail. Despite the paranoia about underplanning, few people discuss the problems caused by overplanning.

Don't mistakenly conclude that we are dismissing the paramount importance of planning in the software development process. On the contrary, this entire book exists to stress the importance of planning—efficient, useful planning.

As much as software planners hate to acknowledge it, the amount of effort we put into project planning does not always correlate well with project success. It is rather heretical to make this assertion in a culture in which more planning is virtually a religious doctrine, but unless you are willing to rely upon faith, the assertion is empirically evident.

The fact is that we do see “seat-of-the-pants” software projects that succeed spectacularly. Sure, many of them do fail. Planning enthusiasts typically point to these to urge project managers to allocate a bigger piece of the project pie to planning. However, it is undeniable that many exhaustively planned projects fail as well. Of course, this does not normally call into question the benefit of planning, but rather proves their argument that even more planning is needed next time. It is like the “any evidence proves my position” logic of the Spanish inquisition.

By the time overplanning is complete, the project is probably late and overbudget. Inevitably, those planning cost overruns will be passed on to development with the assumption that all that extensive planning should reduce development costs. Unfortunately, the reverse is more often true. The large amount of bad planning, and the budget stresses it introduces, make it even more difficult for development to succeed.

It is dangerous to buy into the idea that failure is always due to lack of planning and that success is always the result of great planning. Although planning experts like to proclaim that the day of the heroic programmer is past, there are still heroes out there. Project success is sometimes the result of heroic efforts by developers to overcome a lot of bad planning. At other times, success or failure is due to a combination of reasons that are completely unrelated to planning.

While there are many cases in which too little planning leads to project inefficiency and even project failure, there are also many cases in which too much planning becomes counterproductive and self-destructive. Clearly there are many other factors beyond mere quantity of the planning effort that play critical roles in determining project outcome.

There is a proportional relationship between project size and planning requirements. The success rate of seat-of-the-pants projects does generally diminish as projects get larger. But even in larger projects, the fact that a greater degree of planning is necessary does not negate the fact that some projects of equivalent size succeed better with a relatively smaller planning expenditure.

As the amount of planning goes beyond the optimum for its size, it passes a point of diminishing returns. At that point, it takes just as long to plan a new feature as it would to add it “on the fly.” From then on, the planning cost/benefit drops dramatically. Eventually, additional planning actually becomes a counterproductive stress, reducing overall productivity.

Certainly there is a great benefit to planning when you are on the early side of the curve. However, as you reach the point of diminishing returns, the benefit falls. After that point, it costs increasingly more to research and specify minutia up front than it would to research that at the appropriate time during development.

The ten-to-one ratio that authors commonly cite to reflect the cost of fixing a problem after planning is complete does not consider how close that item is to the point of diminishing returns. They fail to consider that planning a feature past the point of diminishing returns is more expensive than planning it during development, provided the code has acceptable levels of maintainability. They cite the ten-to-one benefit of planning as if it were constant across the entire curve.

The point of diminishing planning returns is attributable to the fact that as the planning effort delves into greater detail, that information takes exponentially longer to obtain and document. The return on that investment in planning quickly decreases. This is compounded by the fact that as more detailed planning is performed, it is far more likely to be scrapped later as the actual work unfolds and uncovers new information.

The insidious thing about overplanning is that it creates the illusion of a complete plan and makes it far more difficult to obtain truly useful information during development. It makes it much harder for the developers to find the useful needle in the haystack of information, if it is there at all.

It also makes it much harder to go back to the client for the real information that the developers need. “You have the plan,” or “Didn't we spec that all out for you?” are the responses they are likely to get to their questions, followed by the typical management conclusion, “I guess we have to plan in more detail next time.”

Figure 3-2 illustrates the law of diminishing returns as it applies to software planning. The point of diminishing returns typically falls in the 70% to 80% range of the total planning effort. In practice, the actual level of diminishing returns may be higher or lower, depending upon the nature of the project and the people involved.

Figure 3-2. Planning benefits.

Also assumed in this analysis is that the requirements are prioritized so that the more important ones—the ones that impact on other decisions—are made early in the planning effort. The last 20% to 30% should be low-impact details that are more efficiently fleshed out later in the development process.

Certainly, there is a chance that a major requirement will turn out to be hiding within that last 20% to 30% left unspecified until too late. This could also happen if the requirement were simply missed or if circumstances changed. In any event, one must play the odds to some extent rather than succumb to self-defeating overplanning. The cost of uncovering such a hidden requirement may be larger than the cost of fixing it.

In the adaptive process, the goal is to do formal planning until you reach the point of diminishing returns. After that point, you rely upon evolutionary development to obtain the remaining specifications in a much more cost-effective manner. It is therefore not useful to cite or think of a particular planning level at which the point of diminishing returns is reached, but rather to establish an objective target blueprint which, when complete, helps assure that the essential information has been obtained to proceed from planning to evolutionary development.

Planning Quality

The previous comments on overplanning are all true enough when you assume your planning is good planning. The fact is that most planning is not very good. This makes the situation far worse since too much bad planning is far worse than too much good planning. The reason that many projects succeed with only modest planning may be in part because those teams haven't invested a huge and unproductive effort in poor planning. Quality must be considered as well as quantity.

In the same way, more bad software specifications are prescribed repeatedly to cure what ails software projects. This is done even when the problems are noncorrectable by planning or when bad planning itself is the cause. One must consider both quality and quantity in planning and recognize that less planning can be better. Good planning is planning with a minimum amount of superfluous material.

Therefore, good planning is self-limiting. When there is too much good planning detail, at some point it becomes bad planning which is no longer cost-effective. It would be much more economical and sensible to obtain that detail during development.

So the common perception that projects fail mostly because of insufficient planning is only partly true. It would be more accurate to say that the project failed because of insufficient good planning or too much bad planning.

While it is rare for anyone to question the benefit of planning in any form, I am not the only author to stress that the quality of a plan is more important than quantity. In Anti Patterns (Brown, 1998), the authors discuss various development, architectural, and management anti patterns that plague software projects. Of those anti patterns relating to planning, many more are a result of too much planning than too little planning.

Here are some of the anti patterns that are symptomatic of too much (bad) planning.

Cover Your Assets

This is the pattern in which the planners document every possible specification and alternative in an attempt to make sure that they aren't accused of being incomplete. In our prevailing climate in which the answer to every problem is more planning, in which more planning is better planning, planners feel an acute need to demonstrate that they specified every possible nuance. By doing so, they are covered and protected from finger pointing when the project goes south—possibly as a consequence of all their attempts to cover their assets.

The problem with the “cover your assets” antipattern is that it indulges in too much planning to create an illusion of personal security. It pays too little regard to quality, and to the effect of overplanning on the budget and on subsequent development.

Analysis Paralysis

Analysis paralysis is another complication of the “more is better” disease. The planners are so preoccupied with striving for perfection and completeness in planning that it leads to project gridlock. The planning documents become so voluminous with detail that they turn the process into an incomprehensible, slowly moving traffic jam.

This time, in an effort to reach the highest possible level of detail, the planners inefficiently overspecify and place stress on the timeline.

Viewgraph Engineering

This antipattern is the result of an obsession with creating high-level metadata in the planning phase. The planners become enamored with their full-color glossy object models, state diagrams, and flow charts and the tools that create them. They start to imagine that this metadata provides a blueprint when, in fact, it is only marginally useful to developers, if at all.

Death by Planning

This anti pattern is another variation of the other excess planning varieties. It describes the situation in which excessive planning leads to overly complex schedules and project-killing levels of management overhead.

The best way to ensure the quality of your plan is to determine first what should be included in your blueprint and then direct your efforts toward producing that content. Then the best way to assess whether your blueprint truly provides quality content is to ask your customers, the project developers, whether it specifies too little, or more importantly, too much information.

The Perfect Plan

The overplanning anti patterns are really symptoms of an underlying belief in the perfect plan. Just as there is no perfect plan for a crime, there is also no perfect plan for a software product.

Sometimes it is possible to come close to perfection, but it costs a lot. It is like reaching the speed of light. As you get closer, the energy expended increases exponentially. That is why large-scale waterfall projects must allocate tremendous amounts of resources to project planning.

How close the plan comes to perfection also depends upon the particular nature of the project and the people involved. If the requirements of the project are relatively straightforward, and if the experts themselves understand their requirements up front, and if the stars come into cosmic alignment, the planners can produce a plan that truly provides developers with what they need. In these cases, it is possible to produce a specification that you can ship off to a team of developers working in a foreign country to implement.

It is a common mistake, however, for some planners to assert that this is always possible. They believe it religiously or as scientific fact. They believe it through wishful thinking or out of a desperate need to believe. They believe it out of naiveté or they believe it because they worked on a previous project that went well. They believe it because it improves their credibility in their profession or workplace. Maybe they believe it because they hope for zero risk.

Occasionally you do get away with a crime. Maybe the plan was perfect. That doesn't mean you can plan every crime to perfection. That cocky attitude will land you in jail fast. Or maybe you just got lucky once and credited yourself with perfect planning.

The truth is that most of the time our best-laid plans don't serve us very well. There is a commonly paraphrased military expression that no plan survives the first contact with the enemy. This is especially true in software planning and development. It could be modified to say that no specification survives the first contact with the user.

Although the perfect plan is not always achievable, that does not mean that we should not strive for the best plan in every project. Dwight D. Eisenhower said that planning is essential, but plans are inconsequential. Since plans are essential, we must do our best to come up with the best plan possible.

“While we sometimes get firm requirements, a software perversity law seems to dictate that the firmer the specifications are, the more likely they are to be wrong. The fact is that the demand for firm and unchanging requirements is mostly wishful thinking and may even be an unconscious attempt to avoid blame for poor results.” (Humphrey, 1989)⁵

Unless we have an unlimited planning budget or simply no alternative, the best compromise is to call it good enough when our design effort reaches the point of diminishing returns.

Even though we can't plan every detail perfectly, we can create a really good plan, or at least a much better plan than what we may have been producing in the past for the same amount of effort. The fact that we have limited planning resources only underscores the importance of expending our planning budget efficiently and effectively.

With the acceptance that the best we can do is a good, cost-effective plan, not a perfect plan, also comes some realistic acceptance of the need for enough flexibility to improve it throughout the development process.

The trick is knowing when to call the plan good enough. This is where all the personal politics kick in. This is where things get ugly. This is where project dysfunction such as rigor mortis takes root.

Rigor Mortis

Rigor mortis often follows extreme overplanning in the quest for perfection. Rigor mortis exists when a plan becomes rigidly set in stone. After rigor mortis sets in, it becomes very difficult to make changes, no matter how beneficial they may be to the overall project, even if failure to make changes is killing the project.

Rigor mortis occurs for many reasons:

• Paranoid fear of scope creep

  The Bookcase Book Bookcase

  Bob had just published a how-to book on building bookcases. He asks his handyman George to build him a new bookcase for his bookcase book—using a blueprint he published in the book. George glances at the plan and attempts to offer some suggestions. Bob cuts him off, tersely telling George that he is expected to just build the bookcase according to his detailed diagram. George eventually gives up in frustration and goes to work.

  At the end of the show, Bob comes in and admires the fruition of his plan. The bookcase looks great. It is sturdy and attractive, constructed with great craftsmanship. He gloats and brags to George about how well his plan worked out. He goes to his desk, picks up the copy of his bookcase book, and goes to put it on the shelf. The book doesn't fit. All the shelves are too small!

  George has the dignity and grace to merely offer a satisfied smile, collect his tools, and leave without saying a word.

• Personal investment in proving your plan was “perfect”

• Pressure to avoid cost increases or appearing obstructionist

• The fear that management or the client will view change as proof of previous inefficiency or mistakes

When companies invest heavily in planning, both emotionally and financially, rigor mortis, or least resistance to change, is hard to avoid. Inflexibility is a typical problem in most companies. Even though the authors may proclaim that their plan is a “living document,” this seldom happens in practice. Firstly, the effort to maintain the document is typically so great that it inhibits the revision process. Secondly, most development coding is not maintainable enough to accommodate change. Thirdly, there is a certain emotional reluctance by the authors to make changes. To admit to the need for a change is to admit that their plan was not perfect. For some managers, they would rather force an incredibly bad design upon the project rather than allow any inference of failure on their part.

This was humorously enacted in an episode of the old Newhart television show:

Certainly this represents a dilemma that many of us in software development can relate to. Often times we have worked for the manager or project planner who has worked hard on his or her specification, and is reluctant or even hostile when the developers suggest changes.⁶ To admit that one's plan is not perfect is an admission of fallibility. Further, any such changes must be avoided to prevent scope creep and having to go back to the client for potentially embarrassing approvals. But since plans are seldom perfect, we end up with software that meets the plan perfectly but doesn't hold any books.

Rigor mortis should be avoided in project planning and management. It is simply not an effective risk-management technique in the long run.

Perhaps, however, there is such a thing as a perfect plan, athough it is just not necessarily an exhaustive plan. A perfect plan is just right planning that achieves an appropriate level of detail as efficiently as possible, a plan which accommodates changes and enhancements without difficulty.

Just Right Planning

The perfect plan meets the Goldilocks criterion. It is just right. It is neither too hot nor too cold. So how much planning is optimum? That is a very difficult judgment call in most planning efforts. If the planning effort suffers from bad planning practices, no amount is ever enough. The planners sense this but just keep planning more and more, ignoring that the planning is becoming increasingly counterproductive.

If you have a good blueprint, however, then you have a huge advantage. When that blueprint is complete, you should have reached the point of diminishing returns. You have acquired most of the essential information. The blueprint should be your objective measure of completeness and quality.

“One of the most difficult challenges we face in the requirements process is making the requirements detailed enough to be well understood without over constraining the system and predefining a whole host of things that may be better off left to others downstream in the process.” (Leffingwell, 2000)⁷

We have examined the opposing syndromes of under- and overplanning. Weighing the relative risks, I fear the risk of overplanning far more than the risk of underplanning. I am confident I can mitigate the risks of underplanning with intelligent application design. I know that through flexible programming techniques I can develop software that can easily accommodate all but the most major scope creep. It is relatively easy to catch major requirements with even cursory planning. If you develop a fragile house of cards to meet the exact specification, it is likely to collapse as soon as maintenance changes or enhancements are required. An application that cannot easily and quickly accommodate changing specifications during development is not worth shipping.

On the other hand, overplanning is a nemesis for which I have little defense. If the planning department spends over half the project budget and puts it behind schedule I have little recourse except to cut development corners or burn the midnight oil. If they never ask me what I need and give me a lot of stuff that I can't use directly, I have to go back to the client and re-engineer anyway. If management wises up to that and cuts off lines of communication, then I end up making guesses and getting hammered for errors discovered late in the project. If the plan specifies approaches that are impractical or inefficient, I must battle rigor mortis for each change or struggle to deliver inefficient requirements. When scope creep occurs, as it always will despite any plan, I am less able politically and technically to respond effectively because of the heavy investment in overplanning.

If forced to choose between erring on one side or the other, I would err on the side of underplanning and rely upon my maintainable program design and implementation strategies to mitigate the risks of scope creep. It is reasonable, though, for organizations with less confidence in their development team to err on the other side.

Therefore, if your project suffers from scope creep, you might want to avoid a futile and counterproductive effort to eliminate all scope creep through overplanning. Instead, you might ask why your project is so adversely impacted by scope creep and look for planning and development strategies to embrace and accommodate change more effectively.

Inside the Blueprint

Up until now the software blueprint has been a conceptual construct. It has been defined as something that provides developers with the information they need to produce software most efficiently. It is differentiated from a traditional functional specification in that those documents often contain a great deal of information that developers don't need and lack information that is critical to them. Typical specifications also do not normally have the level of internal consistency and rigor demanded by a software blueprint.

In order to give you an idea of what details to put into a software blueprint, we must discuss a specific implementation of a software blueprint. I will give you some details of what I include in my implementation of a software blueprint, but of course you are free to create your own version of a blueprint to meet your particular needs.

Your blueprint will probably contain at least these minimum elements and features since they are fundamental and universal. You may very well decide to extend it, but you should always stay true to the goal of providing only essential information in the most developer-friendly format possible. You should validate your blueprint by interviewing your developers to learn what was directly useful and what was not.

A complete and effective blueprint should contain the following essential information that developers require to construct a program:

• A Data Dictionary

• A modular organizational structure

• A narrative to describe the project

• Narratives to describe each module

• Prototype or mock-up screens

• Details of the fields that are referenced in each module

• Details with pseudocode for all operations in each module

A Data Dictionary

The Data Dictionary establishes a clear, unambiguous vocabulary. It uniquely defines each variable in the system. This is much more than a glossary. Nor is it a database schema. It contains variables that may or may not be stored in the database. Items may be stored or calculated. The Data Dictionary will be discussed in greater detail later in the book.⁸

Each Data Dictionary item should include:

• A name that uniquely identifies the item across all project modules

• The type of the item, whether a basic type like string or currency, or a derived type such as phone number or Social Security Number

• If importing the item from a legacy system:

  • The source of this item

  • Any rules needed to convert values from source to database. The rules should be unambiguous pseudocode or formulas.

• The database location, if any, where this item will be stored. Note that not all items in the dictionary are stored in the database.

• Any additional notes to describe the item

A Modular Organizational Structure

Just as it is not enough to simply list the rooms that will be included in a new house, it is not enough to simply list the various components of the planned system. They must be organized in the blueprint in a manner that reflects how they are related. Object models are great for this purpose, but the structure of the planned system need not necessarily correspond one-to-one with the technical object model used to create it.

A Narrative to Describe the Project

This is, in part or whole, the Vision Document. It provides an overview of the project and puts the various modules into context.

Narratives to Describe Each Module

These can be logical or physical modules. Early in the planning process they may be logical modules organized along major functional boundaries, but later they may be rearranged into physical modules. Examples include major program areas such as Order Entry, Order Processing, and Customer Service.

Prototype or Mock-up Screens

For each module that has a user interface associated, a prototype⁹ should be included. This kind of module can be window, form, dialog, or a printed report. The prototype screen lays out the design of the screen or report. Each data element should be keyed to fields (see the next section) that, in turn, link unambiguously to Data Dictionary items.¹⁰

Details of the Fields That Are Referenced in Each Module

One can begin with a simple listing for fields, but before developers can begin to produce a product, there is certain information they almost always require. If this detail is not obtained during planning, then it must be either obtained during development or guesses must be made.

Note that a field may or may not be displayed to the user. Fields can also be used internally for calculations or intermediate results. Fields are also not necessarily database fields. Likewise, all database fields are not necessarily displayed in the module field list if they don't pay a relevant role. Key fields, for example, are not needed in the Data Dictionary provided they have no real world meaning.¹¹ Fields are linked to a Data Dictionary item, which, in turn, can be mapped to a database field.

Therefore, for each field in the module, the following information should be obtained as early as possible for efficiency and accuracy:

• The label that is used to describe the field on the form or report

• The unique item that is the Data Dictionary Item to which this field is mapped

• A key to identify the field on the associated prototype or mock-up

• The position indicating the relative position of the field on the form or report

• The format of how the field should be displayed

• If the module is an editable form or dialog as opposed to a printed report:

  • The control used or proposed to display the field

  • Whether the field is read-only or has any specific security permissions

  • Whether entry is required for the field

  • Whether the entry may be an empty value

  • The minimum and maximum values allowed, whether number, date, or string

  • Any specific values that the field allows

  • The default value of the field

• Any additional notes to describe the field

Details with Pseudocode for All Operations in Each Module

An operation is a logical procedure of some sort. It can be initiated by a physical button or menu item on a form, or it can be a procedure that takes place within events or as part of internal business logic. Pseudocode is used to unambiguously define the operational logic. Pseudocode in a blueprint should reference only items defined in the Data Dictionary.

• The name of the operation

• The label that describes the field on the form or report on which it appears

• If the operation is activated by the user:

  • The control used to activate the functionality

  • Any security groups who do not have permission for that operation

• Psuedocode to document the required functionality

• Any additional notes to describe the operation

This blueprint represents the organization of doors, windows, and walls of the typical software application. For the most part, with this blueprint in hand, any development team can build the forms, reports, database, modules, and other program structures with little need to reference metadata or seek clarification from the business experts.

For those of you who think in terms of objects, you can equate modules to objects, fields to properties, and operations to methods. In fact, these modules, fields, and operations may very well translate into true objects during architecture and design, but they are kept more abstract so that the concepts are generic across all project planning activities. We will discuss this further when we introduce UML in Chapter 6.

Planning in general and creating an effective blueprint in particular are communications challenges. It is all about translating the business requirements into development specifications as quickly and efficiently as possible. The next chapter will explore the communications issues inherent in software planning.