Project Planning for Successful Software Development
- The Pathology of Poor Planning
- 2 The Vision Document
- 3 The Functional Specification
- 4 Recognizing a Bad Plan
- 5 Excellence Is Attainable
- 6 A Software Blueprint
This chapter is from the book
This chapter is from the book
This chapter is from the book
We've got big trouble in Software City. In the famous musical The Music Man, a fast-talking salesman comes to a small town named River City, Iowa. He tells the gullible folks that they "got big trouble." He warns them that the new pool hall will induce their children to fall prey to bad habits. The only thing that will save them, he declares, is to provide a wholesome alternative, such as a boys' marching band. Of course, it just so happens that the salesman deals in a fine line of quality band uniforms and instruments.
Like The Music Man, I'm here to tell you that we got big trouble in Software City. But unlike The Music Man, I'm not saying that just to peddle a textbook equivalent of 76 trombones and 110 cornets. While some authors applaud the state of the industry, I have serious concerns. While there are particular teams that do experience success on certain projects, such successes are not the norm.
Sources such as objective statistics, self-grading, client reaction, and anecdotal experience all confirm that the rate of failed software development projects is embarrassingly high. Failure rates from 80 to 90 percent are commonly cited and generally accepted. Of course, the accuracy of these figures depends on how success or failure is measured, but by any measure, the situation is not good. The need to improve our software development processes is very real.
To make the picture worse, the cited failure rates reflect only projects that overtly failed to meet objectives. They do not include "successful" projects that cost far more than they might have, or took far longer to complete than was necessary. I disagree with authors who claim that failure rates are artificially high because they are based on unrealistic expectations (DeMarco, 1995). On the contrary, the expectations for minimal success across the industry are far too low. Furthermore, success is most often measured by short-term deliverables, not by long-term maintainability. If failure to maintain were also considered, the picture would look even more dismal.
Poor performance is a pervasive and recurring problem that affects the credibility, effectiveness, and long-term survival of the software industry. Despite an undeniable need, fewer potential clients are willing to risk precious capital on custom development projects. They have either heard of, or they have experienced first hand, the development nightmares that all too commonly occur in these projects. As of this writing, the market for custom application development continues to diminish. I don't believe this is due to a lack of need, but rather to a lack of faith in our industry to deliver.
Clients are not the only ones who are disenchanted. Developers are perhaps even more frustrated. Talented developers don't want to waste their talent over and over again on doomed projects. New developers get disillusioned with the business and find other work. Managers burn out from having to fight the same client relationship fires all day long. Owners of development shops sometimes just give up application development and change their business to provide more profitable and less problematic software services.
We face the real possibility that the fledgling software industry will collapse into a much smaller number of players producing mass-market solutions. It may never mature into the lucrative and diverse industry that we would all like it to become.
This should not, and need not, be the future of our industry.1 Custom software development can and should be a thriving, rewarding, and cost-effective industry for both the consumer and for the provider. How is it that an industry that specializes in organizing and streamlining other business practices can itself be so abysmally unorganized and inefficient?
When a problem is identified, our first reaction is to look around for someone or something to blame. Where does the blame lie here? Do we blame the people involved for creating this situation? Have they simply failed to implement the excellent processes and practices that are well-documented in the literature? Many would say yes. However, intelligent, talented, and experienced professionals staff many failed projects. We cannot simply dismiss them all as incompetent. We should search for root causes of the problems, evolve effective countermeasures, and apply them. We should create a continuous process improvement ethic.
Are the "best practices" that we turn to for guidance perhaps not as good as proponents would like us to believe? I am not willing to say that. As I said before, the practices and approaches recommended in the literature are mostly sound, practical, and based on proven experience. We can't minimize their value and effectiveness.
So what is it then? What is out of kilter in the software development industry? In my opinion, the source of the problem is not individuals or any specific practices, but overcomplication. In our conscientious efforts to minimize risk, adopt best practices, meet budget constraints, prevent wasted effort and so on, we introduce overly restrictive and complicated schemes which do not meet the fundamental requirements for productive planning. Our solutions, in part, make worse the very problems we are trying to avoid. We expend far too much time and effort in project planning and management with too little payback.
The key to improved success is to simplify the process. Eliminate unnecessary effort. Expose and eradicate the sources of those problems that force the application of overly complex and counterproductive practices. This is best accomplished by focusing on the production of a simple but effective software blueprint during the project-planning phase.
2.1 The Pathology of Poor Planning
In the human body, certain materials called allergens stimulate allergic responses. The immune system responds by creating antibodies to combat the allergens. As a side effect, the victim can suffer histamine responses like itchiness and a runny nose, or even more severe life-threatening allergic reactions. In a similar fashion, relatively minor problems identified in a software project often stimulate the production of management countermeasures. Too often, these responses cause even worse, and sometimes fatal, reactions in the project.
The allergens in software projects are almost always buggy code and missed deadlines. In effect, they have lower than acceptable quality and productivity. These symptoms may in turn have been caused or aggravated by poor planning. They stimulate typical responses including more elaborate planning and a tightening of restrictions on the developer's ability to improvise. When this becomes a cyclical pattern, projects suffer. Either they become too congested to even breathe, or at the very least they suffer diminished ability to perform at peak capacity.
Software development is uniquely challenging, but it is not rocket science.2 By that I mean that what we do is not that complicated, or at least it should not be. We analyze requirements and translate them into a relatively simple coding language. We're not putting a space shuttle into orbit or producing a blockbuster motion picture. So how can we have so much difficulty? How can we fall prey to the same few common modes of failure over and over again in our industry?
In the human body, the different systems interact intimately. Similarly, planning, development, management, and the client together form the systems of a software project. These systems interact so intimately that any cure must be holistic in nature, considering the response of all systems to stresses or remedies. Each of these areas has a very different perspective on the typical recurring problems we face in software projects.
Management Perspective
From its perspective, management may sometimes recognize that the marketing department overcommitted deliverables. More often, it simply sees that the developers failed to deliver a satisfactory product. Its dissatisfaction may be based upon profitability, budget, timing, functionality, maintainability, usability, or any number of metrics. When this happens, management has limited courses of action available. It can replace or retrain developers if it suspects that the developers lack competence. It can insist upon more planning if it assesses that the planning was lacking. And it can tighten up management controls to ensure that developers follow the plan more closely. Most often, management applies a combination of these remedies in shotgun fashion, hoping to eliminate any problems next time around.
Figure 2-133 Industry insiders are usually the last to recognize and correct their systemic problems.
){kind=link}
Planners' Perspective
From the planners' perspective, we see a slight variation. When projects run into trouble, planners also see a fairly limited number of causes. Planners often feel that marketing doomed planning efforts right from the start by prematurely promising features, approaches, and budget. They feel that the planning process was rushed, inadequately budgeted, and impatiently supported by management. Planners feel that the development group did not fully utilize their planning documentation and that management allowed them to deviate too far from the planned specification. The project planner's remedy for these problems almost always involves more planning next time with greater management control over development.
Developers' Perspective
Developers feel they were handed unrealistic deadlines, constraints, and requirements. They feel they were given a poor project plan that did not meet their needs. They feel neglected and ignored by management when they ask for support to augment or adjust the plan, and then they feel hammered upon when deadlines are missed. Often they feel that management undermined development efforts by understaffing or moving critical staff in midproject. Developers typically want software planning, but they want better, not necessarily more, software planning. They want management, but they seek more responsive and supportive management, not more restrictive, counterproductive management.
Clients' Perspective
From the clients' viewpoint, they just want the software done well, and of course completed as quickly and inexpensively as possible. Poor planning is the first thing they blame if there are problems in the project. They usually feel that the planners did not understand their needs thoroughly enough and did not document adequately. They often complain that although they got what they specified, they did not get what was needed.
Sounds kind of like the interaction between parents and their kids, doesn't it? The parents just want their kids to get an "A" in school. The kids may want that also, but when the kids get only a "C," the parents blame themselves for not taking a more active role. They grimly tighten up and create more restrictions for their kids and impose penalties if the kids deviate. They try to gather more measurable data by requesting more frequent grade reports from the teacher. When increased parental management conflicts with other demands upon them, the kids divert energy into working around those constraints and end up getting a "D" next time. Both sides have taken what they consider to be reasonable action at each step, but the interaction still degenerates into an increasingly counterproductive and hostile relationship.
This same sort of situation occurs over and over again in software development. As a consultant, I was sometimes called upon to save failed projects. Management invariably insisted that the reasons for failure lie within planning and development. If they blame themselves, it is because they didn't manage enough. They vow to be more diligent and proactive about increased project management next time.
The developers want both more and less management at the same time. What they really mean, even if they haven't realized it, is that they want more supportive management.
When you talk to the planners, they lament that they didn't have time to do more software planning and puzzle over why the developers have such problems after all the "good planning" they did do.
When I came into these situations, it was usually only after management had already tried replacing developers several times and had tightened up management practices drastically. It still wasn't working, and probably had gotten worse. In some of these cases, management had become so desperate that it was willing to give me a free hand. In those situations, I was able to succeed dramatically.
In other cases, management insisted that I fix the project while adhering to all of the management controls and requirements it had put in place. I declined those projects. I knew there was no way I could succeed where other fine developers had failed if I was locked into the same underlying causes of failure.
Accordingly, for the software planning techniques in this book to succeed, management may need to "loosen up." That does not necessarily mean managing less, but managing differently, managing smarter. This book is called Planning Smarter, not "Managing Smarter," so it is not specifically about project management. However, smarter planning cannot succeed without smarter management, so it will touch on some management issues and practices.
"Poor management can increase software costs more rapidly than any other factor." (Robertson, 1999)4
Planning smarter can indirectly encourage smarter management. When managers and planners cooperate well, successes are transmitted and reinforced throughout the system.
Is the developer the blameless victim in all this? Certainly not! There are times when developers fail to deliver. Yes, there are hack programmers out there. Between teaching, hiring, and consulting, I have seen some amazingly bad ones. The bad practices that programmers can adopt are truly astounding. However, better planning and tighter project management generally do not even begin to fix coding problems. The only things that can mitigate these problems are training and talented, experienced developers with good programming standards and coding reviews. These are the types of problems and solutions that my first book addressed (Gill, 2000).
Quite often, however, the developers are not the problem, or at least not the only problem. No developer, however talented and conscientious, can develop a project efficiently from a bad plan. The rule of "garbage in, garbage out" is universal. Sometimes, through heroic efforts, developers manage to succeed despite working from a nearly useless project plan. However, in many cases a bad plan all but guarantees development problems. A mediocre programmer may be able to succeed if given a good plan, but he has no chance at all working from a bad plan.
"Attracting the best people is vital, but it is also essential to support them with an effectively managed software process." (Humphrey, 1989)5
Certainly, a bad programmer could mess up the best-laid plan, but it is not common to have a great plan and a poor developer. More frequently, you have the worst case of bad planning followed by bad implementation. In this case, the planners usually don't see the true nature of their complicity. Since the programmer obviously failed to achieve his or her goals, management fails to appreciate that the plan might have made success quite difficult. Managers conclude instead that they simply failed to exert enough control over the developers. They resolve to spend more time and money next time to maintain greater oversight and constraints upon the developers, limiting the ability of good developers to succeed despite the bad plan.
"A lot of software projects fail, but we software developers are not such dummies that our sheer incompetence can account for them all." (DeMarco, 1998)6
It is worth emphasizing again that the one thing that management, planning, and development all normally agree on is that more planning is better. More planning is the solution to all problems. It is the panacea of the software industry. When projects run into trouble, management almost invariably concludes that more planning is needed and tighter controls must be put in place to adhere to it. Managers reason that if they had invested more in planning, the good developers could be more efficient and the less experienced developers would encounter fewer problems.
The flaw in this is that it assumes that planning is always good. This mistaken assumption leads to the erroneous conclusion that more planning must be better. The reality is that planning is often quite bad. It can be as appallingly bad as the worst coding samples. When you try to fix the problems caused by bad planning with even more bad planning, the problems compound.
When a project has been poorly planned, it is usually no secret. Everyone, at least all the developers, knows it very early in the project and becomes quickly resigned to yet another Death March even before the ink has dried on the functional specification.
"A Death March is a project that is doomed from the start. Everyone knows it and yet everyone just resigns to march along dutifully to certain doom." (Yourdon, 1999)7
It can quickly reach the point at which no planning may be better than all that bad planning. No matter how skilled the developers may be, the quality of the plan they receive is critical to their success. Typical software planning often fails to provide the developers with what they need to meet expectations. That may seem like a harsh assessment. After all, the industry has almost universally agreed upon the merits of current planning tools and techniques. However, it is also commonly accepted that the majority of software projects fail to meet their goals.
To explain this contradiction, most software professionals would conclude that their peers simply fail to apply well-known and proven planning principles. I am not willing to accept that explanation. My follow-up question is, why do they fail to apply them?
Walk into any software shop and you hear almost everyone there talk about project planning. Everyone appreciates the importance of doing it effectively. Many attempt to apply standard planning methodologies to their projects. They usually fail.
Some talk about it incessantly and never do it. Some read all the literature and continually bemoan the fact that their shop doesn't implement this or that practice. Some produce incredibly complex planning documents that don't seem to provide much benefit.
In my mind, if it isn't getting done well, you must simplify the process rather than make it larger and more complicated. Threatening, educating, and licensing people into compliance isn't going to have much real benefit if the practices are simply too complex or cumbersome to apply effectively. To eliminate the pathology of poor planning, we must simplify project requirements to the essential elements required to specify the desired software. Then we must put processes in place to obtain that information in the most efficient and direct way possible. The first example of a well-known document that is typically overcomplicated and overinflated is the famous Vision Document.