- Technical Debt and the Flywheel
- The Perils of Jumping in Place
- The Causes of Unsustainable Development
- Summary
This chapter is from the book
This chapter is from the book
This chapter is from the book
The Causes of Unsustainable Development
Why is unsustainable development so common in the software industry? It’s because software development is a complex undertaking. Figure 2-2depicts some of the factors that contribute to the complexity. There are a number of project stresses, which you can’t control, that are trying to push your project toward unsustainability. Your only defense is to employ a number of things that you can control, or at least influence, to counteract the stresses and keep the project on a sustainable path. Get one or more of these wrong and your project is going to start into the death spiral. The project stresses are going to be different for every project, and over time the stresses are bound to change. Therefore, constant diligence is required on your part to keep the balance.
)
Figure 2-2 Software projects are subjected to a number of project stresses that can’t be controlled. These stresses are a constantly changing set of forces that push your project towards unsustainability. You can employ a number of defenses that you can control or influence to try and keep the project on a sustainable path.
Project Stresses
User Requirements
User requirements continually evolve and change. It is virtually impossible to predict what users are going to require more than some small number of weeks to months in the future.
External Dependencies
Today’s software depends on operating systems, libraries, third-party components, etc. Software usually also depends on computer hardware or hardware interfaces (especially if it’s embedded). The steady advance of technology means that changes to these interfaces are a given. Managing these changes is a challenge and requires constant diligence and foresight. A change to any of these might set a software team back for days, weeks, or months as the new version of the operating system, compiler, device driver, libraries, web browser component, etc. is integrated and tested. And for many software teams, especially where it is impossible to dictate the end user’s computing environment, the team must support the old and the new interfaces. This adds to development complexity and to the risk that the software will break in unexpected ways.
Competition
It is relatively easy to set up a software organization today because the barriers to entry are low. All that is needed are some bright people, a few computers, and a connection to the Internet to get started.
Unfortunately, what many companies lack is a workable business plan. They do not have a clear idea of their true market worth because they overestimate or do not know their market size, and they almost always suffer from an inability to adequately sell and market their work. They may be good at getting initial funding and making some initial sales, but most software organizations struggle to stay in business. Their very existence and sometimes more innovative products make it difficult for other companies to thrive.
Disruptive Technologies
The constant advance of technology means there is a need for continual innovation, or at least a need to keep up. Given that products need to meet users’ demands over the long term, it is tempting to think that in every case it makes perfect sense to always do only what your customer wants you to do. In many cases, however, this may actually be a long-term recipe for disaster. Because of the constant advance of technology, it is possible for disruptive technologies and associated ways of doing business to eliminate established businesses, businesses that in most cases have done an excellent job of listening to their customers! [Christensen 2003] [Christensen and Raynor 2003] The problem is that most customers are not visionaries and are restricted by their current problems. Hence, they are really good at asking for 10% more than what they currently have, but if another solution, usually built on newer technology, comes along that offers them 50% more in one increment, chances are they’ll take it.
Disruptive technologies usually start out looking like a toy to established businesses and their customers. Because they do not appear to be a threat at first or require a new way of doing business, the established businesses will ignore them right up until the time when their customers realize the advantages of the new offerings and decide to change. By then, of course, it’s too late for the established business.
The Internet is an example of a disruptive technology. The Internet is a godsend in terms of providing a ubiquitous communication medium. Software and related services such as support and consulting can cheaply and easily be offered over the Internet. All businesses today must learn how to wisely use the Internet. They must also learn how to deal with the threats of the Internet, because it also provides an easily accessible means for hackers to exploit an internal network, for illicit software and serial numbers to be distributed, and also a forum for malicious people to exploit security flaws in software and distribute those exploits to unsuspecting users.
Disruptive Business Models
Equally disruptive to most organizations is a disruptive business model. Dell Computers is an example of this type of disruption. Dell does not innovate technologically; their innovation is the ability to produce computers on demand by clustering their suppliers together and taking all orders over the Internet. This gives them the ability to compete on price and responsiveness. Established PC manufacturers such as HP, Compaq, and IBM have not been able to keep pace or duplicate Dell’s business.
Another example of a disruptive business model is open source software. The open source community is dedicated to producing software for free. Although virtually all of this software is only usable by highly technical people (typically other programmers) today, there is no question that open source will play an increasingly disruptive role in the software industry as usability plays an increasingly important role in open source projects. Microsoft has a well-established monopoly on the desktop market that is now being eroded in some sectors, such as in governments, because Microsoft cannot compete against free software. However, what remains to be seen with open source projects such as Linux is if the business model is sustainable, whether the open source community can keep itself going almost exclusively through support and services as claimed.
I believe it is important for all software developers and organizations to embrace open source. As I will argue later in this book, it is important that the software community stop repeatedly implementing commodity software components so that everyone can concentrate on their unique value. Open source is an ideal mechanism to make this happen. In addition, I believe the tools used for software development have not kept pace with the demands of modern software development. Many organizations and teams put a great deal of effort into building their own infrastructure (custom-built tools, bug tracking, and configuration management systems, etc.) when this type of infrastructure should be shared.
Cost Management
Organizations need to carefully manage how much they spend on soft- ware projects. Because of the complexity of software development, it is hard to control costs through greater efficiency. Achieving high efficiency is difficult because of factors such as the mindset of the organization, its processes, training (which does not emphasize efficiency), deadlines, etc. Hence, the easiest way to reduce costs is to employ cheaper people. This explains the increasing use of outsourcing of software development to countries such as India and China where labor is cheaper. But outsourcing is the easy way out. More people in the industry need to recognize the need for efficiency, because efficiency is at the heart of sustainable de- velopment.
Project Controls
Collaboration
Working effectively with other people in your organization and your customers is a crucial aspect of software development. If people are off doing their own thing, then the project as a whole suffers. Good things happen when people can cooperatively work together toward a common objective.
Methodology
The development methodology, the principles (mindset) and practices (project planning, design, testing, coding, etc.) you use, has a dramatic impact on the success of a project. Methodology directly influences efficiency, change tolerance, responsiveness, and software quality.
As described in the Introduction, an important factor is that we use the wrong metaphor for software development (that software development is like building a building). Because the metaphor is incorrect, teams either approach software development using plan-driven development, because that is what is taught to them in school as the approach used by engineers, or,if that doesn’t work, they use variants of an ad-hoc code-then-fix approach.
Software teams who approach software development with little or no discipline commonly employ code-then-fix. Sometimes this is a reaction to practices that do not work for their project, and sometimes it is a lack of focus on technical excellence. These teams spend most of their time writing software and very little time on planning and analysis. Software developed this way is fragile and brittle and will resist change because even if good coding practices are followed, the lack of a set of sound practices will lead to a great deal of dead code and thrashing.
Plan-driven development is derived from engineering as it is applied to the design and construction of bridges and buildings. It is best typified by the ISO-9000 set of standards. The plan-driven approach is what is taught in schools to software developers because there is the mistaken belief that all the complexities in software and the target ecosystem can be understood before any code is written. This is rarely true over the short term and definitely false over the longer term.
Much of the emphasis in plan-driven development is on producing documentation of requirements, designs, test plans, etc. While documentation has value as an important means of communication, the main point is missed: that the real value comes from the process you have to go through to produce the documentation, not from the documentation itself. That is, having to work through a design with your peers is a valuable exercise because it fosters collaboration, understanding, and creativity; producing the document that describes the results of the discussion is less valuable, and the actual document is of even lower value because it must be maintained. By stressing the need to create documentation, plan-driven approaches achieve a mixed result, where on the positive side, stressing documents means that analysis work is required, but the result is too often that the effort spent producing and maintaining the documents detracts from the amount of time spent on analysis. Also, the emphasis on documentation has led many people to believe that the existence of the documents equates with discipline, when in fact discipline should be viewed as ensuring that the analysis work happens in the most efficient manner possible.
Plan-driven development also does not work for software development because it is impossible to put together a plan when the project starts that take into account all the changes in requirements and in the ecosystem that will occur over the course of the project. Also, plan-driven approaches stifle agility because they emphasize ceremony and documentation over having a working product in the erroneous belief that somehow having a pile of documentation describing requirements and design is proof that the project cannot fail, that a disciplined approach is being used, and that by extension any project that does not have this depth of documentation is not disciplined. The problem is that the effort put into the ceremony and documentation is wasted effort; many projects only wind up producing documentation when what they really needed was to get a working product into their user’s hands and get feedback from them since that is the only way to really know if the desired system meets user needs.
Expertise
Expertise with software development and with the project’s ecosystem is a strong indicator of success. Expertise allows teams to apply an increasing amount of intuition and to be able to understand what is important and what is not.
Decision Making
Your ability to make decisions in a complex and constantly evolving ecosystem, often with incomplete information, is crucial to success. An important aspect of decision making is prioritization, where you choose what must be worked on and when.
Leadership
Leadership helps set and communicate a vision, strategy, and tactics. Leadership helps to keep projects on track.
Culture
The culture of an organization shapes the attitudes and priorities of the people who work in it. Culture defines the requirements for leaders, how decision making is performed (and by whom), and the methodology. If a new methodology or leadership or decision-making style is required, then the culture must change.
Simplicity and Reliability
The vast majority of software today is overly complex, hard to use and learn, and has too many features for most of its users. Even commonplace applications like word processors are packed with features that most users never use. For example, one study showed that although the Office97 version of Microsoft Word had 265 functions, only 12 functions were used by more than 75% of users, and 42 were not used at all! [McGrenere 2000]
There have not been enough studies done to prove any ill effects of unnecessary complexity on users. However, I believe this is a significant issue as computing struggles to become more widely adopted, not to mention provide the productivity gains that should come through automation. Complexity is behind at least part of the outsourcing trend because one way to manage complexity is to get someone else to do it for you. Complexity is also why my wife asked me to return a recent Christmas present: I bought some software that claims to help process and manage digital pictures, yet after one look, she said the new software undeniably had some useful features but overall it would mean she would spend more time processing pictures than before.
When it comes to computing and simplicity, there is a huge disparity between what consumers want and what companies would like to sell them [Economist 2004]. Too many software organizations still compete on features and price alone. Unfortunately, they are often encouraged to do so by the most technical users, who are the early adopters of any technology, and by the media, who still publish reviews that contain feature comparison charts.
Related to simplicity is the need for reliability. Software is notoriously unreliable. Many users refuse to buy the first version of any software product, and users are frequently frustrated by version upgrades that break or remove features they relied upon in the previous version. Too many software organizations emphasize features over reliability.
There is not enough emphasis placed on simplicity and reliability. And yet, all one needs to do is look at Apple’s iPod or a Palm PDA. These are simple and reliable devices that are popular because they do a few things very well. What is impressive about them is that a great deal of restraint was shown in their development. It’s easy to add features but very hard to show restraint to keep them out. Is there a future where this type of thinking is applied to software?