- Dealing with the "Fog of Beforehand" One Step at a Time
- Facing the Unknown in Software Development
- How the Agile Manifesto Covers Common Anti-Patterns for Dealing with Friction
- Key Takeaways
This chapter is from the book
This chapter is from the book
This chapter is from the book
Facing the Unknown in Software Development
If you are familiar with software development, you probably have experienced the following situations:
Release dates of products and features are delayed significantly, and sometimes multiple times.
Initial plans need complete rework due to mistakes, assumptions, and new insights gained by doing the work.
Customers are unhappy when viewing the product for the first time, as it was not what they expected.
In my experience, there are two primary reasons why these kinds of situations happen:
Building software products is predominantly an exercise in dealing with uncertainty and complexity. The inherent uncertainty and complexity make it foolish to stick to the plans we came up with before doing any of the work.
We struggle when we must deal with uncertainty and complexity. We try to apply our usual and familiar approaches of spending more time on planning and analysis to gain control. But those methods are ineffective and often just make things worse.
Our typical response when we encounter uncertainty and complexity is to concentrate on more planning and analysis. We start with turning inward, focusing on what we know, because we believe that is where the solution lies. Common examples of this ineffective behavior are spending many hours in Sprint Planning or endlessly revisiting the same Product Backlog Item at refinement.
But is what we know what matters most before starting the work? Or is it what we don’t know and what we believe we know—often erroneously—that matter most? And if our wrong assumptions and what we don’t know carry substantial weight, will we gain the needed knowledge by talking in meeting rooms and going in circles, limited by the knowledge we can have before starting the work?
Why are detailed planning and spending more time on analysis ineffective, and why do they frequently make matters worse? Let me tell you a story about a deceptively simple software problem that turned out to be too complex to manage using traditional project management methods. This story illustrates how software development is often similar to finding your way back in Vlieland, with every step of the way shaping the subsequent path to follow.
Locked in by Four Digits
“You took the shortest route to the exit.” Project Manager Willem-Jan Ageling was given this grim message by his colleagues when they reassigned him to a crucial project within the organization. He was only a few months in at his new company, a successful payment processing company in the Netherlands. The problem Willem-Jan faced was deceptively simple.
Willem-Jan was tasked with increasing the length of the client identification number. Up to now, the client number had a maximum of four digits, and the company was nearing the limit of 9999 customers. If it didn’t fix this problem before the number of customers reached the limit, then it wouldn’t be able to onboard any new customers.
The decision to create a four-digit client number had been made 20 years earlier. The original developers who had made that decision were no longer working at the company. They probably thought that by the time they reached the limit of those four digits, the company would be so flush with money that fixing it would not pose any problem. If only they had known how much stress and how many headaches their decision would cause 20 years later!
Willem-Jan’s assignment was straightforward: expand the client number so the business could continue growing without bumping into the artificial ceiling imposed by the four-digit limit. He managed the project using traditional project management techniques that he had mastered over many years of working as a project manager.
Once the project started, things were going swell until the project team entered uncharted territory: the replication of customer data from a legacy database to a new database. No one had done this before, and the necessary steps to make it a success were entirely unclear.
Dealing with the legacy database was challenging. Whenever the project team believed they had found a solution, they bumped into a novel problem. The project team members were on a continuous journey of discovery, similar to my journey on the island of Vlieland. Every step of the way shaped the next steps to take—except they could not change their direction as quickly as we kids in Vlieland, because they needed first to run every change of the plan by the Change Advisory Board (CAB).
Each time the project team discovered something they didn’t know, it meant issuing a change proposal to the CAB. Presenting changes to the CAB meant losing face and explaining why their plans—and by extension, Willem-Jan himself—had failed. Willem-Jan was the unlucky person who had to face the wrath of the CAB whenever this happened.
Willem-Jan became a regular visitor at the CAB meetings, and the ordeal began to annoy him. Constantly being grilled about changes to the plan that were beyond anyone’s control became exceedingly frustrating. He was okay with returning to ask for more changes to the plan but realized all those changes were inevitable. He understood there was nothing he could do to prevent his frequent appearances before the CAB. No matter how well he did his job, he would never be able to make perfect plans that could circumvent all the unknowable obstacles the team faced. He could deal with these obstacles only after the team’s actions revealed their existence.
Based on the current information, which he knew was inadequate and limited, Willem-Jan thought the project would take many more months than initially anticipated. Usually, this would be the moment to pull the plug on the project, but for this project, failure was not an option. Its success was crucial for the future of the company.
In the end, Willem-Jan decided to face the CAB head-on. He gave them a presentation explaining all the uncertainties, risks, and things they would be doing for the first time. He made the CAB aware of the significant and pervasive fog surrounding the work. During his presentation, the extreme uncertainty and complexity of the initiative became evident to everyone. The current situation was so uncertain and rife with so many unknowns that creating an accurate long-term plan based on what the project team knew was impossible.
The CAB gave Willem-Jan the green light to drop the long-term planning. In turn, he switched to making short-term weekly plans—planning based only on what the team knew and not on what they believed they knew. Then, the project team made a new plan every week based on what was discovered and learned. Instead of doing overconfident planning, they switched to doing humble planning—planning based solely on what they knew and were able to foresee.
By dropping the overconfident long-term planning and switching to humble planning, Willem-Jan’s plans became rooted in reality. The plans considered the presence of the fog of beforehand and intentionally limited the fog of speculation by planning based only on what was known. By performing the work and exposing the reality, the planning was adjusted based on what the team discovered.
Ultimately, the project was a big success, despite all the risk, uncertainty, and things the team was doing for the first time. Sometimes there was good progress, and occasionally new information was discovered that disrupted the playing field. But since everyone was aware of the uncertain nature of the situation, Willem-Jan was no longer being grilled by the CAB to explain every change. He could apply his energy and focus to those parts of the project actually under his control.
As mentioned in the opening paragraph of this section, Willem-Jan was told by his colleagues that taking on this project would most likely lead to his departure from the company. And it was, indeed, Willem-Jan Ageling’s final project as a project manager at the company—not because he got fired as his colleagues predicted, but because the project was a great success and opened Willem-Jan’s eyes to a different way of working. After the project, Willem-Jan decided to stop working as a project manager and continue his career as a Scrum Master.
As this story shows, it was only after the CAB embraced uncertainty and complexity that the project got on a better path toward success. The project was managed by responding to changes as they presented themselves and swiftly overcoming obstacles instead of trying to fight uncertainty and complexity through more planning and better analysis. The barriers for making changes to the plan were removed by dropping the CAB’s change approval process.
In theory, extending a client number would usually be categorized as trivial and easy work. But in practice, even seemingly simple changes sometimes can be a herculean undertaking where the team is clueless about how to pull it off. Willem-Jan’s story is a beautiful example of being unable to come up with a definitive plan without first doing the work, despite the deceptive simplicity of the problem at hand.
Now that we’ve linked the Vlieland “De Dropping” story to the realm of software development, let’s turn to the history of military warfare. There’s a lot software developers can learn from the military’s approach to solving these kinds of planning problems.
How Excessive Planning and Preparation Can Lead to Defeat
On October 14, 1806, the Prussian army suffered a crushing defeat by the French army in the twin battles of Jena and Auerstedt. When the battle started, the Prussian forces vastly exceeded the French troops. At Auerstedt, the Prussians outnumbered the French with a ratio greater than 2:1. Despite superior numbers, the Prussians were unable to make a strong stand against the French.
In preparation for the encounter, the Prussians drafted, discussed, and contemplated five different plans to best their French opponents, but they struggled to come up with an unified plan of battle. If more time spent preparing and planning won every battle, then surely the Prussians should have emerged victorious.
While the Prussians were busy pondering and reflecting on their best course of action, the nimble French army, led by Napoleon, seized the initiative. All that time spent planning and deliberating buried the Prussian infantry with instructions and information. The inevitable result: confusion and inaction among the Prussian soldiers. The bloated, elaborate Prussian plans weighed their troops down and became nothing more than delayed responses to Napoleon’s swift, unforeseen decisions and movements. The battle was lost before it had even started. At the end of the battle, at least three Prussians lost their lives for every French casualty.
What on paper should have been a straightforward victory turned into a bitter, unexpected defeat that shook the Prussian army’s organizational structure to its core. The Prussians who survived the battle realized the ineffectiveness of their approach and the need for significant reforms.
One of the Prussians who survived the battle of Jena–Auerstedt was a young Carl von Clausewitz, quoted at the beginning of this chapter: “The enemy of a good plan is the dream of a perfect plan.” Everybody knows too little planning is ineffective. But von Clausewitz’s quote perfectly captures how too much planning can also be futile. He witnessed firsthand the failings of excessive analysis, communication, and planning that resulted in the Prussian downfall and led to his incarceration as a prisoner of war in France.
von Clausewitz would later play a crucial role in successfully transforming the Prussian army. And after the Prussians became part of the German army, their novel military doctrine in dealing with uncertainty and complexity became one of the most effective in the world. When the French and the Germans clashed again in the Franco-Prussian war in 1870, the tables had turned. The German army crushed the French troops decisively. What had changed in the German army to make this victory possible?
What Can We Learn from the German Army?
Before I elaborate on the revolutionary changes in the German army, please note that it isn’t my intent to glorify war in any fashion. I compare building software products to warfare because the complex work of building a product shares the same challenges you encounter during a battle. You never have enough information, time, or understanding to produce a perfect plan before the fight starts.
But that is where the comparison stops. It would be nonsensical and disrespectful to suggest building software products entails anything similar to the horrors and tragedy of war. In battle, everyone is fighting for their lives and trying to outwit the opponent. War is messy, uncertain, cruel, and chaotic. Rarely anything pans out as initially predicted. Because of what is at stake, successful execution of strategy and plans on the battlefield is the ultimate exercise in dealing with uncertainty and complexity.
General von Clausewitz invented the concept of friction to explain why it is so difficult for plans to be executed successfully and to produce the desired results on the battlefield. Here’s a quote from von Clausewitz that perfectly captures how friction can throw sand in your gears when you’re tackling even the simplest of tasks:
Everything in war is very simple, but the simplest thing is difficult. Friction, as we choose to call it, is the force that makes the apparently easy so difficult. Friction distinguishes real war from war on paper.
von Clausewitz argues that a central characteristic of friction is that it is difficult to predict how it will manifest itself because it is the sum of an infinity of petty circumstances:
This tremendous friction, which cannot, as in mechanics, be reduced to a few points, is everywhere in contact with chance, and brings about effects that cannot be measured, just because they are largely due to chance. One, for example, is the weather. Fog can prevent the enemy from being seen in time, a gun from firing when it should, a report from reaching the commanding officer. Rain can prevent a battalion from arriving, make another late by keeping it not three but eight hours on the march, ruin a cavalry charge by bogging down the horses in mud, etc.
Just a little bit of fog can bring about all these unanticipated effects. But you don’t even need a physical fog for friction to occur. Friction occurs when independent minds try to realize a common goal in a rapidly changing, complex environment. In such an environment, we never have perfect information at our disposal. And even if we do, people will process that information differently and may arrive at different conclusions. Friction causes our plans to be imperfect and our execution flawed. It explains our inability to predict the results of our actions and plans.
Friction is ultimately rooted in human finitude. Humans are limited by their comprehension, processing speed, and individual biases, and they are affected by their environment, emotions, stress, and personal interests. All of these many factors rooted in human finitude can combine to produce significant amounts of friction.
Humans have limited knowledge and act as independent agents according to their own will. They are affected by unpredictable events and have imperfect information at their disposal. Even with the proper understanding, inadequate information transfer may produce confusion. Even if we transfer information perfectly, individuals may have different interpretations, agendas, and priorities that create misunderstandings and generate noise. The unpredictability of the environment and the complexity may also work together to produce noise, create chance effects, and make it difficult to obtain data.
In short, personal interests, uncertainty, complexity, and the combination of emotions and stress work together to produce friction. The situation and the context determine the extent to which friction plays a role and affects our ability to make the right decisions.
I define friction as follows: Friction is anything that increases the chance of surprises happening.
The more friction we experience, the more frequently we will encounter surprises. The famous mathematician and computer scientist Claude Shannon established the link between information and surprise. The more information an event contains, the more surprising the event is. Discovering more surprises means we’re unearthing more information we have to incorporate in our plans and actions to produce the desired results.
In the eye-opening book The Art of Action, military historian and consultant Stephen Bungay investigated the lessons that we can draw from the German army and ways that we can use this understanding to overcome friction in the business world. Bungay argues we can apply the same military principles of dealing with uncertainty and complexity to deliver better results in the realm of business. He identified three primary gaps amplified by friction that can prevent your plans from achieving success (Figure 1.1):
Knowledge gap: The difference between what we’d like to know and what we actually know.
Alignment gap: The difference between what we want people to do and what they actually do.
Effects gap: The difference between what we expect our actions to achieve and what they actually achieve.
)
Figure 1.1 The three gaps that are increased by friction and increase the number of surprises we can expect to encounter. (From Stephen Bungay, The Art of Action, Nicholas Brealey Publishing, 2010)
The ideal response to these sources of uncertainty and complexity does not come naturally to us. The main problem is that the default response we fall back on to tackle these three gaps just makes matters worse. The usual approach creates its own mist on top of the natural fog that already exists. The fog of beforehand is worsened even more by injecting the fog of speculation.
When there is a knowledge gap, we know less than we would like to know. We usually react by spending more time planning, analyzing, and debating ways to close the gap. The problem with this approach is that when you lack information to make an adequate plan, no amount of planning or analysis can conjure new information on the unknown out of thin air.
Acting based on conclusions drawn from noise, instead of the signal, is risky and dangerous. The signal is the meaningful information we’re trying to detect and act upon. Noise, like trying to predict the weather based on the number of garden gnome statues that populate our lawn, does not contain information you can use to make better decisions.
The more we over-analyze, the more noise becomes entrenched in our knowledge and plans. As a result, what we believe we know finds its way into our plans, resulting in additional fog that clouds and slows our judgment—the fog of speculation. The fog of speculation is dangerous because it often makes us believe we know more than we actually do. When we believe the noise is true, then we believe we are no longer speculating. The danger is that we may mistake the noise for reality. Speculation can conceal what we do not know, making it even more challenging to adjust our plans.
To make things worse, we also get weighed down by the heavy anchor of over-contemplation. The unwieldy burden of overconfident planning stifles our ability to respond if reality turns out to be quite different than it was initially envisioned in our plans. It is difficult for people to abandon carefully devised plans when reality makes them obsolete. After all, you have invested so much time and cognitive effort in making those brilliant plans. How can they be wrong? The Prussians who were taken by surprise by the French at Jena–Auerstedt could tell you a thing or two about that.
When there is an alignment gap, people do something different than what they were told to do. Even when people know exactly what they should do, they still sometimes make mistakes, especially in stressful situations. The usual response to this problem is to issue more instructions and spend more time on elaboration. If people do something other than what we expected, we need to give better instructions and issue those instructions more frequently—also known as micro-management.
More and better instructions are like receiving a thick instruction manual when we purchase a new product. Who bothers to read all the pages in the manual? And even if we do, we will not understand or remember everything in there. Our situation and context can change so rapidly that the effort required to dig through a large body of knowledge will make it impossible to respond adequately and effectively. And more often than not, the instructions will not cover the exact situation in front of us, as it could not be predicted upfront.
When there is an effects gap, our actions produce different results than we initially expected. The usual response is to impose metrics and tighter controls on our teams to ensure they deliver the desired results next time around. Implementing more controls bakes in constraints and limitations on how teams can respond. Adhering to the controls often becomes the goal, preventing the team from responding swiftly and in the best way.
As customers, we are all too familiar with the customer service agent following controls and procedures instead of trying to help us out. The customer service agent cannot help us because there are rules in place they must follow. Following the rules becomes more important than achieving the optimal customer outcome.
In short, when we experience high friction, we will undoubtedly encounter many surprises in our plans, execution, and results. Our usual response is to spend more time planning, issuing instructions, or imposing tight controls, which makes all three gaps even worse.
These anti-patterns to friction are so common that they even take a central place in the value statements of the Agile Manifesto.