This chapter is from the book
This chapter is from the book
This chapter is from the book
Applying Stability and Throughput
The correlation of good scores in these measures with high-quality results is important. It offers us an opportunity to use them to evaluate changes to our process, organization, culture, or technology.
Imagine, for example, that we are concerned with the quality of our software. How could we improve it? We could decide to make a change to our process. Let us add a change approval board (CAB).
Clearly the addition of extra review and sign-offs are going to adversely impact on throughput, and such changes will inevitably slow down the process. However, do they increase stability?
For this particular example the data is in. Perhaps surprisingly, change approval boards don’t improve stability. However, the slowing down of the process does impact stability adversely.
We found that external approvals were negatively correlated with lead-time, deployment frequency, and restore-time, and had no correlation with change fail rate. In short, approval by an external body (such as a manager or CAB) simply doesn’t work to increase the stability of production systems, measured by time to restore service and change fail rate. However, it certainly slows things down. It is, in fact, worse than having no change approval process at all.5
My real point here is not to poke fun at change approval boards, but rather to show the importance of making decisions based on evidence rather than guesswork.
It is not obvious that CABs are a bad idea. They sound sensible, and in reality that is how many, probably most, organizations try to manage quality. The trouble is that it doesn’t work.
Without effective measurement, we can’t tell that it doesn’t work; we can only make guesses.
If we are to start applying a more evidence-based, scientifically rational approach to decision-making, you shouldn’t take my word, or the word of Forsgren and her co-authors, on this or anything else.
Instead, you could make this measurement for yourself, in your team. Measure the throughput and stability of your existing approach, whatever that may be. Make a change, whatever that may be. Does the change move the dial on either of these measures?
You can read more about this correlative model in the excellent Accelerate book. It describes the approach to measurement and the model that is evolving as research continues. My point here is not to duplicate those ideas, but to point out the important, maybe even profound, impact that this should have on our industry. We finally have a useful measuring stick.
We can use this model of stability and throughput to measure the effect of any change.
We can see the impact of changes in organization, process, culture, and technology. “If I adopt this new language, does it increase my throughput or stability?”
We can also use these measures to evaluate different parts of our process. “If I have a significant amount of manual testing, it is certainly going to be slower than automated testing, but does it improve stability?”
We still have to think carefully. We need to consider the meaning of the results. What does it mean if something reduces throughput but increases stability?
Nevertheless, having meaningful measures that allow us to evaluate actions is important, even vital, to taking a more evidence-based approach to decision-making.