- Principle #1: Begin with the Decision in Mind
- Principle #2: Be Transparent and Agile
- Principle #3: Be Predictive, Not Reactive
- Principle #4: Test, Learn, and Continuously Improve
- Summary
This chapter is from the book
This chapter is from the book
This chapter is from the book
Principle #4: Test, Learn, and Continuously Improve
Most information systems have a single approach to handling any decisions that have been embedded in them. Every transaction is treated the same way, with possible alternative approaches largely eliminated during design to find the “best” approach. Once this singular approach has been implemented, information systems continue to work the way they were originally designed until someone explicitly re-codes them to behave differently. The only way these systems are changed is when an external agent—a human—decides that a change is required. These systems also accumulate large amounts of data about customers, products, and other aspects of the business. This data might show that certain actions are more effective than others, but the system will continue with its programmed behavior regardless—every customer is treated like the first.
This approach is not an effective way to develop Decision Management Systems. When we make decisions about our own lives or interactions, we often assess a large amount of data, either explicitly or implicitly. We learn from this data what is likely to work or not work—the data accumulated provides clues to how an effective decision can be made. A Decision Management System cannot afford to ignore the accumulated historical data.
Decisions involve making a selection from a range of alternative actions and then taking the selected action. It is often not immediately obvious if the decision was made effectively. Some decisions have a significant time to outcome, and no assessment of the effectiveness of the decision will be possible until that time has passed. For instance, an early intervention designed to ensure a customer renews her annual contract cannot be assessed until the customer reaches the renewal point, perhaps many months later. If the action taken turns out to be ineffective, then a different approach will need to be considered. A Decision Management System cannot afford to “single thread” this analysis by only testing one decision making approach at a time.
Whether a decision is a good one or a bad one is a moving target. A decision may be made to discount a particular order for a customer that may be competitive today but much less so tomorrow because a competitor has changed their pricing. As markets, competitors, and consumer behavior shift, they affect the effectiveness of a decision. This constant change in the definition of an effective decision means that Decision Management Systems must optimize their behavior over time, continuously refining and improving how they act.
Decision Management Systems must therefore test, learn, and continuously improve. The analysis and changes may be done by human observers of the Decision Management System or by the system itself in a more automated fashion. Decision Management Systems must collect data about the effectiveness of decision making. They must use this data, and other data collected by traditional information systems, to refine and improve their decision-making approach. Decision Management Systems must allow multiple potential decision-making approaches to be tried simultaneously. These are continually compared to see which ones work and which ones do not. Successful ones persist and evolve, unsuccessful ones are jettisoned. Finally, Decision Management Systems must be built on the basis that their behavior will change and improve over time. Decision Management Systems will not be perfect when implemented but will optimize themselves as time passes.
Collect and Use Information to Improve
The first way Decision Management Systems must learn is through collecting and then using information about the decisions they make. When a Decision Management System makes a decision, it should record what decision it made, as well as how and why it made the decision it did. This decision performance information will allow the long-term effectiveness of a decision to be assessed as it can be integrated with the organization’s performance metrics to see which decisions result in which positive, or negative, performance outcomes. This information allows good decisions to be differentiated from bad ones, better ones from worse ones. It is often said that if you wish to improve something, you must first measure it. Decisions are not an exception to this rule.
Information about the decisions made can and should be combined with the information used to make the decision. This information might be about a customer, a product, a claim, or other transaction. This is the information that is passed to the Decision Management System so that it can make a decision. Combining this information with the decision performance information will identify differences in performance that are caused by differences in the information used to drive the decision. For instance, a decision-making approach may work well for customers with income below a certain level and poorly for those above it. Storing, integrating, analyzing, and using this data to improve decision-making is the first building block in building Decision Management Systems that continuously improve.
Support Experimentation (Test and Learn)
When a Decision Management System is being defined, it may not be clear what approach will result in the best outcomes for the organization. Several alternative approaches might all be valid candidates for “best approach.” Simulation and modeling of these approaches, and testing them against historical data, might show which approach is most likely to be superior. Even if the historical data points to a clear winner, the approach is going to be used against new data and may not perform as well in these circumstances.
A Decision Management System, therefore, needs to be able to run experiments, choosing between multiple defined approaches for real transactions. The approach used for each transaction can be recorded, and this information will allow the approaches to be compared to see which is superior. This comparison may not be definitive, and one approach may be better for some segments of a customer base, while a second works better for other segments. Results from these experiments can then be used to update the Decision Management System with the most successful approach or combination of approaches. Because Decision Management Systems handle repeatable decisions, there will always be more decisions to be made that will be able to take advantage of this improved approach.
Optimize Over Time
In a static world, one round of experimentation might be enough to find the best approach. A set of experiments could be conducted and the most effective approach selected. As long as nothing changes, this approach will continue to be most effective. However, the effectiveness of a decision-making approach can vary over time for many reasons, and you have little or no control over this. The old “best” approach may degrade suddenly or gradually, and when it does, you will need to have alternatives. Even when experimentation finds a clear winner, a Decision Management System needs to keep experimenting to see whether any of the alternative approaches have begun to outperform the previous winner. Alternatives approaches could be those rejected as inferior initially or new ones developed specifically to see whether a new approach would be superior in the changing circumstances. The effect of this continuous and never-ending experimentation is to optimize results over time by continually refining and improving decision-making approaches.