- Agile Testing Quadrants
- Challenging the Quadrants
- Using Other Influences for Planning
- Planning for Test Automation
- Summary
This chapter is from the book
This chapter is from the book
This chapter is from the book
Challenging the Quadrants
Over the years, many people have challenged the validity of the Quadrants or adjusted them slightly to be more meaningful to them. We decided to share a couple of these stories because we think it is valuable to continuously challenge what we “know” to be true. That is how we learn and evolve to improve and meet changing demands.
Gojko’s Challenge to the Quadrants
- Gojko Adzic, an author and strategic software delivery consultant, challenges the validity of the Quadrants in the current software delivery era.
The agile testing quadrants model is probably the one thing that everyone remembers about the original Agile Testing book. It was an incredibly useful thinking tool for the software delivery world then—2008. It helped me facilitate many useful discussions on the big picture missing from typical programmers’ view of quality, and it helped many testers figure out what to focus on. The world now, as of 2014, looks significantly different. There has been a surge in the popularity of continuous delivery, DevOps, Big Data analytics, lean startup delivery, and exploratory testing. The Quadrants model is due for a serious update.
One of the problems with the original Quadrants model is that it was easily misunderstood as a sequence of test types—especially that there is some kind of division between things before and things after development.
This problem is even worse now than in 2008. With the surge in popularity of continuous delivery, the dividing line is getting more blurred and is disappearing. With shorter iterations and continuous delivery, it’s generally difficult to draw the line between activities that support the team and those that critique the product. Why would performance tests not be aimed at supporting the team? Why are functional tests not critiquing the product? Why is UAT separate from functional testing? I always found the horizontal dimension of the Quadrants difficult to justify, because critiquing the product can support the team quite effectively if it is done in a timely way. For example, specification by example helps teams to completely merge functional tests and UAT into something that is continuously checked during development. Many teams I worked with recently run performance tests during development, primarily not to mess things up with frequent changes. These are just two examples where things on the right side of the Quadrants are now used more to support the team than anything else. With lean startup methods, products get a lot of critiquing even before a single line of production code is written.
Dividing tests into those that support development and those that evaluate the product does not really help to facilitate useful discussions anymore, so we need a different model—in particular, one that helps to address the eternal issue of so-called nonfunctional requirements, which for many people actually means, “It’s going to be a difficult discussion, so let’s not have it.” The old Quadrants model puts “ilities” into a largely forgotten quadrant of technical tests after development. But things like security, performance, scalability, and so on are not really technical; they imply quite a lot of business expectations, such as compliance, meeting service-levels agreements, handling expected peak loads, and so on. They are also not really nonfunctional, as they imply quite a lot of functionality such as encryption, caching, and work distribution. This of course is complicated by the fact that some expectations in those areas are not that easy to define or test for—especially the unknown unknowns. If we treat these as purely technical concerns, the business expectations are often not explicitly stated or verified. Instead of nonfunctional, these concerns are often dysfunctional. And although many “ilities” are difficult to prove before the software is actually in contact with its real users, the emergence of A/B split testing techniques over the last five years has made it relatively easy, cheap, and low risk to verify those things in production.
Another aspect of testing not really captured well by the first book’s Quadrants is the surge in popularity and importance of exploratory testing. In the old model, exploratory testing is something that happens from the business perspective in order to evaluate the product (often misunderstood as after development). In many contexts, well documented in Elisabeth Hendrickson’s book on exploratory testing (Hendrickson, 2013) and James Whittaker’s book How Google Tests Software (Whittaker et al., 2012), exploratory testing can be incredibly useful for the technical perspective as well and, more importantly, is something that should be done during development.
The third aspect that is not captured well by the early Quadrants is the possibility to quantify and measure software changes through usage analytics in production. The surge in popularity of Big Data analytics, especially combined with lean startup and continuous delivery models, enables teams to test relatively cheaply things that were very expensive to test ten years ago—for example, true performance impacts. When the original Agile Testing book came out, serious performance testing often meant having a complete hardware copy of the production system. These days, many teams de-risk those issues with smaller, less risky continuous changes, whose impact is measured directly on a subset of the production environment. Many teams also look at their production log trends to spot unexpected and previously unknown problems quickly.
We need to change the model (Figure 8-3) to facilitate all those discussions, and I think that the current horizontal division isn’t helping anymore. The context-driven testing community argues very forcefully that looking for expected results isn’t really testing; instead, they call that checking. Without getting into an argument about what is or isn’t testing, I found the division to be quite useful for many recent discussions with clients. Perhaps that is a more useful second axis for the model: the difference between looking for expected outcomes and analyzing unknowns, aspects without a definite yes/no answer, where results require skillful analytic interpretation. Most of the innovation these days seems to happen in the second part anyway. Checking for expected results, from both a technical and business perspective, is now pretty much a solved problem.
)
Figure 8-3 Gojko Adzic’s version of the agile testing quadrants
Thinking about checking expected outcomes versus analyzing outcomes that weren’t predefined helps to explain several important issues facing software delivery teams today:
Security concerns could be split easily into functional tests for compliance such as encryption, data protection, authentication, and so on (essentially all checking for predefined expected results), and penetration/investigations (not predefined). This will help to engage the delivery team and business sponsors in a more useful discussion about describing the functional part of security up front.
Performance concerns could be divided into running business scenarios to prove agreed-upon service levels and capacity, continuous delivery style (predefined), and load tests (where will it break?). This will help to engage the delivery team and business in defining performance expectations and prevent people from treating performance as a purely technical concern. By avoiding the support the team/evaluate the product divisions, we allow a discussion of executing performance tests in different environments and at different times.
Exploration would become much more visible and could be clearly divided between technical and business-oriented exploratory tests. This can support a discussion of technical exploratory tests that developers should perform or that testers can execute by reusing existing automation frameworks. It can also support an overall discussion of what should go into business-oriented exploratory tests.
Build-measure-learn product tests would fit into the model nicely, and the model would facilitate a meaningful discussion of how those tests require a defined hypothesis and how that is different from just pushing things out to see what happens through usage analytics.
We can facilitate a conversation on how to spot unknown problems by monitoring production logs as a way of continuously testing technical concerns that are difficult to check and expensive to automate before deployment, but still useful to support the team. By moving the discussion away from supporting development or evaluating the product toward checking expectations or inspecting the unknown, we would also have a nice way of differentiating those tests from business-oriented production usage analytics.
Most importantly, by using a different horizontal axis, we can raise awareness about a whole category of things that don’t fit into typical test plans or test reports but are still incredibly valuable. The early Quadrants were useful because they raised awareness about a whole category of things in the upper-left corner that most teams weren’t really thinking of but are now taken as common sense. The 2010s Quadrants need to help us raise awareness about some more important issues for today.
Elisabeth Hendrickson also presented an alternative to the existing Quadrants in her talk about “The Thinking Tester” (Hendrickson, 2012). It is similar to Gojko’s version but has a different look. You can see in Figure 8-4 that she relabeled the vertical columns to “confirm” and “investigate,” while the horizontal rows still represent business and technology.
)
Figure 8-4 Elisabeth Hendrickson’s version of the agile testing quadrants
The top left quadrant represents the expectations of the business, which could be in the form of executable (automated) specifications. Others might be represented by paper prototypes or wireframes. At the top right are tests that help investigate risks concerning the external quality of the product. It is very much like the original quadrant’s idea of exploratory testing, scenarios, or usability testing. Like Gojko’s model, the bottom right quadrant highlights the risks of the internal working of the system.
Both of these alternative models provide value. We think there is room for multiple variations to accommodate a spectrum of needs. For example, organizations that are able to adopt continuous delivery are able to think in this space, but many organizations are years from accomplishing that. Check the bibliography for Part III for links to additional testing quadrant models. Use them to help make sure your team covers all the different types of tests you need in order to deliver the right value for your customers.