- Testable Software
- Benefits of Testability
- Testability Defined
- Summary
This chapter is from the book
This chapter is from the book
This chapter is from the book
Testability Defined
Testability is a quality attribute among other “ilities” like reliability, maintainability, and usability. Just like the other quality attributes, it can be broken down into more fine-grained components (Figure 4.2). Observability and controllability are the two cornerstones of testability. Without them, it’s hard to say anything about correctness. The remaining components described next made it to the model based on my practical experience, although I hope that their presence isn’t surprising or controversial.
)
Figure 4.2 The testability quality attribute decomposed.
When a program element (see “Program Elements”) is testable, it means that it can be put in a known state, acted on, and then observed. Further, it means that this can be done without affecting any other program elements and without them interfering. In other words, it’s about making the black box of testing somewhat transparent and adding some control levers to it.
Observability
In order to verify that whatever action our tested program element has been subjected to has had an impact, we need to be able to observe it. The best test in the world isn’t worth anything unless its effects can be seen. Software can be observed using a variety of methods. One way of classifying them is in order of increasing intrusiveness.
The obvious, but seldom sufficient, method of observation is to examine whatever output the tested program element produces. Sometimes that output is a sequence of characters, sometimes a window full of widgets, sometimes a web page, and sometimes a rising or falling signal on the pin of a chip.
Then there’s output that isn’t always meant for the end users. Logging statements, temporary files, lock files, and diagnostics information are all output. Such output is mostly meant for operations and other more “technical” stakeholders. Together with the user output, it provides a source of information for nonintrusive testing.
To increase observability beyond the application’s obvious and less obvious output, we have to be willing to make some intrusions and modify it accordingly. Both testers and developers benefit from strategically placed observation points and various types of hooks/seams for attaching probes, changing implementations, or just peeking at the internal state of the application. Such modifications are sometimes frowned upon, as they result in injection of code with the sole purpose of increasing observability. At the last level, there’s a kind of observability that’s achievable only by developers. It’s the ability to step through running code using a debugger. This certainly provides maximum observability at the cost of total intrusion. I don’t consider this activity testing, but rather writing code. And you certainly don’t want debugging to be your only means of verifying that your code works.
Too many observation points and working too far from production code may result in the appearance of Heisenbugs—bugs that tend to disappear when one tries to find and study them. This happens because the inspection process changes something in the program’s execution. Excessive logging may, for example, hide a race condition because of the time it takes to construct and output the information to be logged.
Logging, by the way, is a double-edged sword. Although it’s certainly the easiest way to increase observability, it may also destroy readability. After all, who hasn’t seen methods like this:
void performRemoteReboot(String message) {
if (log.isDebugEnabled()) {
log.debug("In performRemoteReboot:" + message);
}
log.debug("Creating telnet client");
TelnetClient client = new TelnetClient("192.168.1.34");
log.debug("Logging in");
client.login("rebooter", "secret42");
log.debug("Rebooting");
client.send("/sbin/shutdown -r now '" + message + "'");
client.close();
log.debug("done");
}
As developers, we need to take observability into account early. We need to think about what kind of additional output we and our testers may want and where to add more observation points.
Observability and information hiding are often at odds with each other. Many languages, most notably the object-oriented ones, have mechanisms that enable them to limit the visibility of code and data to separate the interface (function) from the implementation. In formal terms, this means that any proofs of correctness must rely solely on public properties and not on “secret” ones (Meyer 1997). On top of that, the general opinion among developers seems to be that the kind of testing that they do should be performed at the level of public interfaces. The argument is sound: if tests get coupled to internal representations and operations, they get brittle and become obsolete or won’t even compile with the slightest refactoring. They no longer serve as the safety net needed to make refactoring a safe operation.
Although all of this is true, the root cause of the problem isn’t really information hiding or encapsulation, but poor design and implementation, which, in turn, forces us to ask the question of the decade: Should I test private methods?3
Old systems were seldom designed with testability in mind, which means that their program elements often have multiple areas of responsibility, operate at different levels of abstraction at the same time, and exhibit high coupling and low cohesion. Because of the mess under the hood, testing specific functionality in such systems through whatever public interfaces they have (or even finding such interfaces) is a laborious and slow process. Tests, especially unit tests, become very complex because they need to set up entire “ecosystems” of seemingly unrelated dependencies to get something deep in the dragon’s lair working.
In such cases we have two options. Option one is to open up the encapsulation by relaxing restrictions on accessibility to increase both observability and controllability. In Java, changing methods from private to package scoped makes them accessible to (test) code in the same package. In C++, there’s the infamous friend keyword, which can be used to achieve roughly a similar result, and C# has its InternalsVisibleTo attribute.
The other option is to consider the fact that testing at a level where we need to worry about the observability of deeply buried monolithic spaghetti isn’t the course of action that gives the best bang for the buck at the given moment. Higher-level tests, like system tests or integration tests, may be a better bet for old low-quality code that doesn’t change that much (Vance 2013).
With well-designed new code, observability and information hiding shouldn’t be an issue. If the code is designed with testability in mind from the start and each program element has a single area of responsibility, then it follows that all interesting abstractions and their functionality will be primary concepts in the code. In object-oriented languages this corresponds to public classes with well-defined functionality (in procedural languages, to modules or the like). Many such abstractions may be too specialized to be useful outside the system, but in context they’re most meaningful and eligible for detailed developer testing. The tale in the sidebar contains some examples of this.
Controllability
Controllability is the ability to put something in a specific state and is of paramount importance to any kind of testing because it leads to reproducibility. As developers, we like to deal with determinism. We like things to happen the same way every time, or at least in a way that we understand. When we get a bug report, we want to be able to reproduce the bug so that we may understand under what conditions it occurs. Given that understanding, we can fix it. The ability to reproduce a given condition in a system, component, or class depends on the ability to isolate it and manipulate its internal state.
Dealing with state is complex enough to mandate a section of its own. For now, we can safely assume that too much state turns reproducibility, and hence controllability, into a real pain. But what is state? In this context, state simply refers to whatever data we need to provide in order to set the system up for testing. In practice, state isn’t only about data. To get a system into a certain state, we usually have to set up some data and execute some of the system’s functions, which in turn will act on the data and lead to the desired state.
Different test types require different amounts of state. A unit test for a class that takes a string as a parameter in its constructor and prints it on the screen when a certain method is called has little state. On the other hand, if we need to set up thousands of fake transactions in a database to test aggregation of cumulative discounts, then that would qualify as a great deal of state.
Deployability
Before the advent of DevOps, deployability seldom made it to the top five quality attributes to consider when implementing a system. Think about the time you were in a large corporation that deployed its huge monolith to a commercial application server. Was the process easy? Deployability is a measure of the amount of work needed to deploy the system, most notably, into production. To get a rough feeling for it, ask: “How long does it take to get a change that affects one line of code into production?” (Poppendieck & Poppendieck 2006).
Deployability affects the developers’ ability to run their code in a production-like environment. Let’s say that a chunk of code passes its unit tests and all other tests on the developer’s machine. Now it’s time to see if the code actually works as expected in an environment that has more data, more integrations, and more complexity (like a good production-like test environment should have). This is a critical point. If deploying a new version of the system is complicated and prone to error or takes too much time, it won’t be done. A typical process that illustrates this problem is manual deployment based on a list of instructions. Common traits of deployment instructions are that they’re old, they contain some nonobvious steps that may not be relevant at all, and despite their apparent level of detail, they still require a large amount of tacit knowledge. Furthermore, they describe a process that’s complex enough to be quite error prone.
Being unable to deploy painlessly often punishes the developers in the end. If deployment is too complicated and too time consuming, or perceived as such, they may stop verifying that their code runs in environments that are different from their development machines. If this starts happening, they end up in the good-old “it works on my machine” argument, and it never makes them look good, like in this argument between Tracy the Tester and David the Developer:
Tracy: I tried to run the routine for verifying postal codes in Norway. When I entered an invalid code, nothing happened.
David: All my unit tests are green and I even ran the integration tests!
Tracy: Great! But I expected an error message from the system, or at least some kind of reaction.
David: But really, look at my screen! I get an error message when entering an invalid postal code. I have a Norwegian postal code in my database.
Tracy: I notice that you’re running build 273 while the test environment runs 269. What happened?
David: Well . . . I didn’t deploy! It would take me half a day to do it! I’d have to add a column to the database and then manually dump the data for Norway. Then I’d have to copy the six artifacts that make up the system to the application server, but before doing that I’d have to rebuild three of them. . . . I forgot to run the thing because I wanted to finish it!
The bottom line is that developers are not to consider themselves finished with their code until they’ve executed it in an environment that resembles the actual production environment.
Poor deployability has other adverse effects as well. For example, when preparing a demo at the end of an iteration, a team can get totally stressed out if getting the last-minute fixes to the demo environment is a lengthy process because of a manual procedure.
Last, but not least, struggling with unpredictable deployment also makes critical bug fixes difficult. I don’t encourage making quick changes that have to be made in a very short time frame, but sometimes you encounter critical bugs in production and they have to be fixed immediately. In such situations, you don’t want to think about how hard it’s going to get the fix out—you just want to squash the bug.
Isolability
Isolability, modularity, low coupling—in this context, they’re all different sides of the same coin. There are many names for this property, but regardless of the name, it’s about being able to isolate the program element under test—be it a function, class, web service, or an entire system.
Isolability is a desirable property from both a developer’s and a tester’s point of view. In modular systems, related concepts are grouped together, and changes don’t ripple across the entire system. On the other hand, components with lots of dependencies are not only difficult to modify, but also difficult to test. Their tests will require much setup, often of seemingly unrelated dependencies, and their interactions with the outside world will be artificial and hard to make sense of.
Isolability applies at all levels of a system. On the class level, isolability can be described in terms of fan-out, that is, the number of outgoing dependencies on other classes. A useful design rule of thumb is trying to achieve a low fan-out. In fact, high fan-out is often considered bad design (Borysowich 2007). Unit testing classes with high fan-out is cumbersome because of the number of test doubles needed to isolate the class from all collaborators.
Poor isolability at the component level may manifest itself as difficulty setting up its surrounding environment. The component may be coupled to other components by various communication protocols such as SOAP or connected in more indirect ways such as queues or message buses. Putting such a component under test may require that parts of it be reimplemented to make the integration points interchangeable for stubs. In some unfortunate cases, this cannot be done, and testing such a component may require that an entire middleware package be set up just to make it testable.
Systems with poor isolability suffer from the sum of poorness of their individual components. So if a system is composed of one component that makes use of an enterprise-wide message bus, another component that requires a very specific directory layout on the production server (because it won’t even run anywhere else), and a third that requires some web services at specific locations, you’re in for a treat.
Smallness
The smaller the software, the better the testability, because there’s less to test. Simply put, there are fewer moving parts that need to be controlled and observed, to stay consistent with this chapter’s terminology. Smallness primarily translates into the quantity of tests needed to cover the software to achieve a sufficient degree of confidence. But what exactly about the software should be “small”? From a testability perspective, two properties matter the most: the number of features and the size of the codebase. They both drive different aspects of testing.
Feature-richness drives testing from both a black box and a white box perspective. Each feature somehow needs to be tested and verified from the perspective of the user. This typically requires a mix of manual testing and automated high-level tests like end-to-end tests or system tests. In addition, low-level tests are required to secure the building blocks that comprise all the features. Each new feature brings additional complexity to the table and increases the potential for unfortunate and unforeseen interactions with existing features. This implies that there are clear incentives to keep down the number of features in software, which includes removing unused ones.
A codebase’s smallness is a bit trickier, because it depends on a number of factors. These factors aren’t related to the number of features, which means that they’re seldom observable from a black box perspective, but they may place a lot of burden on the shoulders of the developer. In short, white box testing is driven by the size of the codebase. The following sections describe properties that can make developer testing cumbersome without rewarding the effort from the feature point of view.
Singularity
If something is singular, there’s only one instance of it. In systems with high singularity, every behavior and piece of data have a single source of truth. Whenever we want to make a change, we make it in one place. In the book The Pragmatic Programmer, this has been formulated as the DRY principle: Don’t Repeat Yourself (Hunt & Thomas 1999).
Testing a system where singularity has been neglected is quite hard, especially from a black box perspective. Suppose, for example, that you were to test the copy/paste functionality of an editor. Such functionality is normally accessible in three ways: from a menu, by right-clicking, and by using a keyboard shortcut. If you approached this as a black box test while having a limited time constraint, you might have been satisfied with testing only one of these three ways. You’d assume that the others would work by analogy. Unfortunately, if this particular functionality had been implemented by two different developers on two different occasions, then you wouldn’t be able to assume that both are working properly.
)
This example is a bit simplistic, but this scenario is very common in systems that have been developed by different generations of developers (which is true of pretty much every system that’s been in use for a while). Systems with poor singularity appear confusing and frustrating to their users, who report a bug and expect it to be fixed. However, when they perform an action similar to the one that triggered the bug by using a different command or accessing it from another part of the system, the problem is back! From their perspective, the system should behave consistently, and explaining why the bug has been fixed in two out of three places inspires confidence in neither the system nor the developers’ ability.
To a developer, nonsingularity—duplication—presents itself as the activity of implementing or changing the same data or behavior multiple times to achieve a single result. With that comes maintaining multiple instances of test code and making sure that all contracts and behavior are consistent.
Level of Abstraction
The level of abstraction is determined by the choice of programming language and frameworks. If they do the majority of the heavy lifting, the code can get both smaller and simpler. At the extremes lie the alternatives of implementing a modern application in assembly language or a high-level language, possibly backed by a few frameworks. But there’s no need to go to the extremes to find examples. Replacing thread primitives with thread libraries, making use of proper abstractions in object-oriented languages (rather than strings, integers, or lists), and working with web frameworks instead of implementing Front Controllers4 and parsing URLs by hand are all examples of raising the level of abstraction. For certain types of problems and constructs, employing functional or logic programming greatly raises the level of abstraction, while reducing the size of the codebase.
The choice of the programming language has a huge impact on the level of abstraction and plays a crucial role already at the level of toy programs (and scales accordingly as the complexity of the program increases). Here’s a trivial program that adds its two command-line arguments together. Whereas the C version needs to worry about string-to-integer conversion and integer overflow ...
#include <stdio.h>
#include <stdlib.h>
int main(int argc, char *argv[])
{
int augend = atoi(argv[1]);
int addend = atoi(argv[2]);
// Let's hope that we don't overflow...
printf("*drum roll* ... %d", augend + addend);
}
... its Ruby counterpart will work just fine for large numbers while being a little more tolerant with the input as well.
puts "*drum roll* ... #{ARGV[0].to_i + ARGV[1].to_i}"
From a developer testing point of view, the former program would most likely give rise to more tests, because they’d need to take overflow into account. Generally, as the level of abstraction is raised, fewer tests that cover fundamental building blocks, or the “plumbing,” are needed, because such things are handled by the language or framework. The user won’t see the difference, but the developer who writes the tests will.
Efficiency
In this context, efficiency equals the ability to express intent in the programming language in an idiomatic way and making use of that language’s functionality to keep the code expressive and concise. It’s also about applying design patterns and best practices. Sometimes we see signs of struggle in codebases being left by developers who have fought valorously reinventing functionality already provided by the language or its libraries. You know inefficient code when you see it, right after which you delete 20 lines of it and replace them with a one-liner, which turns out idiomatic and simple.
Inefficient implementations increase the size of the codebase without providing any value. They require their tests, especially unit tests, because such tests need to cover many fundamental cases. Such cases wouldn’t need testing if they were handled by functionality in the programming language or its core libraries.
Reuse
Reuse is a close cousin of efficiency. Here, it refers to making use of third-party components to avoid reinventing the wheel. A codebase that contains in-house implementations of a distributed cache or a framework for managing configuration data in text files with periodic reloading5 will obviously be larger than one that uses tested and working third-party implementations.
This kind of reuse reduces the need for developer tests, because the functionality isn’t owned by them and doesn’t need to be tested. Their job is to make sure that it’s plugged in correctly, and although this, too, requires tests, they will be fewer in number.
A Reminder about Testability
Have you ever worked on a project where you didn’t know what to implement until the very last moment? Where there were no requirements or where iteration planning meetings failed to result in a shared understanding about what to implement in the upcoming two or three weeks? Where the end users weren’t available?
Or maybe you weren’t able to use the development environment you needed and had to make do with inferior options. Alternatively, there was this licensed tool that would have saved the day had but somebody paid for it.
Or try this: the requirements and end users were there and so was the tooling, but nobody on the team knew how to do cross-device mobile testing.
After having dissected the kind of testability the developer is exposed to the most, I’m just reminding that there are other facets of testability that we mustn’t lose sight of.