Like this article? We recommend
Like this article? We recommend
Like this article? We recommend
The Challenge
I love to tell people to build tests that document capability, but I believe a lot more work is needed in this area. More often than not, I encounter tests that simply aren't very readable or maintainable. One reason is that the tests often aren't read by someone outside of the day-to-day perimeter of the project team. The testers, programmers, and others working on the project daily have an intimate knowledge of the product. The tests they produce often assume that knowledge.
Someone without that rich, shared knowledge may need to invest significant effort to understand the tests. Worse, comprehending the tests may require insights that the testers don't have. The unfortunate reality is that the "someone" is often us, returning to our own efforts from a few months or even a few weeks ago. We end up scratching our heads and digging around in order to uncover the buried assumptions.
Mike Cohn, acclaimed author of several books and owner of Mountain Goat Software, has written a new book specifically about Agile development: Succeeding with Agile: Software Development Using Scrum. Chapter 13, which covers the product backlog, includes a section about specification by example. Given the goal of this book, Mike chose to provide a short, simple example for exposition purposes. However, Mike and I explored this example in further depth and agreed that it could use further refinement.
The following table presents five rows of examples to capture the business rules for time-off requests. Each example is a specific test case that embodies a bit of the specification.
|
Days Accrued |
Days Requested |
Days Employed |
Approved? |
|
10 |
11 |
364 |
no |
|
10 |
11 |
366 |
yes |
|
10 |
11 |
365 |
no |
|
10 |
15 |
366 |
yes |
|
10 |
16 |
366 |
no |
This table can be pumped into a test tool such as FitNesse, which will verify that the "Approved?" column is appropriate for each row (test case) of data.
What does this table express? Obviously, in some cases, time off is approved, but in some cases it isn't. Reading into the table a little further, I can gather that this table is only concerned with scenarios in which the days requested is greater than days accrued. Immediately that suggests an appropriate name for these five test cases: "Test Approval for Excess Time-Off Requests Based on Seniority."
A little more analysis of the test rows reveals that time off is not approved for employees with 365 or 364 days of employment. The apparent intent is that employees with a year or less of employment are not approved for time-off requests in excess of what they have accrued. For employees with more than a year, it looks like they can make requests for time off that are no more than five days over their days accrued. It's also possible to misread this table in a number of ways, but most misreadings would be deliberately obtuse and pointless.
Here's the story from Succeeding with Agile: "As an employee who has been here more than a year, I want automatic approval of a time-off request that is up to five days more than I've currently accrued." Indeed, our analysis of the test table shows that it appears to capture the details behind this rule.
Piecing together the business rule from the test data wasn't a huge amount of effort, but it wasn't free either. It didn't help that significant values such as "5 days more" and "365" are magic numbers. Magic numbers require effort for a reader to be able to associate these kinds of values with their corresponding meanings: "limit on request overages for senior employees" and "one year." In some cases, the reader might not even be able to make the appropriate correlations—note that the number 5 is significant, but it doesn't even appear in the test data.
The use of 365 days to represent one year is also interesting. For a good tester, this should immediately trigger a concern: Are we missing a test that deals with employees hired within a year prior to a leap day?
We could add the story description to appear along with the test table. FitNesse is a wiki as well as a test tool, and thus the story description would become part of the test page. But such comments are often equally obtuse, and frequently become lies as requirements (and thus the test data) change. Also, this is a simpler example; in many cases, it's fairly difficult to summarize that detail succinctly in an accurate description.
Tests should speak to us, not make us work to understand them.
You should always seek to eliminate the compulsion to annotate (comment) tests, by designing tests that require minimal digging and analysis. Some general recommendations:
- Name each test succinctly to describe the goal it verifies. A tool like FitNesse allows you to identify and organize tests by their names.
- Use "precondition" tables that describe the data context. Rather than describing the setup for a given test textually, devise additional tables that both verify and tell the story.
- Embed expressions to help correlate data. In our time-off example, the number "11" might be better represented as "10 + 1," which helps to explain the source of the number 11.
- Use variables to help describe the data. Variables can replace magic numbers, imparting instantaneous meaning.
- Design more granular tests. Shorter tests promote test names that express more precisely what each test verifies. In turn, this means that test names are a little less likely to get out of sync with the actual tests.
Using these recommendations, I redesigned the tests for time-off requests. Notice that the following code lines have multiple test pages and two different table layouts:
SuiteVacationApproval
TestEmployeesAreNewUntilAfter365DaysEmployed
| New Employee Verifier |
| Days Employed | Is New? |
| 1 | yes |
| 2 | yes |
| 364 | yes |
| 365 | yes |
| 365 + 1| no |
| 365 + 2| no |
| 365 + 5000 | no |
TestVacationDeniedWhenNewbieAccruedFewerDaysThanRequested
!define NewEmployeeDaysEmployed { 365 }
| New Employee Verifier |
| Days Employed | Is New? |
| ${NewEmployeeDaysEmployed} | yes |
| Days Accrued | Days Requested | Days Employed | Approved ? |
| 10 | 10 + 1 | ${NewEmployeeDaysEmployed| | false |
| 10 | 10 + 10 | ${NewEmployeeDaysEmployed| | false |
| 2 | 2 + 1 | ${NewEmployeeDaysEmployed| | false |
TestVacationApprovedWhenOldEmployeeAccruedUpToFiveDaysMoreThanRequested
!define OldEmployeeDaysEmployed { 366 }
| New Employee Verifier |
| Days Employed | Is New? |
| ${OldEmployeeDaysEmployed} | no |
| Days Accrued | Days Requested | Days Employed | Approved ? |
| 10 | 10 + 1 | ${OldEmployeeDaysEmployed} | true |
| 10 | 10 + 5 | ${OldEmployeeDaysEmployed} | true |
| 10 | 10 + 5 + 1 | ${OldEmployeeDaysEmployed} | false |
The "New Employee Verifier" table captures a business rule of its own on a separate test page. This table also serves to document and verify the use of either an old employee or a new employee on the other two test pages. Suppose someone changes the definition of a new employee (perhaps the evil HR department is increasing the probationary time to two years). They will document that change under this test:
TestEmployeesAreNewUntilAfter365DaysEmployed
renaming it at the same time to this name:
TestEmployeesAreNewUntilAfterTwoYearsEmployed
The impact of this change is that one of the other two tests will fail once the feature is implemented. It might otherwise take a bit of time for someone seeing this failure to track down its cause, but adding the "New Employee Verifier" table as a precondition means that it will fail immediately, thus pinpointing the precise source of the problem.
There's still room for improvement: The "5" and "1" are magic numbers in these tests that we might want to eliminate. Coming back to this example months after I designed it, I also found that it could use some better names. But the important thing is that I've recognized the need to critique my tests continually for understandability, and incrementally improve them where they're deficient.
Useful Agile tests, like code, will ultimately be read many more times than they are written. Using an Agile process, the tests will also be read by a larger number of people. To succeed, you will need to revisit and refine the tests continually to improve their readability. The iterative nature of Agile provides the opportunity to do so.
I recommend that tests always be reviewed by an uninvolved party. Ask him or her to paraphrase, out loud, what a given test expresses. The questions asked by this independent party will help you to improve your tests to the point where they can be long-lived and trustworthy artifacts.
Jeff Langr is a veteran software developer with over a quarter century of professional software development experience. He has written two books, including Agile Java: Crafting Code with Test-Driven Development. Jeff contributed to "Uncle Bob" Martin's acclaimed book Clean Code: A Handbook of Agile Software Craftsmanship and has written more than 80 articles on software development. Most recently, Jeff is working with Tim Ottinger on the Agile in a Flash card project. You can find out more about Jeff at his Langr Software Solutions website.