Automated tests are paid for three times: at writing, at each run, then at each evolution of the code they check. The third cost easily escapes the initial estimate, even though the lifespan of a suite depends on it.
They are checks written as code, which the delivery pipeline replays without intervention at each change. The decision to write one is taken check by check. What deserves a script is what will be replayed many times, whose expected result can be stated without ambiguity and whose subject is stable enough for the test to survive the next deliveries. Each check retained then moves as close to the code as the feared defect allows: unit test, integration or contract test, end-to-end test. Whatever requires judgement, whatever changes every week and whatever will soon be removed remains entrusted to a person, or waits.
The checks that benefit from being automated
Three criteria combine, and a fourth decides between them.
Repetition comes first. A check replayed at each code change pays back its writing. A check useful once a year is done by hand, from a written procedure.
Next comes the existence of a verdict. A test compares an obtained result with an expected result. If nobody can write this expected result, because a rate table is still under discussion or a screen will be judged “by eye”, the script will check nothing useful: the rule has to be settled first.
The third criterion is the stability of the subject tested. A calculation rule set by a contract changes little. The layout of a screen being designed can change at each workshop, and one change is enough to make the test fail when no defect exists.
Severity decides between the remaining candidates: priority goes to behaviours whose failure would be expensive, such as an invoiced amount, an access right or a regulatory deadline. How to link these risks to an overall test strategy is described in our page software quality and testing; this article sticks to the decision to automate.
At what level to place each check
The test pyramid, presented in our software quality and testing page, gives a distribution principle. The table below reads it the other way round, starting from what the team fears.
| Feared defect | Level at which to check it | What this level does not see |
|---|---|---|
| A wrong business rule: rounding, rate table, deadline, missing value | Unit test, without database or network | Everything to do with assembly |
| A query that returns the wrong rows, a forgotten schema migration | Integration test, on a database with the same engine as production | The other services and the interface |
| Two services that no longer agree on the format of an exchange | Contract test, run on both sides of the interface | Overall behaviour on real data |
| A vital journey broken once the application is assembled and deployed | End-to-end test, for a small number of journeys | The component behind the failure |
| Response times that degrade under traffic | Load test | Functional correctness |
A test placed too high still detects the defect, but later, more slowly and without designating the faulty component. A rate table checked through the interface requires a browser, an account and data for each case. The same rate table, as a unit test, is checked case by case without starting anything other than the calculation code.
A test placed too low reassures wrongly. Unit tests that replace the database with a mock can all pass while the real query is wrong.
The last two lines of the table follow their own rules. The choice of journeys, the data and the stability of the suite are covered in our article on end-to-end testing. Framing a campaign and reading its results are covered in the one devoted to load testing.
The upkeep cost of automated tests is decided at writing
A test suite is code: it is read, refactored and ages like the rest. Four choices, made when the test is written, set its upkeep burden.
What the test observes counts first. A test that checks a result visible from outside (a returned value, a record created, a message emitted) survives an internal rewrite. A test that checks the manner (a given method called, in a given order) is likely to fail at each refactoring, even though no behaviour has changed. The signal can be spotted without tools: when a refactoring with no functional effect forces dozens of tests to be fixed, the suite is coupled to the implementation.
Test doubles weigh next. Replacing a dependency with a mock or stub speeds up the test and isolates it. Each mock is also an assumption about that dependency’s behaviour, which nothing updates when it evolves. Mocks are therefore reserved for boundaries that are slow or outside the team, and a contract test checks that the assumption still holds.
Test data follows the same logic. A test that creates the data it needs can be read on its own and runs in any order. A data set common to the whole suite becomes a shared dependency: modifying it for one test can make others fail.
Duration remains. Agree on a maximum time for the verdict that precedes each integration: this time budget bounds what can run at that moment and sends the rest to a less frequent run. Parallel execution and the selection of tests affected by a change fall under the pipeline described in our page DevOps and platform engineering.
A test that no longer pays for its upkeep is deleted. Kept as a precaution, it weighs on every change, just like the rest of technical debt.
When to refrain, or wait
Choosing not to automate is a decision like any other, provided it is taken with full knowledge of the facts. Several situations justify it:
- the code is a prototype, intended to be thrown away once its hypothesis has been verified (if the prototype is kept, the decision is reopened);
- the interface is being designed: the business rules behind it are tested now, the screens wait until they are stabilised;
- the check requires judgement, on the readability of a screen, the consistency of a journey or the reaction to a situation nobody had foreseen;
- the check is rare and running it manually costs little, such as an annual closing check;
- the existing code does not lend itself to testing without being restructured.
The last case calls for a clarification. Wanting to cover an old application entirely before touching it paralyses the team. Characterisation tests, which freeze the current behaviour around the sole area to be modified, are enough to start with.
In each of these situations, the check remains: it is manual, described in writing, and someone answers for it. The decision not to automate is recorded with the condition that will cause it to be reviewed.
Rereading the existing suite before enlarging it
In an application that already has tests, the first step adds nothing. Record, for each group of tests, three pieces of information: its run time, the last real defect it detected, and the number of times it had to be fixed while the application was working. Groups that are slow, silent and repeatedly repaired should be moved down a level or deleted. Areas where a defect reached production without any test failing designate what remains to be automated.