Skip to content
Technology transformation expertise firm · ARDIANE Group Innovate, secure, succeed

Engineering

Deciding what deserves automated tests, and at what level

In brief

Automated tests are checks written as code and replayed without intervention at each change to an application. Each one is paid for at writing, at execution and in upkeep: Management Square proposes here criteria for deciding what to automate, at what level to place each check and in which cases to refrain.

Page contents

Automated tests are paid for three times: at writing, at each run, then at each evolution of the code they check. The third cost easily escapes the initial estimate, even though the lifespan of a suite depends on it.

They are checks written as code, which the delivery pipeline replays without intervention at each change. The decision to write one is taken check by check. What deserves a script is what will be replayed many times, whose expected result can be stated without ambiguity and whose subject is stable enough for the test to survive the next deliveries. Each check retained then moves as close to the code as the feared defect allows: unit test, integration or contract test, end-to-end test. Whatever requires judgement, whatever changes every week and whatever will soon be removed remains entrusted to a person, or waits.

The checks that benefit from being automated

Three criteria combine, and a fourth decides between them.

Repetition comes first. A check replayed at each code change pays back its writing. A check useful once a year is done by hand, from a written procedure.

Next comes the existence of a verdict. A test compares an obtained result with an expected result. If nobody can write this expected result, because a rate table is still under discussion or a screen will be judged “by eye”, the script will check nothing useful: the rule has to be settled first.

The third criterion is the stability of the subject tested. A calculation rule set by a contract changes little. The layout of a screen being designed can change at each workshop, and one change is enough to make the test fail when no defect exists.

Severity decides between the remaining candidates: priority goes to behaviours whose failure would be expensive, such as an invoiced amount, an access right or a regulatory deadline. How to link these risks to an overall test strategy is described in our page software quality and testing; this article sticks to the decision to automate.

At what level to place each check

The test pyramid, presented in our software quality and testing page, gives a distribution principle. The table below reads it the other way round, starting from what the team fears.

Feared defect Level at which to check it What this level does not see
A wrong business rule: rounding, rate table, deadline, missing value Unit test, without database or network Everything to do with assembly
A query that returns the wrong rows, a forgotten schema migration Integration test, on a database with the same engine as production The other services and the interface
Two services that no longer agree on the format of an exchange Contract test, run on both sides of the interface Overall behaviour on real data
A vital journey broken once the application is assembled and deployed End-to-end test, for a small number of journeys The component behind the failure
Response times that degrade under traffic Load test Functional correctness

A test placed too high still detects the defect, but later, more slowly and without designating the faulty component. A rate table checked through the interface requires a browser, an account and data for each case. The same rate table, as a unit test, is checked case by case without starting anything other than the calculation code.

A test placed too low reassures wrongly. Unit tests that replace the database with a mock can all pass while the real query is wrong.

The last two lines of the table follow their own rules. The choice of journeys, the data and the stability of the suite are covered in our article on end-to-end testing. Framing a campaign and reading its results are covered in the one devoted to load testing.

The upkeep cost of automated tests is decided at writing

A test suite is code: it is read, refactored and ages like the rest. Four choices, made when the test is written, set its upkeep burden.

What the test observes counts first. A test that checks a result visible from outside (a returned value, a record created, a message emitted) survives an internal rewrite. A test that checks the manner (a given method called, in a given order) is likely to fail at each refactoring, even though no behaviour has changed. The signal can be spotted without tools: when a refactoring with no functional effect forces dozens of tests to be fixed, the suite is coupled to the implementation.

Test doubles weigh next. Replacing a dependency with a mock or stub speeds up the test and isolates it. Each mock is also an assumption about that dependency’s behaviour, which nothing updates when it evolves. Mocks are therefore reserved for boundaries that are slow or outside the team, and a contract test checks that the assumption still holds.

Test data follows the same logic. A test that creates the data it needs can be read on its own and runs in any order. A data set common to the whole suite becomes a shared dependency: modifying it for one test can make others fail.

Duration remains. Agree on a maximum time for the verdict that precedes each integration: this time budget bounds what can run at that moment and sends the rest to a less frequent run. Parallel execution and the selection of tests affected by a change fall under the pipeline described in our page DevOps and platform engineering.

A test that no longer pays for its upkeep is deleted. Kept as a precaution, it weighs on every change, just like the rest of technical debt.

When to refrain, or wait

Choosing not to automate is a decision like any other, provided it is taken with full knowledge of the facts. Several situations justify it:

  • the code is a prototype, intended to be thrown away once its hypothesis has been verified (if the prototype is kept, the decision is reopened);
  • the interface is being designed: the business rules behind it are tested now, the screens wait until they are stabilised;
  • the check requires judgement, on the readability of a screen, the consistency of a journey or the reaction to a situation nobody had foreseen;
  • the check is rare and running it manually costs little, such as an annual closing check;
  • the existing code does not lend itself to testing without being restructured.

The last case calls for a clarification. Wanting to cover an old application entirely before touching it paralyses the team. Characterisation tests, which freeze the current behaviour around the sole area to be modified, are enough to start with.

In each of these situations, the check remains: it is manual, described in writing, and someone answers for it. The decision not to automate is recorded with the condition that will cause it to be reviewed.

Rereading the existing suite before enlarging it

In an application that already has tests, the first step adds nothing. Record, for each group of tests, three pieces of information: its run time, the last real defect it detected, and the number of times it had to be fixed while the application was working. Groups that are slow, silent and repeatedly repaired should be moved down a level or deleted. Areas where a defect reached production without any test failing designate what remains to be automated.

FAQ

Frequently asked questions

Another question? Ask us

Which automated tests should you start with on an application that has none?

With the area the team is about to modify. Characterisation tests there freeze the current behaviour, as it is, before any intervention: they then flag any unintended change. Unit tests are added as the code is restructured and becomes testable. Trying to cover the whole application before touching it delays deliveries without further reducing the risk of the next change.

Do automated tests replace acceptance testing?

They take over its repetitive part: checking, at each delivery, that what worked still works. Acceptance testing keeps what requires judgement, starting with the question of whether the application meets the business need. Acceptance testing relieved of regression checks can concentrate on what is new and on unexpected situations.

Should tests be written before the code?

Test-driven development (TDD) consists in writing a test that fails, then the code that makes it pass, then refactoring the whole. This discipline leads to code that is testable by construction, since each function is born with its test. It is a team choice: a test written just after the code has the same verification value if it checks a behaviour and fails when that behaviour is broken.

What is a regression test?

A regression test checks that a change has not altered a behaviour that previously worked. The term designates a use, without prejudging the level: any automated test kept and replayed at each change plays this role, whether it is a unit, integration or end-to-end test. A regression campaign carried out by hand before each delivery is a good candidate for automation, provided each check is moved down to the level that suits it.

A project to bring to completion?

A first conversation is enough to understand your challenge, identify the right people and tell you frankly what we can bring.