Skip to content
Technology transformation expertise firm · ARDIANE Group Innovate, secure, succeed

Glossary

Mutation testing: definition, mutants, mutation score and limits

Domain

Engineering

In brief

Mutation testing is a technique for evaluating an automated test suite: a tool introduces small faults into the code, one at a time, and checks that at least one test fails for each. The share of faults detected, called the mutation score, indicates how well the tests can catch a defect, which code coverage does not measure.

Page contents

The blind spot of code coverage

A test suite can execute every line of a program without checking anything it produces. Mutation testing starts from this weakness. The documentation of PIT, a mutation testing tool for Java and the JVM (pitest.org), puts it this way: conventional coverage, by lines, statements or branches, only measures which code the tests execute, not their ability to detect a fault in it. It therefore only identifies with certainty the code that is not tested.

Definition of mutation testing

Mutation testing is a technique for evaluating a test suite. A tool automatically introduces small faults into the application’s code, one at a time, then reruns the tests on each modified version. If a test fails, the fault has been detected. If they all pass, the suite has let a change in behaviour slip through.

This description is common to the documentation of PIT and of Stryker (stryker-mutator.io), a tool available for JavaScript and TypeScript, C# and Scala, read on 12 November 2026.

Killed, surviving, equivalent mutant: the vocabulary and the score

  • Mutant: a version of the code carrying a fault introduced by the tool.
  • Mutation operator (mutator): a transformation rule. PIT cites replacing >= with >, negating a condition, removing a method call or modifying a return value.
  • Killed mutant: at least one test fails in its presence.
  • Surviving mutant (survived): all tests pass. Both tools report separately the case where no test executes the modified line (no coverage).
  • Equivalent mutant: its behaviour does not differ from that of the original code, and no test can kill it. Stryker’s documentation states that the tool has no reliable way of identifying it.
  • Mutation score: the share of mutants detected. Stryker divides the mutants detected, killed or stopped by a timeout, by the valid mutants, excluding compilation and runtime errors.

Example: a bike rental company’s free half-hour

Suppose a bike rental company whose pricing offers the first half-hour free: the code returns zero if the duration is less than or equal to thirty minutes. Two tests, for ten minutes and for one hour, execute every line of the calculation. The tool replaces “less than or equal to” with “strictly less than”: both tests still pass, and the mutant survives. The case of exactly thirty minutes was not tested.

Let us continue with hypothetical figures. Out of 20 valid mutants, 16 are detected: the score is 16 / 20, or 80%. The added test kills one more mutant: 17 / 20, or 85%. If one of the three remaining survivors is equivalent and is still counted, the score will not exceed 19 / 20, or 95%.

Mutation testing, code coverage, test levels: not to be confused

Code coverage says which lines the tests execute. The mutation score says whether a fault introduced into those lines makes a test fail. PIT’s reports present the two measures side by side.

Mutation testing is not an additional test level. It does not add a layer to the test pyramid: it evaluates the tests found there, starting with unit tests, which PIT reruns against the mutants. It indicates the strength of the safety net of tests that refactoring relies on.

Nor does it say whether the behaviour verified is the one the business expects, a question addressed by the examples discussed in behavior driven development.

Then there is the cost. PIT’s documentation describes the technique as computationally expensive and advises limiting the analysis to the code currently being modified.

Taken as a target, a mutation score can also encourage tests written for the figure: our software quality and testing page explains why the coverage rate is read there as a signal.

FAQ

Frequently asked questions

Another question? Ask us

What does mutation testing involve?

Mutation testing involves automatically introducing small faults into an application's code, then rerunning the tests on each modified version, called a mutant. A mutant is said to be killed if at least one test fails, and to have survived if all tests pass. This description can be found in the documentation of two tools, PIT for Java and Stryker for JavaScript, C# and Scala.

How is the mutation score calculated?

The mutation score relates the mutants detected by the tests to the number of mutants taken into account. Stryker's documentation calculates it by dividing the mutants detected, killed or stopped by a timeout, by the valid mutants, that is, those that caused neither a compilation error nor a runtime error. The same tool publishes a second score, restricted to code covered by the tests: the convention used must be specified before comparing two figures.

What is an equivalent mutant?

An equivalent mutant is a modified version of the code whose behaviour does not differ from that of the original code, so that no test can make it fail. Stryker's documentation states that the tool has no reliable way of identifying these equivalent mutants. It leaves two options: find them by hand and rewrite the code concerned, or accept a mutation score below 100%.

Do mutation testing and code coverage measure the same thing?

Mutation testing and code coverage measure two distinct things. According to PIT's documentation, code coverage only indicates which code the tests execute, without saying whether those tests would detect a fault in that code. Mutation testing introduces faults and observes whether a test fails. Fully covered code can therefore let mutants survive.

Why does mutation testing take a long time to run?

Mutation testing takes a long time to run because tests are rerun for each mutant produced. PIT's documentation speaks of a computationally expensive process, whose duration depends on the size of the code as well as the quality and speed of the test suite. It advises limiting the analysis to the code currently being modified.

A project to bring to completion?

A first conversation is enough to understand your challenge, identify the right people and tell you frankly what we can bring.