The blind spot of code coverage
A test suite can execute every line of a program without checking anything it produces. Mutation testing starts from this weakness. The documentation of PIT, a mutation testing tool for Java and the JVM (pitest.org), puts it this way: conventional coverage, by lines, statements or branches, only measures which code the tests execute, not their ability to detect a fault in it. It therefore only identifies with certainty the code that is not tested.
Definition of mutation testing
Mutation testing is a technique for evaluating a test suite. A tool automatically introduces small faults into the application’s code, one at a time, then reruns the tests on each modified version. If a test fails, the fault has been detected. If they all pass, the suite has let a change in behaviour slip through.
This description is common to the documentation of PIT and of Stryker (stryker-mutator.io), a tool available for JavaScript and TypeScript, C# and Scala, read on 12 November 2026.
Killed, surviving, equivalent mutant: the vocabulary and the score
- Mutant: a version of the code carrying a fault introduced by the tool.
- Mutation operator (mutator): a transformation rule. PIT cites replacing
>=with>, negating a condition, removing a method call or modifying a return value. - Killed mutant: at least one test fails in its presence.
- Surviving mutant (survived): all tests pass. Both tools report separately the case where no test executes the modified line (no coverage).
- Equivalent mutant: its behaviour does not differ from that of the original code, and no test can kill it. Stryker’s documentation states that the tool has no reliable way of identifying it.
- Mutation score: the share of mutants detected. Stryker divides the mutants detected, killed or stopped by a timeout, by the valid mutants, excluding compilation and runtime errors.
Example: a bike rental company’s free half-hour
Suppose a bike rental company whose pricing offers the first half-hour free: the code returns zero if the duration is less than or equal to thirty minutes. Two tests, for ten minutes and for one hour, execute every line of the calculation. The tool replaces “less than or equal to” with “strictly less than”: both tests still pass, and the mutant survives. The case of exactly thirty minutes was not tested.
Let us continue with hypothetical figures. Out of 20 valid mutants, 16 are detected: the score is 16 / 20, or 80%. The added test kills one more mutant: 17 / 20, or 85%. If one of the three remaining survivors is equivalent and is still counted, the score will not exceed 19 / 20, or 95%.
Mutation testing, code coverage, test levels: not to be confused
Code coverage says which lines the tests execute. The mutation score says whether a fault introduced into those lines makes a test fail. PIT’s reports present the two measures side by side.
Mutation testing is not an additional test level. It does not add a layer to the test pyramid: it evaluates the tests found there, starting with unit tests, which PIT reruns against the mutants. It indicates the strength of the safety net of tests that refactoring relies on.
Nor does it say whether the behaviour verified is the one the business expects, a question addressed by the examples discussed in behavior driven development.
Then there is the cost. PIT’s documentation describes the technique as computationally expensive and advises limiting the analysis to the code currently being modified.
Taken as a target, a mutation score can also encourage tests written for the figure: our software quality and testing page explains why the coverage rate is read there as a signal.