Why software quality deteriorates despite testing
Few organisations lack tests. Yet many find that software quality is not improving: acceptance phases grow longer, production incidents persist, and every release remains an ordeal. The contradiction is only apparent. It is not the quantity of tests that is lacking, but their place and their nature.
Four situations recur. Tests are concentrated at the end of the cycle, in a manual acceptance phase that schedule pressure often ends up cutting short. Automation has focused on the user interface, which produces slow, fragile suites whose failures are often false alarms. These unstable tests are rerun or ignored, and the team stops believing in the red signal. Finally, quality is regarded as the business of a separate test team, which receives the software once it has been written.
The result is late feedback. A defect introduced on a Monday is discovered several weeks later, by someone else, in a context the developer has forgotten. Any software quality approach aims first of all to shorten that delay.
What software quality covers: dimensions and test levels
The quality of software is not limited to its functional conformity. The ISO/IEC 25010 standard describes a software product quality model that distinguishes, among other things, reliability, performance, security, compatibility and maintainability. A serious test strategy states, for each of the dimensions that matter in your context, how it is verified.
For the functional dimension, the test pyramid provides the distribution principle.
| Level | What it verifies | Characteristics | Place in the whole |
|---|---|---|---|
| Unit tests | A unit of code isolated from its dependencies | Very fast, precise about the cause of a failure | The base, the widest |
| Integration and contract tests | Collaboration between components, APIs, data access | Slower, require a partial environment | The middle |
| End-to-end tests | A complete journey through the interface | Slow, sensitive to screen changes and environments | The top, narrow |
The pyramid is not a ratio rule to be applied mechanically. It expresses an economic logic: the higher a test sits, the more it costs to write, run and maintain, and the less it helps locate the cause of a failure. End-to-end tests remain necessary, but for a small number of journeys whose failure would be serious.
Our approach: a risk-based test strategy
Our consultants work within your teams; decisions remain yours.
Start from the risks, not the tools. We establish with the business and operations what would be serious: a wrong calculation, unavailability at a critical period, a data leak, slowness that makes the user give up. Each risk is assigned to the lowest test level capable of detecting it. The resulting strategy fits in a few pages and specifies the production release criteria.
Automate in the right order. Automation starts where it is stable and cost-effective: business rules, APIs, interface contracts. End-to-end journeys come next, in limited number. Two subjects determine success and are addressed from the outset: the availability of environments and the production of controlled test data.
Put continuous quality in place. Every change to the code triggers a series of controls: compilation, static analysis, automated tests, dependency checks. Blocking thresholds prevent the integration of a change that degrades what already exists. This set-up relies on the delivery pipeline described on our DevOps and platform engineering page. It is only worthwhile if feedback is fast: a suite that runs for hours is no longer consulted.
Make the team the owner of quality. Developers write and maintain the tests for their code. Testing specialists bring the method, design the difficult cases and carry out exploratory testing. Peer code review complements the automated controls.
Performance and security: tests in their own right
These two dimensions are often postponed, even though their defects are costly once in production.
Performance tests answer three distinct questions. The load test checks that the application can handle the expected traffic. The stress test looks for the breaking point and observes how the system degrades. The endurance test reveals defects that only appear over time, such as memory consumption that grows without end. None of them makes sense without an explicit objective: we link them to the service level objectives defined with operations, following the approach presented on our SRE and observability page.
Security tests combine several complementary techniques: static analysis of source code, dynamic analysis of the running application, analysis of third-party components to identify known vulnerabilities, and penetration tests carried out by specialists. The first three can be automated in the delivery pipeline. For the financial entities subject to it, the DORA regulation, Regulation (EU) 2022/2554, devotes a chapter to digital operational resilience testing: the test strategy must then be aligned with the testing programme required by that text.
AI to assist testing: uses and limits
Generative AI has several concrete uses in testing: proposing test cases from a specification, including edge cases the team had not thought of, producing synthetic test data, explaining a failure from the logs, or helping to adapt scripts after an interface change. AI agents can also navigate an application to explore its screens.
These uses have three limits, which we set out from the framing stage. The first concerns the reference: a model that writes tests from the code checks that the code does what it does, not what it should do. The second is the false sense of security given by a large number of generated tests that are barely reviewed. The third concerns the confidentiality of the code and data sent to the model.
We therefore promise no quantified gain. The benefit depends on the quality of the specifications, the discipline of review and the scope chosen. It is measured on a limited trial before any wider roll-out.
Testing ideas that seem good but are not
- Automating the manual acceptance phase as it stands. Converting manual scenarios into interface scripts produces the most costly suite to maintain. The checks must first be moved down to the right level.
- Managing by coverage rate. A numerical coverage target produces tests written for the figure, which execute the code without verifying anything.
- Tolerating unstable tests. A test that fails for no reason is fixed or removed. Leaving it in place teaches the team to ignore failures.
- Postponing performance and security to the end. Defects discovered at that stage often affect the architecture and are then costly to fix.
The indicators of improving quality
We select with you a small number of measures, tracked over time and with no target value set in advance.
- Escaped defects. The share of defects discovered in production compared with those detected beforehand. This is the most direct measure of test effectiveness.
- Change failure. The share of production releases followed by an incident, an urgent fix or a rollback.
- Feedback time. The time between a code change and the verdict of the automated controls.
- Test reliability. The share of test failures that correspond to no real defect.
- Vulnerability handling. The time between the detection of a known vulnerability and its correction.
These indicators are read together. A feedback time that shortens while escaped defects increase signals tests that have been removed rather than repaired. It is their joint evolution, examined regularly with the team, that tells whether software quality is really improving.