Definition of SRE
SRE stands for Site Reliability Engineering. It is a discipline born at Google in the early 2000s. It consists in treating the operation of IT systems as a software engineering problem, entrusted to engineers who write code to automate what used to be done by hand.
The definition of SRE rests on a guiding idea: reliability is specified, measured and managed through quantified objectives, instead of being pursued without limit. The acronym also refers to the role. An SRE is the engineer responsible for the reliability of a service.
What SRE is for
Every organisation that operates digital services knows the same tension. Development teams want to ship fast. Operations teams want a stable service, and know that change is the leading cause of incidents. Without a common rule, the trade-off is settled by a balance of power.
SRE provides a measurable arbitration rule, accepted in advance by both parties. It also serves two other purposes: reducing the share of manual work in operations, and drawing a lasting lesson from every incident.
SLI, SLO, SLA: three nested notions
| Acronym | Meaning | Nature | Example |
|---|---|---|---|
| SLI | Service Level Indicator | A measure | Share of requests served without error |
| SLO | Service Level Objective | An internal target set on the SLI | 99.9% of requests successful over 30 days |
| SLA | Service Level Agreement | A contractual commitment | Level guaranteed to the customer, with penalties |
A good indicator is measured from the user’s point of view: availability, response time, correctness of the result. A processor’s utilisation rate is not one. The internal objective is set higher than the contractual commitment, so as to keep a margin before the penalty.
Aiming for 100% is a methodological error. Each additional decimal place costs much more than the previous one, and beyond a certain threshold the user no longer perceives the difference, since their own network or device is less reliable than the service.
The error budget
The error budget is the complement of the objective. With an SLO of 99.9%, it is 0.1%: the share of failure the organisation accepts over the period.
Its value lies in turning a debate of opinions into a management rule. As long as the budget is not used up, the product team ships at its own pace and takes measured risks. When it is exhausted, risky production releases are suspended and effort shifts to reliability. The rule is written in calm conditions, before the incident, and approved by those responsible for the product as well as by those responsible for operations.
Other SRE practices
- Reducing repetitive manual work. Operations tasks with no lasting value are measured and automated as a priority.
- Alerts based on SLOs. On-call staff are called upon when the user’s experience is threatened, not at every variation of a technical metric.
- Blameless incident review. The report describes the sequence of events and the system’s flaws, so that everyone can speak freely.
- Progressive deployments. A new version is exposed to a fraction of traffic before being rolled out generally.
- Observability. Metrics, logs and traces make it possible to understand behaviour that nobody had anticipated.
SRE and DevOps: not to be confused
DevOps is a culture: bringing development and operations closer together, automating, measuring, sharing responsibility for the service. It does not prescribe how to achieve this. SRE is a precise way of implementing it, with codified practices and a role. The two are not opposed.
SRE also differs from two related notions. The ITIL framework describes IT service management processes (incidents, changes, problems): it sets an organisational framework, whereas SRE brings engineering practices. Platform engineering, for its part, focuses first on the tooling offered to developers, whereas SRE concentrates on the reliability of services in production.
A concrete example: calculating an error budget
Take an online payment service whose availability objective is set at 99.9% over 30 days. A 30-day period contains 43,200 minutes. The error budget, 0.1% of that total, represents about 43 minutes of unavailability.
On the tenth day, a faulty deployment causes a 30-minute outage. More than two thirds of the budget is used up. The agreed rule applies: the major production release planned for the following week is postponed, and the team first deals with the cause of the incident, in this case a missing test in the deployment pipeline. Nobody had to impose this decision. It follows from an objective that product and operations had set together.
To define your reliability objectives and tool up their monitoring, see our page on SRE and observability.