SLIs, SLOs and Error Budgets

"Is the service reliable?" is a hard question until you agree on what to measure and what number is good enough. SLIs and SLOs give you that agreement. The error budget then turns the target into a clear rule: while you have budget left, ship features; when it runs out, fix reliability first. These ideas come from Google's Site Reliability Engineering (SRE) practice and are now used widely.

Definitions#

Start with the vocabulary, because three terms sound alike but mean different things.

SLA vs SLO#

The SLA and SLO in that table are easy to mix up. An SLA is a business promise; an SLO is an internal engineering target. Set the SLO tighter than the SLA, so you see trouble and react before you owe customers money. Many internal services have SLOs but no SLA at all.

Choosing SLIs#

Everything else builds on the SLI, so pick it first. A good SLI measures what users feel, not what the servers feel. CPU at 90% isn't a problem if every request is still fast. Write each SLI as a ratio of good events to valid events, so it's always between 0% and 100%. The right SLIs depend on what kind of service you run:

Some tips for choosing well:

• Measure as close to the user as you can, e.g. at the load balancer, not deep inside the app.
• Express latency as a threshold: "99% of requests finish in under 300 ms". Don't use an average.
• Start with two or three SLIs per service. More SLIs mean more noise.
• Exclude invalid events, such as requests that fail because the client sent bad input (4xx).

Setting SLOs#

With an SLI chosen, you set a target for it. An SLO is a trade-off between reliability and speed of change. Each extra "nine" costs much more effort and leaves less room for risk. 100% is the wrong target: users can't tell 100% from 99.99% through their own networks, and a 100% target means you can never deploy.

Pick a target from what users need and what the service achieves today. Then adjust it after a few months of data.

The nines table#

To get a feel for what each target means, translate it into downtime. This table uses a 30-day month and a 365-day year.

At four nines, a human can't even respond to a page before the monthly budget is gone. Targets that high need automatic failover and rollback.

Two ways to measure availability#

The nines table counts minutes, but that's only one way to measure. Availability can be counted in time or in requests, and the two formulas answer slightly different questions.

Time-based availability is the share of time the service was up:

$A = \frac{\text{uptime}}{\text{uptime} + \text{downtime}}$

A service that was down for 43 minutes in a 30-day month (43,200 minutes) has $A = 43{,}157 / 43{,}200 \approx 99.9\%$. This form suits the nines table above. It treats a minute at 3 a.m. the same as a minute at peak traffic, and it needs a rule for what counts as "down".

Request-based (aggregate) availability is the share of requests that succeeded:

$A = \frac{\text{successful requests}}{\text{total valid requests}}$

If 9,990,000 of 10,000,000 valid requests succeed, $A = 99.9\%$. This form weights each failure by how many users it hit, and it also counts partial outages, where only some requests fail. It doesn't work well for services with little traffic, where a few failed requests swing the number. Most request/response SLOs use the request-based form, which is why the error budget below counts failed requests.

Error budgets#

The error budget is the amount of unreliability your SLO allows: $1 - \text{SLO}$. It's the same idea as the downtime column above, but you can count it in requests as well as minutes.

The math#

For a request-based SLO, the budget is a number of failed requests.

$\text{budget} = (1 - \text{SLO}) \times \text{total requests}$

For example, a 99.9% SLO over a month with 10 million requests allows $0.001 \times 10{,}000{,}000 = 10{,}000$ failed requests. If 4,000 requests have failed so far, you've used 40% of the budget and have 60% left. The same calculation in code:

def error_budget_report(
    slo: float, total: int, failed: int
) -> dict[str, float]:
    allowed = (1 - slo) * total
    used = failed / allowed
    # round() hides float noise such as 0.39999999999999963
    return {
        "allowed_failures": round(allowed),
        "budget_used": round(used, 4),
        "budget_left": round(1 - used, 4),
    }


print(error_budget_report(slo=0.999, total=10_000_000, failed=4_000))
# {'allowed_failures': 10000, 'budget_used': 0.4, 'budget_left': 0.6}

Error budget policy#

Knowing how much budget is left only helps if everyone agrees what to do with that number. An error budget policy is a written agreement on what happens when the budget runs low. Product and engineering leads agree on it in advance, so nobody argues about it during an outage. A typical policy says:

1. While budget remains, the team ships features at its normal pace.
2. When the budget is spent, the team freezes risky releases. Only bug fixes and reliability work ship.
3. If one incident used a large part of the budget, the team writes a postmortem.
4. The freeze lifts when the service is back within its SLO for the window.

The budget also works in the other direction. If you always end the month with most of the budget unused, you can take more risk, such as faster rollouts or more experiments.

Burn rate#

How much budget is left tells you where you stand; burn rate tells you how fast that's changing. Burn rate is how fast you're using the error budget, compared with the speed that would use exactly all of it by the end of the window.

$\text{burn rate} = \frac{\text{observed error rate}}{1 - \text{SLO}}$

With a 99.9% SLO, the allowed error rate is 0.1%. Here's how different error rates translate:

Burn rate is the best signal for alerting on an SLO. A burn rate of 1 is exactly on budget. A burn rate of 14.4 for one hour uses 2% of a 30-day budget, which is a common threshold for paging someone. The alerting note shows how to build alerts on it.

Pitfalls#

Most SLO problems come from targets nobody uses.

• Too many SLOs: nobody watches them all. Keep a few per user journey.
• Targets set from hope, not data: measure first, then set the target.
• No policy: an SLO without an agreed response to a spent budget is only a number on a dashboard.
• Measuring servers, not users: host uptime can be 100% while users see errors.
• SLO equal to the SLA: you get no warning before a contract breach.

Key takeaways#

• An SLI measures user experience as good events over valid events.
• An SLO is a target for an SLI over a window; an SLA is a contract and should be looser than the SLO.
• Each extra nine cuts allowed downtime by 10x; 99.9% allows about 43 minutes a month.
• Error budget = $1 - \text{SLO}$; spend it on change, and freeze risky releases when it runs out.
• Burn rate tells you how fast the budget is going and is the right basis for SLO alerts.