Alerting and Dashboards

Telemetry (the logs, metrics and traces your services emit) only helps if someone looks at it at the right time. Alerts tell a human that something needs attention now. Dashboards help that human find out what's wrong once they look. Bad alerts wake you up for nothing and teach the team to ignore the pager. Good alerts are rare, urgent and always come with a next step.

Alert on symptoms, not causes#

A symptom is something users feel: errors, slow pages, missing data. A cause is something inside the system: high CPU, a full disk, a restarted pod. Alert on symptoms, because they always matter to users. A cause alert fires when nothing is wrong for users, and it misses every failure you didn't predict. Here's how the swap looks in practice:

Cause metrics still have a place. Put them on dashboards for diagnosis. Alert on a cause only if it predicts certain user pain soon, such as a disk that will be full in four hours.

Page vs ticket#

Even a good symptom alert doesn't always need to wake someone up. Route each problem one of three ways, based on urgency.

If an alert pages and the right response is "look at it tomorrow", change it to a ticket.

What makes an alert actionable#

Once an alert is worth paging for, it also has to be useful when it arrives. An actionable alert is one where the person who receives it knows what to do. Before you add an alert, check that it passes every item below.

• It reflects real or near-certain user impact.
• A human must act; automation can't handle it.
• The alert name and summary say what is broken and for whom.
• It links to a runbook: a short page with steps to diagnose and mitigate.
• It links to the dashboard for the affected service.
• It fires rarely enough that people still take it seriously.

The on-call process note shows what the responder does after the page arrives.

Thresholds and burn-rate alerts#

The next question is when exactly an alert should fire. A simple threshold alert fires when a metric crosses a fixed line, such as "error rate above 1% for 5 minutes". It's easy to write, but hard to tune. A low threshold pages for short blips. A high threshold misses slow leaks that still use up the whole error budget.

Burn-rate alerts#

Burn-rate alerts fix both problems. A burn-rate alert fires based on how fast you use your error budget (the amount of failure your SLO allows) (see SLIs, SLOs and Error Budgets). A burn rate of 1 uses the budget up exactly at the end of the SLO period; 14.4 uses it 14.4 times faster. This ties every page directly to the SLO. The Google SRE Workbook suggests these starting values for a 30-day SLO:

Why two windows#

Each row in that table has two windows. The alert fires only when both windows are above the burn rate. The long window proves the problem is big enough to matter. The short window proves it is still happening, so the alert stops soon after you fix the problem. Without the short window, a fixed problem can keep the alert firing for up to an hour.

A Prometheus example#

Here's what the first row of the table looks like as a real alert rule. This example assumes a 99.9% SLO, so the allowed error ratio is 0.001. A recording rule computes the error ratio for each window. The alert compares two of them with the burn-rate threshold.

groups:
  - name: checkout-slo
    rules:
      # Error ratio over 1 hour. Repeat for 5m, 30m, 6h and 3d.
      - record: checkout:error_ratio:rate1h
        expr: |
          sum(rate(http_requests_total{route="/checkout",status=~"5.."}[1h]))
          / sum(rate(http_requests_total{route="/checkout"}[1h]))

      - alert: CheckoutErrorBudgetFastBurn
        expr: |
          checkout:error_ratio:rate1h > (14.4 * 0.001)
          and checkout:error_ratio:rate5m > (14.4 * 0.001)
        labels:
          severity: page
        annotations:
          summary: "Checkout is using error budget 14.4x too fast"
          runbook_url: "https://wiki.example.com/runbooks/checkout-errors"
          dashboard: "https://grafana.example.com/d/checkout"

Alertmanager then routes on the severity label: page goes to the pager, ticket goes to the team's queue.

Alert fatigue#

Alert fatigue happens when people get so many alerts that they stop reacting to them. A real outage then gets the same slow response as the noise. Fatigue is a reliability risk, not only an annoyance.

To keep the pager quiet:

1. Review every page after each on-call shift.
2. Delete alerts that fired but needed no action.
3. Change alerts that could wait into tickets.
4. Group related alerts, so one failure sends one page, not twenty.
5. Fix flaky alerts at once; don't let the team learn to ignore them.
6. Track pages per shift as a team health metric.

Runbooks#

The actionable checklist above asked for a runbook link. A runbook is a short document that tells the responder what to do for one specific alert. Put its link in the alert itself, so the responder doesn't need to search at 3 a.m. A useful runbook covers:

• what the alert means and what users see
• which dashboards and log queries to check first
• safe mitigation steps, such as roll back, disable a flag or fail over
• when and to whom to escalate

Update the runbook after every incident where it was wrong or missing a step.

Dashboard layout#

The alert gets someone to look, and the dashboard is where they look first. A good dashboard answers "is it broken, and where?" in a few seconds. Put the most important signals at the top and detail lower down. Each panel should answer one question.

Top row: the golden signals#

The four golden signals from the Google SRE Book are a strong default for the top row of a service dashboard.

Add an SLO panel next to them: current SLI, error budget left and burn rate.

Lower rows: drill-down#

Below the top row, break the same signals down so you can find where the problem is:

• per endpoint or route
• per region, zone or host
• per dependency: database, cache, downstream APIs
• deploy markers on every graph, so you can see if a change lines up with a spike

Link each panel to related logs and traces. Keep dashboards few and owned: a stale dashboard with broken panels is worse than none during an incident.

Key takeaways#

• Page on symptoms users feel; keep cause metrics on dashboards for diagnosis.
• Every page needs a human action, a runbook link and a dashboard link; otherwise make it a ticket or delete it.
• Multi-window burn-rate alerts tie paging to the SLO and catch both fast and slow budget loss.
• Alert fatigue is a reliability risk; review and prune alerts after every shift.
• Put the golden signals at the top of each dashboard and drill down by route, region and dependency below.