SLO-based alerting that does not burn out a small on-call team

The on-call rotation for a small team is fragile. With four or five engineers sharing it, each person is on call one week in four or five, and a noisy week costs more than sleep. It costs trust in the alerts. Once engineers learn that most pages resolve themselves, they start acknowledging and waiting, and the one page that matters gets the same treatment as the forty that did not. The fix is not better thresholds on server metrics. It is alerting on what users experience, using service level objectives.

Why server metrics make bad pages

A typical inherited alert configuration pages when CPU is above 80 percent for five minutes, when memory is above 90 percent, when disk is above 85 percent and when any health check fails. Each of those sounds reasonable. In practice:

• High CPU during a nightly batch job is normal and harmless.
• Memory near its limit is how many runtimes and databases are designed to work.
• A single failed health check from one monitoring location is often a network blip on the monitor's side.

Meanwhile, the incidents that actually hurt users, such as a checkout endpoint returning errors for ten percent of requests or search latency tripling after a deploy, may not trip any of these thresholds at all. Server metrics are causes. Users feel symptoms. Page on symptoms, and use causes for diagnosis.

Choosing indicators

A service level indicator is a measurement of user experience expressed as a ratio of good events to total events. For most web applications, two kinds cover the majority of what matters:

1. Availability. The proportion of requests that succeed, typically counting any 5xx response or timeout as a failure.
2. Latency. The proportion of requests served faster than a threshold, for example 95 percent of page requests under 800 milliseconds.

Measure them at the edge, from load balancer or CDN logs, so that you capture what users see including failures that never reached your application. Split them by user journey rather than by server: checkout, sign-in, search and the API used by the mobile app might each deserve their own indicator, because a failure in checkout is worth far more attention than a failure on the careers page.

Setting objectives you can defend

An objective is the target for an indicator over a window, for example 99.9 percent of checkout requests succeed over 28 days. The number should reflect what the business needs, not what the infrastructure can theoretically achieve. Each additional nine costs substantially more in redundancy and engineering attention.

The difference between the objective and perfection is the error budget. At 99.9 percent over 28 days, you can afford about 40 minutes of total failure, or a longer period of partial failure. That budget is useful beyond alerting: when it is healthy, the team can ship aggressively; when it is nearly spent, releases slow down and reliability work takes priority.

An error budget turns an argument about reliability into arithmetic. Everyone can see how much room is left.

Burn-rate alerts

Instead of paging when the error rate crosses a fixed line, page when you are consuming the error budget too fast. The standard approach uses two windows to balance speed and noise:

• Fast burn. Page if the last hour's error rate would exhaust roughly two percent of the monthly budget, confirmed over a five-minute window. This catches outages quickly.
• Slow burn. Open a ticket, not a page, if the last six hours would exhaust around five percent of the budget. This catches slow degradations without waking anyone.

- alert: CheckoutFastBurn
  expr: |
    (1 - sli:checkout_success:ratio_rate1h) > (14.4 * 0.001)
    and
    (1 - sli:checkout_success:ratio_rate5m) > (14.4 * 0.001)
  labels:
    severity: page

The short confirmation window prevents a brief spike from paging, and the long window ensures that when an alert fires, users have genuinely been affected. When we moved one SaaS client from threshold alerts to burn-rate alerts, pages dropped from an average of 31 a week to four, and every one of the four corresponded to something a customer would have noticed.

Routing, runbooks and review

Fewer, better alerts only help if the ones that remain are actionable. We apply three rules:

1. Every paging alert links to a runbook explaining what it means, how to confirm the impact, and the first three things to check.
2. Cause-based alerts such as disk space or certificate expiry go to a ticket queue during working hours, unless they predict an outage within the next few hours.
3. Every page is reviewed weekly. If it was not actionable, the alert is changed or deleted that week.

The weekly review is the part teams skip, and it is the part that keeps the system healthy. Alerts accumulate naturally; only deliberate pruning keeps them useful.

Getting started without a platform team

You do not need a large observability stack to start. Most CDN and load balancer providers expose request logs with status codes and timings, and a managed metrics service can compute the ratios. A realistic first iteration for a small team is two journeys, two indicators each, fast and slow burn alerts, and a weekly review. That can be in place in two to three weeks.

Our uptime monitoring service sets up external checks and SLO dashboards, and for clients who would rather not carry a pager at all, our monthly care plans include our own on-call rotation working from the same objectives.

Quieter nights, clearer incidents

If your team is tired of alerts that do not mean anything, we can help you rebuild them around what your users experience. Send us an outline of your current monitoring and we will quote a fixed price within a day. Reach out here.