SLO + alerts

This workflow turns "checkout is slow sometimes" into a measurable promise and the alerts that defend it. Time: about 15 minutes per SLO.

Step 1 — Decide what to measure

Pick one user-visible signal per service. Common choices:

You can have more than one SLO per service, but at most one of each kind. Adding "availability" and "latency" SLOs to one HTTP service is normal; adding three different latency SLOs to the same service is over-engineering.

Step 2 — Pick the target

Don't pick 100%. Don't pick something you can't measure.

A common starting place for a user-facing HTTP API:

• Availability: 99.9% success rate, 30-day window. Error budget: 43 m/month.
• Latency: 99% of requests under 500 ms, 30-day window. Error budget: 7 h 12 m/month.

For an internal API, 99% / 99.5% targets are more honest. The point of an SLO is that the team can hit it almost all the time; if it pages constantly, the target is wrong, not the team.

Step 3 — Create the SLO in Orbtrace

1. 1

   Open Admin → SLOs

   Sidebar → Admin → SLOs. Click + Add SLO.

2. 2

   Pick the service

   Dropdown of services. Pick checkout-api.

3. 3

   Pick the kind

   Latency p99 or Success rate.

4. 4

   Set the target and window

   Latency: < 500 ms. Window: 30 days.

   The page shows a live preview of what your error budget would have been over the last 30 days had this SLO been in place. Use that to sanity-check the target — if you would have burned 12× the budget last month, your target is too tight.

5. 5

   Save

   The SLO appears in the list. From here on it powers:

   • The colour of the service detail KPI tiles.
   • The colour of spans in the trace waterfall.
   • The SLO burn-rate alerts you'll wire up in step 4.

Step 4 — Wire the four-window burn-rate alerts

Burn-rate alerts answer "how fast am I eating my budget?". Four windows catch four kinds of incident:

1. 1

   Open Alerts → New

   Sidebar → Alerts → + New rule.

2. 2

   Pick the SLO mode

   A toggle at the top says Threshold | SLO burn rate. Switch to SLO burn rate.

3. 3

   Pick the SLO

   Dropdown lists every SLO you've defined. Pick the one from step 3.

4. 4

   Confirm the four windows

   Defaults are good. Each window has its own severity defaulted to P0 / P1 / P2 / P3. Override if you need to.

5. 5

   Set labels

   At minimum: team=payments (or whichever team owns this), service=checkout-api. Routing uses labels to find the right channel.

6. 6

   Set annotations

   summary: "Checkout-api is burning its latency SLO". runbook_url: link to your team's runbook. The annotation values appear in every Slack/PagerDuty notification.

7. 7

   Save

   The rule appears in Alerts → Rules, one row per window. Each window is independently silenceable.

Step 5 — Make sure it can reach a human

Once. Right now. Click Channels → your team's Slack channel → Send test message. Verify the message lands.

Repeat for PagerDuty if you're paging on P0/P1.

Then click Routing → "Test routing" → enter labels team=payments severity=P1 → confirm it resolves to the channels you expect.

Step 6 — Watch for a week, then tune

Set yourself a reminder for one week from now. Then open Alerts → Analytics. For the new SLO:

• How many times did it fire?
• How many were real incidents?
• How many were silenced and forgotten?

If the answer is "more than 5 noise pages", widen the threshold or the for-duration. If the answer is "zero firings ever", you may have over-shot the target on the optimistic side.

Common mistakes

• SLO on a metric you can't move. "Total errors" is not an SLO; "error rate" is. The team needs a knob to turn.
• Window too short. 30 days is the standard. 1-hour windows produce alert spam.
• Multiple latency targets on one service. Pick one. Composite SLOs confuse on-call.
• No labels. Routing without labels routes everything to a default channel. Tag every rule with team and service.

Next: Run a Time-Travel Replay.