Alerts

Alerting in Orbtrace is broken into eleven subpages. They look like a lot — they're not. Read them in the order on this page; each builds on the one before.

The big picture

An alert rule says "if metric X crosses threshold Y for Z minutes, fire an alert". When an alert fires, it goes through routing which decides what channel it goes to (Slack, PagerDuty, email, webhook…). If nobody acknowledges within the escalation policy's first step, it goes to the next step. A schedule decides which person is on-call at the moment of the alert. A silence suppresses an alert for a fixed window. A maintenance window suppresses every alert for a planned outage.

That's it. The rest is plumbing.

Subpage 1 — Rules

The list of every alert rule in the system. Columns: name, severity, status (firing / ok / silenced / muted), the metric query, the for-duration, last-fired-at, "edit" / "duplicate" / "delete".

Anatomy of a rule

Multi-window, multi-burn-rate alerts

For SLO-based alerts, the rule editor offers an "SLO burn rate" mode that wires up the four-window Google-SRE-handbook style alert in one click: fast-burn (14× over 1h), short-burn (6× over 6h), slow-burn (3× over 24h), low-burn (1× over 3d). Use this for every SLO.

Subpage 2 — Channels

A channel is a destination — Slack workspace, PagerDuty service, email distribution list, generic webhook.

For each channel: name, type, the type-specific config (bot token, service key, URL), a "Send test message" button. Channels are the what; routing is the which.

Subpage 3 — Routing

A tree of rules that maps (severity, labels) → channel(s). Read top-to-bottom; first match wins.

Example tree:

- match: severity=P0
  channels: [pagerduty-primary, slack-incidents]
- match: severity=P1 AND team=payments
  channels: [pagerduty-payments]
- match: severity=P1
  channels: [slack-incidents]
- match: severity=P2
  channels: [slack-warnings]
- default:
  channels: [email-sre]

The page shows the tree, lets you drag rows to reorder, and provides a "What channels does an alert with labels X go to?" tester at the top — paste labels, see the routing decision.

Subpage 4 — Escalation

For each severity (or for a specific routing branch), define an N-step escalation policy: "notify channel A; if not acked in 10 minutes, notify channel B; if not acked in 20 minutes, notify channel C".

Each step references a channel and a wait time. Steps run in order; the chain stops the moment someone acks. P0 typically has three steps; info-only alerts have none.

Subpage 5 — Schedules

Define on-call rotations: a team, a list of people, and how the rotation rolls (weekly handoff, daily handoff, custom). Schedules feed escalations: a step can reference "the on-call from schedule X" instead of a specific person.

The page renders the rotation as a calendar — each person's shift coloured. You can override (e.g. swap a shift, take a vacation) right on the calendar.

Subpage 6 — Silences

A silence suppresses one or more alerts for a fixed window. Each silence has:

• A matcher (labels). E.g. service=checkout-api AND severity=P2.
• A window — start and end timestamps.
• A reason — free text. Required.
• The author.

Silences exist for "I know about this, stop paging me until I fix it". They auto-expire — you cannot create a permanent silence.

Subpage 7 — Maintenance windows

Like a silence, but at the service/dashboard level and with a calendar shape. Use for planned outages (database upgrades, network maintenance). During a maintenance window, every alert from the affected services is suppressed and incidents from them are auto-tagged maintenance.

The page renders maintenance windows on a calendar (multi-month) so you see clashes.

Subpage 8 — Audit

Every action that mutates an alerting object — create rule, edit channel, fire silence, ack alert, resolve incident — produces an audit row. Columns: timestamp, actor, action, target, before/after diff.

Filter by actor, by action type, by target. Use for "who silenced the checkout alert?" post-mortems.

Subpage 9 — Analytics

Aggregate statistics about your alerting health:

• Volume per channel over time.
• Top-N noisy rules (most firings per week).
• Median MTTA (mean time to acknowledge) by team.
• Alert-fatigue heatmap — a per-rule × per-week colour grid; a rule firing every week with no resolutions is paging fatigue waiting to happen.

If a rule sits in "top 5 noisiest" for three weeks running, it's almost always badly tuned. Fix the threshold or kill the rule.

Subpage 10 — GitOps

Lets you version-control your alert rules in a Git repository.

• Connect a repo via SSH key or GitHub App.
• Orbtrace writes a YAML representation of every rule to the repo on change.
• Edits made in the Git repo are reconciled back into Orbtrace.

If your team prefers PR-reviewed alert changes, turn this on; otherwise leave it off.

Subpage 11 — New

The new-rule editor. Two paths:

• From scratch — empty query, you fill it in. Same shape as the Metrics explorer.
• From a metric query — when you click "Create alert" on any metric chart, you land here with the query pre-filled.

The right side of the editor shows a live preview: the current value, the threshold line, and a small chart of the last 24 h with would-have-fired regions shaded rose. The preview is the cheapest way to catch a too-sensitive threshold before saving.

Recommended set-up order

For a new install, go through the subpages in this order:

1. 1

   Channels first

   Set up your Slack channel(s) and your PagerDuty service. You can't route without channels.

2. 2

   Schedules

   Build your on-call rotations.

3. 3

   Escalation

   Wire severities → escalation steps → schedules.

4. 4

   Routing

   Map labels to channels and escalations.

5. 5

   Rules

   Now add rules. Start with availability (is the service responding) and latency (is p99 within SLO). Add more only after you watch the first ones for a week.

6. 6

   Maintenance + Silences

   Use these reactively, not preemptively. Don't pre-silence things "just in case".

7. 7

   Analytics review

   Two weeks in, look at the analytics page. Tune the loudest rules.

Next: Incidents.