Incidents & RCA
When an alert fires, an incident opens. Causal RCA writes a paragraph that says what caused it, with span citations. This page explains every part of the incident screen.
An incident in Orbtrace is a record of "something went wrong". It collects the alerts, the affected services, the traces, the logs, the deploys, and — the headline feature — the Causal RCA paragraph: a grounded AI explanation of why, with span IDs you can click and verify.
How an incident gets opened
Three ways:
- Automatically from an alert — the most common. When an alert rule fires, an incident is created and the alert is attached. If a related alert fires within 10 minutes for the same service, it attaches to the same incident instead of opening a new one.
- Automatically from anomaly detection — Orbtrace's anomaly detector watches every service's golden signals. A novel anomaly that doesn't match an existing alert can still open an incident.
- Manually — click "Open incident" on the incidents list or from any drawer. Use for "I'm investigating something the system didn't catch".
The incidents list
A table at /incidents, one row per incident. Columns:
- SeverityInherited from the highest-severity attached alert.
- Status
open(red),mitigating(amber),resolved(green),cancelled(slate). - TitleAuto-generated from the first attached alert, editable.
- ServicesAffected services as pills.
- Started / DurationWhen it began, how long it's been open.
- On-callWho from the schedule is responding.
- RCAA small badge: 🤖 if RCA has produced an analysis, ⏳ if it's still running, ⏸ if RCA is disabled.
Filters: by status, severity, service, date range, has-RCA.
The incident detail page
Click any row → the detail page. Top to bottom:
① Header
- Severity pill, status pill, title (click to rename).
- Started-at, current duration.
- Buttons: Acknowledge (changes status to mitigating), Resolve, Cancel (it was a false positive), Export post-mortem.
② Timeline
A vertical timeline of every event attached to this incident:
- Alert fired.
- Alert acknowledged by Alice.
- New alert fired (attached).
- Deploy
a8f21brolled back. - Status changed to
mitigating. - RCA produced.
- Status changed to
resolved. - Post-mortem exported.
Click any event for the underlying detail. The timeline is the canonical "what happened, in order".
③ Causal RCA panel — the headline
A wide panel containing the AI-written analysis. Structure:
- SummaryOne sentence. "checkout-api p99 latency jumped from 220 ms to 2.1 s at 14:02; the change correlates with deploy a8f21b which modified
OrderRepository.findByUser." - EvidenceA bulleted list. Each bullet cites at least one span ID (clickable) and at least one log line (clickable).
- Probable root causeA more detailed paragraph explaining the AI's best hypothesis, with the same span/log citations woven in.
- Recommended next steps"Roll back deploy a8f21b" / "Re-introduce the index hint" / "Check the JVM heap on checkout-api-3".
- ConfidenceA percentage. Below 60% Orbtrace adds the disclaimer "Low confidence — verify before acting".
- CostUSD spent on this RCA call. Visible to administrators.
Every span ID in the RCA text is clickable — it opens the trace detail page with the cited span focused. Every log link does the same for logs.
If you disagree with the RCA — say it cites the wrong cause — click "Re-run RCA with feedback". You can add a one-paragraph correction; the AI runs again with your feedback in the prompt. The corrected analysis replaces the original.
④ Affected services
Each affected service shows: current health pill, the alert(s) firing on it, links to its service detail.
⑤ Attached alerts
The list of alerts that landed on this incident. Each row: severity, query, value, fired-at, ack'd-by.
⑥ Traces panel
A pre-filtered list of traces from the incident's window, sorted slowest first. Click any to open the trace drawer.
⑦ Logs panel
A pre-filtered list of ERROR-level logs from the incident's services and window. Same drawer as Logs.
⑧ Deploys panel
Every deploy across affected services in the 24 h before the incident started. If RCA suspects a deploy, that row is highlighted.
⑨ Notes
A text area for human commentary. Markdown supported. Use during the incident for "I just rolled back X". Notes appear in the timeline.
⑩ Post-mortem export
Click Export post-mortem to render a Markdown document containing: title, severity, duration, timeline, RCA, notes, attached charts (PNG snapshots). Drop into Confluence / Notion / GitHub Wiki.
Anomaly settings (a detour)
The anomaly detector that occasionally opens an incident on its own has settings under Admin → Anomaly:
- Per-service sensitivity (low / medium / high).
- Per-metric thresholds for "novel" — typically STL decomposition residual > N standard deviations.
- A list of historical false positives the detector should not flag again.
Tune these only after a week of running on real traffic.
Tips
- Acknowledge fast, then investigate. An open-and-unack'd incident is the most expensive thing on the screen — the on-call clock ticks against MTTA.
- Trust but verify RCA. It cites spans for a reason. Click them. The 1-in-20 wrong RCA is faster to spot if you train the muscle of clicking on every citation.
- Write the notes as you work, not after. Future-you (and your post-mortem) will thank you.
- Export the post-mortem before resolving. Once you resolve, the time pressure drops and you'll forget half the timeline.
Next: Time-Travel Replay.