Investigate an incident

This is the canonical investigation flow. Memorise it once. You'll spend most of your Orbtrace life in this loop.

1. 1

   Acknowledge from the notification

   The notification (Slack message, PagerDuty alert, email) contains a deep link directly into the incident detail. Click it. The page opens with time picker already set to the incident window.

   Click Acknowledge at the top. MTTA stops.

2. 2

   Read the RCA panel — verify before acting

   The Causal RCA panel near the top of the incident has the AI's hypothesis. Read it. Then click at least one cited span to verify the AI didn't hallucinate. Span IDs are clickable.

   If RCA looks right → skip to step 5. If RCA looks wrong → continue, you'll do the investigation manually.

3. 3

   Look at the affected services

   Scroll to "Affected services". Click the worst one — that opens its service detail.

   Read the four charts in this order: RPS, then latency, then error rate, then saturation.

   • RPS unchanged + latency up + errors up → downstream is the cause.
   • RPS unchanged + latency up + errors flat → resource exhaustion (CPU/GC/lock).
   • RPS down + errors up → the service is failing to accept work; check upstream.
   • RPS up + latency up + errors flat → traffic spike, maybe legitimate, maybe a retry loop.
4. 4

   Drill into the suspect

   Two paths, depending on what you saw:

   Path A — slow downstream suspected. Scroll the service-detail page to the "Operations" table. Sort by p99 desc. The top row is the suspect endpoint. Click it → operation detail → scroll to the "Downstream calls" panel → the top row is the culprit downstream. Click it. That opens the downstream service's detail page. Repeat.

   Path B — resource exhaustion suspected. From the service detail, click the saturation chart. If CPU is pinned at 100%, look at the service's deploy timeline — recent deploys often introduce a hot loop. If memory is steadily climbing without recovering, you have a leak. Pivot to logs: filter service:<name> AND level:ERROR over the incident window. The pattern clustering view will surface "NEW" templates — those are usually the unhandled paths the new deploy introduced.

5. 5

   Confirm with a trace

   Even when you think you know the answer, open one real slow trace. The trace waterfall is the single most truthful artefact in the product — it shows what one request actually did.

   Look for: the longest bar (where time went), the bar with status=ERROR (where it broke), the dashed lines (where async links were stitched). The shape of the waterfall is the story of the request.

6. 6

   Decide on mitigation

   Mitigation is stop the bleeding, not fix the cause. Common mitigations:

   • Roll back the suspect deploy. Click the deploy chip on the service-detail page → "Open in CI" → roll back.
   • Scale the suspect service. If saturation is the cause, more pods help.
   • Disable a feature flag. If the new code path is gated, flip the flag off.
   • Drain a degraded node. If one Kubernetes node is hot, cordon and drain.

   Capture what you did in the incident's Notes field. Future-you will need this.

7. 7

   Watch the recovery

   Open the service detail page and watch the four charts in real time. Set the time picker to "Last 15 m" and turn auto-refresh on (30 s).

   When the metric you broke — usually p99 latency or error rate — drops back inside SLO and stays there for at least one observation window, you can mark the incident mitigating → resolved.

8. 8

   Export the post-mortem

   Click Export post-mortem on the incident. You get a Markdown doc with timeline, RCA, notes, attached charts. Drop into your post-mortem tool. Optionally — and powerfully — also run a Time-Travel Replay of the counterfactual ("what if we hadn't deployed?") and attach the result.

9. 9

   Tune what made this hard

   After the dust settles, look at the Alerts → Analytics page. Did the right alert fire? Did it fire too late? Too early? Adjust the threshold. Did you discover a missing alert? Create it now — not next quarter.

   If RCA was wrong, click "Re-run RCA with feedback" and tell the AI what it missed. The correction trains future RCAs implicitly via the embedding store.

When the AI's RCA looks suspicious

Two red flags:

• No span citations — RCA is supposed to ground every claim in evidence. If the summary has no clickable IDs, the prompt didn't retrieve enough context.
• Confidence under 60% — Orbtrace flags this itself with a "Low confidence — verify before acting" banner. Don't act on a low-confidence RCA without independently confirming it.

In either case, fall back to the manual investigation steps (3–5) above.

Time-to-resolution averages, by how you investigated

These are observed numbers from beta deployments — your mileage will vary:

• Following the RCA's pointer when RCA was right → median 6 minutes.
• Manual investigation when you know the system well → median 14 minutes.
• Manual investigation when you've been on the team less than 3 months → median 38 minutes.

The difference between those last two is mostly knowing where to click. This page exists to compress that.

Next: Define an SLO and an alert for it.