Run a replay

The first time you run Replay, the concept feels slippery. This page walks through a worked example, end to end, so the abstraction sticks.

The scenario

It's Tuesday 14:32. Yesterday at 14:02 you had a P1 incident: checkout-api p99 latency jumped from 220 ms to 2.1 s for 14 minutes, and rolled back when deploy a8f21b was reverted. Your team's post-mortem is at 15:30. The question on everyone's mind: "would skipping the deploy have prevented the incident?". Replay answers it.

Step-by-step

1. 1

   Open the incident

   Sidebar → Incidents → click yesterday's checkout-api incident. The detail page opens.

2. 2

   Click "Open in Replay"

   Top-right of the incident page. The Replay editor opens with the anchor pre-filled to the incident's start time.

3. 3

   Verify the actual outcome

   The "Actual outcome" field is pre-filled with p99(http.server.duration{service=checkout-api}). That's the metric the incident broke. Leave it.

4. 4

   Write the counterfactual

   Counterfactual field: paste What if deploy a8f21b had not been merged? and press Enter. The page recognises the deploy ID (because the deploy was captured by the CI webhook) and adds a chip you can click for the deploy's commit / author / changes summary.

5. 5

   Curate the context

   Below the counterfactual is a list of context items Orbtrace has gathered:

   • Deploy a8f21b (the one we're asking about — leave checked).
   • Deploy e3c440 from 2h before (leave unchecked — orthogonal change).
   • Three similar past incidents on checkout-api (leave checked — they teach the model what "looks normal").
   • Feature flag flip on discount-engine 6 min before incident (uncheck — unrelated).
   • The actual incident's RCA paragraph (leave checked — it gives the AI the original analysis).

   The more focused your context, the higher Replay's confidence. Be ruthless.

6. 6

   Run

   Click Run replay. A spinner appears for ~15 seconds while the LLM calls the provider.

7. 7

   Read the side-by-side chart

   Two charts stacked. Top: the actual p99 latency that day — flat at 220 ms, then a spike at 14:02 up to 2.1 s, returning at 14:16 when the deploy was rolled back.

   Bottom: the hypothesised p99 — flat at 220 ms the whole time, with a thin shaded confidence band around it.

   Visually, the spike disappears. That's the answer.

8. 8

   Read the probabilistic summary

   At the top: "Probably (87% confidence) checkout-api p99 would have stayed under 280 ms during the 14:02-14:16 window if deploy a8f21b had not been merged."

   87% is high. The number is high because the AI found three similar past incidents where reverting a deploy returned the metric to baseline — past behaviour informs the prediction.

9. 9

   Read the reasoning paragraph

   Two paragraphs that explain the AI's logic:

   "Deploy a8f21b modified OrderRepository.findByUser by removing an index hint that previously forced a covering index scan (cited spans b3f2…, c81a…). Three structurally similar past incidents on checkout-api (id-203, id-188, id-141) involved index-related deploy changes; in all three, reverting the deploy returned p99 to within ±10% of baseline within two minutes. The remaining 13% confidence is reserved for the possibility that an upstream change (not in your captured deploy log) coincidentally caused the latency increase."

10. 10

    Check sensitivity

    The sensitivity table at the bottom: "Removing 'similar past incidents' context drops confidence to 64%. Removing 'RCA paragraph' context drops confidence to 78%."

    The first row tells you the past incidents are the load-bearing context. Worth knowing.

11. 11

    Save and attach to post-mortem

    Click Save replay. It gets a name and a URL. Paste the URL into the post-mortem doc.

    Done — total elapsed time: about 4 minutes.

What to do if the chart and the summary disagree

If the bottom chart still shows a spike but the summary says "the incident would not have occurred", trust the chart — read the reasoning paragraph for what the model is hedging on. Usually the model is being cautious about the magnitude but agreeing on the direction.

If the result is "the counterfactual would not have helped" (no visible difference in the bottom chart), that's a real finding — it means the deploy was not the root cause. Your post-mortem just shifted scope.

How to ask a different counterfactual

Common rephrasings worth trying:

• "What if checkout-api had been scaled to 6 replicas instead of 3?" — tests capacity counterfactuals.
• "What if the feature flag new_discount_logic had been off?" — tests flag-based counterfactuals.
• "What if traffic had been 2× higher during this window?" — tests load counterfactuals (useful before peak events).

Each is a separate replay. Each costs $0.10-$0.40. Run all three if it's a serious post-mortem.

When NOT to run Replay

• You haven't even finished investigating the actual incident — Replay's value comes from comparing real and counterfactual. Finish the real first.
• The incident is older than a month — telemetry may have been dropped by retention; the AI has less context. Confidence will drop.
• You're just curious, no post-mortem deadline — Replay costs money. Use it when the answer changes a decision.

Next: Search syntax.