Span stitching
Tune the async-stitching engine that reconnects orphan spans — scheduler cadence, candidate windows, the confidence floor, and the three scorer weights. Plus the two independent ways to turn it off and which one to use.
The OpenTelemetry SDK propagates trace context automatically across direct calls, but queue-based hops lose it: a Kafka/SQS/RabbitMQ/Redis-Streams message crosses a service boundary and the consumer starts a brand-new trace with no parent. The result is an orphan span — and a hole in the trace waterfall where the producer→consumer link should be.
Span stitching closes that hole after ingest. A background job walks recent orphans, finds the producer span that most likely emitted each one by message-identity fingerprint + timing, and — when the match clears a confidence bar — writes the inferred parent into a companion table. The trace view then draws a dashed, stitched edge with the confidence percentage, so a reconstructed link is always visually distinct from a real propagated one. The same reconstruction is what fills missing edges on the service map.
You configure all of this at Admin → Span stitching. Every knob is live: a saved change is picked up on the engine's next tick, and the cadence itself reschedules without a restart.
Two ways to turn it off — and they're not the same
This is the single most common point of confusion, so read it before touching anything.
| Control | Where | Effect | Needs restart? |
|---|---|---|---|
ORBTRACE_STITCHING_ENABLED=false | .env (boot) | The engine never starts — no background thread, no lock, no scans, zero overhead. | Yes |
| Stitching enabled checkbox | Admin → Span stitching | A soft pause. The engine stays alive but skips each tick while off. | No |
Which one should I use?
Env var = is the engine installed. Checkbox = is it working right now. Use the env var to ship stitching fully off from infrastructure-as-code (e.g. a minimal 4 GB box that doesn't need it). Use the checkbox to pause and resume during an incident or a tuning experiment. The checkbox only has any effect when the env var is left at its default (on) — if the engine was killed at boot, the admin screen tells you so and disables the toggle.
Scheduler — how often, and how far back
How often the engine wakes, and how much history each pass scans.
| Knob | Default | What it does |
|---|---|---|
| Tick interval | 30s | How often the engine wakes to look for new orphans. Lower = stitched edges appear sooner, at the cost of more frequent scans. Floored at 1s to stop the scheduler hot-spinning. |
| Look-back window | 5m | How far into the past each tick scans for orphans. Must comfortably exceed tick + your ingestion lag, or an orphan that lands between two ticks can slip out of the window and never get stitched. If spans take a while to reach storage, widen this rather than shortening the tick. |
| Orphan scan limit / tick | 5000 | A safety cap on how many orphans one tick processes, bounding the cost of a single pass. If you consistently hit it, orphans are arriving faster than they're cleared — raise the limit or shorten the tick so the backlog doesn't age out of the look-back window. |
Candidate matching — which spans get considered
For each orphan, which producer spans are even eligible to be its parent.
| Knob | Default | What it does |
|---|---|---|
| Candidate window ± | 5s | The time bracket around the orphan's start to pull possible parents from. Widen it to catch slow async hops (a job that sat in a queue for seconds); but the wider it is, the more candidates are scored — more cost, and more room for a coincidental match. |
| Candidate limit / orphan | 200 | A hard cap on how many candidates are scored for a single orphan, bounding the worst case when a busy window holds thousands of spans. If a high-throughput service's links look missed, a wide window combined with this cap may be truncating the real parent out of contention — narrow the window first, then raise the cap. |
Confidence & scorer — what counts as a match
| Knob | Default | What it does |
|---|---|---|
| Confidence floor | 0.5 | The minimum composite score a candidate must reach before the engine writes the stitch. Higher = fewer but more trustworthy links; lower = more coverage, more risk of a wrong parent. |
| Clock-skew tolerance | 2s | Slack added to the temporal window to absorb host clock differences and ingestion jitter, so a parent whose clock is slightly off isn't wrongly excluded. Raise it when producers and consumers run on hosts you don't tightly NTP-sync. |
| Max overshoot | 5s | The largest gap allowed between the parent span's end and the orphan's start before the temporal score decays to zero. In plain terms: how long can the async hop reasonably take? Set it to the realistic upper bound of your queue dwell time. |
Rolling it out
Start with the confidence floor around 0.3, open a few traces that should have an async hop, and eyeball the stitched edges and their confidence badges. Once you trust the matches, raise the floor to cut noise. Tuning by looking at real traces beats tuning by the numbers alone.
Signal weights — how the confidence score is built
The composite confidence is a weighted average of three signals. The three weights must sum to 1.0 — they're a mean, so any other sum pushes the score outside the 0–1 range the floor compares against (the form blocks a save until they balance).
| Signal | Default | Meaning | When to raise it |
|---|---|---|---|
| Temporal | 0.50 | How well the timing lines up: the parent ends, the child starts shortly after, within the overshoot bound. | Low-throughput systems where timing alone is unambiguous. |
| Fingerprint | 0.35 | Match on message identity — message IDs, keys, and propagated headers. The strongest "literally the same message" signal. | When your transport reliably carries message IDs/headers. |
| Content | 0.15 | Softer corroboration from span shape: kind pairing (PRODUCER→CONSUMER), names, attribute consistency. A tie-breaker. | Rarely the primary lever; keep it small. |
What to watch after a change
The engine exposes Micrometer counters (visible in your metrics backend, and dogfooded into Orbtrace itself):
orbtrace.stitching.stitches— links successfully written. Should rise after lowering the floor or widening windows.orbtrace.stitching.rejects— candidate pairs scored below the floor. A high reject rate next to few stitches means the floor is too strict for your weights, or the windows are too narrow to reach the real parent.orbtrace.stitching.orphans— orphans found per tick. If this tracks the scan limit, you're capped.
But the decisive check is always the traces themselves: open a few that should have an async hop and confirm the dashed stitched edge appears with a sensible confidence percentage.
Where the links live
Stitched parents are written to a span_stitch companion table and joined into the trace tree at read time — the original spans are never mutated. That's why a stitched edge is always marked as inferred, and why turning stitching off (or reverting its config) never corrupts your raw telemetry.
Next: Admin pages.