Traces
How to find the right trace in the list (histogram + columns + autocomplete), then how to read the waterfall and span attributes once you've opened one.
If Logs answer "what did this one service say?", traces answer "where did the whole request spend its time?". This page covers both halves: the trace list and the trace detail.
The trace list
The page opens with a search bar, a histogram, a column-rich results table, and a detail drawer that opens on click. Reads almost identically to the Logs screen — only what's plotted is different.
Search bar
Same syntax as everywhere (Search syntax). The autocomplete suggests trace-specific fields:
service:checkout-api— only traces whose root span belongs to this service.service.in:checkout-api— only traces that touch this service anywhere.duration:>1000— trace duration in milliseconds.status:error— traces with at least one ERROR-status span.operation:POST /checkout— root operation.http.status_code:>=500— works on any span attribute.
Histogram
A 28-bucket bar chart. Two stacked layers:
- Total traces (cyan)Count of matching traces per bucket.
- Error traces (rose, on top)Of those, how many had at least one error span.
A growing rose layer is the signature of an outage. Drag-select to zoom.
Columns in the list
- TimestampOf the root span. Local TZ.
- ServiceRoot service.
- OperationRoot operation name (e.g.
POST /checkout). - DurationTotal wall-clock time from root start to root end. Rose if it exceeded the per-service SLO target.
- SpansCount of spans in this trace.
- Status
OK(emerald),ERROR(rose),UNSET(slate). - Trace IDThe 32-char hex ID. Click to open the trace; click the copy icon to put it on your clipboard.
Columns are sortable and customisable; you can pin any span attribute as a column.
The waterfall — what it is, how to read it
Click a row → the detail page opens, dominated by a tall waterfall chart on the left.
How the waterfall is built
Every span is one horizontal bar:
- Vertical position — depth in the call tree. Children sit below their parent.
- Horizontal position — span start time, on a shared time axis.
- Bar length — span duration.
So a 3-second trace spans a 3-second-wide chart. A child that started 100 ms into its parent and lasted 50 ms sits 100 ms in and is 50 ms wide. The shape of the waterfall is the shape of where the request spent its time.
Bar colours
- EmeraldSpan succeeded, duration inside the service's p95 baseline.
- AmberSpan succeeded but is slower than the service's p95 baseline — "tail latency".
- RoseSpan is slower than the p99 baseline OR has
status=ERROR. - SlateSpan has no recorded baseline yet — usually a new code path.
The baseline is computed per service.name + operation over the last 7 days — see the per-service settings under Admin → SLOs to override.
Bar markings
- A small flag icon at the start of a bar = the span has logged "events" attached (e.g. an exception). Hover to read them inline.
- A dashed line from a child to its parent = the link was reconstructed by Async Stitching (the parent was inferred from queue messages, not from a direct call). Hover to see the stitching confidence percentage.
- A 🔗 icon at the end of a bar = there's a "follows-from" link to another trace.
Controls above the waterfall
- Zoom sliderCompress or expand the time axis. Useful for very wide traces.
- Collapse all / Expand allToggle every subtree open/closed.
- Show only errorsHides every emerald and amber span, leaves rose only.
- Show only slowHides every span faster than p95.
- Critical pathHighlights the path that contributed the most to total duration. The rest of the waterfall dims.
For very large traces (> 1000 spans), only the first 1000 spans render at once; a "load next 1000" button is at the bottom. The waterfall is virtualised so even 50k-span traces stay responsive.
Selecting a span
Click any bar. The right side of the page becomes the span detail drawer.
The span detail drawer
- HeaderService · operation · duration · status pill · span ID (copyable).
- TimingA small chart with three numbers: self time (span minus children), child time, queue wait time.
- AttributesTwo-column table of every attribute. Filter pencil on each row.
- EventsLogged events on this span (exceptions, custom checkpoints). Each event shows its relative offset from span start.
- LinksOther spans this span references (e.g. follows-from links across traces). Click to jump.
- Cross-pillar linksButtons "View logs from this span" (opens Logs filtered by
trace_id+ the span's time window) and "View metrics from this service" (opens Metrics for the service).
How to read a waterfall fast
A trained eye scans a waterfall in seconds. The pattern matters more than the numbers.
- One wide bar at the top, many short belowThe root operation is slow on its own — likely CPU work in the entry service.
- One wide child bar inside an otherwise tight parentA specific downstream is the bottleneck — read the wide child's attributes (probably
db.statementorhttp.url). - Many narrow bars arranged in a stairstepSequential calls that should have been parallel — a classic N+1 query pattern.
- A long gap between a parent's end and the previous child's endThe parent is waiting on something — a lock, a sleep, a queue. Look at the gap, not the bars.
- A burst of rose at the bottom of the waterfallA leaf service is failing; the rose colour propagates upward as errors bubble up.
Cross-pillar pivots
Most investigations end with "what was happening in logs / metrics at the same time?". From any open trace:
- Top of page → "Open logs window" — opens Logs filtered by
trace_idand the trace's full time window. - Span drawer → "Open service detail" — opens the service detail for that span's service, time picker pre-set to the trace's window.
- Top of page → "Find similar" — runs a pgvector embedding search over recent traces and shows the 10 most similar. Useful for "is this incident a recurrence of something we've seen?".
What this screen is not
- It is not a profiler. Spans show wall-clock time, not CPU samples. For "where did my CPU go?" use language-specific profiling.
- It is not a request log. It records the spans your apps chose to emit; un-instrumented code is invisible. Add custom spans (see Instrument your apps) where the gaps are.
Next: Metrics.