How the numbers are calculated
Exactly what every headline number on the Services and SLO screens means, the formula behind it, and the honest caveats — request rate, error rate, percentiles, health, burn rate, error budget, Apdex, saturation, and "deviating".
Every number Orbtrace shows is computed from your telemetry — there's no magic. This page is the reference for what each one actually means, so you never have to guess. Where a number is easy to misread, we say so plainly.
Traffic, errors, availability (RED)
These three "golden signals" on a service are computed over the inbound-request spans the service handled in the selected window — SERVER spans (synchronous requests it served) plus CONSUMER spans (message-driven work it picked up). The service's own outbound calls (CLIENT) and internal work (INTERNAL) are deliberately excluded, so these numbers mean what their labels say — exactly as Google's SRE golden-signals and Datadog/OTel define them.
- Request rateRequests per second — inbound request spans in the window ÷ window length. This is the real request rate (not span throughput): a request that fans out to ten DB/client calls still counts as one request, so it's directly comparable across services and to other APMs.
- Error rateErrored requests ÷ total requests, over inbound spans. A request counts as an error when its span status code is
STATUS_CODE_ERROR. A downstream call this service makes that fails (aCLIENTspan) does not by itself count — only the inbound request's own outcome does. - AvailabilitySuccessful requests ÷ total requests = 1 − error rate. This is request-success over the window, not time-based uptime. "99.9% available" here means 99.9% of inbound requests succeeded, not "down for 43 minutes this month".
A service that serves no inbound requests (a pure background worker, a cron job, a pure client) has no request traffic by definition, so it shows an empty RED rather than a number invented from its internal spans.
Percentiles (p50 / p95 / p99)
Latency percentiles are computed over the inbound-request span durations (the same SERVER + CONSUMER spans as the RED signals above), using an approximate-quantile sketch (Doris percentile_approx on raw spans, or the equivalent quantile-union on the rollup tables) at a fixed compression of 2048. The same basis and compression are used everywhere — the service list, the detail page, the waterfall baselines, the SLO/burn-rate engine, and the morning-brief metrics — so every p95 you see for a service agrees. p95 here is the latency distribution of the requests the service served, not of its internal/outbound spans.
Health (the per-service colour)
The health pill is the first match, top to bottom, of this cascade. Thresholds are tunable per service/tier; the defaults are shown.
- CRITICALEither the service is breaching its SLO (severity > 1) and has less than 7 days of error budget left, or it's been silent for 30+ minutes. Note these are two very different situations sharing one colour — a totally silent service shows CRITICAL even with zero errors.
- BURNINGBreaching its SLO (p95 over target) but with 7+ days of budget left, or runway unknown.
- ERRORINGRequest error rate ≥ 1% (the inbound-request error rate above — a log-only error number shown elsewhere does not drive this).
- SILENTEmitted in the last 24h but nothing in the last ~15 minutes (and under 30 min, or it would be CRITICAL).
- DEVIATINGA traffic or latency deviation fired (see below) without any threshold breach.
- HEALTHYLive traffic, no breach, no errors, no deviation.
- UNKNOWNNo live trace signal — e.g. an SLO is defined but the service isn't currently sending.
SLO target and severity
Every service has an effective SLO even if you never set one, resolved in this order: an explicit per-service contract → the namespace-tier default → the global default. The source is shown on the row (an explicit contract reads differently from a "(default)" one).
Severity = current p95 ÷ target p95. Above 1.0 means p95 is over target (e.g. 1.5 = 50% over). Severity is a latency-overshoot ratio — it is not the same as burn rate or error budget, and because a global default target always exists, a busy service can show BURNING against a default you didn't explicitly set. If that surprises you, pin an explicit SLO.
Burn rate (multi-window)
Burn rate answers "how fast am I spending my error budget?". For each window:
exceedance fraction = (requests slower than the p95 SLO target) ÷ (requests in window)
burn rate = exceedance fraction ÷ 0.05The 0.05 is the error budget implied by a 95% objective — so a burn of 1× means you're spending budget exactly as fast as allowed, 2× burns a 30-day budget in 15 days, 14.4× burns it in ~2 days. The four windows trip at: 5m and 1h → 14.4× (fast), 30m and 6h → 6× (slow). A page fires when 1h and 5m both trip; a ticket when 6h and 30m both trip (the dual-window combo suppresses single-window false positives — this is the Google SRE Workbook pattern).
One caveat to know: the 0.05 budget is fixed to a 95% (p95) objective, so burn rate always measures against your p95 target even if you also set a p99.
Error-budget runway ("6.1d budget left")
Runway estimates how many days of budget remain at the current pace, computed as roughly window ÷ burn rate and capped at 365 days. It's only shown for breaching services. Orbtrace uses the most precise model your data supports — a remaining-budget model off the trailing 30-day distribution when available, falling back to a simpler full-window model, falling back to a linear 30 ÷ (severity − 1) heuristic when there's no live exceedance sample. The fallbacks can give somewhat different numbers, so treat runway as a directional signal ("hours" vs "weeks"), not a precise countdown.
Apdex
Apdex is a 0–1 user-satisfaction score:
Apdex = (satisfied + tolerated / 2) ÷ totalover the window, where a request is satisfied at ≤ T, tolerated between T and 4T, and frustrated above 4T. Bands: ≥0.94 excellent, ≥0.85 good, ≥0.70 fair, ≥0.50 poor, below that unacceptable.
The important detail: Orbtrace sets T = your p95 SLO target, not a separately chosen Apdex threshold. So Apdex here is tightly coupled to your SLO — a healthy service sits near the top of the scale by construction, and a service with no explicit SLO is scored against whatever default p95 applies. Read Apdex as "satisfaction relative to your p95 SLO", not an independent benchmark.
Saturation (CPU / memory)
Saturation reads process/runtime resource metrics your apps emit, picking the first available source in this order: container → Kubernetes pod → process → language runtime → host. The gauge is usage ÷ limit where a limit metric exists (container/pod tiers). Runtime tiers (JVM heap, Go memory, …) often have no companion limit, so those show usage without a saturation percentage. The panel shows which source it resolved from — the same "memory" gauge can mean usage/limit on one service and bare usage on another, depending on what that service emits. CPU is shown as a ratio (utilization tiers) or in cores (counter-derived tiers).
"Deviating" vs anomaly alerts
Two different systems use the word "anomaly" — they are not the same:
- The services-list DEVIATING badge comes from a simple ratio check: the last 60 minutes vs the preceding 24-hour baseline, flagged when latency-p95 or request-rate is ≥ 2× (spike) or ≤ 0.5× (drop). It's deliberately crude and can fire on normal daily ramps.
- The anomaly engine that feeds RCA and alerting is a separate z-score / seasonal model (
|observed − mean| ÷ stddev). It's more rigorous and is what drives Incidents and anomaly alerts.
So a "DEVIATING" service isn't necessarily a statistical anomaly — it just moved a lot versus yesterday.
A note on "nines"
When availability is displayed with extra decimals (99.95%), that's a display-precision convention, not a status threshold — there is no hidden "three nines" verdict. And remember from RED above: this availability is request-success over the window, not time-based uptime.