Integration patterns

Orbtrace ships the product — the server, the UI, and the storage (Apache Doris). It does not ship an OpenTelemetry Collector. You run the Collector, and it writes your telemetry into Orbtrace's Doris.

This page is about how the Collector sits in your architecture. If you just want one service reporting in, start with Instrument your apps — that's the copy-paste path. Come back here when you're deciding the layout.

The mental model

Three facts that make the rest of this page easy:

• The Collector is otel/opentelemetry-collector-contrib. Not a fork, not our build. The only requirement is the contrib distribution, because the Doris exporter ships only there.
• The exporter is contrib's own doris exporter. It writes batches into Doris over Stream Load, Doris's high-throughput HTTP load API.
• Orbtrace owns the database schema, not the Collector. Orbtrace creates and migrates every table when it starts. The exporter must therefore run with create_schema: false — more on this below, because getting it wrong is the single most common mistake.

Which pattern fits you?

Both start from the same reference config that ships in your deploy bundle at compose/otelcol-config.yaml. It already contains the batching, queueing, sampling, and per-signal Doris wiring described on this page — copy from it rather than typing config by hand.

Pattern A — one Collector, greenfield

1. 1

   Get the config file

   The Collector needs a config file — otelcol-config.yaml. Don't write it from scratch: it ships ready-made in your Orbtrace deploy bundle, in the compose/ directory, with all the batching, queueing, sampling, and per-signal Doris wiring on this page already in place. The bundle is a free public download.

   Download the bundle, unpack it, and cd into its compose/ directory — that's where otelcol-config.yaml lives.

2. 2

   Run the Collector

   From that compose/ directory, start the Collector container next to the Orbtrace stack:

   docker run -d --name orbtrace-otelcol \
     --network orbtrace-net \
     -p 4317:4317 -p 4318:4318 \
     -v "$PWD/otelcol-config.yaml:/etc/otelcol-contrib/config.yaml" \
     --tmpfs /var/lib/otelcol/queue:rw,mode=1777 \
     otel/opentelemetry-collector-contrib:latest

3. 3

   Point your apps at it

   Set every service's OTEL_EXPORTER_OTLP_ENDPOINT to this Collector — http://orbtrace-host:4317. Your service appears in Orbtrace within seconds. Per-language SDK setup is on Instrument your apps.

Pattern B — add Orbtrace to a pipeline you already run

You already have a Collector, or a vendor agent that speaks OTLP. Don't replace it. Add Orbtrace as a second destination, so the same telemetry also lands in Doris. Your existing backend keeps receiving everything; Orbtrace gets a copy. This is the clean way to trial Orbtrace, or to migrate off another vendor by running both side by side.

Two hard requirements:

1. Your Collector must be the contrib build. The doris exporter ships only in otel/opentelemetry-collector-contrib.
2. Every Doris exporter must run with create_schema: false. Orbtrace already created the tables; a schema-creating exporter races the Orbtrace migrator and splits the schema.

Here is the minimal correct wiring. Everything Orbtrace-specific is in bold-comment lines:

exporters:
  doris/traces:
    endpoint: http://orbtrace-host:8030      # Doris FE — HTTP / Stream Load port
    database: orbtrace
    username: ${env:DORIS_USER}
    password: ${env:DORIS_PASSWORD}
    table: { traces: otel_traces }
    create_schema: false                     # REQUIRED — Orbtrace owns the schema
 
  doris/logs:
    endpoint: http://orbtrace-host:8030
    database: orbtrace
    username: ${env:DORIS_USER}
    password: ${env:DORIS_PASSWORD}
    table: { logs: otel_logs }
    create_schema: false
 
  doris/metrics:
    endpoint: http://orbtrace-host:8030
    database: orbtrace
    username: ${env:DORIS_USER}
    password: ${env:DORIS_PASSWORD}
    table: { metrics: otel_metrics }         # prefix — Doris stores five per-type tables
    create_schema: false
 
service:
  pipelines:
    traces:  { exporters: [your_existing_exporter, doris/traces] }
    logs:    { exporters: [your_existing_exporter, doris/logs] }
    metrics: { exporters: [your_existing_exporter, doris/metrics] }

This is the minimum. The production settings that make it fast and durable — batching, send queues, retries, Stream Load headers — are in The recommended production config below, ready to copy. Or just start from compose/otelcol-config.yaml, which already has them.

The Doris exporter, field by field

Every doris/* block shares the same connection fields. Here is what each one is for.

Why Orbtrace owns the schema

Orbtrace's tables are not the stock exporter tables. Its first migration creates the base tables with the exact DDL the exporter expects — same table model, same distribution, same column types — then layers Orbtrace's own columns on top: stitched-trace edges, anomaly scores, sampling decisions. That is what powers span stitching, RCA, and replay. If the exporter created the schema instead, you would get the plain tables and lose those features. If both tried, you would get two half-built schemas. Hence create_schema: false, everywhere, always.

One exporter per signal: traces, logs, metrics

The three signals are stored differently in Doris, so they load differently. This is the reason for three exporters.

This layout is not Orbtrace-invented — it is exactly what Apache Doris recommends for observability data in its own observability docs: an append-only DUPLICATE KEY model, RANDOM distribution, time_series compaction, VARIANT columns for attribute bags, and inverted indexes on the queried fields. Orbtrace ships that schema and adds its own columns on top.

Traces

The heaviest signal and the one with the most processing. Traces flow through tail sampling (keep every error and slow trace, sample the rest), then land in otel_traces.

otel_traces is DISTRIBUTED BY RANDOM, which makes load_to_single_tablet: "true" valid and worthwhile — each batch goes to one tablet instead of fanning out across all of them, which cuts write amplification and compaction pressure and raises effective write concurrency. This is the single biggest Stream Load throughput lever for traces and logs.

Logs

Same storage shape as traces — RANDOM, single-tablet load on. Logs get two extra processors you should keep:

• A timestamp fix. Some SDKs emit logs with a zero timestamp. Doris has no partition for 1970-01-01, so Stream Load rejects the entire batch, the retry queue fills, and healthy services' logs start failing too. The fix backfills the receive-time timestamp. Pattern B customers must copy it, or one buggy SDK anywhere in the fleet takes down log ingest.
• A severity filter. Below-INFO log noise is dropped at the Collector before it ever reaches Doris. Tune the floor with ORBTRACE_LOG_DROP_BELOW (an OTTL severity constant, default SEVERITY_NUMBER_INFO); an escape-hatch attribute keeps individual records past it.

Metrics

table: { metrics: otel_metrics } is a prefix: OpenTelemetry has five metric shapes (gauge, sum, histogram, exponential histogram, summary), and Doris stores each in its own table — otel_metrics_gauge, otel_metrics_sum, and so on. You still declare one doris/metrics exporter; it routes each datapoint to the right table.

Metrics are the one signal where cardinality hurts. A single user_id label carrying a fresh value per request creates one new time series per user, and the metric tables grow without bound. Two processors defend against this — one deletes known bad label names, one folds UUID-shaped values into a single bucket. They run on metrics only; traces and logs keep their full values, because there the detail is the whole point.

Metrics use load_to_single_tablet: "false". Doris's general advice is "true" for any RANDOM table, but on Nivorbit's benchmark the flip gained nothing for the metric tables and slightly worsened the tail, so it stays off. Re-measure on your own hardware before changing it.

The recommended production config

The minimal Pattern B block connects, but it sends tiny, unbuffered loads. Below is the full recommended setup — all three signals, each with its own exporter and its own pipeline. That per-signal split is the whole reason for three exporters; logs and metrics are not optional add-ons, they each need a block. The three exporters are near-identical: only the values in the per-signal table below differ.

extensions:
  file_storage:                       # one shared on-disk queue; survives a Doris outage/restart
    directory: /var/lib/otelcol/queue
    timeout: 1s
    compaction:                       # bbolt files never shrink without this
      directory: /var/lib/otelcol/queue
      on_start: true
      on_rebound: true
      rebound_needed_threshold_mib: 100
      rebound_trigger_threshold_mib: 10
 
processors:
  batch:
    send_batch_size: 100000           # aim for large loads, not frequent tiny ones
    send_batch_max_size: 200000
    timeout: 5s
  # memory_limiter, cardinality guards, log-timestamp fix and tail_sampling are
  # defined in the reference config — see "Processors you shouldn't drop" below.
 
exporters:
  # ── TRACES — heaviest signal → most consumers, deepest queue ──
  doris/traces:
    endpoint: http://orbtrace-host:8030
    database: orbtrace
    username: ${env:DORIS_USER}
    password: ${env:DORIS_PASSWORD}
    table: { traces: otel_traces }
    create_schema: false              # required — Orbtrace owns the schema
    timeout: 300s
    log_response: true                # the only place dropped-row counts show up
    headers: { load_to_single_tablet: "true", max_filter_ratio: "0.01" }
    sending_queue: { enabled: true, storage: file_storage, num_consumers: 20, queue_size: 1000 }
    retry_on_failure: { enabled: true, initial_interval: 5s, max_interval: 30s, max_elapsed_time: 30s, randomization_factor: 0.5 }
 
  # ── LOGS — same block; only num_consumers / queue_size change ──
  doris/logs:
    endpoint: http://orbtrace-host:8030
    database: orbtrace
    username: ${env:DORIS_USER}
    password: ${env:DORIS_PASSWORD}
    table: { logs: otel_logs }
    create_schema: false
    timeout: 300s
    log_response: true
    headers: { load_to_single_tablet: "true", max_filter_ratio: "0.01" }
    sending_queue: { enabled: true, storage: file_storage, num_consumers: 10, queue_size: 500 }
    retry_on_failure: { enabled: true, initial_interval: 5s, max_interval: 30s, max_elapsed_time: 30s, randomization_factor: 0.5 }
 
  # ── METRICS — single-tablet OFF (measured no-gain); smaller queue ──
  doris/metrics:
    endpoint: http://orbtrace-host:8030
    database: orbtrace
    username: ${env:DORIS_USER}
    password: ${env:DORIS_PASSWORD}
    table: { metrics: otel_metrics }
    create_schema: false
    timeout: 300s
    log_response: true
    headers: { load_to_single_tablet: "false", max_filter_ratio: "0.01" }
    sending_queue: { enabled: true, storage: file_storage, num_consumers: 5, queue_size: 200 }
    retry_on_failure: { enabled: true, initial_interval: 5s, max_interval: 30s, max_elapsed_time: 30s, randomization_factor: 0.5 }
 
service:
  extensions: [file_storage]
  pipelines:
    # Each signal is its OWN pipeline feeding its OWN exporter — that is the point.
    # Processor chains below are the recommended set (defined in the reference config).
    traces:  { receivers: [otlp], processors: [memory_limiter, transform/instance_id, attributes/cardinality_denylist, transform/cardinality_safety, tail_sampling, batch], exporters: [doris/traces] }
    logs:    { receivers: [otlp], processors: [memory_limiter, transform/instance_id, attributes/cardinality_denylist, transform/cardinality_safety, transform/log_timestamp_fix, filter/logs, batch], exporters: [doris/logs] }
    metrics: { receivers: [otlp], processors: [memory_limiter, transform/instance_id, attributes/cardinality_denylist, transform/cardinality_safety, batch], exporters: [doris/metrics] }

The only three values that change per signal:

What each setting does, and when to change it

Advanced tuning (opt-in)

Reach for these only when a signal tells you to. Defaults are correct for most deployments.

• Doris Group Commit. Server-side batching: Doris merges many concurrent Stream Loads into one internal commit (group commit manual), producing fewer tablet versions and far less compaction pressure. It is Doris's recommended answer for exactly the log/high-concurrency-small-load shape observability produces — a published benchmark shows ~16× throughput on 1 MB loads. Add group_commit: "async_mode" to a signal's headers (defaults: flush every 10 s or 64 MB, tunable per table). Two caveats: (1) Stream Load + Group Commit has reported intermittent lost rows on Doris 4.0.2–4.1.0 (apache/doris#63160) — validate exact row-count parity on your build first. (2) async_mode writes to a WAL and silently falls back to normal load if WAL disk runs low, so watch WAL size. Turn it on only if tablet-version counts climb week-over-week and the single-tablet and queue levers are already in place.
• load_to_single_tablet for metrics. Off by default (measured no-gain). If your hardware differs from ours, A/B it before flipping.
• Batch size vs BE limit. Raise send_batch_size for more throughput, but keep it under the streaming_load_json_max_mb ceiling described above.
• Queue depth and consumers. Deeper queue_size buffers longer Doris outages; more num_consumers raises write parallelism. Scale both with ingest rate, and size the volume to match.
• Network egress. The Doris exporter sends Stream Load bodies uncompressed. On a high-volume pipeline crossing a network boundary, size the link for raw event bytes, and keep the Collector close to Doris.

Deployment architectures

Patterns A and B answer what connects to Doris. This answers where the Collector layer lives. These are the OpenTelemetry community's standard layouts — agent and gateway — applied to Orbtrace.

Single host or VMs

One Collector next to the Orbtrace stack is the whole architecture. Every service — on that host or on other VMs — points OTEL_EXPORTER_OTLP_ENDPOINT at it. A single contrib Collector sustains tens of thousands of events per second before it needs help.

Kubernetes — node agents plus a gateway

The industry-standard Kubernetes layout is two tiers, and it is what we recommend.

• Agent tier — a DaemonSet. One Collector per node; pods send to their local node agent. The agent adds pod, namespace, deployment and node labels with the k8sattributes processor — exactly the metadata you filter by in Orbtrace. Local delivery keeps SDK latency and cross-node traffic low.
• Gateway tier — a small Deployment. Two or more replicas do the expensive, stateful work: tail sampling and the Doris exporters with their durable queues. Keeping Doris credentials in one tier also keeps your security review small.
• The one non-obvious rule. Tail sampling must see all spans of a trace on the same replica. With more than one gateway, the agents must export with loadbalancing keyed on routing_key: traceID. Plain round-robin or client-IP stickiness splits a trace across replicas and breaks every multi-service sampling decision.
• Zero-code instrumentation. Run the OpenTelemetry Operator and annotate workloads with its Instrumentation CRD to inject auto-instrumentation without touching your images. It can also manage the Collector tiers themselves.

Start small: one gateway Deployment is enough. Add the DaemonSet when you want per-node metadata; split tail sampling to scaled gateways only when one replica saturates.

OpenShift

The same two tiers, with two platform notes:

• Operator: use the Red Hat build of OpenTelemetry from OperatorHub — it is the same OpenTelemetry Operator, packaged for OpenShift.
• Security: the contrib image runs as non-root, so the tiers run under the default restricted-v2 SCC with no grants. Give the gateway's queue volume write access with securityContext.fsGroup: 10001.

Orbtrace's own OpenShift install is on the OpenShift page; this section is only the pipeline.

Service mesh (Istio, Linkerd)

A mesh does not replace SDK instrumentation — the two answer different questions.

• Keep instrumenting your apps. Envoy sidecars only see traffic between pods. They cannot break a request into your handler, DB call and queue publish, and they cannot carry trace context through your code. An uninstrumented app in a mesh still produces disconnected one-hop spans.
• Agree on W3C traceparent. The mesh and your SDKs must share one context-propagation format. W3C Trace Context is the standard and every OTel SDK's default; prefer it over legacy B3.
• Mesh spans are an optional supplement. Istio's Telemetry API can export Envoy spans over OTLP — point it at your agent/gateway tier, at a low sampling percentage or not at all to start. With instrumented apps, Envoy spans mostly duplicate the picture while multiplying volume.
• One egress. App SDKs, mesh telemetry, and node agents all funnel through the same Collector layer to Doris. There is never a second path into Orbtrace to secure or debug.

Cost-aware sampling

Tail sampling keeps every error trace, every slow trace, every SLO-violating and anomaly-tagged trace, and samples the rest. The probabilistic floor for "the rest" is driven from Orbtrace: operators set a per-service monthly trace cap in /admin/sampling, and Orbtrace publishes the resolved policy as YAML at /api/sampling/policy.yaml. This defends Doris disk and keeps signal-to-noise high without you editing Collector config per service.

Static policy (default). The reference config ships fixed policies — keep errors, keep slow traces, keep force-tagged traces, sample everything else. Good for a single host or a first deploy.

Dynamic policy (production). The gateway pulls the live per-service policy and hot-reloads when it changes:

processors:
  tail_sampling:
    decision_wait: 30s            # must exceed the longest trace you promise to keep
    num_traces: 100000            # ≥ trace rate × decision_wait
    decision_cache:
      sampled_cache_size: 500000
      non_sampled_cache_size: 1000000
    policies: ${httppoll://orbtrace:8080/api/sampling/policy.yaml?interval=5m&auth_env=ORBTRACE_INGEST_TOKEN}

Processors you shouldn't drop

Pattern A users get these for free from the reference config. Pattern B users managing their own Collector must copy them — they keep bad data out of Doris:

• memory_limiter — first in every pipeline. Refuses new data when the heap runs hot, so the Collector sheds load instead of crashing.
• transform/instance_id — backfills service.instance.id from k8s.pod.name or host.name, so Orbtrace's per-instance breakdown resolves instead of collapsing to "(unknown)".
• attributes/cardinality_denylist + transform/cardinality_safety — delete per-user label names and fold UUID-shaped values, so metrics don't explode into unbounded time series.
• transform/log_timestamp_fix — backfills zero timestamps from buggy SDKs, so one bad SDK can't reject whole log batches.

What Orbtrace does not touch

Orbtrace reads from Doris. It never reconfigures your apps or your Collector for you — you own the pipeline. Tuning beyond this reference config (custom processors, multi-gateway fan-out) is a separate engagement, not a prerequisite.