Telemetry

The runner is instrumented with OpenTelemetry, and telemetry is a no-op by default. Development, test, and CI need no collector at all.

Turning it on takes two variables, not one

Here’s the trap: telemetry is only active when three conditions hold at once.

enabled = ROOTPILOT_OTEL_ENABLED && !OTEL_SDK_DISABLED && exporter !== 'none'

And the exporter defaults to none. Which means:

# NOT enough: the exporter stays `none` and nothing is emitted
ROOTPILOT_OTEL_ENABLED=true

# works
ROOTPILOT_OTEL_ENABLED=true
ROOTPILOT_OTEL_EXPORTER=otlp
OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.example.com

Enabling only the master switch produces silence, not an error

There is no warning. If you enabled ROOTPILOT_OTEL_ENABLED and see no spans, the exporter is the first thing to check.

Variables

Variable	Default	Notes
ROOTPILOT_OTEL_ENABLED	false	Master switch. Accepts true or 1.
ROOTPILOT_OTEL_EXPORTER	none	otlp, console, or none. Must move off none.
OTEL_SDK_DISABLED	false	The standard OTel kill switch. true overrides the master switch.
OTEL_EXPORTER_OTLP_ENDPOINT	none	Collector URL.
OTEL_EXPORTER_OTLP_HEADERS	none	Headers, for an authenticated collector.
OTEL_EXPORTER_OTLP_PROTOCOL	http/protobuf
OTEL_SERVICE_NAME	rootpilot-service
OTEL_SERVICE_VERSION	none	Absent ⇒ the service.version attribute simply isn’t emitted.
OTEL_TRACES_SAMPLER_ARG	1	Sampling ratio, between 0 and 1.

console is handy for quickly checking that instrumentation is alive without standing up a collector.

How initialization happens

Through a preload, not a call in code:

node --import @rootpilot/otel/register apps/runner/dist/index.js

The preload runs before the application’s module graph loads, which is what lets OTel’s auto-instrumentation patch http and ws first. The runner image’s CMD is already exactly that command, so you don’t need to do anything beyond setting the variables.

It’s also why the service name and version come from the environment: at that point there’s no build inlining them and no in-code caller to pass them.

What’s instrumented

Span	When
rootpilot.handle_invoke_batch	Receiving a batch of invocations, under the remote context extracted from the message metadata.
rootpilot.run_op	Each operation, with MCP and operation attributes.

The remote context is the detail that matters: it makes the trace distributed, joining the control-plane’s span to the runner’s in a single trace. There are also per-tenant metrics.

Correlation with logs

The logger correlates trace_id and span_id onto log lines. With telemetry on, you can go from a span in your backend straight to that operation’s log lines.

Flush on shutdown

Telemetry is flushed on SIGTERM, alongside the runner’s drain. Keep terminationGracePeriodSeconds in mind: if SIGKILL arrives first, the last spans are lost.

Fleet observability, without OTel

Even with telemetry off, the runner reports state to the control-plane through the handshake:

• accounts[]: the cloud context derived at boot.
• ConnectorStatus{served, health}: what each connector is serving and how it fared in the credential self-check.
• ConnectorStatus.ops: which operations that connector serves.

The last one exists because a flat list of served operations doesn’t say whose they are, and without the owner, the control-plane can’t discount the operations of a connector with a broken credential.

New knowledge enters through the handshake, not a new operation

Handshake fields sit outside the catalog hash, so an optional field there neither invalidates the golden hash nor affects an older cattle runner, which simply omits it, and the control-plane degrades. That's why the status surface grows there.