Troubleshooting

The runner won’t connect

certificate has expired or a hostname mismatch

Development certificates last 30 days. Regenerate:

npm run dev:certs

The SAN covers host.docker.internal. If it still fails, RUNNER_TLS_INSECURE_SKIP_VERIFY=true unblocks it. The tunnel keeps validating the runner’s certificate; only the runner stops checking the server’s. A development escape hatch, not for production.

is not in the cert's altnames

You connected through a hostname that isn’t in the tunnel certificate’s SAN. Even if it resolves to the right IP, TLS refuses. In production the hostname is tunnel.rootpilot.sh.

Nothing happens, and there’s no error

Confirm a control-plane tunnel is listening at RUNNER_TUNNEL_URL. In development the default is :8443 on the host; on Linux, Compose already maps host.docker.internal via host-gateway.

A 1006 → exited 0 → restart cycle

The tunnel on the other side is restarting. Stabilize it and the runner stops cycling. The exited 0 is the reconnect timer being unref’d; restart: unless-stopped brings it back when the tunnel returns.

Boot fails immediately

The runner schema-validates configuration and dies with the reason. The common ones:

Message	What to do
`either provide RUNNER_CERT_PEM+RUNNER_KEY_PEM (dev shortcut) or RUNNER_ENROLL_URL`	Pick one of the two identity paths.
`RUNNER_ATTESTATION_MODE=bootstrap-token requires RUNNER_BOOTSTRAP_TOKEN`	The token is missing. The fail-fast is intentional: without it the symptom becomes a 401 loop that looks like a network problem.
`RUNNER_ATTESTATION_MODE=aws-sts requires AWS_REGION`	The region is not guessed, on purpose.
`RUNNER_SECRET_STORE_MODE=vault requires RUNNER_VAULT_ADDR`	Incomplete Vault configuration.
`attestation mode 'eks-oidc' not implemented`	`eks-oidc` does not exist. On EKS, use `aws-sts`.
`op "…" claimed by both datadog and grafana`	Both observability providers registered. Pick one in `RUNNER_OBSERVABILITY`.
`Could not locate the bindings file`	The image was built with `--ignore-scripts`. `better-sqlite3` is a native addon and needs its postinstall.

A tool returns the wrong shape

Typical symptoms: entries.map is not a function, or “op not implemented”.

You’re in synthetic, which is the default and is only a 3-operation seed whose shapes don’t match the real connectors.

For real data: RUNNER_CONNECTORS=real plus the keys.
For the whole catalog with no credentials: RUNNER_CONNECTORS=demo.

A connector disappeared

The signal is in the boot log:

connector not served {connectorId: "azion", missing: ["AZION_TOKEN"]}

Three causes, in order of frequency:

The key never reached the container. Compose pass-through is by name and explicit: a variable absent on the host is a key omitted in the container. Check that the name is in the docker-compose.yml list.
The name differs. The azion connector declares AZION_TOKEN; a vault holding AZION_API_TOKEN produces exactly the missing-key symptom. Check the exact names in Connector credentials.
The secret isn’t in the store. In remote backends, “not found” resolves to undefined rather than an exception, on purpose, so the lazy model works.

The EKS health check says `skipped`

skipped: no cluster configured (EKS_CLUSTER)

EKS_CLUSTER is missing. Worth distinguishing: the connector shows up as unconfigured in Fleet, which is different from broken, and equally blind.

I enabled OTel and see no spans

The master switch alone isn’t enough: the exporter defaults to none. See Telemetry.

Why a field comes back `null`

This is the most frequent support question, and the answer matters: null here means “I don’t know”, never “it is zero”. The distinction is deliberate, and it came out of a class of incident where a fabricated 0 was read as health.

Three rules the edge follows:

1. Before returning null, try the other source. Having an error rate doesn’t end the latency reading: if percentiles don’t arrive ready-made and raw spans are available, they’re computed from those, and the result says where it came from (percentileSource: 'spans').

2. A null that survives carries the reason code. Fields like percentilesUnavailable and instancesUnavailable exist so that “wasn’t emitted” and “I didn’t query” stop looking identical.

3. An empty payload proves whether the question existed. A service name matching no APM service comes back with serviceResolved: false and a candidate list, instead of the same blanket null a real, quiet service would produce.

And a metric derived from a zero denominator is null, never 0: a ratio without a denominator is undefined, and 0% is the opposite of “I don’t know”: it leads to the opposite action.

An error log contains `[INFO]`

Not a bug. Datadog sends all stderr to status:error, so a search for errors can return lines whose text says otherwise.

The edge exposes the divergence with textLevel and levelMismatch, rather than rewriting level. The source’s classification is a fact about the source, and silently correcting it would erase information.

The runner died and didn’t come back

If the certificate went 6 minutes past expiry (5 of control-plane grace, 1 of skew margin), identity is unrecoverable: the renewal attestation proves possession with the certificate that died.

The runner exits the process on purpose, so the supervisor brings up a replacement that enrolls from scratch. Confirm your supervisor is configured for that. Without it, the effect is a runner that vanishes.

The control-plane-side message:

presented cert expired beyond the renewal grace window

I lost learned state after a kill

Expected on a hard reclaim (SIGKILL, OOM, spot with no drain). The periodic flush (RUNNER_LEARNED_FLUSH_INTERVAL_MS, default 60 s) bounds the loss; it doesn’t eliminate it.

Remember too that the local learned-store is a cache: the source of truth is the control-plane. On Kubernetes, size terminationGracePeriodSeconds above RUNNER_DRAIN_TIMEOUT_MS (default 120 s) so the drain can complete.

How to observe a runner that exposes no port

There is no HTTP probe, because the runner is an outbound client. The available signals are:

Boot logs: boot: liveness, boot: ready, and any connector not served.
The control-plane’s Fleet view, which sees the runner connected and the per-connector status reported in the handshake.
OTel, if enabled correctly.