Enrollment and mTLS

The runner boots with no secret in the image. It builds its own identity at boot, proves who it is to the control-plane, and receives a short-lived mTLS certificate. It’s the only way a cattle fleet gets identity without scattering keys around.

The flow

Boot is a state machine: liveness → config → secrets → enroll → connect → ready. The enroll step does this:

1. Generates an EC P-256 keypair. The private key exists in memory only, as PKCS#8, and is never written to disk.
2. Generates a nonce and builds a CSR with it in the challengePassword field, which is the proof of possession.
3. Gathers platform attestation, per RUNNER_ATTESTATION_MODE.
4. POSTs to /api/enroll with { version, attestation, csr, challengeNonce }.
5. The control-plane validates the attestation, signs the CSR, and returns a certificate with the tenant’s SPIFFE id inside: spiffe://rootpilot/tenant/<slug>/runner.
6. The runner opens the WSS tunnel with mTLS using that certificate.

The one call without mTLS

POST /api/enroll uses server-side TLS only, because it is the call that exists precisely to obtain the client certificate. Everything after it is mTLS. The client never logs the token or the CSR.

The tenant comes from the certificate, not from configuration

This is worth insisting on, because it’s the guarantee holding up everything else: the attestation material determines the tenant, not an environment variable. Enrollment returns a certificate with the SPIFFE id inside, and that’s what the control-plane reads on every call.

RUNNER_TENANT, which shows up in the fleet scripts, picks no tenant at all: it picks the vault folder to read credentials from. Getting that slug wrong will not connect you to the wrong organization.

Renewal: early, not at the deadline

The certificate is valid for at most 1 hour. RUNNER_CERT_RENEWAL=on is the default.

Renewal fires at 2/3 of the certificate’s lifetime, not near expiry. The reason is operational: renewing right at notAfter would turn any transient network instability into identity loss. Renewing at 2/3 leaves a third of the lifetime as a retry window.

Retries back off with a 5-minute ceiling and ±25% jitter. Without the jitter, the entire fleet would hit the control-plane in the same second.

How renewal proves identity

Renewal is not a RUNNER_ATTESTATION_MODE. Renewing isn’t a configuration choice: it’s what the runner does when it already has a certificate.

It signs the new CSR’s nonce with the current private key and sends the current certificate along. The control-plane validates the chain against the CA, checks the signature with the certificate’s public key, and extracts the tenant from the certificate, never from the request body. Reading it from the body would turn renewal into a way to switch tenants.

The configured mode still applies to the initial enrollment, and to the case where renewal fails through to expiry.

When identity is lost

The control-plane grants a 5-minute grace for a just-expired certificate in the renewal attestation. Adding 1 minute of clock-skew margin, identity is considered lost 6 minutes past expiry.

After that, the control-plane answers presented cert expired beyond the renewal grace window, and it will answer that forever. The runner has no way back: the renewal attestation proves possession with the certificate that died.

The behavior is then to exit the process, so the supervisor brings up a fresh runner that enrolls from scratch. Without that exit, the process would become a zombie: up, serving nothing, invisible to any probe.

Renewal requires a control-plane with the verifier

Against an older control-plane, renewal fails and the runner degrades to the old behavior (expiring in 1 hour) logging cert renewal failed rather than dying quietly. RUNNER_CERT_RENEWAL=off remains available for that case.

The development shortcut

There is a second identity path, and it is development only: supplying a ready-made certificate through RUNNER_CERT_PEM + RUNNER_KEY_PEM (+ RUNNER_CA_PEM). When the first two are present, the runner skips enrollment entirely.

npm run dev:certs   # mints CA + tunnel cert + client cert into dev-certs/, 30 days

The log announces the shortcut:

using provisioned cert (dev/test shortcut) {spiffeId: "spiffe://rootpilot/tenant/demo/runner"}

With a provisioned certificate, renewal does not apply: there’s no enrollment to repeat.

Never mount dev-certs/ against production

Those PEMs are signed by the development CA. Mounting them would make the runner take the shortcut instead of enrolling, and mTLS would fail against the production CA with an error that says none of this. That’s why docker-compose.fleet.yml redefines volumes with !override.

Configuration validation covers this

The schema requires one of the two paths, and fails fast when both are missing:

either provide RUNNER_CERT_PEM+RUNNER_KEY_PEM (dev shortcut) or RUNNER_ENROLL_URL (enrollment)