Skip to content
RootPilotRootPilotRootPilot Edge

Observability backend

The observability block (metrics, logs, traces, topology, monitors, rum and synthetics) can be served by Datadog or by Grafana Cloud. Both declare the same catalog operations, under vendor-neutral names.

That neutrality is what makes a second backend cost “write a connector” instead of “rewrite diagnostics”. It is also what makes the two mutually exclusive.

Terminal window
RUNNER_OBSERVABILITY=datadog   # default
RUNNER_OBSERVABILITY=grafana
RUNNER_OBSERVABILITY=none      # tenant without observability

The choice is explicit, not inferred

Section titled “The choice is explicit, not inferred”

The runner does not pick the provider by looking at which credential is present. That’s deliberate: inferring would produce a “my Datadog stopped responding” with no visible cause in the configuration.

datadog is the default, so every existing runner behaves identically without touching anything.

Datadog

Section titled “Datadog”
Terminal window
RUNNER_CONNECTORS=real
DD_API_KEY=…
DD_APP_KEY=…
# DD_SITE=datadoghq.com   # optional: datadoghq.eu / us3 / us5 / ap1

Grafana Cloud

Section titled “Grafana Cloud”
Terminal window
RUNNER_CONNECTORS=real RUNNER_OBSERVABILITY=grafana \
  GRAFANA_CLOUD_TOKEN=… \
  GRAFANA_PROM_URL=https://prometheus-prod-XX-prod-REGION.grafana.net/api/prom

Unlike Datadog, Grafana exposes one endpoint per datasource: Mimir, Loki, and Tempo have distinct hosts, each authenticating over HTTP Basic, with the username being the instance’s numeric ID and the password being the Access Policy token. Without _USER, the token is sent as a Bearer token.

Only the token and Mimir are required. Mimir is the backbone: the service graph and the Faro and Synthetics metrics live there. Loki, Tempo, and Alerting degrade operation by operation, with actionable errors.

Variable Required What for
GRAFANA_CLOUD_TOKEN yes Access Policy token.
GRAFANA_PROM_URL yes Mimir.
GRAFANA_PROM_USER no The instance’s numeric ID.
GRAFANA_LOKI_URL / _USER no Logs.
GRAFANA_TEMPO_URL / _USER no Traces.
GRAFANA_STACK_URL no Alerting API (monitors).
GRAFANA_SYNTHETICS_URL no Defaults to https://synthetic-monitoring-api.grafana.net.

Label mapping

Section titled “Label mapping”

This is where a Grafana installation works for the first customer and breaks for the second.

service and env are first-class concepts in Datadog. In Prometheus and Loki they are the customer’s label convention, and it might be service, app, job, or OTel’s service_name. The defaults below are a convention, not a guarantee.

Variable Default
GRAFANA_SERVICE_LABEL service
GRAFANA_ENV_LABEL env
GRAFANA_NAMESPACE_LABEL namespace
GRAFANA_HOST_LABEL instance
GRAFANA_ROUTE_LABEL http_route

Discover yours with Mimir’s /api/v1/labels endpoint.

Checkly: the specialist that takes over synthetics

Section titled “Checkly: the specialist that takes over synthetics”
Terminal window
CHECKLY_API_KEY=…
# CHECKLY_ACCOUNT_ID=…

checkly is dedicated to one capability. When present, it takes over synthetics from whichever general provider is configured, and it coexists with both Datadog and Grafana.

Here, unlike the provider choice, inferring from a credential is safe: the specialist only adds a source, it doesn’t contend for someone else’s reading.

CHECKLY_ACCOUNT_ID is required in practice for a user token, because without it the API answers 401 without saying why. For an account or service token it is unnecessary.

Vercel: orthogonal, not competing

Section titled “Vercel: orthogonal, not competing”
Terminal window
VERCEL_TOKEN=…
VERCEL_TEAM_ID=…

Vercel does not contend for operation ownership with GitHub, because the two answer different facts: GitHub answers “which PR merged”, Vercel answers “which artifact went live”. So Vercel creates its own operations, with vendor-distinct names, and joins the connector set directly, with no arbitration.

VERCEL_TEAM_ID is required in practice for a team token (without it the API answers 403 without explaining) and absent on a personal account. Use a read-only token scoped to the team.

The test: reuse the operation, or create a new one?

Section titled “The test: reuse the operation, or create a new one?”

If you’re writing a second connector for a capability that already has an owner, the test is semantic:

Would the consumer already using this operation accept the new connector’s answer as equivalent?

Yes → reuse the operation. That’s Grafana in metrics.*: p99 is p99. catalogHash stays intact, nothing changes for consumers, and the connector goes through arbitration, because it contends for operation ownership.

No → a new operation, with a vendor-distinct name. That’s Vercel. catalogHash changes deliberately, and the connector joins directly, without arbitration, because the operations are disjoint.

An operation with more than one implementation needs a contract

Section titled “An operation with more than one implementation needs a contract”

When two implementations serve the same operation, a shared name isn’t enough: a declared output contract is required. It’s what makes “swapping the operation’s owner” safe.

The contract is per operation, not per capability, and it is a floor, not a ceiling: an extra field passes, a declared field that’s missing downgrades the capability to opaque.

When adding or changing an operation served by two or more connectors, update the contract and run the conformance suite against the implementations, with both a populated and an empty scenario.