Docs

Observability

Last Light has first-class OpenTelemetry support: distributed traces and metrics across every workflow run, phase, agent execution, PI event stream, and chat turn. It's disabled by default — nothing leaves the harness until you opt in.

Enabling it

Set LASTLIGHT_OTEL_ENABLED=true and point the standard OTLP exporter env vars at your collector: 

LASTLIGHT_OTEL_ENABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=https://otel-collector.example.com
OTEL_EXPORTER_OTLP_HEADERS=authorization=Bearer <token>
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=prod

The export uses the standard OTLP/HTTP protocol, so any OTEL-compatible backend works — Grafana Tempo/Mimir, Honeycomb, Datadog, Jaeger, Grafana Cloud, or a self-hosted OpenTelemetry Collector. Every LASTLIGHT_OTEL_* and OTEL_* variable is listed on the Configuration page.

Setting OTEL_* env vars alone does not turn telemetry on — LASTLIGHT_OTEL_ENABLED=true is the master switch. This keeps a stray OTEL_EXPORTER_OTLP_ENDPOINT in your environment from silently shipping data.

What's exported

Instrumentation spans the whole execution path so you can trace a single issue or PR from event to outcome:

  • Traces — a nested span tree per run: lastlight.workflow.run → lastlight.workflow.phase → lastlight.agent.execute → a span per model turn → a span per tool call, tagged with the workflow name, phase, repo, issue number, model, sandbox backend, and stop reason, plus per-turn/per-run tokens and cost.
  • Metrics — execution duration, cost (USD), input/output tokens, run and error counts, dimensioned by workflow, phase, repo, model, backend, and success/stop-reason.
  • Chat turns — each in-process Slack chat turn, alongside the same execution metrics.
OpenInference / Arize Phoenix. Spans carry OpenInference attributes (openinference.span.kind of CHAIN / AGENT / LLM / TOOL, plus llm.model_name, token counts, and cost), so an OpenInference-aware backend such as Phoenix renders a run as a proper agent tree with per-turn tokens and cost instead of a flat span. The flat pi.* span events are also kept as a fallback. Phoenix ingests traces but not OTLP metrics — set LASTLIGHT_OTEL_METRICS_ENABLED=false (below) for a traces-only backend.

Disabling metrics

Metrics are on by default. Some backends accept traces but reject the OTLP metrics signal (Arize Phoenix, for example, 404s the metrics endpoint). Set LASTLIGHT_OTEL_METRICS_ENABLED=false (or otel.metrics: false in the overlay config.yaml) to skip the metric exporter entirely — traces still flow, and nothing is sent to a metrics endpoint that would error.

Content redaction

By default Last Light exports metadata only: workflow and phase names, repo, sandbox backend, model, success / stop reason, timing, token counts, and cost. Prompt text, message content, tool-call arguments, and tool outputs are redacted.

Set LASTLIGHT_OTEL_INCLUDE_CONTENT=true to include that content (truncated) — useful for debugging agent behaviour, but it can export sensitive data.

Only enable content with a trusted collector. It can put source code, secrets that appear in tool output, and full prompts on the wire. Auth headers (OTEL_EXPORTER_OTLP_HEADERS) are always treated as secrets and never appear in the dashboard Config tab.

Sandbox telemetry

With LASTLIGHT_OTEL_FORWARD_TO_SANDBOX=true (the default), the agent emits its own telemetry from inside each workflow sandbox too — not just the harness. How that reaches your collector depends on the sandbox backend:

docker backend — in-network collector

Docker sandboxes run with a default-deny egress firewall, so they can't dial an arbitrary collector directly. Instead, Last Light runs a small OpenTelemetry Collector inside the sandbox network. Sandboxes export to it by a fixed internal address; it re-exports to your real backend over a separate, trusted outbound network: 

sandbox ──OTLP──▶ in-network collector ──OTLP──▶ your backend
              (fixed internal IP)        (trusted outbound leg)

This design has three properties worth calling out:

  • Secrets stay host-side. The real backend endpoint and its auth headers live only on the collector's config (on the host). They are never forwarded into the untrusted sandbox.
  • Any backend works. The sandbox only ever dials one fixed internal endpoint, so a collector on any port or scheme (e.g. https://collector:4318) just works — no egress-allowlist or firewall changes needed.
  • No new exfil surface. The collector only ever forwards to its configured backend; sandbox traffic can't redirect where it exports, only influence the span content sent to your own backend.

gondolin / none backends

On the gondolin (micro-VM) and none backends the agent runs in the harness process and already inherits the harness's OTEL setup, so the allowlisted OTEL_* env vars are forwarded directly. The gondolin VM additionally adds your collector hosts — parsed from the endpoint vars and LASTLIGHT_OTEL_COLLECTOR_HOSTS — to its egress allowlist. Private and cloud-metadata hosts remain blocked.

To keep all telemetry harness-only and never touch the sandbox, set LASTLIGHT_OTEL_FORWARD_TO_SANDBOX=false.

Failure handling

If the OTEL SDK fails to initialise (bad endpoint, unreachable collector), Last Light logs a warning and continues without telemetry — observability never takes the harness down. Set LASTLIGHT_OTEL_STRICT=true to make such a failure fatal at startup instead, which is useful when telemetry is a hard requirement.

Human feedback on the trace

When somebody reacts 👍 or 👎 to something Last Light wrote, that score is exported as a lastlight.feedback.signal span attached to the original run's trace — a late child of that run's span, reconstructed from the trace and span ids the run recorded. So the grade appears on the very trace it grades, beside the prompts, tokens and cost that produced the output, instead of in a disconnected trace of its own. The span carries OpenInference's EVALUATOR kind and Langfuse's langfuse.score.* attributes. See Feedback signals.

← Previous Per-repo config Next → Feedback signals