Evals

Introduction

Last Light Evals is a CLI that runs Last Light's real production workflows — issue triage, build, PR review — end to end against a mocked GitHub, grades the results deterministically, and prints a model-comparison scorecard on cost, latency, and pass rate. It's SWE-bench compatible and ships with sample datasets so you can run it in one command.

The Evals dashboard overview — tier cards and a table of recent runs newest-first, each with its model, git SHA, score, and cost.
The dashboard overview: every run, newest first, with score and cost.
See it live: evals.lastlight.dev is a real, published dashboard you can browse right now — runs, model comparisons, trend lines, and inspectable judge traces.

Why run it against the real workflows?

The point is to measure what actually ships. These evals load the actual workflow YAML, prompts, skills, and agent loop from the lastlight package — unmodified. There are only two deviations from production, and they're the two you can't do unattended:

  1. GitHub is mocked, not bypassed. The agent's github_* tool calls hit an in-process fake REST API that seeds fixtures and records mutations; git push goes to a local bare repo. The agent still believes it is talking to GitHub.
  2. Approval gates are disabled so runs never pause waiting for a human.
The one invariant: if a change makes the eval diverge from production in any other way, it's wrong. The whole value is that a green (or red) score reflects the shipping workflow, not a lab replica.

What you get

  • A model comparison. Force a set of models across every workflow step and see pass rate, total cost, P50 latency, and token usage side by side.
  • Deterministic grading. Triage is graded on recorded GitHub mutations; code-fix on held-out tests (SWE-bench's resolved criterion). The only LLM judge is a single, scoped one for PR-review quality.
  • A JSON-driven dashboard. Every run writes a scorecard.json that a local SPA renders live — overview, per-model trend lines, per-instance rows, and inspectable judge traces.
  • Your own workflows and data. Point the harness at a deployment overlay to eval forked workflows, a per-step model config, and your own dataset cases.

The three tiers

TierWorkflowGraded on
triage issue-triage (cheap, fast) Behavioral — labels / comments / state vs the gold expectation.
code-fix build (heavy, full cycle) Execution — held-out tests must go red → green and stay green.
pr-review pr-review An LLM judge scores the posted review against a human gold set (F1).

New to it? Install it and run your first eval in a couple of minutes — no GitHub credentials required.