Evals
Running evals
run is the default command and it's the same shape for every tier.
This page covers what's shared — selecting tiers, the flags, parallelism, and
isolation. Each tier then has its own page for how it seeds and grades:
Triage, Code fix, and
PR review.
Selecting tiers
Give run one or more tiers, or none to pick interactively (a
non-TTY run falls back to the cheapest tier). From a Separate-layout workspace
the overlay and datasets auto-detect; a Plain workspace adds
--overlay ..
lastlight-evals run # default model, default tier
lastlight-evals run triage # one tier
lastlight-evals run triage code-fix # multiple tiers → combined tabbed report
lastlight-evals run --compare # cross-vendor set (key-gated)
lastlight-evals run triage --runs 3 # repeat each case 3× (worst-case verdict, mean metrics)
lastlight-evals run pr-review --limit 3 # only the first 3 cases (controlled / cheap)Grading, in one sentence per tier
Grading is deterministic — the one exception being the PR-review judge. Each tier page has the detail; the summary:
| Tier | Graded on |
|---|---|
triage | Behavioral — recorded GitHub mutations (labels, comments, state) vs the gold expectation. |
code-fix | Execution — held-out tests must go red → green and stay green (SWE-bench's resolved criterion). |
pr-review | Judge — an LLM judge scores the posted review against a human gold set (F1). |
Run flags
| Flag | Meaning |
|---|---|
--mode <models|config> | Comparison axis (default models; a TTY with no flags asks). See Models. |
--overlay <dir> | Layer a deployment's workflows / skills + evals over the built-ins. Repeatable in --mode config. |
--model <m[,m2]> | Select model(s), fuzzy-matched; in config mode, override each config's default. |
--compare | Cross-vendor set (only models whose provider key is present). |
--instance <id[,id]> | Run only these exact instance_id(s) (or EVAL_INSTANCE) — not a substring. |
--limit <n> | First n instances per tier (applied after --instance). |
--runs <n> | Repeat each case n× — worst-case verdict, mean metrics. |
--f-beta <n> | PR-review F-beta β (default 1 = F1). See PR review. |
--judge-with-diff | PR-review: feed the PR diff to the judge (off by default). |
--no-inject-context | PR-review: don't inject synthetic repo context into the checkout (a clean A/B control; or EVAL_INJECT_CONTEXT=0). See PR review. |
--sandbox <backend> | none (default, in-process, fast/CI) or gondolin (QEMU micro-VM isolation; or EVAL_SANDBOX). |
--serial | Force serial execution across provider families. |
--datasets <dir> | Extra datasets root to discover tiers from. |
--models-file <f> | Use an explicit models.json. |
--no-open | Don't open / auto-serve the dashboard (also implied by CI=1). |
Parallelism & exit codes
envKey)
run concurrently but serial within a family, so one provider's rate
limit is never hammered. Config runs stay serial. Force serial everywhere with
--serial. The runner exits non-zero only if the
harness itself errors — a weak model scoring badly is the measurement, not a
build failure.
Isolation
The default --sandbox none runs the agent in-process: fast and
CI-friendly, but with no filesystem restriction — a capable model could read
host gold data. --sandbox gondolin runs the agent's bash/file
tools in a QEMU micro-VM that only sees its own workspace, while keeping
github_* in-process so the fake GitHub still works (this is why
gondolin, not docker/smol, is the
supported backend). It needs QEMU with hardware acceleration and adds a one-time
cold start; a fail-fast preflight aborts rather than hanging.