Evals

Code fix

The code-fix tier runs the real build workflow through a full cycle: the agent reads an issue, edits code in a seeded workspace, and opens a PR against the in-process fake GitHub. It's the heavy tier — it clones and runs real repositories — and it's graded the way SWE-bench grades: by executing held-out tests. 

lastlight-evals run code-fix                     # default model (heavy)
lastlight-evals run code-fix --model haiku       # one model
lastlight-evals run code-fix --compare           # cross-vendor set
A code-fix comparison scorecard: nine models ranked by resolved rate, with bar charts for total cost and P50 latency, token usage, and error count, plus per-instance rows with resolved/tests/pr-opened badges.
A code-fix comparison — resolved rate, cost, and latency across nine models.

How it's graded (execution)

Grading is execution-based: held-out tests are copied into the agent's final workspace and run. A case is resolved only if

  • every test named in FAIL_TO_PASS now passes (the bug is fixed), and
  • every test named in PASS_TO_PASS still passes (nothing regressed).

That's SWE-bench's resolved criterion. Held-out tests live in datasets/code-fix/tests/<id>/, kept out of the seeded repo so the agent can't edit them. A non-TAP test runner falls back to suite mode — the case resolves iff the test command exits 0. The dashboard's per-row tests button shows the captured setup log + TAP for resolved and unresolved cases alike, and files shows the agent's diff.

Two shapes of case

ShapeWhat's on disk
Vendored fixture instances.json + repos/<id>/ (the fixture at its base commit) + tests/<id>/ (held-out tests). Self-contained and fast.
Git-source A real repo + base_commit + test_patch, no vendored repos/<id>/. At run time the repo is cloned into the gitignored ./.eval-cache/ and checked out at base_commit.

Adding a code-fix case from a real PR

The usual path is a git-source case authored from a merged PR: 

lastlight-evals add-case --pr <url> --code-fix --dry-run   # propose; don't write
lastlight-evals add-case --pr <url> --code-fix             # write into ./datasets/code-fix (or --overlay)

It derives repo, base_commit (the merge-base of the base branch and PR head), head_commit, the test diff, and the issue fixture. Two grading modes: suite (default — run the repo's own test_cmd, resolved iff it exits 0) or hold-out (--hold-out, SWE-bench style — hide the PR's test files and auto-detect FAIL_TO_PASS by running tests at base vs head). Details on the Authoring cases page.

A code-fix case runs real repository code during validation and grading — only use trusted repos. For stronger isolation, run under --sandbox gondolin.