Evals

Authoring cases

add-case turns a real GitHub PR or issue into an eval case. It does the mechanical extraction via gh + git — base/head SHAs, test diffs, applied labels, red→green verdicts — and you refine the fuzzy parts (which findings are really gold, which severity, which labels are triage-relevant).

add-case is the one command that reads real GitHub. It requires gh on PATH and authenticated (gh auth login) and checks both up front. A PR can seed two different evals, so --pr requires a kind — --review or --code-fix.

The three flows

lastlight-evals add-case --pr <url> --review --dry-run     # propose a pr-review case; don't write
lastlight-evals add-case --pr <url> --code-fix --dry-run   # propose a code-fix (build) case
lastlight-evals add-case --issue <url> --dry-run           # propose a triage case

--dry-run prints the proposed instance JSON so you (or an agent skill) can read the evidence and curate before writing. Drop it to write into ./datasets/<tier> (or --overlay <dir>'s evals/datasets). Re-running the same PR/issue replaces by id, so accumulating a dataset is idempotent.

--pr --review → a pr-review case

Extracts the PR fixture (refs + SHAs, title, truncated body) and a candidate review_gold from the PR's human review — inline comments and substantive review bodies, with bot / nit / reply / LGTM / outdated noise filtered out. It prints an evidence block (author · file:line [severity] first-line); you assign real severities and prune non-actionable comments. Anti-spoil: the review lands only in review_gold, never in the fixture the mocked GitHub would show the agent.

--pr --code-fix → a build case

Two grading modes:

  • Suite (default) — nothing held out; graded by running the repo's own test_cmd against the agent's final tree, resolved iff it exits 0. add-case warns if the suite is already green at base or doesn't pass at head.
  • Hold-out (--hold-out) — SWE-bench style: the PR's test files are hidden and applied only at grade time; auto-detects FAIL_TO_PASS by running tests at base (red) vs head (green).

It derives repo, base_commit (the merge-base of the base branch and PR head — the true fork point), head_commit, the gold patch (reference only), the test diff, and the issue fixture. No repos/<id>/ is vendored — at run time the repo is cloned into ./.eval-cache/ and checked out at base_commit.

A code-fix case runs real repo code during validation and grading — only use trusted repos.

--issue → a triage case

Derives the issue seed + problem statement, seeds the issue without its triage labels (so the agent triages fresh), and sets expect_github.labels_added from the labels that were actually applied — read from the issue events API, so the evidence shows who applied each (maintainer vs bot). You then assign triage_gold ({ category, state }) per your deployment's taxonomy and prune non-triage labels.

add-case flags

FlagMeaning
--pr <url>GitHub PR url (pick --review or --code-fix).
--reviewWith --pr → a pr-review case (gold = the PR's human review).
--code-fixWith --pr → a code-fix (build) case.
--issue <url>GitHub issue url → a triage case.
--tier <name>Target tier dir (defaults per kind).
--id <slug>instance_id (default derived from repo + number).
--datasets <dir>Datasets root to write into. Default ./datasets, else ./evals/datasets.
--overlay <dir>Write into <dir>/evals/datasets instead.
--severity <level>(--review) default severity for candidate gold comments (low|medium|high|critical, default medium).
--include-bots(--review) keep bot review comments (default: skipped).
--test-cmd "<cmd>"(--code-fix) held-out test command (default node --test).
--setup-cmd "<cmd>"(--code-fix) install/build before tests (e.g. "npm ci").
--hold-out(--code-fix) SWE-bench mode: hold the PR's test files out and grade named FAIL_TO_PASS.
--pass-list(--code-fix, hold-out) enumerate every green test in PASS_TO_PASS (default ["*"]).
--no-validate(--code-fix) don't run the repo's tests to validate — just scaffold.
--dry-runPrint the proposed instance JSON; don't write.
-h, --helpShow help.
Building a PR-review dataset from a set of gold PRs? Run each through --review --dry-run, curate, then write; replace-by-id makes accumulation safe. A per-PR curation sub-agent (via the lastlight-evals skill) can do them in parallel.