Evals

Improving the score

Running an eval tells you a score. The improvement loop raises it — mine the failures, propose a few minimal fixes, re-measure, keep the best or revert, repeat — until you hit a target or plateau. It's driven by the lastlight-evals-loop skill (say "raise the pr-review F1"), and it's built to do this without gaming the eval.

The method — mine weaknesses → propose a few minimal candidates → keep the one that survives a blind held-out gate — follows Self-Harness: Harnesses That Improve Themselves, adapted to keep the anti-gaming guardrails below. The key adaptation: because this loop can see the gold answers, it ranks candidates on the train split and confirms only the winner on held-out — once — so screening many candidates can't quietly overfit the blind set.

The trap this avoids. The loop can see both the gold answer and the agent's answer for every case, so the lazy path is to overfit — hardcode a repo's finding into a prompt, or edit a gold answer to force a pass. That lifts the number and ruins the reviewer. Every guardrail below exists to force generic, cross-cutting changes that would help on repos the loop has never seen.

Scope today is the pr-review tier — its judge trace gives the agent-vs-gold detail that diagnosis needs (which real findings were missed, which posted findings were noise). The pattern extends to triage and code-fix.

The one rule: one change kept per round

A round may explore a few minimal candidates, but at most one is ever kept and committed to the overlay — the rest are reverted. So which edit moved the number is always attributable, and no overfit edit rides along with a good one. Candidates are measured each on their own branch, and the blind held-out set is spent once per round, never once per candidate.

The loop

  1. Split the tier's cases into a train set (you diagnose on these — traces visible) and a held-out set (blind — you never read its traces; only its aggregate F1 gates a keep). Fixed for the whole loop.
  2. Baseline — run both splits, record F1 for each.
  3. Diagnose (mine) the train failures with mine-failures.ts: it clusters missed real issues (false negatives → recall loss, weighted by severity) and noise (false positives → precision loss) into a ranked signature bundle, so you target the systematic pattern with the most headroom, not a one-off.
  4. Propose a few (2–4) minimal, diverse candidate fixes for the top pattern, each at the lowest lever that could move the whole cluster (below).
  5. Audit each (below) — reject anything case-specific or answer-leaking.
  6. Rank on train, confirm on held-out once — pick the best candidate by its train lift, then give that one winner a single blind held-out check. Keep it only if train improves and held-out doesn't regress; revert the rest.
  7. Journal the pattern, every candidate's delta, the winner, and the decision. Repeat until the target or a plateau.

What keeps it honest

GuardrailWhat it prevents
Held-out splitThe empirical anti-overfit gate. A change is kept only if train improves and the blind held-out set doesn't regress. A train gain with a held-out drop is overfitting → revert.
Generality auditorAn adversarial sub-agent rejects any prompt/skill edit that names a specific repo, instance, or file — a fix must apply to any repo.
No-gold-leak auditorInjected repo context must read as plausible maintainer guidance written without knowledge of this PR's bug — never "look for finding X". This is the key check for the context lever.
Lever ladder + sign-offGeneric edits auto-apply; the game-able levers (per-repo context, gold edits) stop for human sign-off. Gold is edited only when it's demonstrably wrong — never to force a pass.
Never touch coreChanges live in the overlay and, with sign-off, the dataset — the real workflow always runs unmodified.

The three levers

Every change lands in one of three places, preferred lowest-first:

  1. Prompts / skills / persona (generic — auto) — the reviewer's rubric, precision bar, or what-to-check list, edited in the overlay so it applies to every repo. The highest-leverage lever, and the default.
  2. Repo context (portable — signed off) — a synthetic AGENTS.md the harness injects into the checkout. A generic block helps every repo; a per-repo block is the "add this to your repo" recommendation.
  3. The eval itself (rare — signed off) — fixing a review_gold entry, only when the gold is demonstrably wrong or incomplete, with the evidence named.

Measuring a change

Two read-only helpers back the loop. mine-failures.ts turns a train scorecard into the ranked failure-signature bundle you diagnose from. diff-runs.ts diffs two scorecards — per-case F1 before → after, the arm summary delta, and, given the train / held-out id lists, the keep-or-revert verdict: keep only when train improved and held-out held. Both read only the ids you pass, so the held-out split stays blind. 

# Diagnose: rank the train-split failures by impact.
npx tsx scripts/mine-failures.ts <train-scorecard>.json --train <train-ids> --keywords

# Decide: the keep-or-revert verdict for the winning candidate.
npx tsx scripts/diff-runs.ts <baseline>.json <candidate>.json \
    --train <train-ids> --heldout <heldout-ids>

# → per-case deltas, arm summary, and:
# VERDICT: KEEP — train ↑ and held-out held  (or)  REVERT — OVERFIT: train ↑ but held-out regressed
# --symmetric swaps in the non-regressive gate: neither split may regress, one must improve.
The loop produces two kinds of durable output: workflow improvements (better prompts/skills for every repo) and per-repo recommendations (context a maintainer can commit). Both are backed by a measured, held-out lift — not a number that only looks good on the cases you tuned against.