Evals

PR review

The pr-review tier measures review quality. The real pr-review workflow posts a review, and an LLM judge scores it against a human-verified gold set → precision, recall, and F1. It's the one tier graded by a judge — matching free-text findings to semantic gold comments can't be done deterministically. Triage and code-fix stay judge-free.

The gold set can come from two sources, and the tier runs the same either way:

  • Your own reviewed PRs — build a dataset from real PRs your team has reviewed. This is the everyday path for tuning a deployment.
  • Martian's Code Review Bench — an optional public benchmark of 50 PRs that also lets you see where a run would rank against published tools.
A pr-review scorecard: F1 with precision and recall, cost and latency, and a per-instance table where each row shows matched-vs-gold counts, false positives, and a judge button.
A pr-review scorecard — F1 headline, per-instance precision/recall, and a judge button on every row.

Option A — bring your own gold PRs

Author cases from real PRs your team has already reviewed with add-case --pr <url> --review. It pulls the PR fixture and a candidate review_gold from the human review (inline comments + substantive review bodies, with bot / nit / reply / LGTM noise filtered out) and prints an evidence block; you assign real severities and prune non-actionable comments. 

lastlight-evals add-case --pr <url> --review --dry-run   # propose; read the evidence, curate
lastlight-evals add-case --pr <url> --review             # write into ./datasets/pr-review (or --overlay)
lastlight-evals run pr-review --model <model>            # run against your dataset

Re-running a PR replaces by id, so building up a dataset one PR at a time is safe. Cases you author are byte-compatible with Martian imports, so the two can coexist in one instances.json. The full authoring flow — severities, the anti-spoil guarantee, curating for many PRs at once — is on the Authoring cases page.

Option B — benchmark against Martian's Code Review Bench

Martian's Code Review Bench is an optional public benchmark: 50 real merged PRs (Sentry, Grafana, Cal.com, Discourse, Keycloak), each carrying inlined golden_comments with base/head SHAs pinned. Because they're large real-repo PRs, the tier ships empty — you import them: 

npx tsx scripts/import-martian.ts            # resolve all 50 via gh (pins base/head SHAs)
npx tsx scripts/import-martian.ts --limit 3  # a quick subset first
npx tsx scripts/import-martian.ts --dry-run  # print the first resolved instance, don't write

It needs an authenticated gh and network. Entries with no golden_comments or an unrecognized source URL are skipped and logged — never silently dropped. Then run the tier exactly as above: lastlight-evals run pr-review. The payoff of using Martian's set is the leaderboard ranking below.

How the judge grades

Whichever source the gold comes from, grading is a two-step LLM judge:

  1. Extract the review's distinct, concrete findings (dropping praise and summaries).
  2. Match each finding to a gold comment — same underlying issue?

From the matches: precision = matched ÷ posted, recall = matched ÷ gold, combined as an F-beta. The headline is F1 (β = 1, precision and recall weighted equally — Martian's leaderboard metric). Pass --f-beta 0.5 (or EVAL_F_BETA=0.5) to weight precision 2×; the dashboard relabels the column F{β} to match. 

lastlight-evals run pr-review --model <model>                   # full tier
lastlight-evals run pr-review --model <model> --limit 3         # first 3 cases (controlled)
lastlight-evals run pr-review --model <model> --f-beta 0.5      # weight precision 2×
lastlight-evals run pr-review --model <model> --judge-with-diff # give the judge the diff
Gold sets are incomplete — treat the score as relative. A gold set (yours or Martian's) can't list every valid finding, so a real-but-unlisted finding scores as a false positive and understates precision. That's why the default is F1, not the precision-weighted F0.5. Don't re-attribute F0.5 to Martian — their leaderboard headline is F1.

The judge is independent — and inspectable

The judge model is independent of the models under test (a strong default per your provider key, overridable with EVAL_JUDGE_MODEL; one-shot, temperature 0). A judge failure marks the case errored (ungraded), never a silent zero. By default it's diff-blind — it reads only the posted review, mirroring Martian's offline judge; --judge-with-diff feeds it the PR diff for higher-fidelity matching (and marks those grades diff-aware, trading away leaderboard parity).

Every per-instance row has a judge button that opens the judge's full working — the findings it extracted, the gold set, and the finding ↔ gold pairing (matched / false positive / missed) — so an F1 score is never a black box:

The judge trace modal: findings extracted from the review, each tagged matched or false positive, the gold set with which finding matched each, and expandable raw judge replies.
The judge trace — extracted findings ↔ gold, false positives, and the raw replies behind the score.

Give the reviewer repo context

A PR is locked at its head SHA, so the reviewing agent only sees the code — not the conventions a maintainer carries in their head. The harness can inject a synthetic AGENTS.md / CLAUDE.md into the checkout so the agent reads it (the coding agent auto-loads it, walking up from its working directory — no prompt change needed). Two presence-based sources, on by default:

  • Generic — <overlay>/repo-context/AGENTS.md, injected into every reviewed repo. For guidance that helps on any codebase (e.g. "prefer concrete, line-anchored findings; suppress pure style nits").
  • Per-repo — datasets/pr-review/context/<instance_id>/AGENTS.md, injected only for that case's repo. This is the portable "add this to your repo and reviews improve" recommendation — honest because it's exactly what a maintainer could commit.

It appends to a real AGENTS.md / CLAUDE.md if the repo ships one (never shadowing it), and records which context each case saw on the scorecard. Toggle it off for a clean control run to measure the lift: 

lastlight-evals run pr-review --overlay instance                      # inject (default)
lastlight-evals run pr-review --overlay instance --no-inject-context  # control (or EVAL_INJECT_CONTEXT=0)
Proving that a piece of context lifts review quality — generically or for one repo — is exactly what the improvement loop drives, alongside prompt and skill changes.

Where would this rank? (Martian only)

When your run covers Martian PRs, each scorecard also computes a leaderboard sidecar: where would this run place against the benchmark's tools, over the same PRs this run covered (micro-averaged F1; only tools with data on every one of those PRs are shown). It's a like-for-like slice, not the full leaderboard — and it only appears for Martian-sourced cases, since your own PRs aren't on it.

A 'Where would this rank' bar chart placing Lastlight among Martian's Code Review Bench tools — Augment, Cubic, Qodo, Gemini, Claude Code, Devin, CodeRabbit and others — with the Lastlight row highlighted.
The leaderboard sidecar — a Martian run ranked against published tools over the same PRs.