Evals

Triage

The triage tier runs the real issue-triage workflow: the agent reads a freshly-seeded issue and decides how to triage it — which labels to apply, whether to ask for more info, whether to close it. It's the cheapest and fastest tier, so it's the best one for smoke-testing the harness or a forked workflow. 

lastlight-evals run triage                       # default model
lastlight-evals run triage --model haiku         # one model
lastlight-evals run triage --compare             # cross-vendor set
lastlight-evals run triage --runs 3              # repeat each case (worst-case verdict)
A triage comparison scorecard: nine models ranked by behavioral pass rate, with cost and latency bars, and per-instance rows showing the exact label / category / state checks each case asserts.
A triage comparison — behavioral pass rate, and the exact label / category / state checks per case.

How it's graded (behavioral)

Triage grading is behavioral: it asserts the GitHub mutations the agent actually made — recorded by the in-process fake GitHub — against the instance's expectation. A case passes only if every expected mutation happened and every forbidden one didn't:

  • expect_github.labels_added / labels_absent — labels the triage must (or must not) apply.
  • expect_github.comment_matches — a regex the posted comment must match (e.g. asking for a repro).
  • expect_github.issue_closed — whether the issue should end up closed.
  • triage_gold — the expected { category, state } classification per your deployment's taxonomy.

Each per-instance row in the dashboard shows exactly which checks it asserted (label:bug, category=bug, state=ready-for-agent, …), so a pass or fail is legible at a glance. Only the fields present on an instance are checked, so a case can assert as much or as little as you want.

Adding a triage case

Two ways. Hand-write a SweBenchInstance in datasets/triage/instances.json (the issue seed, triage_gold, and expect_github), or scaffold from a real resolved issue: 

lastlight-evals add-case --issue <url> --dry-run   # propose a triage case; don't write
lastlight-evals add-case --issue <url>             # write into ./datasets/triage (or --overlay)

add-case --issue pulls the issue content and the labels that were actually applied — read from the issue events API, so the evidence shows who applied each (maintainer vs bot) — seeds the issue without its triage labels so the agent triages fresh, and sets expect_github.labels_added / issue_closed. You then assign triage_gold per your taxonomy. The full flow is on the Authoring cases page.

On tiny synthetic fixtures a model can occasionally pick the wrong owner/repo for a GitHub call (there's no real remote to infer from) — that surfaces as a behavioral miss. Stronger models fare better; it's exactly the kind of thing the eval is meant to reveal.