Evals

Drive it with Claude

The default, recommended way to use Last Light Evals is to let Claude Code — or any agent tool that supports skills — drive it for you. Last Light ships a lastlight-evals skill that knows the whole CLI: you describe what you want in plain language, and the skill scaffolds the workspace, authors cases, runs the evals, and reads the scorecard back to you.

The commands in the rest of these docs are what the skill runs under the hood. You can run them by hand — see Install & first run — but you rarely need to. Reach for the raw CLI when you want precise control; reach for the skill for everything else.

Install the CLI and skills

Install the lastlight CLI, then install the bundled, version-matched skills: 

npm i -g lastlight          # or run it via npx
lastlight skills install    # → ~/.claude/skills (user scope)

lastlight skills install uses the claude plugin marketplace when one is present, otherwise it copies the skills into ~/.claude/skills. Add --scope project to install them into ./.claude/skills for a single repo. The evals harness itself (lastlight-evals) is a separate package the skill installs / invokes as needed.

Ask for what you want

Open Claude Code in a working directory and describe the goal — the skill is triggered by intent, so you don't need to remember command names or flags:

Ask Claude…What the skill does
"set up a Last Light evals workspace" Scaffolds an evals workspace — links it to your deployment overlay, or makes a self-contained one.
"run evals" / "compare models on triage" Picks the right tier + models, runs it, and opens the dashboard at the scorecard.
"create an eval case from this PR / issue" Runs add-case against the GitHub URL, curates the fuzzy parts, and writes the case.
"build a PR-review dataset from these gold PRs" Authors a pr-review dataset from your reviewed PRs — often a curation sub-agent per PR, in parallel.
"fork the build workflow and eval the change" Forks the workflow into your overlay, edits it, and re-runs the evals so you can read the scorecard delta.
"raise the pr-review F1 toward a target" Runs the improvement loop — diagnose, propose one generic fix, re-measure on a blind held-out split, keep or revert.
Invoke it explicitly with /lastlight-evals, or just state the goal and let it trigger. It runs against any model the underlying coding agent supports — Claude, GPT, Gemini, and more.

Because GitHub is mocked when evals run, the skill needs only a model provider API key to run and grade — no GitHub credentials. (The one exception is authoring a case from a real PR/issue, which reads GitHub through your gh login.) The next page covers those keys and a first manual run.