Evals

Install & first run

Getting a first eval on the board takes two things: Node 24+ and one model provider API key. GitHub is mocked, so you do not need any GitHub credentials to run evals.

This is the manual path. The default way to use the harness is to let a skill drive it — see Drive it with Claude. The commands below are what that skill runs for you.

Install

node --version                  # need >= 24
npm install -g lastlight-evals  # pulls in lastlight (core) + agentic-pi
lastlight-evals --version       # prints the evals + bundled lastlight core versions
Prefer not to install globally? Every command works with npx lastlight-evals ….

Set a provider key

Set at least one provider key in your environment or a .env file in the directory you run from (auto-loaded, KEY=VALUE, no quotes). The compare set only runs the models whose key is present, so one key is enough to start. 

# .env
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
FIREWORKS_API_KEY=fw-...      # GLM / DeepSeek / GPT-OSS (open models)
OPENROUTER_API_KEY=sk-or-...
Don't put a real GITHUB_TOKEN in .env. The harness sets its own dummy token and unsets the GitHub App vars so no real installation token is ever minted. (The one command that does read real GitHub is add-case, which uses your gh login — see Authoring cases.)

Run your first eval

The sample triage tier is the cheapest, fastest way to see the whole pipeline — work list → run → grade → scorecard → dashboard: 

lastlight-evals run triage

When it finishes (in fact, while it's still running) the CLI starts a tiny local server and opens the dashboard deep-linked at your run, so you watch the scorecard fill in live. The runner exits non-zero only if the harness itself errors — a weak model scoring badly is the measurement, not a build failure.

Where results land

Each run gets its own timestamped folder so runs accumulate instead of overwriting: 

./eval-results/<tierKey>/<runId>/
  scorecard.json      # structured roll-up (per model + per instance), live-updated
  predictions.jsonl   # SWE-bench predictions shape
  sessions/…          # per-phase agent session logs

tierKey is <tier>, <tier>-compare, or <tier>-config, and runId is <timestamp>-<git-sha>. Override the root with LASTLIGHT_EVALS_OUT. Set CI=1 to keep the browser from opening.

Next: turn this into your workspace — with your own workflows and datasets — via init and overlays.