Evals

Models

The model registry lives in models.json. Resolution order: an explicit --models-file <path> wins, else an overlay's evals/models.json if present, else the built-in registry.

models.json

{
  "default": "openai/gpt-5.4-mini",
  "compare": [
    { "id": "openai/gpt-5.5",              "label": "GPT-5.5",         "provider": "OpenAI",           "envKey": "OPENAI_API_KEY" },
    { "id": "anthropic/claude-opus-4-8",   "label": "Claude Opus 4.8", "provider": "Anthropic",        "envKey": "ANTHROPIC_API_KEY" },
    { "id": "fireworks/accounts/fireworks/models/glm-5p2", "label": "GLM-5.2", "provider": "Fireworks (open)", "envKey": "FIREWORKS_API_KEY" }
  ]
}
  • default — the single model a bare run uses.
  • compare — the cross-vendor set --compare fans out over. Each entry runs only if its envKey is present, so the set auto-trims to whatever keys you have. An entry with no key is a silent no-op, not an error.

Per entry: id is an agentic-pi provider/model spec (Fireworks ids use the long fireworks/accounts/fireworks/models/<x> form), label is the scorecard display name, provider groups families for parallelism, and envKey is the gating variable. A new provider just needs its key set plus a registry id — no harness change.

Selecting models on the CLI

lastlight-evals run triage --model haiku            # fuzzy substring match against id/label
lastlight-evals run triage --model openai/gpt-5.5   # exact provider/model id
lastlight-evals run triage --model glm,deepseek     # comma-list
lastlight-evals run --compare                       # whole compare set (key-gated)
EVAL_MODELS="openai/gpt-5.5,anthropic/claude-sonnet-4-6" lastlight-evals run   # env override

Two run types: --mode

A run compares N arms along one of two axes. In a TTY with no model flags you're asked which; otherwise pick with --mode.

ModeWhat variesUse it to…
models (default) One model, forced across every workflow step. --model / --compare pick the set. Compare models head-to-head. Lands in eval-results/<tier>/ (or <tier>-compare/).
config A deployment's real per-step config — the models/variants maps from an overlay's config.yaml, merged over core's default exactly as prod, so each phase can run on a different model. Measure a real deployment config. The arm is the config/overlay. Lands in eval-results/<tier>-config/ on its own trend line. 
lastlight-evals run code-fix --mode config --overlay examples/overlay
lastlight-evals run code-fix --mode config --overlay A --overlay B   # 2 configs side-by-side
lastlight-evals run code-fix --mode config --overlay A --model haiku # override a config's default

examples/overlay/ ships as a ready-to-run sample. Both run types share all downstream machinery — work list, scorecard, dashboard — they differ only in how the model is chosen per step. A config run's dashboard view adds a Per-step models panel showing each phase's resolved model.