Workspaces & overlays

Running the shipped samples is a good smoke test. To eval your deployment — your forked workflows, your prompts, your dataset cases — scaffold a workspace with init, then layer an overlay over the built-ins. (New to overlays? The Configuration docs cover what an instance/ overlay is and how it overrides the built-in config, workflows, prompts, skills, and persona.)

Scaffold with init

init creates a workspace seeded from the built-in samples. It is non-interactive when there's no TTY (agent / CI / piped), so it never blocks on a prompt. There are two layouts.

Recommended: point at an existing deployment overlay

If you already have a Last Light deployment repo (the private instance/ overlay), clone it in as its own checkout and let the runner auto-detect everything:

lastlight-evals init my-evals --clone your-org/lastlight-instance
cd my-evals && lastlight-evals run     # bare run — overlay + datasets auto-detected

This produces:

• instance/ — your overlay, cloned as its own git checkout (git-ignored here; cd instance && git pull to update).
• evals/datasets/ — seeded from the built-in triage + code-fix samples; this is what you edit.
• evals/models.json — a copy of the built-in model registry.
• .gitignore + README.md.

The runner auto-detects ./instance as the overlay (it looks for instance/config.yaml) and ./evals/datasets as the dataset root, so a bare lastlight-evals run just works — no --overlay flag. Re-running init --clone is idempotent; it won't re-clone an existing instance/.

Self-contained: workspace is the overlay

No existing overlay repo? Scaffold a workspace that is its own overlay — it gets workflows/, skills/, agent-context/ placeholders and its own config.yaml alongside evals/:

lastlight-evals init my-evals          # or: lastlight-evals init  (→ ./lastlight-evals-workspace)
cd my-evals && lastlight-evals run --overlay .

In a TTY it offers git init plus a private gh repo create.

Fork the default workflows

An overlay's workflows/, skills/, and agent-context/ shadow the core built-ins by logical name — the same deployment-overlay mechanism production uses. To customize a workflow, fork it into your overlay with the lastlight CLI. This works from an evals workspace with no core checkout, because the defaults it forks from ship inside the lastlight package:

lastlight fork                       # list forkable workflows + agent-context (marks what's already forked)
lastlight fork <workflow>            # copy a workflow YAML + every prompt & skill its phases reference
lastlight fork agent-context         # copy soul.md / rules.md / security.md (the persona)
lastlight fork agent-context soul.md # just one persona file
#   --force to overwrite an existing copy; --home <dir> to target a specific working dir

Run it from the workspace root to target ./instance, or from inside ./instance. Then edit the copied files under instance/… and re-run the evals to measure the effect — overlay workflows and skills layer over core, and overlay datasets under <overlay>/evals/datasets/ shadow the built-in tiers of the same name.