Workflow Engine

Purpose

The workflow engine is the part of Last Light that decides what to run, in what order, with what inputs, and how to recover when the process dies. It is workflow-agnostic — the runner doesn’t know build.yaml from issue-triage.yaml. It loads a definition, executes phases, calls out to the Sandbox for each agent session, persists state to the state database, and handles every gate and loop the YAML can declare.

Every behaviour in Last Light — build, triage, review, explore, health, security, answer, verify, qa-test, demo — is a YAML file consumed by this engine.

Public contract

export async function runSimpleWorkflow(
  workflowName: string,
  request: SimpleWorkflowRequest,
  config: ExecutorConfig,
  callbacks: RunnerCallbacks,
  db: StateDb,
  models?: ModelConfig,
  approvalConfig?: ApprovalGateConfig,
  bootstrapLabel = "lastlight:bootstrap",
  variants?: VariantConfig,
  concurrency?: { maxWorkflows: number; maxQueueWaitMs: number },
): Promise<WorkflowResult>;

interface WorkflowResult {
  success: boolean;
  phases: PhaseResult[];
  paused?: boolean;     // hit an approval or reply gate
  queued?: boolean;     // held back by the concurrency cap (no phases ran)
  prNumber?: number;    // build cycle that produced a PR
}

The engine’s own WorkflowResult (above) is unchanged. runWorkflow (src/workflows/runner.ts) returns a server-only intersection, WorkflowResult & { backpressure?: boolean } — see Concurrency cap and admission below. backpressure is set only on the k8s sandbox backend; it never appears on gondolin/docker/smol/none runs.

src/workflows/simple.ts:84–310 is the entry point everything funnels through — webhook dispatch, CLI, cron, admin resume.

YAML schema

Workflow level (packages/workflow-engine/src/core/schema.ts:476–617, re-exported as src/workflows/schema.ts):

{
  kind: string;          // "agent" by default; "build" / "triage" / etc. for categorisation
  name: string;          // unique workflow name; lookup key
  description?: string;
  trigger?: string;      // informational
  pr_scoped?: boolean;   // this workflow runs against a PULL REQUEST — see below
  // Runtime policy (issue #368) — see "Runtime-policy keys" below
  git_access?: "read" | "issues-write" | "review-write" | "repo-write";
  workspace?: "per-run" | "per-target-reuse" | "per-target-recreate";
  prepopulate_synth_branch?: boolean;
  prepopulate_pr_head_ref?: boolean;
  pr_fix_shaped?: boolean;
  variables?: Record<string, string>;
  classification?: {     // how the intent classifier routes to this workflow (issue #164)
    intent: string;      //   the intent token this workflow owns (unique; not a control intent)
    description: string; //   the category paragraph merged into the composed classifier prompt
    examples?: string[]; //   optional one-line classifier examples
  };
  chat?: {               // how the CHAT agent advertises this workflow to a human
    trigger?: string;    //   the phrase to tell a user to type, e.g. "triage owner/repo"
    summary: string;     //   one line naming what they get
    deflect?: string[];  //   user phrasings to deflect here rather than attempt in-process
    reply?: string;      //   override for the deflection reply (default: name the trigger)
  };
  phases: PhaseDefinition[];
}

pr_scoped: true is the first of six keys here the runner acts on. It puts a workflow inside the PR-scoped dispatch gate: the run lock shared with every other PR-scoped workflow, the per-head-SHA dedup, escalation, and the resolved PrState snapshot on context.prState. prScopedWorkflows() (src/workflows/pr-scope.ts) derives the set from this metadata, memoised on the loader’s asset version, so a forked or renamed workflow keeps its gate by carrying its own key. It is metadata on the workflow because that is where the fact lives: the harness previously held a hardcoded set of four names while the handlers are operator-configurable through routes.github.*, so remapping a route to a fork silently dropped the whole gate for it — and every consequence of the gate is a refusal, so nothing looked wrong until two agents pushed the same branch (issue #256). validateAssets now warns at boot when a configured routes.github.pr_* target does not declare it. The four packaged members are pr-fix, dependabot-ci-fix, dependabot-pr-merge and pr-review; those four names are also honoured without the key, for overlays that forked them before it existed, with a warning naming the file. See 05-router.md → the PR-scoped dispatch gate.

Runtime-policy keys

The five keys beside pr_scoped answer the same question for the rest of a workflow’s runtime behaviour: what its GitHub token may do, how its sandbox workspace is keyed, and how its checkout is pre-populated. All five default to the value a workflow got by being absent from the hardcoded tables they replaced, so a workflow declaring none of them behaves exactly as an unlisted one did.

workflowTargetPolicy() (src/workflows/target-policy.ts) derives them from the loaded definitions and memoises on the loader’s asset version, exactly as prScopedWorkflows() does; gitAccessProfileForWorkflow() keeps its signature and becomes a lookup over the same map.

The reason they are metadata is the reason pr_scoped is. Each was a literal name table in compiled core code — a switch in runner.ts and four Sets in target-policy.ts — so a workflow shipped only in a deployment overlay could never register itself into any of them. It got a read token (cannot submit a review, cannot push) and no head-ref pinning (reviews the base branch), and the only fix was a core patch naming an arm core does not ship (issue #368). Note that git_access differs from pr_scoped in kind: it is not a gate that only adds restrictions, it decides what the minted token can do, so an overlay declaring repo-write mints itself a push token. That is deliberate — overlays already own prompts, skills and the agent persona.

Two loader invariants, both throwing at boot:

• pr_fix_shaped ⇒ pr_scoped. handlePrFix reads the head branch, the fork verdict and the CI evidence off the PrState snapshot that only the PR-scoped dispatch gate resolves; outside the gate there is nothing to fix.
• workspace: per-target-recreate ⇒ a pre-populate source. Recreate-from-base only means anything for a workflow whose workspace is pre-cloned.

The cutover was hard — the tables were deleted, not unioned in, so unlike pr_scoped there is no legacy-name floor. All sixteen packaged workflows declare their behaviour explicitly and tests/workflows/target-policy.test.ts pins the derived policy against the deleted tables entry-for-entry. For an overlay that forked a built-in before these keys existed, validateAssets warns at boot when a workflow reachable from a configured routes.github.* declares none of the five.

The optional classification block makes a workflow self-describing to the router: its description/examples are composed into the classifier prompt (workflows/prompts/classifier.md), and its intent becomes routable via the router’s getWorkflowByIntent fallback — so adding a workflow (even in an overlay) can add a new intent with no core change. See 05-router.md → Build-intent classifier.

The optional chat block is its counterpart for the chat agent: it is what the bot tells a human to type, composed into the chat system prompt from the enabled workflow set. It is a separate, explicit opt-in rather than a derivation from classification, because “the classifier can tag a message with this intent” and “a human should be told to type this” are different questions that diverge in both directions (demo is classifiable but unroutable from Slack; the two dependabot workflows are routable but would arrive with no PR number). An entry may omit trigger and give reply instead, for a workflow that must be explained but never typed — repo-health is cron-only and has no classification block at all. See 11-chat.md → Advertised capabilities.

The same block feeds the Slack rows of the dashboard trigger table (getWorkflowTriggers, src/workflows/triggers.ts), resolved through routes.slack so an operator’s remap lands on the fork. Those rows gate on the classification.intent (what the Slack switch dispatches on) rather than the trigger phrase, so a chat: entry without an intent contributes an explanation to the agent and no trigger row. They were a hand-kept list of five while the router routed nine, which is why the dashboard showed no Slack trigger for verify / qa-test / demo / answer.

Phase level (schema.ts:84–182):

{
  name: string;                         // unique within workflow
  label?: string;                       // dashboard display
  type?: "context" | "agent" | "bash" | "script"   // default "agent"
        | "fanout" | "post-review" | "survey-units";  // these three run via app-registered handlers
  prompt?: string;                      // path to template, e.g. "prompts/architect.md"
  command?: string;                     // type: bash — deterministic shell command (templated)
  script?: string;                      // type: script — inline source (templated)
  runtime?: "js" | "ts" | "python";     // type: script — js/ts → node, python → uv run (default "js")
  timeout_seconds?: number | { from: string };  // phase budget; unset → sandbox.agentTimeoutSeconds / commandTimeoutSeconds / untilBashTimeoutSeconds
  skill?: string;                       // single skill name; sugar for `skills: [<name>]`
  skills?: string[];                    // per-phase bundle: <workspaceRoot>/.lastlight-skills/<phase>/<name>/
                                        // may coexist with `prompt`; mutually exclusive with `skill`
  model?: string;                       // can be "{{models.architect}}"
  variant?: string;                     // reasoning effort; can be "{{variants.fix}}"
  approval_gate?: string;               // pause gate name
  approval_artifact?: string;           // handoff doc this gate approves (e.g. architect-plan.md)
  approval_gate_message?: string;       // template rendered when pausing ({{approvalUrl}} deep-links the focused view)
  depends_on?: string[];                // declaring this ANYWHERE disables chain synthesis for the WHOLE workflow
  trigger_rule?:
    | "all_success" | "one_success"     // DAG firing conditions
    | "none_failed"                     // no dep FAILED — an all-skipped set passes
    | "none_failed_min_one_success"     // no dep failed AND at least one succeeded
    | "all_done";
  branches?: FanoutBranch[];            // type: fanout only — this OR branches_from, rejected elsewhere
  branches_from?: { file: string; max: number };  // fanout only — the branch list is a run-time manifest (see below)
  branch?: FanoutBranch;                // fanout only, with branches_from — the per-item template ({{item.*}})
  max_concurrent?: number | { from: string; default: number };  // fanout width, clamped by the backend ceiling
  on_branch_soft_failure?: { retries: number; then: "fail" | "complete" };  // per-BRANCH; not generic_loop's key
  on_branch_gate_failure?: { retries: 0 | 1 };  // fanout only — re-run a branch whose until_bash said no, once
  skip_satisfied_branches?: boolean;    // fanout only — run each until_bash first; a branch whose gate already closes starts no session
  output_var?: string;                  // alias for {{this.field}} in later phases
  unrestricted_egress?: boolean;        // bypass strict allowlist for this phase
  web_search?: boolean;                 // enable agentic-pi web tools
  command_policy?: CommandPolicy;       // allow|log|block per bash command class (install, install-scratch, test, host); see "Command policy"
  requires_sandbox?: "docker" | "gondolin" | "none";  // skip phase (non-failing) if active backend differs
  sandbox_image?: "default" | "qa";     // docker only: "qa" runs on lastlight-sandbox-qa (Playwright+Chromium+ffmpeg); skips if unbuilt
  skip_if?: string | string[];          // skip phase (non-failing) when any expression matches the render context
  loop?: PhaseLoop;                     // reviewer-fix loop
  generic_loop?: GenericLoop;           // until-condition loop
  on_output?: OutputRule[];             // contains_BLOCKED → fail; requires_marker → fail if final output lacks this marker (postcondition against silent no-op "successes")
  on_success?: { set_phase: string };   // terminal marker
  messages?: PhaseMessages;             // per-event reply templates
}

Cron level (kind: cron, same file). A cron definition is not runnable itself — it is a schedule plus what to run:

{
  kind: "cron";
  name: string;          // unique cron name; the dashboard/CLI handle
  schedule: string;      // croner expression
  workflow?: string;     // dispatch this AgentWorkflow on each tick
  handler?: string;      // OR run this host-side handler (src/cron/handlers.ts)
  context?: Record<string, unknown>;   // static context merged into each tick
  condition?: { unless: string };      // named predicate; true ⇒ do not register
}

workflow and handler are mutually exclusive and one is required — a Zod refinement, because both would silently pick one and neither would register a cron that ticks into the void. handler: exists for periodic work that cannot be done by an agent at all (harness-only data, or a Slack post); the trade-offs and the failure rule are in Integrations → Cron.

Defined with Zod; loaded and cached by loader.ts.

Phase types

Seven: context (no execution), agent (one LLM session), fanout (N LLM sessions, concurrently, in one workspace), the deterministic bash / script pair (a command, no LLM), post-review (in-process PR-review submission, no sandbox), and survey-units (in-process per-unit review survey: one bounded model call per unit, no sandbox, no agent).

The engine owns only the generic kinds. post-review, survey-units and any other app-specific type are dispatched through a Map<string, PhaseTypeHandler> injected on EnginePorts.handlers (phase-executor.ts, registered in runner.ts) — which is the seam to extend when a deployment needs a step the engine should not know about.

• context — a checkpoint. The runner writes a phase_history row and moves on. Used to mark dashboard pipeline stages without spending tokens (runner.ts:480–491).

• agent — render the user prompt (from prompt: if set, else auto-generate a nudge toward the primary skill), stage any declared skills into the per-phase bundle <workspaceRoot>/.lastlight-skills/<phase>/<name>/ (mapped via --skill/skillPaths), call executeAgent() in the Sandbox, capture output. Iterates if loop: or generic_loop: is declared. See Phases & Prompts and Skills for the prompt/skill mechanics.

• bash — run a deterministic shell command (command:) inside the sandbox container via executeCommand() (no LLM). Built on DockerSandbox.runCommand (the non-agent sibling of runAgent: docker exec --user agent -w <cwd> … sh -c <cmd>), in the same workspace agent phases use (the host workDir persists across phases by taskId). The command is template-rendered first (so it can reference {{phaseOutputs.<name>}}, {{branch}}, …) then a post-render validateShellCommand guard rejects any leftover {{ marker. Exit 0 = success; a non-zero exit fails the phase and cascades like any phase failure. stdout is exposed downstream like an agent phase (output_var → {{phaseOutputs.<name>}}); upstream string outputs are also forwarded as LL_OUT_<PHASE> env vars. The run is mirrored to a session jsonl (command → bash tool_use, output → tool_result) so it appears in the dashboard + lastlight session log like an agent turn, with turns: 0 and no model cost. On gondolin/none it falls back to a host spawnSync.

• script — same machinery as bash, but runs an inline program (script:) with the runtime in runtime: — js/ts → node (TS via --experimental-strip-types), python → uv run. The source is written to a workspace-root sibling beside the skill bundle (.lastlight-scripts/<phase>/script.<ext>, never inside the repo git tree). Python sources may carry a PEP 723 # /// script inline-dependency block, resolved by uv from PyPI (on the strict egress allowlist) into a cached venv. See Sandbox.

• post-review — a first-class, in-process PR-review submission (PhaseExecutor.runPostReview; no sandbox). It reads the reviewer agent’s .lastlight/pr-review/findings.json for the review content only ({ skip?, summary, event, findings[] }) from the persisted host checkout, and supplies every other fact from the harness’s own run context: the PR number (ctx.prNumber/ctx.issueNumber), the base ref (ctx.baseBranch), and the head SHA + diff (git on the checkout). It anchors each finding to a changed line via src/engine/github/review-poster.ts — deriving the line from the finding’s verbatim existingCode excerpt rather than trusting the number the model counted (own hunks → whole head file → a unique match elsewhere in the diff, which re-files a finding filed against the wrong half of a declaration/implementation pair) — demotes off-diff findings to the body, and posts one review through GitHubClient (App auth in prod; a bearer token + config.githubApiBaseUrl against the eval mock, which serves no App-token or diff endpoint). A genuine failure — missing findings after a real review, or a GitHub error surviving the body-only retry — fails the phase; a legitimate skip succeeds without posting. Idempotent on resume, and the idempotency is narrower than it reads: a handler phase writes no executions row, so shouldRunPhase never skips it and every resume or retry re-executes it. What it must not re-post is the review this run posted, which is not the same fact as “a review exists on the head SHA” — a maintainer’s explicit @bot review on a head we reviewed yesterday is a request for exactly that. resolveReviewPost (src/engine/pr-decisions.ts, the same module the dispatch gate asks) tells them apart on the review the dispatch snapshot saw: still the same one ⇒ prior, and an explicit request overrides it; appeared since ⇒ ours, and nothing overrides that. The two used to be one guard, so an explicit re-review ran the whole pipeline, reported succeeded and posted nothing. This replaced the earlier in-sandbox type: script poster, which depended on the AI hand-writing pr_number/base_ref/head_sha into the JSON and silently exit 0’d on any mismatch.

  The APPROVE floor (approvalHeldBack), applied after the ledger fold: an APPROVE becomes a COMMENT while a human’s standing review is CHANGES_REQUESTED (their latest APPROVED / CHANGES_REQUESTED / DISMISSED — a later COMMENTED does not lift it; read in the same listReviews pass as the bot’s history) or a finding an earlier review posted is still open in the ledger. Without an APPROVE the last-light/review check concludes neutral.

  The document may also carry an optional split verdict (issue #271’s fix 7) — verdict: { spec, standards }, each pass / fail / unknown. A fail on either axis stops the review being an APPROVE; it becomes a COMMENT. Nothing else changes: the event is never escalated to REQUEST_CHANGES on a heuristic (that would flip the last-light/review check to failure and shut a merge gate), and among the non-APPROVE events an explicit event still wins. unknown means not assessed and does not block — most PRs state no acceptance criteria, and blocking on unknown would stop the reviewer approving anything. The point is that a change clean by every standards check but not doing what the issue asked is a case one event cannot express, and 58 of 59 production approvals carried zero findings. An absent verdict is today’s behaviour exactly, and the handler discards the field entirely unless review.analysis.enabled — so the inertness is structural rather than a promise about what a prompt writes.

  A document may also be marked incomplete: { phase, reason } — the conservation floor (lastlight-facts findings --repair, in reconcile) writes one when no earlier phase produced findings.json but hypotheses exist. The handler never posts such a document as a clean review: it posts the document’s own not-assessed summary plus the reason, on every posting branch.

• survey-units — the per-unit review survey (src/workflows/handlers/survey-units.ts), the survey of the evidence pipeline (review.analysis.enabled). Runs on the harness, with no sandbox and no agent: it reads .lastlight/pr-review/units.json (written by the preceding units bash phase, lastlight-facts units) from the host checkout and makes one bounded, non-agentic model call per unit — prompt: (required by the schema) as the system prompt, the unit’s pre-rendered request as the user message — up to review.analysis.surveyUnitConcurrency at once, all under one whole-phase deadline (timeout_seconds, from surveyUnitsTimeoutSeconds). Each reply is written back to the workspace for units-ingest; replies are cached under <stateDir>/unit-survey-cache/<owner>/<repo>/. The phase writes one virtual transcript (so it reads like an agent session in the dashboard) and, unlike post-review, an executions row with its cost. It succeeds on every path inside the phase — no units, every call failing, the deadline — because units-ingest records each unanswered obligation. Needing a host-readable workspace, review.analysis.enabled is refused at config load on kubernetes. See Configuration and docs/plans/pr-review-units-sites.md.

fanout — N agent sessions, one workspace

- name: site-review
  type: fanout
  prompt: prompts/review-site.md
  depends_on: [site-plan]
  trigger_rule: all_done
  model: "{{#if models.review-site}}{{models.review-site}}{{/if}}{{#if !models.review-site}}{{models.review-survey}}{{/if}}"
  max_concurrent: { from: siteConcurrency, default: 6 }
  on_branch_soft_failure: { retries: 1, then: complete }
  on_branch_gate_failure: { retries: 1 }
  branches_from:
    file: .lastlight/pr-review/sites/branches.json   # relative to the agent cwd, like context_file
    max: 16                                          # the spend cap — required
  branch:
    name: "{{item.id}}"
    model: "{{#if item.pair}}{{models.review-site-pair}}{{/if}}"
    context_file: ".lastlight/pr-review/sites/{{item.id}}.md"
    until_bash: lastlight-facts sites --check {{item.id}} --dir .lastlight/pr-review --repo .

A fan-out declares either a static branches: list or branches_from: + a branch: template (the schema refuses both, neither, and a branch: alone). A static fan-out’s list is fixed in the YAML; a dynamic one reads its list from a JSON manifest an earlier phase wrote into the workspace — { "items": [ { "id": "site-001", …scalars } ] } — and renders branch: once per item, with the item in scope as {{item.*}}. The manifest is a file, not an upstream output, because phaseOutputs is empty across a resume boundary while the per-PR workspace survives it.

• Resolution is the pure engine helper resolveDynamicBranches (packages/workflow-engine/src/core/dynamic-branches.ts); the handler reads the file host-side through hostAgentCwd inside the session, before any pre-gate, and everything after that runs on the resolved list unchanged. max_concurrent then clamps against the resolved count.
• Every field but until_bash renders against { ...runContext, item }, so model: can mix config and item vars. until_bash renders against { item } alone, and the schema restricts its placeholders to {{item.<key>}}: it is shell, and the run context carries issue titles and comment bodies. For the same reason a manifest string value must match [A-Za-z0-9._/-]* — the manifest lives in a workspace the agent can write.
• Rendered names pass the same rule a static name does (below) and must be unique. An empty render of any field means “unset” — so a primary above takes the phase’s model:.
• Edge cases. Zero items: the phase succeeds as a no-op (0/0 branches, no branch rows). More than max: the extra items are dropped with a warn. An unreadable or invalid manifest, a bad or duplicate name, or an unsafe value fails the phase — loudly, with one row naming the error, before any branch starts — never the run. A backend whose workspace the harness cannot read (kubernetes) fails it too.
• The plan is recorded before any branch starts, in scratch.fanout[<phase>] = { dynamic, planned: [{ name, model }], truncated }. The dashboard draws not-yet-started branches from it; a resumed run re-resolves the same manifest (the per-PR workspace keeps it) and the ledger dedups the finished branches.
• Branch result rows carry modelTemplate (branch.model ?? phase.model), so a consumer recording per-row models (the evals scorecard) needs no copy of a branch list that exists only at run time.

Each branch inherits the phase’s prompt / skills / model / variant and may override any of them. Branch names are ledger keys, so they are alphanumeric-with-hyphens (no underscores, which PhaseRef uses as its separator), must be unique, and may not end in the reserved -retry / -check / -regate suffixes.

A branch’s skills REPLACES the phase’s — it does not union with it (branchPhase() in handlers/fanout.ts), so an override has to re-list everything it still wants. pr-review.yaml uses none: every branch shares one prompt head, byte-identical, so the branches share one provider-side cached prefix, and what differs per branch arrives last, as its context_file.

(The {{#if}} pair in the model: line above is load-bearing: a bare unset {{models.review-site}} renders empty, and an empty model: resolves to the default model — not to models.review-survey, which is the fallback the phase actually wants.)

context_file — a path in a prompt is not a path. A branch may name a workspace file, relative to the AGENT’S OWN CWD, whose contents the harness reads and appends to that branch’s rendered prompt. The model never resolves it. This exists because it was measured: across three stored pr-review runs on 2026-08-22 (the since-removed agent survey fan-out), 27 of 133 attempts to open the per-family obligations block resolved against the workspace ROOT rather than the checkout and hit ENOENT (23 of 120 branches never recovered), while all 98 relative reads succeeded. The model’s only absolute anchor by its first turn is its skill bundle at <workspaceRoot>/.lastlight-skills/…, one level above the checkout the deterministic phases write in — so a seeded pass silently became an unseeded one. The harness resolves the path against ProvisionResult.hostAgentCwd, the host end of the very cwd a type: bash phase runs in, so producer and consumer share one base by construction. It is appended last, which is the prompt-cache ordering the fan-out already depends on. An unreadable or escaping path (absolute, or containing ..) appends a loud NOT AVAILABLE notice naming it, never silence: “nobody looked” must never render as “looked, found none”. The one carve-out is kubernetes, whose hostAgentCwd is an in-pod path this process cannot see at all — there the read is not attempted and the branch is handed the path to open itself, with the mis-anchoring trap named.

site-review (see Configuration) is the case context_file and branches_from make possible: one branch per site the preceding site-plan bash phase formed — its manifest, sites/branches.json, lists only real sites, up to review.analysis.siteTop (site-001 …) and, with models.review-site-pair set, a second investigator per site (site-001-b …, pair: true). Every branch shares one slot-generic prompt (prompts/review-site.md), and everything that differs per branch — the site, its id, its output file — arrives in the brief site-plan wrote for it. A PR with two sites runs two investigators, not sixteen.

(Until #423 the fan-out declared sixteen static slots and site-plan pre-wrote an empty line for every unused one, which the generic skip_satisfied_branches: true key then pre-gated: it runs each branch’s until_bash before any agent starts, so a branch whose gate already closes is reported done, like a resume dedup, and starts no session. The key remains for any fan-out whose earlier phase settles some branches in code; site-review no longer needs it.)

Why one node instead of N parallel phases. Real DAG concurrency is parked behind four hard blockers, and the WP5 parking rationale records why it can be sidestepped: “every hard blocker exists because each phase provisions its own sandbox against a shared workspace.” A fan-out inside one phase has none of them — one node, one provision, one current_phase, one artifact harvest, one dispose.

Execution order, and each step is load-bearing:

1. One withAgentSession — a single prepareRun, so one GitHub token mint and one workspace provision for the whole fan-out.
2. Skill bundles staged per branch, sequentially, before the join. The bundle is already keyed per phase so concurrent readers cannot collide; staging is filesystem work and is serialised rather than reasoned about.
3. Branches run through a bounded mapPool. A branch that died on a provider error (stopReason: error_agent — the agent loop caught a failed model call, a 404/5xx/rate limit, not a crash or a kill) is re-run once, whatever on_branch_soft_failure says, as <phase>_branch_<name>_retry; at ten or sixteen concurrent sessions per PR it is the common branch failure.
4. until_bash gates run after the join, sequentially. InProcessSandbox.runCommand is a spawnSync — it blocks the event loop — so interleaving a gate with the agent turns would serialise the entire fan-out on none, the very backend the fan-out exists to speed up. Each gate records a <phase>_branch_<name>_check ledger row with condition_met / condition_not_met, and can never fail the phase.
5. on_branch_gate_failure: { retries: 1 } re-runs, once, every branch whose gate ran and said no — concurrently, through the same pool — then runs those branches’ gates again (a second _check row under the same label). The re-run is its own ledger row and phase window, <phase>_branch_<name>_regate (not _retry: the soft retry may already hold that key, and a done row is skipped on dedup). Its prompt is the first attempt’s prompt with the gate’s command and verbatim output appended last, so it shares the cached prefix; the instruction is to fix flagged rows in place and answer only what the output names. A gate that timed out or could not run, a branch that hard-failed, and a branch deduped on resume are never re-run. A re-run that fails leaves the first attempt’s result standing. Absent the key the gate stays purely observational — every fan-out’s behaviour before it existed. pr-review’s site-review declares it (it was introduced on the since-removed survey fan-out, where an observational gate let a half-done hypotheses file — 2 of 12 seeded checks answered — go downstream as if complete).
6. One harvest, one dispose.

The fan-out’s verdict, and the workflow’s. The node fails only when every branch failed. Otherwise it succeeds, and each failed branch row keeps success: false but carries tolerated: true — the scheduler counts a tolerated row as visible, not as a workflow failure. Before it, one site investigator’s provider 404 marked a pr-review run failed after the review had posted, and a failed run leaves the head unassessed (assessedHeadShaByWorkflow counts succeeded runs only), so the review sweep re-dispatched a full review.

Concurrency is min(max_concurrent, backend ceiling), and the clamp is logged when the host has the last word:

A ceiling of 1 runs the branches as a chain — byte-identical in behaviour to declaring them as sequential phases, which is what makes the type safe to ship with gondolin as the production default.

What it keeps, and what it gives up. Each branch goes through the same dedup ledger under its own <phase>_branch_<name> key, so resume, dedup, per-branch cost attribution and the dashboard’s longest-prefix grouping all still work, and a branch opens and closes its own reporter window so per-branch duration is measurable. What it gives up is a DAG node per branch: no per-branch trigger_rule, and no approval_gate (a fan-out cannot pause mid-flight — the schema refuses it, along with loop: / generic_loop:, whose iteration shape the branches already are).

Isolation between branches is by disjoint output paths, not by separate checkouts — they share the one workspace. A fan-out whose branches write the same file is a data race the engine will not catch.

The bash/script deterministic types share the agent phase’s dedup ledger (runCommandPhase → runPhaseLedger), so they get an executions row and dedup on resume like everything else. They also inherit the run’s minted GITHUB_TOKEN (scoped by the workflow’s permission profile). When the harness configures a GitHub API base-url override (config.githubApiBaseUrl, set only by the eval harness to point at its mock), runSandboxedCommand forwards it into the command env as GITHUB_API_URL, and post-review reads it directly to build its GitHubClient; in production both are unset, so GitHub calls fall back to api.github.com.

Scheduling — one DAG, one node at a time

Chain synthesis is automatic:

// src/workflows/runner.ts:330
function hasDependencies(definition): boolean {
  return definition.phases.some(p => p.depends_on?.length);
}

Every workflow is a DAG, and there is exactly one scheduler (core/scheduler.ts). buildDag(phases, { chainIfNoDeps: true }) produces the node graph: a workflow that declares no depends_on anywhere gets a synthesized previous-phase chain, reproducing linear semantics including the failure cascade. Declaring depends_on on any phase disables that synthesis for the whole workflow, so a file that declares one edge must declare them all.

getReadyNodes() returns every node whose dependencies are satisfied per trigger_rule — but the scheduler runs one at a time, in declaration order:

// Sequential: run the earliest-declared ready node, one at a time.
const node = ready[0];

• One workspace per run. Every phase and every loop iteration shares the single ctx.taskId; the sandbox workspace persists between phases, which is how the executor reads the architect’s architect-plan.md.
• Loop iterations run sequentially within their node — each fix cycle reads the previous reviewer verdict.

There is no cross-phase concurrency. Two prior claims in this document — that concurrent phases run via Promise.allSettled() and that each takes a phase-scoped ${taskId}-${phaseName} — described code removed by issue #94, which deliberately collapsed two forked schedulers into one and deferred real concurrency (“via git worktrees”) to a later issue. That issue is still open and parked behind four hard blockers, of which the worst silently overwrites a paused run with succeeded: see the WP5 parking rationale in docs/plans/deterministic-pr-levers.md.

Concurrency exists in exactly one place, and it is inside a node: type: fanout (below).

Gated skips

Before running the ready nodes, the scheduler filters them (core/scheduler.ts). Three declarations can take a node out:

The first two gate on the phase being unavailable; skip_if gates on it being unnecessary. All three take the same path: the node records skipped, a recordSkippedPhase row lands, messages.on_skipped_done is surfaced so a human sees why, and the run still records succeeded.

That last part is the whole reason skip_if exists rather than being expressed with the pre-existing on_output.contains_BLOCKED: { action: fail }. A phase whose correct outcome is “there is nothing for the next phase to do” must not paint the run red, because failed has four mechanical consequences: messages.on_failure posts to the PR (actively wrong — “leaving it for a human” on a flaky test), the dashboard’s Retry button targets failed/cancelled and so offers a retry that cannot succeed, the cost and failure stats are polluted, and latestSucceededForTrigger ignores failed runs, which defeats the already-handled-this-SHA dedup so the same dead end is re-diagnosed on every webhook re-fire. Recording the skip succeeded fixes all four, and turns the fourth into a positive: the dedup starts working for exactly the cases that must not be re-attempted.

- name: fix
  skip_if:                                          # one string, or a list (OR-ed)
    - "scratch.fixMarkers.diagnosis.class == 'flaky'"
    - "scratch.fixMarkers.diagnosis.class == 'infra-dependent'"
    - "scratch.fixMarkers.diagnosis.class == 'upstream-broken'"

dependabot-ci-fix carries a second, simpler list on the phase above, reading a plain context value rather than a parsed marker:

- name: diagnose
  skip_if:
    - "reason == 'dirty'"
    - "reason == 'behind'"
    - "reason == 'blocked'"

There is nothing to diagnose when no check is failing — the PR is merge-BLOCKED, and the repair (merge the base in, regenerate the conflicted lockfile) is mechanical. Diagnosing anyway was worse than wasteful: the phase’s taxonomy is CI-failure-shaped, so an agent shown green checks and no failing job honestly answered infra-dependent, which is one of the three stopping rows above — the fix phase was skipped and the conflict left in place, on a succeeded run. A bare OR-ed list is sufficient despite the grammar having no negation, because reason carries CI’s verdict first (conclusion === "failing" ? "checks-failing" : mergeableState in src/cron/dependabot-discovery.ts), so these three values imply CI is not settled-failing; a PR that is both red and behind arrives as checks-failing and is still diagnosed. The fix rows then read a scratch path that is absent, and an absent variable fails open, so the fix phase runs.

Expressions use the until: grammar (see Loop expression evaluator below), evaluated against { ...ctx, phaseOutputs, scratch } — the same values a prompt template can render. output is empty here (the phase has not run), so a bare output.contains(...) is meaningless; read an upstream phase instead. AND is expressible by collapsing to a single expression; the production consumer is a class list, which is why one expression per class reads better than one compound one.

Read a PARSED value out of scratch, not prose out of phaseOutputs. The production rows above used to be phaseOutputs.diagnosis.contains('class=flaky'), and every way that can go wrong, it did: phaseOutputs.diagnosis is the agent’s entire output and the match is a plain substring, so an agent writing “this is not class=flaky, it is reproducible” skipped the phase; a {{priorAttempts}} line replayed from an earlier attempt matched too; class=flaky-timeout matched while class=probably-flaky did not; and phaseOutputs is empty across a resume boundary, so the guard failed open and ran a full sandbox on exactly the verdicts it exists to stop. scratch is reloaded from the run row each iteration, and the marker harvest has already parsed and validated the class — so the guard reads the same value the decision layer does.

There is no negation, and a conditional row — “skip on class=flaky, unless this PR has already deferred twice” — is therefore not expressible at all. The flaky cap needs exactly that, and resolves it without an operator: its second term is a fact about the PR’s history known before the run starts, so the harness composes the run’s guard list from it — promoteFlakyDiagnosis (src/workflows/simple.ts) drops the flaky expression once flakyDeferrals >= fix.maxFlakyDeferrals, on a shallow copy of the loader’s cached definition. The other two rows are unconditional. This is the general shape for any guard whose second term is run-level rather than phase-level: compose the list, don’t grow the grammar.

Two scoping rules:

• Evaluated when the node becomes ready, i.e. after its dependencies are terminal — which is what makes reading an upstream phase’s output well-defined.
• phaseOutputs is empty across a resume boundary (a phase skipped as already-done contributes nothing to the in-memory map), so a guard that must survive resume should read scratch, which the run store rehydrates.

Every expression fails open: an unrecognised form and an absent variable both evaluate false, so a malformed or not-yet-populated guard runs the phase rather than silently swallowing it.

Loops

Two flavours.

loop — reviewer/fix cycle

- name: reviewer
  prompt: prompts/reviewer.md
  loop:
    max_cycles: 2
    on_request_changes:
      fix_prompt: prompts/fix.md
      re_review_prompt: prompts/re-reviewer.md
      fix_model: "{{models.executor}}"
      fix_variant: "{{variants.fix}}"

Iteration naming — built by PhaseRef.format() and resolved by phaseIndexInDefinition (both in src/workflows/phase-ref.ts):

reviewer                    ← first review
reviewer_fix_1              ← fix cycle 1
reviewer_recheck_1          ← re-review after fix 1
reviewer_fix_2              ← fix cycle 2
reviewer_recheck_2          ← re-review after fix 2 (max_cycles)

n is the 1-based cycle; fix_k and recheck_k pair within a cycle. The legacy bare-numeric re-review form (reviewer_2) is dropped — neither produced nor recognized on resume.

The runner parses the verdict line — ^\s*VERDICT:\s*(APPROVED|REQUEST_CHANGES) — from the reviewer’s output via the single pure parser parseReviewerVerdict (src/workflows/verdict.ts) and either advances or enters the next fix cycle.

generic_loop — until-condition cycle

- name: socratic
  prompt: prompts/explore-ask.md
  generic_loop:
    max_iterations: 8            # or { from: <ctx path>, default: N } — see below
    until: "output.contains('READY')"
    gate_kind: "reply"           # pause after each iteration; user reply feeds next
    scratch_key: "socratic"      # accumulate Q&A under workflow_runs.scratch.socratic
    fresh_context: false         # pass {{previousOutput}} to next iteration
    interactive: true
    on_soft_failure:             # optional; absent = hard-fail on any non-success
      retries: 1                 # re-run a soft (empty) iteration up to N times
      then: complete             # then: fail (default) | complete

Iteration naming: ${phaseName}_iter_${n}; a soft-failure retry is ${phaseName}_iter_${n}_retry (its own ledger row) and the until_bash exit check is ${phaseName}_iter_${n}_check (its own ledger row too — see “Recording a loop iteration” below). The until-condition is evaluated by loop-eval.ts — see below.

Recording a loop iteration. The iteration is persisted the moment its work finishes — phase_history entry (iteration N — work complete) plus the iteration’s output_text — before the exit condition is evaluated. Everything after the agent turn belongs to the loop, not the iteration. until_bash then opens its own executions row (recordStart → command → recordFinish + recordOutputText), because it is a real sandbox command that can run for minutes and an open row with a start time is how every renderer draws “in flight, since N”. The row deliberately bypasses runPhaseLedger/shouldRunPhase — a condition must be re-evaluated on every ask, never replayed from a dedup hit — and its success records whether the check ran, with the verdict in stop_reason as condition_met / condition_not_met. A red gate is the loop working as designed (it is what earns the agent another iteration), so it must not read as a failure: the dashboard gives condition_not_met a muted unmet tone and the CLI a ↻. When the condition is met a second entry (iteration N — condition met) is appended for the same label; readers fold a repeated label last-wins.

Without this, prod run 49c101aa spent 6m48s of a 20m31s run inside until_bash with nothing recording it, while the run advertised currentPhase: "diagnose" — a phase that had ended 12 minutes earlier — and omitted the completed fix_iter_1 from phase_history entirely.

on_soft_failure — by default any non-success iteration hard-fails the whole run, which is wrong for a long interactive loop (one degenerate turn would discard all accumulated state). A soft outcome is a clean exit that produced no usable output — mapStopReason returns "unknown" / "error_truncated" — as opposed to a hard crash (terminated / error_fatal / error_tool / error_exit_*); the split is the generic isSoftOutcome(result) classifier, shared with the reviewer loop’s fallback recovery. When declared, a soft iteration re-runs up to retries times; if still soft, then: complete ends the loop as if until matched (advancing downstream with the work so far, recorded as success so the run’s anyFailed rollup stays green) while then: fail keeps the hard-fail. Only explore.yaml’s socratic phase opts in.

Approval gates and reply gates

Both are pause points. The difference is who resumes them.

Approval gate: phase declares approval_gate: post_architect. If config.approval["post_architect"] === true (Configuration), the runner persists the phase, writes a pending row to workflow_approvals, sets workflow_runs.status = "paused", and returns { paused: true }. Resume comes from a GitHub comment (@last-light approve), a Slack slash command, or the dashboard — the Router routes those to skill: approval-response, which calls back into runSimpleWorkflow().

If the gate name is not in APPROVAL_GATES, the phase proceeds without pausing. Gates are positive enable only.

One dispatch path raises gates the operator left off. A run dispatched through an autonomy stage carries that stage’s gates map on its context, and runWorkflow unions it onto the effective approval map — but true leaves only (enabledGatesFrom, src/workflows/simple.ts). The filter is the whole safety argument, and a naive spread is not add-only: gateEnabled tests === true, so a true leaf can raise a gate the operator left off, while a false leaf spread over an operator’s true would clear one. Dropping the false is the clamp — the base map underneath still carries the operator’s value, so the leaf resolves straight back to it, the same shape the repo layer’s policy blocks use. Note what this means for reading gateEnabled’s === true: it guards a missing gate, not an overwritten one, so the clamp has to happen before the union rather than being caught by it. With no stage gates the caller’s own map is passed by identity, so a run that crossed no stage behaves exactly as it did before the mechanism existed. The visible effect is that an autonomous build pauses at post_architect while an @bot build on the same repo does not: same workflow, same repo, different dispatch. See Router and Configuration.

Approving an artifact: a gate can name the handoff doc it’s asking a human to approve via approval_artifact: architect-plan.md (also valid inside loop:). The filename is stored on the workflow_approvals row (State). The gate message can deep-link a focused approval view with {{approvalUrl}} — pauseForApproval injects the new approvalId into the message context and the helper renders ${publicUrl}/admin/?approval=<id> (empty without PUBLIC_URL, so the message still posts; identical for GitHub- and Slack-initiated runs). That view (GET /admin/api/approvals/:id) enriches the approval with an artifactRef derived from the run (context.owner + bare repo + buildAssetIssueKey): server storage mode embeds the artifact editor (edit + save the store doc, then approve); repo mode links out to the doc on GitHub. Both resolve through the same dashboard approve/reject path.

Reply gate: declared as gate_kind: "reply" on a generic_loop. The phase pauses, the next free-form maintainer message on the same issue or Slack thread becomes the next iteration’s input. No @last-light mention required — the router’s reply-gate short-circuit (see Router) feeds it in as a skill: explore-reply. The harness merges the reply into scratch[scratch_key] and re-enters the same phase for iteration n+1.

Template engine — data flow

Full template syntax lives in Phases & Prompts. Here, just the data flow:

TemplateContext = {
  // run-scoped (built once in simple.ts:248–279)
  owner, repo, issueNumber, prNumber, branch, taskId, issueDir,
  issueTitle, issueBody, issueLabels, commentBody, sender,
  bootstrapLabel, contextSnapshot, models, variants,
  ...request.extra,

  // phase-scoped (merged per phase in runner.ts)
  phaseOutputs,    // { [phaseName | output_var]: string | object }
  fixCycle,        // loop only
  iteration,       // generic_loop only
  previousOutput,  // generic_loop with fresh_context: false
  scratch,         // mutable from workflow_runs.scratch
}

Phase A’s output reaches Phase B by being stored in phaseOutputs[A] (in memory during a linear run). Phase B’s prompt template reads it with ${A.output} or {{A.field}}. DAG runs scope outputs per-phase taskId to keep concurrent workspaces clean.

Scratch state

workflow_runs.scratch is the only mutable JSON we keep on a run. What lives there:

• Loop accumulators — scratch.socratic.iteration, scratch.socratic.qa, etc.
• Pointers to large outputs — scratch.<key>.lastOutputExecutionId points at an executions row whose output_text holds the actual text. Inlining 50 KB of LLM output into the scratch JSON every iteration would balloon the state database for no good reason.
• Free-form workflow state — reply-gate-merged user responses, intermediate flags.

Mutations go through db.updateWorkflowRunScratch(workflowId, patch).

taskId scoping

// src/workflows/simple.ts:64–74
function workflowScopedTaskId(repo, number, workflowName, workflowId) {
  const suffix = workflowId.slice(0, 8);
  return number !== undefined
    ? `${repo}-${number}-${workflowName}-${suffix}`
    : `${repo}-${workflowName}-${suffix}`;
}

• Every phase uses this base, unmodified. The sandbox workspace persists across the whole run, so files like .lastlight/issue-42/architect-plan.md survive between phases. Nothing appends a phase name: the per-phase ${taskId}-${phaseName} clones went with issue #94’s scheduler unification, and re-introducing them would break every handoff that reads an earlier phase’s output off disk.
• Loop iterations — reuse the parent phase’s taskId. Fix cycles read the reviewer’s verdict from the same disk.
• Fan-out branches — likewise. Every branch of a type: fanout phase runs against the one provisioned workspace; isolation between them is by disjoint output paths and a per-branch skill bundle, not by separate checkouts.
• Resume — stored in workflow_runs.context.taskId. A resumed run lands in the exact same sandbox directory.

Idempotency

// runner.ts:210–225
const dedupKey = `${workflowName}:${phaseName}`;
const status = db.shouldRunPhase(dedupKey, triggerId, workflowRunId);
if (status === "running") {
  // verify the container is still alive; if not, mark stale
}
if (status === "done") {
  return { skipped: true, reason: "done" };
}

Completed phases are never re-run on resume. In-flight phases are checked for liveness — if a sandbox container disappeared while the process was down, db.markStaleAsFailed() flips the row and the runner re-enters the phase. Worst case: a phase runs twice; the prompts are written to tolerate that.

Resume protocol

Two distinct entry points.

resumeOrphanedWorkflows() (resume.ts:276–315) — called at Harness boot. Scans workflow_runs for rows with status running (paused is left alone — those are awaiting humans). First it repairs gate-stranded runs: a run reading succeeded while its currentPhase is still waiting_approval and its approval is still pending is put back to paused (restorePaused), so answering the approval resumes it. For each running run:

1. Increment restart_count. If > 3 (MAX_RESTART_RESUMES), mark the run failed and skip. This is the crash-loop circuit breaker.
2. Mark stale execution rows failed.
3. Call resumeSimpleRun() in the background (non-blocking). It rebuilds the template context from the row: resume’s own fields (bare repo, taskId, branch, refreshed issue, effective models) over the dispatch’s persisted context (restoredDispatchContext), so a resumed or admitted pr-review keeps analysisEnabled and its PR snapshot.

Approval / reply gate resume — simple.ts:317–397 handles inbound approval responses. Fetches the workflow_approvals row, updates its status, flips the run back to running, calls runSimpleWorkflow() again. The runner’s nextPhaseAfter(definition, run.currentPhase) walks the phase array to the position after the last completed phase and starts there — completed phases are skipped via shouldRunPhase() === "done".

For reply gates the runner sets currentPhase to the phase before the loop owner so nextPhaseAfter() lands back on the looping phase for the next iteration.

Terminal transitions — the run store’s observers

A run’s terminal transition (succeeded / failed / cancelled) is a persisted fact, and anything that projects it outward hangs off that fact rather than off the in-memory promise that produced it. WorkflowRunStore exposes addTerminalObserver, notified wherever a run is finished — simple.ts, resume.ts, the queued-run TTL expiry and the admin cancel — so a projection installed once covers all of them and a ninth terminal path cannot be added without one. The contract is strict and stated on the store: an observer is synchronous, never throws, and never re-enters the store; any real work is fired and forgotten with a .catch attached, which is both where the rejection goes and what satisfies the floating-promise lint gate.

Two observers are installed at boot. The first completes the last-light/review check (see Router); the second is the terminal half of the software-factory stage pipeline (src/engine/stage-observer.ts).

The stage observer is the second and final call site of advanceStage. The first is the dispatch gate, which moves the issue onto the stage’s running label before the run exists; until this one landed, nothing moved it off again, and agent-building was a dead end — every issue the pipeline ever picked up sat there while the run that owned it had long since finished. On a terminal transition it maps:

paused is structurally unreachable here, and that is load-bearing rather than incidental. TerminalRunObserver is typed to receive only the three statuses above, so a run sitting on an approval gate moves no label at all. That is the correct behaviour: an HITL pause means the work is still in flight while a human decides, so advancing the label there would report a build as finished mid-run — and then move it a second time when the resumed run actually finished. The type is what prevents it; every status is named explicitly rather than swept up by an else, so a fourth terminal status becomes a compile error instead of a silent mis-labelling.

It is idempotent with no bookkeeping. The observer can fire more than once for one run — a restart-resumed run that finishes twice, a retry, a cancel racing a finish — and needs no dedup state, because advanceStage is naturally idempotent at both ends: re-adding a label the issue already carries is a no-op for addLabels, and removeLabel swallows the 404 that means “already gone”. A second firing re-asserts the same two facts. Recording an “already advanced” flag would buy nothing and add a write that could itself fail or go stale; the durable fact is the run row, and the label is its projection.

Three things make it a no-op rather than an error: a run whose stored context names no _stage (an @last-light build comment carries none, and a human-asked build must never acquire stage labels it did not enter the pipeline through, so the check is a by-name read of the JSON rather than a mirrored type); a _stage naming a stage an operator has since renamed or removed (debug, not warn — there is no label vocabulary left to move within); and chat-only mode, where there is no GitHub to project onto. A degraded advance — the new label on but the old one still there, or neither — is logged as a warning, because the issue’s stage now misreports to the human reading the tracker, but it never fails the run: that fact is already in the database.

Concurrency cap and admission

A global cap bounds how many sandboxed runs execute at once, so a burst of triggers can’t swamp the host. It is enforced at the single dispatch funnel (runSimpleWorkflow) and defaults to concurrency.maxWorkflows = 4 (env MAX_CONCURRENT_WORKFLOWS; see Configuration).

Enqueue. When a fresh trigger arrives and countRunning() >= maxWorkflows, the run row is created with status queued instead of running, runSimpleWorkflow posts an enqueue ack (“…is queued — it’ll start automatically when a slot frees”), and returns { success: true, queued: true, phases: [] } — no phases run. A duplicate trigger on an already-queued run is a no-op (returns the same queued result; it must not fall through to runWorkflow, which would execute outside the cap). Resumes and orphan restarts bypass the cap: they re-enter runWorkflow directly, finishing in-flight work rather than re-queuing behind it.

The enqueue ack is transient. It is a promise about the future, so it must not outlive the run’s stay in the queue. RunnerCallbacks.postComment resolves to the created GitHub comment id (void on Slack, or when the post failed), and simple.ts stashes it at scratch.queuedAck.commentId. Whichever way the run then leaves the queue, the admission controller resolves that comment — see below. Without this the ack is the only visible trace of a run that starts and then legitimately no-ops (e.g. a re-triage of an already-triaged issue), which reads as “it queued and never came back”.

Admission. createAdmissionController (src/workflows/admission.ts) promotes queued runs to running as slots free, reusing resumeSimpleRun (a queued run’s stored context is shaped exactly like a resume’s, and no phase has run yet, so the ledger runs them all). A promoted run that stops at an approval gate stays paused — the runner reports a gate stop as success: true, paused: true, so resumeSimpleRun checks paused before success, and finishRun itself refuses to flip a paused row to succeeded. Promotion is FIFO by started_at and guarded by a compare-and-set (admitRun: WHERE status = 'queued'), so its two triggers race safely:

1. Event-driven — admitNext() runs in dispatchWorkflow’s finally block, so a just-finished run immediately pulls the next queued one in.
2. Periodic sweep — a setInterval(sweep, 15s) also TTL-expires queued runs older than concurrency.maxQueueWaitMs (default 1 hr; env MAX_QUEUE_WAIT_MS) — transitioning them to cancelled with a “dropped from queue after waiting too long” reason in context.error — before admitting. The sweeper starts after boot’s orphan recovery, so a run that was queued when the harness crashed is picked up on the first tick.

Both exits resolve the enqueue ack, so it never lingers as a stale promise:

• Admitted → retractQueuedAck deletes the ack comment. It has been honoured, and whatever the run itself posts is now the real answer.
• TTL-expired → postExpiryAck rewrites the ack in place to the drop notice. Editing adds nothing to the thread and GitHub does not notify watchers on edits, so this does not reintroduce the comment flood that made expiry silent on GitHub in the first place. A Slack-originated run instead gets a normal thread reply (a human explicitly asked for the work); a GitHub run that left no ack stays silent, its drop visible only in the dashboard and lastlight workflow list.

The queued?: boolean on WorkflowResult propagates up through dispatchWorkflow and the dispatcher: queued runs suppress the spurious “completed” reply, leave an in-progress PR check untouched (the terminal review comment lands once admission runs the workflow), and are cancellable like any live run. The dashboard shows a queued run with a neutral status badge and includes it in the active filter alongside running/paused.

k8s backpressure requeue. The Kubernetes sandbox backend has no tuned concurrency cap of its own — the cluster namespace’s ResourceQuota is the authority, so the harness admits freely (gated only by a sanity-fuse K8S_SANITY_FUSE = 1000, not maxWorkflows) and finds out capacity is exhausted only when a phase’s Pod create is rejected. runWorkflow surfaces that as backpressure: true on its result (a server-layer intersection on WorkflowResult — the engine type itself is unchanged). simple.ts reacts by calling db.runs.requeueRunning(), transitioning the run running → queued instead of failed — a third transition alongside the fresh-trigger → queued enqueue and the approval-gate → paused pause. The AdmissionController then promotes it again like any other queued run, in a backpressure mode (backpressureMode: config.sandbox === "kubernetes") that gates on K8S_SANITY_FUSE and promotes one queued run per admitNext() call instead of filling up to maxWorkflows — each promotion doubles as a quota probe. See Sandbox → Concurrency for the k8s-side half of this mechanism (the QuotaExceededError → stopReason: "error_quota" mapping).

Loop expression evaluator

// src/workflows/loop-eval.ts
export function evalUntilExpression(expr: string, ctx: LoopEvalContext): boolean;
export function evalSkipIf(exprs: readonly string[], ctx: LoopEvalContext): string | undefined;

A custom mini-DSL (not eval()). Accepts:

• output.contains('text') — substring match on the iteration’s output
• a.b.c.contains('text') — the same against any dotted path in the context (output is the degenerate one-segment case). Strings and numbers only: stringifying an object yields "[object Object]", which is a substring match waiting to surprise someone
• a.b.c.startsWith('text') — an anchored match (leading whitespace ignored), for a marker a tool prints as its first line; contains cannot tell it from the same text quoted later in the output
• variable == 'value' / variable != 'value' — equality / inequality
• variable == true / == false — boolean coercion of bare literals
• Dotted keys for nested access: scratch.socratic.ready == true

Unrecognised expressions return false (safe default — the loop runs until max_iterations; a skip_if guard runs its phase).

One grammar, two consumers. evalUntilExpression drives a generic_loop’s until:; evalSkipIf OR-s a phase’s skip_if list and returns the first matching expression, so the scheduler can name it in the skip reason. The longer dotted path exists for skip_if, which needs to read a sibling value (scratch.fixMarkers.diagnosis.class == '…') that the loop never did.

The cancel check is that getRun at the top of every scheduler iteration: a cancelled row stops the run at the next phase boundary. It holds because a cancel is final — finishRun refuses to move a cancelled row, so the failWorkflow a killed phase triggers cannot overwrite it with failed.

runScope.scratch is refreshed from the run row on each iteration, inside the getRun the cancel check already makes — so a guard reading scratch sees what a phase harvest wrote through onPhaseEnd, not the value the scope was constructed with. Without that refresh a scratch guard is a silent no-op on the fresh path: it reads state that predates the phase it is guarding on.

until_bash is the alternative: a shell command whose exit code (0 → stop) drives the loop. It runs inside the sandbox (via executeCommand with writeSession: false) against the persisted workspace — not on the harness host. {{}} markers in the command are rejected before execution to prevent template-after-render injection (validateShellCommand), so the command is necessarily a literal string — it cannot be varied per backend.

Its budget is phase.timeout_seconds, else sandbox.untilBashTimeoutSeconds (30 in config/default.yaml — there is no code default). Thirty seconds is a trap: it kills any real build/test suite mid-run and reports a false red, so a phase whose gate is the repo’s own CI commands must carry an explicit value. Both fix workflows read theirs from gate.timeoutSeconds — see “Templated phase budgets” below.

The two are not exclusive, and the order between them is load-bearing: until is evaluated first, and a match short-circuits until_bash entirely (phase-executor.ts: if (!conditionMet && loop.until_bash)). A loop can therefore declare both — a cheap expression that names the cases where there is nothing left to check, and the expensive command for everything else.

The fix family’s push short-circuit. Both pr-fix and dependabot-ci-fix use exactly that pairing:

generic_loop:
  max_iterations: { from: fix.localIterations, default: 2 }
  until: "output.contains('outcome=pushed tried=')"
  until_bash: "if [ -f .git/lastlight-verify.sh ]; then bash …; else … exit 1; fi"

outcome=pushed in a CI_FIX_COMPLETE marker means the commit is already on the branch — put there by github_publish, not by a git push the agent ran (see Sandbox → “Invariant: the published commit is built by GitHub, not by git”); for this gate the two are the same fact, and the marker name is unchanged because the harness parses it.

GitHub’s checks started against that commit the moment it landed and are the strictly better authority — the real CI environment rather than a sandbox approximation of it, warm rather than a cold container, and covering the matrix legs the sandbox cannot reproduce. Re-running the local gate at that point cannot change anything: its exit code decides only whether to spend another agent iteration, and a pushed fix has nothing left to iterate on. Without the short-circuit a real run (49c101aa) pushed at 11:03:31, saw GitHub go fully green at 11:06:25, and still sat in a fresh container from 11:04:00 to 11:10:48 running the same suite a third time.

It fires on pushed only. no-change / gave-up pushed nothing, so there is no new commit, no new check run and no external authority — the local gate is the only evidence that exists, and its red verdict is precisely what earns the agent the next iteration. Short-circuiting all three outcomes would end every loop at iteration 1 and make fix.localIterations dead config.

The needle carries tried= because outcome=pushed alone also appears in a rendered {{priorAttempts}} journal line from an earlier attempt (renderAttemptLine emits … | outcome=pushed gate=green), which the prompt replays into the run — the same replayed-line false match that forced the fix phase’s skip_if rows off phaseOutputs. scratch is not an option inside a loop (it is refreshed per phase node, not per iteration, so it is stale there), but renderAttemptLine deliberately never renders tried=, so outcome=pushed tried= matches a live marker and nothing else. A marker that reorders its fields simply fails to match and the gate runs as before — the fallback is the previous behaviour, which is the right direction for a cost optimisation to fail in.

What this gives up is stated plainly rather than left implicit: after a push the harness no longer has an independent check on the agent’s self-reported gate=green. That is not a reintroduction of the sh-versus-bash defect (see Sandbox → the push gate), because the gate in this flow runs after the push and therefore never gated it — the agent’s self-report was already the only thing between a bad fix and the branch. What catches a bad fix is unchanged: GitHub’s checks go red, pr.checks_failed re-dispatches the fix family, and fix.maxAttempts / fix.maxCostUsd bound the retries. Naming bash still matters for the unpushed path, which is where the loop’s iterations live.

Templated phase budgets

timeout_seconds and generic_loop.max_iterations accept either a plain positive integer or a reference into the run’s template context:

timeout_seconds: { from: gate.timeoutSeconds }
generic_loop:
  max_iterations: { from: fix.localIterations }

from is the same dotted lookup {{a.b}} performs (lookupContextKey), so fix.localIterations resolves against the EFFECTIVE, already repo-clamped fix block the runner seeds on every run’s context — a repository that lowered its own budget in .lastlight/lastlight.yml is honoured with no code path of its own. Resolution happens once, before the first iteration, in resolveTemplatedNumber (core/templated-number.ts).

There is no default (issue #385): every budget lives in config/default.yaml, so a from that resolves to nothing usable — key absent, non-numeric, zero or negative — fails the workflow load with an error naming the phase and the path, rather than falling back to a literal buried in YAML. The shape is still an object rather than a bare "{{gate.timeoutSeconds}}" because an unresolved template renders to the empty string, which would leave the engine inventing a kill timeout. A resolved non-integer is rounded UP (truncating a suite’s budget downward is the direction that turns a passing gate red).

The runner seeds the resolved timeouts on every run’s context: gate.timeoutSeconds / gate.maxTimeoutSeconds / gate.phaseTimeoutSeconds (after the repo clamp) and timeouts.agentSeconds / timeouts.commandSeconds / timeouts.untilBashSeconds. A phase with no timeout_seconds takes the matching sandbox.*TimeoutSeconds value.

Before this, both keys were parsed, per-repo clamped, CLI-displayed and read by nothing: the operative numbers were literals in the YAML, whose comments asked a human to keep the two in step (issue #256).

A loop node honours the phase’s on_output.requires_marker and its messages.on_start / on_success. The postcondition is checked once, against the last iteration’s output (the turn that reports the outcome; an earlier one is by definition mid-loop), and its absence fails the phase exactly as it does for a non-loop phase. Reaching max_iterations without the condition is not a failure — a fix loop that runs out of iterations reports outcome=gave-up, which is a correct outcome — only the absent sign-off is. Two paths are exempt because they produce no fresh turn to sign off: a deduplicated (already-completed) phase on resume, and the on_soft_failure: complete advance.

Command policy

An agent phase (and a fan-out branch) may declare which classes of bash command its agent may run (issues #403, #404):

command_policy:
  install: block                                   # allow | log | block
  install-scratch: { from: probeScratchInstallPolicy }
  test: { from: probeTestPolicy }
  host: log
  reason: "…"                                      # model-facing text for a blocked call

• Classes. install — a package-manager install in the checkout; install-scratch — an install whose directory is outside it (cd /tmp/probe && npm i x, --prefix, -g), falling back to install’s mode when unset; test — test runners and the project’s test/lint/typecheck scripts; host — a command reaching outside the workspace: find/ls/du/tree/ grep -r/rg rooted at /, ~ or an absolute path outside the workspace root, a read of a known package location (~/.nvm, ~/.npm, ~/.cache, ~/.yarn, ~/.pnpm-store, a global node_modules, /opt), PATH= / NODE_PATH= naming an outside directory, or node -e requiring an outside absolute path. The workspace root is the cwd’s parent (so the sibling skill bundle and AGENTS.md stay reachable) — the guest mount under gondolin. Scratch dirs (/tmp, /var/folders) and /opt/lastlight are never host. An absent class is allow. The schema is .strict(): an unknown class or mode is a load error.
• Modes. allow runs it. log runs it and emits a command_policy event. block refuses it — the model gets reason as the tool result and carries on — and emits the same event. Each event becomes a structured command policy log line (with the phase), a rendered system line in the phase’s session transcript, and command_policy.* span attributes.
• Templated modes. { from: <ctx key>, default?: <mode> }, resolved by phaseConfigFor against the run context. A present value that is not a mode throws (default or not); an absent key uses default, or throws without one.
• Fan-out. A branch’s command_policy replaces the phase’s whole — no per-class merge, like model and skills.
• Enforcement is agentic-pi’s (a pi tool_call handler over a pattern table), on every sandbox backend — see Sandbox. A pattern guard, not a security boundary: sh -c "$(…)" or a script under another name gets past it.

pr-review.yaml sets site-review and select to install: block, test: block, host: block (site-review’s install-scratch follows the probe mode, block when probes are off), and review and falsify set host to block and log respectively (falsify moves to block once an eval arm’s log shows nothing legitimate is caught). review blocks test always and install only when review.analysis.enabled (context key reviewInstallPolicy; with the pipeline off the pr-review skill’s install-to-probe affordance stands). falsify blocks install in every mode and reads test / install-scratch from probeTestPolicy / probeScratchInstallPolicy: log under probes: full (prepare installed the tree), block under probes: static.

Invariants

• The runner is workflow-agnostic. It learns about a workflow by loading YAML; it has no per-workflow branches. Any change to “what happens” is a YAML change, not a code change.
• The engine never chains one workflow into another. A workflow run ends; it does not start a successor. The software-factory pipeline’s stage chaining (ready-for-agent → agent-building → ready-for-human / agent-blocked) is router-mediated, not an engine feature: the harness writes a label, GitHub delivers an issues.labeled webhook, and the router decides — independently, from scratch — whether that label enters a stage. There is no on_complete key, no successor field, and nothing in the YAML schema that names another workflow. A re-implementation that reads the pipeline as evidence the engine chains runs will build a scheduler the harness does not have, and will lose the property that makes the chain safe: every hop crosses the router and the dispatch gate again.
• Completed phases never re-run. shouldRunPhase() is checked at the top of every phase entry; resume relies on it.
• Idempotency is per-(workflow_run_id, phase_name). Not per-phase globally. Two runs of the same workflow on the same issue are independent.
• output_var aliases are unprotected. Two phases writing to overlapping output_vars will clobber each other silently. Convention: use distinct, descriptive aliases.
• One workspace per run — phases, loop iterations and fan-out branches all share ctx.taskId. Every inter-phase handoff reads the previous phase’s output off that one disk, so a per-phase clone would break them all.
• DAG nodes run one at a time. The only concurrency in the engine is within a type: fanout node. If you need two phases to overlap, that is the parked worktree project, not a scheduler tweak.
• Scratch points at outputs; doesn’t inline them. A phase output of any size lives in executions.output_text. Scratch stores the row id.
• Approval gates are positive enable. A gate name not in APPROVAL_GATES is silently disabled — the phase proceeds. There is no enable-all shortcut.
• The verdict marker is exact. ^\s*VERDICT:\s*(APPROVED|REQUEST_CHANGES) on the first matching line of reviewer output. Variant phrasing (“looks good”, “approved!”) is not recognised; reviewer prompts are written to produce the literal marker.
• Restart-count is the circuit breaker. Three failed resumes and the run is failed permanently. Resist the urge to raise the limit without thinking about what’s actually crashing.
• A gated skip keeps the run green. requires_sandbox, sandbox_image and skip_if all record the node skipped and leave the run succeeded. failed is reserved for malfunction — a correct “there is nothing to do here” is not one, and recording it as one mis-fires four downstream mechanisms (see Gated skips).
• A skipped node is not succeeded. So an all_success trigger_rule downstream of a gated phase will not fire. That caveat is identical for all three gates; a graph that must proceed past one needs all_done, none_failed (proceed unless a dep genuinely failed, even when every dep skipped) or none_failed_min_one_success (the same, but at least one dep must have succeeded).
• A run’s config is frozen at dispatch. runWorkflow takes the target repo’s .lastlight/ layer as a trailing, defaulted parameter (defaulted so runWorkflow.length stays 9 — the frozen lastlight/evals surface pinned by evals-contract.test.ts) and derives the effective models / variants / approval gates from it. A resume restores that layer from the run row rather than re-resolving it from the repo’s default branch, so an edit made while the run was paused, queued or dead can’t retarget it half-way through. See Configuration and State.
• Per-run asset resolution never mutates the module globals. When a repo layer applies, the runner builds an AssetResolver over getAssetLayers() + makeLayer("repo", …) and passes that through EnginePorts.assets (loadPromptTemplate / resolveSkillPaths) for the whole run — not configureWorkflowAssets, which would leak one run’s repo layer into another’s prompts. populateCache() additionally refuses to read workflow or cron YAML from a repo layer, so “a repo may not define workflows” is structural rather than conventional.

Current implementation

Rebuild notes

• Schema first, executor second. Defining the YAML schema cleanly (in TypeScript, Go, whatever) is what makes the runner’s behaviour predictable. Don’t let optional fields with implicit defaults proliferate — every default is a future surprise.
• Linear is the default; DAG is the special case. Most workflows don’t need a graph. Adding depends_on to a phase should be a deliberate choice that opts in to the concurrency cost.
• Persist resume state, not in-flight state. The runner stores where it is (current phase, scratch, restart count) — not the conversation buffer or the sandbox process. Those are reconstructed from disk + DB on resume.
• A single state store, not two. Both the relational DB (resume substrate) and the JSONL event logs (event stream) are needed — see State — but the runner only talks to the resume store. Mixing them creates ordering bugs.
• Approval gates as data, not code. Whether a gate fires depends on configuration, not on whether the phase exists. A re-implementation that hard-codes which gates are enabled is denying operators a knob they need.
• Restart-count circuit breaker is non-optional. Crash loops are a certainty; if the runner re-enters an OOMing phase forever, it will eventually deplete the database with stale executions and consume every cent of the LLM bill. Pick a number, default to it, surface the count in the dashboard.
• The verdict marker is an interface. It’s the contract between prompts and code — both sides know exactly what to produce and look for. Other parse markers (READY, BLOCKED) follow the same pattern. Make them exact.