Feedback signals

Last Light writes a lot: triage summaries, PR review findings, fix reports, answers in Slack. Feedback signals let a human grade any of it with a single emoji reaction. The score is recorded against the specific workflow run that produced the output, so you can watch what a change to a prompt or a skill actually did to quality — instead of guessing.

It is analytical only. Nothing about the agent's behaviour changes when you thumb something down; the signal is data for you, not an instruction for it.

What the reactions mean

👀 is recorded but scores zero, and is excluded from every average. That is deliberate: 👀 is Last Light's own "I've seen it" ack emoji, so people copy the idiom back onto its comments to mean "noted". Counting that as criticism would quietly poison the dataset.

Anything outside the table is ignored rather than stored — a 🍕 is not feedback, and recording it would only dilute the numbers it sat next to. Reactions from the bot itself never count.

Slack works out of the box

Slack sends a real reaction_added event, so a Slack signal is recorded within a second of the click, with no polling and no API budget. It is on by default.

You do need to grant one scope. Add the reactions:read bot scope and subscribe to reaction_added and reaction_removed (both are in the reference manifest at deploy/slack/slack-manifest.json), then re-consent the app — Slack does not grant a new scope to an existing install. Until you do, no reaction event is delivered and the feature is simply dormant; nothing else is affected. In channels the bot must be a member to receive reactions; DMs are already covered.

GitHub has to be polled, and is off by default

GitHub delivers no webhook for emoji reactions — there is no reaction event, and none of the issue_comment actions cover one. So the GitHub half has to ask, on a schedule, and it ships off:

feedback:
  enabled: true              # Slack signals, live
  github: true               # opt into the GitHub poller
  pollSchedule: "*/30 * * * *"
  windowDays: 14             # how long a comment stays pollable
  maxAnchorsPerTick: 500     # = 5 GraphQL requests per tick
  retentionDays: 90
  otel: true

What keeps that affordable is that the cost is a property of the data, not of the schedule. Last Light polls individual bot comments — never issues, never repositories. Each is discovered once, when the run that wrote it finishes, and retires from the rotation after windowDays (reactions arrive within hours; nobody thumbs a two-month-old review). Reactions are then read in batches of 100 through a single GraphQL query that costs one rate-limit point and returns who reacted. A steady state of a few hundred live comments is single-digit points per tick, against a 5,000/hour budget — maxAnchorsPerTick / 100 is the exact request bound.

One thing GitHub makes impossible: a pull-request review body has no reactions endpoint and no picker. Only its inline comments can be reacted to — which is the better granularity anyway, since a 👎 then names one finding rather than a whole review.

Where the signals go

The Feedback tab in the admin dashboard charts the score over time, ranks workflows, and lists the raw signals; each run's detail panel carries a badge with what people said about it. The same data is on the admin API at /admin/api/feedback/signals, /feedback/summary, /feedback/daily and /admin/api/workflow-runs/:id/feedback.

And onto the trace, if you run OpenTelemetry

With telemetry enabled, each signal is also exported as a lastlight.feedback.signal span — attached to the original run's trace, as a late child of that run's span. So in Arize Phoenix or Langfuse the score appears on the very trace it grades, next to the prompts, tokens and cost that produced it, rather than in a disconnected trace of its own. Last Light remembers each run's trace and span ids for exactly this reason.

The span carries OpenInference's EVALUATOR kind (which Phoenix reads today) alongside langfuse.score.user_feedback. One caveat worth stating plainly: Langfuse does not yet map langfuse.score.* to first-class Scores on its OTLP ingest path, so for now those attributes ride along on a correctly-placed span rather than becoming a Score object. When Langfuse ships that mapping it will start working with no change here. Metrics (lastlight.feedback.signals, lastlight.feedback.score) are emitted either way.

The reactor's identity is treated as content: it is only attached to a span when LASTLIGHT_OTEL_INCLUDE_CONTENT=true, like every other content value.

Signals recorded while telemetry was off are not marked exported, so switching OpenTelemetry on later backfills them onto the traces they belong to at the next restart — rather than starting your backend from zero.

Removing a reaction

Take a reaction away and the signal is retracted, not deleted — it stops counting toward every score immediately, but the row stays. "Somebody thumbed this and then thought better of it" is itself worth knowing.