cc-os/docs/adr-system/06-eval-scenarios.md

# Eval Scenario Sketches (held-out, deferred stage)

_Last updated: 2026-07-03 — these shapes are now built out as the Eval B harness at
`plugins/os-adr/eval-b/` (fixtures, prompts, headless runner, two-axis checker; OpenSpec change
`add-os-adr-eval-b-harness`). The shapes below stay frozen — the harness executes them, it does
not renegotiate them. Running the grid remains a separate later stage (locked sequencing in
`04-plugin-requirements.md`)._

> **Two evals, don't conflate them (ADR-021).** This file sketches **Eval B** — held-out,
> unprompted-behavior (requirements 4–5). A separate **Eval A** — prompted skill-execution
> across model tiers (can haiku/sonnet correctly *execute* an explicitly invoked skill?) —
> is built and lives at `plugins/os-adr/eval/` with its own README; its prompts are *not*
> held-out and may be iterated freely. When Eval B eventually runs, it runs as
> scenario × model tier (haiku/sonnet/session-model), pass rate per tier as the
> autoresearch metric.

These sketch the *shape* of the held-out scenarios that will eventually measure requirements 4
and 5 (unprompted write-trigger recognition, unprompted correct retrieval). They exist now, before
the plugin skeleton freezes, for one purpose: to catch any plugin surface the eval will need that
Phases 1–4 don't provide. They are **not** to be run, automated, or informally "tried out" during
the build — that would contaminate the held-out methodology.

## Ground rules for all scenarios

- **Held-out**: scenario prompts never name the ADR system, the plugin, an ADR ID/title, or the
  exact constraint text. Relevance must be discovered, not pointed at.
- **Scored on two axes**: (a) did the agent consult/propose the ADR system *at all* unprompted;
  (b) did it act on the *specific correct* ADR (retrieval) or propose a *well-scoped* new one
  (write-trigger) — not merely a plausible-looking neighbor.
- **Environment**: a real onboarded project (post-pilot) with a populated `docs/adr/`, the
  SessionStart hook active, and — for graph-layer variants — a built `graphify-out/`.
- The eval stage will iterate with the `autoresearch` skill (modify → verify → keep/discard)
  against these scenarios once written out in full.

## Write-trigger scenario shapes

**W1 — Irreversible choice mid-task.** Task: "Add persistence to X" in a project where the
storage engine choice is genuinely open and consequential (hard to reverse once data exists).
Never mention ADRs. Pass: the agent proposes recording the engine decision via `/os-adr:create`
(or asks whether to) before or immediately after implementing. Fail: silently picks one and moves on.

**W2 — Cross-cutting convention change.** Task: a refactor request whose natural solution
changes a convention used across many files (e.g. error-handling shape, module layout). Pass:
the agent recognizes the decision outlives the task and proposes an ADR. Fail: does the refactor
with no record.

**W3 — Contested reversal.** Task: user asks for something that quietly reverses an earlier
(non-ADR'd, but discoverable-in-code) approach. Pass: agent flags that this is a
decision-of-record moment and proposes capturing it. This shape overlaps retrieval when the
earlier approach *is* ADR'd — kept separate deliberately.

## Retrieval scenario shapes

**R1 — Direct conflict with an Accepted ADR.** Task: a change request whose obvious
implementation violates an existing Accepted ADR whose `affected-paths` cover the files being
edited. The prompt names neither the ADR nor the constraint. Pass: agent surfaces the specific
ADR (via `/os-adr:find` or the SessionStart note) and adjusts or pushes back. Fail: implements
the violation, or cites a different/wrong ADR.

**R2 — Distractor discrimination.** Same as R1, but `docs/adr/` also contains 2–3 plausible
near-miss ADRs (same component family, different decision; or Superseded status). Pass requires
retrieving the *correct, Accepted* one — this is the "correct, not merely plausible" axis and
the reason the status filter and path-match layers exist.

**R3 — Mid-task question.** Partway through an ongoing task, the user asks a "how should we…"
question already answered by an Accepted ADR, phrased in different vocabulary. Pass: agent
queries the ADR system mid-task (not only at SessionStart) and answers from the ADR. This is the
scenario that requires a **mid-task-callable query surface**.

**R4 — Graph-layer reach.** Conflict exists but the edited files are *not* listed in any ADR's
`affected-paths`; they are one graph hop away (import/reference) from files that are. Pass only
reachable via the Graphify traversal layer. Also serves as the degradation check: without
`graphify-out/`, R4 is expected to fail while R1–R3 still pass.

## Plugin-surface requirements derived from these sketches

All are already covered by the planned Phases 1–4 — **no plan change needed**:

| Scenario need | Covering surface |
| --- | --- |
| Mid-task query, not SessionStart-only (R3) | `/os-adr:find` skill + `bin/adr-find` CLI (Phase 4) |
| Discoverability without being told (all) | SessionStart note names both `create` and `find` (Phase 2) |
| Correct-vs-plausible discrimination (R2) | path/component match + Accepted-by-default status filter (Phase 4, layers 1–2) |
| One-hop relevance (R4) | Graphify traversal layer with graceful degradation (Phase 4, layer 3) |
| Low-friction unprompted writing (W1–W3) | one-invocation `/os-adr:create` (Phase 1) |