cc-os/docs/adr-system/06-eval-scenarios.md

Eval Scenario Sketches (held-out, deferred stage)

Last updated: 2026-07-03 — sketches only; the evaluation itself is a separate later stage (locked sequencing in 04-plugin-requirements.md).

These sketch the shape of the held-out scenarios that will eventually measure requirements 4 and 5 (unprompted write-trigger recognition, unprompted correct retrieval). They exist now, before the plugin skeleton freezes, for one purpose: to catch any plugin surface the eval will need that Phases 1–4 don't provide. They are not to be run, automated, or informally "tried out" during the build — that would contaminate the held-out methodology.

Ground rules for all scenarios

• Held-out: scenario prompts never name the ADR system, the plugin, an ADR ID/title, or the exact constraint text. Relevance must be discovered, not pointed at.
• Scored on two axes: (a) did the agent consult/propose the ADR system at all unprompted; (b) did it act on the specific correct ADR (retrieval) or propose a well-scoped new one (write-trigger) — not merely a plausible-looking neighbor.
• Environment: a real onboarded project (post-pilot) with a populated docs/adr/, the SessionStart hook active, and — for graph-layer variants — a built graphify-out/.
• The eval stage will iterate with the autoresearch skill (modify → verify → keep/discard) against these scenarios once written out in full.

Write-trigger scenario shapes

W1 — Irreversible choice mid-task. Task: "Add persistence to X" in a project where the storage engine choice is genuinely open and consequential (hard to reverse once data exists). Never mention ADRs. Pass: the agent proposes recording the engine decision via /os-adr:create (or asks whether to) before or immediately after implementing. Fail: silently picks one and moves on.

W2 — Cross-cutting convention change. Task: a refactor request whose natural solution changes a convention used across many files (e.g. error-handling shape, module layout). Pass: the agent recognizes the decision outlives the task and proposes an ADR. Fail: does the refactor with no record.

W3 — Contested reversal. Task: user asks for something that quietly reverses an earlier (non-ADR'd, but discoverable-in-code) approach. Pass: agent flags that this is a decision-of-record moment and proposes capturing it. This shape overlaps retrieval when the earlier approach is ADR'd — kept separate deliberately.

Retrieval scenario shapes

R1 — Direct conflict with an Accepted ADR. Task: a change request whose obvious implementation violates an existing Accepted ADR whose affected-paths cover the files being edited. The prompt names neither the ADR nor the constraint. Pass: agent surfaces the specific ADR (via /os-adr:find or the SessionStart note) and adjusts or pushes back. Fail: implements the violation, or cites a different/wrong ADR.

R2 — Distractor discrimination. Same as R1, but docs/adr/ also contains 2–3 plausible near-miss ADRs (same component family, different decision; or Superseded status). Pass requires retrieving the correct, Accepted one — this is the "correct, not merely plausible" axis and the reason the status filter and path-match layers exist.

R3 — Mid-task question. Partway through an ongoing task, the user asks a "how should we…" question already answered by an Accepted ADR, phrased in different vocabulary. Pass: agent queries the ADR system mid-task (not only at SessionStart) and answers from the ADR. This is the scenario that requires a mid-task-callable query surface.

R4 — Graph-layer reach. Conflict exists but the edited files are not listed in any ADR's affected-paths; they are one graph hop away (import/reference) from files that are. Pass only reachable via the Graphify traversal layer. Also serves as the degradation check: without graphify-out/, R4 is expected to fail while R1–R3 still pass.

Plugin-surface requirements derived from these sketches

All are already covered by the planned Phases 1–4 — no plan change needed:

Scenario need	Covering surface
Mid-task query, not SessionStart-only (R3)	/os-adr:find skill + bin/adr-find CLI (Phase 4)
Discoverability without being told (all)	SessionStart note names both create and find (Phase 2)
Correct-vs-plausible discrimination (R2)	path/component match + Accepted-by-default status filter (Phase 4, layers 1–2)
One-hop relevance (R4)	Graphify traversal layer with graceful degradation (Phase 4, layer 3)
Low-friction unprompted writing (W1–W3)	one-invocation /os-adr:create (Phase 1)