## Why The `os-adr` plugin's two hardest requirements — unprompted write-trigger recognition and unprompted correct retrieval (`04-plugin-requirements.md` reqs 4–5) — have never been measured. Eval A (`plugins/os-adr/eval/`) only proves the plugin executes correctly once explicitly invoked; it says nothing about whether an agent notices, on its own, that the ADR system is relevant. The scenario *shapes* for this held-out eval already exist (`docs/adr-system/06-eval-scenarios.md`) but no fixtures, prompts, checker, or runner exist yet. Building it now closes the last unmeasured requirement in the locked rollout order. ## What Changes - Build a new eval harness, `plugins/os-adr/eval-b/`, sibling to but structurally distinct from Eval A, for the 7 held-out scenarios (W1–W3 write-trigger, R1–R4 retrieval). - Author full scenario prompts/fixtures from the frozen shapes in `06-eval-scenarios.md` — scenario *shapes* are not renegotiated by this change; only their execution artifacts are new. - Fixture project(s): a realistic, onboarded project state (populated `docs/adr/` incl. a Superseded pair for R2 distractors, a built `graphify-out/` for R4) that the SessionStart hook recognizes as ADR-initialized. - A **headless runner** (real `claude -p` in a real sandbox cwd) as the primary execution mode, not an in-session Agent-tool subagent — because the SessionStart hook firing for real is part of what's under test (Eval A's subagent shortcut is not valid here; see design.md). - A deterministic-first, two-axis checker: (a) did the agent consult/propose the ADR system unprompted at all, (b) did it act on/write the *specific correct* thing. Axis (a) is mechanically checkable (tool-call/transcript grep for `/os-adr:*` or `docs/adr/` reads/writes); axis (b) requires a small LLM-judged correctness check against the fixture's known-correct answer, kept as thin and rubric-bound as possible. - Sandbox/grid tooling mirroring Eval A's shape (`bin/sandbox`, `bin/check`, results.tsv) adapted for headless execution and the two-axis score. - Wiring for the later `autoresearch` iteration stage (not run by this change — building the harness only). Out of scope for this change: actually running the eval grid, iterating retrieval/trigger heuristics, or adding/altering scenario shapes in `06-eval-scenarios.md`. ## Capabilities ### New Capabilities - `adr-eval-b-harness`: fixtures, held-out scenario prompts, headless sandbox/runner tooling, and a two-axis deterministic-first checker for the unprompted write-trigger and retrieval eval. ### Modified Capabilities (none — this adds a new, separate eval surface; it does not change the behavior of `adr-authoring`, `adr-retrieval`, `adr-session-awareness`, or any shipped plugin capability) ## Impact - New directory: `plugins/os-adr/eval-b/` (fixtures, scenarios, bin/, runner assets, README). - No changes to plugin source (`lib/adr/`, `bin/adr-*`, `hooks/`, `skills/`) — this is a pure test-harness addition. - Depends on `claude` CLI headless mode (`claude -p`) being available in the build/dev environment for the runner to execute against a real sandbox with hooks firing. - Once built, unlocks the deferred final stage of the locked os-adr rollout order (running the grid + `autoresearch` iteration), tracked separately in `CLAUDE.md`.