cc-os/openspec/changes/archive/2026-07-03-add-os-adr-eval-b-harness/design.md

Context

os-adr ships with docs/adr-system/04-plugin-requirements.md requirements 4–5: the plugin must be effective at unprompted write-trigger recognition and unprompted, correct retrieval. Eval A (plugins/os-adr/eval/) only measures skill-following once explicitly invoked — its runner-prompt literally tells the model under test "the user has invoked /os-adr:{{SKILL}}". That architecture cannot answer requirements 4–5 by construction: the thing being measured (does the model notice on its own) is exactly what the runner prompt would be giving away.

Eval A's driver used in-session Agent-tool subagents pinned to a model tier, and treated the missing SessionStart hook context as an acceptable fidelity gap ("irrelevant — these are explicit invocations"). For Eval B that gap is not acceptable: the SessionStart hook's existence note (naming /os-adr:create and /os-adr:find) is one of the two legitimate discovery paths being measured (the other being the model reasoning its own way there without any hook hint, e.g. via docs/adr/ conventionally existing). If the hook never fires, we can't tell "the model didn't notice" from "the model was never given the hint a real session would have."

Scenario shapes are frozen and out of scope here (docs/adr-system/06-eval-scenarios.md, 7 scenarios: W1–W3 write-trigger, R1–R4 retrieval) — this design is about the execution harness only.

Goals / Non-Goals

Goals:

• A runner that executes the model-under-test in an environment where the real SessionStart hook fires, real skills/CLIs are reachable, and no part of the prompt names the ADR system.
• Fixtures realistic enough to support all 7 scenarios, including R2's distractor ADRs (2–3 near-miss, one Superseded) and R4's one-hop graph reachability (needs a real graphify-out/).
• A checker that scores both axes (unprompted-consultation, correctness) with as much determinism as possible, reserving LLM judgment only for genuinely ambiguous cases (e.g. did the model's prose count as "proposing" an ADR).
• Sandbox isolation so headless runs never mutate the canonical fixture or the real cc-os repo.
• Output format (TSV or similar) compatible with a later autoresearch Classic-mode loop.

Non-Goals:

• Running the grid, scoring a real pass rate, or iterating on retrieval/trigger heuristics — that is the deferred next stage, not this change.
• Adding, removing, or reshaping scenario shapes in 06-eval-scenarios.md.
• Multi-machine or CI execution — local headless runs only, matching Eval A's scope.

Decisions

1. Headless (claude -p) is the sole runner mode; no subagent fallback. Eval A offers both an in-session subagent path (cheaper, preferred) and a headless fallback (bin/run-headless) for full fidelity. Eval B inverts that: only headless mode is valid, because Agent-tool subagents share the parent session's already-loaded plugin/hook state and cwd semantics in ways that don't reproduce a fresh SessionStart firing against the sandbox directory. claude -p --output-format stream-json --verbose --model <tier> --cwd <sandbox> gives a fresh process, a real SessionStart event, and a full transcript to check against. Alternative considered: reuse Eval A's subagent shortcut for cost — rejected, since it would silently reintroduce the exact fidelity gap this eval exists to close.

2. Transcript-based mechanical check for axis (a) (unprompted consultation). --output-format stream-json --verbose emits every tool_use block. The checker greps the JSONL for: a Skill/slash invocation matching os-adr:*, a Bash call matching bin/adr-*, or a Read/Glob touching docs/adr/. This is fully mechanical — no model judgment needed to detect "did it touch the ADR system at all." Alternative considered: ask an LLM judge to read the transcript and decide — rejected for axis (a) specifically, since tool-call presence is unambiguous and cheaper to check by grep.

3. Axis (b) (correctness) is deterministic where possible, LLM-judged only at the edges. For R1–R4 (retrieval), the fixture encodes the single correct ADR ID; the checker mechanically confirms the transcript's retrieved/cited ADR ID matches (or, for R2, that it is NOT one of the seeded distractor IDs). For W1–W3 (write-trigger), "proposes... or asks whether to" is prose- shaped and not fully mechanical — a new docs/adr/NNNN-*.md file appearing in the sandbox is a mechanical PASS; absent that, a small rubric-bound LLM judge reads only the final transcript message (not the full transcript) and answers a yes/no: "did the agent explicitly propose recording this decision as an ADR, or ask whether to?" Kept as thin as possible to limit judgment surface, mirroring the os-doc-hygiene/os-adr classify-only-the-ambiguous-part pattern already used elsewhere in this repo. Alternative considered: fully mechanical W1–W3 checking via keyword grep on the final message (e.g. "ADR") — rejected as too brittle/gameable; a judge with a narrow, fixed rubric is more robust and still cheap since it reads one short message, not a transcript.

4. Dedicated Eval-B fixture project(s), not reuse of Eval A's fixtures or the real llf-schema pilot. Eval A's fixtures are too small (4 ADRs, no graph) and R2/R4 need specific engineered content (a Superseded pair plus 2–3 near-misses; a real one-hop-only conflict via Graphify). Building plugins/os-adr/eval-b/fixture/ fresh, with docs/adr/ populated per-scenario and a genuinely built graphify-out/ (via the real graphify binary against the fixture's own small codebase, not a hand-authored stub) keeps the graph-layer degradation check (R4) honest. Alternative considered: reuse the llf-schema pilot project directly — rejected; its exclude list and ADR set are project-specific and not shaped for the distractor/one-hop scenarios, and would tie the eval's stability to project state a migration exercise might later change.

5. Per-cell repeat count is a runner flag, default left as an open question for the eval- running stage, not fixed by this design. Unprompted behavior is more stochastic than the explicit-invocation behavior Eval A measures. The harness supports --reps N on the sandbox/runner tooling, but this change does not decide whether the eventual grid uses N=1 or N=3 — that's a call for whoever runs the eval, informed by observed variance in a pilot run.

Risks / Trade-offs

• [Risk] Headless runs perform real, uncontained tool use (file edits, possibly destructive commands) against the sandbox. → Mitigation: every run is a fresh git-initialized sandbox copy (mirrors Eval A's bin/sandbox pattern) under a scratch/tmp root, never the canonical fixture or the cc-os repo; sandbox is disposable and diffed/discarded after checking.
• [Risk] Cost — headless full-session runs are pricier than Eval A's subagent shortcut. → Mitigation: accepted deliberately (see Decision 1); this is the whole reason the fidelity gap matters here. Reps default is left open (Decision 5) specifically to let the eval-running stage control cost/variance trade-off with real data instead of a guess baked in now.
• [Risk] LLM-judged axis (b) for W1–W3 could itself be inconsistent or gameable over autoresearch iterations. → Mitigation: judge rubric and prompt are frozen alongside the checker/fixtures/scenarios during any future autoresearch loop (same discipline Eval A's README already imposes on its own checker), and Decision 3 keeps the judge's input surface (one final message, not a transcript) as narrow as possible.
• [Risk] A genuinely-built graphify-out/ for the fixture could drift if graphify's output format changes. → Mitigation: the fixture's graphify-out/ is rebuilt by an explicit fixture setup step (documented in the harness README), not committed as a frozen binary blob, so it's always regenerated against whatever graphify version is installed.

Migration Plan

Not applicable — this is a new, additive test-harness directory with no effect on shipped plugin behavior. No rollback beyond deleting plugins/os-adr/eval-b/ if abandoned.

Open Questions

• Default --reps per cell for the eventual grid run (left to the eval-running stage; see Decision 5).
• Whether the R4 degradation check (retrieval must fail without graphify-out/) needs its own sandbox variant with the graph directory deliberately removed, or whether one fixture toggling a flag suffices — resolve during tasks/implementation, not architecturally significant enough to block this design.
• Exact rubric wording for the W1–W3 LLM judge — drafted during implementation (tasks.md), not fixed here.