cc-os/plugins/os-adr/eval-b

os-adr Eval B — unprompted write-trigger & retrieval (held-out)

Last updated: 2026-07-03 — harness built and self-tested (perfect-run PASS, untouched-sandbox FAIL, both directions for all 7 scenarios). Grid run 2026-07-03, 1 rep/cell: haiku 0/8 PASS (never unprompted-consults the ADR system in any scenario); sonnet 5/8 PASS (fails W3 — doesn't propose recording the decision; fails R1 — misses the direct-conflict retrieval). R4-nograph FAILed on both tiers as expected (degradation check; only meaningful paired with an R4 PASS, which sonnet has and haiku doesn't). Wording iteration via /autoresearch not yet started.

Measures the two hardest os-adr requirements (docs/adr-system/04-plugin-requirements.md reqs 4–5): does an agent notice on its own that the ADR system is relevant — proposing to record a consequential decision (W1–W3), or retrieving the specific correct decision before changing decided-on behavior (R1–R4)? Scenario shapes are frozen in docs/adr-system/06-eval-scenarios.md. This is not Eval A (../eval/), which measures skill-execution once explicitly invoked — do not conflate them (ADR-021).

HELD-OUT — read this first. The ## Task blocks in scenarios/*.md must never be pasted into an interactive session, "tried out" informally, or fed to a model outside bin/run. The only permitted non-grid execution is bin/self-test, which fabricates transcripts and never uses the Task blocks. Informal trials contaminate the methodology.

Layout

Path	What
fixture/project/	small Ruby webhook-relay codebase, 6-ADR history (incl. a Superseded pair)
scenarios/W1..W3.md	write-trigger prompts (persistence choice, convention change, reversal)
scenarios/R1..R4.md	retrieval prompts (direct conflict, distractors, mid-task Q, graph hop)
bin/build-fixture-graph	rebuild fixture/project/graphify-out/ from scratch (R4 needs it)
bin/sandbox <Sn> <dest>	fresh git-initialized sandbox; only R4 keeps the graph; R4-nograph = degradation variant
bin/run <Sn> <model> <workdir> [--reps N]	headless claude -p runner; the ONLY valid execution mode
bin/check <Sn> <sandbox> [--tsv <model>]	two-axis checker; exit 0/1; TSV mode for autoresearch
bin/self-test [workdir]	model-free both-directions harness self-test
judge-rubric.md	frozen rubric for the W-scenario axis (b) LLM judge fallback

Why headless-only (no Agent-tool subagents)

The SessionStart hook's [os-adr] note is one of the two discovery paths under measurement. In-session subagents don't get a fresh SessionStart against the sandbox cwd, so they can't tell "the model didn't notice" from "the model never got the hint a real session would have." bin/run gives each rep a fresh claude -p process with cwd set to a fresh sandbox; the transcript's system/hook_response event proves the hook context reached the model — the checker treats a transcript without it as an invalid run. (Verified 2026-07-03 with a neutral non-scenario probe: haiku quoted the hook note verbatim from a sandbox session.)

Two-axis scoring

• Axis (a) — unprompted consultation (mechanical). Any tool_use block in the stream-json transcript that invokes an os-adr:* skill, a bin/adr-* CLI, or touches docs/adr/. Prose never counts, so the hook note can't false-positive it.
• Axis (b) — specific correctness.
  • R1–R4 (mechanical): assistant text must cite the scenario's pre-declared correct ADR ID (R1→0002, R2→0003, R3→0005 content match [UTC + ISO-8601], R4→0003) and flag the conflict. R2 fails if the Superseded distractor 0001 is cited as if live.
  • W1–W3 (deterministic-first): PASS mechanically when a new docs/adr/NNNN-*.md matching the scenario's topic exists; only otherwise does a rubric-bound judge (haiku, judge-rubric.md, final message only) decide propose/ask-vs-silent. Stub it with ADR_EVAL_B_JUDGE_CMD (reads prompt on stdin, prints YES/NO).
• Overall PASS = both axes PASS. TSV row: scenario model A:… B:… PASS|FAIL reasons.

Running one cell

eval-b/bin/build-fixture-graph                  # once, before any R4 cell
eval-b/bin/run R2 haiku /tmp/adr-eval-b --reps 3 --results /tmp/adr-eval-b/results.tsv

Each rep: fresh sandbox → claude -p --output-format stream-json --verbose --model <tier> --dangerously-skip-permissions "<Task block>" with cwd = sandbox → transcript.jsonl saved → bin/check appends one TSV row. --reps default is 1; the grid-run stage picks the real repeat count after observing variance in a pilot (design decision, left open deliberately).

The R4 degradation check: bin/run R4-nograph <model> <workdir> — same prompt, graph absent. An axis (b) FAIL there is the expected outcome; it confirms R4's pass genuinely required the Graphify layer (paired with an R4 PASS on the same tier).

Fixture regeneration

graphify-out/ is never committed — rebuild it with bin/build-fixture-graph, which runs the real graphify update against the fixture (pure tree-sitter AST: the fixture's .graphifyignore excludes all markdown, so no Ollama/LLM is involved and the build is reproducible) and then asserts the R4 one-hop invariant (lib/relay/reports.rb is exactly one graph hop from lib/relay/delivery.rb). If graphify's output format changes and the invariant breaks, fix the fixture layout — not the checker — in a human-reviewed change.

The ADR history was generated with the plugin's own CLIs (adr-init/adr-new), so frontmatter and index match the shipped format, including the mechanically-superseded 0001→0003 pair.

Optimizing with /autoresearch (later stage — not part of building this harness)

Same discipline as Eval A: the checker, fixtures, scenario prompts, and judge rubric are frozen during a loop — only plugin surface wording (SKILL.md, hook note text) may move. Metric: pass rate over the {W1..W3, R1..R4} × {haiku, sonnet} grid via bin/check --tsv. If a scenario or checker turns out to be wrong, stop the loop and fix it as a separate, human-reviewed change. Unlike Eval A, every cell costs a real headless session — budget reps accordingly.

Adding a scenario

Add scenarios/X.md (Task block + checker metadata), a class in bin/check, the fixture route in bin/sandbox, extend bin/self-test both directions, and keep the held-out rule: the new Task block gets no informal trial runs.