cc-os/plugins/os-adr/eval-b/README.md

# os-adr Eval B — unprompted write-trigger & retrieval (held-out)

_Last updated: 2026-07-03 — harness built and self-tested (perfect-run PASS,
untouched-sandbox FAIL, both directions for all 7 scenarios); grid not yet run._

Measures the two hardest os-adr requirements (`docs/adr-system/04-plugin-requirements.md`
reqs 4–5): does an agent notice **on its own** that the ADR system is relevant — proposing to
record a consequential decision (W1–W3), or retrieving the specific correct decision before
changing decided-on behavior (R1–R4)? Scenario shapes are frozen in
`docs/adr-system/06-eval-scenarios.md`. This is **not** Eval A (`../eval/`), which measures
skill-*execution* once explicitly invoked — do not conflate them (ADR-021).

> **HELD-OUT — read this first.** The `## Task` blocks in `scenarios/*.md` must never be
> pasted into an interactive session, "tried out" informally, or fed to a model outside
> `bin/run`. The only permitted non-grid execution is `bin/self-test`, which fabricates
> transcripts and never uses the Task blocks. Informal trials contaminate the methodology.

## Layout

| Path | What |
| --- | --- |
| `fixture/project/` | small Ruby webhook-relay codebase, 6-ADR history (incl. a Superseded pair) |
| `scenarios/W1..W3.md` | write-trigger prompts (persistence choice, convention change, reversal) |
| `scenarios/R1..R4.md` | retrieval prompts (direct conflict, distractors, mid-task Q, graph hop) |
| `bin/build-fixture-graph` | rebuild `fixture/project/graphify-out/` from scratch (R4 needs it) |
| `bin/sandbox <Sn> <dest>` | fresh git-initialized sandbox; only `R4` keeps the graph; `R4-nograph` = degradation variant |
| `bin/run <Sn> <model> <workdir> [--reps N]` | headless `claude -p` runner; the ONLY valid execution mode |
| `bin/check <Sn> <sandbox> [--tsv <model>]` | two-axis checker; exit 0/1; TSV mode for autoresearch |
| `bin/self-test [workdir]` | model-free both-directions harness self-test |
| `judge-rubric.md` | frozen rubric for the W-scenario axis (b) LLM judge fallback |

## Why headless-only (no Agent-tool subagents)

The SessionStart hook's `[os-adr]` note is one of the two discovery paths under measurement.
In-session subagents don't get a fresh SessionStart against the sandbox cwd, so they can't
tell "the model didn't notice" from "the model never got the hint a real session would have."
`bin/run` gives each rep a fresh `claude -p` process with cwd set to a fresh sandbox; the
transcript's `system/hook_response` event proves the hook context reached the model — the
checker treats a transcript without it as an invalid run. (Verified 2026-07-03 with a neutral
non-scenario probe: haiku quoted the hook note verbatim from a sandbox session.)

## Two-axis scoring

- **Axis (a) — unprompted consultation (mechanical).** Any `tool_use` block in the
  `stream-json` transcript that invokes an `os-adr:*` skill, a `bin/adr-*` CLI, or touches
  `docs/adr/`. Prose never counts, so the hook note can't false-positive it.
- **Axis (b) — specific correctness.**
  - R1–R4 (mechanical): assistant text must cite the scenario's pre-declared correct ADR ID
    (R1→0002, R2→0003, R3→0005 content match [UTC + ISO-8601], R4→0003) and flag the
    conflict. R2 fails if the Superseded distractor 0001 is cited as if live.
  - W1–W3 (deterministic-first): PASS mechanically when a new `docs/adr/NNNN-*.md` matching
    the scenario's topic exists; only otherwise does a rubric-bound judge (haiku,
    `judge-rubric.md`, final message only) decide propose/ask-vs-silent. Stub it with
    `ADR_EVAL_B_JUDGE_CMD` (reads prompt on stdin, prints YES/NO).
- **Overall PASS** = both axes PASS. TSV row: `scenario  model  A:…  B:…  PASS|FAIL  reasons`.

## Running one cell

```bash
eval-b/bin/build-fixture-graph                  # once, before any R4 cell
eval-b/bin/run R2 haiku /tmp/adr-eval-b --reps 3 --results /tmp/adr-eval-b/results.tsv
```

Each rep: fresh sandbox → `claude -p --output-format stream-json --verbose --model <tier>
--dangerously-skip-permissions "<Task block>"` with cwd = sandbox → `transcript.jsonl` saved →
`bin/check` appends one TSV row. `--reps` default is 1; the grid-run stage picks the real
repeat count after observing variance in a pilot (design decision, left open deliberately).

The R4 degradation check: `bin/run R4-nograph <model> <workdir>` — same prompt, graph absent.
An axis (b) FAIL there is the **expected** outcome; it confirms R4's pass genuinely required
the Graphify layer (paired with an R4 PASS on the same tier).

## Fixture regeneration

`graphify-out/` is never committed — rebuild it with `bin/build-fixture-graph`, which runs the
real `graphify update` against the fixture (pure tree-sitter AST: the fixture's
`.graphifyignore` excludes all markdown, so no Ollama/LLM is involved and the build is
reproducible) and then asserts the R4 one-hop invariant (`lib/relay/reports.rb` is exactly one
graph hop from `lib/relay/delivery.rb`). If graphify's output format changes and the invariant
breaks, fix the fixture layout — not the checker — in a human-reviewed change.

The ADR history was generated with the plugin's own CLIs (`adr-init`/`adr-new`), so frontmatter
and index match the shipped format, including the mechanically-superseded 0001→0003 pair.

## Optimizing with `/autoresearch` (later stage — not part of building this harness)

Same discipline as Eval A: the **checker, fixtures, scenario prompts, and judge rubric are
frozen during a loop** — only plugin surface wording (SKILL.md, hook note text) may move. Metric:
pass rate over the {W1..W3, R1..R4} × {haiku, sonnet} grid via `bin/check --tsv`. If a scenario
or checker turns out to be wrong, stop the loop and fix it as a separate, human-reviewed change.
Unlike Eval A, every cell costs a real headless session — budget reps accordingly.

## Adding a scenario

Add `scenarios/X.md` (Task block + checker metadata), a class in `bin/check`, the fixture route
in `bin/sandbox`, extend `bin/self-test` both directions, and keep the held-out rule: the new
Task block gets no informal trial runs.