cc-os/openspec/changes/archive/2026-07-03-add-os-adr-eval-b-harness/tasks.md

## 1. Fixture project(s)

- [x] 1.1 Scaffold `plugins/os-adr/eval-b/fixture/` as a small, self-contained fixture codebase
      (init as its own git repo, mirroring Eval A's `fixture/project/` pattern)
- [x] 1.2 Author `docs/adr/` content for W1–W3 (a plausible existing convention/decision history
      the scenarios can plausibly reverse or extend) and initialize the ADR index
- [x] 1.3 Author R1's Accepted ADR with `affected-paths` covering specific fixture files whose
      obvious next change would violate it
- [x] 1.4 Extend `docs/adr/` with R2's 2–3 distractor ADRs (near-miss same-component-family,
      plus one Superseded) alongside the correct one
- [x] 1.5 Add an R3 Accepted ADR whose content answers a "how should we…" question phrased in
      different vocabulary than the ADR's own text
- [x] 1.6 Design R4's one-hop layout: files that import/reference an R1-style ADR's
      `affected-paths` without being listed themselves; run the real `graphify` binary to produce
      `graphify-out/` and confirm the hop is actually one edge away in the resulting graph
- [x] 1.7 Document a fixture regeneration step (README) that rebuilds `graphify-out/` from
      scratch rather than committing it as a frozen blob (`bin/build-fixture-graph` +
      "Fixture regeneration" section in `eval-b/README.md`; script asserts the R4 one-hop
      invariant on every rebuild)

## 2. Scenario prompts

- [x] 2.1 Write `eval-b/scenarios/W1.md`..`W3.md` and `R1.md`..`R4.md`: a task prompt per
      scenario that never names the ADR system, plugin, or exact constraint text, plus the
      checker-facing metadata (correct ADR ID for R1–R4; expected new-ADR shape for W1–W3)
      following the pass/fail language already sketched in `06-eval-scenarios.md`
- [x] 2.2 Self-review each prompt against the "held-out" ground rule before running anything
      against it (no informal trial runs — first real execution is the self-test in section 5)
      (R1's "skip the shared client" wording was flagged and removed in this review)

## 3. Headless runner

- [x] 3.1 Write `eval-b/bin/sandbox <scenario> <dest>`: fresh git-initialized sandbox copy of the
      right fixture variant (mirrors Eval A's `bin/sandbox`; `R4-nograph` variant included)
- [x] 3.2 Write `eval-b/bin/run <scenario> <model> <sandbox>`: invokes
      `claude -p --output-format stream-json --verbose --model <tier>` with cwd set to the
      sandbox, passing only the scenario's task prompt (no system-level hints), capturing the
      full JSONL transcript to a file in the sandbox
- [x] 3.3 Confirm the os-adr plugin (and, for R4, graphify) are active in the environment the
      runner executes in, and that a sandbox with an initialized `docs/adr/` triggers the
      SessionStart hook's "present" branch (verified via a neutral non-scenario probe prompt:
      haiku quoted the [os-adr] note verbatim; transcript carries a system/hook_response event)
- [x] 3.4 Add `--reps N` support to the runner for repeated executions of the same cell (default
      left open per design.md Decision 5; document how to override)

## 4. Checker

- [x] 4.1 Write `eval-b/bin/check <scenario> <sandbox> [--tsv <model>]`: parses the JSONL
      transcript for axis (a) (tool_use blocks touching `os-adr:*`, `bin/adr-*`, or
      `docs/adr/` reads/globs)
- [x] 4.2 Implement axis (b) for R1–R4: compare the transcript's cited/acted-on ADR ID against
      the scenario's pre-declared correct ID; fail on distractor match (R2) or on missing
      graph-layer reach (R4)
- [x] 4.3 Implement axis (b) for W1–W3: mechanical PASS if a new `docs/adr/NNNN-*.md` file
      exists in the sandbox matching the scenario's expected shape; otherwise invoke the narrow
      LLM judge (final message only) with a fixed rubric to decide propose/ask-vs-silent
      (rubric frozen in `eval-b/judge-rubric.md`; stubbable via ADR_EVAL_B_JUDGE_CMD)
- [x] 4.4 Emit the TSV row format (scenario, model tier, axis-a, axis-b, pass/fail) for
      `autoresearch` compatibility
- [x] 4.5 Write the R4 degradation-check variant: same scenario run against a sandbox with
      `graphify-out/` removed, expected to FAIL where the graph-layer path was required
      (`bin/sandbox R4-nograph` + `bin/check R4-nograph` alias)

## 5. Self-test (mirrors Eval A's own-both-directions check)

- [x] 5.1 For each of the 7 scenarios, hand-simulate a "perfect" transcript/sandbox state (an
      agent that does the right thing) and confirm `bin/check` scores PASS on both axes
      (scripted as `eval-b/bin/self-test`; includes the W judge-fallback path with a stub judge,
      plus a one-off smoke test of the real haiku judge)
- [x] 5.2 For each of the 7 scenarios, check an untouched sandbox (no ADR consultation at all)
      and confirm `bin/check` scores axis (a) FAIL (also covers R4-nograph, the R2
      superseded-distractor trap, and the missing-hook-context invalid-run case)
- [x] 5.3 Confirm sandbox isolation: after a run, diff the canonical fixture and the cc-os repo
      and verify neither was modified (fixture digest check inside self-test; git status clean
      of unexpected entries)

## 6. Documentation

- [x] 6.1 Write `eval-b/README.md` mirroring Eval A's README shape: layout table, how to run a
      single cell, how the two-axis scoring works, explicit warning that scenario prompts are
      held-out and must not be informally tried out outside this harness's own self-test
- [x] 6.2 Cross-link from `CLAUDE.md`'s os-adr section once this change is applied and archived
      (status line update per this repo's "keep this file current" convention) — Eval B bullet
      added to the os-adr component section, "Remaining" rollout line updated, and
      `06-eval-scenarios.md` status line now points at the built harness