cc-os/openspec/changes/archive/2026-07-03-add-os-adr-eval-b-harness/tasks.md

1. Fixture project(s)

• 1.1 Scaffold plugins/os-adr/eval-b/fixture/ as a small, self-contained fixture codebase (init as its own git repo, mirroring Eval A's fixture/project/ pattern)
• 1.2 Author docs/adr/ content for W1–W3 (a plausible existing convention/decision history the scenarios can plausibly reverse or extend) and initialize the ADR index
• 1.3 Author R1's Accepted ADR with affected-paths covering specific fixture files whose obvious next change would violate it
• 1.4 Extend docs/adr/ with R2's 2–3 distractor ADRs (near-miss same-component-family, plus one Superseded) alongside the correct one
• 1.5 Add an R3 Accepted ADR whose content answers a "how should we…" question phrased in different vocabulary than the ADR's own text
• 1.6 Design R4's one-hop layout: files that import/reference an R1-style ADR's affected-paths without being listed themselves; run the real graphify binary to produce graphify-out/ and confirm the hop is actually one edge away in the resulting graph
• 1.7 Document a fixture regeneration step (README) that rebuilds graphify-out/ from scratch rather than committing it as a frozen blob (bin/build-fixture-graph + "Fixture regeneration" section in eval-b/README.md; script asserts the R4 one-hop invariant on every rebuild)

2. Scenario prompts

• 2.1 Write eval-b/scenarios/W1.md..W3.md and R1.md..R4.md: a task prompt per scenario that never names the ADR system, plugin, or exact constraint text, plus the checker-facing metadata (correct ADR ID for R1–R4; expected new-ADR shape for W1–W3) following the pass/fail language already sketched in 06-eval-scenarios.md
• 2.2 Self-review each prompt against the "held-out" ground rule before running anything against it (no informal trial runs — first real execution is the self-test in section 5) (R1's "skip the shared client" wording was flagged and removed in this review)

3. Headless runner

• 3.1 Write eval-b/bin/sandbox <scenario> <dest>: fresh git-initialized sandbox copy of the right fixture variant (mirrors Eval A's bin/sandbox; R4-nograph variant included)
• 3.2 Write eval-b/bin/run <scenario> <model> <sandbox>: invokes claude -p --output-format stream-json --verbose --model <tier> with cwd set to the sandbox, passing only the scenario's task prompt (no system-level hints), capturing the full JSONL transcript to a file in the sandbox
• 3.3 Confirm the os-adr plugin (and, for R4, graphify) are active in the environment the runner executes in, and that a sandbox with an initialized docs/adr/ triggers the SessionStart hook's "present" branch (verified via a neutral non-scenario probe prompt: haiku quoted the [os-adr] note verbatim; transcript carries a system/hook_response event)
• 3.4 Add --reps N support to the runner for repeated executions of the same cell (default left open per design.md Decision 5; document how to override)

4. Checker

• 4.1 Write eval-b/bin/check <scenario> <sandbox> [--tsv <model>]: parses the JSONL transcript for axis (a) (tool_use blocks touching os-adr:*, bin/adr-*, or docs/adr/ reads/globs)
• 4.2 Implement axis (b) for R1–R4: compare the transcript's cited/acted-on ADR ID against the scenario's pre-declared correct ID; fail on distractor match (R2) or on missing graph-layer reach (R4)
• 4.3 Implement axis (b) for W1–W3: mechanical PASS if a new docs/adr/NNNN-*.md file exists in the sandbox matching the scenario's expected shape; otherwise invoke the narrow LLM judge (final message only) with a fixed rubric to decide propose/ask-vs-silent (rubric frozen in eval-b/judge-rubric.md; stubbable via ADR_EVAL_B_JUDGE_CMD)
• 4.4 Emit the TSV row format (scenario, model tier, axis-a, axis-b, pass/fail) for autoresearch compatibility
• 4.5 Write the R4 degradation-check variant: same scenario run against a sandbox with graphify-out/ removed, expected to FAIL where the graph-layer path was required (bin/sandbox R4-nograph + bin/check R4-nograph alias)

5. Self-test (mirrors Eval A's own-both-directions check)

• 5.1 For each of the 7 scenarios, hand-simulate a "perfect" transcript/sandbox state (an agent that does the right thing) and confirm bin/check scores PASS on both axes (scripted as eval-b/bin/self-test; includes the W judge-fallback path with a stub judge, plus a one-off smoke test of the real haiku judge)
• 5.2 For each of the 7 scenarios, check an untouched sandbox (no ADR consultation at all) and confirm bin/check scores axis (a) FAIL (also covers R4-nograph, the R2 superseded-distractor trap, and the missing-hook-context invalid-run case)
• 5.3 Confirm sandbox isolation: after a run, diff the canonical fixture and the cc-os repo and verify neither was modified (fixture digest check inside self-test; git status clean of unexpected entries)

6. Documentation

• 6.1 Write eval-b/README.md mirroring Eval A's README shape: layout table, how to run a single cell, how the two-axis scoring works, explicit warning that scenario prompts are held-out and must not be informally tried out outside this harness's own self-test
• 6.2 Cross-link from CLAUDE.md's os-adr section once this change is applied and archived (status line update per this repo's "keep this file current" convention) — Eval B bullet added to the os-adr component section, "Remaining" rollout line updated, and 06-eval-scenarios.md status line now points at the built harness