cc-os/openspec/changes/archive/2026-07-03-add-os-adr-eval-b-harness/design.md

## Context

`os-adr` ships with `docs/adr-system/04-plugin-requirements.md` requirements 4–5: the plugin
must be effective at *unprompted* write-trigger recognition and *unprompted, correct* retrieval.
Eval A (`plugins/os-adr/eval/`) only measures skill-*following* once explicitly invoked — its
runner-prompt literally tells the model under test "the user has invoked `/os-adr:{{SKILL}}`".
That architecture cannot answer requirements 4–5 by construction: the thing being measured
(does the model notice on its own) is exactly what the runner prompt would be giving away.

Eval A's driver used in-session Agent-tool subagents pinned to a model tier, and treated the
missing SessionStart hook context as an acceptable fidelity gap ("irrelevant — these are
explicit invocations"). For Eval B that gap is not acceptable: the SessionStart hook's existence
note (naming `/os-adr:create` and `/os-adr:find`) is one of the two legitimate discovery paths
being measured (the other being the model reasoning its own way there without any hook hint,
e.g. via `docs/adr/` conventionally existing). If the hook never fires, we can't tell "the model
didn't notice" from "the model was never given the hint a real session would have."

Scenario shapes are frozen and out of scope here (`docs/adr-system/06-eval-scenarios.md`, 7
scenarios: W1–W3 write-trigger, R1–R4 retrieval) — this design is about the execution harness
only.

## Goals / Non-Goals

**Goals:**
- A runner that executes the model-under-test in an environment where the real SessionStart
  hook fires, real skills/CLIs are reachable, and no part of the prompt names the ADR system.
- Fixtures realistic enough to support all 7 scenarios, including R2's distractor ADRs (2–3
  near-miss, one Superseded) and R4's one-hop graph reachability (needs a real `graphify-out/`).
- A checker that scores both axes (unprompted-consultation, correctness) with as much
  determinism as possible, reserving LLM judgment only for genuinely ambiguous cases (e.g. did
  the model's prose count as "proposing" an ADR).
- Sandbox isolation so headless runs never mutate the canonical fixture or the real cc-os repo.
- Output format (TSV or similar) compatible with a later `autoresearch` Classic-mode loop.

**Non-Goals:**
- Running the grid, scoring a real pass rate, or iterating on retrieval/trigger heuristics —
  that is the deferred next stage, not this change.
- Adding, removing, or reshaping scenario shapes in `06-eval-scenarios.md`.
- Multi-machine or CI execution — local headless runs only, matching Eval A's scope.

## Decisions

**1. Headless (`claude -p`) is the sole runner mode; no subagent fallback.**
Eval A offers both an in-session subagent path (cheaper, preferred) and a headless fallback
(`bin/run-headless`) for full fidelity. Eval B inverts that: only headless mode is valid, because
Agent-tool subagents share the parent session's already-loaded plugin/hook state and cwd
semantics in ways that don't reproduce a fresh SessionStart firing against the sandbox directory.
`claude -p --output-format stream-json --verbose --model <tier> --cwd <sandbox>` gives a fresh
process, a real SessionStart event, and a full transcript to check against.
*Alternative considered*: reuse Eval A's subagent shortcut for cost — rejected, since it would
silently reintroduce the exact fidelity gap this eval exists to close.

**2. Transcript-based mechanical check for axis (a) (unprompted consultation).**
`--output-format stream-json --verbose` emits every tool_use block. The checker greps the JSONL
for: a Skill/slash invocation matching `os-adr:*`, a `Bash` call matching `bin/adr-*`, or a
`Read`/`Glob` touching `docs/adr/`. This is fully mechanical — no model judgment needed to
detect "did it touch the ADR system at all."
*Alternative considered*: ask an LLM judge to read the transcript and decide — rejected for axis
(a) specifically, since tool-call presence is unambiguous and cheaper to check by grep.

**3. Axis (b) (correctness) is deterministic where possible, LLM-judged only at the edges.**
For R1–R4 (retrieval), the fixture encodes the single correct ADR ID; the checker mechanically
confirms the transcript's retrieved/cited ADR ID matches (or, for R2, that it is NOT one of the
seeded distractor IDs). For W1–W3 (write-trigger), "proposes... or asks whether to" is prose-
shaped and not fully mechanical — a new `docs/adr/NNNN-*.md` file appearing in the sandbox is a
mechanical PASS; absent that, a small rubric-bound LLM judge reads only the final transcript
message (not the full transcript) and answers a yes/no: "did the agent explicitly propose
recording this decision as an ADR, or ask whether to?" Kept as thin as possible to limit judgment
surface, mirroring the `os-doc-hygiene`/`os-adr` classify-only-the-ambiguous-part pattern already
used elsewhere in this repo.
*Alternative considered*: fully mechanical W1–W3 checking via keyword grep on the final message
(e.g. "ADR") — rejected as too brittle/gameable; a judge with a narrow, fixed rubric is more
robust and still cheap since it reads one short message, not a transcript.

**4. Dedicated Eval-B fixture project(s), not reuse of Eval A's fixtures or the real llf-schema
pilot.**
Eval A's fixtures are too small (4 ADRs, no graph) and R2/R4 need specific engineered content
(a Superseded pair plus 2–3 near-misses; a real one-hop-only conflict via Graphify). Building
`plugins/os-adr/eval-b/fixture/` fresh, with `docs/adr/` populated per-scenario and a genuinely
built `graphify-out/` (via the real `graphify` binary against the fixture's own small codebase,
not a hand-authored stub) keeps the graph-layer degradation check (R4) honest.
*Alternative considered*: reuse the llf-schema pilot project directly — rejected; its exclude
list and ADR set are project-specific and not shaped for the distractor/one-hop scenarios,
and would tie the eval's stability to project state a migration exercise might later change.

**5. Per-cell repeat count is a runner flag, default left as an open question for the eval-
running stage, not fixed by this design.**
Unprompted behavior is more stochastic than the explicit-invocation behavior Eval A measures.
The harness supports `--reps N` on the sandbox/runner tooling, but this change does not decide
whether the eventual grid uses N=1 or N=3 — that's a call for whoever runs the eval, informed by
observed variance in a pilot run.

## Risks / Trade-offs

- **[Risk] Headless runs perform real, uncontained tool use (file edits, possibly destructive
  commands) against the sandbox.** → Mitigation: every run is a fresh git-initialized sandbox
  copy (mirrors Eval A's `bin/sandbox` pattern) under a scratch/tmp root, never the canonical
  fixture or the cc-os repo; sandbox is disposable and diffed/discarded after checking.
- **[Risk] Cost — headless full-session runs are pricier than Eval A's subagent shortcut.** →
  Mitigation: accepted deliberately (see Decision 1); this is the whole reason the fidelity gap
  matters here. Reps default is left open (Decision 5) specifically to let the eval-running
  stage control cost/variance trade-off with real data instead of a guess baked in now.
- **[Risk] LLM-judged axis (b) for W1–W3 could itself be inconsistent or gameable over
  `autoresearch` iterations.** → Mitigation: judge rubric and prompt are frozen alongside the
  checker/fixtures/scenarios during any future `autoresearch` loop (same discipline Eval A's
  README already imposes on its own checker), and Decision 3 keeps the judge's input surface
  (one final message, not a transcript) as narrow as possible.
- **[Risk] A genuinely-built `graphify-out/` for the fixture could drift if `graphify`'s output
  format changes.** → Mitigation: the fixture's `graphify-out/` is rebuilt by an explicit fixture
  setup step (documented in the harness README), not committed as a frozen binary blob, so it's
  always regenerated against whatever `graphify` version is installed.

## Migration Plan

Not applicable — this is a new, additive test-harness directory with no effect on shipped plugin
behavior. No rollback beyond deleting `plugins/os-adr/eval-b/` if abandoned.

## Open Questions

- Default `--reps` per cell for the eventual grid run (left to the eval-running stage; see
  Decision 5).
- Whether the R4 degradation check (retrieval must fail without `graphify-out/`) needs its own
  sandbox variant with the graph directory deliberately removed, or whether one fixture toggling
  a flag suffices — resolve during tasks/implementation, not architecturally significant enough
  to block this design.
- Exact rubric wording for the W1–W3 LLM judge — drafted during implementation (tasks.md), not
  fixed here.