## Purpose

Evaluation harness B: held-out, unprompted-behavior testing for the os-adr plugin. Tests whether the plugin's core write/retrieval operations surface naturally without coaching, across multiple model tiers, with deterministic-first scoring (mechanical checks before fallback to LLM judge).

## Requirements

### Requirement: Held-out scenario fixtures
The harness SHALL provide a dedicated fixture project (or projects) under
`plugins/os-adr/eval-b/fixture/` that supports all 7 held-out scenarios (W1–W3, R1–R4) without
any scenario prompt naming the ADR system, an ADR ID, or the exact constraint text being tested.

#### Scenario: R2 distractor set present
- **WHEN** the R2 fixture's `docs/adr/` is inspected
- **THEN** it contains the one correct Accepted ADR plus 2–3 near-miss ADRs (same component
  family, different decision, or Superseded status) that a naive retrieval could mistakenly cite

#### Scenario: R4 graph reachability is real, not stubbed
- **WHEN** the R4 fixture's `graphify-out/` is inspected
- **THEN** it was produced by running the real `graphify` binary against the fixture's own
  codebase (not hand-authored), and the conflicting files are one import/reference hop away from
  files listed in the relevant ADR's `affected-paths`, not directly listed themselves

### Requirement: Headless runner with real hook firing
The harness SHALL execute each scenario via a headless `claude -p` process with its working
directory set to a fresh sandbox copy of the fixture, so that the real SessionStart hook fires
for the model under test. In-session Agent-tool subagents SHALL NOT be used as the execution
mode for this eval.

#### Scenario: SessionStart hook context reaches the model under test
- **WHEN** a scenario is run via the headless runner against an ADR-initialized sandbox
- **THEN** the transcript shows the SessionStart hook's additionalContext was present in the
  model's context before its first action

#### Scenario: Sandbox isolation
- **WHEN** any scenario run completes (pass or fail)
- **THEN** the canonical fixture directory and the cc-os repo itself show no modifications —
  only the disposable sandbox copy was touched

### Requirement: Two-axis deterministic-first checker
The harness SHALL score each run on two independent axes: (a) whether the model consulted or
proposed the ADR system at all, unprompted, and (b) whether it acted on or wrote the specific
correct thing rather than a merely plausible one. Axis (a), and axis (b) for retrieval scenarios,
SHALL be scored mechanically from the transcript and sandbox file state; axis (b) for
write-trigger scenarios MAY fall back to a narrow, rubric-bound LLM judge only when no new ADR
file was mechanically created.

#### Scenario: Axis (a) mechanical detection
- **WHEN** the checker scans a scenario's `stream-json` transcript
- **THEN** it detects axis (a) as true if any tool_use block invokes an `os-adr:*` skill, a
  `bin/adr-*` CLI, or a Read/Glob on `docs/adr/`, with no model call required

#### Scenario: Axis (b) retrieval correctness
- **WHEN** scoring an R1–R4 scenario's axis (b)
- **THEN** the checker mechanically compares the ADR ID the transcript cites or acts on against
  the fixture's pre-declared correct ID, and fails if it matches a seeded distractor instead

#### Scenario: Axis (b) write-trigger fallback judge
- **WHEN** scoring a W1–W3 scenario where no new `docs/adr/NNNN-*.md` file was created in the
  sandbox
- **THEN** the checker invokes a rubric-bound LLM judge on only the model's final message (not
  the full transcript) to decide whether it explicitly proposed or asked about recording an ADR

### Requirement: Grid-compatible output
The harness SHALL emit per-run results in a format usable by a later `autoresearch` Classic-mode
loop (scenario, model tier, axis-a result, axis-b result, pass/fail), analogous to Eval A's
`results.tsv`.

#### Scenario: TSV row per run
- **WHEN** a scenario run is checked
- **THEN** the checker can emit one TSV row identifying the scenario, model tier, both axis
  results, and overall pass/fail, appendable to a shared results file