cc-os/openspec/specs/adr-eval-b-harness/spec.md

Purpose

Evaluation harness B: held-out, unprompted-behavior testing for the os-adr plugin. Tests whether the plugin's core write/retrieval operations surface naturally without coaching, across multiple model tiers, with deterministic-first scoring (mechanical checks before fallback to LLM judge).

Requirements

Requirement: Held-out scenario fixtures

The harness SHALL provide a dedicated fixture project (or projects) under plugins/os-adr/eval-b/fixture/ that supports all 7 held-out scenarios (W1–W3, R1–R4) without any scenario prompt naming the ADR system, an ADR ID, or the exact constraint text being tested.

Scenario: R2 distractor set present

• WHEN the R2 fixture's docs/adr/ is inspected
• THEN it contains the one correct Accepted ADR plus 2–3 near-miss ADRs (same component family, different decision, or Superseded status) that a naive retrieval could mistakenly cite

Scenario: R4 graph reachability is real, not stubbed

• WHEN the R4 fixture's graphify-out/ is inspected
• THEN it was produced by running the real graphify binary against the fixture's own codebase (not hand-authored), and the conflicting files are one import/reference hop away from files listed in the relevant ADR's affected-paths, not directly listed themselves

Requirement: Headless runner with real hook firing

The harness SHALL execute each scenario via a headless claude -p process with its working directory set to a fresh sandbox copy of the fixture, so that the real SessionStart hook fires for the model under test. In-session Agent-tool subagents SHALL NOT be used as the execution mode for this eval.

Scenario: SessionStart hook context reaches the model under test

• WHEN a scenario is run via the headless runner against an ADR-initialized sandbox
• THEN the transcript shows the SessionStart hook's additionalContext was present in the model's context before its first action

Scenario: Sandbox isolation

• WHEN any scenario run completes (pass or fail)
• THEN the canonical fixture directory and the cc-os repo itself show no modifications — only the disposable sandbox copy was touched

Requirement: Two-axis deterministic-first checker

The harness SHALL score each run on two independent axes: (a) whether the model consulted or proposed the ADR system at all, unprompted, and (b) whether it acted on or wrote the specific correct thing rather than a merely plausible one. Axis (a), and axis (b) for retrieval scenarios, SHALL be scored mechanically from the transcript and sandbox file state; axis (b) for write-trigger scenarios MAY fall back to a narrow, rubric-bound LLM judge only when no new ADR file was mechanically created.

Scenario: Axis (a) mechanical detection

• WHEN the checker scans a scenario's stream-json transcript
• THEN it detects axis (a) as true if any tool_use block invokes an os-adr:* skill, a bin/adr-* CLI, or a Read/Glob on docs/adr/, with no model call required

Scenario: Axis (b) retrieval correctness

• WHEN scoring an R1–R4 scenario's axis (b)
• THEN the checker mechanically compares the ADR ID the transcript cites or acts on against the fixture's pre-declared correct ID, and fails if it matches a seeded distractor instead

Scenario: Axis (b) write-trigger fallback judge

• WHEN scoring a W1–W3 scenario where no new docs/adr/NNNN-*.md file was created in the sandbox
• THEN the checker invokes a rubric-bound LLM judge on only the model's final message (not the full transcript) to decide whether it explicitly proposed or asked about recording an ADR

Requirement: Grid-compatible output

The harness SHALL emit per-run results in a format usable by a later autoresearch Classic-mode loop (scenario, model tier, axis-a result, axis-b result, pass/fail), analogous to Eval A's results.tsv.

Scenario: TSV row per run

• WHEN a scenario run is checked
• THEN the checker can emit one TSV row identifying the scenario, model tier, both axis results, and overall pass/fail, appendable to a shared results file