cc-os/plugins/os-adr/eval/README.md

os-adr Eval A — prompted skill-execution across model tiers

Last updated: 2026-07-03 — harness built and self-tested (perfect-run PASS, untouched-sandbox FAIL, for all six scenarios).

Measures whether a given model tier (haiku, sonnet, …) can correctly execute the os-adr skills when explicitly invoked. This is not the held-out unprompted-behavior eval (requirements 4–5, "Eval B") — those scenarios stay deferred and must not be run informally (see docs/adr-system/06-eval-scenarios.md). Eval A prompts are not held-out; they may be read, discussed, and iterated against freely.

Metric: pass rate over the scenario × model grid, scored by the deterministic checker. The model-sensitive surface under test is skill-following (reading SKILL.md and doing what it says: right CLI, right args, right judgment calls). Skill dispatch, the CLIs, hook, and index are deterministic and covered by the plugin's own 62 tests.

Layout

Path	What
fixture/project/	4-ADR project (incl. a superseded pair) — scenarios S1–S4
fixture/legacy-project/	monolithic DECISIONS.md, no docs/adr/ — scenarios S5–S6
scenarios/S1..S6.md	task prompt (verbatim block) + what the checker asserts
bin/sandbox <Sn> <dest>	fresh git-initialized sandbox copy of the right fixture
bin/check <Sn> <sandbox> [--tsv <model>]	deterministic scorer; exit 0/1; TSV mode for autoresearch
runner-prompt.md	the prompt template for the model under test
bin/run-headless <Sn> <model> <workdir>	claude -p fallback runner (costlier)

Scenarios: S1 create · S2 create+supersede · S3 find/conflict · S4 find/distractor · S5 init · S6 migrate+fills. S3/S4 need the model's final answer saved to <sandbox>/ANSWER.md (the runner prompt instructs the model to write it; if a subagent only returns text, the driver saves that text to ANSWER.md before checking).

Running the grid in-session (preferred — cheaper than claude -p)

A driver session (any model) runs, for each cell of {S1..S6} × {haiku, sonnet}:

1. eval/bin/sandbox S3 /tmp/adr-eval/S3-haiku-r1
2. Render runner-prompt.md ({{SANDBOX}}, {{PLUGIN_ROOT}} = the plugin dir, {{SKILL}}, {{TASK}} = the scenario file's Task block) and spawn it as an Agent tool subagent with model: pinned to the tier under test. Spawn independent cells in parallel.
3. Ensure ANSWER.md exists (write the subagent's returned text there if it didn't).
4. eval/bin/check S3 /tmp/adr-eval/S3-haiku-r1 --tsv haiku >> results.tsv

12 cells ≈ 12 subagent runs per round. Pass rate = grep -c PASS results.tsv / total.

Fidelity caveats vs a real interactive session (accepted for Eval A): no SessionStart hook context (irrelevant — these are explicit invocations), no slash-command dispatch (deterministic plumbing), subagent system prompt differs slightly from an interactive session. The headless runner exists when full fidelity matters.

Optimizing with /autoresearch (Classic mode)

Invoke in a driver session at the cc-os root:

/autoresearch
Goal: raise the os-adr skill-execution pass rate on weak models by tightening SKILL.md wording
Scope: plugins/os-adr/skills/*/SKILL.md ONLY — never edit bin/, lib/, tests/, or anything under eval/
Metric: pass rate over the {S1..S6} x {haiku, sonnet} grid via plugins/os-adr/eval/bin/check --tsv
Verify: run the in-session grid per plugins/os-adr/eval/README.md, append all 12 TSV lines to the
  round's results.tsv, report pass rate and per-cell failures
Iterations: 5

Ground rules the loop must respect:

• Scope is the guard against metric-gaming: the checker, fixtures, scenarios, and runner prompt are frozen during a loop. If a scenario or checker turns out to be wrong, stop the loop and fix it as a separate, human-reviewed change.
• Keep/discard on the haiku column first — a wording change that helps haiku and is neutral for sonnet is a keep; one that regresses sonnet is a discard.
• Failures in results.tsv column 4 name the exact broken invariant (e.g. "0001 status ... != Superseded") — feed that string into the next modify step.
• SKILL.md edits must not contradict the plugin's invariants.md (e.g. never suggest hand-editing the index). Run ruby tests/all.rb after each accepted edit as a regression gate.

Adding a scenario

Add scenarios/S7.md (Task block + criteria), a S7 class in bin/check, the fixture route in bin/sandbox, then self-test both directions: simulate a perfect run with the CLIs (must PASS) and check an untouched sandbox (must FAIL).