# os-adr Eval A — prompted skill-execution across model tiers

_Last updated: 2026-07-03 — harness built and self-tested (perfect-run PASS, untouched-sandbox
FAIL, for all six scenarios)._

Measures whether a **given model tier** (haiku, sonnet, …) can correctly execute the os-adr
skills when explicitly invoked. This is **not** the held-out unprompted-behavior eval
(requirements 4–5, "Eval B") — those scenarios stay deferred and must not be run informally
(see `docs/adr-system/06-eval-scenarios.md`). Eval A prompts are not held-out; they may be
read, discussed, and iterated against freely.

**Metric:** pass rate over the scenario × model grid, scored by the deterministic checker.
The model-sensitive surface under test is skill-*following* (reading SKILL.md and doing what
it says: right CLI, right args, right judgment calls). Skill *dispatch*, the CLIs, hook, and
index are deterministic and covered by the plugin's own 62 tests.

## Layout

| Path | What |
| --- | --- |
| `fixture/project/` | 4-ADR project (incl. a superseded pair) — scenarios S1–S4 |
| `fixture/legacy-project/` | monolithic `DECISIONS.md`, no `docs/adr/` — scenarios S5–S6 |
| `scenarios/S1..S6.md` | task prompt (verbatim block) + what the checker asserts |
| `bin/sandbox <Sn> <dest>` | fresh git-initialized sandbox copy of the right fixture |
| `bin/check <Sn> <sandbox> [--tsv <model>]` | deterministic scorer; exit 0/1; TSV mode for autoresearch |
| `runner-prompt.md` | the prompt template for the model under test |
| `bin/run-headless <Sn> <model> <workdir>` | `claude -p` fallback runner (costlier) |

Scenarios: S1 create · S2 create+supersede · S3 find/conflict · S4 find/distractor ·
S5 init · S6 migrate+fills. S3/S4 need the model's final answer saved to `<sandbox>/ANSWER.md`
(the runner prompt instructs the model to write it; if a subagent only returns text, the
driver saves that text to `ANSWER.md` before checking).

## Running the grid in-session (preferred — cheaper than `claude -p`)

A driver session (any model) runs, for each cell of {S1..S6} × {haiku, sonnet}:

1. `eval/bin/sandbox S3 /tmp/adr-eval/S3-haiku-r1`
2. Render `runner-prompt.md` (`{{SANDBOX}}`, `{{PLUGIN_ROOT}}` = the plugin dir,
   `{{SKILL}}`, `{{TASK}}` = the scenario file's Task block) and spawn it as an **Agent tool
   subagent with `model:` pinned to the tier under test**. Spawn independent cells in parallel.
3. Ensure `ANSWER.md` exists (write the subagent's returned text there if it didn't).
4. `eval/bin/check S3 /tmp/adr-eval/S3-haiku-r1 --tsv haiku >> results.tsv`

12 cells ≈ 12 subagent runs per round. Pass rate = `grep -c PASS results.tsv` / total.

**Fidelity caveats vs a real interactive session** (accepted for Eval A): no SessionStart
hook context (irrelevant — these are explicit invocations), no slash-command dispatch
(deterministic plumbing), subagent system prompt differs slightly from an interactive session.
The headless runner exists when full fidelity matters.

## Optimizing with `/autoresearch` (Classic mode)

Invoke in a driver session at the cc-os root:

```
/autoresearch
Goal: raise the os-adr skill-execution pass rate on weak models by tightening SKILL.md wording
Scope: plugins/os-adr/skills/*/SKILL.md ONLY — never edit bin/, lib/, tests/, or anything under eval/
Metric: pass rate over the {S1..S6} x {haiku, sonnet} grid via plugins/os-adr/eval/bin/check --tsv
Verify: run the in-session grid per plugins/os-adr/eval/README.md, append all 12 TSV lines to the
  round's results.tsv, report pass rate and per-cell failures
Iterations: 5
```

Ground rules the loop must respect:

- **Scope is the guard against metric-gaming**: the checker, fixtures, scenarios, and runner
  prompt are frozen during a loop. If a scenario or checker turns out to be wrong, stop the
  loop and fix it as a separate, human-reviewed change.
- Keep/discard on the **haiku** column first — a wording change that helps haiku and is
  neutral for sonnet is a keep; one that regresses sonnet is a discard.
- Failures in `results.tsv` column 4 name the exact broken invariant (e.g. "0001 status
  ... != Superseded") — feed that string into the next modify step.
- SKILL.md edits must not contradict the plugin's `invariants.md` (e.g. never suggest
  hand-editing the index). Run `ruby tests/all.rb` after each accepted edit as a regression
  gate.

## Adding a scenario

Add `scenarios/S7.md` (Task block + criteria), a `S7` class in `bin/check`, the fixture route
in `bin/sandbox`, then self-test both directions: simulate a perfect run with the CLIs
(must PASS) and check an untouched sandbox (must FAIL).