84 lines
4.5 KiB
Markdown
84 lines
4.5 KiB
Markdown
# os-adr Eval A — prompted skill-execution across model tiers
|
||
|
||
_Last updated: 2026-07-03 — harness built and self-tested (perfect-run PASS, untouched-sandbox
|
||
FAIL, for all six scenarios)._
|
||
|
||
Measures whether a **given model tier** (haiku, sonnet, …) can correctly execute the os-adr
|
||
skills when explicitly invoked. This is **not** the held-out unprompted-behavior eval
|
||
(requirements 4–5, "Eval B") — those scenarios stay deferred and must not be run informally
|
||
(see `docs/adr-system/06-eval-scenarios.md`). Eval A prompts are not held-out; they may be
|
||
read, discussed, and iterated against freely.
|
||
|
||
**Metric:** pass rate over the scenario × model grid, scored by the deterministic checker.
|
||
The model-sensitive surface under test is skill-*following* (reading SKILL.md and doing what
|
||
it says: right CLI, right args, right judgment calls). Skill *dispatch*, the CLIs, hook, and
|
||
index are deterministic and covered by the plugin's own 62 tests.
|
||
|
||
## Layout
|
||
|
||
| Path | What |
|
||
| --- | --- |
|
||
| `fixture/project/` | 4-ADR project (incl. a superseded pair) — scenarios S1–S4 |
|
||
| `fixture/legacy-project/` | monolithic `DECISIONS.md`, no `docs/adr/` — scenarios S5–S6 |
|
||
| `scenarios/S1..S6.md` | task prompt (verbatim block) + what the checker asserts |
|
||
| `bin/sandbox <Sn> <dest>` | fresh git-initialized sandbox copy of the right fixture |
|
||
| `bin/check <Sn> <sandbox> [--tsv <model>]` | deterministic scorer; exit 0/1; TSV mode for autoresearch |
|
||
| `runner-prompt.md` | the prompt template for the model under test |
|
||
| `bin/run-headless <Sn> <model> <workdir>` | `claude -p` fallback runner (costlier) |
|
||
|
||
Scenarios: S1 create · S2 create+supersede · S3 find/conflict · S4 find/distractor ·
|
||
S5 init · S6 migrate+fills. S3/S4 need the model's final answer saved to `<sandbox>/ANSWER.md`
|
||
(the runner prompt instructs the model to write it; if a subagent only returns text, the
|
||
driver saves that text to `ANSWER.md` before checking).
|
||
|
||
## Running the grid in-session (preferred — cheaper than `claude -p`)
|
||
|
||
A driver session (any model) runs, for each cell of {S1..S6} × {haiku, sonnet}:
|
||
|
||
1. `eval/bin/sandbox S3 /tmp/adr-eval/S3-haiku-r1`
|
||
2. Render `runner-prompt.md` (`{{SANDBOX}}`, `{{PLUGIN_ROOT}}` = the plugin dir,
|
||
`{{SKILL}}`, `{{TASK}}` = the scenario file's Task block) and spawn it as an **Agent tool
|
||
subagent with `model:` pinned to the tier under test**. Spawn independent cells in parallel.
|
||
3. Ensure `ANSWER.md` exists (write the subagent's returned text there if it didn't).
|
||
4. `eval/bin/check S3 /tmp/adr-eval/S3-haiku-r1 --tsv haiku >> results.tsv`
|
||
|
||
12 cells ≈ 12 subagent runs per round. Pass rate = `grep -c PASS results.tsv` / total.
|
||
|
||
**Fidelity caveats vs a real interactive session** (accepted for Eval A): no SessionStart
|
||
hook context (irrelevant — these are explicit invocations), no slash-command dispatch
|
||
(deterministic plumbing), subagent system prompt differs slightly from an interactive session.
|
||
The headless runner exists when full fidelity matters.
|
||
|
||
## Optimizing with `/autoresearch` (Classic mode)
|
||
|
||
Invoke in a driver session at the cc-os root:
|
||
|
||
```
|
||
/autoresearch
|
||
Goal: raise the os-adr skill-execution pass rate on weak models by tightening SKILL.md wording
|
||
Scope: plugins/os-adr/skills/*/SKILL.md ONLY — never edit bin/, lib/, tests/, or anything under eval/
|
||
Metric: pass rate over the {S1..S6} x {haiku, sonnet} grid via plugins/os-adr/eval/bin/check --tsv
|
||
Verify: run the in-session grid per plugins/os-adr/eval/README.md, append all 12 TSV lines to the
|
||
round's results.tsv, report pass rate and per-cell failures
|
||
Iterations: 5
|
||
```
|
||
|
||
Ground rules the loop must respect:
|
||
|
||
- **Scope is the guard against metric-gaming**: the checker, fixtures, scenarios, and runner
|
||
prompt are frozen during a loop. If a scenario or checker turns out to be wrong, stop the
|
||
loop and fix it as a separate, human-reviewed change.
|
||
- Keep/discard on the **haiku** column first — a wording change that helps haiku and is
|
||
neutral for sonnet is a keep; one that regresses sonnet is a discard.
|
||
- Failures in `results.tsv` column 4 name the exact broken invariant (e.g. "0001 status
|
||
... != Superseded") — feed that string into the next modify step.
|
||
- SKILL.md edits must not contradict the plugin's `invariants.md` (e.g. never suggest
|
||
hand-editing the index). Run `ruby tests/all.rb` after each accepted edit as a regression
|
||
gate.
|
||
|
||
## Adding a scenario
|
||
|
||
Add `scenarios/S7.md` (Task block + criteria), a `S7` class in `bin/check`, the fixture route
|
||
in `bin/sandbox`, then self-test both directions: simulate a perfect run with the CLIs
|
||
(must PASS) and check an untouched sandbox (must FAIL).
|