4.0 KiB
| id | date | status | supersedes | superseded-by | affected-paths | affected-components | migration_confidence | migration_source |
|---|---|---|---|---|---|---|---|---|
| 0021 | 2026-07-03 | Accepted | medium | docs/memory-system/03-architecture-decisions.md### ADR-021 — Model-tier skill-execution eval for os-adr (Eval A), run in-session via subagents |
0021 — Model-tier skill-execution eval for os-adr (Eval A), run in-session via subagents
Context
The os-adr build validated every model-touching surface (skill-following, migrate fills, find judgment) only at the session model used to build it (Fable). Confidence that the skills execute correctly on the weakest tier (Haiku) would imply confidence on all stronger tiers. The locked eval methodology (04-plugin-requirements.md) covers only the deferred held-out unprompted-behavior eval (requirements 4–5, "Eval B") and says nothing about model tiers. Separately, headless claude -p runs consume more of the user's subscription credit than in-session work.
Decision
(1) A second, non-held-out eval — Eval A: prompted skill-execution across model tiers — lives at plugins/os-adr/eval/: two fixture projects (generated with the plugin's own CLIs), six scenarios (S1 create, S2 create+supersede, S3 find/conflict, S4 find/distractor, S5 init, S6 migrate+fills), a deterministic Ruby checker (bin/check, structural invariants only — never prose quality), sandbox + headless-runner scripts, and a runner-prompt template. (2) Primary run mode is in-session: a driver session spawns Agent-tool subagents with model: pinned to the tier under test; each subagent reads the SKILL.md file directly (uniform across tiers; dispatch is deterministic plumbing). Headless claude -p is the fidelity fallback only. (3) The autoresearch skill (Classic mode) wraps the grid as its metric to iterate SKILL.md wording only — checker, fixtures, scenarios, and runner prompt are frozen during a loop, as the guard against metric-gaming. (4) Eval B gains a model axis when it eventually runs: scenario × model tier, pass rate per tier as the autoresearch metric.
- Rationale: The deterministic core (CLIs, hook, index) is model-independent and already tested; what varies by tier is instruction-following, so the cheap, high-signal eval is exactly that surface with machine-checkable pass criteria (the plugin's invariants double as scoring rules). In-session subagents preserve the user's credits and parallelize; the fidelity gap (no SessionStart context, no slash dispatch) is irrelevant for explicitly invoked skills. Eval A prompts are deliberately not held-out — only Eval B's are — so they can be iterated on freely without contaminating the deferred methodology.
Consequences
A new non-held-out Eval A harness was built to test prompted skill-execution across model tiers (including the weakest, Haiku) using six scenarios and a deterministic structural checker, run primarily via in-session Agent-tool subagents rather than headless claude -p to preserve subscription credit and enable parallelism; headless runs remain a fidelity fallback. The autoresearch skill wraps this grid to iterate SKILL.md wording only, with checker/fixtures/scenarios frozen as a guard against metric-gaming; the grid itself had not yet been run against haiku/sonnet as of this ADR.
Alternatives rejected
- Headless-only harness: higher subscription cost per run, no parallel driver, no benefit for prompted-execution scoring; kept as fallback for full-fidelity checks.
- Folding tier-testing into Eval B: conflates two questions (can the model execute a skill it was told to run vs. does it recognize when to run one) and would burn held-out scenarios on mechanical failures.
- LLM-judged scoring: reintroduces the model-capability variable into the scorer; structural invariants are sufficient and reproducible.
- Cross-references:
plugins/os-adr/eval/README.md(procedure + autoresearch invocation),docs/adr-system/06-eval-scenarios.md(Eval B sketches, now noting the model axis), ADR-020 (pilot gate the S6 fixture mirrors).