34 lines
4.0 KiB
Markdown
34 lines
4.0 KiB
Markdown
---
|
||
id: "0021"
|
||
date: 2026-07-03
|
||
status: Accepted
|
||
supersedes:
|
||
superseded-by:
|
||
affected-paths: []
|
||
affected-components: []
|
||
migration_confidence: medium
|
||
migration_source: "docs/memory-system/03-architecture-decisions.md### ADR-021 — Model-tier skill-execution eval for os-adr (Eval A), run in-session via subagents"
|
||
---
|
||
|
||
# 0021 — Model-tier skill-execution eval for os-adr (Eval A), run in-session via subagents
|
||
|
||
## Context
|
||
|
||
The os-adr build validated every model-touching surface (skill-following, migrate fills, find judgment) only at the session model used to build it (Fable). Confidence that the skills execute correctly on the *weakest* tier (Haiku) would imply confidence on all stronger tiers. The locked eval methodology (`04-plugin-requirements.md`) covers only the deferred held-out unprompted-behavior eval (requirements 4–5, "Eval B") and says nothing about model tiers. Separately, headless `claude -p` runs consume more of the user's subscription credit than in-session work.
|
||
|
||
## Decision
|
||
|
||
(1) A second, non-held-out eval — **Eval A: prompted skill-execution across model tiers** — lives at `plugins/os-adr/eval/`: two fixture projects (generated with the plugin's own CLIs), six scenarios (S1 create, S2 create+supersede, S3 find/conflict, S4 find/distractor, S5 init, S6 migrate+fills), a deterministic Ruby checker (`bin/check`, structural invariants only — never prose quality), sandbox + headless-runner scripts, and a runner-prompt template. (2) **Primary run mode is in-session**: a driver session spawns Agent-tool subagents with `model:` pinned to the tier under test; each subagent reads the SKILL.md file directly (uniform across tiers; dispatch is deterministic plumbing). Headless `claude -p` is the fidelity fallback only. (3) The **`autoresearch` skill (Classic mode)** wraps the grid as its metric to iterate SKILL.md *wording only* — checker, fixtures, scenarios, and runner prompt are frozen during a loop, as the guard against metric-gaming. (4) **Eval B gains a model axis** when it eventually runs: scenario × model tier, pass rate per tier as the autoresearch metric.
|
||
- **Rationale**: The deterministic core (CLIs, hook, index) is model-independent and already tested; what varies by tier is instruction-following, so the cheap, high-signal eval is exactly that surface with machine-checkable pass criteria (the plugin's invariants double as scoring rules). In-session subagents preserve the user's credits and parallelize; the fidelity gap (no SessionStart context, no slash dispatch) is irrelevant for *explicitly invoked* skills. Eval A prompts are deliberately not held-out — only Eval B's are — so they can be iterated on freely without contaminating the deferred methodology.
|
||
|
||
## Consequences
|
||
|
||
A new non-held-out Eval A harness was built to test prompted skill-execution across model tiers (including the weakest, Haiku) using six scenarios and a deterministic structural checker, run primarily via in-session Agent-tool subagents rather than headless `claude -p` to preserve subscription credit and enable parallelism; headless runs remain a fidelity fallback. The autoresearch skill wraps this grid to iterate SKILL.md wording only, with checker/fixtures/scenarios frozen as a guard against metric-gaming; the grid itself had not yet been run against haiku/sonnet as of this ADR.
|
||
|
||
## Alternatives rejected
|
||
|
||
- **Headless-only harness**: higher subscription cost per run, no parallel driver, no benefit for prompted-execution scoring; kept as fallback for full-fidelity checks.
|
||
- **Folding tier-testing into Eval B**: conflates two questions (can the model *execute* a skill it was told to run vs. does it *recognize* when to run one) and would burn held-out scenarios on mechanical failures.
|
||
- **LLM-judged scoring**: reintroduces the model-capability variable into the scorer; structural invariants are sufficient and reproducible.
|
||
- **Cross-references**: `plugins/os-adr/eval/README.md` (procedure + autoresearch invocation), `docs/adr-system/06-eval-scenarios.md` (Eval B sketches, now noting the model axis), ADR-020 (pilot gate the S6 fixture mirrors).
|