55 lines
3.3 KiB
Markdown
55 lines
3.3 KiB
Markdown
|
## Why
|
|||
|
|
|
|||
|
|
The `os-adr` plugin's two hardest requirements — unprompted write-trigger recognition and
|
|||
|
|
unprompted correct retrieval (`04-plugin-requirements.md` reqs 4–5) — have never been measured.
|
|||
|
|
Eval A (`plugins/os-adr/eval/`) only proves the plugin executes correctly once explicitly
|
|||
|
|
invoked; it says nothing about whether an agent notices, on its own, that the ADR system is
|
|||
|
|
relevant. The scenario *shapes* for this held-out eval already exist
|
|||
|
|
(`docs/adr-system/06-eval-scenarios.md`) but no fixtures, prompts, checker, or runner exist yet.
|
|||
|
|
Building it now closes the last unmeasured requirement in the locked rollout order.
|
|||
|
|
|
|||
|
|
## What Changes
|
|||
|
|
|
|||
|
|
- Build a new eval harness, `plugins/os-adr/eval-b/`, sibling to but structurally distinct from
|
|||
|
|
Eval A, for the 7 held-out scenarios (W1–W3 write-trigger, R1–R4 retrieval).
|
|||
|
|
- Author full scenario prompts/fixtures from the frozen shapes in `06-eval-scenarios.md` —
|
|||
|
|
scenario *shapes* are not renegotiated by this change; only their execution artifacts are new.
|
|||
|
|
- Fixture project(s): a realistic, onboarded project state (populated `docs/adr/` incl. a
|
|||
|
|
Superseded pair for R2 distractors, a built `graphify-out/` for R4) that the SessionStart hook
|
|||
|
|
recognizes as ADR-initialized.
|
|||
|
|
- A **headless runner** (real `claude -p` in a real sandbox cwd) as the primary execution mode,
|
|||
|
|
not an in-session Agent-tool subagent — because the SessionStart hook firing for real is part
|
|||
|
|
of what's under test (Eval A's subagent shortcut is not valid here; see design.md).
|
|||
|
|
- A deterministic-first, two-axis checker: (a) did the agent consult/propose the ADR system
|
|||
|
|
unprompted at all, (b) did it act on/write the *specific correct* thing. Axis (a) is
|
|||
|
|
mechanically checkable (tool-call/transcript grep for `/os-adr:*` or `docs/adr/` reads/writes);
|
|||
|
|
axis (b) requires a small LLM-judged correctness check against the fixture's known-correct
|
|||
|
|
answer, kept as thin and rubric-bound as possible.
|
|||
|
|
- Sandbox/grid tooling mirroring Eval A's shape (`bin/sandbox`, `bin/check`, results.tsv) adapted
|
|||
|
|
for headless execution and the two-axis score.
|
|||
|
|
- Wiring for the later `autoresearch` iteration stage (not run by this change — building the
|
|||
|
|
harness only).
|
|||
|
|
|
|||
|
|
Out of scope for this change: actually running the eval grid, iterating retrieval/trigger
|
|||
|
|
heuristics, or adding/altering scenario shapes in `06-eval-scenarios.md`.
|
|||
|
|
|
|||
|
|
## Capabilities
|
|||
|
|
|
|||
|
|
### New Capabilities
|
|||
|
|
- `adr-eval-b-harness`: fixtures, held-out scenario prompts, headless sandbox/runner tooling, and
|
|||
|
|
a two-axis deterministic-first checker for the unprompted write-trigger and retrieval eval.
|
|||
|
|
|
|||
|
|
### Modified Capabilities
|
|||
|
|
(none — this adds a new, separate eval surface; it does not change the behavior of
|
|||
|
|
`adr-authoring`, `adr-retrieval`, `adr-session-awareness`, or any shipped plugin capability)
|
|||
|
|
|
|||
|
|
## Impact
|
|||
|
|
|
|||
|
|
- New directory: `plugins/os-adr/eval-b/` (fixtures, scenarios, bin/, runner assets, README).
|
|||
|
|
- No changes to plugin source (`lib/adr/`, `bin/adr-*`, `hooks/`, `skills/`) — this is a pure
|
|||
|
|
test-harness addition.
|
|||
|
|
- Depends on `claude` CLI headless mode (`claude -p`) being available in the build/dev
|
|||
|
|
environment for the runner to execute against a real sandbox with hooks firing.
|
|||
|
|
- Once built, unlocks the deferred final stage of the locked os-adr rollout order (running the
|
|||
|
|
grid + `autoresearch` iteration), tracked separately in `CLAUDE.md`.
|