128 lines
8.6 KiB
Markdown
128 lines
8.6 KiB
Markdown
## Context
|
||
|
||
`os-adr` ships with `docs/adr-system/04-plugin-requirements.md` requirements 4–5: the plugin
|
||
must be effective at *unprompted* write-trigger recognition and *unprompted, correct* retrieval.
|
||
Eval A (`plugins/os-adr/eval/`) only measures skill-*following* once explicitly invoked — its
|
||
runner-prompt literally tells the model under test "the user has invoked `/os-adr:{{SKILL}}`".
|
||
That architecture cannot answer requirements 4–5 by construction: the thing being measured
|
||
(does the model notice on its own) is exactly what the runner prompt would be giving away.
|
||
|
||
Eval A's driver used in-session Agent-tool subagents pinned to a model tier, and treated the
|
||
missing SessionStart hook context as an acceptable fidelity gap ("irrelevant — these are
|
||
explicit invocations"). For Eval B that gap is not acceptable: the SessionStart hook's existence
|
||
note (naming `/os-adr:create` and `/os-adr:find`) is one of the two legitimate discovery paths
|
||
being measured (the other being the model reasoning its own way there without any hook hint,
|
||
e.g. via `docs/adr/` conventionally existing). If the hook never fires, we can't tell "the model
|
||
didn't notice" from "the model was never given the hint a real session would have."
|
||
|
||
Scenario shapes are frozen and out of scope here (`docs/adr-system/06-eval-scenarios.md`, 7
|
||
scenarios: W1–W3 write-trigger, R1–R4 retrieval) — this design is about the execution harness
|
||
only.
|
||
|
||
## Goals / Non-Goals
|
||
|
||
**Goals:**
|
||
- A runner that executes the model-under-test in an environment where the real SessionStart
|
||
hook fires, real skills/CLIs are reachable, and no part of the prompt names the ADR system.
|
||
- Fixtures realistic enough to support all 7 scenarios, including R2's distractor ADRs (2–3
|
||
near-miss, one Superseded) and R4's one-hop graph reachability (needs a real `graphify-out/`).
|
||
- A checker that scores both axes (unprompted-consultation, correctness) with as much
|
||
determinism as possible, reserving LLM judgment only for genuinely ambiguous cases (e.g. did
|
||
the model's prose count as "proposing" an ADR).
|
||
- Sandbox isolation so headless runs never mutate the canonical fixture or the real cc-os repo.
|
||
- Output format (TSV or similar) compatible with a later `autoresearch` Classic-mode loop.
|
||
|
||
**Non-Goals:**
|
||
- Running the grid, scoring a real pass rate, or iterating on retrieval/trigger heuristics —
|
||
that is the deferred next stage, not this change.
|
||
- Adding, removing, or reshaping scenario shapes in `06-eval-scenarios.md`.
|
||
- Multi-machine or CI execution — local headless runs only, matching Eval A's scope.
|
||
|
||
## Decisions
|
||
|
||
**1. Headless (`claude -p`) is the sole runner mode; no subagent fallback.**
|
||
Eval A offers both an in-session subagent path (cheaper, preferred) and a headless fallback
|
||
(`bin/run-headless`) for full fidelity. Eval B inverts that: only headless mode is valid, because
|
||
Agent-tool subagents share the parent session's already-loaded plugin/hook state and cwd
|
||
semantics in ways that don't reproduce a fresh SessionStart firing against the sandbox directory.
|
||
`claude -p --output-format stream-json --verbose --model <tier> --cwd <sandbox>` gives a fresh
|
||
process, a real SessionStart event, and a full transcript to check against.
|
||
*Alternative considered*: reuse Eval A's subagent shortcut for cost — rejected, since it would
|
||
silently reintroduce the exact fidelity gap this eval exists to close.
|
||
|
||
**2. Transcript-based mechanical check for axis (a) (unprompted consultation).**
|
||
`--output-format stream-json --verbose` emits every tool_use block. The checker greps the JSONL
|
||
for: a Skill/slash invocation matching `os-adr:*`, a `Bash` call matching `bin/adr-*`, or a
|
||
`Read`/`Glob` touching `docs/adr/`. This is fully mechanical — no model judgment needed to
|
||
detect "did it touch the ADR system at all."
|
||
*Alternative considered*: ask an LLM judge to read the transcript and decide — rejected for axis
|
||
(a) specifically, since tool-call presence is unambiguous and cheaper to check by grep.
|
||
|
||
**3. Axis (b) (correctness) is deterministic where possible, LLM-judged only at the edges.**
|
||
For R1–R4 (retrieval), the fixture encodes the single correct ADR ID; the checker mechanically
|
||
confirms the transcript's retrieved/cited ADR ID matches (or, for R2, that it is NOT one of the
|
||
seeded distractor IDs). For W1–W3 (write-trigger), "proposes... or asks whether to" is prose-
|
||
shaped and not fully mechanical — a new `docs/adr/NNNN-*.md` file appearing in the sandbox is a
|
||
mechanical PASS; absent that, a small rubric-bound LLM judge reads only the final transcript
|
||
message (not the full transcript) and answers a yes/no: "did the agent explicitly propose
|
||
recording this decision as an ADR, or ask whether to?" Kept as thin as possible to limit judgment
|
||
surface, mirroring the `os-doc-hygiene`/`os-adr` classify-only-the-ambiguous-part pattern already
|
||
used elsewhere in this repo.
|
||
*Alternative considered*: fully mechanical W1–W3 checking via keyword grep on the final message
|
||
(e.g. "ADR") — rejected as too brittle/gameable; a judge with a narrow, fixed rubric is more
|
||
robust and still cheap since it reads one short message, not a transcript.
|
||
|
||
**4. Dedicated Eval-B fixture project(s), not reuse of Eval A's fixtures or the real llf-schema
|
||
pilot.**
|
||
Eval A's fixtures are too small (4 ADRs, no graph) and R2/R4 need specific engineered content
|
||
(a Superseded pair plus 2–3 near-misses; a real one-hop-only conflict via Graphify). Building
|
||
`plugins/os-adr/eval-b/fixture/` fresh, with `docs/adr/` populated per-scenario and a genuinely
|
||
built `graphify-out/` (via the real `graphify` binary against the fixture's own small codebase,
|
||
not a hand-authored stub) keeps the graph-layer degradation check (R4) honest.
|
||
*Alternative considered*: reuse the llf-schema pilot project directly — rejected; its exclude
|
||
list and ADR set are project-specific and not shaped for the distractor/one-hop scenarios,
|
||
and would tie the eval's stability to project state a migration exercise might later change.
|
||
|
||
**5. Per-cell repeat count is a runner flag, default left as an open question for the eval-
|
||
running stage, not fixed by this design.**
|
||
Unprompted behavior is more stochastic than the explicit-invocation behavior Eval A measures.
|
||
The harness supports `--reps N` on the sandbox/runner tooling, but this change does not decide
|
||
whether the eventual grid uses N=1 or N=3 — that's a call for whoever runs the eval, informed by
|
||
observed variance in a pilot run.
|
||
|
||
## Risks / Trade-offs
|
||
|
||
- **[Risk] Headless runs perform real, uncontained tool use (file edits, possibly destructive
|
||
commands) against the sandbox.** → Mitigation: every run is a fresh git-initialized sandbox
|
||
copy (mirrors Eval A's `bin/sandbox` pattern) under a scratch/tmp root, never the canonical
|
||
fixture or the cc-os repo; sandbox is disposable and diffed/discarded after checking.
|
||
- **[Risk] Cost — headless full-session runs are pricier than Eval A's subagent shortcut.** →
|
||
Mitigation: accepted deliberately (see Decision 1); this is the whole reason the fidelity gap
|
||
matters here. Reps default is left open (Decision 5) specifically to let the eval-running
|
||
stage control cost/variance trade-off with real data instead of a guess baked in now.
|
||
- **[Risk] LLM-judged axis (b) for W1–W3 could itself be inconsistent or gameable over
|
||
`autoresearch` iterations.** → Mitigation: judge rubric and prompt are frozen alongside the
|
||
checker/fixtures/scenarios during any future `autoresearch` loop (same discipline Eval A's
|
||
README already imposes on its own checker), and Decision 3 keeps the judge's input surface
|
||
(one final message, not a transcript) as narrow as possible.
|
||
- **[Risk] A genuinely-built `graphify-out/` for the fixture could drift if `graphify`'s output
|
||
format changes.** → Mitigation: the fixture's `graphify-out/` is rebuilt by an explicit fixture
|
||
setup step (documented in the harness README), not committed as a frozen binary blob, so it's
|
||
always regenerated against whatever `graphify` version is installed.
|
||
|
||
## Migration Plan
|
||
|
||
Not applicable — this is a new, additive test-harness directory with no effect on shipped plugin
|
||
behavior. No rollback beyond deleting `plugins/os-adr/eval-b/` if abandoned.
|
||
|
||
## Open Questions
|
||
|
||
- Default `--reps` per cell for the eventual grid run (left to the eval-running stage; see
|
||
Decision 5).
|
||
- Whether the R4 degradation check (retrieval must fail without `graphify-out/`) needs its own
|
||
sandbox variant with the graph directory deliberately removed, or whether one fixture toggling
|
||
a flag suffices — resolve during tasks/implementation, not architecturally significant enough
|
||
to block this design.
|
||
- Exact rubric wording for the W1–W3 LLM judge — drafted during implementation (tasks.md), not
|
||
fixed here.
|