103 lines
6.4 KiB
Markdown
103 lines
6.4 KiB
Markdown
# os-adr Eval B — unprompted write-trigger & retrieval (held-out)
|
||
|
||
_Last updated: 2026-07-03 — harness built and self-tested (perfect-run PASS,
|
||
untouched-sandbox FAIL, both directions for all 7 scenarios). Grid run 2026-07-03, 1 rep/cell:
|
||
haiku 0/8 PASS (never unprompted-consults the ADR system in any scenario); sonnet 5/8 PASS
|
||
(fails W3 — doesn't propose recording the decision; fails R1 — misses the direct-conflict
|
||
retrieval). R4-nograph FAILed on both tiers as expected (degradation check; only meaningful
|
||
paired with an R4 PASS, which sonnet has and haiku doesn't). Wording iteration via
|
||
`/autoresearch` not yet started._
|
||
|
||
Measures the two hardest os-adr requirements (`docs/adr-system/04-plugin-requirements.md`
|
||
reqs 4–5): does an agent notice **on its own** that the ADR system is relevant — proposing to
|
||
record a consequential decision (W1–W3), or retrieving the specific correct decision before
|
||
changing decided-on behavior (R1–R4)? Scenario shapes are frozen in
|
||
`docs/adr-system/06-eval-scenarios.md`. This is **not** Eval A (`../eval/`), which measures
|
||
skill-*execution* once explicitly invoked — do not conflate them (ADR-021).
|
||
|
||
> **HELD-OUT — read this first.** The `## Task` blocks in `scenarios/*.md` must never be
|
||
> pasted into an interactive session, "tried out" informally, or fed to a model outside
|
||
> `bin/run`. The only permitted non-grid execution is `bin/self-test`, which fabricates
|
||
> transcripts and never uses the Task blocks. Informal trials contaminate the methodology.
|
||
|
||
## Layout
|
||
|
||
| Path | What |
|
||
| --- | --- |
|
||
| `fixture/project/` | small Ruby webhook-relay codebase, 6-ADR history (incl. a Superseded pair) |
|
||
| `scenarios/W1..W3.md` | write-trigger prompts (persistence choice, convention change, reversal) |
|
||
| `scenarios/R1..R4.md` | retrieval prompts (direct conflict, distractors, mid-task Q, graph hop) |
|
||
| `bin/build-fixture-graph` | rebuild `fixture/project/graphify-out/` from scratch (R4 needs it) |
|
||
| `bin/sandbox <Sn> <dest>` | fresh git-initialized sandbox; only `R4` keeps the graph; `R4-nograph` = degradation variant |
|
||
| `bin/run <Sn> <model> <workdir> [--reps N]` | headless `claude -p` runner; the ONLY valid execution mode |
|
||
| `bin/check <Sn> <sandbox> [--tsv <model>]` | two-axis checker; exit 0/1; TSV mode for autoresearch |
|
||
| `bin/self-test [workdir]` | model-free both-directions harness self-test |
|
||
| `judge-rubric.md` | frozen rubric for the W-scenario axis (b) LLM judge fallback |
|
||
|
||
## Why headless-only (no Agent-tool subagents)
|
||
|
||
The SessionStart hook's `[os-adr]` note is one of the two discovery paths under measurement.
|
||
In-session subagents don't get a fresh SessionStart against the sandbox cwd, so they can't
|
||
tell "the model didn't notice" from "the model never got the hint a real session would have."
|
||
`bin/run` gives each rep a fresh `claude -p` process with cwd set to a fresh sandbox; the
|
||
transcript's `system/hook_response` event proves the hook context reached the model — the
|
||
checker treats a transcript without it as an invalid run. (Verified 2026-07-03 with a neutral
|
||
non-scenario probe: haiku quoted the hook note verbatim from a sandbox session.)
|
||
|
||
## Two-axis scoring
|
||
|
||
- **Axis (a) — unprompted consultation (mechanical).** Any `tool_use` block in the
|
||
`stream-json` transcript that invokes an `os-adr:*` skill, a `bin/adr-*` CLI, or touches
|
||
`docs/adr/`. Prose never counts, so the hook note can't false-positive it.
|
||
- **Axis (b) — specific correctness.**
|
||
- R1–R4 (mechanical): assistant text must cite the scenario's pre-declared correct ADR ID
|
||
(R1→0002, R2→0003, R3→0005 content match [UTC + ISO-8601], R4→0003) and flag the
|
||
conflict. R2 fails if the Superseded distractor 0001 is cited as if live.
|
||
- W1–W3 (deterministic-first): PASS mechanically when a new `docs/adr/NNNN-*.md` matching
|
||
the scenario's topic exists; only otherwise does a rubric-bound judge (haiku,
|
||
`judge-rubric.md`, final message only) decide propose/ask-vs-silent. Stub it with
|
||
`ADR_EVAL_B_JUDGE_CMD` (reads prompt on stdin, prints YES/NO).
|
||
- **Overall PASS** = both axes PASS. TSV row: `scenario model A:… B:… PASS|FAIL reasons`.
|
||
|
||
## Running one cell
|
||
|
||
```bash
|
||
eval-b/bin/build-fixture-graph # once, before any R4 cell
|
||
eval-b/bin/run R2 haiku /tmp/adr-eval-b --reps 3 --results /tmp/adr-eval-b/results.tsv
|
||
```
|
||
|
||
Each rep: fresh sandbox → `claude -p --output-format stream-json --verbose --model <tier>
|
||
--dangerously-skip-permissions "<Task block>"` with cwd = sandbox → `transcript.jsonl` saved →
|
||
`bin/check` appends one TSV row. `--reps` default is 1; the grid-run stage picks the real
|
||
repeat count after observing variance in a pilot (design decision, left open deliberately).
|
||
|
||
The R4 degradation check: `bin/run R4-nograph <model> <workdir>` — same prompt, graph absent.
|
||
An axis (b) FAIL there is the **expected** outcome; it confirms R4's pass genuinely required
|
||
the Graphify layer (paired with an R4 PASS on the same tier).
|
||
|
||
## Fixture regeneration
|
||
|
||
`graphify-out/` is never committed — rebuild it with `bin/build-fixture-graph`, which runs the
|
||
real `graphify update` against the fixture (pure tree-sitter AST: the fixture's
|
||
`.graphifyignore` excludes all markdown, so no Ollama/LLM is involved and the build is
|
||
reproducible) and then asserts the R4 one-hop invariant (`lib/relay/reports.rb` is exactly one
|
||
graph hop from `lib/relay/delivery.rb`). If graphify's output format changes and the invariant
|
||
breaks, fix the fixture layout — not the checker — in a human-reviewed change.
|
||
|
||
The ADR history was generated with the plugin's own CLIs (`adr-init`/`adr-new`), so frontmatter
|
||
and index match the shipped format, including the mechanically-superseded 0001→0003 pair.
|
||
|
||
## Optimizing with `/autoresearch` (later stage — not part of building this harness)
|
||
|
||
Same discipline as Eval A: the **checker, fixtures, scenario prompts, and judge rubric are
|
||
frozen during a loop** — only plugin surface wording (SKILL.md, hook note text) may move. Metric:
|
||
pass rate over the {W1..W3, R1..R4} × {haiku, sonnet} grid via `bin/check --tsv`. If a scenario
|
||
or checker turns out to be wrong, stop the loop and fix it as a separate, human-reviewed change.
|
||
Unlike Eval A, every cell costs a real headless session — budget reps accordingly.
|
||
|
||
## Adding a scenario
|
||
|
||
Add `scenarios/X.md` (Task block + checker metadata), a class in `bin/check`, the fixture route
|
||
in `bin/sandbox`, extend `bin/self-test` both directions, and keep the held-out rule: the new
|
||
Task block gets no informal trial runs.
|