5.8 KiB
| summary | last_updated | tags | source | date | |||||
|---|---|---|---|---|---|---|---|---|---|
| os-adr Eval B (unprompted write-trigger & retrieval) grid results for haiku/sonnet — haiku never self-triggers the ADR system, sonnet passes 5/8 with two near-miss failures (confirmed by 2026-07-04 re-run) — read before designing any follow-up eval on unprompted-consultation behavior. | 2026-07-04 |
|
cc-os | 2026-07-03 |
os-adr Eval B grid results (2026-07-03)
Eval B measures whether a model notices on its own that an Architecture Decision Record
system is relevant — without being explicitly told to invoke it. Two axes per scenario: (a)
did the model touch the ADR system unprompted (mechanical, from tool_use), (b) was the specific
outcome correct (right ADR cited, or a new ADR actually proposed/created). See
cc-os/plugins/os-adr/eval-b/README.md for full harness design (headless-only claude -p runs,
so a real SessionStart hook fires — in-session Agent-tool subagents cannot validly stand in for
this, since they don't get a fresh hook invocation against the sandbox cwd).
Grid (1 rep/cell)
| Scenario | Haiku | Sonnet |
|---|---|---|
| W1 (persistence choice) | FAIL | PASS |
| W2 (convention change) | FAIL | PASS |
| W3 (reversal) | FAIL | FAIL — doesn't propose recording the decision |
| R1 (direct conflict) | FAIL | FAIL — misses the direct-conflict retrieval |
| R2 (distractor) | FAIL | PASS |
| R3 (mid-task question) | FAIL | PASS |
| R4 (graph one-hop) | FAIL | PASS |
| R4-nograph (degradation check) | FAIL (expected) | FAIL (expected) |
Haiku: 0/8. Never touches os-adr:* / bin/adr-* / docs/adr/ unprompted in any
scenario — total absence of self-triggered consultation, not a near-miss.
Sonnet: 5/8. Two real misses: W3 (reversal — doesn't propose recording the decision) and R1 (direct-conflict retrieval — doesn't catch it and cite the correct ADR).
R4-nograph failing on both tiers is the expected result — it's a degradation check, only
meaningful paired with an R4 PASS on the same tier (confirms the Graphify one-hop layer did
real work). Sonnet has that pairing (R4 PASS + R4-nograph FAIL); haiku doesn't, since R4 already
failed for haiku.
Observations
- This reads as a prompting/wording issue, not a capability gap. Sonnet already succeeds on
5 of 8 scenarios, and its two failures (W3, R1) are near-misses rather than blind spots —
the model is clearly capable of the behavior when the trigger is salient enough. Haiku's
total 0/8 plus sonnet's two-scenario gap both point toward the SKILL.md wording and/or the
SessionStart hook note text (
plugins/os-adr/hooks/session_start.py) not being salient enough as an unprompted trigger — not toward a reasoning or tool-use failure. - Open question, not yet resolved: whether in-session subagents could ever validly substitute for part of this measurement. Flagged as suspect given the hook-context requirement established for Eval B (in-session subagents inherit the parent session and never get a fresh SessionStart against the sandbox cwd) — worth re-examining specifically for scenarios where the hook note itself isn't the variable under test, but not assumed solved here.
- Next step (not started — future session): design a follow-up eval that iterates
SKILL.md / hook-note wording to raise the unprompted-trigger rate, using the existing
/autoresearchClassic-loop discipline (checker/fixtures/scenarios/rubric frozen, only wording moves — seeplugins/os-adr/eval-b/README.md). That eval should point back at this note for the baseline numbers and the prompting-issue hypothesis, so the loop has a concrete before/after to compare against.
Confirmation run (2026-07-04)
Full grid re-run after fixing stale plugin caches (os-adr's installed cache had drifted — missing
SessionStart hook + 3 CLIs) and switching skill registration to namespaced commands
(/os-adr:find, not bare /find). Baseline confirmed: haiku 0/8, sonnet 5/8 with the same two
behavioral failures (W3, R1). All 16 cells reproduced (one cell, W2/sonnet, needed a clean
re-run after a harness error — no transcript captured on the first attempt — and PASSed on
re-run, matching baseline).
New information beyond confirmation:
- Sonnet W3 is an axis-b failure: A:PASS / B:FAIL — it consults the ADR system unprompted, then doesn't propose recording the reversal. The wording target is the create-skill's "when to record" guidance, NOT trigger salience. R1 remains a pure axis-a (trigger) failure.
- R4-nograph/sonnet PASSed (expected FAIL) — sonnet found the correct ADR without the graph layer this rep. At 1 rep this weakens (doesn't refute) the graph-layer-value evidence.
- Haiku's first axis-a pass (W2): one flicker of unprompted consultation — keep as a lower-tier canary cell in any wording loop.
- Variance is real at 1 rep/cell (haiku W2 axis-a and sonnet W1 flipped between attempts) — wording loops need ~3 reps on target cells.
Procedure and efficiency/quality lessons for the follow-up loop: running-autoresearch-skill-evals.
2026-07-04 — follow-up experiment complete: the wording loop closed the gap — final grid sonnet 8/8, haiku 7/8 (from 5/8 / 0/8). The prompting-issue hypothesis above is confirmed. Hypotheses, per-iteration results, winning wording, and open questions (channel ablation never run; R4-nograph no longer differentiates): os-adr-eval-b-wording-experiment-hypotheses. These baseline numbers are superseded.
Source
- cc-os repo,
plugins/os-adr/eval-b/(harness) andplugins/os-adr/eval-b/README.md(status) - Full TSV:
/tmp/adr-eval-b-grid/results.tsv(ephemeral — not committed, per harness design) - Doc updates committed at cc-os
5b399d5