SecondBrain/2026-07-03-os-adr-eval-b-grid-results-and-observations.md

summary	tags	source	date
os-adr Eval B (unprompted write-trigger & retrieval) grid results for haiku/sonnet — haiku never self-triggers the ADR system, sonnet passes 5/8 with two near-miss failures — read before designing any follow-up eval on unprompted-consultation behavior.	scope/global type/reference project/cc-os tool/os-adr domain/llm-evaluation	cc-os	2026-07-03

os-adr Eval B grid results (2026-07-03)

Eval B measures whether a model notices on its own that an Architecture Decision Record system is relevant — without being explicitly told to invoke it. Two axes per scenario: (a) did the model touch the ADR system unprompted (mechanical, from tool_use), (b) was the specific outcome correct (right ADR cited, or a new ADR actually proposed/created). See cc-os/plugins/os-adr/eval-b/README.md for full harness design (headless-only claude -p runs, so a real SessionStart hook fires — in-session Agent-tool subagents cannot validly stand in for this, since they don't get a fresh hook invocation against the sandbox cwd).

Grid (1 rep/cell)

Scenario	Haiku	Sonnet
W1 (persistence choice)	FAIL	PASS
W2 (convention change)	FAIL	PASS
W3 (reversal)	FAIL	FAIL — doesn't propose recording the decision
R1 (direct conflict)	FAIL	FAIL — misses the direct-conflict retrieval
R2 (distractor)	FAIL	PASS
R3 (mid-task question)	FAIL	PASS
R4 (graph one-hop)	FAIL	PASS
R4-nograph (degradation check)	FAIL (expected)	FAIL (expected)

Haiku: 0/8. Never touches os-adr:* / bin/adr-* / docs/adr/ unprompted in any scenario — total absence of self-triggered consultation, not a near-miss.

Sonnet: 5/8. Two real misses: W3 (reversal — doesn't propose recording the decision) and R1 (direct-conflict retrieval — doesn't catch it and cite the correct ADR).

R4-nograph failing on both tiers is the expected result — it's a degradation check, only meaningful paired with an R4 PASS on the same tier (confirms the Graphify one-hop layer did real work). Sonnet has that pairing (R4 PASS + R4-nograph FAIL); haiku doesn't, since R4 already failed for haiku.

Observations

• This reads as a prompting/wording issue, not a capability gap. Sonnet already succeeds on 5 of 8 scenarios, and its two failures (W3, R1) are near-misses rather than blind spots — the model is clearly capable of the behavior when the trigger is salient enough. Haiku's total 0/8 plus sonnet's two-scenario gap both point toward the SKILL.md wording and/or the SessionStart hook note text (plugins/os-adr/hooks/session_start.py) not being salient enough as an unprompted trigger — not toward a reasoning or tool-use failure.
• Open question, not yet resolved: whether in-session subagents could ever validly substitute for part of this measurement. Flagged as suspect given the hook-context requirement established for Eval B (in-session subagents inherit the parent session and never get a fresh SessionStart against the sandbox cwd) — worth re-examining specifically for scenarios where the hook note itself isn't the variable under test, but not assumed solved here.
• Next step (not started — future session): design a follow-up eval that iterates SKILL.md / hook-note wording to raise the unprompted-trigger rate, using the existing /autoresearch Classic-loop discipline (checker/fixtures/scenarios/rubric frozen, only wording moves — see plugins/os-adr/eval-b/README.md). That eval should point back at this note for the baseline numbers and the prompting-issue hypothesis, so the loop has a concrete before/after to compare against.

Source

• cc-os repo, plugins/os-adr/eval-b/ (harness) and plugins/os-adr/eval-b/README.md (status)
• Full TSV: /tmp/adr-eval-b-grid/results.tsv (ephemeral — not committed, per harness design)
• Doc updates committed at cc-os 5b399d5