diff --git a/2026-07-03-os-adr-eval-b-grid-results-and-observations.md b/2026-07-03-os-adr-eval-b-grid-results-and-observations.md new file mode 100644 index 0000000..ad904bb --- /dev/null +++ b/2026-07-03-os-adr-eval-b-grid-results-and-observations.md @@ -0,0 +1,72 @@ +--- +summary: os-adr Eval B (unprompted write-trigger & retrieval) grid results for haiku/sonnet — haiku never self-triggers the ADR system, sonnet passes 5/8 with two near-miss failures — read before designing any follow-up eval on unprompted-consultation behavior. +tags: + - scope/global + - type/reference + - project/cc-os + - tool/os-adr + - domain/llm-evaluation +source: cc-os +date: 2026-07-03 +--- + +# os-adr Eval B grid results (2026-07-03) + +Eval B measures whether a model notices **on its own** that an Architecture Decision Record +system is relevant — without being explicitly told to invoke it. Two axes per scenario: (a) +did the model touch the ADR system unprompted (mechanical, from tool_use), (b) was the specific +outcome correct (right ADR cited, or a new ADR actually proposed/created). See +`cc-os/plugins/os-adr/eval-b/README.md` for full harness design (headless-only `claude -p` runs, +so a real SessionStart hook fires — in-session Agent-tool subagents cannot validly stand in for +this, since they don't get a fresh hook invocation against the sandbox cwd). + +## Grid (1 rep/cell) + +| Scenario | Haiku | Sonnet | +|---|---|---| +| W1 (persistence choice) | FAIL | **PASS** | +| W2 (convention change) | FAIL | **PASS** | +| W3 (reversal) | FAIL | FAIL — doesn't propose recording the decision | +| R1 (direct conflict) | FAIL | FAIL — misses the direct-conflict retrieval | +| R2 (distractor) | FAIL | **PASS** | +| R3 (mid-task question) | FAIL | **PASS** | +| R4 (graph one-hop) | FAIL | **PASS** | +| R4-nograph (degradation check) | FAIL (expected) | FAIL (expected) | + +**Haiku: 0/8.** Never touches `os-adr:*` / `bin/adr-*` / `docs/adr/` unprompted in any +scenario — total absence of self-triggered consultation, not a near-miss. + +**Sonnet: 5/8.** Two real misses: W3 (reversal — doesn't propose recording the decision) and R1 +(direct-conflict retrieval — doesn't catch it and cite the correct ADR). + +`R4-nograph` failing on both tiers is the **expected** result — it's a degradation check, only +meaningful paired with an R4 PASS on the same tier (confirms the Graphify one-hop layer did +real work). Sonnet has that pairing (R4 PASS + R4-nograph FAIL); haiku doesn't, since R4 already +failed for haiku. + +## Observations + +- **This reads as a prompting/wording issue, not a capability gap.** Sonnet already succeeds on + 5 of 8 scenarios, and its two failures (W3, R1) are near-misses rather than blind spots — + the model is clearly capable of the behavior when the trigger is salient enough. Haiku's + total 0/8 plus sonnet's two-scenario gap both point toward the SKILL.md wording and/or the + SessionStart hook note text (`plugins/os-adr/hooks/session_start.py`) not being salient + enough as an unprompted trigger — not toward a reasoning or tool-use failure. +- **Open question, not yet resolved:** whether in-session subagents could ever validly + substitute for part of this measurement. Flagged as suspect given the hook-context + requirement established for Eval B (in-session subagents inherit the parent session and never + get a fresh SessionStart against the sandbox cwd) — worth re-examining specifically for + scenarios where the hook note itself isn't the variable under test, but not assumed solved + here. +- **Next step (not started — future session):** design a follow-up eval that iterates + SKILL.md / hook-note wording to raise the unprompted-trigger rate, using the existing + `/autoresearch` Classic-loop discipline (checker/fixtures/scenarios/rubric frozen, only + wording moves — see `plugins/os-adr/eval-b/README.md`). **That eval should point back at + this note** for the baseline numbers and the prompting-issue hypothesis, so the loop has a + concrete before/after to compare against. + +## Source + +- cc-os repo, `plugins/os-adr/eval-b/` (harness) and `plugins/os-adr/eval-b/README.md` (status) +- Full TSV: `/tmp/adr-eval-b-grid/results.tsv` (ephemeral — not committed, per harness design) +- Doc updates committed at cc-os `5b399d5` diff --git a/journal/2026-07-03.md b/journal/2026-07-03.md index 68107ff..9d088b4 100644 --- a/journal/2026-07-03.md +++ b/journal/2026-07-03.md @@ -394,3 +394,10 @@ tags: [scope/global, type/log] **Reason:** other **Vault notes touched:** (none) + +## Session — 2026-07-03T21:32:58Z + +**Project:** /tmp/adr-eval-b-grid/R2-sonnet-r1 +**Reason:** other +**Vault notes touched:** +(none)