--- summary: Hypotheses for the os-adr Eval B wording experiment (trigger-conditioned phrasing, packaging channel, two-step trigger chain) and the explicit map from grid outcomes to hypothesis verdicts — the tracking pattern any skill-wording eval should copy. tags: - scope/global - type/reference - project/cc-os - tool/os-adr - tool/autoresearch - domain/llm-evaluation source: cc-os date: 2026-07-04 last_updated: 2026-07-04 related: - os-adr-eval-b-grid-results-and-observations - running-autoresearch-skill-evals --- # os-adr Eval B wording experiment — hypotheses and result mapping Design note for the 2026-07-04 `/autoresearch` wording loop against the Eval B baseline (haiku 0/8, sonnet 5/8 failing W3 + R1 — see [[os-adr-eval-b-grid-results-and-observations]]). Procedure discipline lives in [[running-autoresearch-skill-evals]]; this note holds *what we believe and how we'll know*. ## Framing Eval A (prompted) passes 100% on both tiers → the instructions are **followable**. Eval B (unprompted) fails → the problem is **situation recognition**: the model is told *what* exists (inventory: "this project has ADRs, `/os-adr:find` and `/os-adr:create` exist") but never classifies its *current moment* as ADR-relevant. Analogy that framed it: telling someone "the car is in the driveway" vs "when you're ready to go home, walk out to the car." ## Hypotheses **H1 — Phrasing type (primary):** Trigger-conditioned instructions ("**when** you are about to make an architecture-level choice, or change an approach that already exists in the codebase → run `/os-adr:find`") outperform declarative inventory statements, in whatever channel they appear. The model needs recognition cues to pattern-match its current task against, not facts to spontaneously recall. **H2 — Channel (packaging):** Where the trigger phrasing lives determines whether it's present at decision time. Candidate channels: SessionStart hook note (seen once, may fade), project CLAUDE.md line (in context every session), skill `description` frontmatter (ambiently visible in the skill list). Sub-hypothesis: the SessionStart hook may be redundant once a trigger-phrased CLAUDE.md line exists — testable by ablation. **H3 — Two-step trigger chain (instruction placement):** Each condition must reference only what the model can observe *at that point*. "Reversing a prior decision" is unknowable before consulting — so step 1 triggers on locally observable conditions (making/changing an architecture-level choice → `find`), and step 2 conditions on find's *output* (an Accepted ADR covers the approach you're changing → record a supersession). Step-2 wording belongs in the find/create skill bodies (visible when find's results are on screen), not the session-start note (long out of attention by then). ## How grid outcomes map to hypotheses | Outcome | Verdict | |---|---| | Sonnet R1 flips to PASS (axis a) after trigger-phrased entry surfaces | Confirms H1 for the consultation trigger | | Sonnet W3 flips to PASS (axis b) after step-2 wording in find/create bodies | Confirms H3 (placement) — W3's axis-a already passed, so entry-surface changes can't explain it | | Ablation: CLAUDE.md-only arm holds sonnet 8/8 without the hook note | Confirms H2 sub-hypothesis (hook redundant); keep hook only if its removal regresses | | Ablation: hook-only arm regresses vs CLAUDE.md-only | Channel strength ordering: persistent-context > session-start injection | | Haiku moves off 0/8 with the winning sonnet packaging | H1/H2 generalize down-tier; W2 (haiku's one axis-a flicker) is the expected first mover | | Haiku stays 0/8 after ~3 iterations of the sonnet-winning wording | Capability floor for unprompted triggering at haiku tier — record it and stop iterating | | W1/sonnet control cell regresses in any arm | Wording change rejected regardless of target-cell gains | | Any target cell moves on <2 of 3 reps | Noise, not signal — do not accept the change (variance demonstrated 2026-07-03/04) | ## Meta-lesson (why this note exists) Track hypothesis → expected-outcome mapping **before** running a wording loop. It forces each grid cell to have a declared meaning, prevents post-hoc rationalization of noisy flips, and lets later evals zero in faster by reusing confirmed mechanisms (e.g., if H1 confirms here, start the *next* plugin's unprompted-behavior wording at trigger-conditioned phrasing instead of rediscovering it). Pair this note's pattern with the procedure in [[running-autoresearch-skill-evals]]. ## Results (2026-07-04 — experiment complete, 5 iterations) Final full grid: **sonnet 8/8** (was 5/8), **haiku 7/8** (was 0/8; the one miss is a W3 axis-b judge-boundary flicker — it consulted, and passed the same cell in the mid-experiment confirmation). Per-iteration: 1. **Iter 1 — trigger-conditioned rewrite, all channels** (hook note, find/create descriptions + bodies, fixture CLAUDE.md): R1/sonnet FAIL→3/3 PASS; haiku W2 canary 0→3/3 PASS; W3/sonnet 1/3 (axis-b persists); W1 control holds. **H1 CONFIRMED** — and it reaches haiku, so haiku's 0/8 was packaging, not a capability floor. 2. **Iter 2 — "task not complete until superseding ADR exists"**: W3 still 1/3. Transcript analysis showed the model behaving *well* (catches the conflict, offers options with the ADR attached to the reversal path) but phrasing the offer conditionally inside an option list — judge flicker at the rubric boundary, not forgetting. 3. **Iter 3 — explicit-unconditional recording offer** ("state plainly: this reverses ADR-NNNN; I'll record a superseding ADR — never a parenthetical in an option list"): W3/sonnet 3/3 PASS. **H3 CONFIRMED** (placement in the find skill's act-on-findings step, where find's output is on screen). 4. **Iter 4 — lower threshold "modifying existing behavior → find first"**: R4/haiku 0→3/3; R1/haiku still 0/3. Transcript: haiku classified R1's task as a *pure addition* (new bypass method), so any semantic trigger missed it. 5. **Iter 5 — fully mechanical trigger** ("before your first edit to any existing file → run find on those paths; additions count — a new method can bypass a decided constraint"): R1/haiku axis-a 3/3, full 2/3; R2 distractor control holds (no over-triggering); sonnet unaffected. **Verdicts:** H1 confirmed (trigger-conditioned ≫ inventory phrasing, both tiers). H3 confirmed (each rule lives where its precondition is visible; step-2 wording in the find skill fixed W3). **H2 NOT tested** — channels were never ablated (all-on throughout); whether the SessionStart hook is redundant next to a trigger-phrased CLAUDE.md remains open. Also open: R4-nograph now PASSES on both tiers — the strengthened wording made the graph layer unnecessary for that scenario, so the degradation check no longer differentiates. **Tier-generalization lesson:** each tier down needs a more *mechanical* trigger. Sonnet acts on "architecture-level choice"; haiku needs "before editing any existing file, run find — additions count." The mechanical rule subsumes the semantic one and costs one cheap deterministic CLI call per task — lead with it in future packaging. **Deployment note:** the fixture CLAUDE.md section is the template for real-project adoption (candidate: emit it from `/os-adr:init`/`migrate`); winning wording committed to cc-os (hook `session_start.py`, find/create SKILL.mds, eval-b fixture CLAUDE.md).