os-adr Eval B wording experiment — hypotheses and result mapping

Design note for the 2026-07-04 /autoresearch wording loop against the Eval B baseline (haiku 0/8, sonnet 5/8 failing W3 + R1 — see os-adr-eval-b-grid-results-and-observations). Procedure discipline lives in running-autoresearch-skill-evals; this note holds what we believe and how we'll know.

Framing

Eval A (prompted) passes 100% on both tiers → the instructions are followable. Eval B (unprompted) fails → the problem is situation recognition: the model is told what exists (inventory: "this project has ADRs, /os-adr:find and /os-adr:create exist") but never classifies its current moment as ADR-relevant. Analogy that framed it: telling someone "the car is in the driveway" vs "when you're ready to go home, walk out to the car."

Hypotheses

H1 — Phrasing type (primary): Trigger-conditioned instructions ("when you are about to make an architecture-level choice, or change an approach that already exists in the codebase → run /os-adr:find") outperform declarative inventory statements, in whatever channel they appear. The model needs recognition cues to pattern-match its current task against, not facts to spontaneously recall.

H2 — Channel (packaging): Where the trigger phrasing lives determines whether it's present at decision time. Candidate channels: SessionStart hook note (seen once, may fade), project CLAUDE.md line (in context every session), skill description frontmatter (ambiently visible in the skill list). Sub-hypothesis: the SessionStart hook may be redundant once a trigger-phrased CLAUDE.md line exists — testable by ablation.

H3 — Two-step trigger chain (instruction placement): Each condition must reference only what the model can observe at that point. "Reversing a prior decision" is unknowable before consulting — so step 1 triggers on locally observable conditions (making/changing an architecture-level choice → find), and step 2 conditions on find's output (an Accepted ADR covers the approach you're changing → record a supersession). Step-2 wording belongs in the find/create skill bodies (visible when find's results are on screen), not the session-start note (long out of attention by then).

How grid outcomes map to hypotheses

Outcome	Verdict
Sonnet R1 flips to PASS (axis a) after trigger-phrased entry surfaces	Confirms H1 for the consultation trigger
Sonnet W3 flips to PASS (axis b) after step-2 wording in find/create bodies	Confirms H3 (placement) — W3's axis-a already passed, so entry-surface changes can't explain it
Ablation: CLAUDE.md-only arm holds sonnet 8/8 without the hook note	Confirms H2 sub-hypothesis (hook redundant); keep hook only if its removal regresses
Ablation: hook-only arm regresses vs CLAUDE.md-only	Channel strength ordering: persistent-context > session-start injection
Haiku moves off 0/8 with the winning sonnet packaging	H1/H2 generalize down-tier; W2 (haiku's one axis-a flicker) is the expected first mover
Haiku stays 0/8 after ~3 iterations of the sonnet-winning wording	Capability floor for unprompted triggering at haiku tier — record it and stop iterating
W1/sonnet control cell regresses in any arm	Wording change rejected regardless of target-cell gains
Any target cell moves on <2 of 3 reps	Noise, not signal — do not accept the change (variance demonstrated 2026-07-03/04)

Meta-lesson (why this note exists)

Track hypothesis → expected-outcome mapping before running a wording loop. It forces each grid cell to have a declared meaning, prevents post-hoc rationalization of noisy flips, and lets later evals zero in faster by reusing confirmed mechanisms (e.g., if H1 confirms here, start the next plugin's unprompted-behavior wording at trigger-conditioned phrasing instead of rediscovering it). Pair this note's pattern with the procedure in running-autoresearch-skill-evals.

Status

2026-07-04: hypotheses locked, experiment starting (baseline commit cc-os c645faa). Results to be appended here per iteration.

4.6 KiB Raw Blame History