os-adr Eval B wording experiment (2026-07-04) — hypotheses and results

Results Grid/Threshold

Passing criterion: Majority pass (≥2 of 3 reps) on all target cells (W1–W3 write-triggers, R1–R4 retrieval) across both tiers; control cells (W1/sonnet, R2 distractor) hold for regression detection.

Final full grid (5-iteration wording loop, 2026-07-04):

Sonnet 8/8 — all scenarios pass (was 5/8). Haiku 7/8 — all but W3 axis-b pass (was 0/8; the one miss is a W3 axis-b judge-boundary flicker — model consulted, passed same cell mid-experiment confirmation, suggesting marginal rubric sensitivity).

Hypothesis→verdict mapping (the results interpretation):

Outcome	Verdict
Sonnet R1 flips to PASS (axis a) after trigger-phrased entry surfaces	Confirms H1 for consultation trigger
Sonnet W3 flips to PASS (axis b) after step-2 wording in find/create bodies	Confirms H3 (placement) — W3's axis-a already passed, so entry-surface changes can't explain it
Ablation: CLAUDE.md-only arm hypothetically holds sonnet 8/8 without hook note	Would confirm H2 sub-hypothesis (hook redundant) — never tested
Ablation: hook-only arm would regress vs CLAUDE.md-only	Would show channel strength ordering: persistent-context > session-start injection — never tested
Haiku moves off 0/8 with sonnet-winning packaging	Confirms H1/H2 generalize down-tier; W2 (haiku's one axis-a flicker) is the expected first mover — confirmed
Haiku stays low after 3+ iterations of the sonnet-winning wording	Would indicate capability floor for unprompted triggering at haiku tier — not reached; haiku reached 7/8
W1/sonnet control cell regresses in any arm	Wording change rejected regardless of target-cell gains — held across all iterations
Any target cell moves on <2 of 3 reps	Noise, not signal — wording change not accepted — enforced; variance demonstrated

Final verdicts: H1 CONFIRMED (trigger-conditioned phrasing ≫ inventory phrasing, both tiers). H3 CONFIRMED (each rule lives where its precondition is visible). H2 NOT TESTED (channels never ablated; SessionStart hook redundancy remains open). Tier-generalization lesson: sonnet acts on "architecture-level choice" (semantic trigger); haiku needs "before editing any existing file, run find — additions count" (mechanical trigger). The mechanical rule subsumes the semantic one.

Measurement Setup

Hypotheses (Three Framing Questions)

H1 — Phrasing type (primary): Trigger-conditioned instructions ("when you are about to make an architecture-level choice, or change an approach that already exists in the codebase → run /os-adr:find") outperform declarative inventory statements, in whatever channel they appear. The model needs recognition cues to pattern-match its current task against, not facts to spontaneously recall.

H2 — Channel (packaging): Where the trigger phrasing lives determines whether it's present at decision time. Candidate channels: SessionStart hook note (seen once, may fade), project CLAUDE.md line (in context every session), skill description frontmatter (ambiently visible in the skill list). Sub-hypothesis: the SessionStart hook may be redundant once a trigger-phrased CLAUDE.md line exists — testable by ablation (never run).

H3 — Two-step trigger chain (instruction placement): Each condition must reference only what the model can observe at that point. "Reversing a prior decision" is unknowable before consulting — so step 1 triggers on locally observable conditions (making/changing an architecture-level choice → find), and step 2 conditions on find's output (an Accepted ADR covers the approach you're changing → record a supersession). Step-2 wording belongs in the find/create skill bodies (visible when find's results are on screen), not the session-start note (long out of attention by then).

Baseline and Target Scenarios

Baseline (from os-adr-eval-b-grid-results-and-observations): haiku 0/8, sonnet 5/8 failing W3 + R1. Problem: situation recognition — the model is told what exists (inventory: "this project has ADRs, /os-adr:find and /os-adr:create exist") but never classifies its current moment as ADR-relevant. Framing: telling someone "the car is in the driveway" vs "when you're ready to go home, walk out to the car."

Per-iteration results (5 iterations, 2026-07-04):

Iter 1 — trigger-conditioned rewrite, all channels (hook note, find/create descriptions + bodies, fixture CLAUDE.md): R1/sonnet FAIL→3/3 PASS; haiku W2 canary 0→3/3 PASS; W3/sonnet 1/3 (axis-b persists); W1 control holds. H1 CONFIRMED; haiku's 0/8 was packaging, not capability floor.
Iter 2 — "task not complete until superseding ADR exists": W3 still 1/3. Transcript: model behaves well (catches conflict), but phrases offer conditionally inside option list — judge flicker at rubric boundary.
Iter 3 — explicit-unconditional recording offer ("state plainly: this reverses ADR-NNNN; I'll record a superseding ADR — never parenthetical in option list"): W3/sonnet 3/3 PASS. H3 CONFIRMED (placement in find skill's act-on-findings step).
Iter 4 — lower threshold "modifying existing behavior → find first": R4/haiku 0→3/3; R1/haiku still 0/3. Transcript: haiku classified R1 as pure addition (semantic trigger missed it).
Iter 5 — fully mechanical trigger ("before your first edit to any existing file → run find on those paths; additions count — a new method can bypass a decided constraint"): R1/haiku axis-a 3/3, full 2/3; R2 distractor control holds (no over-triggering); sonnet unaffected.

Validity and Limitations

How to Interpret These Results

This eval was optimized against (wording loop running 5 iterations over the same grid) — the sonnet 8/8 / haiku 7/8 are training-set scores, not held-out. What these numbers support: "the wording direction is sound, and haiku is reachable via more-mechanical phrasing." What they do NOT support: "the wording generalizes to other codebases, languages, or unprompted scenarios not in this fixture." The grid-results baseline (haiku 0/8, sonnet 5/8) was held-out; once wording tuning started, this note's measurements became training-set. Confidence caveats: Sonnet 8/8 is clean (all 3 reps PASS on all cells). Haiku 7/8 includes one axis-b judge-boundary case (W3) where the model passed mid-experiment but flickered back to FAIL at the end; if the judge rubric sensitivity is the blocker rather than the model behavior, this is a measurement artifact, not a real gap. R4-nograph now PASSES on both tiers — the strengthened wording made the graph layer unnecessary for that scenario, so the degradation check no longer differentiates (open question: does the improved wording help in cases where the graph layer remains necessary elsewhere).

Weaknesses of This Eval (Its Ladder Level)

Single fixture (Ruby webhook-relay) in one language — generalization to other codebases unknown. 3 reps on target cells, 1 rep on controls — variance is smaller but not eliminated (W3 haiku axis-b flicker evident). No ambiguity axis: all scenarios are clear-cue boundary cases; does not test whether the model mistakes clear-cue for ambiguous-cue or over-triggers on low-confidence edges. Distractor testing minimal (R2 only) — cannot assess false-positive rate under heavy decision-clutter. Axis-b evaluation of W1–W3 relies on frozen rubric (judge-rubric.md); judge-boundary sensitivity suspected. Ablation surfaces never tested: channel redundancy (hook vs CLAUDE.md), whether in-session subagents could ever validly substitute for part of this (suspect invalid, but not definitively tested). This is a training-set measurement — if results were used to update SKILL.md / hook-note wording (they were), the grid becomes invalid as a held-out measurement. The next eval (Eval C, ambiguity-ladder discrimination) will test generalization via the ladder mechanism.

Hardening Path / Next Measurement

Eval C (in design, not yet built) will run an ambiguity-ladder discrimination eval on a second fixture (different language/domain to test fixture generalization) with paired positive/negative scenarios at each ladder level and per-level pass bars. See eval-methodology-ladder for the ladder approach and eval-methodology-irl-feedback-loop for production validation via session auditing.

Deployment and Evolution

Good-Enough Gate

Sonnet: Cleared for real-project pilot rollout. The wording shipped in cc-os (hook session_start.py, find/create SKILL.mds, eval-b fixture CLAUDE.md). Projects should adopt via the trigger-phrased "Architecture decisions" CLAUDE.md section (copy from eval-b/fixture/project/CLAUDE.md; candidate: emit from /os-adr:init/migrate).

Haiku: Resolved 2026-07-06 — cleared for pilot, with a characterized caveat. The stability check ran 7 reps total (not the planned 3; the runner agent's first batches included failures it later excluded, so all reps are counted here to avoid peeking bias): axis (a) consultation 7/7 PASS — rock solid; axis (b) recording-offer ~5/7 PASS (~75–85%). One FAIL was plausibly infra (session died, no final message to judge); one was a genuine behavioral miss: haiku consulted, then asked a clarifying question instead of unconditionally proposing the superseding ADR — the same conditional-phrasing failure mode iteration 3 fixed on sonnet, improved but not eliminated on haiku. Verdict: intermittent flicker, not a hard gap. The grid claim stays 7/8 with W3 as an ~1-in-4/5 axis-b miss, not 8/8. Practical impact is soft: haiku always consults; occasionally the user must say "yes, record it" rather than receiving an unconditional offer. Acceptable for daily use given production auditing (see eval-methodology-irl-feedback-loop); Eval C's per-level pass bars should anticipate exactly this borderline-behavior flicker at 1 rep.

Next steps: Real-project migration/adoption one at a time via /os-adr:migrate (pilot projects first, then cc-os retrofit, then wider). For each onboarded project, add the trigger-phrased "Architecture decisions" CLAUDE.md section. Establish a session-audit backlog (see eval-methodology-irl-feedback-loop) to surface silent misses and feed them back as new eval scenarios for Eval C.

Meta-lesson

Track hypothesis → expected-outcome mapping before running a wording loop. Forces each grid cell to have declared meaning; prevents post-hoc rationalization of noisy flips; lets later evals reuse confirmed mechanisms (e.g., if H1 confirms, start the next plugin's unprompted-behavior wording at trigger-conditioned phrasing). Pair this note's pattern with running-autoresearch-skill-evals procedure.

os-adr-eval-b-grid-results-and-observations — baseline held-out grid (2026-07-03) before wording optimization
running-autoresearch-skill-evals — skill-wording eval loop procedure and efficiency lessons
eval-methodology-ladder — evaluation ladder approach and progression logic
eval-methodology-irl-feedback-loop — production validation via session auditing

12 KiB Raw Permalink Blame History Unescape Escape