SecondBrain/os-adr-eval-b-wording-experiment-hypotheses.md

---
type: eval-results
title: os-adr Eval B wording experiment (2026-07-04) — hypotheses and results
summary: Five-iteration wording loop optimizing trigger-phrasing closed the Eval B gap from sonnet 5/8 / haiku 0/8 to sonnet 8/8 / haiku 7/8 — confirms prompting/wording issue; defines next ladder level.
tags:
  - type/eval-results
  - domain/llm-evaluation
  - tool/os-adr
  - tool/autoresearch
  - project/cc-os
scope: global
last_updated: 2026-07-06
date: 2026-07-04
related:
  - os-adr-eval-b-grid-results-and-observations
  - running-autoresearch-skill-evals
  - eval-methodology-ladder
source: cc-os
---

# os-adr Eval B wording experiment (2026-07-04) — hypotheses and results

## Results Grid/Threshold

**Passing criterion:** Majority pass (≥2 of 3 reps) on all target cells (W1–W3 write-triggers, R1–R4 retrieval) across both tiers; control cells (W1/sonnet, R2 distractor) hold for regression detection.

**Final full grid (5-iteration wording loop, 2026-07-04):**

**Sonnet 8/8** — all scenarios pass (was 5/8). **Haiku 7/8** — all but W3 axis-b pass (was 0/8; the one miss is a W3 axis-b judge-boundary flicker — model consulted, passed same cell mid-experiment confirmation, suggesting marginal rubric sensitivity).

**Hypothesis→verdict mapping (the results interpretation):**

| Outcome | Verdict |
|---|---|
| Sonnet R1 flips to PASS (axis a) after trigger-phrased entry surfaces | Confirms H1 for consultation trigger |
| Sonnet W3 flips to PASS (axis b) after step-2 wording in find/create bodies | Confirms H3 (placement) — W3's axis-a already passed, so entry-surface changes can't explain it |
| Ablation: CLAUDE.md-only arm hypothetically holds sonnet 8/8 without hook note | Would confirm H2 sub-hypothesis (hook redundant) — **never tested** |
| Ablation: hook-only arm would regress vs CLAUDE.md-only | Would show channel strength ordering: persistent-context > session-start injection — **never tested** |
| Haiku moves off 0/8 with sonnet-winning packaging | Confirms H1/H2 generalize down-tier; W2 (haiku's one axis-a flicker) is the expected first mover — **confirmed** |
| Haiku stays low after 3+ iterations of the sonnet-winning wording | Would indicate capability floor for unprompted triggering at haiku tier — **not reached; haiku reached 7/8** |
| W1/sonnet control cell regresses in any arm | Wording change rejected regardless of target-cell gains — **held across all iterations** |
| Any target cell moves on <2 of 3 reps | Noise, not signal — wording change not accepted — **enforced; variance demonstrated** |

Final **verdicts:** H1 CONFIRMED (trigger-conditioned phrasing ≫ inventory phrasing, both tiers). H3 CONFIRMED (each rule lives where its precondition is visible). H2 NOT TESTED (channels never ablated; SessionStart hook redundancy remains open). **Tier-generalization lesson:** sonnet acts on "architecture-level choice" (semantic trigger); haiku needs "before editing any existing file, run find — additions count" (mechanical trigger). The mechanical rule subsumes the semantic one.

---

## Measurement Setup

**Hypotheses (Three Framing Questions)**

**H1 — Phrasing type (primary):** Trigger-conditioned instructions ("**when** you are about to make an architecture-level choice, or change an approach that already exists in the codebase → run `/os-adr:find`") outperform declarative inventory statements, in whatever channel they appear. The model needs recognition cues to pattern-match its current task against, not facts to spontaneously recall.

**H2 — Channel (packaging):** Where the trigger phrasing lives determines whether it's present at decision time. Candidate channels: SessionStart hook note (seen once, may fade), project CLAUDE.md line (in context every session), skill `description` frontmatter (ambiently visible in the skill list). Sub-hypothesis: the SessionStart hook may be redundant once a trigger-phrased CLAUDE.md line exists — testable by ablation (never run).

**H3 — Two-step trigger chain (instruction placement):** Each condition must reference only what the model can observe *at that point*. "Reversing a prior decision" is unknowable before consulting — so step 1 triggers on locally observable conditions (making/changing an architecture-level choice → `find`), and step 2 conditions on find's *output* (an Accepted ADR covers the approach you're changing → record a supersession). Step-2 wording belongs in the find/create skill bodies (visible when find's results are on screen), not the session-start note (long out of attention by then).

**Baseline and Target Scenarios**

Baseline (from [[os-adr-eval-b-grid-results-and-observations]]): haiku 0/8, sonnet 5/8 failing W3 + R1. Problem: **situation recognition** — the model is told *what* exists (inventory: "this project has ADRs, `/os-adr:find` and `/os-adr:create` exist") but never classifies its *current moment* as ADR-relevant. Framing: telling someone "the car is in the driveway" vs "when you're ready to go home, walk out to the car."

**Per-iteration results (5 iterations, 2026-07-04):**

1. **Iter 1** — trigger-conditioned rewrite, all channels (hook note, find/create descriptions + bodies, fixture CLAUDE.md): R1/sonnet FAIL→3/3 PASS; haiku W2 canary 0→3/3 PASS; W3/sonnet 1/3 (axis-b persists); W1 control holds. **H1 CONFIRMED**; haiku's 0/8 was packaging, not capability floor.
2. **Iter 2** — "task not complete until superseding ADR exists": W3 still 1/3. Transcript: model behaves well (catches conflict), but phrases offer conditionally inside option list — judge flicker at rubric boundary.
3. **Iter 3** — explicit-unconditional recording offer ("state plainly: this reverses ADR-NNNN; I'll record a superseding ADR — never parenthetical in option list"): W3/sonnet 3/3 PASS. **H3 CONFIRMED** (placement in find skill's act-on-findings step).
4. **Iter 4** — lower threshold "modifying existing behavior → find first": R4/haiku 0→3/3; R1/haiku still 0/3. Transcript: haiku classified R1 as *pure addition* (semantic trigger missed it).
5. **Iter 5** — fully mechanical trigger ("before your first edit to any existing file → run find on those paths; additions count — a new method can bypass a decided constraint"): R1/haiku axis-a 3/3, full 2/3; R2 distractor control holds (no over-triggering); sonnet unaffected.

---

## Validity and Limitations

**How to Interpret These Results**

This eval **was optimized against** (wording loop running 5 iterations over the same grid) — the sonnet 8/8 / haiku 7/8 are **training-set scores**, not held-out. What these numbers support: "the wording direction is sound, and haiku is reachable via more-mechanical phrasing." What they do NOT support: "the wording generalizes to other codebases, languages, or unprompted scenarios not in this fixture." The grid-results baseline (haiku 0/8, sonnet 5/8) was held-out; once wording tuning started, this note's measurements became training-set. **Confidence caveats:** Sonnet 8/8 is clean (all 3 reps PASS on all cells). Haiku 7/8 includes one axis-b judge-boundary case (W3) where the model passed mid-experiment but flickered back to FAIL at the end; if the judge rubric sensitivity is the blocker rather than the model behavior, this is a measurement artifact, not a real gap. R4-nograph now PASSES on both tiers — the strengthened wording made the graph layer unnecessary for that scenario, so the degradation check no longer differentiates (open question: does the improved wording help in cases where the graph layer remains necessary elsewhere).

**Weaknesses of This Eval (Its Ladder Level)**

Single fixture (Ruby webhook-relay) in one language — generalization to other codebases unknown. 3 reps on target cells, 1 rep on controls — variance is smaller but not eliminated (W3 haiku axis-b flicker evident). No ambiguity axis: all scenarios are clear-cue boundary cases; does not test whether the model mistakes clear-cue for ambiguous-cue or over-triggers on low-confidence edges. Distractor testing minimal (R2 only) — cannot assess false-positive rate under heavy decision-clutter. Axis-b evaluation of W1–W3 relies on frozen rubric (`judge-rubric.md`); judge-boundary sensitivity suspected. Ablation surfaces never tested: channel redundancy (hook vs CLAUDE.md), whether in-session subagents could ever validly substitute for part of this (suspect invalid, but not definitively tested). **This is a training-set measurement** — if results were used to update SKILL.md / hook-note wording (they were), the grid becomes invalid as a held-out measurement. The next eval (Eval C, ambiguity-ladder discrimination) will test generalization via the ladder mechanism.

**Hardening Path / Next Measurement**

Eval C (in design, not yet built) will run an ambiguity-ladder discrimination eval on a second fixture (different language/domain to test fixture generalization) with paired positive/negative scenarios at each ladder level and per-level pass bars. See [[eval-methodology-ladder]] for the ladder approach and [[eval-methodology-irl-feedback-loop]] for production validation via session auditing.

---

## Deployment and Evolution

**Good-Enough Gate**

**Sonnet:** Cleared for real-project pilot rollout. The wording shipped in cc-os (hook `session_start.py`, find/create SKILL.mds, eval-b fixture CLAUDE.md). Projects should adopt via the trigger-phrased "Architecture decisions" CLAUDE.md section (copy from `eval-b/fixture/project/CLAUDE.md`; candidate: emit from `/os-adr:init`/`migrate`).

**Haiku:** Resolved 2026-07-06 — cleared for pilot, with a characterized caveat. The stability check ran 7 reps total (not the planned 3; the runner agent's first batches included failures it later excluded, so all reps are counted here to avoid peeking bias): axis (a) consultation 7/7 PASS — rock solid; axis (b) recording-offer ~5/7 PASS (~75–85%). One FAIL was plausibly infra (session died, no final message to judge); one was a genuine behavioral miss: haiku consulted, then asked a clarifying question instead of unconditionally proposing the superseding ADR — the same conditional-phrasing failure mode iteration 3 fixed on sonnet, improved but not eliminated on haiku. Verdict: intermittent flicker, not a hard gap. The grid claim stays **7/8 with W3 as an ~1-in-4/5 axis-b miss**, not 8/8. Practical impact is soft: haiku always consults; occasionally the user must say "yes, record it" rather than receiving an unconditional offer. Acceptable for daily use given production auditing (see [[eval-methodology-irl-feedback-loop]]); Eval C's per-level pass bars should anticipate exactly this borderline-behavior flicker at 1 rep.

**Next steps:** Real-project migration/adoption one at a time via `/os-adr:migrate` (pilot projects first, then cc-os retrofit, then wider). For each onboarded project, add the trigger-phrased "Architecture decisions" CLAUDE.md section. Establish a session-audit backlog (see [[eval-methodology-irl-feedback-loop]]) to surface silent misses and feed them back as new eval scenarios for Eval C.

## Meta-lesson

Track hypothesis → expected-outcome mapping **before** running a wording loop. Forces each grid cell to have declared meaning; prevents post-hoc rationalization of noisy flips; lets later evals reuse confirmed mechanisms (e.g., if H1 confirms, start the *next* plugin's unprompted-behavior wording at trigger-conditioned phrasing). Pair this note's pattern with [[running-autoresearch-skill-evals]] procedure.

## Related

- [[os-adr-eval-b-grid-results-and-observations]] — baseline held-out grid (2026-07-03) before wording optimization
- [[running-autoresearch-skill-evals]] — skill-wording eval loop procedure and efficiency lessons
- [[eval-methodology-ladder]] — evaluation ladder approach and progression logic
- [[eval-methodology-irl-feedback-loop]] — production validation via session auditing