SecondBrain/os-adr-eval-b-wording-experiment-hypotheses.md

---
summary: Hypotheses for the os-adr Eval B wording experiment (trigger-conditioned phrasing, packaging channel, two-step trigger chain) and the explicit map from grid outcomes to hypothesis verdicts — the tracking pattern any skill-wording eval should copy.
tags:
  - scope/global
  - type/reference
  - project/cc-os
  - tool/os-adr
  - tool/autoresearch
  - domain/llm-evaluation
source: cc-os
date: 2026-07-04
last_updated: 2026-07-04
related:
  - os-adr-eval-b-grid-results-and-observations
  - running-autoresearch-skill-evals
---

# os-adr Eval B wording experiment — hypotheses and result mapping

Design note for the 2026-07-04 `/autoresearch` wording loop against the Eval B baseline
(haiku 0/8, sonnet 5/8 failing W3 + R1 — see
[[os-adr-eval-b-grid-results-and-observations]]). Procedure discipline lives in
[[running-autoresearch-skill-evals]]; this note holds *what we believe and how we'll know*.

## Framing

Eval A (prompted) passes 100% on both tiers → the instructions are **followable**. Eval B
(unprompted) fails → the problem is **situation recognition**: the model is told *what*
exists (inventory: "this project has ADRs, `/os-adr:find` and `/os-adr:create` exist") but
never classifies its *current moment* as ADR-relevant. Analogy that framed it: telling
someone "the car is in the driveway" vs "when you're ready to go home, walk out to the car."

## Hypotheses

**H1 — Phrasing type (primary):** Trigger-conditioned instructions ("**when** you are about
to make an architecture-level choice, or change an approach that already exists in the
codebase → run `/os-adr:find`") outperform declarative inventory statements, in whatever
channel they appear. The model needs recognition cues to pattern-match its current task
against, not facts to spontaneously recall.

**H2 — Channel (packaging):** Where the trigger phrasing lives determines whether it's
present at decision time. Candidate channels: SessionStart hook note (seen once, may fade),
project CLAUDE.md line (in context every session), skill `description` frontmatter
(ambiently visible in the skill list). Sub-hypothesis: the SessionStart hook may be
redundant once a trigger-phrased CLAUDE.md line exists — testable by ablation.

**H3 — Two-step trigger chain (instruction placement):** Each condition must reference only
what the model can observe *at that point*. "Reversing a prior decision" is unknowable
before consulting — so step 1 triggers on locally observable conditions (making/changing an
architecture-level choice → `find`), and step 2 conditions on find's *output* (an Accepted
ADR covers the approach you're changing → record a supersession). Step-2 wording belongs in
the find/create skill bodies (visible when find's results are on screen), not the
session-start note (long out of attention by then).

## How grid outcomes map to hypotheses

| Outcome | Verdict |
|---|---|
| Sonnet R1 flips to PASS (axis a) after trigger-phrased entry surfaces | Confirms H1 for the consultation trigger |
| Sonnet W3 flips to PASS (axis b) after step-2 wording in find/create bodies | Confirms H3 (placement) — W3's axis-a already passed, so entry-surface changes can't explain it |
| Ablation: CLAUDE.md-only arm holds sonnet 8/8 without the hook note | Confirms H2 sub-hypothesis (hook redundant); keep hook only if its removal regresses |
| Ablation: hook-only arm regresses vs CLAUDE.md-only | Channel strength ordering: persistent-context > session-start injection |
| Haiku moves off 0/8 with the winning sonnet packaging | H1/H2 generalize down-tier; W2 (haiku's one axis-a flicker) is the expected first mover |
| Haiku stays 0/8 after ~3 iterations of the sonnet-winning wording | Capability floor for unprompted triggering at haiku tier — record it and stop iterating |
| W1/sonnet control cell regresses in any arm | Wording change rejected regardless of target-cell gains |
| Any target cell moves on <2 of 3 reps | Noise, not signal — do not accept the change (variance demonstrated 2026-07-03/04) |

## Meta-lesson (why this note exists)

Track hypothesis → expected-outcome mapping **before** running a wording loop. It forces
each grid cell to have a declared meaning, prevents post-hoc rationalization of noisy
flips, and lets later evals zero in faster by reusing confirmed mechanisms (e.g., if H1
confirms here, start the *next* plugin's unprompted-behavior wording at trigger-conditioned
phrasing instead of rediscovering it). Pair this note's pattern with the procedure in
[[running-autoresearch-skill-evals]].

## Status

- 2026-07-04: hypotheses locked, experiment starting (baseline commit cc-os `c645faa`).
  Results to be appended here per iteration.