summary: Haiku never self-triggers the ADR system unprompted; sonnet passes 5/8 with two near-miss failures confirmed by 2026-07-04 re-run — baseline for follow-up wording optimization.
Eval B measures whether a model notices **on its own** that an Architecture Decision Record system is relevant — without being explicitly told to invoke it.
**Passing criterion:** Model must (a) touch the ADR system unprompted (mechanical detection from tool_use), and (b) produce correct outcome (right ADR cited for retrieval, or new ADR actually proposed for writes). For the 2026-07-03 baseline, passing is per-tier all-scenarios aggregate before any wording optimization.
**Summary:** Haiku 0/8 (never unprompted-consults the ADR system in any scenario). Sonnet 5/8 (failures: W3 reversal / R1 direct-conflict retrieval). R4-nograph degradation checks failed as expected — only meaningful paired with R4 PASS on the same tier.
## Measurement Setup
**Hypotheses / Scenarios Tested**
Two scenario groups per the frozen shapes in `docs/adr-system/06-eval-scenarios.md`:
- **W1–W3 (write-triggers):** Should the model propose recording an architecture decision? W1: persistence choice (new decision to record). W2: convention change (decision to change established convention). W3: reversal (decision to reverse prior decision).
- **R1–R4 (retrieval):** Should the model consult existing decisions? R1: direct conflict (new approach contradicts an Accepted ADR). R2: distractor (scenario mentioning decisions but not directly conflicting). R3: mid-task question (ask at decision point, not upfront). R4: one-hop graph reach (correct ADR findable via Graphify graph traversal from affected files).
Correctness axis (b) for R1–R4: cite the correct existing ADR. For W1–W3: actually propose recording a new ADR (narrow rubric via `judge-rubric.md`, with haiku judge fallback).
Consultation axis (a, all scenarios): did the model touch any `os-adr:*` / `bin/adr-*` / `docs/adr/` unprompted? Mechanical detection from transcript tool_use blocks.
**Fixture and Sampling**
Single Ruby fixture: dedicated webhook-relay with 6-ADR history (Superseded pair + near-miss distractors, generated via the plugin's own CLIs). R4's one-hop graph reach uses a real `graphify update` AST build (model-free, rebuilt via `eval-b/bin/build-fixture-graph`, never committed; Graphify-less R4-nograph variant tests degradation). One rep per cell (1 rep/cell) for baseline discovery.
**Experimental Control (Frozen Surfaces)**
Frozen during this eval: checker (`eval-b/bin/check`, two-axis deterministic-first), fixture (6-ADR Ruby webhook-relay), scenarios (W1–W3, R1–R4, R4-nograph), judge rubric (`judge-rubric.md` for axes-b evaluation of W1–W3). Varying: model tiers (haiku/sonnet) to establish tier-specific performance. Run mode: headless-only (`claude -p` per rep with cwd = sandbox, so real SessionStart hook fires — in-session Agent-tool subagents are invalid here). Rep count 1 chosen for baseline discovery, not for confidence (variance is expected at 1 rep/cell).
## Validity and Limitations
**How to Interpret These Results**
This is a **held-out baseline** (not yet optimized against). The 5/8 / 0/8 numbers mean "wording iteration is possible and worthwhile" — they support the prompting-issue hypothesis, not final deployment readiness. At 1 rep per cell, expect variance: haiku W2 flickered to axis-a PASS on re-run, demonstrating that single-rep cells are noisy. Do not cite sonnet's 5/8 as "80% accuracy" — the two failures (W3, R1) are near-misses (sonnet consults, then misses one output), not blind spots, and axis-specific targeting (H3 for W3, H1 for R1 — see the hypotheses note) suggests wording fixes are available.
**Weaknesses of This Eval (Its Ladder Level)**
Single fixture (Ruby webhook-relay) in one language — generalization to other codebases unknown. 1 rep/cell: high noise floor, flickers observed (haiku W2 axis-a, sonnet W1 across runs). No ambiguity axis: all scenarios are clear-cue boundary cases (not a progression from clear to ambiguous to over-trigger territory). Single distractor scenario (R2) — cannot assess false-positive rate under heavy decision-clutter. Axis-b evaluation of W1–W3 relies on a frozen rubric (`judge-rubric.md`); axis-a is mechanical (tool_use), so it's the reliable axis. Open question: whether in-session subagents could ever validly substitute for part of this measurement (suspected invalid due to hook-context requirement, but not definitively tested). This baseline is now obsolete — the 2026-07-04 wording loop closed the gap; see [[os-adr-eval-b-wording-experiment-hypotheses]] for the follow-up grid and the upgraded numbers.
## Confirmation Run (2026-07-04)
Full grid re-run (2026-07-04) after fixing stale plugin caches (os-adr's installed cache had drifted — missing SessionStart hook + 3 CLIs) and switching skill registration to namespaced commands (`/os-adr:find`, not bare `/find`). **Baseline confirmed: haiku 0/8, sonnet 5/8 with the same two behavioral failures (W3, R1).** All 16 cells reproduced (W2/sonnet needed clean re-run after harness error and PASSed on re-run, matching baseline).
Confirmation run added: **Sonnet W3 is an axis-b failure** (A:PASS / B:FAIL) — consults the ADR system, then doesn't propose recording the reversal; target is create-skill's "when to record" guidance, not trigger salience. **R4-nograph/sonnet PASSed** (expected FAIL) — found the correct ADR without the graph layer this rep; at 1 rep this weakens (doesn't refute) graph-layer-value evidence. **Haiku W2** flickered to axis-a PASS — keep as lower-tier canary in follow-up loops. **Variance is real at 1 rep/cell** (haiku W2 axis-a and sonnet W1 flipped between attempts).
## Deployment and Evolution
**Good-Enough Gate**
This baseline does NOT clear deployment. It is a held-out measurement establishing the prompting/wording issue hypothesis. The 5/8 / 0/8 indicates that wording optimization is possible (near-misses, not blind spots), and the follow-up wording loop (2026-07-04) confirms this — final grid improved to sonnet 8/8, haiku 7/8. Do not deploy based on this 5/8 / 0/8 baseline; consult the follow-up loop results in [[os-adr-eval-b-wording-experiment-hypotheses]] for the trained-up numbers and the gate for real-project rollout.
**Hardening Path / Next Measurement**
The follow-up wording experiment (2026-07-04) is complete — see [[os-adr-eval-b-wording-experiment-hypotheses]] for results and open questions (channel ablation never run; R4-nograph no longer differentiates). The next hardening step is Eval C, an ambiguity-ladder discrimination eval testing whether the model mistakes clear-cue for ambiguous-cue; see [[eval-methodology-ladder]] for the progression and [[eval-methodology-irl-feedback-loop]] for production validation.
