--- type: eval-results title: os-adr Eval B (unprompted write-trigger & retrieval) grid results (2026-07-03) summary: Haiku never self-triggers the ADR system unprompted; sonnet passes 5/8 with two near-miss failures confirmed by 2026-07-04 re-run — baseline for follow-up wording optimization. tags: - type/eval-results - domain/llm-evaluation - tool/os-adr - project/cc-os scope: global last_updated: 2026-07-04 date: 2026-07-03 related: - os-adr-eval-b-wording-experiment-hypotheses - running-autoresearch-skill-evals source: cc-os --- # os-adr Eval B grid results (2026-07-03 baseline) ## Results Grid/Threshold Eval B measures whether a model notices **on its own** that an Architecture Decision Record system is relevant — without being explicitly told to invoke it. **Passing criterion:** Model must (a) touch the ADR system unprompted (mechanical detection from tool_use), and (b) produce correct outcome (right ADR cited for retrieval, or new ADR actually proposed for writes). For the 2026-07-03 baseline, passing is per-tier all-scenarios aggregate before any wording optimization. **Grid (1 rep/cell):** | Scenario | Haiku | Sonnet | |---|---|---| | W1 (persistence choice) | FAIL | **PASS** | | W2 (convention change) | FAIL | **PASS** | | W3 (reversal) | FAIL | FAIL — doesn't propose recording the decision | | R1 (direct conflict) | FAIL | FAIL — misses the direct-conflict retrieval | | R2 (distractor) | FAIL | **PASS** | | R3 (mid-task question) | FAIL | **PASS** | | R4 (graph one-hop) | FAIL | **PASS** | | R4-nograph (degradation check) | FAIL (expected) | FAIL (expected) | **Summary:** Haiku 0/8 (never unprompted-consults the ADR system in any scenario). Sonnet 5/8 (failures: W3 reversal / R1 direct-conflict retrieval). R4-nograph degradation checks failed as expected — only meaningful paired with R4 PASS on the same tier. ## Measurement Setup **Hypotheses / Scenarios Tested** Two scenario groups per the frozen shapes in `docs/adr-system/06-eval-scenarios.md`: - **W1–W3 (write-triggers):** Should the model propose recording an architecture decision? W1: persistence choice (new decision to record). W2: convention change (decision to change established convention). W3: reversal (decision to reverse prior decision). - **R1–R4 (retrieval):** Should the model consult existing decisions? R1: direct conflict (new approach contradicts an Accepted ADR). R2: distractor (scenario mentioning decisions but not directly conflicting). R3: mid-task question (ask at decision point, not upfront). R4: one-hop graph reach (correct ADR findable via Graphify graph traversal from affected files). Correctness axis (b) for R1–R4: cite the correct existing ADR. For W1–W3: actually propose recording a new ADR (narrow rubric via `judge-rubric.md`, with haiku judge fallback). Consultation axis (a, all scenarios): did the model touch any `os-adr:*` / `bin/adr-*` / `docs/adr/` unprompted? Mechanical detection from transcript tool_use blocks. **Fixture and Sampling** Single Ruby fixture: dedicated webhook-relay with 6-ADR history (Superseded pair + near-miss distractors, generated via the plugin's own CLIs). R4's one-hop graph reach uses a real `graphify update` AST build (model-free, rebuilt via `eval-b/bin/build-fixture-graph`, never committed; Graphify-less R4-nograph variant tests degradation). One rep per cell (1 rep/cell) for baseline discovery. **Experimental Control (Frozen Surfaces)** Frozen during this eval: checker (`eval-b/bin/check`, two-axis deterministic-first), fixture (6-ADR Ruby webhook-relay), scenarios (W1–W3, R1–R4, R4-nograph), judge rubric (`judge-rubric.md` for axes-b evaluation of W1–W3). Varying: model tiers (haiku/sonnet) to establish tier-specific performance. Run mode: headless-only (`claude -p` per rep with cwd = sandbox, so real SessionStart hook fires — in-session Agent-tool subagents are invalid here). Rep count 1 chosen for baseline discovery, not for confidence (variance is expected at 1 rep/cell). ## Validity and Limitations **How to Interpret These Results** This is a **held-out baseline** (not yet optimized against). The 5/8 / 0/8 numbers mean "wording iteration is possible and worthwhile" — they support the prompting-issue hypothesis, not final deployment readiness. At 1 rep per cell, expect variance: haiku W2 flickered to axis-a PASS on re-run, demonstrating that single-rep cells are noisy. Do not cite sonnet's 5/8 as "80% accuracy" — the two failures (W3, R1) are near-misses (sonnet consults, then misses one output), not blind spots, and axis-specific targeting (H3 for W3, H1 for R1 — see the hypotheses note) suggests wording fixes are available. **Weaknesses of This Eval (Its Ladder Level)** Single fixture (Ruby webhook-relay) in one language — generalization to other codebases unknown. 1 rep/cell: high noise floor, flickers observed (haiku W2 axis-a, sonnet W1 across runs). No ambiguity axis: all scenarios are clear-cue boundary cases (not a progression from clear to ambiguous to over-trigger territory). Single distractor scenario (R2) — cannot assess false-positive rate under heavy decision-clutter. Axis-b evaluation of W1–W3 relies on a frozen rubric (`judge-rubric.md`); axis-a is mechanical (tool_use), so it's the reliable axis. Open question: whether in-session subagents could ever validly substitute for part of this measurement (suspected invalid due to hook-context requirement, but not definitively tested). This baseline is now obsolete — the 2026-07-04 wording loop closed the gap; see [[os-adr-eval-b-wording-experiment-hypotheses]] for the follow-up grid and the upgraded numbers. ## Confirmation Run (2026-07-04) Full grid re-run (2026-07-04) after fixing stale plugin caches (os-adr's installed cache had drifted — missing SessionStart hook + 3 CLIs) and switching skill registration to namespaced commands (`/os-adr:find`, not bare `/find`). **Baseline confirmed: haiku 0/8, sonnet 5/8 with the same two behavioral failures (W3, R1).** All 16 cells reproduced (W2/sonnet needed clean re-run after harness error and PASSed on re-run, matching baseline). Confirmation run added: **Sonnet W3 is an axis-b failure** (A:PASS / B:FAIL) — consults the ADR system, then doesn't propose recording the reversal; target is create-skill's "when to record" guidance, not trigger salience. **R4-nograph/sonnet PASSed** (expected FAIL) — found the correct ADR without the graph layer this rep; at 1 rep this weakens (doesn't refute) graph-layer-value evidence. **Haiku W2** flickered to axis-a PASS — keep as lower-tier canary in follow-up loops. **Variance is real at 1 rep/cell** (haiku W2 axis-a and sonnet W1 flipped between attempts). ## Deployment and Evolution **Good-Enough Gate** This baseline does NOT clear deployment. It is a held-out measurement establishing the prompting/wording issue hypothesis. The 5/8 / 0/8 indicates that wording optimization is possible (near-misses, not blind spots), and the follow-up wording loop (2026-07-04) confirms this — final grid improved to sonnet 8/8, haiku 7/8. Do not deploy based on this 5/8 / 0/8 baseline; consult the follow-up loop results in [[os-adr-eval-b-wording-experiment-hypotheses]] for the trained-up numbers and the gate for real-project rollout. **Hardening Path / Next Measurement** The follow-up wording experiment (2026-07-04) is complete — see [[os-adr-eval-b-wording-experiment-hypotheses]] for results and open questions (channel ablation never run; R4-nograph no longer differentiates). The next hardening step is Eval C, an ambiguity-ladder discrimination eval testing whether the model mistakes clear-cue for ambiguous-cue; see [[eval-methodology-ladder]] for the progression and [[eval-methodology-irl-feedback-loop]] for production validation. ## Source - cc-os repo, `plugins/os-adr/eval-b/` (harness) and `plugins/os-adr/eval-b/README.md` (status) - Full TSV: `/tmp/adr-eval-b-grid/results.tsv` (ephemeral — not committed, per harness design) - Doc updates committed at cc-os `5b399d5` ## Related - [[os-adr-eval-b-wording-experiment-hypotheses]] — follow-up wording loop (2026-07-04) with upgraded numbers and deployment gate - [[eval-methodology-ladder]] — evaluation ladder approach and progression logic - [[running-autoresearch-skill-evals]] — procedure for skill-wording eval loops - [[eval-methodology-irl-feedback-loop]] — production validation and audit backlog