os-adr Eval B grid results (2026-07-03 baseline)

Results Grid/Threshold

Eval B measures whether a model notices on its own that an Architecture Decision Record system is relevant — without being explicitly told to invoke it.

Passing criterion: Model must (a) touch the ADR system unprompted (mechanical detection from tool_use), and (b) produce correct outcome (right ADR cited for retrieval, or new ADR actually proposed for writes). For the 2026-07-03 baseline, passing is per-tier all-scenarios aggregate before any wording optimization.

Grid (1 rep/cell):

Scenario	Haiku	Sonnet
W1 (persistence choice)	FAIL	PASS
W2 (convention change)	FAIL	PASS
W3 (reversal)	FAIL	FAIL — doesn't propose recording the decision
R1 (direct conflict)	FAIL	FAIL — misses the direct-conflict retrieval
R2 (distractor)	FAIL	PASS
R3 (mid-task question)	FAIL	PASS
R4 (graph one-hop)	FAIL	PASS
R4-nograph (degradation check)	FAIL (expected)	FAIL (expected)

Summary: Haiku 0/8 (never unprompted-consults the ADR system in any scenario). Sonnet 5/8 (failures: W3 reversal / R1 direct-conflict retrieval). R4-nograph degradation checks failed as expected — only meaningful paired with R4 PASS on the same tier.

Measurement Setup

Hypotheses / Scenarios Tested

Two scenario groups per the frozen shapes in docs/adr-system/06-eval-scenarios.md:

W1–W3 (write-triggers): Should the model propose recording an architecture decision? W1: persistence choice (new decision to record). W2: convention change (decision to change established convention). W3: reversal (decision to reverse prior decision).
R1–R4 (retrieval): Should the model consult existing decisions? R1: direct conflict (new approach contradicts an Accepted ADR). R2: distractor (scenario mentioning decisions but not directly conflicting). R3: mid-task question (ask at decision point, not upfront). R4: one-hop graph reach (correct ADR findable via Graphify graph traversal from affected files).

Correctness axis (b) for R1–R4: cite the correct existing ADR. For W1–W3: actually propose recording a new ADR (narrow rubric via judge-rubric.md, with haiku judge fallback).

Consultation axis (a, all scenarios): did the model touch any os-adr:* / bin/adr-* / docs/adr/ unprompted? Mechanical detection from transcript tool_use blocks.

Fixture and Sampling

Single Ruby fixture: dedicated webhook-relay with 6-ADR history (Superseded pair + near-miss distractors, generated via the plugin's own CLIs). R4's one-hop graph reach uses a real graphify update AST build (model-free, rebuilt via eval-b/bin/build-fixture-graph, never committed; Graphify-less R4-nograph variant tests degradation). One rep per cell (1 rep/cell) for baseline discovery.

Experimental Control (Frozen Surfaces)

Frozen during this eval: checker (eval-b/bin/check, two-axis deterministic-first), fixture (6-ADR Ruby webhook-relay), scenarios (W1–W3, R1–R4, R4-nograph), judge rubric (judge-rubric.md for axes-b evaluation of W1–W3). Varying: model tiers (haiku/sonnet) to establish tier-specific performance. Run mode: headless-only (claude -p per rep with cwd = sandbox, so real SessionStart hook fires — in-session Agent-tool subagents are invalid here). Rep count 1 chosen for baseline discovery, not for confidence (variance is expected at 1 rep/cell).

Validity and Limitations

How to Interpret These Results

This is a held-out baseline (not yet optimized against). The 5/8 / 0/8 numbers mean "wording iteration is possible and worthwhile" — they support the prompting-issue hypothesis, not final deployment readiness. At 1 rep per cell, expect variance: haiku W2 flickered to axis-a PASS on re-run, demonstrating that single-rep cells are noisy. Do not cite sonnet's 5/8 as "80% accuracy" — the two failures (W3, R1) are near-misses (sonnet consults, then misses one output), not blind spots, and axis-specific targeting (H3 for W3, H1 for R1 — see the hypotheses note) suggests wording fixes are available.

Weaknesses of This Eval (Its Ladder Level)

Single fixture (Ruby webhook-relay) in one language — generalization to other codebases unknown. 1 rep/cell: high noise floor, flickers observed (haiku W2 axis-a, sonnet W1 across runs). No ambiguity axis: all scenarios are clear-cue boundary cases (not a progression from clear to ambiguous to over-trigger territory). Single distractor scenario (R2) — cannot assess false-positive rate under heavy decision-clutter. Axis-b evaluation of W1–W3 relies on a frozen rubric (judge-rubric.md); axis-a is mechanical (tool_use), so it's the reliable axis. Open question: whether in-session subagents could ever validly substitute for part of this measurement (suspected invalid due to hook-context requirement, but not definitively tested). This baseline is now obsolete — the 2026-07-04 wording loop closed the gap; see os-adr-eval-b-wording-experiment-hypotheses for the follow-up grid and the upgraded numbers.

Confirmation Run (2026-07-04)

Full grid re-run (2026-07-04) after fixing stale plugin caches (os-adr's installed cache had drifted — missing SessionStart hook + 3 CLIs) and switching skill registration to namespaced commands (/os-adr:find, not bare /find). Baseline confirmed: haiku 0/8, sonnet 5/8 with the same two behavioral failures (W3, R1). All 16 cells reproduced (W2/sonnet needed clean re-run after harness error and PASSed on re-run, matching baseline).

Confirmation run added: Sonnet W3 is an axis-b failure (A:PASS / B:FAIL) — consults the ADR system, then doesn't propose recording the reversal; target is create-skill's "when to record" guidance, not trigger salience. R4-nograph/sonnet PASSed (expected FAIL) — found the correct ADR without the graph layer this rep; at 1 rep this weakens (doesn't refute) graph-layer-value evidence. Haiku W2 flickered to axis-a PASS — keep as lower-tier canary in follow-up loops. Variance is real at 1 rep/cell (haiku W2 axis-a and sonnet W1 flipped between attempts).

Deployment and Evolution

Good-Enough Gate

This baseline does NOT clear deployment. It is a held-out measurement establishing the prompting/wording issue hypothesis. The 5/8 / 0/8 indicates that wording optimization is possible (near-misses, not blind spots), and the follow-up wording loop (2026-07-04) confirms this — final grid improved to sonnet 8/8, haiku 7/8. Do not deploy based on this 5/8 / 0/8 baseline; consult the follow-up loop results in os-adr-eval-b-wording-experiment-hypotheses for the trained-up numbers and the gate for real-project rollout.

Hardening Path / Next Measurement

The follow-up wording experiment (2026-07-04) is complete — see os-adr-eval-b-wording-experiment-hypotheses for results and open questions (channel ablation never run; R4-nograph no longer differentiates). The next hardening step is Eval C, an ambiguity-ladder discrimination eval testing whether the model mistakes clear-cue for ambiguous-cue; see eval-methodology-ladder for the progression and eval-methodology-irl-feedback-loop for production validation.

Source

cc-os repo, plugins/os-adr/eval-b/ (harness) and plugins/os-adr/eval-b/README.md (status)
Full TSV: /tmp/adr-eval-b-grid/results.tsv (ephemeral — not committed, per harness design)
Doc updates committed at cc-os 5b399d5

os-adr-eval-b-wording-experiment-hypotheses — follow-up wording loop (2026-07-04) with upgraded numbers and deployment gate
eval-methodology-ladder — evaluation ladder approach and progression logic
running-autoresearch-skill-evals — procedure for skill-wording eval loops
eval-methodology-irl-feedback-loop — production validation and audit backlog

8.3 KiB Raw Blame History Unescape Escape