SecondBrain/os-adr-eval-c-frozen-grid-results.md

type	title	summary	tags	scope	last_updated	date	related	source
eval-results	os-adr Eval C frozen grid (2026-07-06) — ambiguity-ladder discrimination baseline	First held-out grid on the taskq fixture — 12/12 cells PASS at 3 reps/cell (haiku 18/18 reps, sonnet 17/18); zero over-trigger ADR creation on any negative at any ladder level. The Eval B wording generalizes to a new fixture, domain, and paired negatives.	type/eval-results domain/llm-evaluation tool/os-adr project/cc-os	global	2026-07-06	2026-07-06	os-adr-eval-b-wording-experiment-hypotheses os-adr-eval-b-grid-results-and-observations eval-methodology-ladder eval-methodology-irl-feedback-loop running-autoresearch-skill-evals	cc-os

os-adr Eval C frozen grid (2026-07-06) — ambiguity-ladder discrimination baseline

Results Grid/Threshold

Passing criterion: per-cell majority (≥2 of 3 reps). Positives PASS on both axes (a: consultation, b: created / cited-governing / judge-proposed). Negatives PASS on axis (b) only — the sole fail line is unneeded ADR creation; consultation (A:yes/no) and truthful citation are informational.

Level	Positive sonnet	Positive haiku	Negative sonnet	Negative haiku
L1 (explicit)	PASS 2/3	PASS 3/3	PASS 3/3 (A:yes 3/3)	PASS 3/3 (A:yes 3/3)
L2 (moderate)	PASS 3/3	PASS 3/3	PASS 3/3 (A:yes 2/3)	PASS 3/3 (A:yes 3/3)
L3 (conceptual)	PASS 3/3	PASS 3/3	PASS 3/3 (A:yes 3/3)	PASS 3/3 (A:yes 3/3)

Summary: 12/12 cells PASS; 35/36 reps PASS (haiku 18/18, sonnet 17/18). No negative rep at any level created an ADR — zero over-triggering on either tier. No positive rep created a duplicate ADR either: every positive pass was via governing-ADR citation (16) or judge-recognized recording proposal (1). Sonnet's single FAIL rep (P1-L1) is a scoring-boundary artifact: it consulted, implemented exactly the ADR-0002 policy, and flagged the un-implemented ADR-0005 escalation — but named only 0005 (not 0002) in its final message, so the strict final-message citation check fell through to the judge. Behaviorally clean; a measurement-boundary flicker analogous to Eval B's W3.

Measurement Setup

Scenarios Tested

Run-set only (job-execution domain, target ADR 0002 with 0001→0002 supersession distractor): P1/N1-L1, P2/N2-L2, P3/N3-L3. Positives sit in already-decided territory — the task restates what an Accepted ADR already decided — so correctness is find → cite → comply. Negatives are non-decisional-but-tempting (test/document/audit framing in ADR-covered domains); correctness is doing the task without fabricating a decision record. The reserve-set (notifications domain) was not touched.

Fixture and Sampling

New Python taskq async job-queue fixture (vs Eval B's Ruby webhook-relay) with a 6-ADR history generated by the plugin's own CLIs; trigger-phrased "Architecture decisions" CLAUDE.md section copied verbatim from eval-b's fixture. 3 reps/cell (36 reps), chosen because 1 rep/cell was demonstrably noisy in both Eval B campaigns. Headless-only (bin/run, fresh claude -p per rep, cwd = sandbox, real SessionStart hook verified in every transcript). All reps counted — including the pre-fix canary rep, rescored (not re-run) after the checker conformance fix.

Experimental Control (Frozen Surfaces)

Scenario Task blocks, fixtures, and judge-rubric.md untouched throughout. Two harness defects were found on the first live rep and fixed BEFORE the grid (1 rep exposure, transcript preserved and rescored):

1. bin/run rep-loop bug — set -euo pipefail + piping bin/check (exit 1 on FAIL) into tee aborted a cell's remaining reps on the first FAIL.
2. Positive axis-b conformance bug — the checker required ADR creation/proposal, contradicting the README's own design ("consulting and/or recording is correct") and the shipped plugin wording: with the positives in already-decided territory, creating an ADR would duplicate the record — exactly what the negatives punish. Fixed by adding a mechanical cited-governing PASS branch (final message cites the governing ADR id) before the unchanged frozen-judge fallback; self-test gained compliance-must-PASS and wrong-citation-must-FAIL guards.

Under the pre-fix checker the grid would have scored ~1/18 positives — all false negatives. B-via distribution proves it: cited-governing 16, judge-proposed 1, created 0.

Validity and Limitations

How to Interpret These Results

This is a held-out measurement — no wording was tuned against Eval C before or during this grid, so unlike Eval B's final numbers these are not training-set scores. What they support: the trigger-conditioned wording shipped from the Eval B loop generalizes across fixture, language (Ruby→Python), domain, decreasing cue explicitness (L1→L3), and — the core discrimination test — paired negatives, on both tiers. The Eval B overfit concern is substantially retired at these ladder levels. What they do not support: behavior on genuinely novel-decision positives (see weaknesses), other languages/project shapes, or production conditions.

Weaknesses of This Eval (Its Ladder Level)

The grid saturated — no ladder level differentiated the tiers, so the ceiling wasn't found; harder rungs (heavier distractor clutter, conflicting-ADR tasks requiring supersession, genuinely novel decisions requiring creation) need new scenario authoring. All run-set positives are compliance-shaped (decision already recorded); the eval therefore never measured unprompted creation of a genuinely new decision — Eval B's W1–W3 remain the only (training-set) evidence for that. The cited-governing check reads the final message only; sonnet's one FAIL shows it can under-credit compliant behavior (~1-in-18 observed). Single fixture per set. Checker fixes were made mid-first-rep (documented above); scenario metadata lines were edited but Task blocks never exposed.

Deployment and Evolution

Good-Enough Gate

Both tiers clear the discrimination gate: zero over-trigger creation across 18 negative reps means the daily-use worry ("will it spam ADRs?") is not supported at any measured ambiguity level. Combined with Eval B (post-wording) and the W3 stability check, os-adr is cleared for real-project rollout on both sonnet and haiku, keeping the known haiku W3 caveat (~1-in-4/5 recording-offer flicker on supersession). Contamination protocol stands: the run-set is now the baseline; the moment anyone tunes wording against these results, measurement moves to the frozen reserve-set.

Hardening Path / Next Measurement

Not another lab rung — the saturated grid says the next signal is production: proceed with the locked rollout order (pilot projects via /os-adr:migrate, then cc-os retrofit) and stand up the eval-methodology-irl-feedback-loop session-audit backlog; real misses become the scenarios for any future rung (novel-decision positives, supersession-conflict positives, heavy-clutter negatives — reserve-set discipline applies).

Related

• os-adr-eval-b-wording-experiment-hypotheses — the wording loop whose generalization this grid tested
• os-adr-eval-b-grid-results-and-observations — original held-out baseline (pre-wording)
• eval-methodology-ladder — ladder design this eval instantiates
• eval-methodology-irl-feedback-loop — where measurement goes after a saturated grid
• running-autoresearch-skill-evals — run-mode and rep-count discipline followed here