SecondBrain/os-adr-eval-c-frozen-grid-results.md

---
type: eval-results
title: os-adr Eval C frozen grid (2026-07-06) — ambiguity-ladder discrimination baseline
summary: First held-out grid on the taskq fixture — 12/12 cells PASS at 3 reps/cell (haiku 18/18 reps, sonnet 17/18); zero over-trigger ADR creation on any negative at any ladder level. The Eval B wording generalizes to a new fixture, domain, and paired negatives.
tags:
  - type/eval-results
  - domain/llm-evaluation
  - tool/os-adr
  - project/cc-os
scope: global
last_updated: 2026-07-06
date: 2026-07-06
related:
  - os-adr-eval-b-wording-experiment-hypotheses
  - os-adr-eval-b-grid-results-and-observations
  - eval-methodology-ladder
  - eval-methodology-irl-feedback-loop
  - running-autoresearch-skill-evals
source: cc-os
---

# os-adr Eval C frozen grid (2026-07-06) — ambiguity-ladder discrimination baseline

## Results Grid/Threshold

**Passing criterion:** per-cell majority (≥2 of 3 reps). Positives PASS on both axes (a: consultation, b: created / cited-governing / judge-proposed). Negatives PASS on axis (b) only — the sole fail line is unneeded ADR creation; consultation (`A:yes/no`) and truthful citation are informational.

| Level | Positive sonnet | Positive haiku | Negative sonnet | Negative haiku |
|---|---|---|---|---|
| L1 (explicit) | PASS 2/3 | PASS 3/3 | PASS 3/3 (A:yes 3/3) | PASS 3/3 (A:yes 3/3) |
| L2 (moderate) | PASS 3/3 | PASS 3/3 | PASS 3/3 (A:yes 2/3) | PASS 3/3 (A:yes 3/3) |
| L3 (conceptual) | PASS 3/3 | PASS 3/3 | PASS 3/3 (A:yes 3/3) | PASS 3/3 (A:yes 3/3) |

**Summary:** 12/12 cells PASS; 35/36 reps PASS (haiku 18/18, sonnet 17/18). No negative rep at any level created an ADR — zero over-triggering on either tier. No positive rep created a duplicate ADR either: every positive pass was via governing-ADR citation (16) or judge-recognized recording proposal (1). Sonnet's single FAIL rep (P1-L1) is a scoring-boundary artifact: it consulted, implemented exactly the ADR-0002 policy, and flagged the un-implemented ADR-0005 escalation — but named only 0005 (not 0002) in its final message, so the strict final-message citation check fell through to the judge. Behaviorally clean; a measurement-boundary flicker analogous to Eval B's W3.

## Measurement Setup

**Scenarios Tested**

Run-set only (job-execution domain, target ADR 0002 with 0001→0002 supersession distractor): P1/N1-L1, P2/N2-L2, P3/N3-L3. Positives sit in already-decided territory — the task restates what an Accepted ADR already decided — so correctness is find → cite → comply. Negatives are non-decisional-but-tempting (test/document/audit framing in ADR-covered domains); correctness is doing the task without fabricating a decision record. The reserve-set (notifications domain) was not touched.

**Fixture and Sampling**

New Python `taskq` async job-queue fixture (vs Eval B's Ruby webhook-relay) with a 6-ADR history generated by the plugin's own CLIs; trigger-phrased "Architecture decisions" CLAUDE.md section copied verbatim from eval-b's fixture. 3 reps/cell (36 reps), chosen because 1 rep/cell was demonstrably noisy in both Eval B campaigns. Headless-only (`bin/run`, fresh `claude -p` per rep, cwd = sandbox, real SessionStart hook verified in every transcript). All reps counted — including the pre-fix canary rep, rescored (not re-run) after the checker conformance fix.

**Experimental Control (Frozen Surfaces)**

Scenario Task blocks, fixtures, and `judge-rubric.md` untouched throughout. Two harness defects were found on the first live rep and fixed BEFORE the grid (1 rep exposure, transcript preserved and rescored):
1. `bin/run` rep-loop bug — `set -euo pipefail` + piping `bin/check` (exit 1 on FAIL) into `tee` aborted a cell's remaining reps on the first FAIL.
2. Positive axis-b conformance bug — the checker required ADR creation/proposal, contradicting the README's own design ("consulting and/or recording is correct") and the shipped plugin wording: with the positives in already-decided territory, creating an ADR would duplicate the record — exactly what the negatives punish. Fixed by adding a mechanical cited-governing PASS branch (final message cites the governing ADR id) before the unchanged frozen-judge fallback; self-test gained compliance-must-PASS and wrong-citation-must-FAIL guards.

Under the pre-fix checker the grid would have scored ~1/18 positives — all false negatives. B-via distribution proves it: cited-governing 16, judge-proposed 1, created 0.

## Validity and Limitations

**How to Interpret These Results**

This is a **held-out measurement** — no wording was tuned against Eval C before or during this grid, so unlike Eval B's final numbers these are not training-set scores. What they support: the trigger-conditioned wording shipped from the Eval B loop generalizes across fixture, language (Ruby→Python), domain, decreasing cue explicitness (L1→L3), and — the core discrimination test — paired negatives, on both tiers. The Eval B overfit concern is substantially retired at these ladder levels. What they do not support: behavior on genuinely novel-decision positives (see weaknesses), other languages/project shapes, or production conditions.

**Weaknesses of This Eval (Its Ladder Level)**

The grid saturated — no ladder level differentiated the tiers, so the ceiling wasn't found; harder rungs (heavier distractor clutter, conflicting-ADR tasks requiring supersession, genuinely novel decisions requiring creation) need new scenario authoring. All run-set positives are compliance-shaped (decision already recorded); the eval therefore never measured unprompted *creation* of a genuinely new decision — Eval B's W1–W3 remain the only (training-set) evidence for that. The cited-governing check reads the final message only; sonnet's one FAIL shows it can under-credit compliant behavior (~1-in-18 observed). Single fixture per set. Checker fixes were made mid-first-rep (documented above); scenario metadata lines were edited but Task blocks never exposed.

## Deployment and Evolution

**Good-Enough Gate**

Both tiers clear the discrimination gate: zero over-trigger creation across 18 negative reps means the daily-use worry ("will it spam ADRs?") is not supported at any measured ambiguity level. Combined with Eval B (post-wording) and the W3 stability check, os-adr is cleared for real-project rollout on both sonnet and haiku, keeping the known haiku W3 caveat (~1-in-4/5 recording-offer flicker on supersession). Contamination protocol stands: the run-set is now the baseline; the moment anyone tunes wording against these results, measurement moves to the frozen reserve-set.

**Hardening Path / Next Measurement**

Not another lab rung — the saturated grid says the next signal is production: proceed with the locked rollout order (pilot projects via `/os-adr:migrate`, then cc-os retrofit) and stand up the [[eval-methodology-irl-feedback-loop]] session-audit backlog; real misses become the scenarios for any future rung (novel-decision positives, supersession-conflict positives, heavy-clutter negatives — reserve-set discipline applies).

## Related

- [[os-adr-eval-b-wording-experiment-hypotheses]] — the wording loop whose generalization this grid tested
- [[os-adr-eval-b-grid-results-and-observations]] — original held-out baseline (pre-wording)
- [[eval-methodology-ladder]] — ladder design this eval instantiates
- [[eval-methodology-irl-feedback-loop]] — where measurement goes after a saturated grid
- [[running-autoresearch-skill-evals]] — run-mode and rep-count discipline followed here