jared
/
cc-os
1
0
Fork 0
Code Issues 19 Pull Requests Packages Projects Releases Wiki Activity
cc-os/plugins/os-adr/eval-c/results/2026-07-06-frozen-grid.tsv

37 lines
1.9 KiB
Plaintext
Raw Normal View History

Record Eval C frozen grid run: 12/12 cells PASS, Eval B wording generalizes First full run of the Eval C discrimination harness (3-level ambiguity ladder, 6-scenario run-set, 3 reps/cell, both sonnet+haiku): 36 headless reps, 12/12 cells PASS at ≥2/3 majority. Haiku 18/18 reps, sonnet 17/18 (P1-L1 rep 1: final message citing only 0005, not governing 0002 — a boundary artifact, not a behavioral miss). Zero over-trigger ADR creation on any negative at any level; all positive passes via governing-ADR citation (16 reps) or judge fallback (1). Two harness defects fixed before running the grid: (1) bin/run pipefail bug that aborted remaining reps after first FAIL (changed checker invocation to || true); (2) positive axis-b previously required ADR creation only, contradicting the README's design statement that "consulting and/or recording is correct" — run-set positives sit in already-decided territory, so find→cite→comply is the correct behavior. Implemented three-branch axis-b check: created (new ADR file), cited-governing (final message cites the governing Accepted ADR), or judge fallback. Added two self-test guards to verify the compliance path. Scenario metadata updated to document the corrected axis-b checker behavior (cited-governing as a valid pass branch). TSV reasons column now records the pass path via `B-via:` field for positives. Verdict: Eval B wording generalizes to a new fixture, new language, new domain, and decreasing cue explicitness. Grid ceiling not found; next signal is production (rollout + IRL session audits). Run-set baseline locked; wording tuning moves measurement to reserve-set.
2026-07-06 16:49:42 +00:00
N1-L1-execution haiku A:yes B:PASS PASS
N1-L1-execution haiku A:yes B:PASS PASS
N1-L1-execution haiku A:yes B:PASS PASS
N1-L1-execution sonnet A:yes B:PASS PASS
N1-L1-execution sonnet A:yes B:PASS PASS
N1-L1-execution sonnet A:yes B:PASS PASS
N2-L2-execution haiku A:yes B:PASS PASS
N2-L2-execution haiku A:yes B:PASS PASS
N2-L2-execution haiku A:yes B:PASS PASS
N2-L2-execution sonnet A:no B:PASS PASS
N2-L2-execution sonnet A:yes B:PASS PASS
N2-L2-execution sonnet A:yes B:PASS PASS
N3-L3-execution haiku A:yes B:PASS PASS
N3-L3-execution haiku A:yes B:PASS PASS
N3-L3-execution haiku A:yes B:PASS PASS
N3-L3-execution sonnet A:yes B:PASS PASS
N3-L3-execution sonnet A:yes B:PASS PASS
N3-L3-execution sonnet A:yes B:PASS PASS
P1-L1-execution haiku A:PASS B:PASS PASS B-via:cited-governing
P1-L1-execution haiku A:PASS B:PASS PASS B-via:cited-governing
P1-L1-execution haiku A:PASS B:PASS PASS B-via:cited-governing
P1-L1-execution sonnet A:PASS B:FAIL FAIL no new ADR, no governing-ADR citation /\b0002\b/ in final message; judge: does not propose recording
P1-L1-execution sonnet A:PASS B:PASS PASS B-via:cited-governing
P1-L1-execution sonnet A:PASS B:PASS PASS B-via:cited-governing
P2-L2-execution haiku A:PASS B:PASS PASS B-via:cited-governing
P2-L2-execution haiku A:PASS B:PASS PASS B-via:cited-governing
P2-L2-execution haiku A:PASS B:PASS PASS B-via:judge-proposed
P2-L2-execution sonnet A:PASS B:PASS PASS B-via:cited-governing
P2-L2-execution sonnet A:PASS B:PASS PASS B-via:cited-governing
P2-L2-execution sonnet A:PASS B:PASS PASS B-via:cited-governing
P3-L3-execution haiku A:PASS B:PASS PASS B-via:cited-governing
P3-L3-execution haiku A:PASS B:PASS PASS B-via:cited-governing
P3-L3-execution haiku A:PASS B:PASS PASS B-via:cited-governing
P3-L3-execution sonnet A:PASS B:PASS PASS B-via:cited-governing
P3-L3-execution sonnet A:PASS B:PASS PASS B-via:cited-governing
P3-L3-execution sonnet A:PASS B:PASS PASS B-via:cited-governing