First full run of the Eval C discrimination harness (3-level ambiguity ladder,
6-scenario run-set, 3 reps/cell, both sonnet+haiku): 36 headless reps, 12/12
cells PASS at ≥2/3 majority. Haiku 18/18 reps, sonnet 17/18 (P1-L1 rep 1: final
message citing only 0005, not governing 0002 — a boundary artifact, not a
behavioral miss). Zero over-trigger ADR creation on any negative at any level;
all positive passes via governing-ADR citation (16 reps) or judge fallback (1).
Two harness defects fixed before running the grid: (1) bin/run pipefail bug that
aborted remaining reps after first FAIL (changed checker invocation to || true);
(2) positive axis-b previously required ADR creation only, contradicting the
README's design statement that "consulting and/or recording is correct" — run-set
positives sit in already-decided territory, so find→cite→comply is the correct
behavior. Implemented three-branch axis-b check: created (new ADR file),
cited-governing (final message cites the governing Accepted ADR), or judge
fallback. Added two self-test guards to verify the compliance path.
Scenario metadata updated to document the corrected axis-b checker behavior
(cited-governing as a valid pass branch). TSV reasons column now records the
pass path via `B-via:` field for positives.
Verdict: Eval B wording generalizes to a new fixture, new language, new domain,
and decreasing cue explicitness. Grid ceiling not found; next signal is
production (rollout + IRL session audits). Run-set baseline locked; wording
tuning moves measurement to reserve-set.
Eval B wording reached 8/8 (sonnet) / 7/8 (haiku) on its run-set
after five iterations; subsequent W3 stability testing confirmed
intermittent flicker but acceptable performance. As Eval B is now
contaminated by iteration, we need a held-out measurement set to
assess generalization to new projects and cues.
Eval C measures whether learned behavior generalizes across
decreasing cue explicitness (explicit → moderate → conceptual
framing) and, critically, whether the model correctly avoids
false positives (over-triggering) when no Accepted ADR is in play.
Six paired scenarios (positive/negative at each level) run against
a job-execution domain fixture; a frozen reserve-set (notifications
domain) becomes the measurement set if anyone tunes wording against
the run-set.
New: `plugins/os-adr/eval-c/` — bin/ (run/check/self-test/sandbox
scripts), fixture/ (taskq async job queue, 6 ADRs, trigger-phrased
CLAUDE.md), scenarios/ (6 run-set), scenarios-reserve/ (6 reserve-
set), README.md (measurement discipline + design), judge-rubric.md.
Model-free self-test passing. First real grid run is a pending
decision.
jared