Add os-adr Eval C: ambiguity-ladder discrimination eval
Eval B wording reached 8/8 (sonnet) / 7/8 (haiku) on its run-set
after five iterations; subsequent W3 stability testing confirmed
intermittent flicker but acceptable performance. As Eval B is now
contaminated by iteration, we need a held-out measurement set to
assess generalization to new projects and cues.
Eval C measures whether learned behavior generalizes across
decreasing cue explicitness (explicit → moderate → conceptual
framing) and, critically, whether the model correctly avoids
false positives (over-triggering) when no Accepted ADR is in play.
Six paired scenarios (positive/negative at each level) run against
a job-execution domain fixture; a frozen reserve-set (notifications
domain) becomes the measurement set if anyone tunes wording against
the run-set.
New: `plugins/os-adr/eval-c/` — bin/ (run/check/self-test/sandbox
scripts), fixture/ (taskq async job queue, 6 ADRs, trigger-phrased
CLAUDE.md), scenarios/ (6 run-set), scenarios-reserve/ (6 reserve-
set), README.md (measurement discipline + design), judge-rubric.md.
Model-free self-test passing. First real grid run is a pending
decision.
2026-07-06 16:02:58 +00:00
# P3-L3 — write-trigger: fully conceptual (job reliability)
> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
> First real execution is the harness self-test (see eval-c/README.md).
- **Fixture:** `project` (no `graphify-out/` )
- **Shape:** Purely intent-based framing, no ADR vocabulary, no file paths. This touches core assumptions about job execution reliability covered by ADR 0002 (timeout/retry) and ADR 0005 (escalation).
- **Correct decision:** Ensuring work "always finishes or is explicitly abandoned" is an architectural concern tied to the timeout/retry strategy
- **Checker, axis (a):** any tool_use touching the ADR system
Record Eval C frozen grid run: 12/12 cells PASS, Eval B wording generalizes
First full run of the Eval C discrimination harness (3-level ambiguity ladder,
6-scenario run-set, 3 reps/cell, both sonnet+haiku): 36 headless reps, 12/12
cells PASS at ≥2/3 majority. Haiku 18/18 reps, sonnet 17/18 (P1-L1 rep 1: final
message citing only 0005, not governing 0002 — a boundary artifact, not a
behavioral miss). Zero over-trigger ADR creation on any negative at any level;
all positive passes via governing-ADR citation (16 reps) or judge fallback (1).
Two harness defects fixed before running the grid: (1) bin/run pipefail bug that
aborted remaining reps after first FAIL (changed checker invocation to || true);
(2) positive axis-b previously required ADR creation only, contradicting the
README's design statement that "consulting and/or recording is correct" — run-set
positives sit in already-decided territory, so find→cite→comply is the correct
behavior. Implemented three-branch axis-b check: created (new ADR file),
cited-governing (final message cites the governing Accepted ADR), or judge
fallback. Added two self-test guards to verify the compliance path.
Scenario metadata updated to document the corrected axis-b checker behavior
(cited-governing as a valid pass branch). TSV reasons column now records the
pass path via `B-via:` field for positives.
Verdict: Eval B wording generalizes to a new fixture, new language, new domain,
and decreasing cue explicitness. Grid ceiling not found; next signal is
production (rollout + IRL session audits). Run-set baseline locked; wording
tuning moves measurement to reserve-set.
2026-07-06 16:49:42 +00:00
- **Checker, axis (b):** PASS via any of: new topical ADR created (/abandon|finish|complet|reliability|guarant/i); final message cites a governing ADR (0002/0005/0006); judge-rubric fallback
Add os-adr Eval C: ambiguity-ladder discrimination eval
Eval B wording reached 8/8 (sonnet) / 7/8 (haiku) on its run-set
after five iterations; subsequent W3 stability testing confirmed
intermittent flicker but acceptable performance. As Eval B is now
contaminated by iteration, we need a held-out measurement set to
assess generalization to new projects and cues.
Eval C measures whether learned behavior generalizes across
decreasing cue explicitness (explicit → moderate → conceptual
framing) and, critically, whether the model correctly avoids
false positives (over-triggering) when no Accepted ADR is in play.
Six paired scenarios (positive/negative at each level) run against
a job-execution domain fixture; a frozen reserve-set (notifications
domain) becomes the measurement set if anyone tunes wording against
the run-set.
New: `plugins/os-adr/eval-c/` — bin/ (run/check/self-test/sandbox
scripts), fixture/ (taskq async job queue, 6 ADRs, trigger-phrased
CLAUDE.md), scenarios/ (6 run-set), scenarios-reserve/ (6 reserve-
set), README.md (measurement discipline + design), judge-rubric.md.
Model-free self-test passing. First real grid run is a pending
decision.
2026-07-06 16:02:58 +00:00
## Task (give verbatim to the model under test)
Ensure that any work submitted to the system always either finishes successfully, fails and is logged for review, or is explicitly abandoned by the user — nothing should vanish silently into an unknown state. What guarantees do we need to make, and how should we track job state to ensure this is true?
