Add os-adr Eval C: ambiguity-ladder discrimination eval
Eval B wording reached 8/8 (sonnet) / 7/8 (haiku) on its run-set
after five iterations; subsequent W3 stability testing confirmed
intermittent flicker but acceptable performance. As Eval B is now
contaminated by iteration, we need a held-out measurement set to
assess generalization to new projects and cues.
Eval C measures whether learned behavior generalizes across
decreasing cue explicitness (explicit → moderate → conceptual
framing) and, critically, whether the model correctly avoids
false positives (over-triggering) when no Accepted ADR is in play.
Six paired scenarios (positive/negative at each level) run against
a job-execution domain fixture; a frozen reserve-set (notifications
domain) becomes the measurement set if anyone tunes wording against
the run-set.
New: `plugins/os-adr/eval-c/` — bin/ (run/check/self-test/sandbox
scripts), fixture/ (taskq async job queue, 6 ADRs, trigger-phrased
CLAUDE.md), scenarios/ (6 run-set), scenarios-reserve/ (6 reserve-
set), README.md (measurement discipline + design), judge-rubric.md.
Model-free self-test passing. First real grid run is a pending
decision.
2026-07-06 16:02:58 +00:00
# P1-L1 — write-trigger: explicit cue (job timeout/retry)
> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
> First real execution is the harness self-test (see eval-c/README.md).
- **Fixture:** `project` (no `graphify-out/` )
- **Shape:** Explicit cues — file path named, decision vocabulary explicit
- **Correct decision:** Should recognize this is deciding/changing the job timeout + retry strategy, covered by ADR 0002
- **Checker, axis (a):** any tool_use touching the ADR system (`os-adr:*` skill, `bin/adr-*` CLI, or read/glob of `docs/adr/` )
Record Eval C frozen grid run: 12/12 cells PASS, Eval B wording generalizes
First full run of the Eval C discrimination harness (3-level ambiguity ladder,
6-scenario run-set, 3 reps/cell, both sonnet+haiku): 36 headless reps, 12/12
cells PASS at ≥2/3 majority. Haiku 18/18 reps, sonnet 17/18 (P1-L1 rep 1: final
message citing only 0005, not governing 0002 — a boundary artifact, not a
behavioral miss). Zero over-trigger ADR creation on any negative at any level;
all positive passes via governing-ADR citation (16 reps) or judge fallback (1).
Two harness defects fixed before running the grid: (1) bin/run pipefail bug that
aborted remaining reps after first FAIL (changed checker invocation to || true);
(2) positive axis-b previously required ADR creation only, contradicting the
README's design statement that "consulting and/or recording is correct" — run-set
positives sit in already-decided territory, so find→cite→comply is the correct
behavior. Implemented three-branch axis-b check: created (new ADR file),
cited-governing (final message cites the governing Accepted ADR), or judge
fallback. Added two self-test guards to verify the compliance path.
Scenario metadata updated to document the corrected axis-b checker behavior
(cited-governing as a valid pass branch). TSV reasons column now records the
pass path via `B-via:` field for positives.
Verdict: Eval B wording generalizes to a new fixture, new language, new domain,
and decreasing cue explicitness. Grid ceiling not found; next signal is
production (rollout + IRL session audits). Run-set baseline locked; wording
tuning moves measurement to reserve-set.
2026-07-06 16:49:42 +00:00
- **Checker, axis (b):** PASS via any of: new topical ADR created (/timeout|retry|backoff|exponential|attempt/i); final message cites governing ADR 0002 (find→cite→comply is correct — the decision is already recorded); judge-rubric fallback
Add os-adr Eval C: ambiguity-ladder discrimination eval
Eval B wording reached 8/8 (sonnet) / 7/8 (haiku) on its run-set
after five iterations; subsequent W3 stability testing confirmed
intermittent flicker but acceptable performance. As Eval B is now
contaminated by iteration, we need a held-out measurement set to
assess generalization to new projects and cues.
Eval C measures whether learned behavior generalizes across
decreasing cue explicitness (explicit → moderate → conceptual
framing) and, critically, whether the model correctly avoids
false positives (over-triggering) when no Accepted ADR is in play.
Six paired scenarios (positive/negative at each level) run against
a job-execution domain fixture; a frozen reserve-set (notifications
domain) becomes the measurement set if anyone tunes wording against
the run-set.
New: `plugins/os-adr/eval-c/` — bin/ (run/check/self-test/sandbox
scripts), fixture/ (taskq async job queue, 6 ADRs, trigger-phrased
CLAUDE.md), scenarios/ (6 run-set), scenarios-reserve/ (6 reserve-
set), README.md (measurement discipline + design), judge-rubric.md.
Model-free self-test passing. First real grid run is a pending
decision.
2026-07-06 16:02:58 +00:00
## Task (give verbatim to the model under test)
The job execution engine in `taskq/execution.py` currently has no timeout or retry logic. Add a 5-minute timeout with exponential backoff retries (starting at 1 second, up to 3 total attempts) so that transient failures have a chance to recover. Should we use asyncio timeouts with a separate retry loop, or wrap the handler call?
