jared
/
cc-os
1
0
Fork 0
Code Issues 19 Pull Requests Packages Projects Releases Wiki Activity
cc-os/plugins/os-adr/eval-c/scenarios-reserve/N4-L1-notifications.md

15 lines
998 B
Markdown
Raw Normal View History

Add os-adr Eval C: ambiguity-ladder discrimination eval Eval B wording reached 8/8 (sonnet) / 7/8 (haiku) on its run-set after five iterations; subsequent W3 stability testing confirmed intermittent flicker but acceptable performance. As Eval B is now contaminated by iteration, we need a held-out measurement set to assess generalization to new projects and cues. Eval C measures whether learned behavior generalizes across decreasing cue explicitness (explicit → moderate → conceptual framing) and, critically, whether the model correctly avoids false positives (over-triggering) when no Accepted ADR is in play. Six paired scenarios (positive/negative at each level) run against a job-execution domain fixture; a frozen reserve-set (notifications domain) becomes the measurement set if anyone tunes wording against the run-set. New: `plugins/os-adr/eval-c/` — bin/ (run/check/self-test/sandbox scripts), fixture/ (taskq async job queue, 6 ADRs, trigger-phrased CLAUDE.md), scenarios/ (6 run-set), scenarios-reserve/ (6 reserve- set), README.md (measurement discipline + design), judge-rubric.md. Model-free self-test passing. First real grid run is a pending decision.
2026-07-06 16:02:58 +00:00
# N4-L1 — negative: debug existing behavior (webhook delivery)
> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
> First real execution is the harness self-test (see eval-c/README.md).
- **Fixture:** `project` (no `graphify-out/`)
- **Shape:** Explicit file/vocabulary reference (like L1-positive) but task is debug/investigate, not design
- **Correct non-trigger:** No new notification delivery strategy being designed; debugging existing flow (ADR 0003) within its scope
- **Checker, axis (a):** informational only (yes/no)
- **Checker, axis (b):** mechanical PASS if NO new ADR file created (truthful citation of governing ADRs is correct behavior — recorded as informational `cited-adr:yes/no`, never a FAIL)
## Task (give verbatim to the model under test)
Debug why a customer's webhook endpoint never received job completion notifications. Review the notification routing flow in taskq/notifications.py and trace where the delivery could fail.