jared
/
cc-os
1
0
Fork 0
Code Issues 19 Pull Requests Packages Projects Releases Wiki Activity
cc-os/plugins/os-adr/eval-c/scenarios-reserve/P5-L2-notifications.md

15 lines
1021 B
Markdown
Raw Normal View History

Add os-adr Eval C: ambiguity-ladder discrimination eval Eval B wording reached 8/8 (sonnet) / 7/8 (haiku) on its run-set after five iterations; subsequent W3 stability testing confirmed intermittent flicker but acceptable performance. As Eval B is now contaminated by iteration, we need a held-out measurement set to assess generalization to new projects and cues. Eval C measures whether learned behavior generalizes across decreasing cue explicitness (explicit → moderate → conceptual framing) and, critically, whether the model correctly avoids false positives (over-triggering) when no Accepted ADR is in play. Six paired scenarios (positive/negative at each level) run against a job-execution domain fixture; a frozen reserve-set (notifications domain) becomes the measurement set if anyone tunes wording against the run-set. New: `plugins/os-adr/eval-c/` — bin/ (run/check/self-test/sandbox scripts), fixture/ (taskq async job queue, 6 ADRs, trigger-phrased CLAUDE.md), scenarios/ (6 run-set), scenarios-reserve/ (6 reserve- set), README.md (measurement discipline + design), judge-rubric.md. Model-free self-test passing. First real grid run is a pending decision.
2026-07-06 16:02:58 +00:00
# P5-L2 — write-trigger: moderate cues (webhook delivery)
> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
> First real execution is the harness self-test (see eval-c/README.md).
- **Fixture:** `project` (no `graphify-out/`)
- **Shape:** Moderate vocabulary (delivery, failures) but no explicit file paths or ADR naming
- **Correct decision:** Webhook delivery reliability is a sub-concern of the notification routing strategy (ADR 0003)
- **Checker, axis (a):** any tool_use touching the ADR system
- **Checker, axis (b):** mechanical PASS if new `docs/adr/NNNN-*.md` exists matching /webhook|deliv|retry|notification/i; otherwise judge-rubric fallback
## Task (give verbatim to the model under test)
Some webhook deliveries are failing silently, and clients never know if their notification actually arrived. We need better failure handling for webhook notifications — should we retry failed webhook deliveries, buffer them, or use a different delivery guarantee model?