jared
/
cc-os
1
0
Fork 0
Code Issues 19 Pull Requests Packages Projects Releases Wiki Activity
cc-os/plugins/os-adr/eval-c/scenarios-reserve/N6-L3-notifications.md

15 lines
1023 B
Markdown
Raw Normal View History

Add os-adr Eval C: ambiguity-ladder discrimination eval Eval B wording reached 8/8 (sonnet) / 7/8 (haiku) on its run-set after five iterations; subsequent W3 stability testing confirmed intermittent flicker but acceptable performance. As Eval B is now contaminated by iteration, we need a held-out measurement set to assess generalization to new projects and cues. Eval C measures whether learned behavior generalizes across decreasing cue explicitness (explicit → moderate → conceptual framing) and, critically, whether the model correctly avoids false positives (over-triggering) when no Accepted ADR is in play. Six paired scenarios (positive/negative at each level) run against a job-execution domain fixture; a frozen reserve-set (notifications domain) becomes the measurement set if anyone tunes wording against the run-set. New: `plugins/os-adr/eval-c/` — bin/ (run/check/self-test/sandbox scripts), fixture/ (taskq async job queue, 6 ADRs, trigger-phrased CLAUDE.md), scenarios/ (6 run-set), scenarios-reserve/ (6 reserve- set), README.md (measurement discipline + design), judge-rubric.md. Model-free self-test passing. First real grid run is a pending decision.
2026-07-06 16:02:58 +00:00
# N6-L3 — negative: audit existing audit trail (notification traceability)
> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
> First real execution is the harness self-test (see eval-c/README.md).
- **Fixture:** `project` (no `graphify-out/`)
- **Shape:** Pure intent (audit/traceability), no new decision needed
- **Correct non-trigger:** Verifying existing notification routing and audit guarantees (ADRs 0003, 0006) — no new architecture choice
- **Checker, axis (a):** informational only (yes/no)
- **Checker, axis (b):** mechanical PASS if NO new ADR file created (truthful citation of governing ADRs is correct behavior — recorded as informational `cited-adr:yes/no`, never a FAIL)
## Task (give verbatim to the model under test)
Trace the notification pathway from job completion through delivery to end users. Verify that the system maintains full audit and traceability: confirm all notifications are routed through the central service and logged with timestamps.