jared
/
cc-os
1
0
Fork 0
Code Issues 18 Pull Requests Packages Projects Releases Wiki Activity
cc-os/plugins/os-adr/eval-b/fixture/project/config/relay.yml

6 lines
144 B
YAML
Raw Normal View History

Add os-adr evaluation harnesses (Eval A and Eval B) Build comprehensive evaluation framework for the os-adr plugin (ADR-021). Eval A: prompted skill-execution evaluation across haiku/sonnet models. Six ADR lifecycle scenarios, deterministic Ruby checker, sandbox runner, /autoresearch loop for wording optimization. Eval B: held-out unprompted-behavior evaluation. Seven scenarios (W1–W3 write-trigger, R1–R4 retrieval), webhook fixture with 6-ADR history (Superseded pair + distractors), headless runner (isolated SessionStart context), two-axis deterministic-first checker (consultation + citation accuracy, AI judge fallback for new-file writes). Both harnesses self-tested model-free. Grid runs deferred per locked rollout order. Updated docs/specs to freeze scenario shapes and document methodologies.
2026-07-03 20:44:42 +00:00
endpoints:
- https://hooks.example.test/orders
- https://hooks.example.test/invoices
dead_letter_path: deadletter.jsonl
log_path: relay.log