jared
/
cc-os
1
0
Fork 0
Code Issues 18 Pull Requests Packages Projects Releases Wiki Activity
cc-os/plugins/os-adr/eval-b/fixture/project
History
jared 00e22f1676 Add os-adr evaluation harnesses (Eval A and Eval B) 
Build comprehensive evaluation framework for the os-adr plugin (ADR-021).

Eval A: prompted skill-execution evaluation across haiku/sonnet models.
Six ADR lifecycle scenarios, deterministic Ruby checker, sandbox runner,
/autoresearch loop for wording optimization.

Eval B: held-out unprompted-behavior evaluation. Seven scenarios
(W1–W3 write-trigger, R1–R4 retrieval), webhook fixture with 6-ADR history
(Superseded pair + distractors), headless runner (isolated SessionStart
context), two-axis deterministic-first checker (consultation + citation
accuracy, AI judge fallback for new-file writes).

Both harnesses self-tested model-free. Grid runs deferred per locked
rollout order. Updated docs/specs to freeze scenario shapes and document
methodologies.
 		2026-07-03 16:44:42 -04:00
..
bin Add os-adr evaluation harnesses (Eval A and Eval B) 2026-07-03 16:44:42 -04:00
config Add os-adr evaluation harnesses (Eval A and Eval B) 2026-07-03 16:44:42 -04:00
docs/adr Add os-adr evaluation harnesses (Eval A and Eval B) 2026-07-03 16:44:42 -04:00
lib Add os-adr evaluation harnesses (Eval A and Eval B) 2026-07-03 16:44:42 -04:00
.gitignore Add os-adr evaluation harnesses (Eval A and Eval B) 2026-07-03 16:44:42 -04:00
.graphifyignore Add os-adr evaluation harnesses (Eval A and Eval B) 2026-07-03 16:44:42 -04:00
README.md Add os-adr evaluation harnesses (Eval A and Eval B) 2026-07-03 16:44:42 -04:00

README.md

relay

A small CLI that delivers webhook payloads to configured endpoints and reports on delivery outcomes.

  • bin/relay send <url> <json-payload> — queue and deliver one payload
  • bin/relay report — summarize failed deliveries

Configuration lives in config/relay.yml. Failed deliveries that exhaust their retries are appended to the dead-letter file for manual replay.

Architecture decisions are recorded in docs/adr/.