cc-os/plugins/os-adr/eval/scenarios/S1.md

# S1 — create: record a fresh decision

- **Fixture:** `project` · **Skill:** `create`
- **Checker asserts:** `docs/adr/0005-*.md` exists, id 0005, status Accepted, non-empty
  Context/Decision/Consequences, non-empty affected-paths, indexed in README.md, ADRs 0001–0004 untouched.

## Task (give verbatim to the model under test)

We've just decided: rendered reports will be cached on disk under `cache/` with mtime-based
invalidation, because re-rendering is too slow for repeat views. We considered memcached
(rejected — an extra service to operate) and doing nothing (rejected — repeat views stay slow).
This affects `lib/reports.rb` and the new `cache/` directory. Record this decision.
-												Add os-adr evaluation harnesses (Eval A and Eval B)

Build comprehensive evaluation framework for the os-adr plugin (ADR-021).

Eval A: prompted skill-execution evaluation across haiku/sonnet models.
Six ADR lifecycle scenarios, deterministic Ruby checker, sandbox runner,
/autoresearch loop for wording optimization.

Eval B: held-out unprompted-behavior evaluation. Seven scenarios
(W1–W3 write-trigger, R1–R4 retrieval), webhook fixture with 6-ADR history
(Superseded pair + distractors), headless runner (isolated SessionStart
context), two-axis deterministic-first checker (consultation + citation
accuracy, AI judge fallback for new-file writes).

Both harnesses self-tested model-free. Grid runs deferred per locked
rollout order. Updated docs/specs to freeze scenario shapes and document
methodologies.

											
										
										
											2026-07-03 20:44:42 +00:00
+								# S1 — create: record a fresh decision
 								- **Fixture:** `project` · **Skill:** `create`
 								- **Checker asserts:** `docs/adr/0005-*.md` exists, id 0005, status Accepted, non-empty
 								  Context/Decision/Consequences, non-empty affected-paths, indexed in README.md, ADRs 0001–0004 untouched.
 								## Task (give verbatim to the model under test)
 								We've just decided: rendered reports will be cached on disk under `cache/` with mtime-based
 								invalidation, because re-rendering is too slow for repeat views. We considered memcached
 								(rejected — an extra service to operate) and doing nothing (rejected — repeat views stay slow).
 								This affects `lib/reports.rb` and the new `cache/` directory. Record this decision.