cc-os/plugins/os-adr/eval/scenarios/S2.md

# S2 — create: supersession bookkeeping

- **Fixture:** `project` · **Skill:** `create`
- **Checker asserts:** new 0005 with `supersedes: 0001`; 0001 flipped to Superseded with
  `superseded-by: 0005`; index updated; 0002–0004 untouched.
- The task never says "supersede 0001" — the model must notice the existing persistence
  decision via the index, as the skill instructs.

## Task (give verbatim to the model under test)

We're replacing SQLite with DuckDB for local persistence — analytical aggregate queries have
outgrown SQLite. Staying on SQLite was rejected because aggregate scans are too slow; Postgres
was rejected again for operational overhead. This affects `lib/store.rb` and
`config/database.yml`. Record this decision — note we already have a persistence decision on file.
-												Add os-adr evaluation harnesses (Eval A and Eval B)

Build comprehensive evaluation framework for the os-adr plugin (ADR-021).

Eval A: prompted skill-execution evaluation across haiku/sonnet models.
Six ADR lifecycle scenarios, deterministic Ruby checker, sandbox runner,
/autoresearch loop for wording optimization.

Eval B: held-out unprompted-behavior evaluation. Seven scenarios
(W1–W3 write-trigger, R1–R4 retrieval), webhook fixture with 6-ADR history
(Superseded pair + distractors), headless runner (isolated SessionStart
context), two-axis deterministic-first checker (consultation + citation
accuracy, AI judge fallback for new-file writes).

Both harnesses self-tested model-free. Grid runs deferred per locked
rollout order. Updated docs/specs to freeze scenario shapes and document
methodologies.

											
										
										
											2026-07-03 20:44:42 +00:00
+								# S2 — create: supersession bookkeeping
 								- **Fixture:** `project` · **Skill:** `create`
 								- **Checker asserts:** new 0005 with `supersedes: 0001`; 0001 flipped to Superseded with
 								  `superseded-by: 0005`; index updated; 0002–0004 untouched.
 								- The task never says "supersede 0001" — the model must notice the existing persistence
 								  decision via the index, as the skill instructs.
 								## Task (give verbatim to the model under test)
 								We're replacing SQLite with DuckDB for local persistence — analytical aggregate queries have
 								outgrown SQLite. Staying on SQLite was rejected because aggregate scans are too slow; Postgres
 								was rejected again for operational overhead. This affects `lib/store.rb` and
 								`config/database.yml`. Record this decision — note we already have a persistence decision on file.