cc-os

Commit Graph

Author	SHA1	Message	Date
jared	5b399d5614	Record Eval A and Eval B grid results in status docs Eval A grid ran clean across all scenarios/tiers. Eval B grid (1 rep/cell) shows a real gap: haiku never unprompted-consults the ADR system in any scenario, while sonnet passes 5/8 (misses W3 write-trigger and R1 direct- conflict retrieval). Updates CLAUDE.md component status and the eval-b README so the next session picks up from here instead of re-running blind. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-03 17:36:37 -04:00
jared	00e22f1676	Add os-adr evaluation harnesses (Eval A and Eval B) Build comprehensive evaluation framework for the os-adr plugin (ADR-021). Eval A: prompted skill-execution evaluation across haiku/sonnet models. Six ADR lifecycle scenarios, deterministic Ruby checker, sandbox runner, /autoresearch loop for wording optimization. Eval B: held-out unprompted-behavior evaluation. Seven scenarios (W1–W3 write-trigger, R1–R4 retrieval), webhook fixture with 6-ADR history (Superseded pair + distractors), headless runner (isolated SessionStart context), two-axis deterministic-first checker (consultation + citation accuracy, AI judge fallback for new-file writes). Both harnesses self-tested model-free. Grid runs deferred per locked rollout order. Updated docs/specs to freeze scenario shapes and document methodologies.	2026-07-03 16:44:42 -04:00

Author

SHA1

Message

Date

jared

5b399d5614

Record Eval A and Eval B grid results in status docs

Eval A grid ran clean across all scenarios/tiers. Eval B grid (1 rep/cell)
shows a real gap: haiku never unprompted-consults the ADR system in any
scenario, while sonnet passes 5/8 (misses W3 write-trigger and R1 direct-
conflict retrieval). Updates CLAUDE.md component status and the eval-b
README so the next session picks up from here instead of re-running blind.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

2026-07-03 17:36:37 -04:00

jared

00e22f1676

Add os-adr evaluation harnesses (Eval A and Eval B)

Build comprehensive evaluation framework for the os-adr plugin (ADR-021).

Eval A: prompted skill-execution evaluation across haiku/sonnet models.
Six ADR lifecycle scenarios, deterministic Ruby checker, sandbox runner,
/autoresearch loop for wording optimization.

Eval B: held-out unprompted-behavior evaluation. Seven scenarios
(W1–W3 write-trigger, R1–R4 retrieval), webhook fixture with 6-ADR history
(Superseded pair + distractors), headless runner (isolated SessionStart
context), two-axis deterministic-first checker (consultation + citation
accuracy, AI judge fallback for new-file writes).

Both harnesses self-tested model-free. Grid runs deferred per locked
rollout order. Updated docs/specs to freeze scenario shapes and document
methodologies.

2026-07-03 16:44:42 -04:00

2 Commits