jared
|
00e22f1676
|
Add os-adr evaluation harnesses (Eval A and Eval B)
Build comprehensive evaluation framework for the os-adr plugin (ADR-021).
Eval A: prompted skill-execution evaluation across haiku/sonnet models.
Six ADR lifecycle scenarios, deterministic Ruby checker, sandbox runner,
/autoresearch loop for wording optimization.
Eval B: held-out unprompted-behavior evaluation. Seven scenarios
(W1–W3 write-trigger, R1–R4 retrieval), webhook fixture with 6-ADR history
(Superseded pair + distractors), headless runner (isolated SessionStart
context), two-axis deterministic-first checker (consultation + citation
accuracy, AI judge fallback for new-file writes).
Both harnesses self-tested model-free. Grid runs deferred per locked
rollout order. Updated docs/specs to freeze scenario shapes and document
methodologies.
|
2026-07-03 16:44:42 -04:00 |
|
jared
|
36abd43107
|
Build os-adr plugin: ADR authoring, session hook, migration, retrieval
Implements OpenSpec change add-os-adr-plugin (archived as
2026-07-03-add-os-adr-plugin; delta specs synced to openspec/specs/).
- plugins/os-adr/: Ruby OO core (Record, Repository, Index, Template,
Detector, Migrator, MigrationReport, Finder) behind thin bin/ CLIs;
Python SessionStart hook via plugin-relative hooks.json; four skills
(create, init, migrate, find); invariants.md; 47 Ruby + 15 Python
model-free tests, all passing.
- Migration pilot passed (threshold 25%): viking 0%, delta-refinery 0%,
llf-schema 8.3%. Retrieval verified against the real llf-schema graph;
resolved that ADRs need no Graphify tag/edge convention (layer 3
expands query paths via node source_files).
- docs/adr-system/: requirements (04), PRD (05), eval sketches (06).
- ADR-020 recorded; CLAUDE.md Implemented Components updated with the
locked rollout order (pilot projects -> cc-os retrofit -> wider).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
2026-07-03 14:24:19 -04:00 |