jared
/
cc-os
1
0
Fork 0
Code Issues 18 Pull Requests Packages Projects Releases Wiki Activity
cc-os/openspec/changes/archive/2026-07-03-add-os-adr-eval-.../proposal.md

3.3 KiB
Raw Blame History

Why

The os-adr plugin's two hardest requirements — unprompted write-trigger recognition and unprompted correct retrieval (04-plugin-requirements.md reqs 4–5) — have never been measured. Eval A (plugins/os-adr/eval/) only proves the plugin executes correctly once explicitly invoked; it says nothing about whether an agent notices, on its own, that the ADR system is relevant. The scenario shapes for this held-out eval already exist (docs/adr-system/06-eval-scenarios.md) but no fixtures, prompts, checker, or runner exist yet. Building it now closes the last unmeasured requirement in the locked rollout order.

What Changes

  • Build a new eval harness, plugins/os-adr/eval-b/, sibling to but structurally distinct from Eval A, for the 7 held-out scenarios (W1–W3 write-trigger, R1–R4 retrieval).
  • Author full scenario prompts/fixtures from the frozen shapes in 06-eval-scenarios.md — scenario shapes are not renegotiated by this change; only their execution artifacts are new.
  • Fixture project(s): a realistic, onboarded project state (populated docs/adr/ incl. a Superseded pair for R2 distractors, a built graphify-out/ for R4) that the SessionStart hook recognizes as ADR-initialized.
  • A headless runner (real claude -p in a real sandbox cwd) as the primary execution mode, not an in-session Agent-tool subagent — because the SessionStart hook firing for real is part of what's under test (Eval A's subagent shortcut is not valid here; see design.md).
  • A deterministic-first, two-axis checker: (a) did the agent consult/propose the ADR system unprompted at all, (b) did it act on/write the specific correct thing. Axis (a) is mechanically checkable (tool-call/transcript grep for /os-adr:* or docs/adr/ reads/writes); axis (b) requires a small LLM-judged correctness check against the fixture's known-correct answer, kept as thin and rubric-bound as possible.
  • Sandbox/grid tooling mirroring Eval A's shape (bin/sandbox, bin/check, results.tsv) adapted for headless execution and the two-axis score.
  • Wiring for the later autoresearch iteration stage (not run by this change — building the harness only).

Out of scope for this change: actually running the eval grid, iterating retrieval/trigger heuristics, or adding/altering scenario shapes in 06-eval-scenarios.md.

Capabilities

New Capabilities

  • adr-eval-b-harness: fixtures, held-out scenario prompts, headless sandbox/runner tooling, and a two-axis deterministic-first checker for the unprompted write-trigger and retrieval eval.

Modified Capabilities

(none — this adds a new, separate eval surface; it does not change the behavior of adr-authoring, adr-retrieval, adr-session-awareness, or any shipped plugin capability)

Impact

  • New directory: plugins/os-adr/eval-b/ (fixtures, scenarios, bin/, runner assets, README).
  • No changes to plugin source (lib/adr/, bin/adr-*, hooks/, skills/) — this is a pure test-harness addition.
  • Depends on claude CLI headless mode (claude -p) being available in the build/dev environment for the runner to execute against a real sandbox with hooks firing.
  • Once built, unlocks the deferred final stage of the locked os-adr rollout order (running the grid + autoresearch iteration), tracked separately in CLAUDE.md.