86 lines
5.7 KiB
Markdown
86 lines
5.7 KiB
Markdown
## 1. Fixture project(s)
|
||
|
||
- [x] 1.1 Scaffold `plugins/os-adr/eval-b/fixture/` as a small, self-contained fixture codebase
|
||
(init as its own git repo, mirroring Eval A's `fixture/project/` pattern)
|
||
- [x] 1.2 Author `docs/adr/` content for W1–W3 (a plausible existing convention/decision history
|
||
the scenarios can plausibly reverse or extend) and initialize the ADR index
|
||
- [x] 1.3 Author R1's Accepted ADR with `affected-paths` covering specific fixture files whose
|
||
obvious next change would violate it
|
||
- [x] 1.4 Extend `docs/adr/` with R2's 2–3 distractor ADRs (near-miss same-component-family,
|
||
plus one Superseded) alongside the correct one
|
||
- [x] 1.5 Add an R3 Accepted ADR whose content answers a "how should we…" question phrased in
|
||
different vocabulary than the ADR's own text
|
||
- [x] 1.6 Design R4's one-hop layout: files that import/reference an R1-style ADR's
|
||
`affected-paths` without being listed themselves; run the real `graphify` binary to produce
|
||
`graphify-out/` and confirm the hop is actually one edge away in the resulting graph
|
||
- [x] 1.7 Document a fixture regeneration step (README) that rebuilds `graphify-out/` from
|
||
scratch rather than committing it as a frozen blob (`bin/build-fixture-graph` +
|
||
"Fixture regeneration" section in `eval-b/README.md`; script asserts the R4 one-hop
|
||
invariant on every rebuild)
|
||
|
||
## 2. Scenario prompts
|
||
|
||
- [x] 2.1 Write `eval-b/scenarios/W1.md`..`W3.md` and `R1.md`..`R4.md`: a task prompt per
|
||
scenario that never names the ADR system, plugin, or exact constraint text, plus the
|
||
checker-facing metadata (correct ADR ID for R1–R4; expected new-ADR shape for W1–W3)
|
||
following the pass/fail language already sketched in `06-eval-scenarios.md`
|
||
- [x] 2.2 Self-review each prompt against the "held-out" ground rule before running anything
|
||
against it (no informal trial runs — first real execution is the self-test in section 5)
|
||
(R1's "skip the shared client" wording was flagged and removed in this review)
|
||
|
||
## 3. Headless runner
|
||
|
||
- [x] 3.1 Write `eval-b/bin/sandbox <scenario> <dest>`: fresh git-initialized sandbox copy of the
|
||
right fixture variant (mirrors Eval A's `bin/sandbox`; `R4-nograph` variant included)
|
||
- [x] 3.2 Write `eval-b/bin/run <scenario> <model> <sandbox>`: invokes
|
||
`claude -p --output-format stream-json --verbose --model <tier>` with cwd set to the
|
||
sandbox, passing only the scenario's task prompt (no system-level hints), capturing the
|
||
full JSONL transcript to a file in the sandbox
|
||
- [x] 3.3 Confirm the os-adr plugin (and, for R4, graphify) are active in the environment the
|
||
runner executes in, and that a sandbox with an initialized `docs/adr/` triggers the
|
||
SessionStart hook's "present" branch (verified via a neutral non-scenario probe prompt:
|
||
haiku quoted the [os-adr] note verbatim; transcript carries a system/hook_response event)
|
||
- [x] 3.4 Add `--reps N` support to the runner for repeated executions of the same cell (default
|
||
left open per design.md Decision 5; document how to override)
|
||
|
||
## 4. Checker
|
||
|
||
- [x] 4.1 Write `eval-b/bin/check <scenario> <sandbox> [--tsv <model>]`: parses the JSONL
|
||
transcript for axis (a) (tool_use blocks touching `os-adr:*`, `bin/adr-*`, or
|
||
`docs/adr/` reads/globs)
|
||
- [x] 4.2 Implement axis (b) for R1–R4: compare the transcript's cited/acted-on ADR ID against
|
||
the scenario's pre-declared correct ID; fail on distractor match (R2) or on missing
|
||
graph-layer reach (R4)
|
||
- [x] 4.3 Implement axis (b) for W1–W3: mechanical PASS if a new `docs/adr/NNNN-*.md` file
|
||
exists in the sandbox matching the scenario's expected shape; otherwise invoke the narrow
|
||
LLM judge (final message only) with a fixed rubric to decide propose/ask-vs-silent
|
||
(rubric frozen in `eval-b/judge-rubric.md`; stubbable via ADR_EVAL_B_JUDGE_CMD)
|
||
- [x] 4.4 Emit the TSV row format (scenario, model tier, axis-a, axis-b, pass/fail) for
|
||
`autoresearch` compatibility
|
||
- [x] 4.5 Write the R4 degradation-check variant: same scenario run against a sandbox with
|
||
`graphify-out/` removed, expected to FAIL where the graph-layer path was required
|
||
(`bin/sandbox R4-nograph` + `bin/check R4-nograph` alias)
|
||
|
||
## 5. Self-test (mirrors Eval A's own-both-directions check)
|
||
|
||
- [x] 5.1 For each of the 7 scenarios, hand-simulate a "perfect" transcript/sandbox state (an
|
||
agent that does the right thing) and confirm `bin/check` scores PASS on both axes
|
||
(scripted as `eval-b/bin/self-test`; includes the W judge-fallback path with a stub judge,
|
||
plus a one-off smoke test of the real haiku judge)
|
||
- [x] 5.2 For each of the 7 scenarios, check an untouched sandbox (no ADR consultation at all)
|
||
and confirm `bin/check` scores axis (a) FAIL (also covers R4-nograph, the R2
|
||
superseded-distractor trap, and the missing-hook-context invalid-run case)
|
||
- [x] 5.3 Confirm sandbox isolation: after a run, diff the canonical fixture and the cc-os repo
|
||
and verify neither was modified (fixture digest check inside self-test; git status clean
|
||
of unexpected entries)
|
||
|
||
## 6. Documentation
|
||
|
||
- [x] 6.1 Write `eval-b/README.md` mirroring Eval A's README shape: layout table, how to run a
|
||
single cell, how the two-axis scoring works, explicit warning that scenario prompts are
|
||
held-out and must not be informally tried out outside this harness's own self-test
|
||
- [x] 6.2 Cross-link from `CLAUDE.md`'s os-adr section once this change is applied and archived
|
||
(status line update per this repo's "keep this file current" convention) — Eval B bullet
|
||
added to the os-adr component section, "Remaining" rollout line updated, and
|
||
`06-eval-scenarios.md` status line now points at the built harness
|