72 lines
4.0 KiB
Markdown
72 lines
4.0 KiB
Markdown
## Purpose
|
||
|
||
Evaluation harness B: held-out, unprompted-behavior testing for the os-adr plugin. Tests whether the plugin's core write/retrieval operations surface naturally without coaching, across multiple model tiers, with deterministic-first scoring (mechanical checks before fallback to LLM judge).
|
||
|
||
## Requirements
|
||
|
||
### Requirement: Held-out scenario fixtures
|
||
The harness SHALL provide a dedicated fixture project (or projects) under
|
||
`plugins/os-adr/eval-b/fixture/` that supports all 7 held-out scenarios (W1–W3, R1–R4) without
|
||
any scenario prompt naming the ADR system, an ADR ID, or the exact constraint text being tested.
|
||
|
||
#### Scenario: R2 distractor set present
|
||
- **WHEN** the R2 fixture's `docs/adr/` is inspected
|
||
- **THEN** it contains the one correct Accepted ADR plus 2–3 near-miss ADRs (same component
|
||
family, different decision, or Superseded status) that a naive retrieval could mistakenly cite
|
||
|
||
#### Scenario: R4 graph reachability is real, not stubbed
|
||
- **WHEN** the R4 fixture's `graphify-out/` is inspected
|
||
- **THEN** it was produced by running the real `graphify` binary against the fixture's own
|
||
codebase (not hand-authored), and the conflicting files are one import/reference hop away from
|
||
files listed in the relevant ADR's `affected-paths`, not directly listed themselves
|
||
|
||
### Requirement: Headless runner with real hook firing
|
||
The harness SHALL execute each scenario via a headless `claude -p` process with its working
|
||
directory set to a fresh sandbox copy of the fixture, so that the real SessionStart hook fires
|
||
for the model under test. In-session Agent-tool subagents SHALL NOT be used as the execution
|
||
mode for this eval.
|
||
|
||
#### Scenario: SessionStart hook context reaches the model under test
|
||
- **WHEN** a scenario is run via the headless runner against an ADR-initialized sandbox
|
||
- **THEN** the transcript shows the SessionStart hook's additionalContext was present in the
|
||
model's context before its first action
|
||
|
||
#### Scenario: Sandbox isolation
|
||
- **WHEN** any scenario run completes (pass or fail)
|
||
- **THEN** the canonical fixture directory and the cc-os repo itself show no modifications —
|
||
only the disposable sandbox copy was touched
|
||
|
||
### Requirement: Two-axis deterministic-first checker
|
||
The harness SHALL score each run on two independent axes: (a) whether the model consulted or
|
||
proposed the ADR system at all, unprompted, and (b) whether it acted on or wrote the specific
|
||
correct thing rather than a merely plausible one. Axis (a), and axis (b) for retrieval scenarios,
|
||
SHALL be scored mechanically from the transcript and sandbox file state; axis (b) for
|
||
write-trigger scenarios MAY fall back to a narrow, rubric-bound LLM judge only when no new ADR
|
||
file was mechanically created.
|
||
|
||
#### Scenario: Axis (a) mechanical detection
|
||
- **WHEN** the checker scans a scenario's `stream-json` transcript
|
||
- **THEN** it detects axis (a) as true if any tool_use block invokes an `os-adr:*` skill, a
|
||
`bin/adr-*` CLI, or a Read/Glob on `docs/adr/`, with no model call required
|
||
|
||
#### Scenario: Axis (b) retrieval correctness
|
||
- **WHEN** scoring an R1–R4 scenario's axis (b)
|
||
- **THEN** the checker mechanically compares the ADR ID the transcript cites or acts on against
|
||
the fixture's pre-declared correct ID, and fails if it matches a seeded distractor instead
|
||
|
||
#### Scenario: Axis (b) write-trigger fallback judge
|
||
- **WHEN** scoring a W1–W3 scenario where no new `docs/adr/NNNN-*.md` file was created in the
|
||
sandbox
|
||
- **THEN** the checker invokes a rubric-bound LLM judge on only the model's final message (not
|
||
the full transcript) to decide whether it explicitly proposed or asked about recording an ADR
|
||
|
||
### Requirement: Grid-compatible output
|
||
The harness SHALL emit per-run results in a format usable by a later `autoresearch` Classic-mode
|
||
loop (scenario, model tier, axis-a result, axis-b result, pass/fail), analogous to Eval A's
|
||
`results.tsv`.
|
||
|
||
#### Scenario: TSV row per run
|
||
- **WHEN** a scenario run is checked
|
||
- **THEN** the checker can emit one TSV row identifying the scenario, model tier, both axis
|
||
results, and overall pass/fail, appendable to a shared results file
|