68 lines
3.8 KiB
Markdown
68 lines
3.8 KiB
Markdown
|
## ADDED Requirements
|
|||
|
|
|
|||
|
|
### Requirement: Held-out scenario fixtures
|
|||
|
|
The harness SHALL provide a dedicated fixture project (or projects) under
|
|||
|
|
`plugins/os-adr/eval-b/fixture/` that supports all 7 held-out scenarios (W1–W3, R1–R4) without
|
|||
|
|
any scenario prompt naming the ADR system, an ADR ID, or the exact constraint text being tested.
|
|||
|
|
|
|||
|
|
#### Scenario: R2 distractor set present
|
|||
|
|
- **WHEN** the R2 fixture's `docs/adr/` is inspected
|
|||
|
|
- **THEN** it contains the one correct Accepted ADR plus 2–3 near-miss ADRs (same component
|
|||
|
|
family, different decision, or Superseded status) that a naive retrieval could mistakenly cite
|
|||
|
|
|
|||
|
|
#### Scenario: R4 graph reachability is real, not stubbed
|
|||
|
|
- **WHEN** the R4 fixture's `graphify-out/` is inspected
|
|||
|
|
- **THEN** it was produced by running the real `graphify` binary against the fixture's own
|
|||
|
|
codebase (not hand-authored), and the conflicting files are one import/reference hop away from
|
|||
|
|
files listed in the relevant ADR's `affected-paths`, not directly listed themselves
|
|||
|
|
|
|||
|
|
### Requirement: Headless runner with real hook firing
|
|||
|
|
The harness SHALL execute each scenario via a headless `claude -p` process with its working
|
|||
|
|
directory set to a fresh sandbox copy of the fixture, so that the real SessionStart hook fires
|
|||
|
|
for the model under test. In-session Agent-tool subagents SHALL NOT be used as the execution
|
|||
|
|
mode for this eval.
|
|||
|
|
|
|||
|
|
#### Scenario: SessionStart hook context reaches the model under test
|
|||
|
|
- **WHEN** a scenario is run via the headless runner against an ADR-initialized sandbox
|
|||
|
|
- **THEN** the transcript shows the SessionStart hook's additionalContext was present in the
|
|||
|
|
model's context before its first action
|
|||
|
|
|
|||
|
|
#### Scenario: Sandbox isolation
|
|||
|
|
- **WHEN** any scenario run completes (pass or fail)
|
|||
|
|
- **THEN** the canonical fixture directory and the cc-os repo itself show no modifications —
|
|||
|
|
only the disposable sandbox copy was touched
|
|||
|
|
|
|||
|
|
### Requirement: Two-axis deterministic-first checker
|
|||
|
|
The harness SHALL score each run on two independent axes: (a) whether the model consulted or
|
|||
|
|
proposed the ADR system at all, unprompted, and (b) whether it acted on or wrote the specific
|
|||
|
|
correct thing rather than a merely plausible one. Axis (a), and axis (b) for retrieval scenarios,
|
|||
|
|
SHALL be scored mechanically from the transcript and sandbox file state; axis (b) for
|
|||
|
|
write-trigger scenarios MAY fall back to a narrow, rubric-bound LLM judge only when no new ADR
|
|||
|
|
file was mechanically created.
|
|||
|
|
|
|||
|
|
#### Scenario: Axis (a) mechanical detection
|
|||
|
|
- **WHEN** the checker scans a scenario's `stream-json` transcript
|
|||
|
|
- **THEN** it detects axis (a) as true if any tool_use block invokes an `os-adr:*` skill, a
|
|||
|
|
`bin/adr-*` CLI, or a Read/Glob on `docs/adr/`, with no model call required
|
|||
|
|
|
|||
|
|
#### Scenario: Axis (b) retrieval correctness
|
|||
|
|
- **WHEN** scoring an R1–R4 scenario's axis (b)
|
|||
|
|
- **THEN** the checker mechanically compares the ADR ID the transcript cites or acts on against
|
|||
|
|
the fixture's pre-declared correct ID, and fails if it matches a seeded distractor instead
|
|||
|
|
|
|||
|
|
#### Scenario: Axis (b) write-trigger fallback judge
|
|||
|
|
- **WHEN** scoring a W1–W3 scenario where no new `docs/adr/NNNN-*.md` file was created in the
|
|||
|
|
sandbox
|
|||
|
|
- **THEN** the checker invokes a rubric-bound LLM judge on only the model's final message (not
|
|||
|
|
the full transcript) to decide whether it explicitly proposed or asked about recording an ADR
|
|||
|
|
|
|||
|
|
### Requirement: Grid-compatible output
|
|||
|
|
The harness SHALL emit per-run results in a format usable by a later `autoresearch` Classic-mode
|
|||
|
|
loop (scenario, model tier, axis-a result, axis-b result, pass/fail), analogous to Eval A's
|
|||
|
|
`results.tsv`.
|
|||
|
|
|
|||
|
|
#### Scenario: TSV row per run
|
|||
|
|
- **WHEN** a scenario run is checked
|
|||
|
|
- **THEN** the checker can emit one TSV row identifying the scenario, model tier, both axis
|
|||
|
|
results, and overall pass/fail, appendable to a shared results file
|