86 lines
5.4 KiB
Markdown
86 lines
5.4 KiB
Markdown
# Eval Scenario Sketches (held-out, deferred stage)
|
||
|
||
_Last updated: 2026-07-03 — these shapes are now built out as the Eval B harness at
|
||
`plugins/os-adr/eval-b/` (fixtures, prompts, headless runner, two-axis checker; OpenSpec change
|
||
`add-os-adr-eval-b-harness`). The shapes below stay frozen — the harness executes them, it does
|
||
not renegotiate them. Running the grid remains a separate later stage (locked sequencing in
|
||
`04-plugin-requirements.md`)._
|
||
|
||
> **Two evals, don't conflate them (ADR-021).** This file sketches **Eval B** — held-out,
|
||
> unprompted-behavior (requirements 4–5). A separate **Eval A** — prompted skill-execution
|
||
> across model tiers (can haiku/sonnet correctly *execute* an explicitly invoked skill?) —
|
||
> is built and lives at `plugins/os-adr/eval/` with its own README; its prompts are *not*
|
||
> held-out and may be iterated freely. When Eval B eventually runs, it runs as
|
||
> scenario × model tier (haiku/sonnet/session-model), pass rate per tier as the
|
||
> autoresearch metric.
|
||
|
||
These sketch the *shape* of the held-out scenarios that will eventually measure requirements 4
|
||
and 5 (unprompted write-trigger recognition, unprompted correct retrieval). They exist now, before
|
||
the plugin skeleton freezes, for one purpose: to catch any plugin surface the eval will need that
|
||
Phases 1–4 don't provide. They are **not** to be run, automated, or informally "tried out" during
|
||
the build — that would contaminate the held-out methodology.
|
||
|
||
## Ground rules for all scenarios
|
||
|
||
- **Held-out**: scenario prompts never name the ADR system, the plugin, an ADR ID/title, or the
|
||
exact constraint text. Relevance must be discovered, not pointed at.
|
||
- **Scored on two axes**: (a) did the agent consult/propose the ADR system *at all* unprompted;
|
||
(b) did it act on the *specific correct* ADR (retrieval) or propose a *well-scoped* new one
|
||
(write-trigger) — not merely a plausible-looking neighbor.
|
||
- **Environment**: a real onboarded project (post-pilot) with a populated `docs/adr/`, the
|
||
SessionStart hook active, and — for graph-layer variants — a built `graphify-out/`.
|
||
- The eval stage will iterate with the `autoresearch` skill (modify → verify → keep/discard)
|
||
against these scenarios once written out in full.
|
||
|
||
## Write-trigger scenario shapes
|
||
|
||
**W1 — Irreversible choice mid-task.** Task: "Add persistence to X" in a project where the
|
||
storage engine choice is genuinely open and consequential (hard to reverse once data exists).
|
||
Never mention ADRs. Pass: the agent proposes recording the engine decision via `/os-adr:create`
|
||
(or asks whether to) before or immediately after implementing. Fail: silently picks one and moves on.
|
||
|
||
**W2 — Cross-cutting convention change.** Task: a refactor request whose natural solution
|
||
changes a convention used across many files (e.g. error-handling shape, module layout). Pass:
|
||
the agent recognizes the decision outlives the task and proposes an ADR. Fail: does the refactor
|
||
with no record.
|
||
|
||
**W3 — Contested reversal.** Task: user asks for something that quietly reverses an earlier
|
||
(non-ADR'd, but discoverable-in-code) approach. Pass: agent flags that this is a
|
||
decision-of-record moment and proposes capturing it. This shape overlaps retrieval when the
|
||
earlier approach *is* ADR'd — kept separate deliberately.
|
||
|
||
## Retrieval scenario shapes
|
||
|
||
**R1 — Direct conflict with an Accepted ADR.** Task: a change request whose obvious
|
||
implementation violates an existing Accepted ADR whose `affected-paths` cover the files being
|
||
edited. The prompt names neither the ADR nor the constraint. Pass: agent surfaces the specific
|
||
ADR (via `/os-adr:find` or the SessionStart note) and adjusts or pushes back. Fail: implements
|
||
the violation, or cites a different/wrong ADR.
|
||
|
||
**R2 — Distractor discrimination.** Same as R1, but `docs/adr/` also contains 2–3 plausible
|
||
near-miss ADRs (same component family, different decision; or Superseded status). Pass requires
|
||
retrieving the *correct, Accepted* one — this is the "correct, not merely plausible" axis and
|
||
the reason the status filter and path-match layers exist.
|
||
|
||
**R3 — Mid-task question.** Partway through an ongoing task, the user asks a "how should we…"
|
||
question already answered by an Accepted ADR, phrased in different vocabulary. Pass: agent
|
||
queries the ADR system mid-task (not only at SessionStart) and answers from the ADR. This is the
|
||
scenario that requires a **mid-task-callable query surface**.
|
||
|
||
**R4 — Graph-layer reach.** Conflict exists but the edited files are *not* listed in any ADR's
|
||
`affected-paths`; they are one graph hop away (import/reference) from files that are. Pass only
|
||
reachable via the Graphify traversal layer. Also serves as the degradation check: without
|
||
`graphify-out/`, R4 is expected to fail while R1–R3 still pass.
|
||
|
||
## Plugin-surface requirements derived from these sketches
|
||
|
||
All are already covered by the planned Phases 1–4 — **no plan change needed**:
|
||
|
||
| Scenario need | Covering surface |
|
||
| --- | --- |
|
||
| Mid-task query, not SessionStart-only (R3) | `/os-adr:find` skill + `bin/adr-find` CLI (Phase 4) |
|
||
| Discoverability without being told (all) | SessionStart note names both `create` and `find` (Phase 2) |
|
||
| Correct-vs-plausible discrimination (R2) | path/component match + Accepted-by-default status filter (Phase 4, layers 1–2) |
|
||
| One-hop relevance (R4) | Graphify traversal layer with graceful degradation (Phase 4, layer 3) |
|
||
| Low-friction unprompted writing (W1–W3) | one-invocation `/os-adr:create` (Phase 1) |
|