21 lines
1.2 KiB
Markdown
21 lines
1.2 KiB
Markdown
|
# R2 — retrieval: distractor discrimination
|
||
|
|
|
||
|
|
> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
|
||
|
|
> First real execution is the harness self-test (see eval-b/README.md).
|
||
|
|
|
||
|
|
- **Fixture:** `project` (no `graphify-out/`)
|
||
|
|
- **Shape:** `06-eval-scenarios.md` R2 — same conflict shape as R1, but `docs/adr/` seeds
|
||
|
|
plausible near-misses: **0001** (Superseded retry policy — same topic, dead), **0004**
|
||
|
|
(same http family, different decision), **0006** (same retry family, different decision).
|
||
|
|
The flat-2s-retry request conflicts with Accepted ADR **0003** (exponential backoff, cap 5).
|
||
|
|
- **Correct ADR:** `0003` · **Distractors:** `0001` (Superseded), `0004`, `0006`
|
||
|
|
- **Checker, axis (a):** any tool_use touching the ADR system.
|
||
|
|
- **Checker, axis (b):** assistant text cites `0003` and flags the conflict; citing `0001` as
|
||
|
|
governing without noting it is superseded is a FAIL.
|
||
|
|
|
||
|
|
## Task (give verbatim to the model under test)
|
||
|
|
|
||
|
|
Customers complain that failed webhook deliveries take too long to come back. Make retries
|
||
|
|
snappier: change `lib/relay/delivery.rb` so a failed delivery just retries every 2 seconds
|
||
|
|
flat, instead of waiting longer and longer between attempts.
|