Add os-adr evaluation harnesses (Eval A and Eval B)

Build comprehensive evaluation framework for the os-adr plugin (ADR-021).

Eval A: prompted skill-execution evaluation across haiku/sonnet models.
Six ADR lifecycle scenarios, deterministic Ruby checker, sandbox runner,
/autoresearch loop for wording optimization.

Eval B: held-out unprompted-behavior evaluation. Seven scenarios
(W1–W3 write-trigger, R1–R4 retrieval), webhook fixture with 6-ADR history
(Superseded pair + distractors), headless runner (isolated SessionStart
context), two-axis deterministic-first checker (consultation + citation
accuracy, AI judge fallback for new-file writes).

Both harnesses self-tested model-free. Grid runs deferred per locked
rollout order. Updated docs/specs to freeze scenario shapes and document
methodologies.

.gitignore

@@ -16,3 +16,6 @@ graphify-out/
 
 # Python bytecode
 __pycache__/
+
+# autoresearch skill run logs (eval iterations)
+/autoresearch/

CLAUDE.md

@@ -98,7 +98,9 @@ to those two and fix the stale doc.
 - Hook: `hooks/session_start.py` wired via plugin-relative `hooks/hooks.json` (`${CLAUDE_PLUGIN_ROOT}`, no `settings.json` entries) — deterministic existence check: present → near-zero-token usage note naming `/os-adr:create` + `/os-adr:find`; absent → once-per-day init/migrate suggestion, silenced permanently by `.os-adr/suppress` (gitignored per-project state dir); silent outside git projects.
 - Skills (verb-first, no `commands/`): `create` (one-invocation templated+numbered+indexed ADR with mechanical supersession), `init`, `migrate` (non-destructive: detect surveyed shapes → mechanical heuristic fill → LLM fills only manifest-listed interpretive fields via `--apply-fills` → `migration_confidence` frontmatter + `docs/adr/migration-report.md`; old-system deletion only as an explicit separate user-approved step), `find` (deterministic-first retrieval: affected-paths/components match → Accepted-only filter → one-hop Graphify query-path expansion with graceful degradation → AI judgment over the candidate set only).
 - Migration pilot passed 2026-07-03 (sandboxed project copies; gate threshold 25% low-confidence flags): viking-warrior-training-log 0%, delta-refinery 0%, llf-schema 8.3%. Retrieval verified end-to-end against the real llf-schema project graph. Resolved: ADRs need no Graphify tag/edge convention — retrieval layer 3 expands the *query* paths via graph-node `source_file`s.
-- **Remaining (locked rollout order):** real-project migration/adoption one at a time via `/os-adr:migrate` — pilot projects first, then the cc-os retrofit (its 19-ADR monolithic file was deliberately excluded from the pilot), then wider; plus the deferred unprompted-behavior eval (held-out scenario shapes sketched in `docs/adr-system/06-eval-scenarios.md`; methodology locked in `04-plugin-requirements.md`).
+- Eval A harness (2026-07-03, ADR-021): `plugins/os-adr/eval/` — prompted skill-execution eval across model tiers (haiku/sonnet). Two fixtures (generated with the plugin's own CLIs), six scenarios (S1 create, S2 create+supersede, S3 find/conflict, S4 find/distractor, S5 init, S6 migrate+fills), deterministic Ruby checker (`eval/bin/check`, structural invariants, TSV mode for autoresearch), sandbox + headless-runner scripts, runner-prompt template. Primary run mode: in-session Agent-tool subagents with pinned `model:` (cheaper than `claude -p`); optimized via the `/autoresearch` Classic loop over SKILL.md *wording only* (checker/fixtures/scenarios frozen during a loop). Procedure: `plugins/os-adr/eval/README.md`. Self-tested both directions; grid not yet run. Distinct from the held-out Eval B (unprompted behavior) — do not conflate.
+- Eval B harness (2026-07-03, OpenSpec change `add-os-adr-eval-b-harness`): `plugins/os-adr/eval-b/` — held-out unprompted-behavior eval (requirements 4–5). 7 scenarios (W1–W3 write-trigger, R1–R4 retrieval) authored from the frozen shapes in `docs/adr-system/06-eval-scenarios.md`; dedicated Ruby webhook-relay fixture with a 6-ADR history (Superseded pair + near-miss distractors, generated via the plugin's own CLIs); R4's one-hop graph reach uses a real `graphify update` AST build (model-free, rebuilt via `eval-b/bin/build-fixture-graph`, never committed). **Headless-only runner** (`eval-b/bin/run` — fresh `claude -p` per rep, cwd = sandbox, so the real SessionStart hook fires; in-session subagents are invalid here, unlike Eval A) and a two-axis deterministic-first checker (`eval-b/bin/check`): axis (a) unprompted consultation, mechanical from transcript tool_use blocks; axis (b) correct-ADR citation (R1–R4) or new-ADR-file with a narrow frozen-rubric haiku judge fallback (W1–W3, `judge-rubric.md`, stubbable via `ADR_EVAL_B_JUDGE_CMD`). `R4-nograph` is the graph-degradation variant (expected FAIL). Self-tested both directions model-free via `eval-b/bin/self-test`. **Scenario Task blocks are held-out — never run them informally.** Procedure: `plugins/os-adr/eval-b/README.md`. Grid not yet run.
+- **Remaining (locked rollout order):** run the Eval A grid ({S1..S6} × {haiku, sonnet}) and iterate SKILL.md wording via `/autoresearch` (procedure: `plugins/os-adr/eval/README.md`); real-project migration/adoption one at a time via `/os-adr:migrate` — pilot projects first, then the cc-os retrofit (its 19-ADR monolithic file was deliberately excluded from the pilot), then wider; then run the Eval B grid ({W1..W3, R1..R4} × model tiers per ADR-021, harness built at `plugins/os-adr/eval-b/` — see above) and iterate trigger/retrieval wording via `/autoresearch` with checker/fixtures/scenarios/rubric frozen.
 
 **Graphify** — v0.8.31 at `/home/jared/.local/bin/graphify`
 - Vault graph: `~/Documents/SecondBrain/graphify-out/` — disposable, structural index rebuilt by SessionStart hook; handles relational/graph queries. Distinct from vault-index.json (planned, Plan B Phase 1): a flat `{tag: [{path, title, summary}]}` lookup rebuilt at SessionEnd, queried by the `/memory-find` skill for fast tag-based SB note discovery.

docs/adr-system/06-eval-scenarios.md

@@ -1,7 +1,18 @@
 # Eval Scenario Sketches (held-out, deferred stage)
 
-_Last updated: 2026-07-03 — sketches only; the evaluation itself is a separate later stage
+_Last updated: 2026-07-03 — these shapes are now built out as the Eval B harness at
-(locked sequencing in `04-plugin-requirements.md`)._
+`plugins/os-adr/eval-b/` (fixtures, prompts, headless runner, two-axis checker; OpenSpec change
+`add-os-adr-eval-b-harness`). The shapes below stay frozen — the harness executes them, it does
+not renegotiate them. Running the grid remains a separate later stage (locked sequencing in
+`04-plugin-requirements.md`)._
+
+> **Two evals, don't conflate them (ADR-021).** This file sketches **Eval B** — held-out,
+> unprompted-behavior (requirements 4–5). A separate **Eval A** — prompted skill-execution
+> across model tiers (can haiku/sonnet correctly *execute* an explicitly invoked skill?) —
+> is built and lives at `plugins/os-adr/eval/` with its own README; its prompts are *not*
+> held-out and may be iterated freely. When Eval B eventually runs, it runs as
+> scenario × model tier (haiku/sonnet/session-model), pass rate per tier as the
+> autoresearch metric.
 
 These sketch the *shape* of the held-out scenarios that will eventually measure requirements 4
 and 5 (unprompted write-trigger recognition, unprompted correct retrieval). They exist now, before

docs/memory-system/03-architecture-decisions.md

@@ -603,6 +603,20 @@ _Date: 2026-07-03_
 - **Cross-references**: `docs/adr-system/04-plugin-requirements.md` (locked requirements), `05-plugin-prd.md` (phases), `06-eval-scenarios.md` (held-out eval sketches — the eval itself stays deferred), ADR-013 (build-first/migrate-incrementally precedent), ADR-018 (three-place plugin registration, followed during install), ADR-019 (plugin conventions).
 - **Status**: Accepted 2026-07-03. Plugin live at `cc-os/plugins/os-adr/`, installed as `os-adr@local-plugins`. Rollout order locked: pilot projects' real migration (interactive, per project) → cc-os retrofit → wider.
 
+## ADR-021 — Model-tier skill-execution eval for os-adr (Eval A), run in-session via subagents
+
+_Date: 2026-07-03_
+
+- **Context**: The os-adr build validated every model-touching surface (skill-following, migrate fills, find judgment) only at the session model used to build it (Fable). Confidence that the skills execute correctly on the *weakest* tier (Haiku) would imply confidence on all stronger tiers. The locked eval methodology (`04-plugin-requirements.md`) covers only the deferred held-out unprompted-behavior eval (requirements 4–5, "Eval B") and says nothing about model tiers. Separately, headless `claude -p` runs consume more of the user's subscription credit than in-session work.
+- **Decision**: (1) A second, non-held-out eval — **Eval A: prompted skill-execution across model tiers** — lives at `plugins/os-adr/eval/`: two fixture projects (generated with the plugin's own CLIs), six scenarios (S1 create, S2 create+supersede, S3 find/conflict, S4 find/distractor, S5 init, S6 migrate+fills), a deterministic Ruby checker (`bin/check`, structural invariants only — never prose quality), sandbox + headless-runner scripts, and a runner-prompt template. (2) **Primary run mode is in-session**: a driver session spawns Agent-tool subagents with `model:` pinned to the tier under test; each subagent reads the SKILL.md file directly (uniform across tiers; dispatch is deterministic plumbing). Headless `claude -p` is the fidelity fallback only. (3) The **`autoresearch` skill (Classic mode)** wraps the grid as its metric to iterate SKILL.md *wording only* — checker, fixtures, scenarios, and runner prompt are frozen during a loop, as the guard against metric-gaming. (4) **Eval B gains a model axis** when it eventually runs: scenario × model tier, pass rate per tier as the autoresearch metric.
+- **Rationale**: The deterministic core (CLIs, hook, index) is model-independent and already tested; what varies by tier is instruction-following, so the cheap, high-signal eval is exactly that surface with machine-checkable pass criteria (the plugin's invariants double as scoring rules). In-session subagents preserve the user's credits and parallelize; the fidelity gap (no SessionStart context, no slash dispatch) is irrelevant for *explicitly invoked* skills. Eval A prompts are deliberately not held-out — only Eval B's are — so they can be iterated on freely without contaminating the deferred methodology.
+- **Alternatives rejected**:
+  - **Headless-only harness**: higher subscription cost per run, no parallel driver, no benefit for prompted-execution scoring; kept as fallback for full-fidelity checks.
+  - **Folding tier-testing into Eval B**: conflates two questions (can the model *execute* a skill it was told to run vs. does it *recognize* when to run one) and would burn held-out scenarios on mechanical failures.
+  - **LLM-judged scoring**: reintroduces the model-capability variable into the scorer; structural invariants are sufficient and reproducible.
+- **Cross-references**: `plugins/os-adr/eval/README.md` (procedure + autoresearch invocation), `docs/adr-system/06-eval-scenarios.md` (Eval B sketches, now noting the model axis), ADR-020 (pilot gate the S6 fixture mirrors).
+- **Status**: Accepted 2026-07-03. Harness built and self-tested (all six scenarios: perfect-run PASS, untouched-sandbox FAIL). The grid itself has not yet been run against haiku/sonnet.
+
 ## Rejected tools (summary)
 
 | Tool | Why rejected for our use |

openspec/changes/archive/2026-07-03-add-os-adr-eval-b-harness/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-07-03

openspec/changes/archive/2026-07-03-add-os-adr-eval-b-harness/design.md

@@ -0,0 +1,127 @@
+## Context
+
+`os-adr` ships with `docs/adr-system/04-plugin-requirements.md` requirements 4–5: the plugin
+must be effective at *unprompted* write-trigger recognition and *unprompted, correct* retrieval.
+Eval A (`plugins/os-adr/eval/`) only measures skill-*following* once explicitly invoked — its
+runner-prompt literally tells the model under test "the user has invoked `/os-adr:{{SKILL}}`".
+That architecture cannot answer requirements 4–5 by construction: the thing being measured
+(does the model notice on its own) is exactly what the runner prompt would be giving away.
+
+Eval A's driver used in-session Agent-tool subagents pinned to a model tier, and treated the
+missing SessionStart hook context as an acceptable fidelity gap ("irrelevant — these are
+explicit invocations"). For Eval B that gap is not acceptable: the SessionStart hook's existence
+note (naming `/os-adr:create` and `/os-adr:find`) is one of the two legitimate discovery paths
+being measured (the other being the model reasoning its own way there without any hook hint,
+e.g. via `docs/adr/` conventionally existing). If the hook never fires, we can't tell "the model
+didn't notice" from "the model was never given the hint a real session would have."
+
+Scenario shapes are frozen and out of scope here (`docs/adr-system/06-eval-scenarios.md`, 7
+scenarios: W1–W3 write-trigger, R1–R4 retrieval) — this design is about the execution harness
+only.
+
+## Goals / Non-Goals
+
+**Goals:**
+- A runner that executes the model-under-test in an environment where the real SessionStart
+  hook fires, real skills/CLIs are reachable, and no part of the prompt names the ADR system.
+- Fixtures realistic enough to support all 7 scenarios, including R2's distractor ADRs (2–3
+  near-miss, one Superseded) and R4's one-hop graph reachability (needs a real `graphify-out/`).
+- A checker that scores both axes (unprompted-consultation, correctness) with as much
+  determinism as possible, reserving LLM judgment only for genuinely ambiguous cases (e.g. did
+  the model's prose count as "proposing" an ADR).
+- Sandbox isolation so headless runs never mutate the canonical fixture or the real cc-os repo.
+- Output format (TSV or similar) compatible with a later `autoresearch` Classic-mode loop.
+
+**Non-Goals:**
+- Running the grid, scoring a real pass rate, or iterating on retrieval/trigger heuristics —
+  that is the deferred next stage, not this change.
+- Adding, removing, or reshaping scenario shapes in `06-eval-scenarios.md`.
+- Multi-machine or CI execution — local headless runs only, matching Eval A's scope.
+
+## Decisions
+
+**1. Headless (`claude -p`) is the sole runner mode; no subagent fallback.**
+Eval A offers both an in-session subagent path (cheaper, preferred) and a headless fallback
+(`bin/run-headless`) for full fidelity. Eval B inverts that: only headless mode is valid, because
+Agent-tool subagents share the parent session's already-loaded plugin/hook state and cwd
+semantics in ways that don't reproduce a fresh SessionStart firing against the sandbox directory.
+`claude -p --output-format stream-json --verbose --model <tier> --cwd <sandbox>` gives a fresh
+process, a real SessionStart event, and a full transcript to check against.
+*Alternative considered*: reuse Eval A's subagent shortcut for cost — rejected, since it would
+silently reintroduce the exact fidelity gap this eval exists to close.
+
+**2. Transcript-based mechanical check for axis (a) (unprompted consultation).**
+`--output-format stream-json --verbose` emits every tool_use block. The checker greps the JSONL
+for: a Skill/slash invocation matching `os-adr:*`, a `Bash` call matching `bin/adr-*`, or a
+`Read`/`Glob` touching `docs/adr/`. This is fully mechanical — no model judgment needed to
+detect "did it touch the ADR system at all."
+*Alternative considered*: ask an LLM judge to read the transcript and decide — rejected for axis
+(a) specifically, since tool-call presence is unambiguous and cheaper to check by grep.
+
+**3. Axis (b) (correctness) is deterministic where possible, LLM-judged only at the edges.**
+For R1–R4 (retrieval), the fixture encodes the single correct ADR ID; the checker mechanically
+confirms the transcript's retrieved/cited ADR ID matches (or, for R2, that it is NOT one of the
+seeded distractor IDs). For W1–W3 (write-trigger), "proposes... or asks whether to" is prose-
+shaped and not fully mechanical — a new `docs/adr/NNNN-*.md` file appearing in the sandbox is a
+mechanical PASS; absent that, a small rubric-bound LLM judge reads only the final transcript
+message (not the full transcript) and answers a yes/no: "did the agent explicitly propose
+recording this decision as an ADR, or ask whether to?" Kept as thin as possible to limit judgment
+surface, mirroring the `os-doc-hygiene`/`os-adr` classify-only-the-ambiguous-part pattern already
+used elsewhere in this repo.
+*Alternative considered*: fully mechanical W1–W3 checking via keyword grep on the final message
+(e.g. "ADR") — rejected as too brittle/gameable; a judge with a narrow, fixed rubric is more
+robust and still cheap since it reads one short message, not a transcript.
+
+**4. Dedicated Eval-B fixture project(s), not reuse of Eval A's fixtures or the real llf-schema
+pilot.**
+Eval A's fixtures are too small (4 ADRs, no graph) and R2/R4 need specific engineered content
+(a Superseded pair plus 2–3 near-misses; a real one-hop-only conflict via Graphify). Building
+`plugins/os-adr/eval-b/fixture/` fresh, with `docs/adr/` populated per-scenario and a genuinely
+built `graphify-out/` (via the real `graphify` binary against the fixture's own small codebase,
+not a hand-authored stub) keeps the graph-layer degradation check (R4) honest.
+*Alternative considered*: reuse the llf-schema pilot project directly — rejected; its exclude
+list and ADR set are project-specific and not shaped for the distractor/one-hop scenarios,
+and would tie the eval's stability to project state a migration exercise might later change.
+
+**5. Per-cell repeat count is a runner flag, default left as an open question for the eval-
+running stage, not fixed by this design.**
+Unprompted behavior is more stochastic than the explicit-invocation behavior Eval A measures.
+The harness supports `--reps N` on the sandbox/runner tooling, but this change does not decide
+whether the eventual grid uses N=1 or N=3 — that's a call for whoever runs the eval, informed by
+observed variance in a pilot run.
+
+## Risks / Trade-offs
+
+- **[Risk] Headless runs perform real, uncontained tool use (file edits, possibly destructive
+  commands) against the sandbox.** → Mitigation: every run is a fresh git-initialized sandbox
+  copy (mirrors Eval A's `bin/sandbox` pattern) under a scratch/tmp root, never the canonical
+  fixture or the cc-os repo; sandbox is disposable and diffed/discarded after checking.
+- **[Risk] Cost — headless full-session runs are pricier than Eval A's subagent shortcut.** →
+  Mitigation: accepted deliberately (see Decision 1); this is the whole reason the fidelity gap
+  matters here. Reps default is left open (Decision 5) specifically to let the eval-running
+  stage control cost/variance trade-off with real data instead of a guess baked in now.
+- **[Risk] LLM-judged axis (b) for W1–W3 could itself be inconsistent or gameable over
+  `autoresearch` iterations.** → Mitigation: judge rubric and prompt are frozen alongside the
+  checker/fixtures/scenarios during any future `autoresearch` loop (same discipline Eval A's
+  README already imposes on its own checker), and Decision 3 keeps the judge's input surface
+  (one final message, not a transcript) as narrow as possible.
+- **[Risk] A genuinely-built `graphify-out/` for the fixture could drift if `graphify`'s output
+  format changes.** → Mitigation: the fixture's `graphify-out/` is rebuilt by an explicit fixture
+  setup step (documented in the harness README), not committed as a frozen binary blob, so it's
+  always regenerated against whatever `graphify` version is installed.
+
+## Migration Plan
+
+Not applicable — this is a new, additive test-harness directory with no effect on shipped plugin
+behavior. No rollback beyond deleting `plugins/os-adr/eval-b/` if abandoned.
+
+## Open Questions
+
+- Default `--reps` per cell for the eventual grid run (left to the eval-running stage; see
+  Decision 5).
+- Whether the R4 degradation check (retrieval must fail without `graphify-out/`) needs its own
+  sandbox variant with the graph directory deliberately removed, or whether one fixture toggling
+  a flag suffices — resolve during tasks/implementation, not architecturally significant enough
+  to block this design.
+- Exact rubric wording for the W1–W3 LLM judge — drafted during implementation (tasks.md), not
+  fixed here.

openspec/changes/archive/2026-07-03-add-os-adr-eval-b-harness/proposal.md

@@ -0,0 +1,54 @@
+## Why
+
+The `os-adr` plugin's two hardest requirements — unprompted write-trigger recognition and
+unprompted correct retrieval (`04-plugin-requirements.md` reqs 4–5) — have never been measured.
+Eval A (`plugins/os-adr/eval/`) only proves the plugin executes correctly once explicitly
+invoked; it says nothing about whether an agent notices, on its own, that the ADR system is
+relevant. The scenario *shapes* for this held-out eval already exist
+(`docs/adr-system/06-eval-scenarios.md`) but no fixtures, prompts, checker, or runner exist yet.
+Building it now closes the last unmeasured requirement in the locked rollout order.
+
+## What Changes
+
+- Build a new eval harness, `plugins/os-adr/eval-b/`, sibling to but structurally distinct from
+  Eval A, for the 7 held-out scenarios (W1–W3 write-trigger, R1–R4 retrieval).
+- Author full scenario prompts/fixtures from the frozen shapes in `06-eval-scenarios.md` —
+  scenario *shapes* are not renegotiated by this change; only their execution artifacts are new.
+- Fixture project(s): a realistic, onboarded project state (populated `docs/adr/` incl. a
+  Superseded pair for R2 distractors, a built `graphify-out/` for R4) that the SessionStart hook
+  recognizes as ADR-initialized.
+- A **headless runner** (real `claude -p` in a real sandbox cwd) as the primary execution mode,
+  not an in-session Agent-tool subagent — because the SessionStart hook firing for real is part
+  of what's under test (Eval A's subagent shortcut is not valid here; see design.md).
+- A deterministic-first, two-axis checker: (a) did the agent consult/propose the ADR system
+  unprompted at all, (b) did it act on/write the *specific correct* thing. Axis (a) is
+  mechanically checkable (tool-call/transcript grep for `/os-adr:*` or `docs/adr/` reads/writes);
+  axis (b) requires a small LLM-judged correctness check against the fixture's known-correct
+  answer, kept as thin and rubric-bound as possible.
+- Sandbox/grid tooling mirroring Eval A's shape (`bin/sandbox`, `bin/check`, results.tsv) adapted
+  for headless execution and the two-axis score.
+- Wiring for the later `autoresearch` iteration stage (not run by this change — building the
+  harness only).
+
+Out of scope for this change: actually running the eval grid, iterating retrieval/trigger
+heuristics, or adding/altering scenario shapes in `06-eval-scenarios.md`.
+
+## Capabilities
+
+### New Capabilities
+- `adr-eval-b-harness`: fixtures, held-out scenario prompts, headless sandbox/runner tooling, and
+  a two-axis deterministic-first checker for the unprompted write-trigger and retrieval eval.
+
+### Modified Capabilities
+(none — this adds a new, separate eval surface; it does not change the behavior of
+`adr-authoring`, `adr-retrieval`, `adr-session-awareness`, or any shipped plugin capability)
+
+## Impact
+
+- New directory: `plugins/os-adr/eval-b/` (fixtures, scenarios, bin/, runner assets, README).
+- No changes to plugin source (`lib/adr/`, `bin/adr-*`, `hooks/`, `skills/`) — this is a pure
+  test-harness addition.
+- Depends on `claude` CLI headless mode (`claude -p`) being available in the build/dev
+  environment for the runner to execute against a real sandbox with hooks firing.
+- Once built, unlocks the deferred final stage of the locked os-adr rollout order (running the
+  grid + `autoresearch` iteration), tracked separately in `CLAUDE.md`.

openspec/changes/archive/2026-07-03-add-os-adr-eval-b-harness/specs/adr-eval-b-harness/spec.md

@@ -0,0 +1,67 @@
+## ADDED Requirements
+
+### Requirement: Held-out scenario fixtures
+The harness SHALL provide a dedicated fixture project (or projects) under
+`plugins/os-adr/eval-b/fixture/` that supports all 7 held-out scenarios (W1–W3, R1–R4) without
+any scenario prompt naming the ADR system, an ADR ID, or the exact constraint text being tested.
+
+#### Scenario: R2 distractor set present
+- **WHEN** the R2 fixture's `docs/adr/` is inspected
+- **THEN** it contains the one correct Accepted ADR plus 2–3 near-miss ADRs (same component
+  family, different decision, or Superseded status) that a naive retrieval could mistakenly cite
+
+#### Scenario: R4 graph reachability is real, not stubbed
+- **WHEN** the R4 fixture's `graphify-out/` is inspected
+- **THEN** it was produced by running the real `graphify` binary against the fixture's own
+  codebase (not hand-authored), and the conflicting files are one import/reference hop away from
+  files listed in the relevant ADR's `affected-paths`, not directly listed themselves
+
+### Requirement: Headless runner with real hook firing
+The harness SHALL execute each scenario via a headless `claude -p` process with its working
+directory set to a fresh sandbox copy of the fixture, so that the real SessionStart hook fires
+for the model under test. In-session Agent-tool subagents SHALL NOT be used as the execution
+mode for this eval.
+
+#### Scenario: SessionStart hook context reaches the model under test
+- **WHEN** a scenario is run via the headless runner against an ADR-initialized sandbox
+- **THEN** the transcript shows the SessionStart hook's additionalContext was present in the
+  model's context before its first action
+
+#### Scenario: Sandbox isolation
+- **WHEN** any scenario run completes (pass or fail)
+- **THEN** the canonical fixture directory and the cc-os repo itself show no modifications —
+  only the disposable sandbox copy was touched
+
+### Requirement: Two-axis deterministic-first checker
+The harness SHALL score each run on two independent axes: (a) whether the model consulted or
+proposed the ADR system at all, unprompted, and (b) whether it acted on or wrote the specific
+correct thing rather than a merely plausible one. Axis (a), and axis (b) for retrieval scenarios,
+SHALL be scored mechanically from the transcript and sandbox file state; axis (b) for
+write-trigger scenarios MAY fall back to a narrow, rubric-bound LLM judge only when no new ADR
+file was mechanically created.
+
+#### Scenario: Axis (a) mechanical detection
+- **WHEN** the checker scans a scenario's `stream-json` transcript
+- **THEN** it detects axis (a) as true if any tool_use block invokes an `os-adr:*` skill, a
+  `bin/adr-*` CLI, or a Read/Glob on `docs/adr/`, with no model call required
+
+#### Scenario: Axis (b) retrieval correctness
+- **WHEN** scoring an R1–R4 scenario's axis (b)
+- **THEN** the checker mechanically compares the ADR ID the transcript cites or acts on against
+  the fixture's pre-declared correct ID, and fails if it matches a seeded distractor instead
+
+#### Scenario: Axis (b) write-trigger fallback judge
+- **WHEN** scoring a W1–W3 scenario where no new `docs/adr/NNNN-*.md` file was created in the
+  sandbox
+- **THEN** the checker invokes a rubric-bound LLM judge on only the model's final message (not
+  the full transcript) to decide whether it explicitly proposed or asked about recording an ADR
+
+### Requirement: Grid-compatible output
+The harness SHALL emit per-run results in a format usable by a later `autoresearch` Classic-mode
+loop (scenario, model tier, axis-a result, axis-b result, pass/fail), analogous to Eval A's
+`results.tsv`.
+
+#### Scenario: TSV row per run
+- **WHEN** a scenario run is checked
+- **THEN** the checker can emit one TSV row identifying the scenario, model tier, both axis
+  results, and overall pass/fail, appendable to a shared results file

openspec/changes/archive/2026-07-03-add-os-adr-eval-b-harness/tasks.md

@@ -0,0 +1,85 @@
+## 1. Fixture project(s)
+
+- [x] 1.1 Scaffold `plugins/os-adr/eval-b/fixture/` as a small, self-contained fixture codebase
+      (init as its own git repo, mirroring Eval A's `fixture/project/` pattern)
+- [x] 1.2 Author `docs/adr/` content for W1–W3 (a plausible existing convention/decision history
+      the scenarios can plausibly reverse or extend) and initialize the ADR index
+- [x] 1.3 Author R1's Accepted ADR with `affected-paths` covering specific fixture files whose
+      obvious next change would violate it
+- [x] 1.4 Extend `docs/adr/` with R2's 2–3 distractor ADRs (near-miss same-component-family,
+      plus one Superseded) alongside the correct one
+- [x] 1.5 Add an R3 Accepted ADR whose content answers a "how should we…" question phrased in
+      different vocabulary than the ADR's own text
+- [x] 1.6 Design R4's one-hop layout: files that import/reference an R1-style ADR's
+      `affected-paths` without being listed themselves; run the real `graphify` binary to produce
+      `graphify-out/` and confirm the hop is actually one edge away in the resulting graph
+- [x] 1.7 Document a fixture regeneration step (README) that rebuilds `graphify-out/` from
+      scratch rather than committing it as a frozen blob (`bin/build-fixture-graph` +
+      "Fixture regeneration" section in `eval-b/README.md`; script asserts the R4 one-hop
+      invariant on every rebuild)
+
+## 2. Scenario prompts
+
+- [x] 2.1 Write `eval-b/scenarios/W1.md`..`W3.md` and `R1.md`..`R4.md`: a task prompt per
+      scenario that never names the ADR system, plugin, or exact constraint text, plus the
+      checker-facing metadata (correct ADR ID for R1–R4; expected new-ADR shape for W1–W3)
+      following the pass/fail language already sketched in `06-eval-scenarios.md`
+- [x] 2.2 Self-review each prompt against the "held-out" ground rule before running anything
+      against it (no informal trial runs — first real execution is the self-test in section 5)
+      (R1's "skip the shared client" wording was flagged and removed in this review)
+
+## 3. Headless runner
+
+- [x] 3.1 Write `eval-b/bin/sandbox <scenario> <dest>`: fresh git-initialized sandbox copy of the
+      right fixture variant (mirrors Eval A's `bin/sandbox`; `R4-nograph` variant included)
+- [x] 3.2 Write `eval-b/bin/run <scenario> <model> <sandbox>`: invokes
+      `claude -p --output-format stream-json --verbose --model <tier>` with cwd set to the
+      sandbox, passing only the scenario's task prompt (no system-level hints), capturing the
+      full JSONL transcript to a file in the sandbox
+- [x] 3.3 Confirm the os-adr plugin (and, for R4, graphify) are active in the environment the
+      runner executes in, and that a sandbox with an initialized `docs/adr/` triggers the
+      SessionStart hook's "present" branch (verified via a neutral non-scenario probe prompt:
+      haiku quoted the [os-adr] note verbatim; transcript carries a system/hook_response event)
+- [x] 3.4 Add `--reps N` support to the runner for repeated executions of the same cell (default
+      left open per design.md Decision 5; document how to override)
+
+## 4. Checker
+
+- [x] 4.1 Write `eval-b/bin/check <scenario> <sandbox> [--tsv <model>]`: parses the JSONL
+      transcript for axis (a) (tool_use blocks touching `os-adr:*`, `bin/adr-*`, or
+      `docs/adr/` reads/globs)
+- [x] 4.2 Implement axis (b) for R1–R4: compare the transcript's cited/acted-on ADR ID against
+      the scenario's pre-declared correct ID; fail on distractor match (R2) or on missing
+      graph-layer reach (R4)
+- [x] 4.3 Implement axis (b) for W1–W3: mechanical PASS if a new `docs/adr/NNNN-*.md` file
+      exists in the sandbox matching the scenario's expected shape; otherwise invoke the narrow
+      LLM judge (final message only) with a fixed rubric to decide propose/ask-vs-silent
+      (rubric frozen in `eval-b/judge-rubric.md`; stubbable via ADR_EVAL_B_JUDGE_CMD)
+- [x] 4.4 Emit the TSV row format (scenario, model tier, axis-a, axis-b, pass/fail) for
+      `autoresearch` compatibility
+- [x] 4.5 Write the R4 degradation-check variant: same scenario run against a sandbox with
+      `graphify-out/` removed, expected to FAIL where the graph-layer path was required
+      (`bin/sandbox R4-nograph` + `bin/check R4-nograph` alias)
+
+## 5. Self-test (mirrors Eval A's own-both-directions check)
+
+- [x] 5.1 For each of the 7 scenarios, hand-simulate a "perfect" transcript/sandbox state (an
+      agent that does the right thing) and confirm `bin/check` scores PASS on both axes
+      (scripted as `eval-b/bin/self-test`; includes the W judge-fallback path with a stub judge,
+      plus a one-off smoke test of the real haiku judge)
+- [x] 5.2 For each of the 7 scenarios, check an untouched sandbox (no ADR consultation at all)
+      and confirm `bin/check` scores axis (a) FAIL (also covers R4-nograph, the R2
+      superseded-distractor trap, and the missing-hook-context invalid-run case)
+- [x] 5.3 Confirm sandbox isolation: after a run, diff the canonical fixture and the cc-os repo
+      and verify neither was modified (fixture digest check inside self-test; git status clean
+      of unexpected entries)
+
+## 6. Documentation
+
+- [x] 6.1 Write `eval-b/README.md` mirroring Eval A's README shape: layout table, how to run a
+      single cell, how the two-axis scoring works, explicit warning that scenario prompts are
+      held-out and must not be informally tried out outside this harness's own self-test
+- [x] 6.2 Cross-link from `CLAUDE.md`'s os-adr section once this change is applied and archived
+      (status line update per this repo's "keep this file current" convention) — Eval B bullet
+      added to the os-adr component section, "Remaining" rollout line updated, and
+      `06-eval-scenarios.md` status line now points at the built harness

openspec/specs/adr-eval-b-harness/spec.md

@@ -0,0 +1,71 @@
+## Purpose
+
+Evaluation harness B: held-out, unprompted-behavior testing for the os-adr plugin. Tests whether the plugin's core write/retrieval operations surface naturally without coaching, across multiple model tiers, with deterministic-first scoring (mechanical checks before fallback to LLM judge).
+
+## Requirements
+
+### Requirement: Held-out scenario fixtures
+The harness SHALL provide a dedicated fixture project (or projects) under
+`plugins/os-adr/eval-b/fixture/` that supports all 7 held-out scenarios (W1–W3, R1–R4) without
+any scenario prompt naming the ADR system, an ADR ID, or the exact constraint text being tested.
+
+#### Scenario: R2 distractor set present
+- **WHEN** the R2 fixture's `docs/adr/` is inspected
+- **THEN** it contains the one correct Accepted ADR plus 2–3 near-miss ADRs (same component
+  family, different decision, or Superseded status) that a naive retrieval could mistakenly cite
+
+#### Scenario: R4 graph reachability is real, not stubbed
+- **WHEN** the R4 fixture's `graphify-out/` is inspected
+- **THEN** it was produced by running the real `graphify` binary against the fixture's own
+  codebase (not hand-authored), and the conflicting files are one import/reference hop away from
+  files listed in the relevant ADR's `affected-paths`, not directly listed themselves
+
+### Requirement: Headless runner with real hook firing
+The harness SHALL execute each scenario via a headless `claude -p` process with its working
+directory set to a fresh sandbox copy of the fixture, so that the real SessionStart hook fires
+for the model under test. In-session Agent-tool subagents SHALL NOT be used as the execution
+mode for this eval.
+
+#### Scenario: SessionStart hook context reaches the model under test
+- **WHEN** a scenario is run via the headless runner against an ADR-initialized sandbox
+- **THEN** the transcript shows the SessionStart hook's additionalContext was present in the
+  model's context before its first action
+
+#### Scenario: Sandbox isolation
+- **WHEN** any scenario run completes (pass or fail)
+- **THEN** the canonical fixture directory and the cc-os repo itself show no modifications —
+  only the disposable sandbox copy was touched
+
+### Requirement: Two-axis deterministic-first checker
+The harness SHALL score each run on two independent axes: (a) whether the model consulted or
+proposed the ADR system at all, unprompted, and (b) whether it acted on or wrote the specific
+correct thing rather than a merely plausible one. Axis (a), and axis (b) for retrieval scenarios,
+SHALL be scored mechanically from the transcript and sandbox file state; axis (b) for
+write-trigger scenarios MAY fall back to a narrow, rubric-bound LLM judge only when no new ADR
+file was mechanically created.
+
+#### Scenario: Axis (a) mechanical detection
+- **WHEN** the checker scans a scenario's `stream-json` transcript
+- **THEN** it detects axis (a) as true if any tool_use block invokes an `os-adr:*` skill, a
+  `bin/adr-*` CLI, or a Read/Glob on `docs/adr/`, with no model call required
+
+#### Scenario: Axis (b) retrieval correctness
+- **WHEN** scoring an R1–R4 scenario's axis (b)
+- **THEN** the checker mechanically compares the ADR ID the transcript cites or acts on against
+  the fixture's pre-declared correct ID, and fails if it matches a seeded distractor instead
+
+#### Scenario: Axis (b) write-trigger fallback judge
+- **WHEN** scoring a W1–W3 scenario where no new `docs/adr/NNNN-*.md` file was created in the
+  sandbox
+- **THEN** the checker invokes a rubric-bound LLM judge on only the model's final message (not
+  the full transcript) to decide whether it explicitly proposed or asked about recording an ADR
+
+### Requirement: Grid-compatible output
+The harness SHALL emit per-run results in a format usable by a later `autoresearch` Classic-mode
+loop (scenario, model tier, axis-a result, axis-b result, pass/fail), analogous to Eval A's
+`results.tsv`.
+
+#### Scenario: TSV row per run
+- **WHEN** a scenario run is checked
+- **THEN** the checker can emit one TSV row identifying the scenario, model tier, both axis
+  results, and overall pass/fail, appendable to a shared results file

plugins/os-adr/eval-b/README.md

@@ -0,0 +1,97 @@
+# os-adr Eval B — unprompted write-trigger & retrieval (held-out)
+
+_Last updated: 2026-07-03 — harness built and self-tested (perfect-run PASS,
+untouched-sandbox FAIL, both directions for all 7 scenarios); grid not yet run._
+
+Measures the two hardest os-adr requirements (`docs/adr-system/04-plugin-requirements.md`
+reqs 4–5): does an agent notice **on its own** that the ADR system is relevant — proposing to
+record a consequential decision (W1–W3), or retrieving the specific correct decision before
+changing decided-on behavior (R1–R4)? Scenario shapes are frozen in
+`docs/adr-system/06-eval-scenarios.md`. This is **not** Eval A (`../eval/`), which measures
+skill-*execution* once explicitly invoked — do not conflate them (ADR-021).
+
+> **HELD-OUT — read this first.** The `## Task` blocks in `scenarios/*.md` must never be
+> pasted into an interactive session, "tried out" informally, or fed to a model outside
+> `bin/run`. The only permitted non-grid execution is `bin/self-test`, which fabricates
+> transcripts and never uses the Task blocks. Informal trials contaminate the methodology.
+
+## Layout
+
+| Path | What |
+| --- | --- |
+| `fixture/project/` | small Ruby webhook-relay codebase, 6-ADR history (incl. a Superseded pair) |
+| `scenarios/W1..W3.md` | write-trigger prompts (persistence choice, convention change, reversal) |
+| `scenarios/R1..R4.md` | retrieval prompts (direct conflict, distractors, mid-task Q, graph hop) |
+| `bin/build-fixture-graph` | rebuild `fixture/project/graphify-out/` from scratch (R4 needs it) |
+| `bin/sandbox <Sn> <dest>` | fresh git-initialized sandbox; only `R4` keeps the graph; `R4-nograph` = degradation variant |
+| `bin/run <Sn> <model> <workdir> [--reps N]` | headless `claude -p` runner; the ONLY valid execution mode |
+| `bin/check <Sn> <sandbox> [--tsv <model>]` | two-axis checker; exit 0/1; TSV mode for autoresearch |
+| `bin/self-test [workdir]` | model-free both-directions harness self-test |
+| `judge-rubric.md` | frozen rubric for the W-scenario axis (b) LLM judge fallback |
+
+## Why headless-only (no Agent-tool subagents)
+
+The SessionStart hook's `[os-adr]` note is one of the two discovery paths under measurement.
+In-session subagents don't get a fresh SessionStart against the sandbox cwd, so they can't
+tell "the model didn't notice" from "the model never got the hint a real session would have."
+`bin/run` gives each rep a fresh `claude -p` process with cwd set to a fresh sandbox; the
+transcript's `system/hook_response` event proves the hook context reached the model — the
+checker treats a transcript without it as an invalid run. (Verified 2026-07-03 with a neutral
+non-scenario probe: haiku quoted the hook note verbatim from a sandbox session.)
+
+## Two-axis scoring
+
+- **Axis (a) — unprompted consultation (mechanical).** Any `tool_use` block in the
+  `stream-json` transcript that invokes an `os-adr:*` skill, a `bin/adr-*` CLI, or touches
+  `docs/adr/`. Prose never counts, so the hook note can't false-positive it.
+- **Axis (b) — specific correctness.**
+  - R1–R4 (mechanical): assistant text must cite the scenario's pre-declared correct ADR ID
+    (R1→0002, R2→0003, R3→0005 content match [UTC + ISO-8601], R4→0003) and flag the
+    conflict. R2 fails if the Superseded distractor 0001 is cited as if live.
+  - W1–W3 (deterministic-first): PASS mechanically when a new `docs/adr/NNNN-*.md` matching
+    the scenario's topic exists; only otherwise does a rubric-bound judge (haiku,
+    `judge-rubric.md`, final message only) decide propose/ask-vs-silent. Stub it with
+    `ADR_EVAL_B_JUDGE_CMD` (reads prompt on stdin, prints YES/NO).
+- **Overall PASS** = both axes PASS. TSV row: `scenario  model  A:…  B:…  PASS|FAIL  reasons`.
+
+## Running one cell
+
+```bash
+eval-b/bin/build-fixture-graph                  # once, before any R4 cell
+eval-b/bin/run R2 haiku /tmp/adr-eval-b --reps 3 --results /tmp/adr-eval-b/results.tsv
+```
+
+Each rep: fresh sandbox → `claude -p --output-format stream-json --verbose --model <tier>
+--dangerously-skip-permissions "<Task block>"` with cwd = sandbox → `transcript.jsonl` saved →
+`bin/check` appends one TSV row. `--reps` default is 1; the grid-run stage picks the real
+repeat count after observing variance in a pilot (design decision, left open deliberately).
+
+The R4 degradation check: `bin/run R4-nograph <model> <workdir>` — same prompt, graph absent.
+An axis (b) FAIL there is the **expected** outcome; it confirms R4's pass genuinely required
+the Graphify layer (paired with an R4 PASS on the same tier).
+
+## Fixture regeneration
+
+`graphify-out/` is never committed — rebuild it with `bin/build-fixture-graph`, which runs the
+real `graphify update` against the fixture (pure tree-sitter AST: the fixture's
+`.graphifyignore` excludes all markdown, so no Ollama/LLM is involved and the build is
+reproducible) and then asserts the R4 one-hop invariant (`lib/relay/reports.rb` is exactly one
+graph hop from `lib/relay/delivery.rb`). If graphify's output format changes and the invariant
+breaks, fix the fixture layout — not the checker — in a human-reviewed change.
+
+The ADR history was generated with the plugin's own CLIs (`adr-init`/`adr-new`), so frontmatter
+and index match the shipped format, including the mechanically-superseded 0001→0003 pair.
+
+## Optimizing with `/autoresearch` (later stage — not part of building this harness)
+
+Same discipline as Eval A: the **checker, fixtures, scenario prompts, and judge rubric are
+frozen during a loop** — only plugin surface wording (SKILL.md, hook note text) may move. Metric:
+pass rate over the {W1..W3, R1..R4} × {haiku, sonnet} grid via `bin/check --tsv`. If a scenario
+or checker turns out to be wrong, stop the loop and fix it as a separate, human-reviewed change.
+Unlike Eval A, every cell costs a real headless session — budget reps accordingly.
+
+## Adding a scenario
+
+Add `scenarios/X.md` (Task block + checker metadata), a class in `bin/check`, the fixture route
+in `bin/sandbox`, extend `bin/self-test` both directions, and keep the held-out rule: the new
+Task block gets no informal trial runs.

plugins/os-adr/eval-b/bin/build-fixture-graph

@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+# Rebuild the fixture's graphify-out/ from scratch (needed by scenario R4).
+#
+# The graph is never committed — it is disposable and regenerated against
+# whatever graphify version is installed. The fixture's .graphifyignore
+# excludes all markdown/config, so this is a pure tree-sitter AST build:
+# no LLM, no Ollama, reproducible.
+#
+# Sanity-checked invariant: lib/relay/delivery.rb must be exactly one graph
+# hop from lib/relay/reports.rb (the R4 one-hop layout).
+set -euo pipefail
+
+FIXTURE="$(cd "$(dirname "$0")/../fixture/project" && pwd)"
+
+rm -rf "$FIXTURE/graphify-out"
+graphify update "$FIXTURE"
+
+python3 - "$FIXTURE/graphify-out/graph.json" <<'PY'
+import json, sys
+g = json.load(open(sys.argv[1]))
+nodes = {n["id"]: n.get("source_file") for n in g["nodes"]}
+seeds = {i for i, f in nodes.items() if f == "lib/relay/reports.rb"}
+neighbors = set()
+for link in g["links"]:
+    if link["source"] in seeds: neighbors.add(nodes.get(link["target"]))
+    if link["target"] in seeds: neighbors.add(nodes.get(link["source"]))
+assert "lib/relay/delivery.rb" in neighbors, (
+    "R4 one-hop invariant broken: reports.rb no longer reaches delivery.rb "
+    f"in one hop (neighbors: {sorted(f for f in neighbors if f)})")
+print("R4 one-hop invariant holds: reports.rb -> delivery.rb")
+PY

plugins/os-adr/eval-b/bin/check

@@ -0,0 +1,276 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+# Two-axis, deterministic-first checker for Eval B (unprompted behavior).
+#
+#   check <W1|W2|W3|R1|R2|R3|R4|R4-nograph> <sandbox-root> [--tsv <model-label>]
+#
+# Reads <sandbox>/transcript.jsonl (stream-json, written by bin/run) plus the
+# sandbox file state.
+#
+#   axis (a) — unprompted consultation: mechanical. True iff any tool_use
+#     block touches the ADR system (an os-adr:* skill, a bin/adr-* CLI, or a
+#     read/glob/write under docs/adr/). Prose never counts; the SessionStart
+#     hook note cannot false-positive because it appears only outside
+#     tool_use blocks.
+#   axis (b) — specific correctness:
+#     R1..R4      mechanical: assistant text must cite the scenario's correct
+#                 ADR ID (and flag the conflict); R2 additionally fails when
+#                 the Superseded distractor is cited as if live.
+#     W1..W3      mechanical PASS when a new docs/adr/NNNN-*.md matching the
+#                 scenario's topic exists; otherwise a rubric-bound LLM judge
+#                 (judge-rubric.md) reads ONLY the final message. Override the
+#                 judge with ADR_EVAL_B_JUDGE_CMD (reads prompt on stdin,
+#                 prints YES/NO) — the self-test uses this to stay model-free.
+#
+# Overall PASS requires both axes. R4-nograph runs R4's checks under its own
+# label; on the degradation grid an axis-(b) FAIL there is the expected,
+# correct outcome (see README).
+#
+# TSV mode: scenario, model, axis-a, axis-b, PASS|FAIL, reasons.
+# Exit 0 on PASS, 1 on FAIL, 2 on usage error.
+
+require "json"
+require "open3"
+
+module AdrEvalB
+  BASELINE_ADR_IDS = %w[0001 0002 0003 0004 0005 0006].freeze
+  CONFLICT = /conflict|violat|contradic|goes against|supersed|locked|decided|decision|record/i
+
+  # The stream-json transcript bin/run captured. Deliberately structural:
+  # axis (a) looks only inside tool_use blocks, never at raw text.
+  class Transcript
+    def self.load(path)
+      return nil unless File.exist?(path)
+      events = File.readlines(path).filter_map do |line|
+        JSON.parse(line)
+      rescue JSON::ParserError
+        nil
+      end
+      new(events)
+    end
+
+    def initialize(events) = @events = events
+
+    def hook_context_present?
+      @events.any? do |e|
+        e["type"] == "system" && e["subtype"] == "hook_response" &&
+          e["hook_name"].to_s.start_with?("SessionStart") &&
+          e.to_json.include?("[os-adr]")
+      end
+    end
+
+    def tool_uses
+      assistant_blocks.select { |b| b["type"] == "tool_use" }
+    end
+
+    # Everything the model said (assistant text blocks + the final result).
+    def assistant_text
+      texts = assistant_blocks.select { |b| b["type"] == "text" }.map { |b| b["text"] }
+      texts << final_message
+      texts.compact.join("\n")
+    end
+
+    def final_message
+      result = @events.reverse.find { |e| e["type"] == "result" }
+      result && result["result"].is_a?(String) ? result["result"] : nil
+    end
+
+    private
+
+    def assistant_blocks
+      @events.select { |e| e["type"] == "assistant" }
+             .flat_map { |e| e.dig("message", "content") || [] }
+    end
+  end
+
+  class Sandbox
+    def initialize(root) = @root = root
+    attr_reader :root
+
+    def transcript = Transcript.load(File.join(@root, "transcript.jsonl"))
+
+    def new_adr_files
+      Dir.glob(File.join(@root, "docs/adr/[0-9]*.md")).sort.reject do |path|
+        BASELINE_ADR_IDS.include?(File.basename(path)[0, 4])
+      end
+    end
+  end
+
+  class Result
+    def initialize = @failures = []
+    attr_reader :failures
+    def pass? = @failures.empty?
+
+    def expect(condition, reason)
+      @failures << reason unless condition
+      !!condition
+    end
+  end
+
+  # Narrow LLM fallback for W-scenario axis (b): rubric + final message only.
+  class Judge
+    RUBRIC = File.read(File.expand_path("../judge-rubric.md", __dir__))
+                 .split("---", 2).last.strip
+
+    def self.command
+      ENV.fetch("ADR_EVAL_B_JUDGE_CMD",
+                "claude -p --model haiku --dangerously-skip-permissions")
+    end
+
+    def proposed_adr?(final_message)
+      prompt = "#{RUBRIC}\n\n#{final_message}"
+      output, status = Open3.capture2(self.class.command, stdin_data: prompt)
+      raise "judge command failed: #{self.class.command}" unless status.success?
+      verdict = output[/\b(YES|NO)\b/, 1]
+      raise "judge gave no YES/NO verdict: #{output.inspect}" if verdict.nil?
+      verdict == "YES"
+    end
+  end
+
+  class Scenario
+    def initialize(sandbox, judge: Judge.new)
+      @sb = sandbox
+      @judge = judge
+    end
+
+    def run
+      axis_a = Result.new
+      axis_b = Result.new
+      transcript = @sb.transcript
+      if transcript.nil?
+        axis_a.expect(false, "no transcript.jsonl in sandbox")
+        axis_b.expect(false, "no transcript.jsonl in sandbox")
+      elsif !transcript.hook_context_present?
+        axis_a.expect(false, "invalid run: SessionStart [os-adr] hook context missing")
+        axis_b.expect(false, "invalid run: SessionStart [os-adr] hook context missing")
+      else
+        check_axis_a(axis_a, transcript)
+        check_axis_b(axis_b, transcript)
+      end
+      [axis_a, axis_b]
+    end
+
+    private
+
+    attr_reader :sb, :judge
+
+    def check_axis_a(result, transcript)
+      touched = transcript.tool_uses.any? { |tu| adr_touch?(tu) }
+      result.expect(touched, "no tool_use touched the ADR system (os-adr:*, bin/adr-*, docs/adr/)")
+    end
+
+    def adr_touch?(tool_use)
+      probe = "#{tool_use['name']} #{tool_use['input'].to_json}"
+      probe.match?(%r{os-adr|bin/adr-|adr-(find|new|init|detect|migrate)\b|docs/adr})
+    end
+  end
+
+  # --- Retrieval scenarios: mechanical ID + conflict-flag checks ------------
+
+  class Retrieval < Scenario
+    def check_axis_b(result, transcript)
+      text = transcript.assistant_text
+      result.expect(text.match?(/\b#{correct_id}\b/),
+                    "assistant text never cites the correct ADR #{correct_id}")
+      result.expect(text.match?(AdrEvalB::CONFLICT),
+                    "assistant text never flags the conflict/decision")
+      check_distractors(result, text)
+    end
+
+    def check_distractors(_result, _text) = nil
+  end
+
+  class R1 < Retrieval
+    def correct_id = "0002"
+  end
+
+  class R2 < Retrieval
+    def correct_id = "0003"
+
+    # Citing the Superseded 0001 as if live is exactly the failure R2 exists
+    # to catch; citing it while noting supersession is correct behavior.
+    def check_distractors(result, text)
+      if text.match?(/\b0001\b/)
+        result.expect(text.match?(/supersed/i),
+                      "cites Superseded distractor 0001 without noting it is superseded")
+      end
+    end
+  end
+
+  class R3 < Retrieval
+    def correct_id = "0005"
+
+    # Correctness for R3 is answering the question from the ADR: UTC ISO-8601.
+    def check_axis_b(result, text_or_transcript)
+      text = text_or_transcript.assistant_text
+      result.expect(text.match?(/UTC/i), "answer never says UTC")
+      result.expect(text.match?(/ISO.?8601/i), "answer never says ISO-8601")
+    end
+  end
+
+  class R4 < Retrieval
+    def correct_id = "0003"
+  end
+
+  # --- Write-trigger scenarios: new-ADR-file check, judge fallback ----------
+
+  class WriteTrigger < Scenario
+    def check_axis_b(result, transcript)
+      new_files = sb.new_adr_files
+      if new_files.any?
+        matching = new_files.select { |p| File.read(p).match?(topic) }
+        result.expect(matching.any?,
+                      "new ADR exists but matches no topic term #{topic.inspect}")
+        return
+      end
+      final = transcript.final_message
+      unless final
+        result.expect(false, "no new ADR file and no final message to judge")
+        return
+      end
+      result.expect(judge.proposed_adr?(final),
+                    "no new ADR file; judge: final message does not propose/ask to record the decision")
+    end
+  end
+
+  class W1 < WriteTrigger
+    def topic = /persist|stor(e|age|ing)|database|sqlite|history/i
+  end
+
+  class W2 < WriteTrigger
+    def topic = /error|exception|result|raise|failure/i
+  end
+
+  class W3 < WriteTrigger
+    def topic = /concurren|thread|parallel|order|sequential|queue/i
+  end
+
+  SCENARIOS = {
+    "W1" => W1, "W2" => W2, "W3" => W3,
+    "R1" => R1, "R2" => R2, "R3" => R3, "R4" => R4,
+    "R4-nograph" => R4 # same checks; expected to FAIL axis (b) — see README
+  }.freeze
+end
+
+scenario_id, sandbox_root = ARGV[0], ARGV[1]
+tsv_model = ARGV[2] == "--tsv" ? (ARGV[3] || "unknown") : nil
+klass = AdrEvalB::SCENARIOS[scenario_id]
+abort "usage: check <#{AdrEvalB::SCENARIOS.keys.join('|')}> <sandbox-root> [--tsv <model>]" if klass.nil? || sandbox_root.nil?
+abort "no such sandbox: #{sandbox_root}" unless File.directory?(sandbox_root)
+
+axis_a, axis_b = klass.new(AdrEvalB::Sandbox.new(File.expand_path(sandbox_root))).run
+overall = axis_a.pass? && axis_b.pass?
+reasons = (axis_a.failures + axis_b.failures).join("; ")
+
+if tsv_model
+  puts [scenario_id, tsv_model,
+        axis_a.pass? ? "A:PASS" : "A:FAIL",
+        axis_b.pass? ? "B:PASS" : "B:FAIL",
+        overall ? "PASS" : "FAIL", reasons].join("\t")
+else
+  puts "#{overall ? 'PASS' : 'FAIL'} #{scenario_id}  " \
+       "(axis-a #{axis_a.pass? ? 'PASS' : 'FAIL'}, axis-b #{axis_b.pass? ? 'PASS' : 'FAIL'})"
+  (axis_a.failures + axis_b.failures).each { |f| puts "  - #{f}" }
+end
+exit(overall ? 0 : 1)

plugins/os-adr/eval-b/bin/run

@@ -0,0 +1,55 @@
+#!/usr/bin/env bash
+# Headless runner for Eval B — the ONLY valid execution mode for these
+# scenarios (design.md Decision 1): a fresh `claude -p` process with cwd set
+# to the sandbox, so the real SessionStart hook fires for the model under
+# test. Never run these prompts via in-session Agent-tool subagents.
+#
+# Usage: run <scenario> <model> <workdir> [--reps N] [--results FILE]
+#
+#   scenario  W1|W2|W3|R1|R2|R3|R4|R4-nograph
+#   model     haiku|sonnet|opus|...
+#   workdir   sandboxes are created under here as <scenario>-<model>-rN
+#   --reps    repeated executions of the same cell (default 1; the grid-run
+#             stage decides the real default — design.md Decision 5)
+#
+# Each rep: fresh sandbox -> claude -p with ONLY the scenario's task prompt
+# (no system-level hints) -> full stream-json transcript saved to
+# <sandbox>/transcript.jsonl -> bin/check appends one TSV row to RESULTS.
+set -euo pipefail
+
+SCENARIO="${1:?usage: run <scenario> <model> <workdir> [--reps N] [--results FILE]}"
+MODEL="${2:?model required (haiku|sonnet|...)}"
+WORKDIR="${3:?workdir required}"
+shift 3
+
+REPS=1
+RESULTS=""
+while [ $# -gt 0 ]; do
+  case "$1" in
+    --reps)    REPS="${2:?--reps needs a number}"; shift 2 ;;
+    --results) RESULTS="${2:?--results needs a path}"; shift 2 ;;
+    *) echo "unknown option: $1" >&2; exit 2 ;;
+  esac
+done
+
+EVAL_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+SCENARIO_FILE="$EVAL_ROOT/scenarios/${SCENARIO%-nograph}.md"
+[ -f "$SCENARIO_FILE" ] || { echo "unknown scenario: $SCENARIO" >&2; exit 2; }
+mkdir -p "$WORKDIR"
+RESULTS="${RESULTS:-$WORKDIR/results.tsv}"
+
+TASK="$(awk '/^## Task/{found=1; next} found' "$SCENARIO_FILE")"
+
+for rep in $(seq 1 "$REPS"); do
+  SANDBOX="$WORKDIR/$SCENARIO-$MODEL-r$rep"
+  "$EVAL_ROOT/bin/sandbox" "$SCENARIO" "$SANDBOX" >/dev/null
+
+  # cwd = sandbox: the SessionStart hook resolves the project root from here.
+  (cd "$SANDBOX" && claude -p \
+      --model "$MODEL" \
+      --output-format stream-json --verbose \
+      --dangerously-skip-permissions \
+      "$TASK" > transcript.jsonl) || echo "claude exited non-zero for $SANDBOX" >&2
+
+  "$EVAL_ROOT/bin/check" "$SCENARIO" "$SANDBOX" --tsv "$MODEL" | tee -a "$RESULTS"
+done

plugins/os-adr/eval-b/bin/sandbox

@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+# Create a fresh, git-initialized sandbox copy of the Eval B fixture.
+# Usage: sandbox <W1|W2|W3|R1|R2|R3|R4|R4-nograph> <dest-dir>
+#
+# Only R4 keeps graphify-out/ (the graph-layer scenario). R4-nograph is the
+# degradation variant: same scenario, graph deliberately absent.
+set -euo pipefail
+
+SCENARIO="${1:?usage: sandbox <scenario> <dest-dir>}"
+DEST="${2:?usage: sandbox <scenario> <dest-dir>}"
+EVAL_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+FIXTURE="$EVAL_ROOT/fixture/project"
+
+case "$SCENARIO" in
+  W1|W2|W3|R1|R2|R3|R4-nograph) GRAPH=no ;;
+  R4)                           GRAPH=yes ;;
+  *) echo "unknown scenario: $SCENARIO" >&2; exit 2 ;;
+esac
+
+if [ "$GRAPH" = yes ] && [ ! -f "$FIXTURE/graphify-out/graph.json" ]; then
+  echo "fixture graph missing — run eval-b/bin/build-fixture-graph first" >&2
+  exit 2
+fi
+
+if [ -e "$DEST" ]; then echo "refusing to overwrite existing $DEST" >&2; exit 2; fi
+mkdir -p "$DEST"
+cp -r "$FIXTURE/." "$DEST/"
+[ "$GRAPH" = no ] && rm -rf "$DEST/graphify-out"
+rm -rf "$DEST/.os-adr"
+
+git -C "$DEST" init -q
+git -C "$DEST" add -A
+git -C "$DEST" -c user.email=eval@local -c user.name=eval commit -qm "fixture baseline"
+echo "$DEST"

plugins/os-adr/eval-b/bin/self-test

@@ -0,0 +1,158 @@
+#!/usr/bin/env bash
+# Self-test the Eval B harness in both directions, model-free:
+#
+#   1. For each scenario, fabricate a "perfect" transcript/sandbox state
+#      (an agent that did the right thing) -> bin/check must PASS.
+#   2. For each scenario, fabricate an "untouched" run (agent never consulted
+#      the ADR system) -> bin/check must FAIL, axis (a) FAIL specifically.
+#   3. Sandbox isolation: the canonical fixture must be byte-identical after
+#      all sandbox operations.
+#
+# The W-scenario LLM judge is stubbed via ADR_EVAL_B_JUDGE_CMD so this never
+# spends model tokens. It never touches the held-out scenario Task blocks.
+#
+# Usage: self-test [workdir]   (default: a fresh mktemp dir)
+set -euo pipefail
+
+EVAL_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+PLUGIN_ROOT="$(cd "$EVAL_ROOT/.." && pwd)"
+WORKDIR="${1:-$(mktemp -d /tmp/adr-eval-b-selftest.XXXXXX)}"
+mkdir -p "$WORKDIR"
+FAILED=0
+
+fixture_digest() {
+  (cd "$EVAL_ROOT/fixture/project" && find . -type f ! -path "./graphify-out/*" -print0 \
+    | sort -z | xargs -0 md5sum | md5sum)
+}
+DIGEST_BEFORE="$(fixture_digest)"
+
+# --- transcript fabrication -------------------------------------------------
+
+hook_event() {
+  printf '%s\n' '{"type":"system","subtype":"hook_response","hook_name":"SessionStart:startup","output":"[os-adr] This project records architecture decisions in docs/adr/"}'
+}
+
+tool_use_event() { # $1 = tool name, $2 = input JSON
+  printf '{"type":"assistant","message":{"role":"assistant","content":[{"type":"tool_use","name":"%s","input":%s}]}}\n' "$1" "$2"
+}
+
+result_event() { # $1 = final text (single line, no quotes needed beyond basic)
+  python3 -c 'import json,sys; print(json.dumps({"type":"result","subtype":"success","result":sys.argv[1]}))' "$1"
+}
+
+perfect_transcript() { # $1 = sandbox, $2 = adr-touching tool input, $3 = final text
+  { hook_event
+    tool_use_event "Bash" "$2"
+    result_event "$3"
+  } > "$1/transcript.jsonl"
+}
+
+untouched_transcript() { # $1 = sandbox
+  { hook_event
+    tool_use_event "Read" '{"file_path":"lib/relay/delivery.rb"}'
+    result_event "Done. I made the change you asked for."
+  } > "$1/transcript.jsonl"
+}
+
+# --- expectations -----------------------------------------------------------
+
+expect() { # $1 = PASS|FAIL, $2 = label, then the check command
+  local want="$1" label="$2"; shift 2
+  local got=PASS
+  "$@" >/dev/null 2>&1 || got=FAIL
+  if [ "$got" = "$want" ]; then
+    echo "ok   $label ($want)"
+  else
+    echo "FAIL $label: wanted $want, got $got"
+    "$@" || true
+    FAILED=1
+  fi
+}
+
+FIND_INPUT='{"command":"ruby '"$PLUGIN_ROOT"'/bin/adr-find --root . --paths lib/relay/delivery.rb"}'
+
+# --- 1. perfect runs must PASS ----------------------------------------------
+
+declare -A FINAL_TEXT=(
+  [R1]="Stopped before implementing: ADR 0002 requires all outbound HTTP to go through the shared HttpClient, so a direct Net::HTTP call would conflict with that decision. I added ping via the wrapper instead."
+  [R2]="Flat 2-second retries would violate ADR 0003 (exponential backoff, capped attempts). The older retry decision 0001 is superseded by 0003, so it does not apply. I did not change the backoff."
+  [R3]="Per ADR 0005, recorded event times are UTC ISO-8601 strings, so the first-seen value is written with Time.now.utc.iso8601 — not local server time."
+  [R4]="A retry-until-success loop in reports.rb would conflict with ADR 0003, which governs delivery retries one hop away: re-sends must follow the exponential backoff schedule. I routed redelivery through the backoff instead."
+)
+
+for sc in R1 R2 R3 R4; do
+  sb="$WORKDIR/perfect-$sc"
+  "$EVAL_ROOT/bin/sandbox" "$sc" "$sb" >/dev/null
+  perfect_transcript "$sb" "$FIND_INPUT" "${FINAL_TEXT[$sc]}"
+  expect PASS "perfect $sc" "$EVAL_ROOT/bin/check" "$sc" "$sb"
+done
+
+declare -A W_TITLE=(
+  [W1]="Use SQLite for delivery history persistence"
+  [W2]="Raise exceptions instead of returning Result objects"
+  [W3]="Allow concurrent delivery dispatch, dropping strict ordering"
+)
+
+for sc in W1 W2 W3; do
+  sb="$WORKDIR/perfect-$sc"
+  "$EVAL_ROOT/bin/sandbox" "$sc" "$sb" >/dev/null
+  printf '{"title": "%s", "context": "c", "decision": "d", "consequences": "q", "alternatives": "a"}' \
+    "${W_TITLE[$sc]}" | ruby "$PLUGIN_ROOT/bin/adr-new" --root "$sb" >/dev/null
+  perfect_transcript "$sb" '{"command":"ruby '"$PLUGIN_ROOT"'/bin/adr-new --root ."}' \
+    "Recorded the decision as a new ADR before implementing."
+  expect PASS "perfect $sc (mechanical new-ADR path)" "$EVAL_ROOT/bin/check" "$sc" "$sb"
+done
+
+# W judge fallback path: no new ADR file, final message proposes recording.
+sb="$WORKDIR/perfect-W1-judge"
+"$EVAL_ROOT/bin/sandbox" W1 "$sb" >/dev/null
+perfect_transcript "$sb" "$FIND_INPUT" \
+  "I implemented SQLite-backed history. This engine choice is hard to reverse — want me to record it as an ADR?"
+expect PASS "perfect W1 (judge fallback, stub YES)" \
+  env ADR_EVAL_B_JUDGE_CMD="printf YES" "$EVAL_ROOT/bin/check" W1 "$sb"
+
+# --- 2. untouched runs must FAIL (axis a) -----------------------------------
+
+for sc in W1 W2 W3 R1 R2 R3 R4 R4-nograph; do
+  sb="$WORKDIR/untouched-$sc"
+  "$EVAL_ROOT/bin/sandbox" "$sc" "$sb" >/dev/null
+  untouched_transcript "$sb"
+  expect FAIL "untouched $sc" \
+    env ADR_EVAL_B_JUDGE_CMD="printf NO" "$EVAL_ROOT/bin/check" "$sc" "$sb"
+  row="$(env ADR_EVAL_B_JUDGE_CMD="printf NO" "$EVAL_ROOT/bin/check" "$sc" "$sb" --tsv selftest || true)"
+  case "$row" in
+    *$'\t'A:FAIL$'\t'*) echo "ok   untouched $sc axis-a FAIL confirmed" ;;
+    *) echo "FAIL untouched $sc: axis-a not FAIL in TSV: $row"; FAILED=1 ;;
+  esac
+done
+
+# R2 distractor trap: cites superseded 0001 as if live -> axis (b) FAIL.
+sb="$WORKDIR/distractor-R2"
+"$EVAL_ROOT/bin/sandbox" R2 "$sb" >/dev/null
+perfect_transcript "$sb" "$FIND_INPUT" \
+  "ADR 0001 says deliveries retry three times, so I set the retries to match that decision."
+expect FAIL "R2 citing superseded distractor 0001" "$EVAL_ROOT/bin/check" R2 "$sb"
+
+# Missing hook context invalidates the run.
+sb="$WORKDIR/nohook-R1"
+"$EVAL_ROOT/bin/sandbox" R1 "$sb" >/dev/null
+{ tool_use_event "Bash" "$FIND_INPUT"; result_event "${FINAL_TEXT[R1]}"; } > "$sb/transcript.jsonl"
+expect FAIL "R1 without SessionStart hook context (invalid run)" "$EVAL_ROOT/bin/check" R1 "$sb"
+
+# --- 3. fixture isolation ---------------------------------------------------
+
+DIGEST_AFTER="$(fixture_digest)"
+if [ "$DIGEST_BEFORE" = "$DIGEST_AFTER" ]; then
+  echo "ok   canonical fixture untouched by all sandbox operations"
+else
+  echo "FAIL canonical fixture was modified"
+  FAILED=1
+fi
+
+echo
+if [ "$FAILED" = 0 ]; then
+  echo "self-test PASS (workdir: $WORKDIR)"
+else
+  echo "self-test FAIL (workdir kept for inspection: $WORKDIR)"
+fi
+exit "$FAILED"

plugins/os-adr/eval-b/fixture/project/.gitignore

@@ -0,0 +1,4 @@
+graphify-out/
+.os-adr/
+deadletter.jsonl
+relay.log

plugins/os-adr/eval-b/fixture/project/.graphifyignore

@@ -0,0 +1,7 @@
+# Keep the project graph AST-only (code files); markdown/docs would route
+# through the Ollama doc pass and make graph rebuilds slow and non-reproducible.
+docs/
+*.md
+config/
+graphify-out/
+.os-adr/

plugins/os-adr/eval-b/fixture/project/README.md

@@ -0,0 +1,12 @@
+# relay
+
+A small CLI that delivers webhook payloads to configured endpoints and
+reports on delivery outcomes.
+
+- `bin/relay send <url> <json-payload>` — queue and deliver one payload
+- `bin/relay report` — summarize failed deliveries
+
+Configuration lives in `config/relay.yml`. Failed deliveries that exhaust
+their retries are appended to the dead-letter file for manual replay.
+
+Architecture decisions are recorded in `docs/adr/`.

plugins/os-adr/eval-b/fixture/project/bin/relay

@@ -0,0 +1,31 @@
+#!/usr/bin/env ruby
+# relay — deliver webhook payloads and report on outcomes.
+#
+#   relay send <url> <json-payload>
+#   relay report
+
+require_relative "../lib/relay"
+
+config = Relay::Config.load
+abort(config.error) if config.err?
+config = config.value
+
+log = Relay::Log.new(config.log_path)
+
+case ARGV[0]
+when "send"
+  url, raw = ARGV[1], ARGV[2]
+  abort("usage: relay send <url> <json-payload>") unless url && raw
+  delivery = Relay::Delivery.new(log: log, dead_letter_path: config.dead_letter_path)
+  queue = Relay::Queue.new(delivery: delivery)
+  queue.enqueue(url, JSON.parse(raw))
+  result = queue.drain
+  abort(result.error) if result.err?
+when "report"
+  reports = Relay::Reports.new(dead_letter_path: config.dead_letter_path)
+  result = reports.render
+  abort(result.error) if result.err?
+  puts result.value
+else
+  abort("usage: relay <send|report>")
+end

plugins/os-adr/eval-b/fixture/project/config/relay.yml

@@ -0,0 +1,5 @@
+endpoints:
+  - https://hooks.example.test/orders
+  - https://hooks.example.test/invoices
+dead_letter_path: deadletter.jsonl
+log_path: relay.log

plugins/os-adr/eval-b/fixture/project/docs/adr/0001-retry-failed-deliveries-a-fixed-three-times.md

@@ -0,0 +1,27 @@
+---
+id: "0001"
+date: 2026-02-10
+status: Superseded
+supersedes:
+superseded-by: "0003"
+affected-paths: [lib/relay/delivery.rb]
+affected-components: [delivery, retries]
+---
+
+# 0001 — Retry failed deliveries a fixed three times
+
+## Context
+
+Early receivers dropped webhooks intermittently; a delivery that fails once usually succeeds on a quick follow-up. We needed some retry behavior before launch and had no data on receiver load patterns yet.
+
+## Decision
+
+Failed deliveries are retried exactly three times, one second apart, then reported as failed. The retry count and spacing are constants in lib/relay/delivery.rb.
+
+## Consequences
+
+Simple and predictable; transient blips are absorbed. Receivers that are down for more than a few seconds still lose the delivery, and rapid-fire retries add load to a receiver that is already struggling.
+
+## Alternatives rejected
+
+No retries rejected: transient network errors were the dominant failure mode in testing. Unbounded retries rejected: a permanently dead endpoint would wedge the queue.

plugins/os-adr/eval-b/fixture/project/docs/adr/0002-route-all-outbound-http-through-the-shared-httpclient.md

@@ -0,0 +1,27 @@
+---
+id: "0002"
+date: 2026-02-18
+status: Accepted
+supersedes:
+superseded-by:
+affected-paths: [lib/relay/http_client.rb, lib/relay/delivery.rb]
+affected-components: [http, delivery]
+---
+
+# 0002 — Route all outbound HTTP through the shared HttpClient
+
+## Context
+
+Two call sites had grown their own Net::HTTP usage with inconsistent timeouts and no shared User-Agent, and a TLS verification bug had to be fixed in both places separately.
+
+## Decision
+
+All outbound HTTP goes through Relay::HttpClient (lib/relay/http_client.rb). No other file constructs Net::HTTP objects directly — timeouts, TLS policy, headers, and instrumentation live in exactly one place.
+
+## Consequences
+
+One place to change HTTP policy; call sites cannot drift. New delivery types must accept the small indirection of going through the wrapper even for trivial requests.
+
+## Alternatives rejected
+
+Per-call-site Net::HTTP rejected: policy drift was the concrete bug that motivated this. Pulling in a full HTTP gem (faraday) rejected: stdlib is sufficient at this scale and keeps the tool dependency-free.

plugins/os-adr/eval-b/fixture/project/docs/adr/0003-retry-failed-deliveries-with-exponential-backoff-capped-at-five-attempts.md

@@ -0,0 +1,27 @@
+---
+id: "0003"
+date: 2026-03-05
+status: Accepted
+supersedes: "0001"
+superseded-by:
+affected-paths: [lib/relay/delivery.rb]
+affected-components: [delivery, retries]
+---
+
+# 0003 — Retry failed deliveries with exponential backoff, capped at five attempts
+
+## Context
+
+Fixed one-second retries hammered receivers that were already struggling: several endpoints throttle by IP, and back-to-back retries during an outage got our sender IP temporarily banned. The fixed-three-times policy from the earlier decision made outages worse, not better.
+
+## Decision
+
+Failed deliveries back off exponentially — 1s, 2s, 4s, 8s, 16s — and give up after five attempts. Retries are never issued back-to-back or in a tight loop; anything that re-sends a failed delivery must go through this backoff schedule.
+
+## Consequences
+
+Struggling receivers get breathing room and IP throttling is no longer triggered. A delivery can now take up to ~31 seconds to fail over all five attempts, so callers must not assume prompt failure.
+
+## Alternatives rejected
+
+Keeping fixed-interval retries rejected: it caused the IP bans. Immediate retry-until-success rejected outright: it is precisely the pattern receivers throttle. Jittered backoff deferred: single-sender volume does not need it yet.

plugins/os-adr/eval-b/fixture/project/docs/adr/0004-cap-outbound-http-timeouts-at-five-seconds.md

@@ -0,0 +1,27 @@
+---
+id: "0004"
+date: 2026-03-12
+status: Accepted
+supersedes:
+superseded-by:
+affected-paths: [lib/relay/http_client.rb]
+affected-components: [http]
+---
+
+# 0004 — Cap outbound HTTP timeouts at five seconds
+
+## Context
+
+A single hung receiver held a delivery open for two minutes with the stdlib default timeouts, blocking everything queued behind it.
+
+## Decision
+
+HttpClient sets open_timeout and read_timeout to five seconds. A receiver that cannot accept a small JSON POST within five seconds is treated as failed and handled by the retry policy.
+
+## Consequences
+
+A hung receiver costs at most ten seconds per attempt. Genuinely slow-but-healthy receivers will see more retries; none have surfaced in practice.
+
+## Alternatives rejected
+
+Per-endpoint configurable timeouts rejected: no current need, and it invites unbounded values creeping back in. Thirty-second timeouts rejected: still long enough to stall the queue noticeably.

plugins/os-adr/eval-b/fixture/project/docs/adr/0005-record-event-times-as-utc-iso-8601-strings.md

@@ -0,0 +1,27 @@
+---
+id: "0005"
+date: 2026-04-02
+status: Accepted
+supersedes:
+superseded-by:
+affected-paths: [lib/relay/log.rb]
+affected-components: [logging]
+---
+
+# 0005 — Record event times as UTC ISO-8601 strings
+
+## Context
+
+Early log lines used local server time with no zone marker. Correlating our log against receivers' logs during an incident required guessing the offset, and a DST change produced apparently out-of-order events.
+
+## Decision
+
+Every recorded event time is a UTC ISO-8601 string (e.g. 2026-04-02T09:15:00Z), produced via Time.now.utc.iso8601. Local time and bare epoch integers are not written anywhere.
+
+## Consequences
+
+Log lines correlate directly with receiver logs and sort lexicographically. Anyone reading logs on a non-UTC machine does the mental offset themselves.
+
+## Alternatives rejected
+
+Epoch seconds rejected: compact but unreadable during incidents. Local time with zone suffix rejected: DST still reorders events within a day.

plugins/os-adr/eval-b/fixture/project/docs/adr/0006-dead-letter-deliveries-that-exhaust-retries-to-a-jsonl-file.md

@@ -0,0 +1,27 @@
+---
+id: "0006"
+date: 2026-04-20
+status: Accepted
+supersedes:
+superseded-by:
+affected-paths: [lib/relay/delivery.rb]
+affected-components: [delivery, retries]
+---
+
+# 0006 — Dead-letter deliveries that exhaust retries to a JSONL file
+
+## Context
+
+A delivery that exhausts its retry attempts was previously just a log line; operators had no way to replay it after the receiver recovered.
+
+## Decision
+
+When a delivery fails its final attempt, Delivery appends the URL, payload, and last error as one JSON line to the dead-letter file (deadletter.jsonl). Replay is a manual operator action, not automatic.
+
+## Consequences
+
+No delivery is silently lost; the report command can enumerate failures. The file grows without rotation, acceptable at current volume.
+
+## Alternatives rejected
+
+Automatic background replay rejected: a recovering receiver would immediately be hit with the full backlog. A database table rejected: the tool has no database and one failure file is greppable.

plugins/os-adr/eval-b/fixture/project/docs/adr/README.md

@@ -0,0 +1,16 @@
+<!-- Generated by os-adr. Do not hand-edit the table: it is regenerated in full on every ADR write. -->
+
+# Architecture Decision Records
+
+One file per decision, `NNNN-kebab-title.md`, created via `/os-adr:create`.
+
+<!-- adr-index:begin -->
+| ID | Title | Status | Date |
+| --- | --- | --- | --- |
+| 0001 | [Retry failed deliveries a fixed three times](0001-retry-failed-deliveries-a-fixed-three-times.md) | Superseded | 2026-02-10 |
+| 0002 | [Route all outbound HTTP through the shared HttpClient](0002-route-all-outbound-http-through-the-shared-httpclient.md) | Accepted | 2026-02-18 |
+| 0003 | [Retry failed deliveries with exponential backoff, capped at five attempts](0003-retry-failed-deliveries-with-exponential-backoff-capped-at-five-attempts.md) | Accepted | 2026-03-05 |
+| 0004 | [Cap outbound HTTP timeouts at five seconds](0004-cap-outbound-http-timeouts-at-five-seconds.md) | Accepted | 2026-03-12 |
+| 0005 | [Record event times as UTC ISO-8601 strings](0005-record-event-times-as-utc-iso-8601-strings.md) | Accepted | 2026-04-02 |
+| 0006 | [Dead-letter deliveries that exhaust retries to a JSONL file](0006-dead-letter-deliveries-that-exhaust-retries-to-a-jsonl-file.md) | Accepted | 2026-04-20 |
+<!-- adr-index:end -->

plugins/os-adr/eval-b/fixture/project/lib/relay.rb

@@ -0,0 +1,11 @@
+require_relative "relay/result"
+require_relative "relay/config"
+require_relative "relay/log"
+require_relative "relay/http_client"
+require_relative "relay/delivery"
+require_relative "relay/queue"
+require_relative "relay/formatter"
+require_relative "relay/reports"
+
+module Relay
+end

plugins/os-adr/eval-b/fixture/project/lib/relay/config.rb

@@ -0,0 +1,29 @@
+require "yaml"
+require_relative "result"
+
+module Relay
+  # Loads config/relay.yml. Returns Result, never raises for a missing or
+  # malformed file.
+  class Config
+    DEFAULT_PATH = File.expand_path("../../config/relay.yml", __dir__)
+
+    def self.load(path = DEFAULT_PATH)
+      return Result.err("config not found: #{path}") unless File.exist?(path)
+
+      data = YAML.safe_load(File.read(path))
+      return Result.err("config is not a mapping") unless data.is_a?(Hash)
+
+      Result.ok(new(data))
+    rescue Psych::SyntaxError => e
+      Result.err("config parse error: #{e.message}")
+    end
+
+    def initialize(data)
+      @data = data
+    end
+
+    def endpoints = @data.fetch("endpoints", [])
+    def dead_letter_path = @data.fetch("dead_letter_path", "deadletter.jsonl")
+    def log_path = @data.fetch("log_path", "relay.log")
+  end
+end

plugins/os-adr/eval-b/fixture/project/lib/relay/delivery.rb

@@ -0,0 +1,53 @@
+require "json"
+require_relative "result"
+require_relative "http_client"
+require_relative "log"
+
+module Relay
+  # Delivers one webhook payload to one endpoint, with retries.
+  #
+  # Failed attempts back off exponentially (1s, 2s, 4s, 8s, 16s) and give up
+  # after MAX_ATTEMPTS; receivers throttle by IP, so attempts are never
+  # retried back-to-back. A delivery that exhausts its attempts is appended
+  # to the dead-letter file for manual replay.
+  class Delivery
+    MAX_ATTEMPTS = 5
+    BASE_BACKOFF = 1 # seconds; doubles each attempt
+
+    def self.attempt_budget = MAX_ATTEMPTS
+
+    def initialize(client: HttpClient.new, log:, dead_letter_path:)
+      @client = client
+      @log = log
+      @dead_letter_path = dead_letter_path
+    end
+
+    def deliver(url, payload)
+      body = JSON.generate(payload)
+      last_error = nil
+
+      MAX_ATTEMPTS.times do |attempt|
+        sleep(BASE_BACKOFF * (2**(attempt - 1))) if attempt.positive?
+        @log.record("attempt", "#{url} ##{attempt + 1}")
+        result = @client.post(url, body)
+        if result.ok?
+          @log.record("delivered", url)
+          return Result.ok(result.value)
+        end
+        last_error = result.error
+        @log.record("failed", "#{url}: #{last_error}")
+      end
+
+      dead_letter(url, payload, last_error)
+      Result.err("gave up after #{MAX_ATTEMPTS} attempts: #{last_error}")
+    end
+
+    private
+
+    def dead_letter(url, payload, error)
+      entry = { "url" => url, "payload" => payload, "error" => error }
+      File.open(@dead_letter_path, "a") { |f| f.puts(JSON.generate(entry)) }
+      @log.record("dead-lettered", url)
+    end
+  end
+end

plugins/os-adr/eval-b/fixture/project/lib/relay/formatter.rb

@@ -0,0 +1,27 @@
+require_relative "result"
+
+module Relay
+  # Renders report rows for terminal output.
+  class Formatter
+    HEADERS = %w[url attempts outcome].freeze
+
+    def render(rows)
+      return Result.err("no rows") if rows.empty?
+
+      widths = column_widths(rows)
+      lines = [format_row(HEADERS, widths)]
+      lines += rows.map { |row| format_row(row, widths) }
+      Result.ok(lines.join("\n"))
+    end
+
+    private
+
+    def column_widths(rows)
+      ([HEADERS] + rows).transpose.map { |col| col.map { |c| c.to_s.length }.max }
+    end
+
+    def format_row(row, widths)
+      row.zip(widths).map { |cell, w| cell.to_s.ljust(w) }.join("  ")
+    end
+  end
+end

plugins/os-adr/eval-b/fixture/project/lib/relay/http_client.rb

@@ -0,0 +1,38 @@
+require "net/http"
+require "uri"
+require_relative "result"
+
+module Relay
+  # The single place outbound HTTP happens. Timeouts, TLS policy, and
+  # instrumentation live here so call sites cannot drift.
+  class HttpClient
+    OPEN_TIMEOUT = 5
+    READ_TIMEOUT = 5
+
+    def post(url, body, headers = {})
+      uri = URI.parse(url)
+      http = Net::HTTP.new(uri.host, uri.port)
+      http.use_ssl = uri.scheme == "https"
+      http.open_timeout = OPEN_TIMEOUT
+      http.read_timeout = READ_TIMEOUT
+
+      request = Net::HTTP::Post.new(uri.request_uri, default_headers.merge(headers))
+      request.body = body
+      response = http.request(request)
+
+      if response.is_a?(Net::HTTPSuccess)
+        Result.ok(response)
+      else
+        Result.err("HTTP #{response.code} from #{uri.host}")
+      end
+    rescue SystemCallError, Net::OpenTimeout, Net::ReadTimeout => e
+      Result.err("#{e.class}: #{e.message}")
+    end
+
+    private
+
+    def default_headers
+      { "Content-Type" => "application/json", "User-Agent" => "relay/1.0" }
+    end
+  end
+end

plugins/os-adr/eval-b/fixture/project/lib/relay/log.rb

@@ -0,0 +1,21 @@
+require "time"
+require_relative "result"
+
+module Relay
+  # Append-only event log. Every recorded event carries the moment it
+  # happened as a UTC ISO-8601 string.
+  class Log
+    def initialize(path)
+      @path = path
+    end
+
+    def record(event, detail = nil)
+      stamp = Time.now.utc.iso8601
+      line = [stamp, event, detail].compact.join("\t")
+      File.open(@path, "a") { |f| f.puts(line) }
+      Result.ok(line)
+    rescue SystemCallError => e
+      Result.err("log write failed: #{e.message}")
+    end
+  end
+end

plugins/os-adr/eval-b/fixture/project/lib/relay/queue.rb

@@ -0,0 +1,31 @@
+require_relative "result"
+require_relative "delivery"
+
+module Relay
+  # Dispatches queued deliveries strictly in order, one at a time.
+  #
+  # This is deliberate, not an oversight: receivers rely on seeing events in
+  # the order they occurred (e.g. "created" before "updated"), and a single
+  # in-process worker is the simplest way to guarantee that. Interleaving
+  # deliveries would silently break receivers' ordering assumptions.
+  class Queue
+    def initialize(delivery:)
+      @delivery = delivery
+      @pending = []
+    end
+
+    def enqueue(url, payload)
+      @pending << [url, payload]
+      Result.ok(@pending.size)
+    end
+
+    def drain
+      results = []
+      until @pending.empty?
+        url, payload = @pending.shift
+        results << @delivery.deliver(url, payload)
+      end
+      Result.ok(results)
+    end
+  end
+end

plugins/os-adr/eval-b/fixture/project/lib/relay/reports.rb

@@ -0,0 +1,34 @@
+require "json"
+require_relative "result"
+require_relative "delivery"
+require_relative "formatter"
+
+module Relay
+  # Summarizes delivery outcomes for the `relay report` command, including
+  # entries the Delivery dead-letter path wrote out.
+  class Reports
+    def initialize(dead_letter_path:, formatter: Formatter.new)
+      @dead_letter_path = dead_letter_path
+      @formatter = formatter
+    end
+
+    def failed_deliveries
+      return Result.ok([]) unless File.exist?(@dead_letter_path)
+
+      entries = File.readlines(@dead_letter_path).map { |line| JSON.parse(line) }
+      Result.ok(entries)
+    rescue JSON::ParserError => e
+      Result.err("dead-letter file corrupt: #{e.message}")
+    end
+
+    def render
+      failures = failed_deliveries
+      return failures if failures.err?
+
+      rows = failures.value.map do |entry|
+        [entry["url"], Delivery.attempt_budget, "dead-lettered"]
+      end
+      rows.empty? ? Result.ok("no failed deliveries") : @formatter.render(rows)
+    end
+  end
+end

plugins/os-adr/eval-b/fixture/project/lib/relay/result.rb

@@ -0,0 +1,12 @@
+module Relay
+  # Uniform error-handling shape used across the whole codebase: every
+  # fallible operation returns a Result; exceptions are reserved for
+  # programmer errors, never expected failures.
+  Result = Struct.new(:ok, :value, :error) do
+    def self.ok(value = nil) = new(true, value, nil)
+    def self.err(error) = new(false, nil, error)
+
+    def ok? = ok
+    def err? = !ok
+  end
+end

plugins/os-adr/eval-b/judge-rubric.md

@@ -0,0 +1,27 @@
+# W-scenario axis (b) judge rubric — FROZEN
+
+This rubric is part of the checker. Like the fixtures and scenario prompts, it is frozen
+during any `autoresearch` loop; if it turns out to be wrong, stop the loop and fix it as a
+separate, human-reviewed change.
+
+The judge reads ONLY the model-under-test's final message (never the full transcript) and
+answers one yes/no question.
+
+---
+
+You are grading one final assistant message from a coding-agent session. The agent was given
+an implementation task in a project that records architecture decisions as ADRs.
+
+Question: in this final message, did the agent explicitly propose recording the decision it
+made (or was asked to make) as an architecture decision record — for example by offering to
+create an ADR, naming an ADR-creation step or command it could run, or asking the user whether
+the decision should be recorded?
+
+Answer YES only if the message explicitly proposes or asks about recording THIS decision as an
+ADR / decision record. Answer NO if the agent implemented silently, only described code
+changes, or mentioned documentation generically (README updates, code comments) without
+proposing a decision record.
+
+Reply with exactly one word: YES or NO.
+
+Final assistant message follows:

plugins/os-adr/eval-b/scenarios/R1.md

@@ -0,0 +1,21 @@
+# R1 — retrieval: direct conflict with an Accepted ADR
+
+> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
+> First real execution is the harness self-test (see eval-b/README.md).
+
+- **Fixture:** `project` (no `graphify-out/`)
+- **Shape:** `06-eval-scenarios.md` R1 — the requested implementation (direct `Net::HTTP` in
+  `lib/relay/delivery.rb`) violates Accepted ADR **0002** (all outbound HTTP through the shared
+  HttpClient), whose `affected-paths` cover the file being edited.
+- **Correct ADR:** `0002`
+- **Checker, axis (a):** any tool_use touching the ADR system.
+- **Checker, axis (b):** assistant text cites `0002` and flags the conflict
+  (/conflict|violat|contradic|goes against|instead|decision|record/i); implementing the direct
+  call without surfacing 0002 is a FAIL.
+
+## Task (give verbatim to the model under test)
+
+Add a lightweight `ping` method to `lib/relay/delivery.rb` that POSTs a tiny heartbeat body
+(`{"ping": true}`) to a given URL. It's fire-and-forget — no retries, no dead-lettering — so
+keep it cheap: just call `Net::HTTP` directly inside the method rather than pulling in any of
+the heavier machinery.

plugins/os-adr/eval-b/scenarios/R2.md

@@ -0,0 +1,20 @@
+# R2 — retrieval: distractor discrimination
+
+> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
+> First real execution is the harness self-test (see eval-b/README.md).
+
+- **Fixture:** `project` (no `graphify-out/`)
+- **Shape:** `06-eval-scenarios.md` R2 — same conflict shape as R1, but `docs/adr/` seeds
+  plausible near-misses: **0001** (Superseded retry policy — same topic, dead), **0004**
+  (same http family, different decision), **0006** (same retry family, different decision).
+  The flat-2s-retry request conflicts with Accepted ADR **0003** (exponential backoff, cap 5).
+- **Correct ADR:** `0003` · **Distractors:** `0001` (Superseded), `0004`, `0006`
+- **Checker, axis (a):** any tool_use touching the ADR system.
+- **Checker, axis (b):** assistant text cites `0003` and flags the conflict; citing `0001` as
+  governing without noting it is superseded is a FAIL.
+
+## Task (give verbatim to the model under test)
+
+Customers complain that failed webhook deliveries take too long to come back. Make retries
+snappier: change `lib/relay/delivery.rb` so a failed delivery just retries every 2 seconds
+flat, instead of waiting longer and longer between attempts.

plugins/os-adr/eval-b/scenarios/R3.md

@@ -0,0 +1,21 @@
+# R3 — retrieval: mid-task "how should we…" question, different vocabulary
+
+> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
+> First real execution is the harness self-test (see eval-b/README.md).
+
+- **Fixture:** `project` (no `graphify-out/`)
+- **Shape:** `06-eval-scenarios.md` R3 — the embedded question ("what format should the time
+  value use") is answered by Accepted ADR **0005** (UTC ISO-8601), phrased in different
+  vocabulary ("write down the moment each attempt happened" vs "record event times").
+- **Correct ADR:** `0005`
+- **Checker, axis (a):** any tool_use touching the ADR system (mid-task query surface).
+- **Checker, axis (b):** assistant text answers with UTC **and** ISO-8601 (both
+  /UTC/i and /ISO.?8601/i must match); local-time or epoch answers FAIL.
+
+## Task (give verbatim to the model under test)
+
+I want `bin/relay report` to show a "first seen" column for each failed delivery. Add the
+moment a delivery first failed to the dead-letter entries in `lib/relay/delivery.rb`, and show
+it in the report. Quick question while you're in there: for that time value we write into the
+file, should it be local server time or something else, and in what format? Set it up whichever
+way is correct for this project.

plugins/os-adr/eval-b/scenarios/R4.md

@@ -0,0 +1,25 @@
+# R4 — retrieval: graph-layer reach (one hop past affected-paths)
+
+> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
+> First real execution is the harness self-test (see eval-b/README.md).
+
+- **Fixture:** `project` **with** `graphify-out/` (the only scenario that keeps it; rebuild via
+  `bin/build-fixture-graph` if missing)
+- **Shape:** `06-eval-scenarios.md` R4 — `lib/relay/reports.rb` is listed in **no** ADR's
+  `affected-paths`, but is one call-graph hop (`Reports#render` → `Delivery.attempt_budget`)
+  from `lib/relay/delivery.rb`, which Accepted ADR **0003** (exponential backoff; never
+  tight-loop re-sends) does cover. The requested retry-until-success loop conflicts with 0003.
+  Reachable only via the Graphify expansion layer: `adr-find --paths lib/relay/reports.rb`
+  returns 0003 (`matched_on: graph`) with the graph present and nothing without it.
+- **Correct ADR:** `0003`
+- **Degradation variant:** `bin/sandbox R4-nograph` strips `graphify-out/`; the same run is
+  then *expected* to FAIL axis (b) — that failure is the graph-layer degradation check.
+- **Checker, axis (a):** any tool_use touching the ADR system.
+- **Checker, axis (b):** assistant text cites `0003` and flags the conflict.
+
+## Task (give verbatim to the model under test)
+
+Add a `redeliver` command to `bin/relay`, implemented in `lib/relay/reports.rb`: for each
+failed delivery in the dead-letter file, immediately re-send it to its URL, and keep re-sending
+in a loop until it goes through. Keep all the new logic inside `lib/relay/reports.rb` — don't
+touch `lib/relay/delivery.rb`.

plugins/os-adr/eval-b/scenarios/W1.md

@@ -0,0 +1,22 @@
+# W1 — write-trigger: irreversible choice mid-task (persistence engine)
+
+> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
+> First real execution is the harness self-test (see eval-b/README.md).
+
+- **Fixture:** `project` (no `graphify-out/`)
+- **Shape:** `06-eval-scenarios.md` W1 — storage engine choice is genuinely open and
+  consequential; no existing ADR covers persistence.
+- **Checker, axis (a):** any tool_use touching the ADR system (`os-adr:*` skill, `bin/adr-*`
+  CLI, or read/glob of `docs/adr/`).
+- **Checker, axis (b):** mechanical PASS if a new `docs/adr/0007-*.md` exists whose text
+  matches the topic pattern /persist|stor(e|age)|database|sqlite|history/i; otherwise the
+  rubric-bound judge decides from the final message whether the agent proposed or asked about
+  recording the decision.
+
+## Task (give verbatim to the model under test)
+
+Delivery history currently only exists while the process is running — `bin/relay report` can
+only show dead-lettered failures, and nothing survives between runs. Add persistence for
+delivery history so a report can cover everything delivered in past runs too. Pick whatever
+storage approach you think is right for a small single-user CLI tool like this, and wire
+`bin/relay report` up to it.

plugins/os-adr/eval-b/scenarios/W2.md

@@ -0,0 +1,18 @@
+# W2 — write-trigger: cross-cutting convention change (error-handling shape)
+
+> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
+> First real execution is the harness self-test (see eval-b/README.md).
+
+- **Fixture:** `project` (no `graphify-out/`)
+- **Shape:** `06-eval-scenarios.md` W2 — the natural solution changes a convention used across
+  many files (the `Relay::Result` return shape); the convention is real in code but not ADR'd.
+- **Checker, axis (a):** any tool_use touching the ADR system.
+- **Checker, axis (b):** mechanical PASS if a new `docs/adr/0007-*.md` exists matching
+  /error|exception|result|raise|failure/i; otherwise the judge decides from the final message.
+
+## Task (give verbatim to the model under test)
+
+I keep finding the Result-struct plumbing noisy — every caller has to unwrap `ok?`/`error` by
+hand. Refactor the codebase so failures raise exceptions instead of returning Result objects,
+and update all the callers accordingly. Feel free to delete `lib/relay/result.rb` once nothing
+uses it.

plugins/os-adr/eval-b/scenarios/W3.md

@@ -0,0 +1,18 @@
+# W3 — write-trigger: contested reversal of a discoverable-in-code approach
+
+> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
+> First real execution is the harness self-test (see eval-b/README.md).
+
+- **Fixture:** `project` (no `graphify-out/`)
+- **Shape:** `06-eval-scenarios.md` W3 — the request quietly reverses the deliberate
+  single-worker in-order dispatch documented in `lib/relay/queue.rb`'s comment (not ADR'd).
+- **Checker, axis (a):** any tool_use touching the ADR system.
+- **Checker, axis (b):** mechanical PASS if a new `docs/adr/0007-*.md` exists matching
+  /concurren|thread|parallel|order|sequential|queue/i; otherwise the judge decides from the
+  final message.
+
+## Task (give verbatim to the model under test)
+
+Dispatch is slow when a lot of deliveries queue up, because they go out one at a time. Change
+`lib/relay/queue.rb` so `drain` processes deliveries concurrently with a small thread pool —
+four workers is plenty.

plugins/os-adr/eval/README.md

@@ -0,0 +1,83 @@
+# os-adr Eval A — prompted skill-execution across model tiers
+
+_Last updated: 2026-07-03 — harness built and self-tested (perfect-run PASS, untouched-sandbox
+FAIL, for all six scenarios)._
+
+Measures whether a **given model tier** (haiku, sonnet, …) can correctly execute the os-adr
+skills when explicitly invoked. This is **not** the held-out unprompted-behavior eval
+(requirements 4–5, "Eval B") — those scenarios stay deferred and must not be run informally
+(see `docs/adr-system/06-eval-scenarios.md`). Eval A prompts are not held-out; they may be
+read, discussed, and iterated against freely.
+
+**Metric:** pass rate over the scenario × model grid, scored by the deterministic checker.
+The model-sensitive surface under test is skill-*following* (reading SKILL.md and doing what
+it says: right CLI, right args, right judgment calls). Skill *dispatch*, the CLIs, hook, and
+index are deterministic and covered by the plugin's own 62 tests.
+
+## Layout
+
+| Path | What |
+| --- | --- |
+| `fixture/project/` | 4-ADR project (incl. a superseded pair) — scenarios S1–S4 |
+| `fixture/legacy-project/` | monolithic `DECISIONS.md`, no `docs/adr/` — scenarios S5–S6 |
+| `scenarios/S1..S6.md` | task prompt (verbatim block) + what the checker asserts |
+| `bin/sandbox <Sn> <dest>` | fresh git-initialized sandbox copy of the right fixture |
+| `bin/check <Sn> <sandbox> [--tsv <model>]` | deterministic scorer; exit 0/1; TSV mode for autoresearch |
+| `runner-prompt.md` | the prompt template for the model under test |
+| `bin/run-headless <Sn> <model> <workdir>` | `claude -p` fallback runner (costlier) |
+
+Scenarios: S1 create · S2 create+supersede · S3 find/conflict · S4 find/distractor ·
+S5 init · S6 migrate+fills. S3/S4 need the model's final answer saved to `<sandbox>/ANSWER.md`
+(the runner prompt instructs the model to write it; if a subagent only returns text, the
+driver saves that text to `ANSWER.md` before checking).
+
+## Running the grid in-session (preferred — cheaper than `claude -p`)
+
+A driver session (any model) runs, for each cell of {S1..S6} × {haiku, sonnet}:
+
+1. `eval/bin/sandbox S3 /tmp/adr-eval/S3-haiku-r1`
+2. Render `runner-prompt.md` (`{{SANDBOX}}`, `{{PLUGIN_ROOT}}` = the plugin dir,
+   `{{SKILL}}`, `{{TASK}}` = the scenario file's Task block) and spawn it as an **Agent tool
+   subagent with `model:` pinned to the tier under test**. Spawn independent cells in parallel.
+3. Ensure `ANSWER.md` exists (write the subagent's returned text there if it didn't).
+4. `eval/bin/check S3 /tmp/adr-eval/S3-haiku-r1 --tsv haiku >> results.tsv`
+
+12 cells ≈ 12 subagent runs per round. Pass rate = `grep -c PASS results.tsv` / total.
+
+**Fidelity caveats vs a real interactive session** (accepted for Eval A): no SessionStart
+hook context (irrelevant — these are explicit invocations), no slash-command dispatch
+(deterministic plumbing), subagent system prompt differs slightly from an interactive session.
+The headless runner exists when full fidelity matters.
+
+## Optimizing with `/autoresearch` (Classic mode)
+
+Invoke in a driver session at the cc-os root:
+
+```
+/autoresearch
+Goal: raise the os-adr skill-execution pass rate on weak models by tightening SKILL.md wording
+Scope: plugins/os-adr/skills/*/SKILL.md ONLY — never edit bin/, lib/, tests/, or anything under eval/
+Metric: pass rate over the {S1..S6} x {haiku, sonnet} grid via plugins/os-adr/eval/bin/check --tsv
+Verify: run the in-session grid per plugins/os-adr/eval/README.md, append all 12 TSV lines to the
+  round's results.tsv, report pass rate and per-cell failures
+Iterations: 5
+```
+
+Ground rules the loop must respect:
+
+- **Scope is the guard against metric-gaming**: the checker, fixtures, scenarios, and runner
+  prompt are frozen during a loop. If a scenario or checker turns out to be wrong, stop the
+  loop and fix it as a separate, human-reviewed change.
+- Keep/discard on the **haiku** column first — a wording change that helps haiku and is
+  neutral for sonnet is a keep; one that regresses sonnet is a discard.
+- Failures in `results.tsv` column 4 name the exact broken invariant (e.g. "0001 status
+  ... != Superseded") — feed that string into the next modify step.
+- SKILL.md edits must not contradict the plugin's `invariants.md` (e.g. never suggest
+  hand-editing the index). Run `ruby tests/all.rb` after each accepted edit as a regression
+  gate.
+
+## Adding a scenario
+
+Add `scenarios/S7.md` (Task block + criteria), a `S7` class in `bin/check`, the fixture route
+in `bin/sandbox`, then self-test both directions: simulate a perfect run with the CLIs
+(must PASS) and check an untouched sandbox (must FAIL).

plugins/os-adr/eval/bin/check

@@ -0,0 +1,233 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+# Deterministic per-scenario checker for the os-adr model-tier eval (Eval A).
+#
+#   check <S1..S6> <sandbox-root> [--tsv <model-label>]
+#
+# Reads the sandbox produced by bin/sandbox after a model run. Scenarios S3/S4
+# additionally require the model's final answer saved to <sandbox>/ANSWER.md.
+# Human mode prints PASS/FAIL + reasons; --tsv prints one tab-separated line
+# (scenario, model, PASS|FAIL, reasons) for autoresearch's *-results.tsv.
+# Exit 0 on PASS, 1 on FAIL, 2 on usage error.
+
+require "date"
+
+module AdrEval
+  PENDING = /_pending LLM fill/i
+
+  # Minimal frontmatter reader — deliberately independent of lib/adr so the
+  # checker cannot inherit a bug from the code it is scoring around.
+  class Frontmatter
+    def self.parse(text)
+      block = text[/\A---\n(.*?)\n---\n/m, 1] or return nil
+      fields = {}
+      block.scan(/^([\w-]+):[ \t]*(.*)$/) { |k, v| fields[k] = v.strip.gsub(/\A"|"\z/, "") }
+      new(fields)
+    end
+
+    def initialize(fields) = @fields = fields
+    def [](key) = @fields[key]
+    def list(key) = (@fields[key] || "").scan(/"([^"]*)"|([^\[\],\s]+)/).flatten.compact
+  end
+
+  class Sandbox
+    def initialize(root, eval_root)
+      @root = root
+      @eval_root = eval_root
+    end
+
+    attr_reader :root
+
+    def read(rel) = File.read(File.join(@root, rel))
+    def exist?(rel) = File.exist?(File.join(@root, rel))
+    def adr_files = Dir.glob(File.join(@root, "docs/adr/[0-9]*.md")).sort
+    def answer = exist?("ANSWER.md") ? read("ANSWER.md") : nil
+
+    def unchanged_from_fixture?(rel, fixture)
+      original = File.join(@eval_root, "fixture", fixture, rel)
+      File.exist?(original) && exist?(rel) && File.read(original) == read(rel)
+    end
+
+    def index = exist?("docs/adr/README.md") ? read("docs/adr/README.md") : ""
+  end
+
+  class Result
+    def initialize = @failures = []
+    def pass? = @failures.empty?
+    attr_reader :failures
+
+    # Returns the condition so callers can bail early on failure.
+    def expect(condition, reason)
+      @failures << reason unless condition
+      !!condition
+    end
+  end
+
+  class Scenario
+    def initialize(sandbox) = @sb = sandbox
+
+    def run
+      result = Result.new
+      check(result)
+      result
+    end
+
+    private
+
+    attr_reader :sb
+
+    # A well-formed, fully-authored ADR file at the given path.
+    def expect_complete_adr(result, path, id:)
+      unless sb.exist?(path)
+        result.expect(false, "missing #{path}")
+        return nil
+      end
+      text = sb.read(path)
+      fm = Frontmatter.parse(text)
+      return nil unless result.expect(!fm.nil?, "#{path}: no frontmatter")
+      result.expect(fm["id"] == id, "#{path}: id #{fm['id'].inspect} != #{id.inspect}")
+      result.expect(text !~ AdrEval::PENDING, "#{path}: pending-fill markers remain")
+      %w[Context Decision Consequences].each do |section|
+        body = text[/^## #{section}\n(.*?)(?=^## |\z)/m, 1].to_s.strip
+        result.expect(!body.empty?, "#{path}: empty #{section} section")
+      end
+      fm
+    end
+
+    def expect_indexed(result, id)
+      result.expect(sb.index.include?(id), "index README.md has no row for #{id}")
+    end
+
+    def expect_fixture_adrs_untouched(result, except: [])
+      %w[0001 0002 0003 0004].each do |id|
+        next if except.include?(id)
+        rel = Dir.glob(File.join(sb.root, "docs/adr/#{id}-*.md")).first
+        next result.expect(false, "fixture ADR #{id} missing") unless rel
+        rel = rel.sub("#{sb.root}/", "")
+        result.expect(sb.unchanged_from_fixture?(rel, "project"), "fixture ADR #{id} was modified")
+      end
+    end
+  end
+
+  class S1 < Scenario # create: new decision recorded via CLI
+    def check(result)
+      fm = expect_complete_adr(result, new_adr_path, id: "0005")
+      return unless fm
+      result.expect(fm["status"] == "Accepted", "0005 status #{fm['status'].inspect} != Accepted")
+      result.expect(!fm.list("affected-paths").empty?, "0005 affected-paths empty")
+      expect_indexed(result, "0005")
+      expect_fixture_adrs_untouched(result)
+    end
+
+    def new_adr_path
+      path = Dir.glob(File.join(sb.root, "docs/adr/0005-*.md")).first
+      path ? path.sub("#{sb.root}/", "") : "docs/adr/0005-*.md"
+    end
+  end
+
+  class S2 < S1 # create with supersession of 0001
+    def check(result)
+      fm = expect_complete_adr(result, new_adr_path, id: "0005")
+      return unless fm
+      result.expect(fm["supersedes"] == "0001",
+                    "0005 supersedes #{fm['supersedes'].inspect} != 0001")
+      old = Frontmatter.parse(sb.read(old_adr_rel))
+      result.expect(old["status"] == "Superseded", "0001 status #{old['status'].inspect} != Superseded")
+      result.expect(old["superseded-by"] == "0005", "0001 superseded-by #{old['superseded-by'].inspect} != 0005")
+      expect_indexed(result, "0005")
+      expect_fixture_adrs_untouched(result, except: %w[0001])
+    end
+
+    def old_adr_rel
+      Dir.glob(File.join(sb.root, "docs/adr/0001-*.md")).first.sub("#{sb.root}/", "")
+    end
+  end
+
+  class FindScenario < Scenario
+    def check(result)
+      answer = sb.answer
+      return unless result.expect(!answer.nil?, "no ANSWER.md saved in sandbox")
+      result.expect(answer.include?("adr-find"), "answer never mentions running adr-find")
+      check_answer(result, answer)
+      expect_fixture_adrs_untouched(result)
+    end
+  end
+
+  class S3 < FindScenario # find: direct path match + conflict recognition
+    def check_answer(result, answer)
+      result.expect(answer.include?("0001"), "answer does not surface ADR 0001")
+      result.expect(answer.match?(/conflict|violat|supersed|contradic|locked/i),
+                    "answer does not flag the conflict with 0001")
+    end
+  end
+
+  class S4 < FindScenario # find: distractor discrimination (0003 not superseded 0002)
+    def check_answer(result, answer)
+      result.expect(answer.include?("0003"), "answer does not surface ADR 0003")
+      if answer.include?("0002")
+        result.expect(answer.match?(/supersed/i),
+                      "answer cites superseded 0002 without noting it is superseded")
+      end
+    end
+  end
+
+  class S5 < Scenario # init on the legacy project
+    def check(result)
+      result.expect(sb.exist?("docs/adr/README.md"), "docs/adr/README.md not created")
+      result.expect(sb.index.include?("<!--"), "index missing generated markers")
+      gitignore = sb.exist?(".gitignore") ? sb.read(".gitignore") : ""
+      result.expect(gitignore.include?(".os-adr"), ".os-adr/ not gitignored")
+      result.expect(sb.unchanged_from_fixture?("DECISIONS.md", "legacy-project"),
+                    "DECISIONS.md was modified")
+    end
+  end
+
+  class S6 < Scenario # migrate the legacy DECISIONS.md, sources untouched, fills applied
+    STATUS_EXPECTATIONS = { "0001" => "Accepted", "0002" => "Accepted", "0003" => "Proposed" }.freeze
+
+    def check(result)
+      result.expect(sb.unchanged_from_fixture?("DECISIONS.md", "legacy-project"),
+                    "DECISIONS.md was modified (sources must stay byte-identical)")
+      result.expect(sb.exist?("docs/adr/migration-report.md"), "migration-report.md missing")
+      result.expect(sb.adr_files.size >= 3, "expected 3 migrated ADRs, found #{sb.adr_files.size}")
+      STATUS_EXPECTATIONS.each do |id, expected|
+        path = Dir.glob(File.join(sb.root, "docs/adr/#{id}-*.md")).first
+        next result.expect(false, "migrated ADR #{id} missing") unless path
+        fm = Frontmatter.parse(File.read(path))
+        result.expect(fm && fm["status"] == expected,
+                      "#{id} status #{fm && fm['status'].inspect} != #{expected} " \
+                      "(SETTLED→Accepted / OPEN→Proposed mapping)")
+      end
+      first = Dir.glob(File.join(sb.root, "docs/adr/0001-*.md")).first
+      if first
+        text = File.read(first)
+        %w[Consequences Alternatives].each do |section|
+          body = text[/^## #{section}.*?\n(.*?)(?=^## |\z)/m, 1].to_s
+          result.expect(body !~ AdrEval::PENDING && !body.strip.empty?,
+                        "0001 #{section} not filled from source material")
+        end
+      end
+      expect_indexed(result, "0001")
+    end
+  end
+
+  SCENARIOS = { "S1" => S1, "S2" => S2, "S3" => S3, "S4" => S4, "S5" => S5, "S6" => S6 }.freeze
+end
+
+scenario_id, sandbox_root = ARGV[0], ARGV[1]
+tsv_model = ARGV[2] == "--tsv" ? (ARGV[3] || "unknown") : nil
+klass = AdrEval::SCENARIOS[scenario_id]
+abort "usage: check <#{AdrEval::SCENARIOS.keys.join('|')}> <sandbox-root> [--tsv <model>]" if klass.nil? || sandbox_root.nil?
+abort "no such sandbox: #{sandbox_root}" unless File.directory?(sandbox_root)
+
+eval_root = File.expand_path("..", __dir__)
+result = klass.new(AdrEval::Sandbox.new(File.expand_path(sandbox_root), eval_root)).run
+
+if tsv_model
+  puts [scenario_id, tsv_model, result.pass? ? "PASS" : "FAIL", result.failures.join("; ")].join("\t")
+else
+  puts result.pass? ? "PASS #{scenario_id}" : "FAIL #{scenario_id}"
+  result.failures.each { |f| puts "  - #{f}" }
+end
+exit(result.pass? ? 0 : 1)

plugins/os-adr/eval/bin/run-headless

@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+# Headless fallback runner (burns more subscription credit than in-session subagents).
+# Usage: run-headless <S1..S6> <model> <workdir> [results.tsv]
+set -euo pipefail
+
+SCENARIO="${1:?usage: run-headless <scenario> <model> <workdir> [results.tsv]}"
+MODEL="${2:?model required (haiku|sonnet|opus)}"
+WORKDIR="${3:?workdir required}"
+RESULTS="${4:-$WORKDIR/results.tsv}"
+
+EVAL_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+PLUGIN_ROOT="$(cd "$EVAL_ROOT/.." && pwd)"
+SANDBOX="$WORKDIR/$SCENARIO-$MODEL-$(date +%s)"
+
+case "$SCENARIO" in
+  S1|S2) SKILL=create ;;
+  S3|S4) SKILL=find ;;
+  S5)    SKILL=init ;;
+  S6)    SKILL=migrate ;;
+  *) echo "unknown scenario $SCENARIO" >&2; exit 2 ;;
+esac
+
+"$EVAL_ROOT/bin/sandbox" "$SCENARIO" "$SANDBOX" >/dev/null
+TASK="$(awk '/^## Task/{found=1; next} found' "$EVAL_ROOT/scenarios/$SCENARIO.md")"
+
+PROMPT="$(awk '/^---$/{found=1; next} found' "$EVAL_ROOT/runner-prompt.md" \
+  | sed -e "s|{{SANDBOX}}|$SANDBOX|g" \
+        -e "s|{{PLUGIN_ROOT}}|$PLUGIN_ROOT|g" \
+        -e "s|{{SKILL}}|$SKILL|g")"
+PROMPT="${PROMPT//\{\{TASK\}\}/$TASK}"
+
+(cd "$SANDBOX" && claude -p --model "$MODEL" --dangerously-skip-permissions "$PROMPT")
+
+"$EVAL_ROOT/bin/check" "$SCENARIO" "$SANDBOX" --tsv "$MODEL" | tee -a "$RESULTS"

plugins/os-adr/eval/bin/sandbox

@@ -0,0 +1,22 @@
+#!/usr/bin/env bash
+# Create a fresh, git-initialized sandbox copy of the right fixture for a scenario.
+# Usage: sandbox <S1|S2|S3|S4|S5|S6> <dest-dir>
+set -euo pipefail
+
+SCENARIO="${1:?usage: sandbox <scenario> <dest-dir>}"
+DEST="${2:?usage: sandbox <scenario> <dest-dir>}"
+EVAL_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+
+case "$SCENARIO" in
+  S1|S2|S3|S4) FIXTURE="$EVAL_ROOT/fixture/project" ;;
+  S5|S6)       FIXTURE="$EVAL_ROOT/fixture/legacy-project" ;;
+  *) echo "unknown scenario: $SCENARIO" >&2; exit 2 ;;
+esac
+
+if [ -e "$DEST" ]; then echo "refusing to overwrite existing $DEST" >&2; exit 2; fi
+mkdir -p "$DEST"
+cp -r "$FIXTURE/." "$DEST/"
+git -C "$DEST" init -q
+git -C "$DEST" add -A
+git -C "$DEST" -c user.email=eval@local -c user.name=eval commit -qm "fixture baseline"
+echo "$DEST"

plugins/os-adr/eval/fixture/legacy-project/DECISIONS.md

@@ -0,0 +1,23 @@
+# Decisions Log
+
+## ADR-1 — Use Redis for the job queue
+
+**Why:** We needed a queue that survives process restarts. The in-memory array we started
+with lost jobs on every deploy.
+
+**Decision:** All background jobs go through Redis lists via the Worker class. SETTLED.
+
+We looked at beanstalkd but nobody on the team had operated it, and at Postgres-based
+queues but we did not want polling load on the main database.
+
+## ADR-2 — Ship as a single container
+
+**Why:** Deploys kept drifting between the app host and the worker host.
+
+**Decision:** App and worker ship in one container image; the entrypoint picks the role. SETTLED.
+
+## ADR-3 — Job payload schema versioning
+
+**Why:** A payload shape change stranded old jobs in the queue.
+
+**Decision:** OPEN — leaning toward an integer `v` field in every payload, not decided yet.

plugins/os-adr/eval/fixture/legacy-project/README.md

@@ -0,0 +1,3 @@
+# Legacy Fixture App
+
+Background job runner. Decision history lives in docs/decisions.md (old format).

plugins/os-adr/eval/fixture/legacy-project/src/worker.rb

@@ -0,0 +1,3 @@
+class Worker
+  def run; end
+end

plugins/os-adr/eval/fixture/project/README.md

@@ -0,0 +1,3 @@
+# Fixture App
+
+A tiny app used as an eval fixture for the os-adr plugin. Persistence in lib/store.rb, HTTP sync in lib/api.rb.

plugins/os-adr/eval/fixture/project/config/database.yml

@@ -0,0 +1,3 @@
+default:
+  adapter: sqlite3
+  database: data/app.db

plugins/os-adr/eval/fixture/project/docs/adr/0001-use-sqlite-for-local-persistence.md

@@ -0,0 +1,27 @@
+---
+id: "0001"
+date: 2026-05-01
+status: Accepted
+supersedes:
+superseded-by:
+affected-paths: [lib/store.rb, config/database.yml]
+affected-components: [store, persistence]
+---
+
+# 0001 — Use SQLite for local persistence
+
+## Context
+
+The app needs durable local storage with zero external services; data volume is small (thousands of rows) and single-user.
+
+## Decision
+
+All persistence goes through SQLite via lib/store.rb; no external database server is introduced.
+
+## Consequences
+
+Zero-dependency deploys and trivial backups; concurrent writers are not supported, which is acceptable for a single-user tool.
+
+## Alternatives rejected
+
+Postgres rejected: operational overhead unjustified at this scale. Flat JSON files rejected: no query capability and easy corruption.

plugins/os-adr/eval/fixture/project/docs/adr/0002-use-json-for-config-exports.md

@@ -0,0 +1,27 @@
+---
+id: "0002"
+date: 2026-05-10
+status: Superseded
+supersedes:
+superseded-by: "0003"
+affected-paths: [config/]
+affected-components: [config-export]
+---
+
+# 0002 — Use JSON for config exports
+
+## Context
+
+Config snapshots must be exportable for support tickets; a machine-readable format was needed.
+
+## Decision
+
+Config exports are serialized as JSON.
+
+## Consequences
+
+Easy to parse anywhere; comments impossible, which support staff dislike.
+
+## Alternatives rejected
+
+YAML rejected at the time over parser inconsistencies.

plugins/os-adr/eval/fixture/project/docs/adr/0003-use-yaml-for-config-exports.md

@@ -0,0 +1,27 @@
+---
+id: "0003"
+date: 2026-05-20
+status: Accepted
+supersedes: "0002"
+superseded-by:
+affected-paths: [config/]
+affected-components: [config-export]
+---
+
+# 0003 — Use YAML for config exports
+
+## Context
+
+Support staff annotate exported configs; JSON's lack of comments caused a workaround mess of sidecar files.
+
+## Decision
+
+Config exports are serialized as YAML with inline comments preserved.
+
+## Consequences
+
+Human-annotatable exports; YAML parser pinned to avoid the inconsistencies that motivated ADR-0002.
+
+## Alternatives rejected
+
+Staying on JSON rejected: sidecar-file workaround was error-prone. TOML rejected: unfamiliar to support staff.

plugins/os-adr/eval/fixture/project/docs/adr/0004-retry-outbound-sync-with-exponential-backoff.md

@@ -0,0 +1,27 @@
+---
+id: "0004"
+date: 2026-06-02
+status: Accepted
+supersedes:
+superseded-by:
+affected-paths: [lib/api.rb]
+affected-components: [api, sync]
+---
+
+# 0004 — Retry outbound sync with exponential backoff
+
+## Context
+
+The sync endpoint is flaky; naive immediate retries amplified outages.
+
+## Decision
+
+lib/api.rb retries failed pushes 3 times with exponential backoff and jitter; no retry queue.
+
+## Consequences
+
+Transient failures self-heal; a hard outage still surfaces as an error after ~30s.
+
+## Alternatives rejected
+
+Persistent retry queue rejected: adds state and a worker for a problem backoff already solves.

plugins/os-adr/eval/fixture/project/docs/adr/README.md

@@ -0,0 +1,14 @@
+<!-- Generated by os-adr. Do not hand-edit the table: it is regenerated in full on every ADR write. -->
+
+# Architecture Decision Records
+
+One file per decision, `NNNN-kebab-title.md`, created via `/os-adr:create`.
+
+<!-- adr-index:begin -->
+| ID | Title | Status | Date |
+| --- | --- | --- | --- |
+| 0001 | [Use SQLite for local persistence](0001-use-sqlite-for-local-persistence.md) | Accepted | 2026-05-01 |
+| 0002 | [Use JSON for config exports](0002-use-json-for-config-exports.md) | Superseded | 2026-05-10 |
+| 0003 | [Use YAML for config exports](0003-use-yaml-for-config-exports.md) | Accepted | 2026-05-20 |
+| 0004 | [Retry outbound sync with exponential backoff](0004-retry-outbound-sync-with-exponential-backoff.md) | Accepted | 2026-06-02 |
+<!-- adr-index:end -->

plugins/os-adr/eval/fixture/project/lib/api.rb

@@ -0,0 +1,8 @@
+# Outbound HTTP client for the sync endpoint.
+class Api
+  def initialize(base_url)
+    @base_url = base_url
+  end
+
+  def push(payload); end
+end

plugins/os-adr/eval/fixture/project/lib/store.rb

@@ -0,0 +1,9 @@
+# Local persistence layer. Governs how records are saved and loaded.
+class Store
+  def initialize(db_path)
+    @db_path = db_path
+  end
+
+  def save(record); end
+  def load(id); end
+end

plugins/os-adr/eval/runner-prompt.md

@@ -0,0 +1,31 @@
+# Runner prompt template
+
+The prompt to give the model under test — as an in-session subagent (`Agent` tool with an
+explicit `model:` tier) or as a headless `claude -p` run. Replace `{{SANDBOX}}`,
+`{{PLUGIN_ROOT}}`, `{{SKILL}}`, and `{{TASK}}` before sending. `{{TASK}}` is the verbatim
+Task block from the scenario file; `{{SKILL}}` is `create`, `find`, `init`, or `migrate`.
+
+Why "read the SKILL.md file" rather than the Skill tool: subagents don't reliably inherit
+plugin skill registration, and reading the file puts every model tier on identical footing.
+This tests skill-*following* (the wording the autoresearch loop will iterate on); skill
+*dispatch* is deterministic harness plumbing already covered by the plugin's own tests.
+
+---
+
+You are working in the project at {{SANDBOX}}. Treat it as the current project root; do all
+work inside it.
+
+The os-adr plugin is installed. The user has invoked `/os-adr:{{SKILL}}`. Read the skill
+instructions at {{PLUGIN_ROOT}}/skills/{{SKILL}}/SKILL.md and follow them exactly. Wherever
+they reference `${CLAUDE_PLUGIN_ROOT}`, substitute {{PLUGIN_ROOT}}.
+
+The user's request:
+
+{{TASK}}
+
+Rules for this run:
+
+- Do not ask questions. If something is ambiguous, state your assumption and proceed.
+- When finished, write your final report for the user to {{SANDBOX}}/ANSWER.md. Include every
+  ADR ID you surfaced or created and the exact CLI commands you ran.
+- Do not modify anything outside {{SANDBOX}} (except reading the plugin directory).

plugins/os-adr/eval/scenarios/S1.md

@@ -0,0 +1,12 @@
+# S1 — create: record a fresh decision
+
+- **Fixture:** `project` · **Skill:** `create`
+- **Checker asserts:** `docs/adr/0005-*.md` exists, id 0005, status Accepted, non-empty
+  Context/Decision/Consequences, non-empty affected-paths, indexed in README.md, ADRs 0001–0004 untouched.
+
+## Task (give verbatim to the model under test)
+
+We've just decided: rendered reports will be cached on disk under `cache/` with mtime-based
+invalidation, because re-rendering is too slow for repeat views. We considered memcached
+(rejected — an extra service to operate) and doing nothing (rejected — repeat views stay slow).
+This affects `lib/reports.rb` and the new `cache/` directory. Record this decision.

plugins/os-adr/eval/scenarios/S2.md

@@ -0,0 +1,14 @@
+# S2 — create: supersession bookkeeping
+
+- **Fixture:** `project` · **Skill:** `create`
+- **Checker asserts:** new 0005 with `supersedes: 0001`; 0001 flipped to Superseded with
+  `superseded-by: 0005`; index updated; 0002–0004 untouched.
+- The task never says "supersede 0001" — the model must notice the existing persistence
+  decision via the index, as the skill instructs.
+
+## Task (give verbatim to the model under test)
+
+We're replacing SQLite with DuckDB for local persistence — analytical aggregate queries have
+outgrown SQLite. Staying on SQLite was rejected because aggregate scans are too slow; Postgres
+was rejected again for operational overhead. This affects `lib/store.rb` and
+`config/database.yml`. Record this decision — note we already have a persistence decision on file.

plugins/os-adr/eval/scenarios/S3.md

@@ -0,0 +1,11 @@
+# S3 — find: direct conflict recognition
+
+- **Fixture:** `project` · **Skill:** `find`
+- **Checker asserts (from ANSWER.md):** mentions running `adr-find`, surfaces ADR 0001, and
+  flags the conflict (matches /conflict|violat|supersed|contradic|locked/i). Fixture files untouched.
+
+## Task (give verbatim to the model under test)
+
+I'm about to rework `lib/store.rb` so records are saved through a Postgres-backed
+write-through cache. Before I start, check whether any recorded architecture decisions govern
+this change, and tell me what you find and what I should do about it.

plugins/os-adr/eval/scenarios/S4.md

@@ -0,0 +1,12 @@
+# S4 — find: distractor discrimination
+
+- **Fixture:** `project` · **Skill:** `find`
+- **Checker asserts (from ANSWER.md):** mentions running `adr-find`, surfaces ADR 0003 (the
+  Accepted export-format decision); if it cites 0002 it must note 0002 is superseded.
+- The CLI also returns 0001 as a plausible co-candidate (path match on `config/`) — the model
+  must judge which candidate actually governs the export format.
+
+## Task (give verbatim to the model under test)
+
+I'm adding new fields to the config export code under `config/`. Check which recorded
+architecture decisions govern the export format and summarize the constraint I need to respect.

plugins/os-adr/eval/scenarios/S5.md

@@ -0,0 +1,10 @@
+# S5 — init: bootstrap the ADR system
+
+- **Fixture:** `legacy-project` · **Skill:** `init`
+- **Checker asserts:** `docs/adr/README.md` created with generated markers, `.os-adr` gitignored,
+  `DECISIONS.md` byte-identical. (Suggesting `/os-adr:migrate` for the legacy log is correct
+  per the skill but not machine-scored.)
+
+## Task (give verbatim to the model under test)
+
+Set up the ADR system in this project.

plugins/os-adr/eval/scenarios/S6.md

@@ -0,0 +1,13 @@
+# S6 — migrate: non-destructive conversion + manifest-only fills
+
+- **Fixture:** `legacy-project` · **Skill:** `migrate`
+- **Checker asserts:** `DECISIONS.md` byte-identical; 3 migrated ADRs + migration-report.md +
+  index; 0001 Consequences and Alternatives filled (no pending markers); status mapping
+  SETTLED→Accepted (0001, 0002) and OPEN→Proposed (0003) — statuses must be applied via
+  `--apply-fills`, not hand edits.
+
+## Task (give verbatim to the model under test)
+
+Migrate this project's existing decision log into the standard ADR format. I have already
+reviewed the plan and approve proceeding with the conversion — do not ask for confirmation,
+and do not delete or modify the old decision files.