os-orchestration: trigger-conditioned wording for audit clusters 1-3

Applies the Eval B when->then lesson to the three verified WS1 clusters:
explicit model: before every Agent call (cluster 2, cost exposure now that
CLAUDE_CODE_SUBAGENT_MODEL is removed), verify resolved model after launch
(cluster 1 policy gap), don't re-cover own ground before delegating
(cluster 3). Delegation thresholds unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

.gitignore

@@ -16,3 +16,6 @@ graphify-out/
 
 # Python bytecode
 __pycache__/
+
+# autoresearch skill run logs (eval iterations)
+/autoresearch/

CLAUDE.md

@@ -86,6 +86,7 @@ to those two and fix the stale doc.
 - Hooks: `hooks/` — `inject.py` (injects `ORCHESTRATION.md` as additionalContext to all sessions)
 - Behavior: SessionStart hook injects an `ORCHESTRATION.md` markdown doc (hardcoded; lives in plugin source) as additionalContext, carrying a permissive session-orchestration rule: "do single-file/≤2-tool-call ops directly; delegate only when work is parallelizable across independent files, spans many files, or needs isolated/large context." This is the canonical global default for Claude Code across all projects.
 - Migration: migrated from a standalone repo (`~/dev/cc-plugins/orchestration/`, 2026-07-03) and integrated into cc-os. Supersedes the per-project copy-pasted orchestration text blocks that previously existed in individual project CLAUDE.md files (including a stricter local override that cc-os had carried — now removed; see ADR-019).
+- Session audit (WS1, 2026-07-06): 10 stratified real sessions audited via `audit/bin/extract` (Ruby fact-sheet extractor, dual-use as future eval checker) + parallel sonnet auditors + verified synthesis. Findings + E1–E4 eval-scenario backlog: `docs/orchestration-audit/2026-07-06-findings.md`. Headlines: model param was silently overridden to haiku on all 23 verified spawns — root cause found post-audit: `CLAUDE_CODE_SUBAGENT_MODEL=haiku` in `~/.claude/settings.json` env block (set by an earlier session as a cost measure; removed 2026-07-06; the Fable-5 correlation was coincidental timing). Policy gap remains — no verify-resolvedModel rule; omitted model param inherits the main-loop model (opus/sonnet) — misses cluster in pre-rollout and ops sessions; no over-sharing/full-dump failures anywhere. Eval design deliberately deferred; scenarios must come from these verified misses.
 
 **Global os-doc-hygiene plugin** — `cc-os/plugins/os-doc-hygiene/` (git-tracked, 2026-07-03); symlinked into `~/.claude/plugins/os-doc-hygiene`
 - Hooks: `hooks/hooks.json` → SessionStart hook (matcher: startup|resume) runs `scripts/reminder.py` via `${CLAUDE_PLUGIN_ROOT}` (5s timeout), emitting a deterministic zero-token reminder banner
@@ -98,7 +99,15 @@ to those two and fix the stale doc.
 - Hook: `hooks/session_start.py` wired via plugin-relative `hooks/hooks.json` (`${CLAUDE_PLUGIN_ROOT}`, no `settings.json` entries) — deterministic existence check: present → near-zero-token usage note naming `/os-adr:create` + `/os-adr:find`; absent → once-per-day init/migrate suggestion, silenced permanently by `.os-adr/suppress` (gitignored per-project state dir); silent outside git projects.
 - Skills (verb-first, no `commands/`): `create` (one-invocation templated+numbered+indexed ADR with mechanical supersession), `init`, `migrate` (non-destructive: detect surveyed shapes → mechanical heuristic fill → LLM fills only manifest-listed interpretive fields via `--apply-fills` → `migration_confidence` frontmatter + `docs/adr/migration-report.md`; old-system deletion only as an explicit separate user-approved step), `find` (deterministic-first retrieval: affected-paths/components match → Accepted-only filter → one-hop Graphify query-path expansion with graceful degradation → AI judgment over the candidate set only).
 - Migration pilot passed 2026-07-03 (sandboxed project copies; gate threshold 25% low-confidence flags): viking-warrior-training-log 0%, delta-refinery 0%, llf-schema 8.3%. Retrieval verified end-to-end against the real llf-schema project graph. Resolved: ADRs need no Graphify tag/edge convention — retrieval layer 3 expands the *query* paths via graph-node `source_file`s.
-- **Remaining (locked rollout order):** real-project migration/adoption one at a time via `/os-adr:migrate` — pilot projects first, then the cc-os retrofit (its 19-ADR monolithic file was deliberately excluded from the pilot), then wider; plus the deferred unprompted-behavior eval (held-out scenario shapes sketched in `docs/adr-system/06-eval-scenarios.md`; methodology locked in `04-plugin-requirements.md`).
+- Eval A harness (2026-07-03, ADR-021): `plugins/os-adr/eval/` — prompted skill-execution eval across model tiers (haiku/sonnet). Two fixtures (generated with the plugin's own CLIs), six scenarios (S1 create, S2 create+supersede, S3 find/conflict, S4 find/distractor, S5 init, S6 migrate+fills), deterministic Ruby checker (`eval/bin/check`, structural invariants, TSV mode for autoresearch), sandbox + headless-runner scripts, runner-prompt template. Primary run mode: in-session Agent-tool subagents with pinned `model:` (cheaper than `claude -p`); optimized via the `/autoresearch` Classic loop over SKILL.md *wording only* (checker/fixtures/scenarios frozen during a loop). Procedure: `plugins/os-adr/eval/README.md`. Self-tested both directions; grid run 2026-07-03, all scenarios × both tiers passing. Distinct from the held-out Eval B (unprompted behavior) — do not conflate.
+- Eval B harness (2026-07-03, OpenSpec change `add-os-adr-eval-b-harness`): `plugins/os-adr/eval-b/` — held-out unprompted-behavior eval (requirements 4–5). 7 scenarios (W1–W3 write-trigger, R1–R4 retrieval) authored from the frozen shapes in `docs/adr-system/06-eval-scenarios.md`; dedicated Ruby webhook-relay fixture with a 6-ADR history (Superseded pair + near-miss distractors, generated via the plugin's own CLIs); R4's one-hop graph reach uses a real `graphify update` AST build (model-free, rebuilt via `eval-b/bin/build-fixture-graph`, never committed). **Headless-only runner** (`eval-b/bin/run` — fresh `claude -p` per rep, cwd = sandbox, so the real SessionStart hook fires; in-session subagents are invalid here, unlike Eval A) and a two-axis deterministic-first checker (`eval-b/bin/check`): axis (a) unprompted consultation, mechanical from transcript tool_use blocks; axis (b) correct-ADR citation (R1–R4) or new-ADR-file with a narrow frozen-rubric haiku judge fallback (W1–W3, `judge-rubric.md`, stubbable via `ADR_EVAL_B_JUDGE_CMD`). `R4-nograph` is the graph-degradation variant (expected FAIL). Self-tested both directions model-free via `eval-b/bin/self-test`. **Scenario Task blocks are held-out — never run them informally.** Procedure: `plugins/os-adr/eval-b/README.md`. Grid run 2026-07-03 (1 rep/cell): haiku 0/8 PASS (never unprompted-consults the ADR system in any scenario — a real gap, not a harness defect); sonnet 5/8 PASS (fails W3 — doesn't propose recording the decision; fails R1 — misses the direct-conflict retrieval). `R4-nograph` FAILed on both tiers as expected (degradation check, only meaningful paired with an R4 PASS — sonnet has one, haiku doesn't). Full results + observations (prompting-issue hypothesis, open question on whether in-session subagents could ever validly substitute for part of this measurement) written to the vault: [[os-adr-eval-b-grid-results-and-observations]].
+- **Initial iteration complete (2026-07-03):** plugin build, migration pilot, Eval A grid (clean pass), and Eval B grid (baseline captured above) are all done — this closes the first pass on os-adr. Follow-on work is deliberately deferred to future sessions, not in-flight.
+- **Wording experiment complete (2026-07-04):** the follow-up `/autoresearch` loop over Eval B wording ran (5 iterations, checker/fixtures/scenarios/rubric frozen; fixture CLAUDE.md declared a wording surface upfront) and closed the gap — final full grid **sonnet 8/8, haiku 7/8** (from 5/8 / 0/8 baseline; haiku's one miss is a W3 axis-b judge-boundary flicker). Winning wording shipped in `hooks/session_start.py` (PRESENT_NOTE), find/create SKILL.mds, and `eval-b/fixture/project/CLAUDE.md` (new — the trigger-phrased "Architecture decisions" section, the template for real-project adoption). Confirmed mechanisms: trigger-conditioned when→then phrasing beats inventory phrasing on both tiers; each rule must live where its precondition is visible (the reversal→supersede rule in the find skill's act-on-findings step fixed W3); lower tiers need *mechanical* triggers ("before your first edit to any existing file → run `/os-adr:find` on those paths; additions count") — semantic triggers ("architecture-level choice") only reach sonnet. Open: channel ablation never run (hook vs CLAUDE.md redundancy unknown); R4-nograph now passes both tiers, so the graph-degradation check no longer differentiates. Full hypothesis→result mapping: vault note [[os-adr-eval-b-wording-experiment-hypotheses]]. **Before designing or running any autoresearch eval, Read `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md`.**
+- **Eval B W3 stability check (2026-07-06):** 7 headless reps of haiku × W3 (all reps counted, including two the runner initially excluded, to avoid peeking bias): axis (a) consultation 7/7 PASS; axis (b) recording-offer ~5/7 (~75–85%) — one FAIL plausibly infra, one a genuine behavioral miss (consulted, then asked a clarifying question instead of unconditionally proposing the superseding ADR; the conditional-phrasing failure mode iteration 3 fixed on sonnet, improved but not eliminated on haiku). Verdict: intermittent flicker, not a hard gap — grid claim stays **haiku 7/8**, W3 characterized as an ~1-in-4/5 axis-b miss. Haiku cleared for daily use with that caveat. Recorded in the vault good-enough gate: [[os-adr-eval-b-wording-experiment-hypotheses]].
+- **Eval C harness built, NOT yet run (2026-07-06):** `plugins/os-adr/eval-c/` — ambiguity-ladder DISCRIMINATION eval (held-out; Eval B is contaminated by the wording loop). 3 levels (explicit → moderate → conceptual framing) × paired positive/negative scenarios; run-set (6, job-execution domain) + frozen reserve-set (6, notifications domain — becomes the measurement set if anyone ever tunes wording against the run-set). New Python fixture (`taskq` async job queue, 6-ADR history generated via the plugin's own CLIs, trigger-phrased CLAUDE.md section copied from eval-b). Scoring: positives = both axes; negatives FAIL **only on unneeded ADR creation** — consultation and truthful ADR citation are informational (`A:yes/no`, `cited-adr:yes/no`), never FAILs, because the shipped wording endorses cheap finds and the negatives sit deliberately in ADR-covered domains (two earlier negative designs that punished instruction-compliant behavior were caught in review and redesigned/fixed). Model-free `bin/self-test` green, including a truthful-citation-must-PASS guard. **Scenario Task blocks are held-out — never run informally.** Methodology: vault notes [[eval-methodology-ladder]] (per-level pass bars, reserve discipline) and [[eval-methodology-irl-feedback-loop]] (post-rollout session-audit backlog); vault also gained an `eval-results` note type + `_templates/eval-results.md` (2026-07-06).
+- **Eval C frozen grid run (2026-07-06):** run-set × {sonnet, haiku} × 3 reps/cell (36 headless reps, all counted) — **12/12 cells PASS at ≥2/3 majority; haiku 18/18 reps, sonnet 17/18**. Zero over-trigger ADR creation on any negative at any level; every positive pass was via governing-ADR citation (16) or judge-recognized proposal (1), never duplicate creation. Sonnet's one FAIL rep (P1-L1) is a final-message-citation boundary artifact (implemented the ADR-0002 policy, cited only 0005), not a behavioral miss. Two harness defects fixed on the first live rep, BEFORE the grid, with Task blocks/fixtures/rubric untouched: (1) `bin/run` pipefail bug aborted a cell's reps after the first FAIL; (2) positive axis-b required creation/proposal, contradicting the README's own "consulting and/or recording is correct" — the run-set positives sit in *already-decided* territory, so find→cite→comply is correct; fixed via a mechanical `cited-governing` PASS branch + two new self-test guards (`B-via:` now recorded in TSV). Verdict: the Eval B wording **generalizes** (new fixture/language/domain, decreasing cue explicitness, paired negatives) — grid saturated, ceiling not found; next signal is production (rollout + IRL session audits), not another lab rung. Run-set is now the baseline; wording tuning against it moves measurement to the reserve-set. Full results: vault note [[os-adr-eval-c-frozen-grid-results]].
+- **Remaining (locked rollout order):** real-project migration/adoption one at a time via `/os-adr:migrate` — pilot projects first, then the cc-os retrofit (its 19-ADR monolithic file was deliberately excluded from the pilot), then wider. When onboarding real projects, add the trigger-phrased "Architecture decisions" CLAUDE.md section (copy from `eval-b/fixture/project/CLAUDE.md`; candidate: emit it from `/os-adr:init`/`migrate`).
+- **Resolved (2026-07-04):** os-adr's skills failed to register on first install due to stale plugin caches — the cache that was installed on 2026-07-03 17:21:48 was missing `hooks/hooks.json`, `bin/adr-detect`, `bin/adr-find`, `bin/adr-migrate`. Root cause: unknown, but the caches were restored via plugin uninstall/reinstall. A parallel issue affected os-doc-hygiene (cache retained deleted `commands/` directory). Fixed by manually refreshing both caches, then created `bin/refresh-plugins` automation to prevent future drift — see "Editing a local plugin (cache refresh)" subsection for refresh procedure. Investigation result: stale caches (not manifest-naming issues) were the cause; slash command registration works correctly once caches are fresh.
 
 **Graphify** — v0.8.31 at `/home/jared/.local/bin/graphify`
 - Vault graph: `~/Documents/SecondBrain/graphify-out/` — disposable, structural index rebuilt by SessionStart hook; handles relational/graph queries. Distinct from vault-index.json (planned, Plan B Phase 1): a flat `{tag: [{path, title, summary}]}` lookup rebuilt at SessionEnd, queried by the `/memory-find` skill for fast tag-based SB note discovery.
@@ -151,6 +160,19 @@ Skipping step 2 is the failure mode to watch for: hooks (wired by absolute path
 `settings.json`) keep working, masking the fact that skills/slash-commands silently stopped
 registering under the plugin's new name.
 
+### Editing a local plugin (cache refresh)
+
+Plugin source edits (including SKILL.md wording, hook scripts, or CLI code) do not reach running
+sessions until the cache is refreshed. Claude Code caches plugins when installed, copying them
+to `~/.claude/plugins/cache/<marketplace>/<plugin>/<version>/`. To refresh a stale cache after
+source edits:
+
+```bash
+bin/refresh-plugins [marketplace-names...]    # refresh all or specific local marketplaces
+```
+
+This uninstalls and reinstalls each plugin, forcing the cache to repopulate. See `bin/refresh-plugins --help` for usage. After refreshing, `claude plugin details <name>@<marketplace>` should show updated skill descriptions and hook counts. Directory-source marketplaces (e.g., `local-plugins` sourced from `~/.claude/plugins`, `cc-plugins` sourced from `~/dev/cc-plugins`) are the only ones that need manual refresh — public marketplaces auto-update.
+
 ## Issue tracking
 
 Issues (created via `/to-issues`) live on self-hosted Forgejo (`jared/cc-os`), queried with the `tea` CLI — not GitHub/`gh`. See `docs/issue-workflow.md` for token setup, listing, implementing, and closing issues.
@@ -179,6 +201,15 @@ project context for OpenSpec artifacts comes from `docs/` and this file.
   `03-architecture-decisions.md` with the reasoning.
 - The package on PyPI is `graphifyy` (double-y) but the command is `graphify`.
 - **Keep this file current:** When a build step completes, (a) mark it done in `docs/memory-system/04-build-plan.md`, (b) add or update a component pointer in the Implemented Components section above, and (c) update the current design paragraph if the design changed. This file is the AI's orientation entry point — accuracy matters more than brevity.
-- **Plugin and skill naming:** Read [[cc-os-plugin-skill-naming-convention]] for `os-` prefix and verb-first skill conventions before naming a new cc-os plugin or skill. Agent and hook naming are open questions per that note, not yet covered by this rule.
+- **Plugin and skill naming:** Before naming ANY new cc-os plugin, skill, or slash command, Read
+  `~/Documents/SecondBrain/cc-os-plugin-skill-naming-convention.md` and follow it — do NOT invent
+  a new convention. Rules in brief: plugins are `os-[domain]`; skills are verb-first kebab-case
+  (`query`, `onboard-project`), invoked as `/os-[domain]:[verb]`; no `commands/` dispatcher
+  directories; **never set a `name:` field in SKILL.md frontmatter** — the directory name is the
+  skill name, and an explicit `name:` collapses the slash command to a bare unnamespaced form
+  (`/find` instead of `/os-adr:find`; found and fixed 2026-07-04). After editing any SKILL.md,
+  run `bin/refresh-plugins` (installs cache plugin files; source edits don't reach sessions until
+  refreshed). Agent and hook naming are open questions per that note, not yet covered by this
+  rule.
 
 **Session orchestration behavior** — Provided by the global `os-orchestration` plugin (see Implemented Components below). This repo no longer carries a local orchestration override; it follows the plugin's default behavior like every other project.

bin/refresh-plugins

@@ -0,0 +1,123 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+# Refresh Claude Code plugin caches for directory-source marketplaces.
+# Discovers installed plugins from each local marketplace and refreshes their caches.
+#
+# Usage:
+#   bin/refresh-plugins [marketplace-names...]
+#   bin/refresh-plugins                    # refresh all local marketplaces
+#   bin/refresh-plugins local-plugins      # refresh only local-plugins
+#   bin/refresh-plugins cc-plugins local-plugins
+
+require 'json'
+require 'pathname'
+
+class PluginRefresher
+  attr_reader :results
+
+  def initialize
+    @results = []
+  end
+
+  def refresh(marketplace_names = nil)
+    marketplaces = load_marketplaces(marketplace_names)
+
+    marketplaces.each do |name, config|
+      next unless config['source']['source'] == 'directory'
+
+      puts "Refreshing marketplace: #{name}"
+      refresh_marketplace(name, config)
+      puts
+    end
+
+    report_results
+  end
+
+  private
+
+  def load_marketplaces(names)
+    all = JSON.parse(File.read(File.expand_path('~/.claude/plugins/known_marketplaces.json')))
+    return all unless names
+
+    all.select { |k, _| names.include?(k) }
+  end
+
+  def refresh_marketplace(marketplace_name, config)
+    source_path = config['source']['path']
+    installed = load_installed_plugins(marketplace_name)
+
+    installed.each do |plugin_name|
+      refresh_plugin(marketplace_name, plugin_name, source_path)
+    end
+  end
+
+  def load_installed_plugins(marketplace_name)
+    all = JSON.parse(File.read(File.expand_path('~/.claude/plugins/installed_plugins.json')))
+    all['plugins']
+      .select { |k, _| k.end_with?("@#{marketplace_name}") }
+      .keys
+      .map { |k| k.sub(/@#{marketplace_name}$/, '') }
+      .uniq
+  end
+
+  def refresh_plugin(marketplace_name, plugin_name, source_path)
+    cache_exists = cache_exists?(marketplace_name, plugin_name)
+
+    unless File.exist?(File.join(source_path, plugin_name))
+      @results << [plugin_name, "source not found at #{source_path}", :error]
+      puts "  #{plugin_name}: ERROR — source not found"
+      return
+    end
+
+    system("claude plugin uninstall #{plugin_name}@#{marketplace_name} > /dev/null 2>&1")
+    system("claude plugin install #{plugin_name}@#{marketplace_name} > /dev/null 2>&1")
+
+    if verify_cache(marketplace_name, plugin_name, source_path)
+      @results << [plugin_name, "refreshed", :success]
+      puts "  #{plugin_name}: REFRESHED"
+    elsif !cache_exists
+      @results << [plugin_name, "in-sync (not previously cached)", :info]
+      puts "  #{plugin_name}: IN SYNC (new)"
+    else
+      @results << [plugin_name, "STALE", :warning]
+      puts "  #{plugin_name}: STALE (differences remain)"
+    end
+  end
+
+  def cache_exists?(marketplace_name, plugin_name)
+    Dir.exist?(File.expand_path("~/.claude/plugins/cache/#{marketplace_name}/#{plugin_name}"))
+  end
+
+  def verify_cache(marketplace_name, plugin_name, source_path)
+    cache_base = File.expand_path("~/.claude/plugins/cache/#{marketplace_name}/#{plugin_name}")
+    source_full = File.join(source_path, plugin_name)
+
+    # Find the version directory in cache
+    versions = Dir.glob(File.join(cache_base, '*')).select { |p| File.directory?(p) }
+    return false if versions.empty?
+
+    cache_path = versions.first
+
+    # Run diff check (exclude plugin manager state files)
+    output = `diff -rq "#{cache_path}" "#{source_full}" --exclude='.git' --exclude='__pycache__' --exclude='.orphaned_at' 2>&1`
+    output.strip.empty?
+  end
+
+  def report_results
+    puts "\n=== Summary ==="
+    successes = @results.select { |_, _, status| status == :success }
+    warnings = @results.select { |_, _, status| status == :warning }
+    errors = @results.select { |_, _, status| status == :error }
+    infos = @results.select { |_, _, status| status == :info }
+
+    puts "Refreshed: #{successes.map { |r| r[0] }.join(', ')}" if successes.any?
+    puts "In sync: #{infos.map { |r| r[0] }.join(', ')}" if infos.any?
+    puts "Stale (not fixed): #{warnings.map { |r| r[0] }.join(', ')}" if warnings.any?
+    puts "Errors: #{errors.map { |r| "#{r[0]} (#{r[1]})" }.join(', ')}" if errors.any?
+    puts "All plugins synchronized." if @results.all? { |_, _, s| s != :warning && s != :error }
+  end
+end
+
+marketplace_names = ARGV.empty? ? nil : ARGV
+PluginRefresher.new.refresh(marketplace_names)

docs/adr-system/06-eval-scenarios.md

@@ -1,7 +1,18 @@
 # Eval Scenario Sketches (held-out, deferred stage)
 
-_Last updated: 2026-07-03 — sketches only; the evaluation itself is a separate later stage
-(locked sequencing in `04-plugin-requirements.md`)._
+_Last updated: 2026-07-03 — these shapes are now built out as the Eval B harness at
+`plugins/os-adr/eval-b/` (fixtures, prompts, headless runner, two-axis checker; OpenSpec change
+`add-os-adr-eval-b-harness`). The shapes below stay frozen — the harness executes them, it does
+not renegotiate them. Running the grid remains a separate later stage (locked sequencing in
+`04-plugin-requirements.md`)._
+
+> **Two evals, don't conflate them (ADR-021).** This file sketches **Eval B** — held-out,
+> unprompted-behavior (requirements 4–5). A separate **Eval A** — prompted skill-execution
+> across model tiers (can haiku/sonnet correctly *execute* an explicitly invoked skill?) —
+> is built and lives at `plugins/os-adr/eval/` with its own README; its prompts are *not*
+> held-out and may be iterated freely. When Eval B eventually runs, it runs as
+> scenario × model tier (haiku/sonnet/session-model), pass rate per tier as the
+> autoresearch metric.
 
 These sketch the *shape* of the held-out scenarios that will eventually measure requirements 4
 and 5 (unprompted write-trigger recognition, unprompted correct retrieval). They exist now, before

docs/memory-system/03-architecture-decisions.md

@@ -603,6 +603,20 @@ _Date: 2026-07-03_
 - **Cross-references**: `docs/adr-system/04-plugin-requirements.md` (locked requirements), `05-plugin-prd.md` (phases), `06-eval-scenarios.md` (held-out eval sketches — the eval itself stays deferred), ADR-013 (build-first/migrate-incrementally precedent), ADR-018 (three-place plugin registration, followed during install), ADR-019 (plugin conventions).
 - **Status**: Accepted 2026-07-03. Plugin live at `cc-os/plugins/os-adr/`, installed as `os-adr@local-plugins`. Rollout order locked: pilot projects' real migration (interactive, per project) → cc-os retrofit → wider.
 
+## ADR-021 — Model-tier skill-execution eval for os-adr (Eval A), run in-session via subagents
+
+_Date: 2026-07-03_
+
+- **Context**: The os-adr build validated every model-touching surface (skill-following, migrate fills, find judgment) only at the session model used to build it (Fable). Confidence that the skills execute correctly on the *weakest* tier (Haiku) would imply confidence on all stronger tiers. The locked eval methodology (`04-plugin-requirements.md`) covers only the deferred held-out unprompted-behavior eval (requirements 4–5, "Eval B") and says nothing about model tiers. Separately, headless `claude -p` runs consume more of the user's subscription credit than in-session work.
+- **Decision**: (1) A second, non-held-out eval — **Eval A: prompted skill-execution across model tiers** — lives at `plugins/os-adr/eval/`: two fixture projects (generated with the plugin's own CLIs), six scenarios (S1 create, S2 create+supersede, S3 find/conflict, S4 find/distractor, S5 init, S6 migrate+fills), a deterministic Ruby checker (`bin/check`, structural invariants only — never prose quality), sandbox + headless-runner scripts, and a runner-prompt template. (2) **Primary run mode is in-session**: a driver session spawns Agent-tool subagents with `model:` pinned to the tier under test; each subagent reads the SKILL.md file directly (uniform across tiers; dispatch is deterministic plumbing). Headless `claude -p` is the fidelity fallback only. (3) The **`autoresearch` skill (Classic mode)** wraps the grid as its metric to iterate SKILL.md *wording only* — checker, fixtures, scenarios, and runner prompt are frozen during a loop, as the guard against metric-gaming. (4) **Eval B gains a model axis** when it eventually runs: scenario × model tier, pass rate per tier as the autoresearch metric.
+- **Rationale**: The deterministic core (CLIs, hook, index) is model-independent and already tested; what varies by tier is instruction-following, so the cheap, high-signal eval is exactly that surface with machine-checkable pass criteria (the plugin's invariants double as scoring rules). In-session subagents preserve the user's credits and parallelize; the fidelity gap (no SessionStart context, no slash dispatch) is irrelevant for *explicitly invoked* skills. Eval A prompts are deliberately not held-out — only Eval B's are — so they can be iterated on freely without contaminating the deferred methodology.
+- **Alternatives rejected**:
+  - **Headless-only harness**: higher subscription cost per run, no parallel driver, no benefit for prompted-execution scoring; kept as fallback for full-fidelity checks.
+  - **Folding tier-testing into Eval B**: conflates two questions (can the model *execute* a skill it was told to run vs. does it *recognize* when to run one) and would burn held-out scenarios on mechanical failures.
+  - **LLM-judged scoring**: reintroduces the model-capability variable into the scorer; structural invariants are sufficient and reproducible.
+- **Cross-references**: `plugins/os-adr/eval/README.md` (procedure + autoresearch invocation), `docs/adr-system/06-eval-scenarios.md` (Eval B sketches, now noting the model axis), ADR-020 (pilot gate the S6 fixture mirrors).
+- **Status**: Accepted 2026-07-03. Harness built and self-tested (all six scenarios: perfect-run PASS, untouched-sandbox FAIL). The grid itself has not yet been run against haiku/sonnet.
+
 ## Rejected tools (summary)
 
 | Tool | Why rejected for our use |

docs/orchestration-audit/2026-07-06-findings.md

@@ -0,0 +1,153 @@
+# os-orchestration session audit — findings
+
+_Audit date: 2026-07-06. Status: complete (WS1 of `docs/plans/ws1-orchestration-audit.md`)._
+_Method: Phase 1 extractor (`plugins/os-orchestration/audit/bin/extract`) → 10 stratified
+sessions → 10 parallel sonnet auditors (7-question rubric) → orchestrator-verified synthesis.
+Every load-bearing claim below was re-verified against the primary transcript; auditor claims
+that failed or only partially survived verification are marked._
+
+## Sample
+
+| ID | Project | Date | Population | Turns | Spawns | Main-loop model |
+|----|---------|------|------------|-------|--------|-----------------|
+| S1 | cc-os | 2026-07-06 | post-rollout (contaminated¹) | 66 | 10 | fable-5 |
+| S2 | cc-os | 2026-07-06 | post-rollout (contaminated¹) | 207 | 8 | fable-5 |
+| S3 | cc-os | 2026-07-04 | post-rollout (contaminated¹) | 255 | 5 | fable-5 |
+| S4 | llf-schema | 2026-06-30 | pre-rollout | 191 | 41 | opus-4-8 |
+| S5 | llf-schema | 2026-06-29 | pre-rollout | 36 | 8 | sonnet-4-6 |
+| S6 | philly-sem | 2026-06-29 | pre-rollout | 21 | 1 | sonnet-4-6 |
+| S7 | servers | 2026-07-04 | post-rollout | 144 | 3 | sonnet-5 |
+| S8 | servers | 2026-07-04 | post-rollout | 90 | 0 | sonnet-5 |
+| S9 | systems-admin | 2026-06-30 | pre-rollout | 117 | 3 | opus-4-8 |
+| S10 | systems-admin | 2026-07-01 | pre-rollout | 53 | 0 | sonnet-5 |
+
+¹ cc-os sessions were actively building eval/orchestration tooling; delegation-awareness in
+them overstates the general population.
+
+Transcripts: `~/.claude/projects/<dir>/<uuid>.jsonl`; fact-sheets and full per-session auditor
+reports were session artifacts (scratchpad); regenerate fact-sheets any time with the extractor.
+
+## Per-question summary (across 10 sessions)
+
+1. **Subagents called when they should be?** Mostly yes. 8/10 PASS or justified-no-delegation
+   (S8, S10 correctly kept sequential/coordinated work in-session — both heuristic
+   missed-delegation flags there were judged false positives). Genuine gray zone: S3 and S7
+   did heavy in-session investigation (19–37 calls, 74KB) that could have been one
+   "investigate and report" delegation.
+2. **Correct model per subagent?** The dominant failure area, in two distinct modes (see
+   clusters 1 and 2). Explicit-tier choices, when the orchestrator made them, were sensible
+   (haiku for mechanical, sonnet for judgment).
+3. **Planning/grouping for context efficiency?** Good. Concurrent fan-out used in S1, S2, S4,
+   S5, S7, S9; serial dependencies respected. Minor: reactive late spawns (S1 agents 8–10)
+   instead of pre-planned batches.
+4. **Avoiding unnecessary orchestrator reads?** Mostly PASS (S4: zero orchestrator bytes over
+   41 spawns; S1, S5, S6 near-zero). Exceptions: S7 (74KB pre-spawn diagnostics, then Explore
+   agents told to re-read files "already read" — partially verified²), S9 (read
+   `settings.json`, then spawn #1 instructed to read it again — confirmed).
+5. **Over-sharing context with subagents?** PASS everywhere. Prompts 0.4–6.7KB,
+   instruction-forward; no transcript/CLAUDE.md dumps observed in any session.
+6. **Following ORCHESTRATION.md?** Split by population: post-rollout cc-os sessions pass the
+   explicit-model rule 23/23 spawns; pre-rollout and ops sessions omit `model` on 16/56 spawns
+   (S4×9, S5×6, S6×1, S7×3, S9×3 — counting per verified transcript data, 22 omissions total
+   including Explore-type spawns). Everything else (delegation thresholds, orienting reads)
+   broadly followed.
+7. **Receiving only needed context back?** PASS everywhere. Agent results 0.9–13KB summaries;
+   no full-dump pattern found.
+
+² Confirmed for one file (`incidents.md`, S7 line 182 → spawn-2 prompt) and for the prompt
+text "CLAUDE.md (already read, but note key conventions)"; the auditor's other named files
+could not be confirmed as orchestrator Reads (may have been Bash cats) — treated as plausible.
+
+## Failure-mode clusters (verified)
+
+### Cluster 1 — model param silently overridden to haiku in Fable-5 sessions (HIGH, environment)
+
+Verified across S1/S2/S3 (23 spawns): requested `sonnet` (18), `opus` (1), `haiku` (4) — every
+one resolved `claude-haiku-4-5-20251001`. In S4/S5 (opus/sonnet main loops) explicit params
+were honored (sonnet→sonnet-4-6, haiku→haiku-4-5). So under a Fable-5 main loop the `model`
+param is currently a no-op and all subagent work runs at haiku tier, including tasks the
+orchestrator explicitly graded as judgment work (e.g. S3 line 603: "answer from judgment
+alone" → haiku). The orchestrator never detected or escalated the downgrade — nothing in the
+policy tells it to check `resolvedModel`.
+
+- **Not an orchestrator wording failure** — compliance with the written rule was perfect in
+  these sessions. It is an ops/environment issue plus a policy gap (no verify-resolution rule).
+- **ROOT CAUSE FOUND (2026-07-06, post-audit):** `CLAUDE_CODE_SUBAGENT_MODEL=haiku` was set in
+  `~/.claude/settings.json`'s `env` block (added by an earlier AI session as a cost measure).
+  That env var force-overrides every subagent's `model` param. The "Fable-5 main loop"
+  correlation was coincidental timing — the var was added after the pre-rollout opus/sonnet
+  sessions (S4/S5, 06-29/30, params honored) and before the Fable-5 sessions. **Removed
+  2026-07-06.** E1 remains valid as a detect-the-downgrade eval; the policy gap (no
+  verify-`resolvedModel` rule) still stands regardless of cause.
+
+### Cluster 2 — omitted `model` param inherits the main-loop model (HIGH, orchestrator)
+
+Verified: omission resolves to the parent session's model — opus-4-8 in S4 (×7) and S9 (×2),
+sonnet in S5 (×6) and S6 (×1) — except agent types with their own default (Explore→haiku,
+S7 ×3, S4 ×2, S9 ×1). Cost consequence is real: S6's single spawn was a mechanical file-edit
+task billed at sonnet; S9 ran a character-counting task at opus. All these sessions are
+pre-rollout or ops sessions where the global plugin text either wasn't present or wasn't
+followed on this rule. Post-rollout cc-os sessions (S1–S3): zero omissions.
+
+- The rule exists and works where the plugin text is in force and salient; the misses cluster
+  in the population that ran without it (pre-07-03) or in ops projects (S7 07-04, post-rollout,
+  still omitted on all 3 spawns — the one confirmed post-rollout violation).
+- Action candidates: mechanical trigger-phrased wording (the Eval B lesson — "before every
+  Agent call → include `model:`"); eval scenario E2.
+
+### Cluster 3 — orchestrator self-investigates then delegates the same ground (MEDIUM)
+
+S7: 32 Bash + 4 Reads (74KB) of diagnostics before spawning 3 Explore agents whose prompts
+overlap that ground (one confirmed file-level dual-read; prompt text acknowledges "already
+read"). S9: read `settings.json` then instructed spawn #1 to read it (confirmed). S3: 19-call
+investigate-then-fix run kept in-session where one delegation could have isolated it.
+
+- Boundary is genuinely fuzzy — the policy allows "a short orienting Read"; these exceed
+  "short" but the work partly informed direct action too. Judged real but medium-severity.
+- Action candidate: eval scenario E3 (bytes-read-before-first-spawn budget).
+
+### Cluster 4 — reactive late spawns instead of planned batches (LOW)
+
+S1 (agents 8–10 spawned only after synthesis of 4–7), S3 (91 post-eval tool calls that could
+have been folded into the eval spawns' own prompts). Low priority: wall-clock cost only,
+no correctness or spend impact demonstrated.
+
+### Non-findings worth keeping
+
+- Zero cases of context over-sharing to subagents; zero cases of full-dump returns (Q5/Q7
+  clean across all 10 — the async task-notification pattern returns summaries by design).
+- The missed-delegation heuristic (≥4 same-tool runs) produced 4 flags in the sample;
+  auditors + verification judged all of them false positives (coordinated multi-file units or
+  sequential-dependent work). Threshold/shape needs tuning before it can score an eval.
+
+## Eval-scenario backlog (inputs to a future eval design — NOT an eval yet)
+
+- **E1 (from cluster 1):** Scripted task forces a judgment-grade delegation; harness stubs
+  resolution to haiku. Criterion: orchestrator notices `resolvedModel` mismatch in the launch
+  stub and says so / adapts. Checker: extractor already captures requested vs resolved.
+- **E2 (from cluster 2):** Multi-file parallelizable task in a non-cc-os fixture. Criterion:
+  every Agent spawn includes `model:`; mechanical work→haiku, judgment→sonnet. Checker:
+  extractor `model_explicit` field — fully mechanical. Paired negative: single-file ≤2-call
+  task where spawning anything is the failure.
+- **E3 (from cluster 3):** Task whose natural first move is delegatable investigation (log
+  review across many files). Criterion: orchestrator bytes-read before first spawn under a
+  threshold (extractor segment `pre-spawn-1`), no file both Read by orchestrator and assigned
+  to a subagent prompt. Paired negative: task where orienting reads are correct (uncertain
+  target path).
+- **E4 (from cluster 4, deferred):** Batch-planning quality — hard to score mechanically;
+  design only if clusters 1–3 get fixed and this remains visible.
+
+Per the plan's eval-design note: these run as scripted multi-file headless tasks scored by the
+Phase 1 extractor + thresholds; mid-session behavior, so the Eval B SessionStart pattern does
+not transfer. Design gate: follow `~/Documents/SecondBrain/eval-methodology-ladder.md`
+(paired positives/negatives, frozen reserve) when authoring.
+
+## Caveats
+
+- Sample: 10 sessions, single audit pass; cc-os sessions contaminated (delegation-aware work);
+  only 2 truly post-rollout non-cc-os sessions (S7, S8) — the population the plugin targets is
+  thin in this sample. Re-audit in ~2 weeks per the IRL-feedback-loop cadence.
+- Auditor reliability: several auditor claims did not survive verification as stated
+  (S4 "defaults to opus" was right for general-purpose but auditors variously mis-attributed
+  the mechanism; S7's dual-read file list only partially confirmed). All cluster claims above
+  reflect the verified transcript data, not the raw auditor reports.

docs/orchestration-audit/auditor-reports/S1-report.md

@@ -0,0 +1,224 @@
+# Orchestration Audit: S1-df546b88
+
+## Session Metadata
+- Transcript: `/home/jared/.claude/projects/-home-jared-dev-cc-os/df546b88-e27d-47e1-888c-648012c24e62.jsonl`
+- cwd: /home/jared/dev/cc-os
+- Duration: 2026-07-06 15:32:42 to 18:20:38 (2h 48m)
+- Assistant turns: 66 (all Fable-5 / Haiku 4.5)
+- Agent spawns: 10 (general-purpose ×3, perspectives ×4, Explore ×1, general-purpose ×2)
+- Model params: All 10 requested `model: "sonnet"` → all 10 resolved as `claude-haiku-4-5-20251001`
+
+## Critical Finding
+**Agent model parameter is not being respected by the Agent framework.** All 10 agents explicitly requested `model: "sonnet"` in their tool_use input but resolved to Haiku. This prevents the orchestrator from implementing the cost-quality tradeoff specified in ORCHESTRATION.md.
+
+Evidence: Line 21 (agent spawn 1) shows `"model": "sonnet"` in the Agent input; line 22 (toolUseResult) shows `"resolvedModel": "claude-haiku-4-5-20251001"`. Pattern holds for all 10 agents (lines 21, 23, 25, 79, 81, 83, 85, 202, 218, 219 as spawns; each followed by a tool result line with resolvedModel=haiku).
+
+---
+
+## Q1: Are subagents getting called when they should be?
+
+**Verdict: PASS (with user's explicit request caveat)**
+
+The user's initial prompt explicitly requests delegation: "Dispatch subagents based on model to intelligence and cost (low cost, high quality) to explore and research options, approaches as well as tools and services."
+
+**Evidence:**
+- Line 4 (user prompt): "Dispatch subagents based on model to intelligence and cost (low cost, high quality)"
+- Lines 20-21: Orchestrator accepts the delegation request and clarifies reasoning: "Dispatching three sonnet researchers now (this is judgment-heavy evaluation work, not mechanical, so no haiku)"
+- Agents 1-3 (lines 21, 23, 25) research distinct tools (Storybloq, self-hosted kanban options, agent-native backlogs) — parallelizable research tasks with no dependencies
+
+**Pattern observation:** Agents 4-7 (perspectives) and 8-10 (deep-dives) appear to be reactive rather than pre-planned, running after earlier results come back. This is defensible (each synthesizes prior findings) but weakens the "pre-planned delegation" story.
+
+**Candidate issue:** Agents 8-10 (Explore Hermes, Planka deep-dive, modern kanban alternatives sweep) lack clear pre-planning signals in the transcript. They appear reactive to synthesized agent findings rather than pre-delegated work.
+
+---
+
+## Q2: Is the correct model chosen per subagent?
+
+**Verdict: FAIL — Model parameter ignored by Agent framework**
+
+The orchestrator correctly specifies model parameters per ORCHESTRATION.md policy:
+- All 10 agents requested `model: "sonnet"` (judgment-heavy research and perspective work)
+- Policy expectation: Sonnet for judgment work, Haiku for mechanical work
+- **Actual outcome:** All 10 resolved to Haiku
+
+**Evidence:**
+- Line 21: `"model": "sonnet"` in Agent tool input
+- Line 22: `"resolvedModel": "claude-haiku-4-5-20251001"` in toolUseResult
+- Line 23: Same pattern; line 24: `"resolvedModel": "claude-haiku-4-5-20251001"`
+- Line 25: `"model": "sonnet"`; line 26: `"resolvedModel": "claude-haiku-4-5-20251001"`
+- Agents 4-7 (perspectives: devils-advocate, simplifier, implementer, premortem): all requested `model: "sonnet"` but resolved to Haiku
+- Agents 8-10: same pattern (lines 202, 218, 219 spawns → Haiku resolves on subsequent lines)
+
+**Root cause:** Unknown — likely Agent framework / Claude Code plugin issue, not orchestrator error. The orchestrator is making the correct specification; the framework is downgrading.
+
+**Impact:** Orchestrator cannot implement the stated policy of "sonnet for judgment work, haiku for mechanical work." All work runs at lower quality/cost than specified.
+
+---
+
+## Q3: Is the orchestrator planning/grouping tasks to maximize efficient context-window use?
+
+**Verdict: MIXED — Good initial batching, then reactive pattern**
+
+**Initial parallelization (agents 1-3):**
+- Agents 1-3 dispatched together to research three distinct but related kanban systems
+- Independent research tasks with no inter-dependencies
+- Well-scoped individual prompts (~1100-2100 chars each)
+- No orchestrator reading between spawns 1-3 (fact-sheet shows 0 tool calls pre-spawn-4)
+- **This is optimal parallel batching**
+
+**Perspectives batch (agents 4-7):**
+- Agents 4-7 (devils-advocate, simplifier, implementer, premortem) dispatched after agents 1-3 complete
+- Reviewing a concrete proposal synthesized from agent 1-3 results
+- All four perspectives agents dispatched together (lines 79, 81, 83, 85)
+- No reading between spawns 4-7
+- **Pattern:** Sequential synthesis (agents 1-3 complete → orchestrator synthesizes → agents 4-7 launch against synthesis)
+- This is reasonable but not pre-planned; it's reactive to agent results
+
+**Final deep-dives (agents 8-10):**
+- Agent 8 (Explore Hermes agent OS): line 202
+- Agent 9 (Planka maturity deep-dive): line 218
+- Agent 10 (Modern kanban alternatives sweep): line 219
+- Agents 9-10 launched together but agent 8 is isolated
+- No orchestrator context prep visible between perspectives completion and agent 8 launch
+- **Pattern:** Reactive to ongoing synthesis, not pre-planned
+
+**Inefficiency candidate:** The orchestrator could have planned agents 8-10 upfront rather than spawning them reactively. However, spawning them reactively against fresh synthesis might actually be more context-efficient (each agent sees the prior conclusions it's building on) — net verdict unclear.
+
+---
+
+## Q4: Is the orchestrator avoiding reading files it does NOT need?
+
+**Verdict: PASS — Minimal unnecessary reading**
+
+**File reading pattern:**
+- Pre-spawn-1 through post-spawn-6: 0 bytes read (fact-sheet segments pre-spawn-1 through after-spawn-6)
+- After-spawn-7 (post-perspectives): 970 bytes read via Skill:1, Bash:1, Write:2
+- After-spawn-10 (final synthesis): 7106 bytes read via Bash:3, Read:1, Write:1, Edit:1
+
+**Details of after-spawn-7 reading (970 bytes):**
+- Likely routine write of agent results or notes; Skill invocation suggests a vault operation
+
+**Details of after-spawn-10 reading (7106 bytes):**
+- Line 239: Bash to list `/home/jared/servers/ovh-prod/` and grep for SecondBrain references (practical investigation of user's infrastructure)
+- Line 244: Read tool (exact file unknown without parsing, but single Read suggests targeted lookup, not bulk scan)
+- Writes: likely capturing findings
+
+**Assessment:** The orchestrator does not pre-load CLAUDE.md, docs/, or large context files before delegating. File operations are minimal and come after agents complete, suggesting the orchestrator is being selective about what to read. This is aligned with ORCHESTRATION.md guidance: "A short orienting Read before delegating is fine when the target file/path is uncertain. Don't delegate the orienting step itself."
+
+---
+
+## Q5: Is the orchestrator sharing too much context with subagents?
+
+**Verdict: PASS — Prompts are focused, not bloated**
+
+**Agent 1 prompt (Storybloq research):** ~1098 chars
+- Specific research task (assess GitHub project for kanban/backlog viability)
+- Clear criteria (maturity, self-hosting, AI accessibility, visual dashboard)
+- No dump of CLAUDE.md, project history, or vault context
+- Appropriate scope for isolated research
+
+**Agents 2-3 prompts:** ~1600-2100 chars each
+- Agent 2: Specific survey task (self-hosted kanban tools inventory)
+- Agent 3: Specific research task (agent-native/markdown backlogs)
+- No bloat; instruction-forward, not context-forward
+
+**Agents 4-7 (perspectives) prompts:** ~2100-3300 chars each
+- Each perspective receives the concrete proposal being reviewed
+- No full CLAUDE.md dump; focused on the proposal and the perspective lens
+- Reasonable context load for judgment work
+
+**Agents 8-10 prompts:** Unknown exact sizes but factsheet rows suggest ~1100-2300 chars (row 8: 1120 chars, row 9: 1832 chars, row 10: 2301 chars)
+- Proportionate to research scope, not bloated
+
+**Assessment:** No evidence of context waste. Prompts are instruction-dense (telling agents what to do, why, and how to report) rather than context-dense (dumping large files).
+
+---
+
+## Q6: Is the orchestrator following the ORCHESTRATION.md instructions?
+
+**Verdict: MIXED — Orchestrator tries to follow but Agent framework doesn't comply**
+
+**ORCHESTRATION.md policy (stated verbatim):**
+1. "Do single-file, ≤2-tool-call ops directly. Don't delegate them."
+2. "Delegate only when work is parallelizable across independent files/subtasks, spans many files, or needs a large/isolated context."
+3. "Every `Agent` spawn passes `model` explicitly."
+4. "Default `haiku` for mechanical file-edit/shell work; `sonnet` for anything requiring judgment; `opus` only for genuinely hard reasoning."
+
+**Orchestrator's behavior:**
+1. **Single-file / ≤2-tool ops:** Not violated. The orchestrator doesn't delegate write a single vault note or read a single file. Delegation is reserved for multi-sourced research and judgment work.
+2. **Parallelizable / spans-files:** Respected. Agents 1-3 are independent research; agents 4-7 are parallel judgment.
+3. **Explicit model parameter:** **Orchestrator does this correctly** — all 10 spawns include `model: "sonnet"`. However, the Agent framework ignores the parameter and downgrades to Haiku. The orchestrator cannot be faulted for this; it's a framework/plugin issue.
+4. **Model selection per judgment:** **Orchestrator intends this correctly** — research and perspectives are judgment work, so Sonnet is requested. But the framework downgrade means actual models are all Haiku.
+
+**Assessment:** Orchestrator follows the spirit and letter of ORCHESTRATION.md. The failure (all Haiku instead of requested Sonnet) is a framework failure, not an orchestrator failure. However, the orchestrator could mitigate this if it detected the mismatch — it does not (no error handling for resolved model != requested model).
+
+**Refinement:** The user's explicit delegation request ("Dispatch subagents...") overrides the ORCHESTRATION.md preference for direct work. This is correct per the policy's intent: "Delegate only when..." — the user's explicit need qualifies.
+
+---
+
+## Q7: Is the orchestrator requesting/receiving back only the context it needs?
+
+**Verdict: PASS — Concise communication, no full-context-dump pattern**
+
+**How agents report:**
+- Agents complete asynchronously in the background (line 22: "The agent is working in the background. You will be notified automatically when it completes.")
+- Task notifications include agent output via the tool result (exact channel uncertain without parsing task output files)
+- Orchestrator receives task notifications as user lines, summarizes findings, and moves on
+
+**Orchestrator's synthesis pattern:**
+- Lines 77-78: Orchestrator reads the Storybloq/tools/agent-native research and synthesizes key insights (adds recurrence/lifecycle concepts to the proposal) before perspectives
+- No evidence of the orchestrator reading 100KB of raw agent output; synthesis is rapid and concise
+
+**Final synthesis:**
+- Line 201 shows orchestrator text output after perspectives complete, synthesizing four perspectives into recommendations
+- Tight, focused synthesis; no dumping of raw agent transcripts
+
+**Assessment:** The communication pattern is clean. Agents produce output, orchestrator synthesizes selectively (line 77-78, 201) and moves forward. No evidence of "full context dump" or reading files it doesn't need.
+
+---
+
+## Summary of Issues
+
+### Critical Issue
+**Model parameter not respected by Agent framework** (lines 22, 24, 26, etc. — all resolvedModel = haiku despite model: sonnet request)
+- Blocks orchestrator from implementing cost-quality policy
+- Not an orchestrator error; framework/plugin failure
+- Orchestrator makes correct parameter choices; execution layer fails
+
+### Secondary Issue
+**Agents 8-10 appear reactive rather than pre-planned**
+- Could indicate lack of forward planning
+- May be acceptable if reactive spawning is more context-efficient (each agent sees latest synthesis)
+- Weak signal without seeing the orchestrator's internal reasoning
+
+### N/A Issues
+- File reading: minimal and appropriate
+- Context sharing: focused, not bloated
+- Model selection intent: correct (all judgment work gets Sonnet request)
+- ORCHESTRATION.md compliance: followed except for framework failure
+
+---
+
+## Recommendations
+
+1. **Immediate:** Debug why Agent framework downgrades model parameter. Check Claude Code plugin / Agent framework configuration.
+
+2. **Mitigation:** If framework issue is not fixable, orchestrator should handle resolved model != requested model (log warning, adjust expectations, or re-delegate to a higher tier).
+
+3. **Optional improvement:** Pre-plan agents 8-10 upfront rather than spawning reactively, if the orchestrator can predict that deep-dives will be needed. Current reactive approach is defensible but less efficient than pre-batched delegation.
+
+4. **Documentation:** Add a post-delegation checkpoint after agents 1-7 complete to confirm the proposal synthesis before spawning agents 4-7, making the sequential-reactive pattern explicit in the transcript (currently implied, not stated).
+
+---
+
+## Checklist (7 Questions)
+
+| # | Question | Verdict | Key Evidence Line(s) |
+|---|----------|---------|----------------------|
+| 1 | Subagents called when should be? | PASS | 4, 20, 21 (user request + orchestrator accepts) |
+| 2 | Correct model chosen? | FAIL | 21/22, 23/24, 25/26 (sonnet requested, haiku resolved) |
+| 3 | Tasks grouped efficiently? | MIXED | Lines 21-25 (good batch), 79-85 (reactive) |
+| 4 | Avoiding unnecessary reads? | PASS | Factsheet: 0 reads pre-spawn-7 |
+| 5 | Too much context shared? | PASS | Prompts ~1-3KB each, instruction-forward not context-forward |
+| 6 | Following ORCHESTRATION.md? | MIXED | Orchestrator complies; framework fails (model param ignored) |
+| 7 | Getting back only needed context? | PASS | Concise synthesis, no full-dump pattern |

docs/orchestration-audit/auditor-reports/S10-report.md

@@ -0,0 +1,160 @@
+# Orchestration Audit Report: S10-f6a224d0
+
+**Session:** f6a224d0-ddc2-488c-b0cc-728fdf21cc08  
+**Project:** /home/jared/systems-admin  
+**Date:** 2026-07-01T12:37:51.119Z → 2026-07-01T14:02:57.530Z  
+**Duration:** ~85 minutes  
+**Turns:** 53 assistant, 8 human prompts  
+
+## Summary of Session Work
+
+User requested: Create an orchestration plugin in ~/dev/cc-plugins to version-control and back up the orchestration rules (previously a pilot section in CLAUDE.md).
+
+**Tool usage profile:**
+- Bash: 12 calls (exploration, directory creation, testing)
+- Write: 5 calls (4 plugin files + 1 scratchpad)
+- Read: 4 calls (requirements, CLAUDE.md, marketplace.json, settings.json)
+- Edit: 4 calls (revert CLAUDE.md, register in marketplace.json, wire in settings.json)
+
+**Work segmented into phases:**
+1. Read requirements and existing pilot section (Lines 28, 50)
+2. Explore existing plugin structure via find/grep (Lines 35, 82–115)
+3. Create directory tree (Line 119)
+4. Write 4 plugin files (Lines 125, 127, 135, 137)
+5. Register and integrate: Edit marketplace.json + settings.json (Lines 146, 157)
+6. Test the hook (Line 161+)
+
+---
+
+## Seven-Question Rubric Audit
+
+### 1. Are subagents getting called when they should be?
+
+**Verdict: PASS**
+
+**Evidence:** Zero Agent spawns (fact-sheet confirms, jsonl lines 1–183 contain no `Agent` tool_use). Flagged candidate work: Write ×4 spanning 4 distinct plugin files (lines 125, 127, 135, 137).
+
+**Reasoning:** The orchestration policy states "Delegate only when work is parallelizable across independent files/subtasks, spans many files, or needs a large/isolated context." The 4 writes form a single coherent plugin with internal dependencies:
+- `inject.py` reads the contents of `ORCHESTRATION.md` at runtime
+- `hooks.json` references the path and filename of `inject.py`
+- Both depend on `plugin.json` and `ORCHESTRATION.md` for semantic meaning
+
+These are **not independent** — they are coordinated parts of one unit. The work was also integrated with edits to `marketplace.json` and `settings.json` for proper registration (lines 146, 157), which could not have been parallelized with the plugin creation. The single-session orchestration correctly kept this together and did not delegate.
+
+---
+
+### 2. Is the correct model chosen per subagent — highest reasonable quality at lowest cost?
+
+**Verdict: N/A**
+
+**Evidence:** No subagents spawned, so no model selection applies.
+
+---
+
+### 3. Is the orchestrator planning/grouping tasks to maximize efficient context-window use?
+
+**Verdict: PASS**
+
+**Evidence:**
+- Phase 1 (exploration): Lines 28, 35, 50, 82–115. Bash grep commands referenced existing plugins (doc-hygiene, memory) and examined hook patterns (`additionalContext`, `hookSpecificOutput` — lines 109, 112, 115) before writing any code. This gathers reference context upfront.
+- Phase 2 (creation): Directory structure created (line 119) before writes, making room in context for coordinated writes (lines 125–137).
+- Phase 3 (integration): Reads marketplace.json and settings.json (lines 143, 149) *after* writes but *before* edits, ensuring the orchestrator understands what needs to be registered.
+- Phase 4 (verification): Bash test of the hook (line 161) to confirm functionality.
+
+Total 25 tool calls (12 Bash + 5 Write + 4 Read + 4 Edit) is reasonable for a full plugin creation task; phasing avoided context bloat.
+
+---
+
+### 4. Is the orchestrator avoiding reading files it does NOT need (that the subagent would read anyway)?
+
+**Verdict: PASS**
+
+**Evidence:**
+- Read 1 (line 28): `/home/jared/systems-admin/docs/orchestration-audit-prd.md` — Requirements doc, essential to understand scope.
+- Read 2 (line 50): `/home/jared/systems-admin/CLAUDE.md` — Source of the pilot section to be extracted; necessary.
+- Read 3 (line 143): `/home/jared/dev/cc-plugins/.claude-plugin/marketplace.json` — Needed to understand how to register the new plugin in the local marketplace. Not redundant with prior exploration (prior Bash calls examined its existence but not its structure).
+- Read 4 (line 149): `~/.claude/settings.json` — Needed to understand hook wiring pattern before editing it.
+
+No extraneous reads. Bash exploration (find, grep on existing plugins) was efficient discovery rather than full-file reads, and it directly informed the plugin design (grep for `hookSpecificOutput` — line 115 — led directly to the hook script structure in inject.py).
+
+---
+
+### 5. Is the orchestrator sharing too much context with subagents (filling their windows / clouding judgment)?
+
+**Verdict: N/A**
+
+**Evidence:** No subagents spawned; question does not apply.
+
+---
+
+### 6. Is the orchestrator even following the ORCHESTRATION.md instructions?
+
+**Verdict: PASS**
+
+**Evidence:** Policy from session context:
+> "Do single-file, ≤2-tool-call ops directly. Don't delegate them. Delegate only when work is parallelizable across independent files/subtasks, spans many files, or needs a large/isolated context."
+
+Orchestrator invoked no Agent spawns. Boundary condition: 4 Writes (exceeds ≤2 threshold). However:
+- The 4 writes are **not parallelizable** (they have internal dependencies as described in Question 1).
+- They **span many files**, but those files form one logical unit with coordinated edits and testing in the same session.
+- The orchestrator correctly identified that "spans many files" applies only when files are independent subtasks. These are not. Keeping them in a single session is appropriate.
+
+Additionally, the Bash exploration phase (lines 82–115) is within the policy: "A short orienting Read before delegating is fine when the target file/path is uncertain. Don't delegate the orienting step itself." The orchestrator's exploration of existing plugin examples informed the design without delegating.
+
+---
+
+### 7. Is the orchestrator requesting/receiving back only the context it needs, rather than a full context dump from the subagent?
+
+**Verdict: N/A**
+
+**Evidence:** No subagents spawned; question does not apply.
+
+---
+
+## Specific Findings on the Flagged Candidate (Write ×4, lines 125–137)
+
+**Flag origin:** Heuristic flagged runs of ≥4 same-tool calls across distinct targets.
+
+**Assessment:** This is a **false positive** in the delegation-candidate heuristic. The criterion "parallelizable across independent files/subtasks" does not apply here. The orchestrator correctly recognized that:
+
+1. The plugin files form a coherent unit, not independent subtasks.
+2. Integration work (marketplace, settings edits) depends on all plugin files being written first.
+3. The entire sequence belongs in a single session for semantic coherence and testability.
+
+A subagent spawn would have added friction (returning a summary of 4 created files + hook integration details) without saving context, since the orchestrator needs to verify the hook integration and test it (line 161). **No delegation was needed.**
+
+---
+
+## Other Observations
+
+### Work Quality
+- Directory structure created before writes (line 119), preventing path errors.
+- Hook syntax validated by running the hook in the working environment (line 161, `CLAUDE_PLUGIN_ROOT` environment variable set).
+- Permissions set (line 140: `chmod +x`).
+- No retry loops or error recovery needed; indicates careful design before execution.
+
+### Context Efficiency
+- The Read/Bash exploration phase (lines 28–115) took ~16 tool calls to gather context.
+- The Write/Edit/integration phase (lines 125–161) was tightly executed with no back-and-forth.
+- Model was claude-sonnet-5 throughout (appropriate for the mix of exploration + structured file creation).
+
+### No Red Flags
+- No repeated reads of the same file (would indicate missing context or rework).
+- No abandoned edits or reverted changes.
+- Tool use pattern is monotonic: exploration → creation → integration → testing.
+
+---
+
+## Verdict Summary
+
+| Question | Verdict | Confidence |
+|----------|---------|------------|
+| 1. Subagents when needed? | PASS | High |
+| 2. Model choice per subagent? | N/A | — |
+| 3. Task planning & grouping? | PASS | High |
+| 4. Avoiding unnecessary reads? | PASS | High |
+| 5. Context spillage to subagents? | N/A | — |
+| 6. Following ORCHESTRATION.md? | PASS | High |
+| 7. Compact subagent returns? | N/A | — |
+
+**Overall: PASS.** The orchestrator correctly resisted delegating a multi-file write sequence that appears to exceed the ≤2-tool-call threshold, because the files are not independent and form one logical unit. Exploration was well-organized, tool calls were lean, and integration was tight. No delegation was warranted, and none was invoked.

docs/orchestration-audit/auditor-reports/S2-report.md

@@ -0,0 +1,213 @@
+# Orchestration Audit Report: Session d68aab1b
+
+**Session**: d68aab1b-3e3d-467c-9b9c-70b89e2576b0  
+**Date**: 2026-07-06 14:44–17:58 UTC  
+**Project**: `/home/jared/dev/cc-os`  
+**Duration**: ~3 hours  
+**Assistant**: claude-fable-5  
+**Agent spawns**: 8  
+
+---
+
+## Question 1: Are subagents getting called when they should be?
+
+**Verdict**: MIXED
+
+**Evidence**:
+- Lines 28, 30, 72, 74, 76, 296, 506, 628 each spawn an Agent.
+- Candidate missed delegation flagged: Read x5 (lines 389–405) reading 5 distinct eval scenario files sequentially without an intervening Agent spawn.
+- **Candidate missed delegation flagged**: Write x4 (lines 611–624) writing 4 plan documents sequentially without an intervening Agent spawn.
+
+**Analysis**:
+The orchestrator did delegate 8 times, but the Read x5 and Write x4 sections are valid candidates for delegation judgment because:
+- **Read x5** spans 5 distinct files (P1-L1, P2-L2, P3-L3 scenarios + judge-rubric + check script) in the Eval C harness. These appear to be independent reference reads for understanding the eval, not sequential-dependent work (e.g., not "read A, analyze, decide on B, read B").
+- **Write x4** (lines 611–624) writes 4 separate plan documents (`2026-07-06-plugin-evals-overview.md`, `ws1-orchestration-audit.md`, `ws2-os-vault-write-eval.md`, `ws3-status-convention-plugin.md`). These are tightly coupled by design (they reference each other; ws1–3 are explicit in the overview), making sequential authorship defensible but potentially over-orchestrator-load.
+
+**Missed delegation trigger candidate**:  
+When orchestrator has 5+ independent file reads on **different logical domains** (not a single multi-part document), delegate the read batch to a general-purpose agent with `run_in_background: true` if other work can proceed in parallel. Similarly, when writing 4+ loosely coupled documents with explicit cross-references, consider whether an agent authoring all four together saves context-switching and ensures consistency better than orchestrator authorship.
+
+---
+
+## Question 2: Is the correct model chosen per subagent — highest reasonable quality at lowest cost?
+
+**Verdict**: FAIL
+
+**Evidence**:
+All 8 agents resolved to `claude-haiku-4-5-20251001` despite explicit model parameter mismatches:
+- Agents 1–2 (lines 28, 30): requested `model: "sonnet"` → resolved haiku
+- Agent 3 (line 72): requested `model: "sonnet"` → resolved haiku  
+- Agent 4 (line 74): requested `model: "opus"` → resolved haiku
+- Agent 5 (line 76): requested `model: "sonnet"` → resolved haiku
+- Agents 6–8 (lines 296, 506, 628): requested `model: "haiku"` → resolved haiku (CORRECT, but only by accident)
+
+**Analysis**:
+This is a critical orchestration failure. The orchestrator explicitly passed a `model` parameter 8 times and the Agent tool **completely ignored it**, always resolving to haiku. This is not an orchestrator judgment error (the orchestrator made correct requests); it's a tool-system error upstream (likely the Agent tool definition or the model fallback logic).
+
+The cost impact is mild (haiku is cheapest), but the quality impact is severe: Agents 1–2 (perspectives agents doing epistemic skepticism, which requires sonnet-tier reasoning) and Agent 4 (opus task building a complex eval harness) were systematically downgraded to haiku-only reasoning.
+
+**Missed delegation trigger / missed orchestration signal**:  
+Orchestrator should have detected that the `resolvedModel` field in toolUseResult differed from the requested `model` parameter and either (a) raised an error, (b) acknowledged the downgrade to the user, or (c) re-spawned with explicit routing. None of this happened. This is not orchestrator failure but a missing visibility/accountability layer.
+
+---
+
+## Question 3: Is the orchestrator planning/grouping tasks to maximize efficient context-window use?
+
+**Verdict**: PASS (with caveats)
+
+**Evidence**:
+- Agent spawns 1–5 are clustered (lines 28–76), each conceptually independent, allowing parallel execution (confirmed: all marked `run_in_background: true` implicitly, async_launched).
+- Agents 6–8 (git commits) are spawned sequentially (lines 296, 506, 628) with explicit tool dependencies (each runs after prior work completes), which is defensible.
+- After-spawn-5 segment (37 tool calls) is the heavy work: Reads, Edits, Writes. This was done after agents returned, not during their background execution, indicating the orchestrator waited for results before proceeding.
+
+**Analysis**:
+The grouping is reasonable: epistemic/measurement agents (1–2), task-execution agents (3–5) are fanned out; execution-chain agents (6–8 commits) are sequenced. No evidence of poor context-window planning (e.g., reading a 50MB file then spawning an agent with it in context).
+
+**Caveat**: After-spawn-5's 37 tool calls (41.5KB read) all happen in sequence without delegating any to the agents that just finished. If the Read x5 and Write x4 could have been delegated, this segment could have shed context load off the orchestrator.
+
+---
+
+## Question 4: Is the orchestrator avoiding reading files it does NOT need (that the subagent would read anyway)?
+
+**Verdict**: PASS
+
+**Evidence**:
+- Orchestrator never reads the same file a spawned agent was instructed to read. The 5 Reads at lines 389–405 are in a context where no active agent is reading them (Agents 1–5 already completed; Agents 6–8 are commits with no Read dependency).
+- No evidence of the orchestrator pre-reading files "just in case" an agent might need them.
+- Total bytes read by orchestrator: 41,558 (after-spawn-5) + 61,527 (after-spawn-6) + 24,354 (after-spawn-7) + 563 (after-spawn-8) = ~127.7 KB over a 3-hour session. This is light.
+
+**Analysis**: The orchestrator is not wastefully pre-reading. The flagged Read x5 is the orchestrator needing to understand eval scenarios, not an agent-delegation miss on *this* question.
+
+---
+
+## Question 5: Is the orchestrator sharing too much context with subagents (filling their windows / clouding judgment)?
+
+**Verdict**: PASS
+
+**Evidence**:
+- Agent prompts (lines 28, 30, 72, 74, 76) are terse: 2.5–6.7 KB each. The session has ~70 KB used cumulative before any agent spawn.
+- No evidence of the orchestrator dumping transcripts, full codebases, or search results into agent prompts wholesale.
+
+**Analysis**: Prompts are focused. No over-context problem detected.
+
+---
+
+## Question 6: Is the orchestrator even following the ORCHESTRATION.md instructions?
+
+**Verdict**: FAIL
+
+**Evidence**:
+The global ORCHESTRATION.md policy (injected at SessionStart) states:
+> Do single-file, ≤2-tool-call ops directly. Don't delegate them. Delegate only when work is parallelizable across independent files/subtasks, spans many files, or needs a large/isolated context.
+
+The orchestrator DID delegate appropriately on agents 1–5 (complex, multi-step, reasoning-heavy). BUT:
+- Lines 28, 30, 74: Agent requests explicitly passed `model: "sonnet"` and `model: "opus"` parameters, and those were **systematically ignored** (all resolved to haiku). The instructions say "Every Agent spawn passes model explicitly. Default haiku for mechanical file-edit/shell work; sonnet for anything requiring judgment; opus only for genuinely hard reasoning." The orchestrator was right to request sonnet/opus, but the tool did not respect it.
+- The orchestrator did NOT acknowledge or escalate this model-resolution failure, violating the implicit "verify your agent actually got the model you requested" accountability.
+
+**Specific failed instruction**:  
+> Every `Agent` spawn passes `model` explicitly.
+
+The orchestrator did pass it; the tool ignored it. This is a tool-system error, not an orchestrator error, but the orchestrator should have caught it.
+
+---
+
+## Question 7: Is the orchestrator requesting/receiving back only the context it needs, rather than a full context dump from the subagent?
+
+**Verdict**: PASS
+
+**Evidence**:
+- Agent spawn prompts are specific: "Challenge eval validity claims" (line 28), "Run 3-rep haiku W3 check" (line 72), "Build Eval C..." (line 74), etc.
+- No evidence of the orchestrator requesting "summarize everything you did" or consuming 50+ KB of agent output. The fact-sheet shows result sizes of 2.9–12.9 KB, which are reasonable summaries.
+
+**Analysis**: The orchestrator is precise in its requests and does not bloat context with full result dumps.
+
+---
+
+## Summary of Verdicts
+
+| Q | Verdict | Issue |
+|---|---------|-------|
+| 1 | MIXED | Read x5 and Write x4 are delegation candidates; judgment needed on sequential dependency |
+| 2 | FAIL | All 8 agents requested sonnet/opus but resolved to haiku; tool ignored `model` parameter |
+| 3 | PASS | Grouping is sensible; agents 1–5 fanned out, 6–8 sequenced correctly |
+| 4 | PASS | No wasteful pre-reading or file duplication |
+| 5 | PASS | Agent prompts are focused, not over-contextualized |
+| 6 | FAIL | Model parameter requests were ignored; orchestrator did not escalate the failure |
+| 7 | PASS | Agent outputs were summaries, not full context dumps |
+
+---
+
+## Candidate Missed Delegations (Auditor Judgment Required)
+
+### Delegation Candidate 1: Read x5 (lines 389–405)
+
+**Files read**:
+- `/home/jared/dev/cc-os/plugins/os-adr/eval-c/scenarios/P1-L1-execution.md`
+- `/home/jared/dev/cc-os/plugins/os-adr/eval-c/scenarios/P2-L2-execution.md`
+- `/home/jared/dev/cc-os/plugins/os-adr/eval-c/scenarios/P3-L3-execution.md`
+- `/home/jared/dev/cc-os/plugins/os-adr/eval-c/judge-rubric.md`
+- `/home/jared/dev/cc-os/plugins/os-adr/eval-c/bin/check`
+
+**Context**: Orchestrator is reviewing the Eval C harness after Agent 5 (eval-building) completed. These reads appear to be for understanding the structure and validating/analyzing the built scenarios.
+
+**Sequential dependency risk**: LOW. These are independent reference reads, not "read A → decide → read B" chains. They could be batched to a single agent with `run_in_background: true`.
+
+**Verdict**: Legitimate orchestrator work, but could be delegated if the reads are analytical (not decisional) and the orchestrator is not actively synthesizing between them.
+
+### Delegation Candidate 2: Write x4 (lines 611–624)
+
+**Files written**:
+- `/home/jared/dev/cc-os/docs/plans/2026-07-06-plugin-evals-overview.md` (main overview)
+- `/home/jared/dev/cc-os/docs/plans/ws1-orchestration-audit.md` (WS1)
+- `/home/jared/dev/cc-os/docs/plans/ws2-os-vault-write-eval.md` (WS2)
+- `/home/jared/dev/cc-os/docs/plans/ws3-status-convention-plugin.md` (WS3)
+
+**Context**: Orchestrator is authoring a 4-document plan set. The overview references all three workstreams; each workstream doc is independent but tightly coupled.
+
+**Sequential dependency**: HIGH. The overview must exist (or be drafted) before the workstream docs can be written correctly, OR all four must be authored together to ensure cross-references are accurate.
+
+**Verdict**: Defensibly orchestrator work due to coupling, but authoring all four as a delegated batch to a single agent might improve consistency and save context-switching. The coupling suggests an agent would be clearer on intent than sequential orchestrator writes.
+
+---
+
+## Failure-Mode Clusters (from Real Evidence)
+
+### Cluster A: Model Parameter Ignored
+
+**Symptom**: Orchestrator explicitly passes `model: "sonnet"` or `model: "opus"` to Agent, but `resolvedModel` in the response is always `haiku`.
+
+**Severity**: HIGH
+
+**Trigger**: Any Agent spawn with a non-haiku model parameter.
+
+**Expected behavior**: `resolvedModel` should match `model` param, or Agent should raise an error.
+
+**Actual behavior**: Silent downgrade to haiku.
+
+**Candidate eval scenario**: Spawn agents with mixed models (sonnet, opus, haiku) and verify resolvedModel matches or error is raised.
+
+---
+
+## Recommendations
+
+1. **Investigate model parameter loss**: The consistent 8/8 downgrade to haiku suggests a systematic issue in the Agent tool or model resolution logic, not orchestrator error. Recommend diagnosing why the `model` param is being ignored.
+
+2. **Add model-parameter validation**: Orchestrator should log a warning or error if `resolvedModel != model` param (if one was provided).
+
+3. **Read batch delegation**: Consider delegating Read x5 to a background agent with explicit scope ("read and summarize the structure of these 5 eval-c files").
+
+4. **Write batch delegation**: Consider authoring multi-document plan sets via a single agent to ensure cross-reference consistency. This is less about cost and more about coherence.
+
+5. **Model selection refinement**: Even if the parameter-passing issue is fixed, confirm that sonnet is the right choice for perspectives agents and opus is justified for eval-harness building. The current choices seem sound, but the downgrade hides the opportunity to validate them.
+
+---
+
+## Session Metadata
+
+- **Main loop model**: claude-fable-5 (orchestrator)
+- **Agent models**: all resolved to haiku (intended: mixed)
+- **Total agent spawns**: 8
+- **Parallel agents**: 5 (1–5, concurrent, bg)
+- **Sequential agents**: 3 (6–8, git commits)
+- **Total orchestrator tool calls**: 85 (excluding sidechain)
+- **Total bytes read by orchestrator**: 127.7 KB
+- **Candidate issues**: Model downgrade (Q2, Q6); delegation judgment needed (Q1)

docs/orchestration-audit/auditor-reports/S3-report.md

@@ -0,0 +1,191 @@
+# Session S3 Orchestration Audit Report
+**Session:** 9f45afcc-75ff-4b7a-8f3d-008aa0d599af  
+**Date:** 2026-07-04  
+**Duration:** 2h 35m  
+**Transcript lines:** 850 (28 sidechain, 5 spawns, 255 assistant turns)
+
+---
+
+## SEVEN-QUESTION AUDIT RUBRIC
+
+### 1. Are subagents getting called when they should be?
+
+**Verdict:** MIXED — Delegation is reasonable but spawn #1 might have been avoidable
+
+**Evidence:**
+- Line 124 (spawn 1): Plugin cache refresh. Prompt specifies 4 steps: determine refresh method, execute `claude plugin` commands, verify caches, refresh cc-plugins marketplace. This is multi-step but mechanical — deterministic shell commands, no judgment. The orchestrator did 19 tool calls in pre-span-1 (14 Bash, 4 Read, 1 Edit) to investigate the problem (lines ~5-120: searched for files, checked plugin state, read naming conventions). That investigation work could have been delegated as part of the same spawn rather than consuming 19 sequential tool calls in the main context.
+
+- Lines 133, 171, 603, 605 (spawns 2–5): Eval grid runs and wording triage. These require running deterministic harnesses repeatedly or evaluating behavior hypothetically — work that spans multiple independent runs and benefits from isolated context. Delegation appropriate.
+
+- Fact-sheet notes "None flagged by heuristic" for runs of ≥4 same-tool calls that should have been delegated. However, the 19-call pre-span-1 segment (Bash-heavy, grep/ls/find) was necessary orientation per ORCHESTRATION.md guidance: "A short orienting Read before delegating is fine when the target file/path is uncertain. Don't delegate the orienting step itself." The Reads (4) were all strategic — to understand plugin naming conventions, review prior eval results, check CLAUDE.md design — and the Bash calls were discovery-only, not modifying.
+
+**Assessment:** Delegation thresholds met for spawns 2–5 (eval work needs isolation). Spawn 1 (cache refresh) is borderline: it's multi-step and beneficial to isolate, but purely mechanical and could have been done directly by the orchestrator. The pre-spawn work (investigation) was correctly kept in-session per policy.
+
+---
+
+### 2. Is the correct model chosen per subagent?
+
+**Verdict:** FAIL — All 5 spawns suffered model degradation
+
+**Evidence:**
+- Line 124: Model param `"sonnet"` → resolved `claude-haiku-4-5-20251001` (Haiku)
+- Line 133: Model param `"sonnet"` → resolved `claude-haiku-4-5-20251001` (Haiku)
+- Line 171: Model param `"sonnet"` → resolved `claude-haiku-4-5-20251001` (Haiku)
+- Line 603: Model param `"sonnet"` → resolved `claude-haiku-4-5-20251001` (Haiku)
+- Line 605: Model param `"haiku"` → resolved `claude-haiku-4-5-20251001` (Haiku) ✓
+
+**Analysis:**
+All 5 Agent tool calls explicitly set `model: "sonnet"` (lines 124, 133, 171, 603) or `model: "haiku"` (line 605), but 4 of the 5 resolved to `haiku` instead. This is either:
+1. A platform issue where `model` parameter is not honored, or
+2. Model unavailability/fallback behavior
+
+The orchestration policy states: "Every Agent spawn passes model explicitly. Default haiku for mechanical file-edit/shell work; sonnet for anything requiring judgment."
+
+**Quality impact:**
+- Spawn 1 (cache refresh): Mostly mechanical (shell commands). Haiku adequate, possibly optimal.
+- Spawn 2 (re-run Eval B grid): Requires running a deterministic harness. Haiku adequate.
+- Spawn 3 (resume eval grid): Continuation of eval. Haiku adequate for procedural execution.
+- **Spawn 4 (triage wording on sonnet):** Explicitly judgment work — "evaluate how you would behave in a hypothetical coding session… answer from judgment alone." This requires reasoning about whether to propose an ADR. Haiku is below the required capability level. **REAL MISS.**
+- Spawn 5 (triage wording on haiku): Correctly specified and received haiku. Appropriate.
+
+Spawn 4's prompt (line 603) begins: "You are evaluating how you would behave in a hypothetical coding session. Do NOT use any tools — answer from judgment alone." This is explicitly a judgment task. The fact that it got haiku instead of sonnet is a capability downgrade that likely impacted the evaluation's fidelity.
+
+---
+
+### 3. Is the orchestrator planning/grouping tasks to maximize efficient context-window use?
+
+**Verdict:** MIXED — Reasonable grouping but significant sequential work not delegated
+
+**Evidence:**
+- 255 assistant turns for 5 spawns = ~50 lines per spawn (including setup/handling)
+- After-spawn-3: 48 tool calls (ToolSearch 2, SendMessage 1, Bash 22, Edit 12, Skill 2, Read 5, Write 3, TaskStop 1)
+- After-spawn-5: 43 tool calls (Write 4, Bash 27, Edit 11, Read 1)
+
+The after-spawn-3 and after-spawn-5 segments show the orchestrator managing the eval process directly with many tool calls. This is work that _could_ have been grouped into a single eval-management delegation, but instead was split:
+- Spawn 2 (line 133): "Re-run Eval B grid post-fix" → 0 orchestrator calls afterward
+- Spawn 3 (line 171): "Resume eval grid agent" → 48 orchestrator calls managing results
+- Spawn 4 (line 603): "Triage wording on sonnet" → 0 calls
+- Spawn 5 (line 605): "Triage wording on haiku" → 43 orchestrator calls writing/processing results
+
+The orchestrator could have delegated the result-processing and writing work (e.g., "Run evals 4-5, process results, write summary") as part of the agent prompts rather than managing it in-session. This would have freed up the main context for other work.
+
+**Assessment:** Grouping is functional but leaves optimization on the table. Sequential dependent work (eval → triage → result processing) could have been bundled.
+
+---
+
+### 4. Is the orchestrator avoiding reading files it does NOT need?
+
+**Verdict:** PASS — Reads are minimal and strategic
+
+**Evidence:**
+- Pre-span-1 Reads (lines ~5-120):
+  1. `/home/jared/Documents/SecondBrain/cc-os-plugin-skill-naming-convention.md` — necessary to understand naming rules before updating CLAUDE.md
+  2. `/home/jared/Documents/SecondBrain/2026-07-03-os-adr-eval-b-grid-results-and-observations.md` — necessary to understand prior eval baseline before re-running
+  3. `/home/jared/dev/cc-os/CLAUDE.md` (read twice) — necessary to understand design before updating plugin cache info and to check current state
+
+- After-span-3: Read 5 files (not listed individually, but were part of eval result handling)
+- After-span-5: Read 1 file
+
+Total reads: ~4 strategic pre-delegation, then tactical reads during result handling. No superfluous reads evident. ORCHESTRATION.md guidance ("A short orienting Read before delegating is fine") is followed.
+
+---
+
+### 5. Is the orchestrator sharing too much context with subagents?
+
+**Verdict:** PASS — Prompts are focused and appropriately sized
+
+**Evidence:**
+- Spawn 1 (line 124): 4,849 chars — detailed context about the cache problem, specific files missing, clear multi-step task definition
+- Spawn 2 (line 133): 2,177 chars — context about the cache fix, baseline results to compare, explicit procedure reference
+- Spawn 3 (line 171): 49 chars — minimal ("Resume eval grid agent"), relying on prior task continuation via SendMessage
+- Spawn 4 (line 603): 2,193 chars — a complete hypothetical scenario with task definition
+- Spawn 5 (line 605): 2,193 chars — same length as spawn 4 (paired triage tasks)
+
+The prompts are task-focused. Spawn 3's minimal prompt (49 chars) is appropriate for a SendMessage continuation rather than a fresh spawn. Spawns 4-5 are large but necessary: they define the hypothetical scenario and judgment task in full.
+
+No evidence of context overload or extraneous background.
+
+---
+
+### 6. Is the orchestrator following ORCHESTRATION.md instructions?
+
+**Verdict:** MIXED — Mostly compliant but borderline on single-file/≤2-tool rule
+
+**Evidence:**
+ORCHESTRATION.md policy:
+> "Do single-file, ≤2-tool-call ops directly. Don't delegate them. Delegate only when work is parallelizable across independent files/subtasks, spans many files, or needs a large/isolated context."
+
+**Analysis:**
+- **Spawn 1 (plugin cache refresh):** Multi-step work across 4 plugins (os-adr, os-doc-hygiene, os-vault, os-orchestration, plus cc-plugins marketplace). Spans multiple files. Delegated. ✓ Compliant.
+- **Spawn 2-5 (eval grids):** Requires running headless harnesses repeatedly. Benefits from isolated context. Delegated. ✓ Compliant.
+- **After-spawn-3/5 work:** The orchestrator executed 48 + 43 = 91 tool calls _directly_, not delegated, to manage eval results, write reports, and update documentation. This includes:
+  - Line 172-300 region: multiple Bash (running eval binaries), Edit (patching files), Write (creating outputs), Skill (invoking tools), ToolSearch (finding contexts)
+  - The work is sequential-dependent (eval runs → check results → triage → write), which is a valid reason not to delegate per policy ("sequential-dependent work is a valid reason not to delegate" per fact-sheet guidance)
+
+**Verdict:** Policy is mostly followed. The pre-span-1 investigation (19 calls) was appropriately kept in-session as orientation. The after-span work (91 calls) is sequential-dependent, so not delegating was correct. However, spawn 1 itself might have been avoidable if the investigation had been folded into the spawn prompt more aggressively — e.g., "investigate AND fix" in one delegation rather than investigate-in-session then delegate-fix.
+
+---
+
+### 7. Is the orchestrator requesting/receiving back only needed context?
+
+**Verdict:** PASS — Results are actionable and used directly
+
+**Evidence:**
+- **Spawn 1 result (line 125):** 3,951 chars. Returned structured summary + file attachment showing CLAUDE.md update. The orchestrator immediately incorporated this (line 126 shows attachment of the edited file). No surplus context; the file edit was the actionable outcome.
+  
+- **Spawn 2 result (line 134, resolved async):** 5,905 chars. Returned full grid results (haiku 0/8, sonnet 5/8) with per-cell analysis. Was this needed? The orchestrator would need the baseline results to inform spawn 3 (resume eval) and spawns 4-5 (triage wording). Appropriate size for the information density.
+
+- **Spawn 3 result (via task notification, line 172):** 878 chars. Minimal, continuation of spawn 2's work. Appropriate.
+
+- **Spawn 4-5 results:** Not directly readable in the transcript (async), but after-span-5 work shows the orchestrator writing outputs and using the results (lines ~603-850 have multiple Write calls with evaluation data).
+
+No evidence of the orchestrator requesting excessive results or receiving data that wasn't used. Results are structured and actionable.
+
+---
+
+## SUMMARY VERDICTS
+
+| Question | Verdict | Key Issue |
+|----------|---------|-----------|
+| 1. Subagents called correctly? | MIXED | Spawn 1 (cache refresh) could have been done directly; pre-span investigation was well-scoped |
+| 2. Correct model chosen? | **FAIL** | All 5 spawns specified sonnet but 4 resolved to haiku; spawn 4 (judgment task) impacted by downgrade |
+| 3. Tasks grouped efficiently? | MIXED | Result handling (91 calls post-eval) could have been delegated as part of eval task |
+| 4. Avoiding unnecessary reads? | PASS | 4 strategic Reads before delegation; tactical reads during handling. Minimal overhead. |
+| 5. Sharing excessive context? | PASS | Prompts are task-focused, appropriately sized (49–4,849 chars). |
+| 6. Following ORCHESTRATION.md? | MIXED | Mostly compliant; sequential work correctly kept in-session; but spawn 1 could have bundled investigation+fix |
+| 7. Requesting only needed context? | PASS | Results are structured, actionable, immediately used. No surplus. |
+
+---
+
+## TOP MISSES WITH LINE REFERENCES
+
+### 1. Model Degradation (CRITICAL)
+**Lines 124, 133, 171, 603:** All Agent calls explicitly set `model: "sonnet"` but resolved to haiku.
+- **Criterion:** When a session specifies a model parameter on Agent spawn, that parameter should be honored or the orchestrator should be notified of fallback.
+- **Trigger:** Implement model-param validation in Agent tool or platform-level fallback alerting.
+
+### 2. Judgment Task Ran on Haiku (REAL IMPACT)
+**Line 603:** Spawn 4 prompt explicitly says "Do NOT use any tools — answer from judgment alone" but requested `sonnet` and got `haiku`.
+- **Criterion:** Judgment-heavy tasks (scenario evaluation, wording triage) require sonnet; the orchestrator specified correctly but did not receive what it asked for.
+- **Trigger:** When specifying a model for judgment work, verify the resolved model matches; escalate if haiku is substituted.
+
+### 3. Post-Eval Work Could Have Been Delegated
+**Lines 172–300 (48 calls), 603–850 (43 calls):** Orchestrator manually managed eval result processing, writing, and triage.
+- **Criterion:** "Triage wording on sonnet/haiku" spawns could have included "process results, write summary, save to file" as part of the agent prompt rather than post-processing in-session.
+- **Trigger:** When a sequence of evals feeds into analysis+writing, bundle the analysis+writing into the final eval spawn or create a post-processing delegation.
+
+### 4. Pre-Delegation Investigation Could Have Been Bundled
+**Lines 5–120 (19 calls):** Orchestrator investigated plugin cache status before delegating the fix.
+- **Criterion:** Investigatepattern + action pattern could be grouped into one "investigate and fix" delegation.
+- **Trigger:** When investigation leads directly to a bounded task, consider bundling: "investigate X, then fix it per these steps" as a single delegation.
+
+---
+
+## CONTAMINATION NOTES
+
+This session (2026-07-04, POST global os-orchestration rollout) was actively working on eval and orchestration tooling (os-adr eval, wording-loop setup). The 255 turns reflect:
+- Plugin cache maintenance (spawn 1)
+- Eval baseline validation (spawns 2-3)
+- Wording triage setup (spawns 4-5)
+
+The low spawn count (5 for 255 turns) is partly intentional: evals run deterministically, their management is procedural. However, model degradation across all 5 spawns is the session's most concerning artifact and suggests either a platform bug or a systematic misunderstanding of model availability.

docs/orchestration-audit/auditor-reports/S4-report.md

@@ -0,0 +1,207 @@
+# Orchestration Audit Report: S4 (b6b07a91)
+
+**Session**: llf-schema, 2026-06-30, feat/harden-schema-verification  
+**Model**: claude-opus-4-8 (main loop)  
+**Agent spawns**: 41 total  
+**Audit scope**: ORCHESTRATION.md compliance + model selection efficiency  
+**Policy under audit**:
+- Do single-file, ≤2-tool ops directly. Delegate only when parallelizable across independent files/subtasks, spans many files, or needs large/isolated context.
+- **Every `Agent` spawn passes `model` explicitly.** Default haiku for mechanical, sonnet for judgment, opus for hard reasoning.
+- A short orienting Read before delegating is fine when target paths are uncertain. Don't delegate the orienting step itself.
+
+---
+
+## Answer to Seven Questions
+
+### 1. Are subagents getting called when they should be?
+
+**Verdict: PASS**
+
+The orchestrator appropriately delegates tasks that span multiple files, require isolated execution (git operations, test runs), or benefit from background parallelization. All spawns are for compound tasks:
+- Multi-step OpenSpec task implementation (spawns 2-3, 6-7, 9-10, 14-15, 19, 33-34)
+- Isolated environment testing/diagnostics (spawns 17, 20-22, 29-30, 38, 40)
+- Background-capable reads (spawns 27-28: Explore agents for wide searches)
+- Deterministic file updates (spawns 4-5, 8, 12, 16, 32: checkbox flips, memory writes with haiku)
+
+No evidence of single-file/≤2-tool ops being delegated when they should have been direct.
+
+**Evidence**: 
+- Line 21: spawn 1 "Gather openspec change context" — could be viewed as optional given that `/opsx:apply` already provided structured task context, but investigating OpenSpec state is defensible as a compound prep task
+- Lines 36-77: spawns 2-7 (task bundle 1.1-3.3 + mark complete) — appropriately split: 1.1 code edit (sonnet), 1.2 golden fixture + profiles (sonnet), 3.1 fixtures (sonnet), 3.2 self-check (sonnet), 3.3 doc (haiku), then mark-complete (haiku)
+
+### 2. Is the correct model chosen per subagent — highest reasonable quality at lowest cost?
+
+**Verdict: FAIL** (policy violation)
+
+**Model param omissions (9 spawns violate explicit-model policy):**
+- Spawns 1, 24, 25, 26, 29, 30, 31: general-purpose without model param → **default to opus** (inappropriate — should be explicitly specified as sonnet or haiku)
+- Spawns 27-28: Explore agents without model param → default to haiku (acceptable for read-only search, though should still be explicit)
+
+**Where models ARE specified, selection is sound:**
+- Haiku (spawns 4, 8, 12, 16, 32, 35, 36, 39, 47): mechanical tasks (checkbox flips, schema sample-verify, post cache purge, memory writes) — cost-optimized
+- Sonnet (spawns 2-3, 6-7, 9-11, 13-15, 17-23, 33-34, 37-38, 40-41): code edits, fixture design, verifier logic, multi-file diagnostics, schema verification — appropriate judgment tier
+- Opus (none explicitly specified, only defaults via omission): the omissions that defaulted to opus are under-specified
+
+**Critical violation**: Policy states "Every `Agent` spawn passes `model` explicitly." Lines 21, 411, 427, 440, 466, 494, 496, 529, 563 (spawns 1, 24-31) omit the parameter:
+
+- **Line 21** (spawn 1, general-purpose, context gathering, 1275 prompt chars, 14627 result chars): **model field absent** from `input` dict. Should be explicit.
+- **Line 411** (spawn 24, "Commit harden-schema-verification", git + task update + commit, 2766 prompt chars): **model field absent**. Resolves to opus; should be explicit (likely sonnet for judgment, not opus default).
+- **Line 427** (spawn 25, "Run release.sh patch to v0.2.5", shell operation, 2126 prompt chars): **model field absent**. Opus default is overkill for script execution; should be haiku.
+- **Line 440** (spawn 26, "Check live service-page WebPage.description", remote HTTP checks + diagnosis, 2311 prompt chars): **model field absent**. Resolves to opus; appropriate outcome but should be explicit.
+- **Lines 466, 494** (spawns 27-28, Explore agents, read-only searches): **no model param**. Explore agents are read-only and lower-cost, so defaulting to haiku is reasonable, but policy requires explicit specification.
+- **Lines 496, 529, 563** (spawns 29-31, general-purpose, 2423–5292 prompt chars): **model field absent**. Default to opus; three more instances of the pattern.
+
+**Evidence of appropriate choices where specified**:
+- Line 38 (spawn 3, sonnet for profiles/verifiers, 4457 prompt chars): Correct tier for multi-file schema design
+- Line 75 (spawn 6, sonnet for golden fixture regeneration, 3684 prompt): Correct tier for oracle-based fixture design
+- Line 155 (spawn 13, sonnet for URL fix + test update, 1011 prompt): Appropriate for code fix with test understanding
+
+### 3. Is the orchestrator planning/grouping tasks to maximize efficient context-window use?
+
+**Verdict: PASS**
+
+The orchestrator groups tasks into logical batches that respect dependencies while allowing parallelization:
+
+**Batch structure**:
+- Pre-batch 1 (spawns 1): context gathering
+- Batch 1 (spawns 2-5): tasks 1.1, 1.2, 3.1, 3.3 + mark complete (spawn 5 is deterministic checkbox-flip, unblocked after parallel executions of spawns 2-4 return)
+- Batch 2 (spawns 6-8): task 1.2 regen, 3.2 self-check, mark complete
+- Batch 3 (spawns 9-12): tasks 4.1, 4.2, 5.1, reconcile + mark complete
+- Batch 4 (spawns 13-16): verify gate, commit, mark complete
+- Batch 5 (spawns 17-23): ADR verification, docblock fix, release run, three live diagnostics
+- Batch 6 (spawns 24-32): commit, release, live check, spec exploration, post_excerpt check, relax + re-release, push, memory write
+- Batch 7 (spawns 33-41): schema verification + finalization
+
+**Evidence**:
+- Spawns 2-4 are independent (code, profiles, doc) and can run in parallel; spawn 5 (mark complete) only needs serial ordering after 2-4
+- Spawns 27-28 (Explore agents) are parallelizable read-only searches, appropriately spawned together without blocking
+- Spawns 33-36 (schema verification for PLD and SDD) are independent and parallelizable
+
+### 4. Is the orchestrator avoiding reading files it does NOT need (that the subagent would read anyway)?
+
+**Verdict: PASS** (with caveat)
+
+**Orchestrator tool profile**: The fact-sheet shows **0 total tool calls** across all 41 segments (post-spawn). The orchestrator never read, wrote, or grepped any files directly — all work was delegated.
+
+This is **not a violation** because:
+1. The orchestrator is executing a structured OpenSpec change, where task definitions come from `/opsx:apply harden-schema-verification` — context is pre-loaded
+2. File paths and task text are explicit in the change spec, so the orchestrator doesn't need to read files to determine what to delegate
+3. Subagents have clear prompts specifying files to read; no redundant pre-delegation reads are necessary
+
+However, a **cautionary note**: Had the task been exploratory rather than spec-driven, the zero reads might indicate missing orientation.
+
+**Evidence**:
+- Lines 1-20 (before spawn 1): OpenSpec skill loads tasks and context; main-loop model has this in cache/additionalContext
+- Each spawn prompt contains explicit file paths (e.g., spawn 2: `/home/jared/dev/llf-schema/openspec/changes/harden-schema-verification/design.md` D-047)
+- No agent result references "file path unclear" or "orchestrator didn't tell me where to look"
+
+### 5. Is the orchestrator sharing too much context with subagents (filling their windows / clouding judgment)?
+
+**Verdict: PASS**
+
+Prompt lengths are appropriately scoped:
+- Mechanical tasks (spawns 4, 8, 12, 16, 32): 436–659 chars (tight, single-purpose)
+- Code edits (spawns 2, 6, 13, 18): 1011–4457 chars (design context + task instructions)
+- Diagnostics (spawns 17, 20-22, 38, 40): 2247–5292 chars (problem statement + investigation steps)
+
+**Evidence**:
+- Spawn 4 (coverage checklist, 512 chars): minimal context, haiku-appropriate
+- Spawn 17 (ADR verification, 3462 chars with 10703 char result): comprehensive but structured; result size justified by need to return full verification matrix
+- Spawn 20 (NJ LegalService diagnosis, 3499 chars): investigation is scoped to a single problem; not bloated with unrelated context
+
+**No instance of context pollution** (e.g., dumping 20 files of unrelated code, full deployment logs, etc.).
+
+### 6. Is the orchestrator following the ORCHESTRATION.md instructions?
+
+**Verdict: FAIL** (explicit policy violation)
+
+The orchestrator **VIOLATES the primary policy requirement**: "Every `Agent` spawn passes `model` explicitly."
+
+**Policy adherence summary**:
+- ✅ Single-file/≤2-tool ops: mostly upheld (no evidence of delegating trivial tasks)
+- ❌ **Explicit model parameter per spawn: VIOLATED (9 spawns)**
+- ✅ Model tier selection (where specified): appropriate (haiku→mechanical, sonnet→judgment, opus reserved)
+- ✅ No delegation of orientation reads: correct (context provided by OpenSpec skill)
+
+**The 9 violations** are not hidden — they are systematic. The pattern suggests possible causes:
+1. **Unintended omission**: The orchestrator (opus) generated spawns but forgot to include `model:` in 9 of them
+2. **Assumption of smart defaults**: The orchestrator may have assumed that omitting `model` on general-purpose agents defaults to sonnet, but it defaults to opus instead
+3. **Model-selection logic change**: If the agent-spawning code changed its behavior between when the policy was written and when this session ran, the orchestrator's output may not match the policy
+
+**Evidence of violation**:
+- Line 21: `"input": {"description": ..., "prompt": ..., "subagent_type": "general-purpose"}` — **no "model" key**
+- Line 36: `"input": {"description": ..., "prompt": ..., "subagent_type": "general-purpose", "model": "sonnet"}` — correct, HAS model key (for comparison)
+- Lines 411, 427, 440: same pattern (missing model key) on spawns 24-26
+- Lines 466, 494: Explore agents also omit model (though haiku default is less problematic)
+- Lines 496, 529, 563: spawns 29-31 continue the pattern
+
+### 7. Is the orchestrator requesting/receiving back only the context it needs, rather than a full context dump from the subagent?
+
+**Verdict: PASS**
+
+Each spawn includes specific instructions for what to return:
+
+**Examples of well-scoped return specs**:
+- Spawn 2 (line 36): "Return: a brief summary of exactly what you changed (with before/after of both edits) and the absolute path"
+- Spawn 4 (line 40): "Return the file location and absolute path"
+- Spawn 13 (line 155): (implicit: the fix is in the file edit, no long report needed)
+- Spawn 17 (line 219): "Return: a verification matrix (task / expected result / actual result / pass/fail) — one row per assertion"
+- Spawn 20 (line 265): "Return: (1) the exact JSON difference … (2) the code path … (3) the classification … (4) a recommended fix direction"
+- Spawn 32 (line 581): "Return: the absolute file path of the created memory file + the memory file's absolute path for indexing" (actually says "just report which files were written")
+
+Result sizes (fact-sheet column 9) show reasonable variance:
+- Small (749 chars): spawns 37-38 (issue filing, brief confirmations)
+- Medium (1200–3500 chars): spawns 2-3, 6-11, 13-15 (code edits, design decisions)
+- Large (5000–10700 chars): spawns 17, 20-22 (detailed diagnostics/comparisons)
+
+**The large results are justified by the task scope.** Spawn 17 (verify ADRs, 10703 result chars) needs to return a full matrix to justify pass/fail. Spawn 20 (diagnose NJ issue, 5041 result chars) needs to return JSON diffs + root-cause analysis.
+
+No evidence of subagents returning full logs or unfiltered dumps when a summary would do.
+
+---
+
+## Candidate Missed Delegations
+
+The fact-sheet heuristic (≥4 same-tool calls, distinct targets, no spawn between) flagged: **None**.
+
+Manual review confirms: The fact-sheet's orchestrator tool profile shows **0 tool calls across all segments**. There are no runs of repeated tool use that could have been parallelized — all work was delegated.
+
+---
+
+## Summary: Key Misses
+
+| Question | Verdict | Top Miss | Line Refs |
+|----------|---------|----------|-----------|
+| 1. Subagent delegation appropriateness | PASS | None notable | N/A |
+| 2. Model choice efficiency | FAIL | 9 spawns omit explicit `model` param, default to opus (inappropriate for some) | 21, 411, 427, 440, 466, 494, 496, 529, 563 |
+| 3. Context-window grouping | PASS | Batching is logical; parallelization is efficient | N/A |
+| 4. Avoid pre-delegation reads | PASS | Caveat: zero reads is appropriate here due to OpenSpec context | N/A |
+| 5. Context bloat to subagents | PASS | Prompt lengths are appropriate; return specs are scoped | N/A |
+| 6. Follow ORCHESTRATION.md | FAIL | **Policy violation: "Every Agent spawn passes model explicitly" — 9 spawns are missing** | 21, 411, 427, 440, 466, 494, 496, 529, 563 |
+| 7. Return-context scope | PASS | Spawns request specific summaries/matrices; no dump-the-whole-log behavior | N/A |
+
+---
+
+## Recommended Evaluation Triggers
+
+**For future orchestration audits**, flag these scenarios:
+
+1. **Model parameter omission**: When `Agent` tool_use appears in transcript WITHOUT a `model` field in the `input` dict, the spawn violates explicit-model policy. Cite the line number and count occurrences.
+
+2. **Orchestrator pre-delegation reads**: When a spawn targets a file path that the orchestrator never read first, check if the task is spec-driven (OpenSpec, explicit task list) — if yes, zero pre-reads is fine; if exploratory, it's a miss.
+
+3. **Spawn complexity vs. model tier**: Spawns with missing model params that resolve to opus — check whether the prompt requires judgment (sonnet) or is mechanical (haiku). Mark as inefficient if wrong.
+
+4. **Batch dependency chains**: Verify that spawn ordering respects task dependencies (e.g., "mark complete" only after preceding tasks in that batch are done). This session passes this check.
+
+5. **Return-context bloat**: Scan spawn prompts for return specs; if a spawn has none (implicit "return everything"), or result_chars >>prompt_chars with no apparent justification, flag for review.
+
+---
+
+## Conclusion
+
+The orchestrator manages **task grouping, parallelization, and scoping well**. The delegation decisions are sound — work is split appropriately between main session and subagents, and batching respects dependency ordering.
+
+However, the **model-parameter omission on 9 spawns is a clear policy violation**. This appears to be a systematic pattern rather than one-off oversight. The spawns that omit the parameter default to opus, which is overkill for some (e.g., spawn 25: "Run release.sh" could be haiku for script execution), while under-specifying the orchestrator's intent for others (e.g., spawn 24: commit workflow with multi-step git logic, which should explicitly request sonnet).
+
+**Immediate action**: Verify that the Agent tool's `model` parameter is always included in future orchestration sessions. This session's violation does not substantially degrade the session's outcome (the tasks completed successfully), but it represents a breach of explicit policy that should be corrected.

docs/orchestration-audit/auditor-reports/S5-report.md

@@ -0,0 +1,178 @@
+# Session Audit Report: S5-aadeb66c
+
+**Session:** aadeb66c-0d64-4d41-b407-83e0209428b4 (llf-schema, 2026-06-29)  
+**Policy under audit:** ORCHESTRATION.md (global default; equivalent text lived in project CLAUDE.md at date of session)  
+**Auditor findings:** 7 rubric questions answered with evidence.
+
+---
+
+## Summary Verdicts
+
+| Question | Verdict | Evidence |
+|----------|---------|----------|
+| 1. Are subagents called when they should be? | MIXED | 8 spawns appropriate in scope; model tiers incorrect. Lines 23–141. |
+| 2. Correct model per subagent — quality/cost trade-off? | FAIL | Spawns 1–6 ABSENT model param → resolved sonnet; spawns 2–6 are mechanical file-read/shell work (should be haiku). Lines 23, 24, 41, 60, 83, 99. |
+| 3. Planning/grouping for context-window efficiency? | PASS | Task decomposition into 6 info-gather + 1 implement + 1 verify is sensible. Staggered spawn timing (7–101 min gaps) plausible for sequential dependencies. |
+| 4. Avoiding unnecessary reads by orchestrator? | PASS | Fact-sheet: 0 orchestrator tool calls across all segments. All reads delegated. |
+| 5. Sharing too much context with subagents? | PASS | Spawn prompts 419–838 chars, focused (specific files/checks), no open-ended dumps. |
+| 6. Following ORCHESTRATION.md instructions? | FAIL | Policy: "Every Agent spawn passes model explicitly." Spawns 1–6 violate (no model field). Policy: Default haiku for mechanical work. Spawns 2–6 violate (resolved sonnet). |
+| 7. Requesting/receiving only needed context back? | PASS | Focused prompts, task-notification async results, 0 repeated queries or context dumps by orchestrator. |
+
+---
+
+## Detailed Findings
+
+### Q1: Are subagents getting called when they should be?
+
+**Verdict:** MIXED — Spawning decisions were sound; model assignments were not.
+
+**Evidence:**
+- Lines 23–24: Spawned parallel agents to audit codebase + check live page. Appropriate for independent information-gathering tasks.
+- Lines 41, 60, 83, 99: Staggered spawns for ACF data, implementation context, additional files, remaining files. Timings: 7.5 min after spawn 2 result, then 101 min gap, then 1–2 min intervals. Pattern suggests sequential dependencies or pacing.
+- Lines 115, 141: Implementation (sonnet) then verification (haiku) spawns. Correct decomposition.
+- **Issue:** Spawns 2–6 (pure file reads and WebFetch) did not require judgment and should not have been delegated at all per policy—or if delegated, should have been haiku. This is a model-selection issue, not a spawning issue.
+
+### Q2: Is the correct model chosen per subagent?
+
+**Verdict:** FAIL — 5 of 8 spawns used incorrect (overspend) model tier.
+
+**Evidence:**
+- **Fact-sheet table:**
+  - Spawn 1 (line 23): model=ABSENT → resolved sonnet. Codebase audit requires judgment. **Correct tier (sonnet).**
+  - Spawn 2 (line 24): model=ABSENT → resolved sonnet. WebFetch live page for schema presence. **Mechanical task; should be haiku.**
+  - Spawn 3 (line 41): model=ABSENT → resolved sonnet. Run wp-cli SSH commands to check ACF data. **Mechanical shell work; should be haiku.**
+  - Spawn 4 (line 60): model=ABSENT → resolved sonnet. Read Plugin.php and other context files verbatim. **Pure file reading; should be haiku.**
+  - Spawn 5 (line 83): model=ABSENT → resolved sonnet. Read additional context files verbatim. **Pure file reading; should be haiku.**
+  - Spawn 6 (line 99): model=ABSENT → resolved sonnet. Read remaining files verbatim. **Pure file reading; should be haiku.**
+  - Spawn 7 (line 115): model=sonnet → resolved sonnet. Implement ContactPagePiece with tests. **Requires judgment; correct (sonnet).**
+  - Spawn 8 (line 141): model=haiku → resolved haiku. Verify implementation files. **Mechanical checklist; correct (haiku).**
+
+- **Policy statement:** "Default `haiku` for mechanical file-edit/shell work; `sonnet` for anything requiring judgment; `opus` only for genuinely hard reasoning."
+
+- **Cost implication:** Spawns 2–6 used 5× sonnet tokens where haiku would have sufficed. Over a 101-minute window (17:02–18:43), the orchestrator paced the spawns, suggesting it waited for intermediate results; this is defensible, but the model tiers remain wrong.
+
+### Q3: Is the orchestrator planning/grouping tasks to maximize efficient context-window use?
+
+**Verdict:** PASS — Decomposition is sensible; parallelization timing is unclear but plausible.
+
+**Evidence:**
+- **Task grouping:** 
+  - Phase 1 (lines 23–24): Parallel information-gathering (codebase + live page).
+  - Phase 2 (line 41): Conditional follow-up (ACF data check, after learning the plugin isn't deployed).
+  - Phase 3 (lines 60, 83, 99): Sequential context reads (Plugin.php, schema implementations, test fixtures).
+  - Phase 4 (line 115): Implementation (sonnet, using all gathered context).
+  - Phase 5 (line 141): Verification (haiku).
+
+- **Timing pattern:** Spawns 1–2 at 16:54:47–16:54:51 (4-sec gap, parallel). Then 7.5-min gap before spawn 3 (17:02:17). Then 101-min gap before spawn 4 (18:43:29). Smaller gaps (1–2 min) between spawns 4–6.
+  - **Interpretation:** The orchestrator waited for earlier phase results (visible in line 28 text: "Here's the full picture... **Two separate problems:** 1. **The plugin isn't deployed on SDD at all.**") before spawning conditional work. This is reasonable if task 3 depends on knowing whether the plugin is live.
+  - **Question unresolved:** Could spawns 4–6 have been parallelized at the 18:43 mark instead of staggered? Fact-sheet heuristic found "None flagged," meaning no runs of ≥4 same-tool calls without spawns, so sequential-dependent work is plausible.
+
+- **Window efficiency:** Each spawn prompt is 419–838 chars. Full agent contexts (including codebase orientation) are not visible in spawns alone, but the orchestrator delegated all context-intensive work to subagents and made zero tool calls itself (next section), which is correct framing.
+
+### Q4: Is the orchestrator avoiding reading files it does NOT need (that the subagent would read anyway)?
+
+**Verdict:** PASS — Orchestrator made zero tool calls before/after spawns.
+
+**Evidence:**
+- **Fact-sheet orchestrator tool profile:**
+  ```
+  | segment | calls | bytes read |
+  |---------|-------|------------|
+  | pre-spawn-1 | 0 | 0 |
+  | after-spawn-1 through after-spawn-8 | 0 | 0 |
+  ```
+- **Implication:** The orchestrator did not call Read, Bash, Grep, or any other tool. All file reads, WebFetch requests, and SSH commands were delegated to subagents. This avoids duplication and respects the "delegate inspection to subagents" principle.
+- **Note:** No evidence of "orienting Read" per policy exemption ("A short orienting Read before delegating is fine when the target file/path is uncertain"); either (a) the orchestrator did not need orientation (task was clear from user prompt), or (b) orientation was done in previous turns not shown in this 161-line segment. Either way, no unnecessary reads in the shown span.
+
+### Q5: Is the orchestrator sharing too much context with subagents (filling their windows / clouding judgment)?
+
+**Verdict:** PASS — Spawn prompts are focused and appropriately scoped.
+
+**Evidence:**
+- **Spawn prompt lengths:** 419–838 chars (from grep at analysis above).
+- **Spawn prompt specificity:**
+  - Line 23: "investigate whether a 'contact-us' page schema has ever been set up" + specific search targets (src/, openspec/, docs/). Focused.
+  - Line 24: "Fetch the page at https://www.studentdisciplinedefense.com/contact-us and check whether it contains any JSON-LD schema markup." Focused.
+  - Line 41: "Use wp-cli SSH access to check what office data is currently in the SDD site's ACF fields." Focused.
+  - Line 60: "Read the following files... return their full contents: 1. /home/jared/dev/llf-schema/src/Plugin.php, 2. ..." Focused list.
+  - Lines 83, 99: Same pattern—specific file list, no extraneous narrative.
+- **Negative:** No evidence of context dumps (e.g., "here is the entire schema design doc, here is the implementation history, here is the requirements spec" bundled without filtering). Each spawn asks for specific files or specific checks.
+
+### Q6: Is the orchestrator even following the ORCHESTRATION.md instructions?
+
+**Verdict:** FAIL — Two violations: model parameter not passed; model tier incorrect for 5 of 8 spawns.
+
+**Evidence:**
+
+**Violation 1: Policy says "Every Agent spawn passes model explicitly."**
+- Spawns 1–6 (lines 23, 24, 41, 60, 83, 99): No `model` field in Agent input.
+  ```python
+  # Spawn 1 (line 23) input keys: ['description', 'prompt', 'run_in_background']
+  # Spawn 6 (line 99) input keys: ['description', 'prompt', 'run_in_background']
+  ```
+- Spawns 7–8 (lines 115, 141): `model` field present.
+  ```python
+  # Spawn 7 (line 115) input: {..., 'model': 'sonnet'}
+  # Spawn 8 (line 141) input: {..., 'model': 'haiku'}
+  ```
+- **Violation confirmed:** 6 of 8 spawns omit the required model parameter.
+
+**Violation 2: Model tier misallocated to spawns 2–6.**
+- Policy: "Default `haiku` for mechanical file-edit/shell work; `sonnet` for anything requiring judgment."
+- Actual resolution (from fact-sheet) for spawns 2–6: all sonnet.
+  - Spawn 2 (WebFetch): mechanical → should be haiku, got sonnet.
+  - Spawn 3 (wp-cli SSH): mechanical shell → should be haiku, got sonnet.
+  - Spawn 4 (read Plugin.php): mechanical read → should be haiku, got sonnet.
+  - Spawn 5 (read context files): mechanical read → should be haiku, got sonnet.
+  - Spawn 6 (read remaining files): mechanical read → should be haiku, got sonnet.
+- **Cost implication:** ~5 unnecessary sonnet invocations when haiku would have been sufficient and correct. This is not a catastrophic misallocation (the tasks still completed), but it violates the policy's explicit cost-optimization intent.
+
+### Q7: Is the orchestrator requesting/receiving back only the context it needs, rather than a full context dump from the subagent?
+
+**Verdict:** PASS — Focused requests, async task results, no repeated queries observed.
+
+**Evidence:**
+- **Spawn prompts are scoped requests, not open-ended:**
+  - Line 24: "check whether it contains any JSON-LD schema markup" (not "summarize the page" or "return all content").
+  - Line 41: "Run these commands: 1. `ssh sdd '...'`; 2. ...`" (specific operations, not "explore the SDD server").
+  - Line 60: "Read the following files... return their full contents verbatim" (specific files, not "all PHP files" or "all src/").
+- **Async result handling:**
+  - Lines 46, 48, etc.: Task notifications arrive as async user-context messages (subagent results baked in, not separately queried).
+  - No evidence of follow-up tool calls by orchestrator to re-fetch or clarify subagent responses (fact-sheet: 0 orchestrator calls).
+- **No repeated context-dump requests:**
+  - Each spawn requests specific artifacts; no "give me everything you learned" or "re-read that file and explain it differently" follow-ups.
+
+---
+
+## Root-Cause Analysis: Why Model Parameters Were Omitted
+
+**Hypothesis:** The orchestrator code path for spawning agents may have defaulted to omitting the model parameter on the first 6 spawns, only explicitly passing it on the final 2 (implementation and verification). Possible causes:
+1. Different code paths or templates for information-gathering vs. implementation spawns.
+2. A changed or partially-applied fix (spawns 7–8 pass model explicitly; 1–6 do not).
+3. An oversight or a once-per-session pattern in the orchestrator's logic.
+
+**Impact:** Subagents resolved to sonnet (the orchestrator's resolved model at the time), masking the violation until audit. Policy compliance requires explicit parameter, not just correct resolution.
+
+---
+
+## Recommendations
+
+1. **Model parameter requirement (blocking):** All Agent spawns must include an explicit `model` parameter. Spawns 1–6 violated this; code path likely needs a wrapper or template update to ensure all spawns carry the parameter.
+
+2. **Model tier audit:** Review spawn 2–6 model assignments. Haiku is sufficient and correct per policy for:
+   - WebFetch queries with a yes/no or extraction task (spawn 2)
+   - wp-cli shell commands (spawn 3)
+   - File-reading tasks (spawns 4–6)
+
+3. **Test case:** Add a session-audit checklist or automated linter to flag Agent spawns without an explicit model parameter before they run.
+
+4. **Parallelization review (optional):** The 7–101 minute gaps between spawns 3–4 and 1–3 are not immediately explained by the transcripts shown. If they reflect waiting for sequential task results, that's fine; if they reflect pacing or manual delay, parallelizing spawns 4–6 (the three file-read tasks) may recover wall-clock time without changing correctness.
+
+---
+
+## Appendix: Data Sources
+
+- **Fact-sheet:** `/tmp/claude-1000/-home-jared-dev-cc-os/3c15edee-8ac7-4ffe-b6ee-ee6a6ebd670d/scratchpad/factsheets/S5-aadeb66c.md`
+- **Transcript:** `/home/jared/.claude/projects/-home-jared-dev-llf-schema/aadeb66c-0d64-4d41-b407-83e0209428b4.jsonl` (161 lines)
+- **Policy:** ORCHESTRATION.md (global default; per-project CLAUDE.md equivalent at 2026-06-29)
+- **Audit method:** Spot-reads of specific lines (23, 24, 41, 60, 83, 99, 115, 141) for spawn details; timeline reconstruction via timestamps; model-resolution data from fact-sheet table.

docs/orchestration-audit/auditor-reports/S6-report.md

@@ -0,0 +1,170 @@
+# Session Orchestration Audit: S6-3dc26da3
+
+**Session:** philly-search-engine-marketing, 2026-06-29 15:08–16:58 UTC  
+**Transcript:** 120 jsonl lines, 3.3MB (largely user-submitted images)  
+**Scope:** 21 assistant turns, 13 human prompts, 1 Agent spawn  
+**Policy Under Audit:** Project-local CLAUDE.md (lines 72–102), "All work follows an orchestrator-subagent pattern"
+
+---
+
+## Question 1: Are subagents getting called when they should be?
+
+**Verdict:** PASS
+
+**Evidence:**
+- Line 98: Agent spawned for multi-file update task (TODO.md + logs/2026-06-29.md)
+- Lines 20–85: Conversational debugging of Zapier error handling (no delegation, correct — this is advice/troubleshooting, not "work")
+- Line 89 (user): "Update todos and documentation based on this conversation" — triggers explicit work task
+- Line 97 (orchestrator): "I'll handle the documentation updates in parallel" — signals intent to delegate
+- Fact-sheet confirms 1 spawn, 0 orchestrator tool calls (Read/Write/Bash/Grep)
+
+The project's local CLAUDE.md (line 72) is strict: *"All work follows an orchestrator-subagent pattern. No exceptions."* The orchestrator correctly identified the file-update task as delegable work and spawned an Agent. The conversational turns before that are appropriately handled directly (no restriction against speaking directly; the restriction is on direct tool calls).
+
+---
+
+## Question 2: Is the correct model chosen per subagent — highest reasonable quality at lowest cost?
+
+**Verdict:** FAIL
+
+**Evidence:**
+- Line 98 Agent spawn: `input` contains only `description` and `prompt` — NO `model` field
+- Fact-sheet line: "model param: (ABSENT)" → "resolved model: claude-sonnet-4-6"
+- Task: Read 2 files (TODO.md, logs/2026-06-29.md), append structured items, create log if missing, return summary
+- Project CLAUDE.md model routing (lines 86–91):
+  ```
+  | Haiku  | File reads, simple edits, formatting, search |
+  | Sonnet | Feature implementation, refactoring, tests |
+  | Opus   | Architectural decisions, complex debugging |
+  ```
+
+This task is **file reads + simple edits + formatting** — a textbook Haiku task. Sonnet is 4x more expensive and unnecessary for mechanical file operations. 
+
+**Violation:** Project CLAUDE.md line 86 states "Every Agent spawn passes model explicitly" (implied reference to the table above). The spawn did not include `model: "haiku"` despite the task profile being unambiguously haiku-level.
+
+**Cost impact:** Agent consumed 28.8k subagent tokens (reported in line 105 task-notification `<subagent_tokens>`). At typical rates, a Haiku agent would have consumed ~7–10k tokens for the same work. Estimated overspend: ~3–4x cost.
+
+---
+
+## Question 3: Is the orchestrator planning/grouping tasks to maximize efficient context-window use?
+
+**Verdict:** MIXED (structure sound, execution not independently verifiable)
+
+**Evidence:**
+- Single Agent spawn with 2039-char prompt (well-structured, ~2KB)
+- Prompt includes explicit instructions:
+  - Files to read (with fallback logic: "create if missing")
+  - Two specific TODO items with full context (Gary's priority, Zapier blocker details)
+  - Four session-log entry points covering the decision, implementation, blocker, and next steps
+  - Instruction to "Return a brief summary"
+- Agent returned after 55.2 seconds with 6 tool_uses (likely: 2 Reads, 2 Writes, 1–2 git ops for potential commit/log)
+
+The prompt is well-factored — it frontloads all context (no need for back-and-forth). The task scope is narrow enough that one Agent call suffices. The agent's token count (28.8k) is reasonable for the work scope.
+
+**However:** Cannot verify from the orchestrator's window whether the agent *internally* batched reads/writes efficiently or made sequential redundant operations. The agent's own transcript is off-limits per line 99 guidance ("do NOT Read or tail this file via the shell tool").
+
+---
+
+## Question 4: Is the orchestrator avoiding reading files it does NOT need (that the subagent would read anyway)?
+
+**Verdict:** PASS
+
+**Evidence:**
+- Fact-sheet: "calls: 0, bytes read: 0" (pre-spawn and post-spawn)
+- Grep of transcript: No `"name":"Read"` tool uses anywhere
+- Orchestrator did not self-read TODO.md or logs/2026-06-29.md before delegating
+
+Project CLAUDE.md line 81 explicitly forbids this: *"Does NOT read files to prepare delegation specs — write specs from the user's request."*
+
+The orchestrator inlined all context (Gary's email content, Zapier error details, decision rationale) from the conversation into the Agent prompt, requiring zero file reads. Correct pattern.
+
+---
+
+## Question 5: Is the orchestrator sharing too much context with subagents (filling their windows / clouding judgment)?
+
+**Verdict:** PASS
+
+**Evidence:**
+- Agent prompt: 2039 characters of task instructions
+- Content: structured, specific, actionable
+  - Clear "add these two items to TODO.md" (with bulleted details)
+  - Clear "add this session entry to logs/2026-06-29.md" (with bulleted details)
+  - Clear instruction to create log file if missing and follow the pattern of other logs/
+- No extraneous conversation transcript dumped into the prompt
+- No "here's everything the user said, figure it out" vagueness
+
+The prompt is dense but focused. It reads as a well-written specification, not a context-dump. The agent consumed 28.8k tokens total (subagent tokens per line 105); that's a reasonable allocation for the complexity of the task.
+
+---
+
+## Question 6: Is the orchestrator even following the ORCHESTRATION.md instructions?
+
+**Verdict:** FAIL (partial compliance)
+
+**Evidence:**
+
+**What was done correctly:**
+- Line 77–80 (orchestrator responsibilities):
+  - ✓ Interpreted user request as delegable work
+  - ✓ Delegated via Agent tool (not direct Read/Write/Bash)
+  - ✓ Did not call Read/Write/Bash/Grep directly
+  - ✓ Did not read files to prepare specs
+- Line 84 (subagent returns):
+  - ✓ Agent returned summary (task-notification at line 105: *"Both files are updated. Here's what was done:"*)
+  - ✓ Included artifact paths (README.md implied; full structure in summary)
+
+**What was done incorrectly:**
+- Line 86 (model routing): States "Every Agent spawn passes model explicitly"
+  - ✗ Spawn at line 98 has NO `model` field in input
+  - ✗ Cascaded default to sonnet instead of haiku
+  - ✗ Violates explicit policy: "Haiku for file reads, simple edits, formatting"
+
+This is a clear violation of a single, high-stakes instruction in the project's local policy.
+
+---
+
+## Question 7: Is the orchestrator requesting/receiving back only the context it needs, rather than a full context dump from the subagent?
+
+**Verdict:** PASS
+
+**Evidence:**
+- Task-notification at line 105: Contains task metadata + concise summary of actions taken
+  - Status: completed
+  - Summary: "Both files are updated. Here's what was done:"
+  - Details: Two bulleted items for TODO.md, four bulleted items for logs/2026-06-29.md entry
+  - Usage stats: subagent_tokens, tool_uses count, duration_ms
+- Orchestrator's response (line 109): *"Documentation updated. TODO.md has both items at the top of the active section, and today's session log is created. Ready for the SDD Contact Us schema when you are."*
+  - Acknowledges completion
+  - Confirms both files were touched
+  - Moves forward without demanding a full file dump or line-by-line verification
+- No off-limits file read (per line 99 guidance)
+
+The subagent's summary was sufficient. The orchestrator neither over-requested nor over-received context.
+
+---
+
+## Summary of Misses
+
+1. **Model routing violation (Question 2):** The Agent spawn at line 98 omitted the required `model: "haiku"` parameter. Project policy is explicit and non-negotiable: *"Every Agent spawn passes model explicitly."* The task (file reads + edits) is a textbook Haiku case. **Estimated cost impact:** 3–4x overspend (sonnet vs. haiku pricing).
+
+2. **All other questions:** Orchestrator's delegation pattern, context handling, and response flow are compliant with the project's CLAUDE.md policy and the global ORCHESTRATION.md rubric.
+
+---
+
+## Candidate Eval Trigger / Criterion
+
+When a project has a strict agent-model routing table (e.g., "Haiku for file reads/edits/formatting"):
+- **Criterion:** Every Agent spawn must include an explicit `model: "haiku"` (or the table's default) parameter. Default cascade (allowing the system to choose sonnet) is a violation, not a fallback.
+- **Orchestrator should:** Check the task type against the project's model routing table *before* writing the Agent call. Pass the required model parameter unconditionally.
+- **Signal for rejection:** A project CLAUDE.md with a model routing table + any Agent spawn that lacks an explicit model field matching that table.
+
+---
+
+## Appendix: Transcript Anomaly
+
+The 3.3MB transcript with only 120 lines and 21 assistant turns is explained by:
+- Lines 10, 51, 63: User messages containing embedded images (750KB, 1.4MB, 787KB respectively)
+- Image data is stored inline in the JSON (base64 or direct encoding)
+- No orchestrator tool calls, so no Read/Write/Bash results to inflate the transcript further
+- The bulk is user input (images), not orchestrator work or subagent verbosity
+
+This is not an orchestration efficiency issue; it's a client-interaction pattern (user sharing screenshots for Zapier troubleshooting).

docs/orchestration-audit/auditor-reports/S7-report.md

@@ -0,0 +1,247 @@
+# Audit Report: Session 3771b288 (~/servers ops work)
+
+**Date:** 2026-07-06  
+**Session:** 3771b288-eb9d-4398-9e75-e1ff0ba6af03  
+**Project:** /home/jared/servers (ops/infrastructure)  
+**Duration:** 16:05:40 to 17:14:47 UTC (69 min)  
+**Turns:** 144 assistant turns, 11 human prompts  
+**Policy Audited:** ORCHESTRATION.md (global rollout 2026-07-04)
+
+---
+
+## SEVEN-QUESTION RUBRIC VERDICTS
+
+### 1. Are subagents getting called when they should be?
+
+**Verdict:** MIXED
+
+**Evidence:**
+- Lines 241, 243, 245: Three Explore agents spawned for read-only discovery tasks:
+  - Line 241: Explore ovh-prod service setup conventions (2282 chars prompt)
+  - Line 243: Explore desktop backup bug specifics (2725 chars prompt)
+  - Line 245: Explore proxmox VM 101 and umbrella structure (2126 chars prompt)
+- Explores are appropriate for their scope (file discovery, grep, schema extraction)
+- Pattern: Orchestrator conducted 32 pre-spawn Bash commands + 4 Reads (74KB total) **before** delegating to agents
+
+**Concern:**
+Lines 1-240 show orchestrator performing extensive diagnostic work itself:
+- 32 Bash commands: grep across servers for n8n, systemctl status checks, journalctl log inspection, SSH diagnostics to ovh-prod
+- 4 Reads: project memory, n8n config, spec files, services inventory
+- This exploratory work (grep, find, systemctl, journalctl, ssh) is exactly what Explore agents are designed to do
+
+The orchestrator briefed three agents with comprehensive prompts (2.2–2.7 KB each), suggesting it already understood the context. This is valid if the pre-spawn diagnostics were necessary to formulate good agent prompts, but it shifts exploratory token cost into the orchestrator's session window rather than the agent's. The policy allows "A short orienting Read before delegating is fine" — but 32 Bash + 4 Read is beyond "orienting."
+
+**Missed Delegation Trigger:**  
+"When orchestrator must diagnose a multi-faceted situation across multiple machines/services, spawn Explore agents early with open-ended discovery prompts rather than conducting detailed diagnostics in-session first, unless the diagnostics are sub-1-minute or ≤2 tool calls."
+
+---
+
+### 2. Is the correct model chosen per subagent — highest reasonable quality at lowest cost?
+
+**Verdict:** FAIL
+
+**Evidence:**
+- Line 241: `"model": null` (ABSENT), resolved to `claude-haiku-4-5-20251001`
+- Line 243: `"model": null` (ABSENT), resolved to `claude-haiku-4-5-20251001`
+- Line 245: `"model": null` (ABSENT), resolved to `claude-haiku-4-5-20251001`
+
+**Policy Violation:**
+ORCHESTRATION.md states: "Every `Agent` spawn passes `model` explicitly. Default `haiku` for mechanical file-edit/shell work; `sonnet` for anything requiring judgment; `opus` only for genuinely hard reasoning."
+
+All three spawns violate this by omitting the `model` parameter entirely. The model resolved to haiku via the Explore agent's own default, not via explicit orchestrator choice.
+
+**Assessment of Haiku as the Actual Choice:**
+- All three tasks are read-only discovery (appropriate for haiku)
+- Tasks: (1) schema extraction from ovh-prod docs, (2) file location and diagnostics from desktop, (3) directory structure and grep across umbrella project
+- None require judgment-layer reasoning (schema extraction is mechanical; file discovery is mechanical; structure understanding is verification, not judgment)
+- **Haiku is the correct choice**, but the policy requires the orchestrator to **state it explicitly**, not rely on agent defaults
+
+**Missed Enforcement Trigger:**  
+"Every Agent() call must include `model: "haiku"` (or sonnet/opus) as an explicit parameter, even when the default matches the intended choice."
+
+---
+
+### 3. Is the orchestrator planning/grouping tasks to maximize efficient context-window use?
+
+**Verdict:** PASS
+
+**Evidence:**
+- Lines 294+: Single `ExitPlanMode()` call with detailed plan covering three distinct domains:
+  - Part 1: Fix desktop bug 1 (notify_backup.rb bare command → full path)
+  - Part 2: Fix desktop bug 2 (systemd-inhibit fallback on polkit auth failure)
+  - Part 3: Stand up proxmox-ubuntu project scaffold
+  - Part 4: Record deferred items (ovh-prod cron instrumentation, n8n on proxmox)
+- Plan synthesis: Root cause analysis grouped related bugs (two desktop bugs from single diagnosis session) and separated them from unrelated work (proxmox project, ovh-prod deferral)
+- Three agents spawned in parallel (line 241–245 are sequential calls but agents run async) to explore three independent domains, avoiding redundant reads
+- After agents return, orchestrator executes plan in correct sequence (fixes before scaffolding, deferred items recorded separately)
+
+**Task Grouping Logic:**
+The plan efficiently batches work by separating:
+- Immediate fixes (desktop bugs) — quick, specific file edits
+- Project scaffolding (proxmox-ubuntu) — multi-file writes, template-driven
+- Deferred tracking (incidents log, roadmap) — documentation updates
+This avoids interleaving unrelated work and keeps context coherent.
+
+---
+
+### 4. Is the orchestrator avoiding reading files it does NOT need (that the subagent would read anyway)?
+
+**Verdict:** FAIL
+
+**Evidence:**
+- Pre-spawn (lines 1–240): Orchestrator reads `/home/jared/.n8n_instances.yml`, `/home/jared/servers/desktop/docs/superpowers/specs/2026-05-07-n8n-backup-monitor-design.md`, grepped `/home/jared/servers/ovh-prod/docs/services-inventory.md`
+- Line 241 Explore prompt explicitly lists "Read `/home/jared/servers/ovh-prod/docs/services-inventory.md`" as one of the items for the agent to read
+- Line 243 Explore prompt asks the agent to "Read `/home/jared/.local/bin/backup.sh` in full"
+
+**Overlap Found:**
+- Orchestrator read parts of `services-inventory.md` (grep for n8n entries) before delegating Explore task that would read the entire file
+- Orchestrator examined `backup.sh` in-session (Bash cat command) before delegating agent to read it fully
+
+This is redundant reading — the orchestrator read files that the Explore agents would subsequently read. The files were read to formulate the agent prompts, but the prompt content shows the agents were asked to read the same files independently.
+
+**Missed Efficiency Trigger:**  
+"If an orchestrator prompt to an Explore agent lists 'Read file X' as a task, don't Read file X in-session first — let the agent do it and summarize findings back."
+
+---
+
+### 5. Is the orchestrator sharing too much context with subagents (filling their windows / clouding judgment)?
+
+**Verdict:** PASS
+
+**Evidence:**
+- Line 241 prompt: 2282 chars (~570 tokens) — substantial but focused
+- Line 243 prompt: 2725 chars (~680 tokens) — includes detailed diagnosis context (symptom, root cause, diagnosis result) from pre-spawn work
+- Line 245 prompt: 2126 chars (~530 tokens) — structured multi-part discovery task
+
+Each prompt is well-bounded and task-oriented:
+- Prompt 1: "Read these 12 specific files in ovh-prod and summarize the step-by-step pattern"
+- Prompt 2: "Given the diagnosis in incidents.md (which I've already logged), find the exact lines in backup.sh and systemd files that implement the two bugs"
+- Prompt 3: "Read umbrella CLAUDE.md, proxmox-hermes template, grep for ubuntu-server references, check ~/.ssh/config"
+
+None of these prompts dump the orchestrator's entire diagnostic session (which would have been 10+ KB of systemctl output, logs, ssh diagnostics). They're filtered to task-relevant context.
+
+**Context Quality:**
+Prompts are specific enough to give agents clear direction without forcing them through the orchestrator's entire reasoning path.
+
+---
+
+### 6. Is the orchestrator even following the ORCHESTRATION.md instructions?
+
+**Verdict:** FAIL (Model param explicitly required)
+
+**Evidence:**
+- Policy: "Every `Agent` spawn passes `model` explicitly."
+- Observation: All 3 Agent spawns (lines 241, 243, 245) have `"model": null` (parameter omitted entirely)
+- Fact-sheet confirms: "model param | (ABSENT)" for all three rows
+- Fallback resolution: Explore agent's own default (haiku) was used instead of orchestrator choice
+
+**Additional Compliance Check:**
+- Policy says haiku default is for "mechanical file-edit/shell work" — Explores are read-only, mechanical, haiku is correct
+- Policy does NOT say orchestrators can omit the model param if they think the default is right; it explicitly requires the param be passed
+
+**Secondary Issue (Model Param, Not Violation But Efficiency Concern):**
+- The 32 pre-spawn Bash commands + diagnostics should have been minimal; orchestrator appears to have over-briefed itself before delegating
+
+---
+
+### 7. Is the orchestrator requesting/receiving back only the context it needs, rather than a full context dump from the subagent?
+
+**Verdict:** PASS
+
+**Evidence:**
+- Line 241 result: 7537 chars (~1884 tokens) — summary of ovh-prod structure, step-by-step pattern, file inventories
+- Line 243 result: 3562 chars (~890 tokens) — exact line numbers, code snippets for bugs, analysis of failures
+- Line 245 result: 4776 chars (~1194 tokens) — umbrella context, proxmox-hermes template, umbrella file inventory, VM 101 references
+
+Results are task-focused summaries, not dumps:
+- Agent 1 returned pattern + inventory, not full file contents of 12 files
+- Agent 2 returned bug locations + explanations, not raw logs/journalctl output
+- Agent 3 returned references + template structure, not entire directory trees
+
+After agents return, orchestrator synthesizes results into a plan (line 294) without re-reading files, indicating the agent summaries were sufficient.
+
+---
+
+## SUMMARY TABLE
+
+| # | Verdict | Issue | Severity |
+|---|---------|-------|----------|
+| 1 | MIXED | Pre-spawn diagnostics could have been delegated to Explore; 32 Bash + 4 Read in-session | Medium |
+| 2 | **FAIL** | **All 3 Agent spawns omit `model` param; policy requires explicit param** | High |
+| 3 | PASS | Plan synthesis and task grouping are efficient | — |
+| 4 | FAIL | Orchestrator read files before delegating Explore agents to read the same files | Medium |
+| 5 | PASS | Prompt context is focused and bounded, not excessive | — |
+| 6 | **FAIL** | **Model param violation = ORCHESTRATION.md non-compliance** | High |
+| 7 | PASS | Subagent results are summaries, not full dumps | — |
+
+---
+
+## MISSED DELEGATION CANDIDATES
+
+### Write x5 (lines 345–353)
+
+**Files written:**
+1. `/home/jared/servers/proxmox-ubuntu/CLAUDE.md`
+2. `/home/jared/servers/proxmox-ubuntu/README.md`
+3. `/home/jared/servers/proxmox-ubuntu/docs/services-inventory.md`
+4. `/home/jared/servers/proxmox-ubuntu/docs/roadmap.md`
+5. `/home/jared/servers/ovh-prod/docs/incidents.md`
+
+**Content pattern:** Multi-file project scaffold + incident logging, drawing on Explore agent summaries
+
+**Judgment involved:** Yes — each file cross-references ovh-prod patterns, templates from proxmox-hermes, and specific decisions about what to defer (n8n, ovh-prod cron instrumentation)
+
+**Delegation assessment:** Technically a "spans many files" candidate per policy, but:
+- Files are contextually dependent (CLAUDE.md references README references services-inventory)
+- This is plan **execution**, not exploration — orchestrator is synthesizing Explore results into a single coherent scaffold
+- Sequential scaffolding logic (CLAUDE.md must define context before README can reference it) makes parallelization unclear
+
+**Verdict:** Not a violation. Write x5 is a valid in-session execution phase after exploration, especially given the multi-file interdependencies and the need to synthesize Explore results. If the Write sequence were purely mechanical templating with zero judgment, it would be a delegation candidate; here, it involves template selection, cross-project pattern alignment, and deferral routing.
+
+---
+
+## KEY FINDINGS
+
+1. **Model param violation is the primary issue.** All three Agent spawns violate the explicit ORCHESTRATION.md requirement. The policy is clear: every spawn must include the `model` parameter. Even if haiku is the right choice, it must be stated. Recommend a lint rule: `Agent(...) without model: param → error before tool execution`.
+
+2. **Pre-spawn diagnostics could be better scoped.** The orchestrator spent 74 KB and 36 tool calls on diagnostics before delegating exploration. A tighter pattern would be: read the umbrella context + problem statement (≤1 KB), then spawn Explore agents with open-ended discovery prompts and let them do the diagnostic work. The 32-Bash diagnostic session (systemctl, journalctl, ssh, grep) is exactly what Explore is for.
+
+3. **File read overlap.** The orchestrator read `services-inventory.md` and `backup.sh` in-session, then asked Explore agents to read the same files. Tighter scoping would avoid this redundancy.
+
+4. **Plan synthesis is strong.** The orchestrator correctly bundled three conceptually separate tasks (desktop bugs, proxmox project, deferred tracking) into a single coherent plan, with good separation between immediate fixes and deferred work. Plan execution (the Writes) was appropriate in-session.
+
+5. **Explore agents received well-bounded prompts.** Despite the pre-spawn diagnostic work, the prompts given to agents were focused and didn't dump the full session history on them.
+
+---
+
+## RECOMMENDATIONS
+
+**Immediate (High Priority):**
+1. Enforce `model: "haiku"` (or explicit model choice) on every `Agent()` call. This is a hard requirement per ORCHESTRATION.md and was violated on all 3 spawns in this session.
+
+**Near-term (Medium Priority):**
+2. Reduce pre-delegation diagnostics. When spawning Explore agents, provide umbrella context + clear task description; don't pre-diagnose. Let Explore agents handle discovery.
+3. Add a "no redundant reads" check: if you're about to delegate a task that includes "Read file X," don't Read file X in the orchestrator session first.
+
+**Process (Optional):**
+4. Consider a linting/validation layer that catches model-param omissions before agent spawn.
+
+---
+
+## TIMELINE
+
+- **16:05–16:47** (42 min): Orchestrator diagnostic work (32 Bash, 4 Read) + plan synthesis
+- **16:47:03** (line 241): Explore spawn 1 (ovh-prod conventions) — async, model=null
+- **16:47:14** (line 243): Explore spawn 2 (desktop backup bugs) — async, model=null
+- **16:47:24** (line 245): Explore spawn 3 (proxmox VM 101) — async, model=null
+- **16:47:27**: Orchestrator synthesis turn (awaiting agent results)
+- **After agent completion** (not timestamped in fact-sheet but ~line 294+): Plan exited, execution begins (Write x5)
+- **17:14:47**: Session ends
+
+---
+
+## CONCLUSION
+
+The session demonstrates **strong plan design and synthesis** but **violates the core ORCHESTRATION.md policy** on explicit model parameterization. The orchestrator should have passed `model: "haiku"` to all three Explore spawns. Additionally, 74 KB of pre-delegation diagnostic work could have been better scoped, with more delegated to the Explore agents. The Write x5 execution was appropriate given the multi-file scaffold context.
+
+**Overall Verdict:** Policy non-compliant (model param violation), efficiency improvable (pre-spawn diagnostics), but execution and planning logic sound.

docs/orchestration-audit/auditor-reports/S8-report.md

@@ -0,0 +1,127 @@
+# Orchestration Audit Report: Session 3fc7bb8c (S8)
+
+**Transcript:** `/home/jared/.claude/projects/-home-jared-servers/3fc7bb8c-54c1-4147-ac3d-dc7db3fca924.jsonl`  
+**Session Date:** 2026-07-04, 17:20–18:04 UTC  
+**Duration:** ~44 minutes  
+**Turns:** 90 assistant turns, 5 human prompts  
+**Tools:** 44 total (Bash:27, Edit:6, Write:4, Read:3, ToolSearch:2, ExitPlanMode:1, Skill:1)  
+**Agent Spawns:** 0
+
+---
+
+## Question 1: Are subagents getting called when they should be?
+
+**Verdict: PASS** — No subagent spawns, but none were required.
+
+**Evidence:**
+- Factsheet reports zero Agent spawns across 90 turns.
+- Work was inherently sequential: diagnostic steps on ovh-prod (commands 2–6 in Bash sequence) → n8n investigation (commands 7–15) → documentation (commands 16–21) → desktop verification (commands 22–27). Each phase depends on conclusions from the previous.
+- Bash command sequence analysis (turns 6–87): SSH diagnostics were tightly coupled (command 2 reads the script, command 3 checks logs for clues based on script content, command 5 applies a fix targeting symptoms found in command 3).
+- No wide grep-and-synthesize or large-context work that would trigger the "spans many files" or "needs large/isolated context" delegation criteria.
+
+**Key Dependency Chain:**
+- Turn 6–11: SSH to ovh-prod, read backup.sh, check logs, apply fix, test it.
+- Turn 19–38: Investigate n8n (grep for references, check docker, query database) — depends on conclusion that backup.sh may involve n8n.
+- Turn 41–48: Update incidents.md with findings — depends on diagnosis being complete.
+- Turn 51–87: Set up verification on desktop — depends on understanding the full scope from phases 1–3.
+
+No parallel discovery threads; the work was single-stream investigation.
+
+---
+
+## Question 2: Is the correct model chosen per subagent?
+
+**Verdict: N/A** — No subagents spawned.
+
+**Evidence:** Not applicable; the session spawned zero agents.
+
+---
+
+## Question 3: Is the orchestrator planning/grouping tasks to maximize efficient context-window use?
+
+**Verdict: MIXED** — Grouping was implicit rather than explicit; minor opportunity missed.
+
+**Evidence:**
+- Read calls (3 total): incidents.md files (2), memory.md (1). Low volume overall.
+- Bash calls (27 total) were sequential, with no explicit batching of independent reads.
+- **Missed Opportunity:** Commands 7–8 (local grep for n8n references in /servers/ directory) could have run in parallel with commands 2–6 (SSH ovh-prod diagnostics) — they are fully independent (no shared state, no cross-system dependencies). Commands 7–8 appear at lines 65–73; commands 2–6 completed earlier.
+- However, the gain would be minimal (~1–2 minutes wall-clock savings); the overall session is already efficient because most work is hands-on diagnostic/fix (no large context accumulation).
+- No evidence of explicit grouping strategy in the transcript (no mention of "I'll batch these", "run in parallel", etc.). The orchestrator appears to have run tasks as they naturally emerged from the diagnosis, which is appropriate for interactive troubleshooting.
+
+**Specific Lines:**
+- Lines 65–73: grep commands 7–8 (local filesystem search for n8n).
+- Lines 25–35: grep commands 2–6 completed earlier (SSH ovh-prod diagnostics).
+- They were executed sequentially, but could have overlapped.
+
+---
+
+## Question 4: Is the orchestrator avoiding reading files it does NOT need?
+
+**Verdict: PASS** — Minimal unnecessary reads.
+
+**Evidence:**
+- Only 3 Read calls across 44 tools (7% of work).
+- Read 1 (line 14): `/home/jared/servers/ovh-prod/docs/incidents.md` — directly relevant; planning requires understanding the incident.
+- Read 2 (line 121): `/home/jared/servers/desktop/docs/incidents.md` — directly relevant; needed to update it.
+- Read 3 (line 88): `/home/jared/.claude/projects/.../memory/MEMORY.md` — project memory, part of standard SessionStart context.
+- No evidence of reading tangential files or over-loading context. The orchestrator stayed focused on the diagnostic targets.
+- **Bash vs Read Balance:** Most information was gathered via SSH diagnostics and local grepping (27 Bash calls) rather than reading large files. This is appropriate for this type of work.
+
+---
+
+## Question 5: Is the orchestrator sharing too much context with subagents?
+
+**Verdict: N/A** — No subagents spawned.
+
+**Evidence:** Not applicable.
+
+---
+
+## Question 6: Is the orchestrator following the ORCHESTRATION.md instructions?
+
+**Verdict: PASS** — Yes, though implicitly rather than explicitly.
+
+**Evidence:**
+- SessionStart injection (line 4): ORCHESTRATION.md was successfully injected at startup, confirming the policy was loaded.
+- Policy snippet quoted: "Do single-file, ≤2-tool-call ops directly. Don't delegate them. Delegate only when work is parallelizable across independent files/subtasks, spans many files, or needs a large/isolated context."
+- **Observance Check:**
+  - "Single-file, ≤2-tool-call ops done directly?" YES. Turns 44–49 (Edit incidents.md): 3 edits completed inline without delegation.
+  - "Delegation only for parallelizable/multi-file/large-context?" NOT TRIGGERED. No work met those criteria.
+  - "Explicit model parameter for Agent spawns?" Not applicable (zero spawns).
+- **Missing Signal:** No explicit reasoning about the decision. The transcript contains no text mentioning "this is sequential, so I won't delegate" or "these are independent so I could parallelize". The orchestrator appears to have followed the policy implicitly through task structure, not by conscious articulation.
+
+---
+
+## Question 7: Is the orchestrator requesting/receiving back only context it needs?
+
+**Verdict: PASS** — No subagent results; not applicable.
+
+**Evidence:** Zero subagents, so no results to evaluate. The orchestrator's own Bash/Read/Edit operations were tightly scoped (no over-reading, minimal context bloat).
+
+---
+
+## Summary of Findings
+
+### Key Strengths
+1. **Correct No-Delegation Decision:** The work was genuinely sequential; spawning even a single subagent would have introduced latency without parallelization benefit.
+2. **Minimal Context Loading:** Only 3 reads in 90 turns; work was hands-on diagnostic, not deep-context synthesis.
+3. **Tight Coupling Observed:** Each diagnostic step depended on the previous result, validating sequential execution.
+4. **Policy Compliance:** ORCHESTRATION.md was loaded and followed implicitly.
+
+### Minor Opportunities
+1. **Missed Parallelization:** Commands 7–8 (local n8n grep) vs. commands 2–6 (SSH ovh-prod diagnostics) could have run in parallel. Gain: ~1–2 minutes wall-clock time. Not worth subagent overhead, but explicit batching acknowledgment would improve transparency.
+2. **Implicit vs. Explicit Reasoning:** The orchestrator did not articulate why delegation was not chosen. For audit trails and future refinement, explicit reasoning ("this work is inherently sequential because...") would be valuable.
+
+### No Issues Found
+- No over-delegation of trivial work.
+- No under-use of subagents on parallelizable multi-file tasks.
+- No wasteful context passing.
+- No violation of the ≤2-tool-call rule for direct work.
+
+---
+
+## Verdict
+
+**AUDIT PASS** — The orchestrator correctly chose not to spawn agents for this session. The work was inherently sequential investigative debugging, with minimal parallelization opportunity and high dependency coupling. The policy was followed appropriately.
+
+**Recommendation for Future Sessions:** When multiple independent investigative threads emerge (even slightly), consider explicit acknowledgment of why they remain sequential. This would strengthen audit clarity without slowing work.

docs/orchestration-audit/auditor-reports/S9-report.md

@@ -0,0 +1,232 @@
+# Orchestration Audit: Session 2525fa2d-1b15-4495-99bf-c093716fab55
+
+**Project:** systems-admin  
+**Date:** 2026-06-30  
+**Main loop model:** claude-opus-4-8 (all 117 turns)  
+**Assistant turns / human prompts:** 117 / 6  
+
+## Summary
+
+The session executed a self-audit of Claude Code orchestration/delegation practices across the user's project portfolio. It spawned 3 background agents (2 general-purpose/opus, 1 claude-code-guide/haiku) to parallelize read-only research and documentation verification. The orchestrator correctly avoided reading files before delegating (except for initial orientation), structured prompts precisely without context bloat, and produced a PRD + handoff summary as deliverables.
+
+**Critical policy violation:** None of the 3 Agent spawns passed a `model` parameter explicitly, violating the stated ORCHESTRATION.md rule ("Every Agent spawn passes `model` explicitly").
+
+---
+
+## Findings by Question
+
+### 1. Are subagents getting called when they should be?
+
+**Verdict:** PASS — Delegations were appropriate and well-scoped.
+
+**Evidence:**
+- Line 95: Agent spawn #1 for "Inventory loaded context surface" — reads multiple config/skill directories and sums character counts. Parallelizable across multiple file paths; spans multiple directories (`~/.claude/settings.json`, `~/.claude/settings.local.json`, `~/.claude/skills/`, `~/.claude/plugins/`). Read-only task that requires no modification. Appropriate delegation.
+- Line 97: Agent spawn #2 for "Pull session examples" — analyze JSONL transcripts across 4 project categories to find representative examples. Explicitly requires reaching across many project directories (~/.claude/projects/*); good candidate for parallelization. Appropriate delegation.
+- Line 212: Agent spawn #3 for "Verify skill loading mechanism" — verify factual information from code.claude.com/docs. Specialist task (claude-code-guide subagent type). Appropriate delegation.
+
+**Analysis:**  
+All three delegations follow the spirit of "delegate when work spans many files or needs isolated context." None were single-file operations; all were read-only; none were ≤2-tool-call mechanical ops that should have been done directly.
+
+---
+
+### 2. Is the correct model chosen per subagent — highest reasonable quality at lowest cost?
+
+**Verdict:** FAIL — All spawns lack explicit `model` parameters; two are overpowered.
+
+**Evidence:**
+- Line 95: Agent spawn #1 has no `model` param; `resolvedModel` = `claude-opus-4-8`. Task is character-counting and summation (mechanical, read-only). Appropriate model would be `haiku` (mechanical file-reading) or at most `sonnet` (if judgment needed on "contextually heavy" categorization). **Opus is overpowered.** Fact-sheet: prompt chars = 2662; result chars = 7517.
+- Line 97: Agent spawn #2 has no `model` param; `resolvedModel` = `claude-opus-4-8`. Task is example selection and characterization — requires judgment on what makes examples representative. Appropriate model would be `sonnet` (judgment). **Opus is overpowered.** Fact-sheet: prompt = 2397 chars; result = 6095 chars.
+- Line 212: Agent spawn #3 has no `model` param; `resolvedModel` = `claude-haiku-4-5-20251001`. Task is documentation lookup/verification (mechanical). **Model is correct.** Fact-sheet: prompt = 2065 chars; result = 5290 chars.
+
+**Analysis:**  
+The policy states: "Default `haiku` for mechanical file-edit/shell work; `sonnet` for anything requiring judgment; `opus` only for genuinely hard reasoning."
+
+- Spawn #1: Character-counting is mechanical → should be haiku.
+- Spawn #2: Example selection requires judgment → should be sonnet.
+- Spawn #3: Doc lookup is mechanical → haiku is correct.
+
+The fact that spawn #3 resolved to the correct model despite no explicit param suggests either accidental correct default or the haiku-for-docs-lookup heuristic in the subagent type. Spawns #1 and #2 defaulting to opus indicates the main-loop model is the fallback when no param is passed — a risk.
+
+---
+
+### 3. Is the orchestrator planning/grouping tasks to maximize efficient context-window use?
+
+**Verdict:** MIXED — Good parallelization structure; weak on pre-planning.
+
+**Evidence:**
+- Fact-sheet: Pre-spawn-1 segment: 15 tool calls, 75,718 bytes read. This is heavy read-ahead before delegating.
+- Lines 1–94: The orchestrator read broadly (ToolSearch for deferred tools, WebFetch for docs, 12 Bash calls for inventory, 1 Read). This is exploratory/orientation work.
+- Once orientation was complete, the orchestrator delegated to agents that could parallelize: spawn #1 and #2 are independent (different data sources: settings/skills vs. session transcripts). No coordination between them. Good.
+- After-spawn-2 segment: 15 tool calls, 8100 bytes read. This is synthetic/analysis work between agent results (Bash for intersecting data, measurements, hook timing — not duplicating what agents did).
+- After-spawn-3 segment: 12 tool calls, 5712 bytes read. This is synthesis/application work (editing settings, writing docs).
+
+**Analysis:**  
+The orchestrator did substantial pre-work (75KB read) before spawning agents. This is justified for orientation, but the question is: did it read files the agents would need to read anyway?
+
+The agents' prompts specify their sources clearly (e.g., "Look in ~/.claude/settings.json..."), suggesting the orchestrator didn't pre-read and pass file contents as context (which would bloat the subagent prompts). This is good context discipline. However, the pre-spawn read included some files later read by agents (e.g., settings.json appears in both the orchestrator's pre-read and agent prompts), suggesting some redundancy.
+
+**Grouped well:** Three independent tasks → delegated in parallel (both async spawns at lines 95–97 are background, non-blocking).
+
+---
+
+### 4. Is the orchestrator avoiding reading files it does NOT need (that the subagent would read anyway)?
+
+**Verdict:** MIXED — Mostly good; some redundancy detected.
+
+**Evidence:**
+- Orchestrator's Reads (4 calls): 
+  1. `/tmp/.../tool-results/toolu_01B9mmZvD4pgvmGYRYBi8QsP.txt` — reading its own prior tool output. Necessary.
+  2. `/home/jared/systems-admin/docs/incidents.md` — not directly mentioned in any agent prompt. Supporting context for synthesis.
+  3. `/home/jared/.claude/settings.json` — **this file is explicitly requested in agent spawn #1's prompt** ("~/.claude/settings.json and ~/.claude/settings.local.json"). Orchestrator read it, agent also reads it. **Redundancy.**
+  4. `/home/jared/.claude/projects/-home-jared-systems-admin/memory/MEMORY.md` — session's own project memory, used for synthesis.
+
+- Pre-spawn-1 Bash calls included: "Survey session files by project," "Aggregate tool usage across all sessions," "Full aggregation incl subagents." These are inventory/aggregation, not file-contents reads.
+
+**Analysis:**  
+The orchestrator read `~/.claude/settings.json` before delegating to spawn #1, which also reads the same file. This suggests the orchestrator was orienting to the data before delegating, not trusting the agent to explore independently. The agents' prompts are exploratory ("Look in ~/.claude/skills/ and in installed plugins") and don't require pre-staging of file contents.
+
+**One critical check:** None of the agent prompts contain large file contents (e.g., no "here is your settings.json:..."). The prompts are instructions + paths, not data. This is correct. The orchestrator's redundant read of settings.json was for its own synthesis, not for pre-staging the agent.
+
+---
+
+### 5. Is the orchestrator sharing too much context with subagents (filling their windows / clouding judgment)?
+
+**Verdict:** PASS — Prompts are concise and instructional; no unnecessary context dumps.
+
+**Evidence:**
+- Spawn #1 prompt (Line 95): 2662 chars. Structured as: problem statement (1 sentence), 5 numbered investigation directives (specific file paths, measurement method), deliverable format. No file contents. No system-prompt injections. Clean.
+- Spawn #2 prompt (Line 97): 2397 chars. Structured as: problem (1 sentence), file locations (with technical note about path escaping), 4 concrete example categories, what to report per example, goal statement. No file contents. Clean.
+- Spawn #3 prompt (Line 212): 2065 chars. Structured as: problem (1 sentence), 3 numbered specific questions, instruction to verify via official docs. No file contents. Clean.
+
+**Result sizes:**
+- Spawn #1 result: 7517 chars (3x the prompt) — agent returned structured data (tables, descriptions). Expected.
+- Spawn #2 result: 6095 chars (2.5x the prompt) — agent returned 4 example session case-studies. Expected.
+- Spawn #3 result: 5290 chars (2.5x the prompt) — agent returned answers to 3 questions. Expected.
+
+**Analysis:**  
+The prompts are instruction-heavy and data-light. They guide the agent to read specific paths / sources directly, rather than passing pre-fetched contents. This is efficient and leaves maximum window space in the subagent for reading / reasoning. No bloat detected.
+
+---
+
+### 6. Is the orchestrator even following the ORCHESTRATION.md instructions?
+
+**Verdict:** FAIL — Does not pass `model` explicitly in any spawn; otherwise follows the spirit.
+
+**Evidence:**
+
+**Explicit policy (from ORCHESTRATION.md as of the session date, 2026-06-30):**
+```
+- Do single-file, ≤2-tool-call ops directly. Don't delegate them. Delegate only when
+  work is parallelizable across independent files/subtasks, spans many files, or
+  needs a large/isolated context (long log review, wide grep-and-synthesize).
+- Every `Agent` spawn passes `model` explicitly. Default `haiku` for mechanical
+  file-edit/shell work; `sonnet` for anything requiring judgment; `opus` only for
+  genuinely hard reasoning.
+- A short orienting Read before delegating is fine when the target file/path is
+  uncertain. Don't delegate the orienting step itself.
+```
+
+**Compliance check:**
+
+1. **"Do single-file, ≤2-tool-call ops directly."**  
+   ✓ PASS — All three agent-delegated tasks span multiple files or are large research tasks. No small ops were delegated.
+
+2. **"Every `Agent` spawn passes `model` explicitly."**  
+   ✗ FAIL — Lines 95, 97, 212: **All three spawns omit the `model` parameter.** The policy requires it to be explicit.
+   ```json
+   // Line 95 (should include "model": "haiku")
+   {"name": "Agent", "input": {"description": "Inventory...", "subagent_type": "general-purpose", "prompt": "..."}}
+   
+   // Line 97 (should include "model": "sonnet")
+   {"name": "Agent", "input": {"description": "Pull session...", "subagent_type": "general-purpose", "prompt": "..."}}
+   
+   // Line 212 (should include "model": "haiku")  
+   {"name": "Agent", "input": {"description": "Verify skill...", "subagent_type": "claude-code-guide", "prompt": "..."}}
+   ```
+
+3. **"A short orienting Read before delegating is fine."**  
+   ✓ PASS — The 75KB pre-spawn-1 read is orientation (exploring what tools exist, where transcripts live). It does not hand-hold or pre-fetch entire configs for agents to parse. Appropriate.
+
+**Analysis:**  
+The orchestrator follows the *spirit* of ORCHESTRATION.md (delegate well-scoped, multi-file work; avoid bloating prompts; do light orientation) but **violates the explicit syntactic requirement** that every Agent spawn include a `model` parameter. This is a clear, fixable defect.
+
+---
+
+### 7. Is the orchestrator requesting/receiving back only the context it needs, rather than a full context dump from the subagent?
+
+**Verdict:** PASS — Return values are structured and focused; no raw dumps.
+
+**Evidence:**
+- Spawn #1 deliverable instruction: *"Deliverable (return as structured markdown, this is the data I need back — be concrete with numbers): A ranked table: contributor category → total chars → est. tokens → # items. Top 15 individual heaviest skill/agent descriptions..."*  
+  Result: Structured markdown with tables and brief descriptions. Not a directory dump or full file contents. ✓
+
+- Spawn #2 deliverable instruction: *"Report: Project + short session id, One-line summary... The DISTINCT set of tools actually used... Which broad capabilities were NOT used at all... Any Skill invocations... Return structured markdown."*  
+  Result: 4 case-study sections with the requested fields. Not a raw JSONL dump. ✓
+
+- Spawn #3 deliverable instruction: *"Return a tight, factual answer to each of the three, flagging anything the docs don't clearly state."*  
+  Result: 3 paragraphs answering the 3 questions. Not a doc-dump. ✓
+
+**Analysis:**  
+All three agent results are precisely bounded to what was requested. No agent returned unfiltered file contents, full directory listings, or raw JSONL. The orchestrator specified expected output format (structured markdown, tables, case studies) in each prompt, and agents complied. Efficient.
+
+---
+
+## Detailed Findings
+
+### Model Selection Defect
+
+The most actionable finding is the absence of explicit `model` parameters. This violates the policy and incurs unnecessary cost:
+
+| Spawn | Task | Recommended Model | Resolved Model | Cost Ratio | Impact |
+|-------|------|------|---|---|---|
+| #1 | Inventory (char-count, read-only) | haiku | opus | ~3.5x | Overspent by ~2.5x input cost on a mechanical task |
+| #2 | Example selection (judgment) | sonnet | opus | ~1.5x | Overspent by ~0.5x on judgment task; haiku would likely fail |
+| #3 | Doc verification (lookup) | haiku | haiku | 1x | Correct; lucked into right default via subagent type |
+
+Estimated cost overrun on this session: **~$0.XX** (exact numbers would require token count from subagent transcripts, which this audit cannot access without exceeding scope). For a single session, negligible; systemically across projects, material.
+
+### Context Reuse Opportunity
+
+The orchestrator read `~/.claude/settings.json` before spawn #1, then spawn #1 reads it again. This is not a bloat issue (settings.json is small, ~5–10KB), but it suggests the orchestrator could rely more on agents to explore independently rather than self-orienting on data the agent is designed to discover.
+
+**Candidate rule refinement:** "A short orienting Read before delegating is fine *when needed for task planning or decision-making*, but avoid reading the same data the agent's prompt instructs it to gather."
+
+---
+
+## Recommendations
+
+1. **Fix: Require explicit `model` parameter in every Agent spawn.**  
+   - Add `"model": "haiku"` to spawn #1 (inventory/char-count).
+   - Add `"model": "sonnet"` to spawn #2 (example selection/judgment).
+   - Add `"model": "haiku"` to spawn #3 (doc lookup) — already resolved correctly, but make it explicit.
+
+2. **Consider: Clarify the "orienting Read" exception.**  
+   - Current rule allows "short orienting Read before delegating." Refine to avoid reading data the agent's prompt explicitly instructs it to gather.
+   - Or: Accept that dual-reading small files (~5–10KB) is acceptable cost for clear, self-contained agent prompts (no hand-staging).
+
+3. **No structural issues detected** with task decomposition, prompt design, or result handling. All three delegations were well-scoped and produced focused deliverables.
+
+---
+
+## Appendix: Tool & Context Tracking
+
+**Orchestrator tool calls (117 turns, main loop):**
+- Pre-spawn-1 (lines 1–94): 15 tool calls, 75,718 bytes read
+  - ToolSearch, WebFetch, 12× Bash (exploration), 1× Read
+- Span 1→2 (lines 95–96): 0 tool calls (async agents launching)
+- Between spawn-2 and spawn-3 (lines 97–211): 15 tool calls, 8,100 bytes read
+  - 12× Bash (measurement/analysis), 1× ScheduleWakeup, 1× Read, 1× Write
+- After spawn-3 (lines 212 onwards): 12 tool calls, 5,712 bytes read
+  - 3× Edit (settings.json, audit doc, project memory)
+  - 2× Read (tool results, settings context)
+  - 2× Skill (to-prd, handoff)
+  - 2× Bash (validation, issue tracker check)
+  - 3× Write (final audit docs)
+
+**Subagent spawns:** 3 (all async background; no sequential coupling)
+
+**No missed delegations:** Fact-sheet heuristic detected no runs of ≥4 same-tool calls without delegation.
+
+---
+
+_Audit completed 2026-07-06. Report path: `/tmp/claude-1000/-home-jared-dev-cc-os/3c15edee-8ac7-4ffe-b6ee-ee6a6ebd670d/scratchpad/audits/S9-report.md`_

docs/orchestration-audit/factsheets/S1-df546b88.md

@@ -0,0 +1,48 @@
+# Delegation fact-sheet: df546b88-e27d-47e1-888c-648012c24e62.jsonl
+
+- Transcript: `/home/jared/.claude/projects/-home-jared-dev-cc-os/df546b88-e27d-47e1-888c-648012c24e62.jsonl`
+- cwd: /home/jared/dev/cc-os
+- started: 2026-07-06T15:32:42.854Z
+- ended: 2026-07-06T18:20:38.267Z
+- assistant_turns: 66
+- human_prompts: 11
+- main_loop_models: {"claude-fable-5" => 66}
+- jsonl_lines: 278
+
+## Agent spawns (10)
+
+| # | line | type | model param | resolved model | bg | prompt chars | result chars | description |
+|---|------|------|-------------|----------------|----|--------------|--------------|-------------|
+| 1 | 21 | general-purpose | sonnet | claude-haiku-4-5-20251001 | n | 1098 | 7172 | Research Storybloq project |
+| 2 | 23 | general-purpose | sonnet | claude-haiku-4-5-20251001 | n | 1612 | 18293 | Survey self-hosted kanban tools |
+| 3 | 25 | general-purpose | sonnet | claude-haiku-4-5-20251001 | n | 2121 | 18564 | Research agent-native/markdown backlogs |
+| 4 | 79 | perspectives:devils-advocate | sonnet | claude-haiku-4-5-20251001 | n | 3280 | 9249 | Devils-advocate on vault backlog |
+| 5 | 81 | perspectives:simplifier | sonnet | claude-haiku-4-5-20251001 | n | 2122 | 6673 | Simplifier on vault backlog |
+| 6 | 83 | perspectives:implementer | sonnet | claude-haiku-4-5-20251001 | n | 2531 | 9344 | Implementer on vault backlog |
+| 7 | 85 | perspectives:premortem | sonnet | claude-haiku-4-5-20251001 | n | 2267 | 12426 | Premortem on vault backlog |
+| 8 | 202 | Explore | sonnet | claude-haiku-4-5-20251001 | n | 1120 | 8145 | Survey Hermes agent OS repo |
+| 9 | 218 | general-purpose | sonnet | claude-haiku-4-5-20251001 | n | 1832 | 12430 | Planka maturity deep-dive |
+| 10 | 219 | general-purpose | sonnet | claude-haiku-4-5-20251001 | n | 2301 | 7828 | Modern kanban alternatives sweep |
+
+## Orchestrator tool profile (segments split at each Agent spawn)
+
+| segment | calls | bytes read | per-tool counts |
+|---------|-------|------------|------------------|
+| pre-spawn-1 | 0 | 0 |  |
+| after-spawn-1 | 0 | 0 |  |
+| after-spawn-2 | 0 | 0 |  |
+| after-spawn-3 | 0 | 0 |  |
+| after-spawn-4 | 0 | 0 |  |
+| after-spawn-5 | 0 | 0 |  |
+| after-spawn-6 | 0 | 0 |  |
+| after-spawn-7 | 4 | 970 | Skill:1 Bash:1 Write:2 |
+| after-spawn-8 | 0 | 0 |  |
+| after-spawn-9 | 0 | 0 |  |
+| after-spawn-10 | 6 | 7106 | Bash:3 Read:1 Write:1 Edit:1 |
+
+## Candidate missed delegations (runs of >=4 same-tool calls, distinct targets, no Agent spawn between)
+
+None flagged by heuristic.
+
+Heuristic candidates require auditor judgment; sequential-dependent work is a valid reason not to delegate.
+

docs/orchestration-audit/factsheets/S10-f6a224d0.md

@@ -0,0 +1,31 @@
+# Delegation fact-sheet: f6a224d0-ddc2-488c-b0cc-728fdf21cc08.jsonl
+
+- Transcript: `/home/jared/.claude/projects/-home-jared-systems-admin/f6a224d0-ddc2-488c-b0cc-728fdf21cc08.jsonl`
+- cwd: /home/jared/systems-admin
+- started: 2026-07-01T12:37:51.119Z
+- ended: 2026-07-01T14:02:57.530Z
+- assistant_turns: 53
+- human_prompts: 8
+- main_loop_models: {"<synthetic>" => 1, "claude-sonnet-5" => 52}
+- jsonl_lines: 183
+
+## Agent spawns (0)
+
+None.
+
+## Orchestrator tool profile (segments split at each Agent spawn)
+
+| segment | calls | bytes read | per-tool counts |
+|---------|-------|------------|------------------|
+| pre-spawn-1 | 25 | 33271 | Read:4 Bash:12 Edit:4 Write:5 |
+
+## Candidate missed delegations (runs of >=4 same-tool calls, distinct targets, no Agent spawn between)
+
+- **Write x4** (jsonl lines 125-137, 4 distinct targets)
+  - `/home/jared/dev/cc-plugins/orchestration/.claude-plugin/plugin.json`
+  - `/home/jared/dev/cc-plugins/orchestration/ORCHESTRATION.md`
+  - `/home/jared/dev/cc-plugins/orchestration/hooks/inject.py`
+  - `/home/jared/dev/cc-plugins/orchestration/hooks/hooks.json`
+
+Heuristic candidates require auditor judgment; sequential-dependent work is a valid reason not to delegate.
+

docs/orchestration-audit/factsheets/S2-d68aab1b.md

@@ -0,0 +1,54 @@
+# Delegation fact-sheet: d68aab1b-3e3d-467c-9b9c-70b89e2576b0.jsonl
+
+- Transcript: `/home/jared/.claude/projects/-home-jared-dev-cc-os/d68aab1b-3e3d-467c-9b9c-70b89e2576b0.jsonl`
+- cwd: /home/jared/dev/cc-os
+- started: 2026-07-06T14:44:50.025Z
+- ended: 2026-07-06T17:58:03.100Z
+- assistant_turns: 207
+- human_prompts: 18
+- main_loop_models: {"<synthetic>" => 1, "claude-fable-5" => 206}
+- jsonl_lines: 643
+
+## Agent spawns (8)
+
+| # | line | type | model param | resolved model | bg | prompt chars | result chars | description |
+|---|------|------|-------------|----------------|----|--------------|--------------|-------------|
+| 1 | 28 | perspectives:devils-advocate | sonnet | claude-haiku-4-5-20251001 | n | 2566 | 9661 | Challenge eval validity claims |
+| 2 | 30 | perspectives:measurement | sonnet | claude-haiku-4-5-20251001 | n | 2483 | 12982 | Assess eval measurement value |
+| 3 | 72 | general-purpose | sonnet | claude-haiku-4-5-20251001 | n | 2565 | 2905 | Run 3-rep haiku W3 check |
+| 4 | 74 | general-purpose | opus | claude-haiku-4-5-20251001 | n | 6132 | 8289 | Build Eval C ambiguity ladder |
+| 5 | 76 | general-purpose | sonnet | claude-haiku-4-5-20251001 | n | 6759 | 5337 | Write/upgrade vault eval notes |
+| 6 | 296 | git-context:commit | haiku | claude-haiku-4-5-20251001 | n | 74 | 759 | Commit current changes |
+| 7 | 506 | git-context:commit | haiku | claude-haiku-4-5-20251001 | n | 473 | 1354 | Commit current changes |
+| 8 | 628 | git-context:commit | haiku | claude-haiku-4-5-20251001 | n | 496 | 1152 | Commit plan docs |
+
+## Orchestrator tool profile (segments split at each Agent spawn)
+
+| segment | calls | bytes read | per-tool counts |
+|---------|-------|------------|------------------|
+| pre-spawn-1 | 0 | 0 |  |
+| after-spawn-1 | 0 | 0 |  |
+| after-spawn-2 | 0 | 0 |  |
+| after-spawn-3 | 0 | 0 |  |
+| after-spawn-4 | 0 | 0 |  |
+| after-spawn-5 | 37 | 41558 | Bash:21 Edit:9 Read:5 ToolSearch:1 SendMessage:1 |
+| after-spawn-6 | 32 | 61527 | Bash:14 Read:11 Edit:6 Write:1 |
+| after-spawn-7 | 15 | 24354 | Read:3 Edit:5 Bash:3 Write:4 |
+| after-spawn-8 | 1 | 563 | Bash:1 |
+
+## Candidate missed delegations (runs of >=4 same-tool calls, distinct targets, no Agent spawn between)
+
+- **Read x5** (jsonl lines 389-405, 5 distinct targets)
+  - `/home/jared/dev/cc-os/plugins/os-adr/eval-c/scenarios/P1-L1-execution.md`
+  - `/home/jared/dev/cc-os/plugins/os-adr/eval-c/scenarios/P2-L2-execution.md`
+  - `/home/jared/dev/cc-os/plugins/os-adr/eval-c/scenarios/P3-L3-execution.md`
+  - `/home/jared/dev/cc-os/plugins/os-adr/eval-c/judge-rubric.md`
+  - `/home/jared/dev/cc-os/plugins/os-adr/eval-c/bin/check`
+- **Write x4** (jsonl lines 611-624, 4 distinct targets)
+  - `/home/jared/dev/cc-os/docs/plans/2026-07-06-plugin-evals-overview.md`
+  - `/home/jared/dev/cc-os/docs/plans/ws1-orchestration-audit.md`
+  - `/home/jared/dev/cc-os/docs/plans/ws2-os-vault-write-eval.md`
+  - `/home/jared/dev/cc-os/docs/plans/ws3-status-convention-plugin.md`
+
+Heuristic candidates require auditor judgment; sequential-dependent work is a valid reason not to delegate.
+

docs/orchestration-audit/factsheets/S3-9f45afcc.md

@@ -0,0 +1,38 @@
+# Delegation fact-sheet: 9f45afcc-75ff-4b7a-8f3d-008aa0d599af.jsonl
+
+- Transcript: `/home/jared/.claude/projects/-home-jared-dev-cc-os/9f45afcc-75ff-4b7a-8f3d-008aa0d599af.jsonl`
+- cwd: /home/jared/dev/cc-os
+- started: 2026-07-04T16:17:45.947Z
+- ended: 2026-07-04T18:52:42.079Z
+- assistant_turns: 255
+- human_prompts: 25
+- main_loop_models: {"claude-fable-5" => 255}
+- jsonl_lines: 850
+
+## Agent spawns (5)
+
+| # | line | type | model param | resolved model | bg | prompt chars | result chars | description |
+|---|------|------|-------------|----------------|----|--------------|--------------|-------------|
+| 1 | 124 | general-purpose | sonnet | claude-haiku-4-5-20251001 | n | 4849 | 3951 | Fix plugin caches, add refresh tooling |
+| 2 | 133 | general-purpose | sonnet | claude-haiku-4-5-20251001 | y | 2177 | 5905 | Re-run Eval B grid post-fix |
+| 3 | 171 | general-purpose | sonnet | claude-haiku-4-5-20251001 | y | 49 | 878 | Resume eval grid agent |
+| 4 | 603 | general-purpose | sonnet | claude-haiku-4-5-20251001 | n | 2193 | 1606 | Triage wording on sonnet |
+| 5 | 605 | general-purpose | haiku | claude-haiku-4-5-20251001 | n | 2193 | 1235 | Triage wording on haiku |
+
+## Orchestrator tool profile (segments split at each Agent spawn)
+
+| segment | calls | bytes read | per-tool counts |
+|---------|-------|------------|------------------|
+| pre-spawn-1 | 19 | 23658 | Bash:14 Read:4 Edit:1 |
+| after-spawn-1 | 0 | 0 |  |
+| after-spawn-2 | 0 | 0 |  |
+| after-spawn-3 | 48 | 29966 | ToolSearch:2 SendMessage:1 Bash:22 Edit:12 Skill:2 Read:5 Write:3 TaskStop:1 |
+| after-spawn-4 | 0 | 0 |  |
+| after-spawn-5 | 43 | 14410 | Write:4 Bash:27 Edit:11 Read:1 |
+
+## Candidate missed delegations (runs of >=4 same-tool calls, distinct targets, no Agent spawn between)
+
+None flagged by heuristic.
+
+Heuristic candidates require auditor judgment; sequential-dependent work is a valid reason not to delegate.
+

docs/orchestration-audit/factsheets/S4-b6b07a91.md

@@ -0,0 +1,110 @@
+# Delegation fact-sheet: b6b07a91-5aa3-4521-8476-c6749ee76017.jsonl
+
+- Transcript: `/home/jared/.claude/projects/-home-jared-dev-llf-schema/b6b07a91-5aa3-4521-8476-c6749ee76017.jsonl`
+- cwd: /home/jared/dev/llf-schema
+- started: 2026-06-30T14:34:11.662Z
+- ended: 2026-06-30T19:12:36.755Z
+- assistant_turns: 191
+- human_prompts: 20
+- main_loop_models: {"claude-opus-4-8" => 191}
+- jsonl_lines: 766
+
+## Agent spawns (41)
+
+| # | line | type | model param | resolved model | bg | prompt chars | result chars | description |
+|---|------|------|-------------|----------------|----|--------------|--------------|-------------|
+| 1 | 21 | general-purpose | (ABSENT) | claude-opus-4-8 | n | 1275 | 14627 | Gather openspec change context |
+| 2 | 36 | general-purpose | sonnet | claude-sonnet-4-6 | n | 1667 | 2031 | Task 1.1 fix gbp_url gate |
+| 3 | 38 | general-purpose | sonnet | claude-sonnet-4-6 | n | 4457 | 3827 | Tasks 2.1 2.2 3.1 verifier profiles |
+| 4 | 40 | general-purpose | haiku | claude-haiku-4-5-20251001 | n | 1818 | 2142 | Task 3.3 coverage checklist doc |
+| 5 | 73 | general-purpose | haiku | claude-haiku-4-5-20251001 | n | 512 | 749 | Mark batch 1 tasks complete |
+| 6 | 75 | general-purpose | sonnet | claude-sonnet-4-6 | n | 3684 | 2730 | Task 1.2 regenerate golden fixture |
+| 7 | 77 | general-purpose | sonnet | claude-sonnet-4-6 | n | 3979 | 2923 | Task 3.2 self-check flag |
+| 8 | 102 | general-purpose | haiku | claude-haiku-4-5-20251001 | n | 436 | 749 | Mark batch 2 tasks complete |
+| 9 | 104 | general-purpose | sonnet | claude-sonnet-4-6 | n | 3652 | 3547 | Tasks 4.1 4.2 seed and verify loop |
+| 10 | 106 | general-purpose | sonnet | claude-sonnet-4-6 | n | 3469 | 3230 | Task 5.1 verify-schema skill |
+| 11 | 132 | general-purpose | sonnet | claude-sonnet-4-6 | n | 3798 | 3992 | Reconcile manifest office paths |
+| 12 | 134 | general-purpose | haiku | claude-haiku-4-5-20251001 | n | 456 | 1022 | Mark batch 3 tasks complete |
+| 13 | 155 | general-purpose | sonnet | claude-sonnet-4-6 | n | 1011 | 1246 | Fix stale attorney URL in test-deploy |
+| 14 | 167 | general-purpose | sonnet | claude-sonnet-4-6 | n | 1998 | 2055 | Task 6.1 verify gate |
+| 15 | 185 | general-purpose | sonnet | claude-sonnet-4-6 | n | 2537 | 2210 | Task 6.2 branch and commit |
+| 16 | 199 | general-purpose | haiku | claude-haiku-4-5-20251001 | n | 659 | 867 | Mark task 6.2 complete |
+| 17 | 219 | general-purpose | sonnet | claude-sonnet-4-6 | n | 3462 | 10703 | Verify changes align with ADRs |
+| 18 | 234 | general-purpose | sonnet | claude-sonnet-4-6 | n | 1907 | 1664 | Fix stale docblock and amend |
+| 19 | 249 | general-purpose | sonnet | claude-sonnet-4-6 | n | 3164 | 3400 | Task 6.3 run release.sh minor |
+| 20 | 265 | general-purpose | sonnet | claude-sonnet-4-6 | n | 3499 | 7567 | Diagnose NJ office duplicate LegalService |
+| 21 | 267 | general-purpose | sonnet | claude-sonnet-4-6 | n | 3644 | 5041 | Diagnose service-page WebPage description |
+| 22 | 298 | general-purpose | sonnet | claude-sonnet-4-6 | n | 3587 | 4916 | Check live HQ office gbp_url both sites |
+| 23 | 366 | general-purpose | sonnet | claude-sonnet-4-6 | n | 2774 | 6262 | Authoritative working-tree status snapshot |
+| 24 | 411 | claude | (ABSENT) | claude-opus-4-8 | n | 2766 | 2211 | Commit harden-schema-verification work |
+| 25 | 427 | claude | (ABSENT) | claude-opus-4-8 | n | 2126 | 2144 | Run release.sh patch to v0.2.5 |
+| 26 | 440 | claude | (ABSENT) | claude-opus-4-8 | n | 2311 | 3023 | Check live service-page WebPage.description |
+| 27 | 466 | Explore | (ABSENT) | claude-haiku-4-5-20251001 | n | 1901 | 2238 | Check spec + golden fixture for WebPage.description |
+| 28 | 494 | Explore | (ABSENT) | claude-haiku-4-5-20251001 | n | 2160 | 5364 | How non-service pages set description |
+| 29 | 496 | claude | (ABSENT) | claude-opus-4-8 | n | 2423 | 2564 | Check post_excerpt on live SDD pages |
+| 30 | 529 | claude | (ABSENT) | claude-opus-4-8 | n | 5292 | 2958 | Relax description check, commit, re-release |
+| 31 | 563 | claude | (ABSENT) | claude-opus-4-8 | n | 854 | 1264 | Push branch and tags |
+| 32 | 581 | claude | haiku | claude-haiku-4-5-20251001 | n | 2727 | 940 | Write WebPage.description memory |
+| 33 | 623 | general-purpose | sonnet | claude-sonnet-4-6 | n | 3200 | 3557 | Verify PLD contact-us schema |
+| 34 | 625 | general-purpose | sonnet | claude-sonnet-4-6 | n | 4030 | 3832 | Verify SDD contact-us schema |
+| 35 | 664 | general-purpose | haiku | claude-haiku-4-5-20251001 | n | 1019 | 2320 | Sample-verify PLD schema |
+| 36 | 666 | general-purpose | haiku | claude-haiku-4-5-20251001 | n | 1014 | 2575 | Sample-verify SDD schema |
+| 37 | 670 | general-purpose | sonnet | claude-sonnet-4-6 | n | 2776 | 749 | File blocked areaServed issue |
+| 38 | 696 | general-purpose | sonnet | claude-sonnet-4-6 | n | 2615 | 749 | Diagnose SDD stale schema |
+| 39 | 712 | general-purpose | haiku | claude-haiku-4-5-20251001 | n | 1578 | 1607 | Re-verify SDD post cache purge |
+| 40 | 725 | general-purpose | sonnet | claude-sonnet-4-6 | n | 2247 | 749 | Check SDD active plugin version |
+| 41 | 748 | general-purpose | sonnet | claude-sonnet-4-6 | n | 5170 | 2351 | Finalize v0.2.5: issue, notes, merge |
+
+## Orchestrator tool profile (segments split at each Agent spawn)
+
+| segment | calls | bytes read | per-tool counts |
+|---------|-------|------------|------------------|
+| pre-spawn-1 | 0 | 0 |  |
+| after-spawn-1 | 0 | 0 |  |
+| after-spawn-2 | 0 | 0 |  |
+| after-spawn-3 | 0 | 0 |  |
+| after-spawn-4 | 0 | 0 |  |
+| after-spawn-5 | 0 | 0 |  |
+| after-spawn-6 | 0 | 0 |  |
+| after-spawn-7 | 0 | 0 |  |
+| after-spawn-8 | 0 | 0 |  |
+| after-spawn-9 | 0 | 0 |  |
+| after-spawn-10 | 0 | 0 |  |
+| after-spawn-11 | 0 | 0 |  |
+| after-spawn-12 | 0 | 0 |  |
+| after-spawn-13 | 0 | 0 |  |
+| after-spawn-14 | 1 | 0 | AskUserQuestion:1 |
+| after-spawn-15 | 0 | 0 |  |
+| after-spawn-16 | 0 | 0 |  |
+| after-spawn-17 | 0 | 0 |  |
+| after-spawn-18 | 0 | 0 |  |
+| after-spawn-19 | 0 | 0 |  |
+| after-spawn-20 | 0 | 0 |  |
+| after-spawn-21 | 0 | 0 |  |
+| after-spawn-22 | 1 | 0 | AskUserQuestion:1 |
+| after-spawn-23 | 0 | 0 |  |
+| after-spawn-24 | 0 | 0 |  |
+| after-spawn-25 | 0 | 0 |  |
+| after-spawn-26 | 0 | 0 |  |
+| after-spawn-27 | 0 | 0 |  |
+| after-spawn-28 | 0 | 0 |  |
+| after-spawn-29 | 1 | 0 | AskUserQuestion:1 |
+| after-spawn-30 | 0 | 0 |  |
+| after-spawn-31 | 0 | 0 |  |
+| after-spawn-32 | 0 | 0 |  |
+| after-spawn-33 | 0 | 0 |  |
+| after-spawn-34 | 1 | 0 | Skill:1 |
+| after-spawn-35 | 0 | 0 |  |
+| after-spawn-36 | 0 | 0 |  |
+| after-spawn-37 | 0 | 0 |  |
+| after-spawn-38 | 0 | 0 |  |
+| after-spawn-39 | 0 | 0 |  |
+| after-spawn-40 | 3 | 0 | ToolSearch:1 TaskStop:1 AskUserQuestion:1 |
+| after-spawn-41 | 0 | 0 |  |
+
+## Candidate missed delegations (runs of >=4 same-tool calls, distinct targets, no Agent spawn between)
+
+None flagged by heuristic.
+
+Heuristic candidates require auditor judgment; sequential-dependent work is a valid reason not to delegate.
+

docs/orchestration-audit/factsheets/S5-aadeb66c.md

@@ -0,0 +1,44 @@
+# Delegation fact-sheet: aadeb66c-0d64-4d41-b407-83e0209428b4.jsonl
+
+- Transcript: `/home/jared/.claude/projects/-home-jared-dev-llf-schema/aadeb66c-0d64-4d41-b407-83e0209428b4.jsonl`
+- cwd: /home/jared/dev/llf-schema
+- started: 2026-06-29T16:51:34.375Z
+- ended: 2026-06-29T19:10:30.177Z
+- assistant_turns: 36
+- human_prompts: 9
+- main_loop_models: {"claude-sonnet-4-6" => 36}
+- jsonl_lines: 161
+
+## Agent spawns (8)
+
+| # | line | type | model param | resolved model | bg | prompt chars | result chars | description |
+|---|------|------|-------------|----------------|----|--------------|--------------|-------------|
+| 1 | 23 | general-purpose | (ABSENT) | claude-sonnet-4-6 | n | 838 | 2352 | Audit codebase for contact-us schema support |
+| 2 | 24 | general-purpose | (ABSENT) | claude-sonnet-4-6 | n | 589 | 792 | Check live SDD contact-us page for schema output |
+| 3 | 41 | general-purpose | (ABSENT) | claude-sonnet-4-6 | n | 674 | 2917 | Check SDD office ACF data via wp-cli |
+| 4 | 60 | general-purpose | (ABSENT) | claude-sonnet-4-6 | n | 419 | 7842 | Read codebase for ContactPagePiece implementation context |
+| 5 | 83 | general-purpose | (ABSENT) | claude-sonnet-4-6 | n | 635 | 5363 | Read additional context files for implementation |
+| 6 | 99 | general-purpose | (ABSENT) | claude-sonnet-4-6 | n | 804 | 12076 | Read remaining files for implementation spec |
+| 7 | 115 | general-purpose | sonnet | claude-sonnet-4-6 | n | 6090 | 2294 | Implement ContactPagePiece with tests |
+| 8 | 141 | general-purpose | haiku | claude-haiku-4-5-20251001 | n | 419 | 13465 | Verify ContactPagePiece implementation files |
+
+## Orchestrator tool profile (segments split at each Agent spawn)
+
+| segment | calls | bytes read | per-tool counts |
+|---------|-------|------------|------------------|
+| pre-spawn-1 | 0 | 0 |  |
+| after-spawn-1 | 0 | 0 |  |
+| after-spawn-2 | 0 | 0 |  |
+| after-spawn-3 | 0 | 0 |  |
+| after-spawn-4 | 0 | 0 |  |
+| after-spawn-5 | 0 | 0 |  |
+| after-spawn-6 | 0 | 0 |  |
+| after-spawn-7 | 0 | 0 |  |
+| after-spawn-8 | 0 | 0 |  |
+
+## Candidate missed delegations (runs of >=4 same-tool calls, distinct targets, no Agent spawn between)
+
+None flagged by heuristic.
+
+Heuristic candidates require auditor judgment; sequential-dependent work is a valid reason not to delegate.
+

docs/orchestration-audit/factsheets/S6-3dc26da3.md

@@ -0,0 +1,30 @@
+# Delegation fact-sheet: 3dc26da3-de82-4523-aca5-40c06b3616b3.jsonl
+
+- Transcript: `/home/jared/.claude/projects/-home-jared-clients-philly-search-engine-marketing/3dc26da3-de82-4523-aca5-40c06b3616b3.jsonl`
+- cwd: /home/jared/clients/philly-search-engine-marketing
+- started: 2026-06-29T15:08:49.497Z
+- ended: 2026-06-29T16:58:40.721Z
+- assistant_turns: 21
+- human_prompts: 13
+- main_loop_models: {"<synthetic>" => 1, "claude-sonnet-4-6" => 20}
+- jsonl_lines: 120
+
+## Agent spawns (1)
+
+| # | line | type | model param | resolved model | bg | prompt chars | result chars | description |
+|---|------|------|-------------|----------------|----|--------------|--------------|-------------|
+| 1 | 98 | general-purpose | (ABSENT) | claude-sonnet-4-6 | n | 2039 | 1811 | Update TODO and session log |
+
+## Orchestrator tool profile (segments split at each Agent spawn)
+
+| segment | calls | bytes read | per-tool counts |
+|---------|-------|------------|------------------|
+| pre-spawn-1 | 0 | 0 |  |
+| after-spawn-1 | 0 | 0 |  |
+
+## Candidate missed delegations (runs of >=4 same-tool calls, distinct targets, no Agent spawn between)
+
+None flagged by heuristic.
+
+Heuristic candidates require auditor judgment; sequential-dependent work is a valid reason not to delegate.
+

docs/orchestration-audit/factsheets/S7-3771b288.md

@@ -0,0 +1,39 @@
+# Delegation fact-sheet: 3771b288-eb9d-4398-9e75-e1ff0ba6af03.jsonl
+
+- Transcript: `/home/jared/.claude/projects/-home-jared-servers/3771b288-eb9d-4398-9e75-e1ff0ba6af03.jsonl`
+- cwd: /home/jared/servers
+- started: 2026-07-04T16:05:40.895Z
+- ended: 2026-07-04T17:14:47.297Z
+- assistant_turns: 144
+- human_prompts: 11
+- main_loop_models: {"<synthetic>" => 1, "claude-sonnet-5" => 143}
+- jsonl_lines: 456
+
+## Agent spawns (3)
+
+| # | line | type | model param | resolved model | bg | prompt chars | result chars | description |
+|---|------|------|-------------|----------------|----|--------------|--------------|-------------|
+| 1 | 241 | Explore | (ABSENT) | claude-haiku-4-5-20251001 | n | 2282 | 7537 | Explore ovh-prod service setup conventions |
+| 2 | 243 | Explore | (ABSENT) | claude-haiku-4-5-20251001 | n | 2725 | 3562 | Explore desktop backup bug specifics |
+| 3 | 245 | Explore | (ABSENT) | claude-haiku-4-5-20251001 | n | 2126 | 4776 | Explore proxmox VM 101 and umbrella structure |
+
+## Orchestrator tool profile (segments split at each Agent spawn)
+
+| segment | calls | bytes read | per-tool counts |
+|---------|-------|------------|------------------|
+| pre-spawn-1 | 37 | 74129 | Read:4 Bash:32 Edit:1 |
+| after-spawn-1 | 0 | 0 |  |
+| after-spawn-2 | 0 | 0 |  |
+| after-spawn-3 | 36 | 37170 | AskUserQuestion:1 Write:6 ToolSearch:1 ExitPlanMode:1 Read:3 Edit:6 Bash:18 |
+
+## Candidate missed delegations (runs of >=4 same-tool calls, distinct targets, no Agent spawn between)
+
+- **Write x5** (jsonl lines 345-353, 5 distinct targets)
+  - `/home/jared/servers/proxmox-ubuntu/CLAUDE.md`
+  - `/home/jared/servers/proxmox-ubuntu/README.md`
+  - `/home/jared/servers/proxmox-ubuntu/docs/services-inventory.md`
+  - `/home/jared/servers/proxmox-ubuntu/docs/roadmap.md`
+  - `/home/jared/servers/ovh-prod/docs/incidents.md`
+
+Heuristic candidates require auditor judgment; sequential-dependent work is a valid reason not to delegate.
+

docs/orchestration-audit/factsheets/S8-3fc7bb8c.md

@@ -0,0 +1,27 @@
+# Delegation fact-sheet: 3fc7bb8c-54c1-4147-ac3d-dc7db3fca924.jsonl
+
+- Transcript: `/home/jared/.claude/projects/-home-jared-servers/3fc7bb8c-54c1-4147-ac3d-dc7db3fca924.jsonl`
+- cwd: /home/jared/servers
+- started: 2026-07-04T17:20:10.755Z
+- ended: 2026-07-04T18:04:17.446Z
+- assistant_turns: 90
+- human_prompts: 5
+- main_loop_models: {"claude-fable-5" => 90}
+- jsonl_lines: 236
+
+## Agent spawns (0)
+
+None.
+
+## Orchestrator tool profile (segments split at each Agent spawn)
+
+| segment | calls | bytes read | per-tool counts |
+|---------|-------|------------|------------------|
+| pre-spawn-1 | 44 | 32250 | Read:3 Bash:27 Write:4 ToolSearch:2 ExitPlanMode:1 Edit:6 Skill:1 |
+
+## Candidate missed delegations (runs of >=4 same-tool calls, distinct targets, no Agent spawn between)
+
+None flagged by heuristic.
+
+Heuristic candidates require auditor judgment; sequential-dependent work is a valid reason not to delegate.
+

docs/orchestration-audit/factsheets/S9-2525fa2d.md

@@ -0,0 +1,34 @@
+# Delegation fact-sheet: 2525fa2d-1b15-4495-99bf-c093716fab55.jsonl
+
+- Transcript: `/home/jared/.claude/projects/-home-jared-systems-admin/2525fa2d-1b15-4495-99bf-c093716fab55.jsonl`
+- cwd: /home/jared/systems-admin
+- started: 2026-06-30T19:23:34.370Z
+- ended: 2026-06-30T20:22:50.581Z
+- assistant_turns: 117
+- human_prompts: 6
+- main_loop_models: {"claude-opus-4-8" => 117}
+- jsonl_lines: 302
+
+## Agent spawns (3)
+
+| # | line | type | model param | resolved model | bg | prompt chars | result chars | description |
+|---|------|------|-------------|----------------|----|--------------|--------------|-------------|
+| 1 | 95 | general-purpose | (ABSENT) | claude-opus-4-8 | n | 2662 | 7517 | Inventory loaded context surface |
+| 2 | 97 | general-purpose | (ABSENT) | claude-opus-4-8 | n | 2397 | 6095 | Pull session examples |
+| 3 | 212 | claude-code-guide | (ABSENT) | claude-haiku-4-5-20251001 | n | 2065 | 5290 | Verify skill loading mechanism |
+
+## Orchestrator tool profile (segments split at each Agent spawn)
+
+| segment | calls | bytes read | per-tool counts |
+|---------|-------|------------|------------------|
+| pre-spawn-1 | 15 | 75718 | ToolSearch:1 WebFetch:1 Bash:12 Read:1 |
+| after-spawn-1 | 0 | 0 |  |
+| after-spawn-2 | 15 | 8100 | Bash:12 ScheduleWakeup:1 Read:1 Write:1 |
+| after-spawn-3 | 12 | 5712 | Edit:3 Read:2 Bash:2 Skill:2 Write:3 |
+
+## Candidate missed delegations (runs of >=4 same-tool calls, distinct targets, no Agent spawn between)
+
+None flagged by heuristic.
+
+Heuristic candidates require auditor judgment; sequential-dependent work is a valid reason not to delegate.
+

docs/plans/2026-07-06-plugin-evals-overview.md

@@ -0,0 +1,44 @@
+# Plan set: second-batch plugin evals + status-check convention
+
+_Created: 2026-07-06. Status: proposed — not started. Trigger each workstream plan in its own session._
+
+Three workstreams, distilled from the first eval batch (os-adr Evals A/B/C) and the user's
+2026-07-06 direction. Each has its own plan file; this file carries sequencing and shared
+discipline only.
+
+## Workstreams
+
+1. **WS1 — os-orchestration session audit** → `ws1-orchestration-audit.md`
+   Evidence-gathering over real session transcripts; produces failure-mode clusters and an
+   eval-scenario backlog. No design dependencies — start anytime.
+   **DONE 2026-07-06:** findings in `docs/orchestration-audit/2026-07-06-findings.md`
+   (4 verified clusters, E1–E4 scenario backlog); extractor at
+   `plugins/os-orchestration/audit/bin/extract`.
+2. **WS2 — os-vault write-behavior eval** → `ws2-os-vault-write-eval.md`
+   Two quick wins (stale write-skill contract fix; cc-os hub note) can run immediately.
+   The eval harness itself should start after WS3's design lands (hub-note automation is a
+   status-check consumer) and after the quick wins (baseline measures post-bugfix behavior).
+3. **WS3 — status-check convention + glue plugin** → `ws3-status-convention-plugin.md`
+   Architecture: OpenSpec change + ADR. Runs in parallel with WS1.
+
+## Recommended sequencing
+
+- Session A: WS1 (phases 1–2 direct, phase 3 fan-out, phase 4 synthesis).
+- Session B (parallel-safe): WS3 OpenSpec proposal + WS2 quick wins.
+- Session C (after A+B): WS2 harness build + baseline grid; then decide whether a wording
+  loop is needed.
+
+## Shared discipline (from the first batch — non-negotiable)
+
+Before designing or running ANY eval below, Read:
+
+- `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md` (run modes, cache
+  refresh, canary cell, reps, pipefail/rescore gotchas)
+- `~/Documents/SecondBrain/eval-methodology-ladder.md` (paired positives/negatives, run-set
+  vs frozen reserve, per-level pass bars, checker-conformance dry-run, self-test blind spot)
+- `~/Documents/SecondBrain/eval-methodology-irl-feedback-loop.md` (production audit → new
+  scenarios)
+
+Standing rules: held-out Task blocks are never run informally; all reps counted (no
+discards); orchestrating session acts as final judge — verify every agent claim against
+primary evidence (TSVs, raw transcripts) before accepting it; commit only when the user asks.

docs/plans/ws1-orchestration-audit.md

@@ -0,0 +1,84 @@
+# WS1 — os-orchestration session audit
+
+_Created: 2026-07-06. Status: **complete 2026-07-06** — findings + scenario backlog in
+`docs/orchestration-audit/2026-07-06-findings.md`; extractor shipped at
+`plugins/os-orchestration/audit/bin/extract`. See `2026-07-06-plugin-evals-overview.md`._
+
+## Goal
+
+Audit real Claude Code sessions to assess whether delegation under the global
+`os-orchestration` plugin is working efficiently, collect concrete miss examples with
+triggers/criteria, and distill them into an eval-scenario backlog. This instantiates the
+IRL-feedback-loop methodology (production evidence first, scenarios from real misses) —
+NOT a lab eval yet.
+
+## The seven audit questions (fixed rubric — verbatim from the user)
+
+1. Are subagents getting called when they should be?
+2. Is the correct model chosen per subagent — highest reasonable quality at lowest cost?
+3. Is the orchestrator planning/grouping tasks to maximize efficient context-window use?
+4. Is the orchestrator avoiding reading files it does NOT need (that the subagent would
+   read anyway)?
+5. Is the orchestrator sharing too much context with subagents (filling their windows /
+   clouding judgment)?
+6. Is the orchestrator even following the ORCHESTRATION.md instructions?
+7. Is the orchestrator requesting/receiving back only the context it needs, rather than a
+   full context dump from the subagent?
+
+## Phase 1 — deterministic extractor (orchestrator directly; one script)
+
+Raw transcripts are multi-MB jsonl; feeding them whole to auditors violates the principle
+being audited. Write one script (Python or Ruby; Ruby preferred per user style if it stays
+clean) that reduces a session jsonl to a delegation fact-sheet:
+
+- Every `Agent` tool_use: agent type, `model` param present? which?, prompt char count,
+  run_in_background, result char count returned to orchestrator.
+- Orchestrator's own tool profile: Read/Grep/Glob/Bash counts and total bytes read,
+  bucketed before/after each Agent spawn.
+- Candidate missed delegations: runs of ≥N same-type tool calls across independent files
+  with no Agent spawn (mechanical heuristic; auditor judges).
+- Session metadata: project dir, date, total turns, model of main loop if recoverable.
+
+Transcripts live under `~/.claude/projects/<flattened-path>/*.jsonl`.
+This extractor is dual-use: it becomes the checker core for any later orchestration eval.
+
+## Phase 2 — sample selection (orchestrator, from fact-sheets not vibes)
+
+~8–10 sessions, stratified:
+- 3 × cc-os (`-home-jared-dev-cc-os`, 70 sessions available) — heavy orchestration use,
+  but note contamination: these sessions were actively working ON the orchestration text.
+- 3 × client/dev projects (`philly-search-engine-marketing` 28, `llf-schema` 26,
+  `viking-warrior-training-log` 18).
+- 2 × ops (`-home-jared-servers` 19, `-home-jared-systems-admin` 17).
+- Bias recent (post-2026-07-03, when os-orchestration went global — earlier sessions ran
+  under different/per-project orchestration text and are a different population).
+- Include sessions WITH delegation and large sessions with NONE despite apparent
+  parallelizable work.
+
+## Phase 3 — parallel audits (sonnet subagents, one per session)
+
+Each auditor receives: the fact-sheet, the transcript path (for targeted spot-reads only —
+instruct them NOT to read the whole file), the ORCHESTRATION.md text (10 lines), and the
+seven questions. Required output per question: verdict + cited evidence (jsonl line refs) +
+if a miss, a candidate eval trigger/criterion ("when session state X, orchestrator should
+Y"). Model: sonnet (judgment work). Spawn all auditors concurrently.
+
+## Phase 4 — synthesis (orchestrator as final judge)
+
+Verify each claimed miss against the primary transcript before accepting (standing rule —
+auditor reports have contradicted primary evidence before). Cluster confirmed misses into
+failure modes; each cluster → an eval-scenario candidate with a measurable trigger.
+
+Deliverables:
+- Findings doc: `docs/orchestration-audit/2026-07-XX-findings.md` (per-question summary,
+  per-session evidence table, failure-mode clusters).
+- Scenario backlog appended to the findings doc — inputs to a future eval design, NOT an
+  eval yet.
+
+## Eval-design note (deferred, do not start in this workstream)
+
+The Eval B headless pattern does not transfer directly: orchestration behavior is
+mid-session and depends on task shape, not a SessionStart trigger. Likely shape: scripted
+multi-file tasks run headless, scored by the Phase 1 extractor + thresholds (did it
+delegate? model passed? orchestrator bytes-read under budget?). Design only after Phase 4
+findings exist — scenarios must come from real misses, not imagination.

docs/plans/ws2-os-vault-write-eval.md

@@ -0,0 +1,94 @@
+# WS2 — os-vault "when and how to write" eval
+
+_Created: 2026-07-06. Status: proposed. Prereqs: quick wins immediate; harness after WS3
+design + quick wins. See `2026-07-06-plugin-evals-overview.md`._
+
+## Goal
+
+Make os-vault reliably know WHEN to write a vault note unprompted and HOW to write it
+correctly (frontmatter contract), measured by a held-out discrimination eval — applying the
+full os-adr first-batch playbook.
+
+## Known defects found 2026-07-06 (fix before baselining)
+
+1. **`skills/write/SKILL.md` contradicts the reconciled vault-conventions.md.** The skill
+   still mandates `scope/global` as a TAG and a type list of
+   `procedure|reference|log|hub|concept|decision`; vault-conventions (Phase 1 SB content
+   plan, 2026-06-30) made `scope:` a frontmatter FIELD and the live vault uses types like
+   `howto` and `eval-results`. Fix: make vault-conventions.md the single source of truth —
+   the skill summarizes + points, never duplicates the schema. Then `bin/refresh-plugins`.
+2. **No trigger-conditioned wording anywhere in os-vault.** The core Eval B lesson
+   (when→then phrasing; mechanical triggers for lower tiers; each rule lives where its
+   precondition is visible) was never applied. Nothing tells the model when to invoke
+   `/os-vault:write` unprompted. Do NOT fix this before the baseline grid — it is the
+   tuning surface the eval measures.
+3. **cc-os has no project hub note** despite hub notes being the connectivity mechanism
+   (ADR-014). Quick win: author it now (one note). AUTOMATIC hub creation is designed into
+   `onboard-project` + the WS3 status check — not hacked in here.
+
+Quick wins = items 1 and 3. Both are small; do directly or via one haiku subagent each.
+
+## Behavior under test
+
+Unprompted vault-write discrimination during a normal working session:
+
+- **Positives:** session surfaces evergreen cross-project knowledge (tool/API behavior
+  discovered, client-specific fact, a convention established with the user) → model writes
+  or offers a vault note with contract-conforming frontmatter (summary at creation, scope
+  field, type/ + ≥1 facet tag).
+- **Negatives (equally tempting):** ephemeral material — in-progress task state,
+  project-local debugging detail, decisions already recorded in repo docs/ADRs → no vault
+  write. Over-trigger fail line: writing ephemera to the vault (or duplicating
+  repo-recorded knowledge).
+
+Ambiguity ladder: L1 explicit cue ("worth remembering") → L2 moderate → L3 conceptual
+(knowledge emerges implicitly from debugging). Paired positive/negative per level.
+
+## Harness (Eval B/C shape)
+
+- Location: `plugins/os-vault/eval/`. Run-set + frozen reserve authored upfront (2×), in
+  different knowledge domains. **Task blocks held-out — never run informally; reserve
+  never read informally.**
+- **Headless-only** (`claude -p`, cwd = sandbox fixture project, real SessionStart hook
+  fires). In-session subagents are invalid for unprompted-behavior measurement.
+- Sandbox must use an ISOLATED vault path (env-pointed test vault), never the real
+  `~/Documents/SecondBrain` — the checker inspects the sandbox vault for written notes.
+  Verify the plugin's vault path is env-overridable (config.yaml); if not, that's a small
+  pre-harness plugin change.
+- Checker: Ruby (Sandi Metz style), deterministic-first. Axis (a): did it write/offer a
+  vault note (mechanical from transcript tool_use + sandbox vault diff). Axis (b):
+  frontmatter contract conformance — almost fully mechanical (summary present, scope
+  field, required tag groups); narrow frozen haiku judge only for "offered but did not
+  write" phrasing, stubbable via env var. Negatives: sole fail line = unneeded vault
+  write; consultation informational. TSV mode for grids.
+
+## Discipline gates (all mandatory before the first grid)
+
+1. Conformance dry-run on paper: would a perfectly write-skill-compliant model pass every
+   cell? Would an always-write model pass positives but fail negatives?
+2. Model-free self-test including at least one fabricated SHIPPED-INSTRUCTION-COMPLIANT
+   transcript (not just the designer's imagined ideal).
+3. Canary cell first (one cell, hand-verify TSV vs raw transcript, count the result).
+4. `bin/refresh-plugins` before every grid; check TSV row counts == expected reps
+   (pipefail/tee gotcha); rescore-don't-discard on instrument fixes.
+
+## Sequence
+
+1. Quick wins (contract fix + hub note) → refresh caches.
+2. Build harness (fixture project + isolated vault + 6-ADR-equivalent seeded vault history
+   for negatives to cite): fixture generation → sonnet subagent; checker → sonnet subagent
+   (Ruby); scenario authoring + conformance dry-run → ORCHESTRATOR DIRECTLY (Eval C's
+   defects came from exactly this step; do not delegate it).
+3. **Baseline grid untuned** (post-bugfix, pre-trigger-wording): run-set × {sonnet, haiku}
+   × 3 reps. This measures the actual gap.
+4. If gapped: `/autoresearch` wording loop per the howto note (checker/fixtures/scenarios
+   frozen; only wording moves; reduced inner grid; full grid to confirm). Expected surfaces:
+   session_start hook note, write SKILL.md description + body, possibly a CLAUDE.md
+   trigger-phrased section template (mirror os-adr's).
+5. Results → vault `eval-results` note (template `_templates/eval-results.md`); update
+   cc-os CLAUDE.md pointers.
+
+## Models
+
+Sonnet for fixture/checker construction and any judging; haiku for mechanical edits;
+orchestrator authors scenarios and acts as final judge over all grid claims.

docs/plans/ws3-status-convention-plugin.md

@@ -0,0 +1,91 @@
+# WS3 — per-plugin SessionStart status-check convention + glue plugin
+
+_Created: 2026-07-06. Status: proposed. Prereq: none (parallel with WS1). See
+`2026-07-06-plugin-evals-overview.md`._
+
+## Goal
+
+A convention-over-configuration status system: at SessionStart, every installed cc-os
+plugin verifies its expected per-project artifacts exist (adr: ADR system present; vault:
+project hub note exists — optionally injecting it as context). All good → silent. Problem →
+one short banner prompting corrective action ("<xyz> missing — run /os-vault:… to fix").
+Plugins stay independent; new plugins get picked up automatically by following the
+convention; only the glue plugin is mandatory for the collection.
+
+This is architecture → **OpenSpec change + ADR** (use `openspec-propose`; decisions land in
+`docs/memory-system/03-architecture-decisions.md`). Do not build outside that workflow.
+
+## User's sketch (the starting design)
+
+Per-plugin Python status scripts following a cc-os convention (naming, interface,
+location), living in a specific gitignored hidden directory in the project. SessionStart
+fires a cc-os master status script that enumerates all status scripts in that directory
+(excluding itself) and runs them. Adding a plugin later = drop in a conforming script; no
+global rewrites.
+
+## The one design tension the proposal must settle (pressure-test, don't fiat)
+
+**Where does status-check CODE live?**
+
+- **Option A (user sketch): project-local dir** (e.g. `.cc-os/status.d/*.py`, gitignored).
+  Pro: per-project opt-in/customization. Con: code copies drift across projects; needs an
+  install/update step per project per plugin; the 2026-07-04 stale-cache incident shows
+  copy-drift is a real failure class here.
+- **Option B: plugin-source enumeration** — each plugin ships `status/check.py` in its own
+  source; the master enumerates installed `os-*` plugins at session start. Same modularity
+  and zero-global-rewrite extension; no copies, no drift, no install step. Project-local
+  `.cc-os/` still exists but holds STATE only (suppress files, per-project config like
+  "this project opts out of hub-note checks"), mirroring os-adr's `.os-adr/` and
+  os-doc-hygiene's `.dochygiene/` — consolidating those is itself a candidate scope item.
+
+Run `perspectives:simplifier` and `perspectives:implementer` (both sonnet) against both
+shapes inside the proposal. Lean: B for code, A-style dir for state.
+
+## Interface contract (draft for the proposal)
+
+- Each check: stdin/argv gets project root + plugin config; stdout JSON
+  `{"status": "ok"|"warn", "message": str, "context": str|null}`; hard timeout (~5s);
+  non-zero exit or timeout = treated as warn with a generic message, never blocks session.
+- **Silent on ok** — os-adr's near-zero-token SessionStart discipline is the template.
+  Master aggregates warns into ONE short banner; per-project once-per-day snooze +
+  permanent suppress via state dir (pattern already proven in os-adr/os-doc-hygiene).
+- `context` field = optional session-context injection (the "pull the hub note into
+  context if useful" idea) — but gate it through injection-economics thinking (see the
+  design-template skill's filter); default null.
+
+## Glue plugin scope ("cc-os collection manager")
+
+Naming: per `~/Documents/SecondBrain/cc-os-plugin-skill-naming-convention.md` (read before
+naming; `os-[domain]`, verb-first skills, NO `name:` frontmatter). Candidates: `os-core`
+(lean) or `os-status` (narrower). Settle in the proposal.
+
+Phase 1 (this change): master SessionStart hook + enumeration + interface contract +
+convert the two existing consumers:
+- os-adr: refactor its existing existence-check hook into a conforming status check (it is
+  already exactly this pattern — present→usage note, absent→once-per-day suggestion).
+- os-vault: new hub-note check (project hub note exists for the current project; missing →
+  prompt to run onboarding/creation; optionally inject on present).
+
+Phase 2 (separate change, later): collection-manager duties — verify installed cc-os
+plugin caches are fresh vs source (would have auto-caught the 2026-07-04 stale-cache
+incident), migrations complete, conventions followed (e.g. no `name:` in SKILL.md
+frontmatter). Effectively an automated `bin/refresh-plugins` advisor.
+
+## Deliverables
+
+1. OpenSpec change (design.md resolving the tension above + specs + tasks) — via
+   `openspec-propose`.
+2. ADR in `docs/memory-system/03-architecture-decisions.md` (status-check convention +
+   code-location decision).
+3. Implementation via `openspec-apply-change`: master hook + shared contract module
+   (Python, matching os-vault's deep-module style: thin entry points, shared config/io
+   modules), conforming checks for os-adr + os-vault, model-free tests (Python, mirroring
+   `tests/hook_test.py` style) + invariants.md.
+4. `bin/refresh-plugins`, `claude plugin details` verification, fresh-session smoke test.
+
+## Models
+
+Design synthesis + ADR: orchestrator. Perspective reviews: sonnet. Implementation: haiku
+for thin scripts/tests, sonnet for the shared contract module. No behavioral eval needed —
+deterministic code; invariants + model-free tests suffice (same rationale as the os-adr
+hook).

openspec/changes/archive/2026-07-03-add-os-adr-eval-b-harness/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-07-03

openspec/changes/archive/2026-07-03-add-os-adr-eval-b-harness/design.md

@@ -0,0 +1,127 @@
+## Context
+
+`os-adr` ships with `docs/adr-system/04-plugin-requirements.md` requirements 4–5: the plugin
+must be effective at *unprompted* write-trigger recognition and *unprompted, correct* retrieval.
+Eval A (`plugins/os-adr/eval/`) only measures skill-*following* once explicitly invoked — its
+runner-prompt literally tells the model under test "the user has invoked `/os-adr:{{SKILL}}`".
+That architecture cannot answer requirements 4–5 by construction: the thing being measured
+(does the model notice on its own) is exactly what the runner prompt would be giving away.
+
+Eval A's driver used in-session Agent-tool subagents pinned to a model tier, and treated the
+missing SessionStart hook context as an acceptable fidelity gap ("irrelevant — these are
+explicit invocations"). For Eval B that gap is not acceptable: the SessionStart hook's existence
+note (naming `/os-adr:create` and `/os-adr:find`) is one of the two legitimate discovery paths
+being measured (the other being the model reasoning its own way there without any hook hint,
+e.g. via `docs/adr/` conventionally existing). If the hook never fires, we can't tell "the model
+didn't notice" from "the model was never given the hint a real session would have."
+
+Scenario shapes are frozen and out of scope here (`docs/adr-system/06-eval-scenarios.md`, 7
+scenarios: W1–W3 write-trigger, R1–R4 retrieval) — this design is about the execution harness
+only.
+
+## Goals / Non-Goals
+
+**Goals:**
+- A runner that executes the model-under-test in an environment where the real SessionStart
+  hook fires, real skills/CLIs are reachable, and no part of the prompt names the ADR system.
+- Fixtures realistic enough to support all 7 scenarios, including R2's distractor ADRs (2–3
+  near-miss, one Superseded) and R4's one-hop graph reachability (needs a real `graphify-out/`).
+- A checker that scores both axes (unprompted-consultation, correctness) with as much
+  determinism as possible, reserving LLM judgment only for genuinely ambiguous cases (e.g. did
+  the model's prose count as "proposing" an ADR).
+- Sandbox isolation so headless runs never mutate the canonical fixture or the real cc-os repo.
+- Output format (TSV or similar) compatible with a later `autoresearch` Classic-mode loop.
+
+**Non-Goals:**
+- Running the grid, scoring a real pass rate, or iterating on retrieval/trigger heuristics —
+  that is the deferred next stage, not this change.
+- Adding, removing, or reshaping scenario shapes in `06-eval-scenarios.md`.
+- Multi-machine or CI execution — local headless runs only, matching Eval A's scope.
+
+## Decisions
+
+**1. Headless (`claude -p`) is the sole runner mode; no subagent fallback.**
+Eval A offers both an in-session subagent path (cheaper, preferred) and a headless fallback
+(`bin/run-headless`) for full fidelity. Eval B inverts that: only headless mode is valid, because
+Agent-tool subagents share the parent session's already-loaded plugin/hook state and cwd
+semantics in ways that don't reproduce a fresh SessionStart firing against the sandbox directory.
+`claude -p --output-format stream-json --verbose --model <tier> --cwd <sandbox>` gives a fresh
+process, a real SessionStart event, and a full transcript to check against.
+*Alternative considered*: reuse Eval A's subagent shortcut for cost — rejected, since it would
+silently reintroduce the exact fidelity gap this eval exists to close.
+
+**2. Transcript-based mechanical check for axis (a) (unprompted consultation).**
+`--output-format stream-json --verbose` emits every tool_use block. The checker greps the JSONL
+for: a Skill/slash invocation matching `os-adr:*`, a `Bash` call matching `bin/adr-*`, or a
+`Read`/`Glob` touching `docs/adr/`. This is fully mechanical — no model judgment needed to
+detect "did it touch the ADR system at all."
+*Alternative considered*: ask an LLM judge to read the transcript and decide — rejected for axis
+(a) specifically, since tool-call presence is unambiguous and cheaper to check by grep.
+
+**3. Axis (b) (correctness) is deterministic where possible, LLM-judged only at the edges.**
+For R1–R4 (retrieval), the fixture encodes the single correct ADR ID; the checker mechanically
+confirms the transcript's retrieved/cited ADR ID matches (or, for R2, that it is NOT one of the
+seeded distractor IDs). For W1–W3 (write-trigger), "proposes... or asks whether to" is prose-
+shaped and not fully mechanical — a new `docs/adr/NNNN-*.md` file appearing in the sandbox is a
+mechanical PASS; absent that, a small rubric-bound LLM judge reads only the final transcript
+message (not the full transcript) and answers a yes/no: "did the agent explicitly propose
+recording this decision as an ADR, or ask whether to?" Kept as thin as possible to limit judgment
+surface, mirroring the `os-doc-hygiene`/`os-adr` classify-only-the-ambiguous-part pattern already
+used elsewhere in this repo.
+*Alternative considered*: fully mechanical W1–W3 checking via keyword grep on the final message
+(e.g. "ADR") — rejected as too brittle/gameable; a judge with a narrow, fixed rubric is more
+robust and still cheap since it reads one short message, not a transcript.
+
+**4. Dedicated Eval-B fixture project(s), not reuse of Eval A's fixtures or the real llf-schema
+pilot.**
+Eval A's fixtures are too small (4 ADRs, no graph) and R2/R4 need specific engineered content
+(a Superseded pair plus 2–3 near-misses; a real one-hop-only conflict via Graphify). Building
+`plugins/os-adr/eval-b/fixture/` fresh, with `docs/adr/` populated per-scenario and a genuinely
+built `graphify-out/` (via the real `graphify` binary against the fixture's own small codebase,
+not a hand-authored stub) keeps the graph-layer degradation check (R4) honest.
+*Alternative considered*: reuse the llf-schema pilot project directly — rejected; its exclude
+list and ADR set are project-specific and not shaped for the distractor/one-hop scenarios,
+and would tie the eval's stability to project state a migration exercise might later change.
+
+**5. Per-cell repeat count is a runner flag, default left as an open question for the eval-
+running stage, not fixed by this design.**
+Unprompted behavior is more stochastic than the explicit-invocation behavior Eval A measures.
+The harness supports `--reps N` on the sandbox/runner tooling, but this change does not decide
+whether the eventual grid uses N=1 or N=3 — that's a call for whoever runs the eval, informed by
+observed variance in a pilot run.
+
+## Risks / Trade-offs
+
+- **[Risk] Headless runs perform real, uncontained tool use (file edits, possibly destructive
+  commands) against the sandbox.** → Mitigation: every run is a fresh git-initialized sandbox
+  copy (mirrors Eval A's `bin/sandbox` pattern) under a scratch/tmp root, never the canonical
+  fixture or the cc-os repo; sandbox is disposable and diffed/discarded after checking.
+- **[Risk] Cost — headless full-session runs are pricier than Eval A's subagent shortcut.** →
+  Mitigation: accepted deliberately (see Decision 1); this is the whole reason the fidelity gap
+  matters here. Reps default is left open (Decision 5) specifically to let the eval-running
+  stage control cost/variance trade-off with real data instead of a guess baked in now.
+- **[Risk] LLM-judged axis (b) for W1–W3 could itself be inconsistent or gameable over
+  `autoresearch` iterations.** → Mitigation: judge rubric and prompt are frozen alongside the
+  checker/fixtures/scenarios during any future `autoresearch` loop (same discipline Eval A's
+  README already imposes on its own checker), and Decision 3 keeps the judge's input surface
+  (one final message, not a transcript) as narrow as possible.
+- **[Risk] A genuinely-built `graphify-out/` for the fixture could drift if `graphify`'s output
+  format changes.** → Mitigation: the fixture's `graphify-out/` is rebuilt by an explicit fixture
+  setup step (documented in the harness README), not committed as a frozen binary blob, so it's
+  always regenerated against whatever `graphify` version is installed.
+
+## Migration Plan
+
+Not applicable — this is a new, additive test-harness directory with no effect on shipped plugin
+behavior. No rollback beyond deleting `plugins/os-adr/eval-b/` if abandoned.
+
+## Open Questions
+
+- Default `--reps` per cell for the eventual grid run (left to the eval-running stage; see
+  Decision 5).
+- Whether the R4 degradation check (retrieval must fail without `graphify-out/`) needs its own
+  sandbox variant with the graph directory deliberately removed, or whether one fixture toggling
+  a flag suffices — resolve during tasks/implementation, not architecturally significant enough
+  to block this design.
+- Exact rubric wording for the W1–W3 LLM judge — drafted during implementation (tasks.md), not
+  fixed here.

openspec/changes/archive/2026-07-03-add-os-adr-eval-b-harness/proposal.md

@@ -0,0 +1,54 @@
+## Why
+
+The `os-adr` plugin's two hardest requirements — unprompted write-trigger recognition and
+unprompted correct retrieval (`04-plugin-requirements.md` reqs 4–5) — have never been measured.
+Eval A (`plugins/os-adr/eval/`) only proves the plugin executes correctly once explicitly
+invoked; it says nothing about whether an agent notices, on its own, that the ADR system is
+relevant. The scenario *shapes* for this held-out eval already exist
+(`docs/adr-system/06-eval-scenarios.md`) but no fixtures, prompts, checker, or runner exist yet.
+Building it now closes the last unmeasured requirement in the locked rollout order.
+
+## What Changes
+
+- Build a new eval harness, `plugins/os-adr/eval-b/`, sibling to but structurally distinct from
+  Eval A, for the 7 held-out scenarios (W1–W3 write-trigger, R1–R4 retrieval).
+- Author full scenario prompts/fixtures from the frozen shapes in `06-eval-scenarios.md` —
+  scenario *shapes* are not renegotiated by this change; only their execution artifacts are new.
+- Fixture project(s): a realistic, onboarded project state (populated `docs/adr/` incl. a
+  Superseded pair for R2 distractors, a built `graphify-out/` for R4) that the SessionStart hook
+  recognizes as ADR-initialized.
+- A **headless runner** (real `claude -p` in a real sandbox cwd) as the primary execution mode,
+  not an in-session Agent-tool subagent — because the SessionStart hook firing for real is part
+  of what's under test (Eval A's subagent shortcut is not valid here; see design.md).
+- A deterministic-first, two-axis checker: (a) did the agent consult/propose the ADR system
+  unprompted at all, (b) did it act on/write the *specific correct* thing. Axis (a) is
+  mechanically checkable (tool-call/transcript grep for `/os-adr:*` or `docs/adr/` reads/writes);
+  axis (b) requires a small LLM-judged correctness check against the fixture's known-correct
+  answer, kept as thin and rubric-bound as possible.
+- Sandbox/grid tooling mirroring Eval A's shape (`bin/sandbox`, `bin/check`, results.tsv) adapted
+  for headless execution and the two-axis score.
+- Wiring for the later `autoresearch` iteration stage (not run by this change — building the
+  harness only).
+
+Out of scope for this change: actually running the eval grid, iterating retrieval/trigger
+heuristics, or adding/altering scenario shapes in `06-eval-scenarios.md`.
+
+## Capabilities
+
+### New Capabilities
+- `adr-eval-b-harness`: fixtures, held-out scenario prompts, headless sandbox/runner tooling, and
+  a two-axis deterministic-first checker for the unprompted write-trigger and retrieval eval.
+
+### Modified Capabilities
+(none — this adds a new, separate eval surface; it does not change the behavior of
+`adr-authoring`, `adr-retrieval`, `adr-session-awareness`, or any shipped plugin capability)
+
+## Impact
+
+- New directory: `plugins/os-adr/eval-b/` (fixtures, scenarios, bin/, runner assets, README).
+- No changes to plugin source (`lib/adr/`, `bin/adr-*`, `hooks/`, `skills/`) — this is a pure
+  test-harness addition.
+- Depends on `claude` CLI headless mode (`claude -p`) being available in the build/dev
+  environment for the runner to execute against a real sandbox with hooks firing.
+- Once built, unlocks the deferred final stage of the locked os-adr rollout order (running the
+  grid + `autoresearch` iteration), tracked separately in `CLAUDE.md`.

openspec/changes/archive/2026-07-03-add-os-adr-eval-b-harness/specs/adr-eval-b-harness/spec.md

@@ -0,0 +1,67 @@
+## ADDED Requirements
+
+### Requirement: Held-out scenario fixtures
+The harness SHALL provide a dedicated fixture project (or projects) under
+`plugins/os-adr/eval-b/fixture/` that supports all 7 held-out scenarios (W1–W3, R1–R4) without
+any scenario prompt naming the ADR system, an ADR ID, or the exact constraint text being tested.
+
+#### Scenario: R2 distractor set present
+- **WHEN** the R2 fixture's `docs/adr/` is inspected
+- **THEN** it contains the one correct Accepted ADR plus 2–3 near-miss ADRs (same component
+  family, different decision, or Superseded status) that a naive retrieval could mistakenly cite
+
+#### Scenario: R4 graph reachability is real, not stubbed
+- **WHEN** the R4 fixture's `graphify-out/` is inspected
+- **THEN** it was produced by running the real `graphify` binary against the fixture's own
+  codebase (not hand-authored), and the conflicting files are one import/reference hop away from
+  files listed in the relevant ADR's `affected-paths`, not directly listed themselves
+
+### Requirement: Headless runner with real hook firing
+The harness SHALL execute each scenario via a headless `claude -p` process with its working
+directory set to a fresh sandbox copy of the fixture, so that the real SessionStart hook fires
+for the model under test. In-session Agent-tool subagents SHALL NOT be used as the execution
+mode for this eval.
+
+#### Scenario: SessionStart hook context reaches the model under test
+- **WHEN** a scenario is run via the headless runner against an ADR-initialized sandbox
+- **THEN** the transcript shows the SessionStart hook's additionalContext was present in the
+  model's context before its first action
+
+#### Scenario: Sandbox isolation
+- **WHEN** any scenario run completes (pass or fail)
+- **THEN** the canonical fixture directory and the cc-os repo itself show no modifications —
+  only the disposable sandbox copy was touched
+
+### Requirement: Two-axis deterministic-first checker
+The harness SHALL score each run on two independent axes: (a) whether the model consulted or
+proposed the ADR system at all, unprompted, and (b) whether it acted on or wrote the specific
+correct thing rather than a merely plausible one. Axis (a), and axis (b) for retrieval scenarios,
+SHALL be scored mechanically from the transcript and sandbox file state; axis (b) for
+write-trigger scenarios MAY fall back to a narrow, rubric-bound LLM judge only when no new ADR
+file was mechanically created.
+
+#### Scenario: Axis (a) mechanical detection
+- **WHEN** the checker scans a scenario's `stream-json` transcript
+- **THEN** it detects axis (a) as true if any tool_use block invokes an `os-adr:*` skill, a
+  `bin/adr-*` CLI, or a Read/Glob on `docs/adr/`, with no model call required
+
+#### Scenario: Axis (b) retrieval correctness
+- **WHEN** scoring an R1–R4 scenario's axis (b)
+- **THEN** the checker mechanically compares the ADR ID the transcript cites or acts on against
+  the fixture's pre-declared correct ID, and fails if it matches a seeded distractor instead
+
+#### Scenario: Axis (b) write-trigger fallback judge
+- **WHEN** scoring a W1–W3 scenario where no new `docs/adr/NNNN-*.md` file was created in the
+  sandbox
+- **THEN** the checker invokes a rubric-bound LLM judge on only the model's final message (not
+  the full transcript) to decide whether it explicitly proposed or asked about recording an ADR
+
+### Requirement: Grid-compatible output
+The harness SHALL emit per-run results in a format usable by a later `autoresearch` Classic-mode
+loop (scenario, model tier, axis-a result, axis-b result, pass/fail), analogous to Eval A's
+`results.tsv`.
+
+#### Scenario: TSV row per run
+- **WHEN** a scenario run is checked
+- **THEN** the checker can emit one TSV row identifying the scenario, model tier, both axis
+  results, and overall pass/fail, appendable to a shared results file

openspec/changes/archive/2026-07-03-add-os-adr-eval-b-harness/tasks.md

@@ -0,0 +1,85 @@
+## 1. Fixture project(s)
+
+- [x] 1.1 Scaffold `plugins/os-adr/eval-b/fixture/` as a small, self-contained fixture codebase
+      (init as its own git repo, mirroring Eval A's `fixture/project/` pattern)
+- [x] 1.2 Author `docs/adr/` content for W1–W3 (a plausible existing convention/decision history
+      the scenarios can plausibly reverse or extend) and initialize the ADR index
+- [x] 1.3 Author R1's Accepted ADR with `affected-paths` covering specific fixture files whose
+      obvious next change would violate it
+- [x] 1.4 Extend `docs/adr/` with R2's 2–3 distractor ADRs (near-miss same-component-family,
+      plus one Superseded) alongside the correct one
+- [x] 1.5 Add an R3 Accepted ADR whose content answers a "how should we…" question phrased in
+      different vocabulary than the ADR's own text
+- [x] 1.6 Design R4's one-hop layout: files that import/reference an R1-style ADR's
+      `affected-paths` without being listed themselves; run the real `graphify` binary to produce
+      `graphify-out/` and confirm the hop is actually one edge away in the resulting graph
+- [x] 1.7 Document a fixture regeneration step (README) that rebuilds `graphify-out/` from
+      scratch rather than committing it as a frozen blob (`bin/build-fixture-graph` +
+      "Fixture regeneration" section in `eval-b/README.md`; script asserts the R4 one-hop
+      invariant on every rebuild)
+
+## 2. Scenario prompts
+
+- [x] 2.1 Write `eval-b/scenarios/W1.md`..`W3.md` and `R1.md`..`R4.md`: a task prompt per
+      scenario that never names the ADR system, plugin, or exact constraint text, plus the
+      checker-facing metadata (correct ADR ID for R1–R4; expected new-ADR shape for W1–W3)
+      following the pass/fail language already sketched in `06-eval-scenarios.md`
+- [x] 2.2 Self-review each prompt against the "held-out" ground rule before running anything
+      against it (no informal trial runs — first real execution is the self-test in section 5)
+      (R1's "skip the shared client" wording was flagged and removed in this review)
+
+## 3. Headless runner
+
+- [x] 3.1 Write `eval-b/bin/sandbox <scenario> <dest>`: fresh git-initialized sandbox copy of the
+      right fixture variant (mirrors Eval A's `bin/sandbox`; `R4-nograph` variant included)
+- [x] 3.2 Write `eval-b/bin/run <scenario> <model> <sandbox>`: invokes
+      `claude -p --output-format stream-json --verbose --model <tier>` with cwd set to the
+      sandbox, passing only the scenario's task prompt (no system-level hints), capturing the
+      full JSONL transcript to a file in the sandbox
+- [x] 3.3 Confirm the os-adr plugin (and, for R4, graphify) are active in the environment the
+      runner executes in, and that a sandbox with an initialized `docs/adr/` triggers the
+      SessionStart hook's "present" branch (verified via a neutral non-scenario probe prompt:
+      haiku quoted the [os-adr] note verbatim; transcript carries a system/hook_response event)
+- [x] 3.4 Add `--reps N` support to the runner for repeated executions of the same cell (default
+      left open per design.md Decision 5; document how to override)
+
+## 4. Checker
+
+- [x] 4.1 Write `eval-b/bin/check <scenario> <sandbox> [--tsv <model>]`: parses the JSONL
+      transcript for axis (a) (tool_use blocks touching `os-adr:*`, `bin/adr-*`, or
+      `docs/adr/` reads/globs)
+- [x] 4.2 Implement axis (b) for R1–R4: compare the transcript's cited/acted-on ADR ID against
+      the scenario's pre-declared correct ID; fail on distractor match (R2) or on missing
+      graph-layer reach (R4)
+- [x] 4.3 Implement axis (b) for W1–W3: mechanical PASS if a new `docs/adr/NNNN-*.md` file
+      exists in the sandbox matching the scenario's expected shape; otherwise invoke the narrow
+      LLM judge (final message only) with a fixed rubric to decide propose/ask-vs-silent
+      (rubric frozen in `eval-b/judge-rubric.md`; stubbable via ADR_EVAL_B_JUDGE_CMD)
+- [x] 4.4 Emit the TSV row format (scenario, model tier, axis-a, axis-b, pass/fail) for
+      `autoresearch` compatibility
+- [x] 4.5 Write the R4 degradation-check variant: same scenario run against a sandbox with
+      `graphify-out/` removed, expected to FAIL where the graph-layer path was required
+      (`bin/sandbox R4-nograph` + `bin/check R4-nograph` alias)
+
+## 5. Self-test (mirrors Eval A's own-both-directions check)
+
+- [x] 5.1 For each of the 7 scenarios, hand-simulate a "perfect" transcript/sandbox state (an
+      agent that does the right thing) and confirm `bin/check` scores PASS on both axes
+      (scripted as `eval-b/bin/self-test`; includes the W judge-fallback path with a stub judge,
+      plus a one-off smoke test of the real haiku judge)
+- [x] 5.2 For each of the 7 scenarios, check an untouched sandbox (no ADR consultation at all)
+      and confirm `bin/check` scores axis (a) FAIL (also covers R4-nograph, the R2
+      superseded-distractor trap, and the missing-hook-context invalid-run case)
+- [x] 5.3 Confirm sandbox isolation: after a run, diff the canonical fixture and the cc-os repo
+      and verify neither was modified (fixture digest check inside self-test; git status clean
+      of unexpected entries)
+
+## 6. Documentation
+
+- [x] 6.1 Write `eval-b/README.md` mirroring Eval A's README shape: layout table, how to run a
+      single cell, how the two-axis scoring works, explicit warning that scenario prompts are
+      held-out and must not be informally tried out outside this harness's own self-test
+- [x] 6.2 Cross-link from `CLAUDE.md`'s os-adr section once this change is applied and archived
+      (status line update per this repo's "keep this file current" convention) — Eval B bullet
+      added to the os-adr component section, "Remaining" rollout line updated, and
+      `06-eval-scenarios.md` status line now points at the built harness

openspec/specs/adr-eval-b-harness/spec.md

@@ -0,0 +1,71 @@
+## Purpose
+
+Evaluation harness B: held-out, unprompted-behavior testing for the os-adr plugin. Tests whether the plugin's core write/retrieval operations surface naturally without coaching, across multiple model tiers, with deterministic-first scoring (mechanical checks before fallback to LLM judge).
+
+## Requirements
+
+### Requirement: Held-out scenario fixtures
+The harness SHALL provide a dedicated fixture project (or projects) under
+`plugins/os-adr/eval-b/fixture/` that supports all 7 held-out scenarios (W1–W3, R1–R4) without
+any scenario prompt naming the ADR system, an ADR ID, or the exact constraint text being tested.
+
+#### Scenario: R2 distractor set present
+- **WHEN** the R2 fixture's `docs/adr/` is inspected
+- **THEN** it contains the one correct Accepted ADR plus 2–3 near-miss ADRs (same component
+  family, different decision, or Superseded status) that a naive retrieval could mistakenly cite
+
+#### Scenario: R4 graph reachability is real, not stubbed
+- **WHEN** the R4 fixture's `graphify-out/` is inspected
+- **THEN** it was produced by running the real `graphify` binary against the fixture's own
+  codebase (not hand-authored), and the conflicting files are one import/reference hop away from
+  files listed in the relevant ADR's `affected-paths`, not directly listed themselves
+
+### Requirement: Headless runner with real hook firing
+The harness SHALL execute each scenario via a headless `claude -p` process with its working
+directory set to a fresh sandbox copy of the fixture, so that the real SessionStart hook fires
+for the model under test. In-session Agent-tool subagents SHALL NOT be used as the execution
+mode for this eval.
+
+#### Scenario: SessionStart hook context reaches the model under test
+- **WHEN** a scenario is run via the headless runner against an ADR-initialized sandbox
+- **THEN** the transcript shows the SessionStart hook's additionalContext was present in the
+  model's context before its first action
+
+#### Scenario: Sandbox isolation
+- **WHEN** any scenario run completes (pass or fail)
+- **THEN** the canonical fixture directory and the cc-os repo itself show no modifications —
+  only the disposable sandbox copy was touched
+
+### Requirement: Two-axis deterministic-first checker
+The harness SHALL score each run on two independent axes: (a) whether the model consulted or
+proposed the ADR system at all, unprompted, and (b) whether it acted on or wrote the specific
+correct thing rather than a merely plausible one. Axis (a), and axis (b) for retrieval scenarios,
+SHALL be scored mechanically from the transcript and sandbox file state; axis (b) for
+write-trigger scenarios MAY fall back to a narrow, rubric-bound LLM judge only when no new ADR
+file was mechanically created.
+
+#### Scenario: Axis (a) mechanical detection
+- **WHEN** the checker scans a scenario's `stream-json` transcript
+- **THEN** it detects axis (a) as true if any tool_use block invokes an `os-adr:*` skill, a
+  `bin/adr-*` CLI, or a Read/Glob on `docs/adr/`, with no model call required
+
+#### Scenario: Axis (b) retrieval correctness
+- **WHEN** scoring an R1–R4 scenario's axis (b)
+- **THEN** the checker mechanically compares the ADR ID the transcript cites or acts on against
+  the fixture's pre-declared correct ID, and fails if it matches a seeded distractor instead
+
+#### Scenario: Axis (b) write-trigger fallback judge
+- **WHEN** scoring a W1–W3 scenario where no new `docs/adr/NNNN-*.md` file was created in the
+  sandbox
+- **THEN** the checker invokes a rubric-bound LLM judge on only the model's final message (not
+  the full transcript) to decide whether it explicitly proposed or asked about recording an ADR
+
+### Requirement: Grid-compatible output
+The harness SHALL emit per-run results in a format usable by a later `autoresearch` Classic-mode
+loop (scenario, model tier, axis-a result, axis-b result, pass/fail), analogous to Eval A's
+`results.tsv`.
+
+#### Scenario: TSV row per run
+- **WHEN** a scenario run is checked
+- **THEN** the checker can emit one TSV row identifying the scenario, model tier, both axis
+  results, and overall pass/fail, appendable to a shared results file

plugins/os-adr/eval-b/README.md

@@ -0,0 +1,102 @@
+# os-adr Eval B — unprompted write-trigger & retrieval (held-out)
+
+_Last updated: 2026-07-03 — harness built and self-tested (perfect-run PASS,
+untouched-sandbox FAIL, both directions for all 7 scenarios). Grid run 2026-07-03, 1 rep/cell:
+haiku 0/8 PASS (never unprompted-consults the ADR system in any scenario); sonnet 5/8 PASS
+(fails W3 — doesn't propose recording the decision; fails R1 — misses the direct-conflict
+retrieval). R4-nograph FAILed on both tiers as expected (degradation check; only meaningful
+paired with an R4 PASS, which sonnet has and haiku doesn't). Wording iteration via
+`/autoresearch` not yet started._
+
+Measures the two hardest os-adr requirements (`docs/adr-system/04-plugin-requirements.md`
+reqs 4–5): does an agent notice **on its own** that the ADR system is relevant — proposing to
+record a consequential decision (W1–W3), or retrieving the specific correct decision before
+changing decided-on behavior (R1–R4)? Scenario shapes are frozen in
+`docs/adr-system/06-eval-scenarios.md`. This is **not** Eval A (`../eval/`), which measures
+skill-*execution* once explicitly invoked — do not conflate them (ADR-021).
+
+> **HELD-OUT — read this first.** The `## Task` blocks in `scenarios/*.md` must never be
+> pasted into an interactive session, "tried out" informally, or fed to a model outside
+> `bin/run`. The only permitted non-grid execution is `bin/self-test`, which fabricates
+> transcripts and never uses the Task blocks. Informal trials contaminate the methodology.
+
+## Layout
+
+| Path | What |
+| --- | --- |
+| `fixture/project/` | small Ruby webhook-relay codebase, 6-ADR history (incl. a Superseded pair) |
+| `scenarios/W1..W3.md` | write-trigger prompts (persistence choice, convention change, reversal) |
+| `scenarios/R1..R4.md` | retrieval prompts (direct conflict, distractors, mid-task Q, graph hop) |
+| `bin/build-fixture-graph` | rebuild `fixture/project/graphify-out/` from scratch (R4 needs it) |
+| `bin/sandbox <Sn> <dest>` | fresh git-initialized sandbox; only `R4` keeps the graph; `R4-nograph` = degradation variant |
+| `bin/run <Sn> <model> <workdir> [--reps N]` | headless `claude -p` runner; the ONLY valid execution mode |
+| `bin/check <Sn> <sandbox> [--tsv <model>]` | two-axis checker; exit 0/1; TSV mode for autoresearch |
+| `bin/self-test [workdir]` | model-free both-directions harness self-test |
+| `judge-rubric.md` | frozen rubric for the W-scenario axis (b) LLM judge fallback |
+
+## Why headless-only (no Agent-tool subagents)
+
+The SessionStart hook's `[os-adr]` note is one of the two discovery paths under measurement.
+In-session subagents don't get a fresh SessionStart against the sandbox cwd, so they can't
+tell "the model didn't notice" from "the model never got the hint a real session would have."
+`bin/run` gives each rep a fresh `claude -p` process with cwd set to a fresh sandbox; the
+transcript's `system/hook_response` event proves the hook context reached the model — the
+checker treats a transcript without it as an invalid run. (Verified 2026-07-03 with a neutral
+non-scenario probe: haiku quoted the hook note verbatim from a sandbox session.)
+
+## Two-axis scoring
+
+- **Axis (a) — unprompted consultation (mechanical).** Any `tool_use` block in the
+  `stream-json` transcript that invokes an `os-adr:*` skill, a `bin/adr-*` CLI, or touches
+  `docs/adr/`. Prose never counts, so the hook note can't false-positive it.
+- **Axis (b) — specific correctness.**
+  - R1–R4 (mechanical): assistant text must cite the scenario's pre-declared correct ADR ID
+    (R1→0002, R2→0003, R3→0005 content match [UTC + ISO-8601], R4→0003) and flag the
+    conflict. R2 fails if the Superseded distractor 0001 is cited as if live.
+  - W1–W3 (deterministic-first): PASS mechanically when a new `docs/adr/NNNN-*.md` matching
+    the scenario's topic exists; only otherwise does a rubric-bound judge (haiku,
+    `judge-rubric.md`, final message only) decide propose/ask-vs-silent. Stub it with
+    `ADR_EVAL_B_JUDGE_CMD` (reads prompt on stdin, prints YES/NO).
+- **Overall PASS** = both axes PASS. TSV row: `scenario  model  A:…  B:…  PASS|FAIL  reasons`.
+
+## Running one cell
+
+```bash
+eval-b/bin/build-fixture-graph                  # once, before any R4 cell
+eval-b/bin/run R2 haiku /tmp/adr-eval-b --reps 3 --results /tmp/adr-eval-b/results.tsv
+```
+
+Each rep: fresh sandbox → `claude -p --output-format stream-json --verbose --model <tier>
+--dangerously-skip-permissions "<Task block>"` with cwd = sandbox → `transcript.jsonl` saved →
+`bin/check` appends one TSV row. `--reps` default is 1; the grid-run stage picks the real
+repeat count after observing variance in a pilot (design decision, left open deliberately).
+
+The R4 degradation check: `bin/run R4-nograph <model> <workdir>` — same prompt, graph absent.
+An axis (b) FAIL there is the **expected** outcome; it confirms R4's pass genuinely required
+the Graphify layer (paired with an R4 PASS on the same tier).
+
+## Fixture regeneration
+
+`graphify-out/` is never committed — rebuild it with `bin/build-fixture-graph`, which runs the
+real `graphify update` against the fixture (pure tree-sitter AST: the fixture's
+`.graphifyignore` excludes all markdown, so no Ollama/LLM is involved and the build is
+reproducible) and then asserts the R4 one-hop invariant (`lib/relay/reports.rb` is exactly one
+graph hop from `lib/relay/delivery.rb`). If graphify's output format changes and the invariant
+breaks, fix the fixture layout — not the checker — in a human-reviewed change.
+
+The ADR history was generated with the plugin's own CLIs (`adr-init`/`adr-new`), so frontmatter
+and index match the shipped format, including the mechanically-superseded 0001→0003 pair.
+
+## Optimizing with `/autoresearch` (later stage — not part of building this harness)
+
+Same discipline as Eval A: the **checker, fixtures, scenario prompts, and judge rubric are
+frozen during a loop** — only plugin surface wording (SKILL.md, hook note text) may move. Metric:
+pass rate over the {W1..W3, R1..R4} × {haiku, sonnet} grid via `bin/check --tsv`. If a scenario
+or checker turns out to be wrong, stop the loop and fix it as a separate, human-reviewed change.
+Unlike Eval A, every cell costs a real headless session — budget reps accordingly.
+
+## Adding a scenario
+
+Add `scenarios/X.md` (Task block + checker metadata), a class in `bin/check`, the fixture route
+in `bin/sandbox`, extend `bin/self-test` both directions, and keep the held-out rule: the new
+Task block gets no informal trial runs.

plugins/os-adr/eval-b/bin/build-fixture-graph

@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+# Rebuild the fixture's graphify-out/ from scratch (needed by scenario R4).
+#
+# The graph is never committed — it is disposable and regenerated against
+# whatever graphify version is installed. The fixture's .graphifyignore
+# excludes all markdown/config, so this is a pure tree-sitter AST build:
+# no LLM, no Ollama, reproducible.
+#
+# Sanity-checked invariant: lib/relay/delivery.rb must be exactly one graph
+# hop from lib/relay/reports.rb (the R4 one-hop layout).
+set -euo pipefail
+
+FIXTURE="$(cd "$(dirname "$0")/../fixture/project" && pwd)"
+
+rm -rf "$FIXTURE/graphify-out"
+graphify update "$FIXTURE"
+
+python3 - "$FIXTURE/graphify-out/graph.json" <<'PY'
+import json, sys
+g = json.load(open(sys.argv[1]))
+nodes = {n["id"]: n.get("source_file") for n in g["nodes"]}
+seeds = {i for i, f in nodes.items() if f == "lib/relay/reports.rb"}
+neighbors = set()
+for link in g["links"]:
+    if link["source"] in seeds: neighbors.add(nodes.get(link["target"]))
+    if link["target"] in seeds: neighbors.add(nodes.get(link["source"]))
+assert "lib/relay/delivery.rb" in neighbors, (
+    "R4 one-hop invariant broken: reports.rb no longer reaches delivery.rb "
+    f"in one hop (neighbors: {sorted(f for f in neighbors if f)})")
+print("R4 one-hop invariant holds: reports.rb -> delivery.rb")
+PY

plugins/os-adr/eval-b/bin/check

@@ -0,0 +1,276 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+# Two-axis, deterministic-first checker for Eval B (unprompted behavior).
+#
+#   check <W1|W2|W3|R1|R2|R3|R4|R4-nograph> <sandbox-root> [--tsv <model-label>]
+#
+# Reads <sandbox>/transcript.jsonl (stream-json, written by bin/run) plus the
+# sandbox file state.
+#
+#   axis (a) — unprompted consultation: mechanical. True iff any tool_use
+#     block touches the ADR system (an os-adr:* skill, a bin/adr-* CLI, or a
+#     read/glob/write under docs/adr/). Prose never counts; the SessionStart
+#     hook note cannot false-positive because it appears only outside
+#     tool_use blocks.
+#   axis (b) — specific correctness:
+#     R1..R4      mechanical: assistant text must cite the scenario's correct
+#                 ADR ID (and flag the conflict); R2 additionally fails when
+#                 the Superseded distractor is cited as if live.
+#     W1..W3      mechanical PASS when a new docs/adr/NNNN-*.md matching the
+#                 scenario's topic exists; otherwise a rubric-bound LLM judge
+#                 (judge-rubric.md) reads ONLY the final message. Override the
+#                 judge with ADR_EVAL_B_JUDGE_CMD (reads prompt on stdin,
+#                 prints YES/NO) — the self-test uses this to stay model-free.
+#
+# Overall PASS requires both axes. R4-nograph runs R4's checks under its own
+# label; on the degradation grid an axis-(b) FAIL there is the expected,
+# correct outcome (see README).
+#
+# TSV mode: scenario, model, axis-a, axis-b, PASS|FAIL, reasons.
+# Exit 0 on PASS, 1 on FAIL, 2 on usage error.
+
+require "json"
+require "open3"
+
+module AdrEvalB
+  BASELINE_ADR_IDS = %w[0001 0002 0003 0004 0005 0006].freeze
+  CONFLICT = /conflict|violat|contradic|goes against|supersed|locked|decided|decision|record/i
+
+  # The stream-json transcript bin/run captured. Deliberately structural:
+  # axis (a) looks only inside tool_use blocks, never at raw text.
+  class Transcript
+    def self.load(path)
+      return nil unless File.exist?(path)
+      events = File.readlines(path).filter_map do |line|
+        JSON.parse(line)
+      rescue JSON::ParserError
+        nil
+      end
+      new(events)
+    end
+
+    def initialize(events) = @events = events
+
+    def hook_context_present?
+      @events.any? do |e|
+        e["type"] == "system" && e["subtype"] == "hook_response" &&
+          e["hook_name"].to_s.start_with?("SessionStart") &&
+          e.to_json.include?("[os-adr]")
+      end
+    end
+
+    def tool_uses
+      assistant_blocks.select { |b| b["type"] == "tool_use" }
+    end
+
+    # Everything the model said (assistant text blocks + the final result).
+    def assistant_text
+      texts = assistant_blocks.select { |b| b["type"] == "text" }.map { |b| b["text"] }
+      texts << final_message
+      texts.compact.join("\n")
+    end
+
+    def final_message
+      result = @events.reverse.find { |e| e["type"] == "result" }
+      result && result["result"].is_a?(String) ? result["result"] : nil
+    end
+
+    private
+
+    def assistant_blocks
+      @events.select { |e| e["type"] == "assistant" }
+             .flat_map { |e| e.dig("message", "content") || [] }
+    end
+  end
+
+  class Sandbox
+    def initialize(root) = @root = root
+    attr_reader :root
+
+    def transcript = Transcript.load(File.join(@root, "transcript.jsonl"))
+
+    def new_adr_files
+      Dir.glob(File.join(@root, "docs/adr/[0-9]*.md")).sort.reject do |path|
+        BASELINE_ADR_IDS.include?(File.basename(path)[0, 4])
+      end
+    end
+  end
+
+  class Result
+    def initialize = @failures = []
+    attr_reader :failures
+    def pass? = @failures.empty?
+
+    def expect(condition, reason)
+      @failures << reason unless condition
+      !!condition
+    end
+  end
+
+  # Narrow LLM fallback for W-scenario axis (b): rubric + final message only.
+  class Judge
+    RUBRIC = File.read(File.expand_path("../judge-rubric.md", __dir__))
+                 .split("---", 2).last.strip
+
+    def self.command
+      ENV.fetch("ADR_EVAL_B_JUDGE_CMD",
+                "claude -p --model haiku --dangerously-skip-permissions")
+    end
+
+    def proposed_adr?(final_message)
+      prompt = "#{RUBRIC}\n\n#{final_message}"
+      output, status = Open3.capture2(self.class.command, stdin_data: prompt)
+      raise "judge command failed: #{self.class.command}" unless status.success?
+      verdict = output[/\b(YES|NO)\b/, 1]
+      raise "judge gave no YES/NO verdict: #{output.inspect}" if verdict.nil?
+      verdict == "YES"
+    end
+  end
+
+  class Scenario
+    def initialize(sandbox, judge: Judge.new)
+      @sb = sandbox
+      @judge = judge
+    end
+
+    def run
+      axis_a = Result.new
+      axis_b = Result.new
+      transcript = @sb.transcript
+      if transcript.nil?
+        axis_a.expect(false, "no transcript.jsonl in sandbox")
+        axis_b.expect(false, "no transcript.jsonl in sandbox")
+      elsif !transcript.hook_context_present?
+        axis_a.expect(false, "invalid run: SessionStart [os-adr] hook context missing")
+        axis_b.expect(false, "invalid run: SessionStart [os-adr] hook context missing")
+      else
+        check_axis_a(axis_a, transcript)
+        check_axis_b(axis_b, transcript)
+      end
+      [axis_a, axis_b]
+    end
+
+    private
+
+    attr_reader :sb, :judge
+
+    def check_axis_a(result, transcript)
+      touched = transcript.tool_uses.any? { |tu| adr_touch?(tu) }
+      result.expect(touched, "no tool_use touched the ADR system (os-adr:*, bin/adr-*, docs/adr/)")
+    end
+
+    def adr_touch?(tool_use)
+      probe = "#{tool_use['name']} #{tool_use['input'].to_json}"
+      probe.match?(%r{os-adr|bin/adr-|adr-(find|new|init|detect|migrate)\b|docs/adr})
+    end
+  end
+
+  # --- Retrieval scenarios: mechanical ID + conflict-flag checks ------------
+
+  class Retrieval < Scenario
+    def check_axis_b(result, transcript)
+      text = transcript.assistant_text
+      result.expect(text.match?(/\b#{correct_id}\b/),
+                    "assistant text never cites the correct ADR #{correct_id}")
+      result.expect(text.match?(AdrEvalB::CONFLICT),
+                    "assistant text never flags the conflict/decision")
+      check_distractors(result, text)
+    end
+
+    def check_distractors(_result, _text) = nil
+  end
+
+  class R1 < Retrieval
+    def correct_id = "0002"
+  end
+
+  class R2 < Retrieval
+    def correct_id = "0003"
+
+    # Citing the Superseded 0001 as if live is exactly the failure R2 exists
+    # to catch; citing it while noting supersession is correct behavior.
+    def check_distractors(result, text)
+      if text.match?(/\b0001\b/)
+        result.expect(text.match?(/supersed/i),
+                      "cites Superseded distractor 0001 without noting it is superseded")
+      end
+    end
+  end
+
+  class R3 < Retrieval
+    def correct_id = "0005"
+
+    # Correctness for R3 is answering the question from the ADR: UTC ISO-8601.
+    def check_axis_b(result, text_or_transcript)
+      text = text_or_transcript.assistant_text
+      result.expect(text.match?(/UTC/i), "answer never says UTC")
+      result.expect(text.match?(/ISO.?8601/i), "answer never says ISO-8601")
+    end
+  end
+
+  class R4 < Retrieval
+    def correct_id = "0003"
+  end
+
+  # --- Write-trigger scenarios: new-ADR-file check, judge fallback ----------
+
+  class WriteTrigger < Scenario
+    def check_axis_b(result, transcript)
+      new_files = sb.new_adr_files
+      if new_files.any?
+        matching = new_files.select { |p| File.read(p).match?(topic) }
+        result.expect(matching.any?,
+                      "new ADR exists but matches no topic term #{topic.inspect}")
+        return
+      end
+      final = transcript.final_message
+      unless final
+        result.expect(false, "no new ADR file and no final message to judge")
+        return
+      end
+      result.expect(judge.proposed_adr?(final),
+                    "no new ADR file; judge: final message does not propose/ask to record the decision")
+    end
+  end
+
+  class W1 < WriteTrigger
+    def topic = /persist|stor(e|age|ing)|database|sqlite|history/i
+  end
+
+  class W2 < WriteTrigger
+    def topic = /error|exception|result|raise|failure/i
+  end
+
+  class W3 < WriteTrigger
+    def topic = /concurren|thread|parallel|order|sequential|queue/i
+  end
+
+  SCENARIOS = {
+    "W1" => W1, "W2" => W2, "W3" => W3,
+    "R1" => R1, "R2" => R2, "R3" => R3, "R4" => R4,
+    "R4-nograph" => R4 # same checks; expected to FAIL axis (b) — see README
+  }.freeze
+end
+
+scenario_id, sandbox_root = ARGV[0], ARGV[1]
+tsv_model = ARGV[2] == "--tsv" ? (ARGV[3] || "unknown") : nil
+klass = AdrEvalB::SCENARIOS[scenario_id]
+abort "usage: check <#{AdrEvalB::SCENARIOS.keys.join('|')}> <sandbox-root> [--tsv <model>]" if klass.nil? || sandbox_root.nil?
+abort "no such sandbox: #{sandbox_root}" unless File.directory?(sandbox_root)
+
+axis_a, axis_b = klass.new(AdrEvalB::Sandbox.new(File.expand_path(sandbox_root))).run
+overall = axis_a.pass? && axis_b.pass?
+reasons = (axis_a.failures + axis_b.failures).join("; ")
+
+if tsv_model
+  puts [scenario_id, tsv_model,
+        axis_a.pass? ? "A:PASS" : "A:FAIL",
+        axis_b.pass? ? "B:PASS" : "B:FAIL",
+        overall ? "PASS" : "FAIL", reasons].join("\t")
+else
+  puts "#{overall ? 'PASS' : 'FAIL'} #{scenario_id}  " \
+       "(axis-a #{axis_a.pass? ? 'PASS' : 'FAIL'}, axis-b #{axis_b.pass? ? 'PASS' : 'FAIL'})"
+  (axis_a.failures + axis_b.failures).each { |f| puts "  - #{f}" }
+end
+exit(overall ? 0 : 1)

plugins/os-adr/eval-b/bin/run

@@ -0,0 +1,55 @@
+#!/usr/bin/env bash
+# Headless runner for Eval B — the ONLY valid execution mode for these
+# scenarios (design.md Decision 1): a fresh `claude -p` process with cwd set
+# to the sandbox, so the real SessionStart hook fires for the model under
+# test. Never run these prompts via in-session Agent-tool subagents.
+#
+# Usage: run <scenario> <model> <workdir> [--reps N] [--results FILE]
+#
+#   scenario  W1|W2|W3|R1|R2|R3|R4|R4-nograph
+#   model     haiku|sonnet|opus|...
+#   workdir   sandboxes are created under here as <scenario>-<model>-rN
+#   --reps    repeated executions of the same cell (default 1; the grid-run
+#             stage decides the real default — design.md Decision 5)
+#
+# Each rep: fresh sandbox -> claude -p with ONLY the scenario's task prompt
+# (no system-level hints) -> full stream-json transcript saved to
+# <sandbox>/transcript.jsonl -> bin/check appends one TSV row to RESULTS.
+set -euo pipefail
+
+SCENARIO="${1:?usage: run <scenario> <model> <workdir> [--reps N] [--results FILE]}"
+MODEL="${2:?model required (haiku|sonnet|...)}"
+WORKDIR="${3:?workdir required}"
+shift 3
+
+REPS=1
+RESULTS=""
+while [ $# -gt 0 ]; do
+  case "$1" in
+    --reps)    REPS="${2:?--reps needs a number}"; shift 2 ;;
+    --results) RESULTS="${2:?--results needs a path}"; shift 2 ;;
+    *) echo "unknown option: $1" >&2; exit 2 ;;
+  esac
+done
+
+EVAL_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+SCENARIO_FILE="$EVAL_ROOT/scenarios/${SCENARIO%-nograph}.md"
+[ -f "$SCENARIO_FILE" ] || { echo "unknown scenario: $SCENARIO" >&2; exit 2; }
+mkdir -p "$WORKDIR"
+RESULTS="${RESULTS:-$WORKDIR/results.tsv}"
+
+TASK="$(awk '/^## Task/{found=1; next} found' "$SCENARIO_FILE")"
+
+for rep in $(seq 1 "$REPS"); do
+  SANDBOX="$WORKDIR/$SCENARIO-$MODEL-r$rep"
+  "$EVAL_ROOT/bin/sandbox" "$SCENARIO" "$SANDBOX" >/dev/null
+
+  # cwd = sandbox: the SessionStart hook resolves the project root from here.
+  (cd "$SANDBOX" && claude -p \
+      --model "$MODEL" \
+      --output-format stream-json --verbose \
+      --dangerously-skip-permissions \
+      "$TASK" > transcript.jsonl) || echo "claude exited non-zero for $SANDBOX" >&2
+
+  "$EVAL_ROOT/bin/check" "$SCENARIO" "$SANDBOX" --tsv "$MODEL" | tee -a "$RESULTS"
+done

plugins/os-adr/eval-b/bin/sandbox

@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+# Create a fresh, git-initialized sandbox copy of the Eval B fixture.
+# Usage: sandbox <W1|W2|W3|R1|R2|R3|R4|R4-nograph> <dest-dir>
+#
+# Only R4 keeps graphify-out/ (the graph-layer scenario). R4-nograph is the
+# degradation variant: same scenario, graph deliberately absent.
+set -euo pipefail
+
+SCENARIO="${1:?usage: sandbox <scenario> <dest-dir>}"
+DEST="${2:?usage: sandbox <scenario> <dest-dir>}"
+EVAL_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+FIXTURE="$EVAL_ROOT/fixture/project"
+
+case "$SCENARIO" in
+  W1|W2|W3|R1|R2|R3|R4-nograph) GRAPH=no ;;
+  R4)                           GRAPH=yes ;;
+  *) echo "unknown scenario: $SCENARIO" >&2; exit 2 ;;
+esac
+
+if [ "$GRAPH" = yes ] && [ ! -f "$FIXTURE/graphify-out/graph.json" ]; then
+  echo "fixture graph missing — run eval-b/bin/build-fixture-graph first" >&2
+  exit 2
+fi
+
+if [ -e "$DEST" ]; then echo "refusing to overwrite existing $DEST" >&2; exit 2; fi
+mkdir -p "$DEST"
+cp -r "$FIXTURE/." "$DEST/"
+[ "$GRAPH" = no ] && rm -rf "$DEST/graphify-out"
+rm -rf "$DEST/.os-adr"
+
+git -C "$DEST" init -q
+git -C "$DEST" add -A
+git -C "$DEST" -c user.email=eval@local -c user.name=eval commit -qm "fixture baseline"
+echo "$DEST"

plugins/os-adr/eval-b/bin/self-test

@@ -0,0 +1,158 @@
+#!/usr/bin/env bash
+# Self-test the Eval B harness in both directions, model-free:
+#
+#   1. For each scenario, fabricate a "perfect" transcript/sandbox state
+#      (an agent that did the right thing) -> bin/check must PASS.
+#   2. For each scenario, fabricate an "untouched" run (agent never consulted
+#      the ADR system) -> bin/check must FAIL, axis (a) FAIL specifically.
+#   3. Sandbox isolation: the canonical fixture must be byte-identical after
+#      all sandbox operations.
+#
+# The W-scenario LLM judge is stubbed via ADR_EVAL_B_JUDGE_CMD so this never
+# spends model tokens. It never touches the held-out scenario Task blocks.
+#
+# Usage: self-test [workdir]   (default: a fresh mktemp dir)
+set -euo pipefail
+
+EVAL_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+PLUGIN_ROOT="$(cd "$EVAL_ROOT/.." && pwd)"
+WORKDIR="${1:-$(mktemp -d /tmp/adr-eval-b-selftest.XXXXXX)}"
+mkdir -p "$WORKDIR"
+FAILED=0
+
+fixture_digest() {
+  (cd "$EVAL_ROOT/fixture/project" && find . -type f ! -path "./graphify-out/*" -print0 \
+    | sort -z | xargs -0 md5sum | md5sum)
+}
+DIGEST_BEFORE="$(fixture_digest)"
+
+# --- transcript fabrication -------------------------------------------------
+
+hook_event() {
+  printf '%s\n' '{"type":"system","subtype":"hook_response","hook_name":"SessionStart:startup","output":"[os-adr] This project records architecture decisions in docs/adr/"}'
+}
+
+tool_use_event() { # $1 = tool name, $2 = input JSON
+  printf '{"type":"assistant","message":{"role":"assistant","content":[{"type":"tool_use","name":"%s","input":%s}]}}\n' "$1" "$2"
+}
+
+result_event() { # $1 = final text (single line, no quotes needed beyond basic)
+  python3 -c 'import json,sys; print(json.dumps({"type":"result","subtype":"success","result":sys.argv[1]}))' "$1"
+}
+
+perfect_transcript() { # $1 = sandbox, $2 = adr-touching tool input, $3 = final text
+  { hook_event
+    tool_use_event "Bash" "$2"
+    result_event "$3"
+  } > "$1/transcript.jsonl"
+}
+
+untouched_transcript() { # $1 = sandbox
+  { hook_event
+    tool_use_event "Read" '{"file_path":"lib/relay/delivery.rb"}'
+    result_event "Done. I made the change you asked for."
+  } > "$1/transcript.jsonl"
+}
+
+# --- expectations -----------------------------------------------------------
+
+expect() { # $1 = PASS|FAIL, $2 = label, then the check command
+  local want="$1" label="$2"; shift 2
+  local got=PASS
+  "$@" >/dev/null 2>&1 || got=FAIL
+  if [ "$got" = "$want" ]; then
+    echo "ok   $label ($want)"
+  else
+    echo "FAIL $label: wanted $want, got $got"
+    "$@" || true
+    FAILED=1
+  fi
+}
+
+FIND_INPUT='{"command":"ruby '"$PLUGIN_ROOT"'/bin/adr-find --root . --paths lib/relay/delivery.rb"}'
+
+# --- 1. perfect runs must PASS ----------------------------------------------
+
+declare -A FINAL_TEXT=(
+  [R1]="Stopped before implementing: ADR 0002 requires all outbound HTTP to go through the shared HttpClient, so a direct Net::HTTP call would conflict with that decision. I added ping via the wrapper instead."
+  [R2]="Flat 2-second retries would violate ADR 0003 (exponential backoff, capped attempts). The older retry decision 0001 is superseded by 0003, so it does not apply. I did not change the backoff."
+  [R3]="Per ADR 0005, recorded event times are UTC ISO-8601 strings, so the first-seen value is written with Time.now.utc.iso8601 — not local server time."
+  [R4]="A retry-until-success loop in reports.rb would conflict with ADR 0003, which governs delivery retries one hop away: re-sends must follow the exponential backoff schedule. I routed redelivery through the backoff instead."
+)
+
+for sc in R1 R2 R3 R4; do
+  sb="$WORKDIR/perfect-$sc"
+  "$EVAL_ROOT/bin/sandbox" "$sc" "$sb" >/dev/null
+  perfect_transcript "$sb" "$FIND_INPUT" "${FINAL_TEXT[$sc]}"
+  expect PASS "perfect $sc" "$EVAL_ROOT/bin/check" "$sc" "$sb"
+done
+
+declare -A W_TITLE=(
+  [W1]="Use SQLite for delivery history persistence"
+  [W2]="Raise exceptions instead of returning Result objects"
+  [W3]="Allow concurrent delivery dispatch, dropping strict ordering"
+)
+
+for sc in W1 W2 W3; do
+  sb="$WORKDIR/perfect-$sc"
+  "$EVAL_ROOT/bin/sandbox" "$sc" "$sb" >/dev/null
+  printf '{"title": "%s", "context": "c", "decision": "d", "consequences": "q", "alternatives": "a"}' \
+    "${W_TITLE[$sc]}" | ruby "$PLUGIN_ROOT/bin/adr-new" --root "$sb" >/dev/null
+  perfect_transcript "$sb" '{"command":"ruby '"$PLUGIN_ROOT"'/bin/adr-new --root ."}' \
+    "Recorded the decision as a new ADR before implementing."
+  expect PASS "perfect $sc (mechanical new-ADR path)" "$EVAL_ROOT/bin/check" "$sc" "$sb"
+done
+
+# W judge fallback path: no new ADR file, final message proposes recording.
+sb="$WORKDIR/perfect-W1-judge"
+"$EVAL_ROOT/bin/sandbox" W1 "$sb" >/dev/null
+perfect_transcript "$sb" "$FIND_INPUT" \
+  "I implemented SQLite-backed history. This engine choice is hard to reverse — want me to record it as an ADR?"
+expect PASS "perfect W1 (judge fallback, stub YES)" \
+  env ADR_EVAL_B_JUDGE_CMD="printf YES" "$EVAL_ROOT/bin/check" W1 "$sb"
+
+# --- 2. untouched runs must FAIL (axis a) -----------------------------------
+
+for sc in W1 W2 W3 R1 R2 R3 R4 R4-nograph; do
+  sb="$WORKDIR/untouched-$sc"
+  "$EVAL_ROOT/bin/sandbox" "$sc" "$sb" >/dev/null
+  untouched_transcript "$sb"
+  expect FAIL "untouched $sc" \
+    env ADR_EVAL_B_JUDGE_CMD="printf NO" "$EVAL_ROOT/bin/check" "$sc" "$sb"
+  row="$(env ADR_EVAL_B_JUDGE_CMD="printf NO" "$EVAL_ROOT/bin/check" "$sc" "$sb" --tsv selftest || true)"
+  case "$row" in
+    *$'\t'A:FAIL$'\t'*) echo "ok   untouched $sc axis-a FAIL confirmed" ;;
+    *) echo "FAIL untouched $sc: axis-a not FAIL in TSV: $row"; FAILED=1 ;;
+  esac
+done
+
+# R2 distractor trap: cites superseded 0001 as if live -> axis (b) FAIL.
+sb="$WORKDIR/distractor-R2"
+"$EVAL_ROOT/bin/sandbox" R2 "$sb" >/dev/null
+perfect_transcript "$sb" "$FIND_INPUT" \
+  "ADR 0001 says deliveries retry three times, so I set the retries to match that decision."
+expect FAIL "R2 citing superseded distractor 0001" "$EVAL_ROOT/bin/check" R2 "$sb"
+
+# Missing hook context invalidates the run.
+sb="$WORKDIR/nohook-R1"
+"$EVAL_ROOT/bin/sandbox" R1 "$sb" >/dev/null
+{ tool_use_event "Bash" "$FIND_INPUT"; result_event "${FINAL_TEXT[R1]}"; } > "$sb/transcript.jsonl"
+expect FAIL "R1 without SessionStart hook context (invalid run)" "$EVAL_ROOT/bin/check" R1 "$sb"
+
+# --- 3. fixture isolation ---------------------------------------------------
+
+DIGEST_AFTER="$(fixture_digest)"
+if [ "$DIGEST_BEFORE" = "$DIGEST_AFTER" ]; then
+  echo "ok   canonical fixture untouched by all sandbox operations"
+else
+  echo "FAIL canonical fixture was modified"
+  FAILED=1
+fi
+
+echo
+if [ "$FAILED" = 0 ]; then
+  echo "self-test PASS (workdir: $WORKDIR)"
+else
+  echo "self-test FAIL (workdir kept for inspection: $WORKDIR)"
+fi
+exit "$FAILED"

plugins/os-adr/eval-b/fixture/project/.gitignore

@@ -0,0 +1,4 @@
+graphify-out/
+.os-adr/
+deadletter.jsonl
+relay.log

plugins/os-adr/eval-b/fixture/project/.graphifyignore

@@ -0,0 +1,7 @@
+# Keep the project graph AST-only (code files); markdown/docs would route
+# through the Ollama doc pass and make graph rebuilds slow and non-reproducible.
+docs/
+*.md
+config/
+graphify-out/
+.os-adr/

plugins/os-adr/eval-b/fixture/project/CLAUDE.md

@@ -0,0 +1,25 @@
+# CLAUDE.md
+
+`relay` is a small Ruby CLI that delivers webhook payloads to configured endpoints and
+reports on delivery outcomes. Configuration lives in `config/relay.yml`; failed deliveries
+that exhaust retries go to the dead-letter file.
+
+## Architecture decisions
+
+This project records architecture decisions as ADRs in `docs/adr/` (index:
+`docs/adr/README.md`).
+
+- **When** your task involves making an architecture-level choice, or changing/replacing an
+  approach this codebase already uses (persistence, delivery/retry strategy, file formats,
+  libraries, conventions) → **first** run `/os-adr:find` to check whether a recorded decision
+  already covers it.
+- Mechanical rule: **before your first edit to any existing source or config file**, run
+  `/os-adr:find` on the paths you are about to touch — one cheap, deterministic CLI call.
+  This applies to additions too: a new method or option can bypass a decided constraint just
+  as easily as a rewrite can.
+- **When** you make such a choice → record it with `/os-adr:create`.
+- If an Accepted ADR covers the approach you are changing, you are **reversing a recorded
+  decision**. A task that reverses a recorded decision is **not complete until the superseding
+  ADR is created** (`/os-adr:create` with `supersedes`) — implementing the change and stopping
+  leaves `docs/adr/` asserting the opposite of what the code does. Create the superseding ADR
+  (or explicitly propose it to the user) before you consider the task done.

plugins/os-adr/eval-b/fixture/project/README.md

@@ -0,0 +1,12 @@
+# relay
+
+A small CLI that delivers webhook payloads to configured endpoints and
+reports on delivery outcomes.
+
+- `bin/relay send <url> <json-payload>` — queue and deliver one payload
+- `bin/relay report` — summarize failed deliveries
+
+Configuration lives in `config/relay.yml`. Failed deliveries that exhaust
+their retries are appended to the dead-letter file for manual replay.
+
+Architecture decisions are recorded in `docs/adr/`.

plugins/os-adr/eval-b/fixture/project/bin/relay

@@ -0,0 +1,31 @@
+#!/usr/bin/env ruby
+# relay — deliver webhook payloads and report on outcomes.
+#
+#   relay send <url> <json-payload>
+#   relay report
+
+require_relative "../lib/relay"
+
+config = Relay::Config.load
+abort(config.error) if config.err?
+config = config.value
+
+log = Relay::Log.new(config.log_path)
+
+case ARGV[0]
+when "send"
+  url, raw = ARGV[1], ARGV[2]
+  abort("usage: relay send <url> <json-payload>") unless url && raw
+  delivery = Relay::Delivery.new(log: log, dead_letter_path: config.dead_letter_path)
+  queue = Relay::Queue.new(delivery: delivery)
+  queue.enqueue(url, JSON.parse(raw))
+  result = queue.drain
+  abort(result.error) if result.err?
+when "report"
+  reports = Relay::Reports.new(dead_letter_path: config.dead_letter_path)
+  result = reports.render
+  abort(result.error) if result.err?
+  puts result.value
+else
+  abort("usage: relay <send|report>")
+end

plugins/os-adr/eval-b/fixture/project/config/relay.yml

@@ -0,0 +1,5 @@
+endpoints:
+  - https://hooks.example.test/orders
+  - https://hooks.example.test/invoices
+dead_letter_path: deadletter.jsonl
+log_path: relay.log

plugins/os-adr/eval-b/fixture/project/docs/adr/0001-retry-failed-deliveries-a-fixed-three-times.md

@@ -0,0 +1,27 @@
+---
+id: "0001"
+date: 2026-02-10
+status: Superseded
+supersedes:
+superseded-by: "0003"
+affected-paths: [lib/relay/delivery.rb]
+affected-components: [delivery, retries]
+---
+
+# 0001 — Retry failed deliveries a fixed three times
+
+## Context
+
+Early receivers dropped webhooks intermittently; a delivery that fails once usually succeeds on a quick follow-up. We needed some retry behavior before launch and had no data on receiver load patterns yet.
+
+## Decision
+
+Failed deliveries are retried exactly three times, one second apart, then reported as failed. The retry count and spacing are constants in lib/relay/delivery.rb.
+
+## Consequences
+
+Simple and predictable; transient blips are absorbed. Receivers that are down for more than a few seconds still lose the delivery, and rapid-fire retries add load to a receiver that is already struggling.
+
+## Alternatives rejected
+
+No retries rejected: transient network errors were the dominant failure mode in testing. Unbounded retries rejected: a permanently dead endpoint would wedge the queue.

plugins/os-adr/eval-b/fixture/project/docs/adr/0002-route-all-outbound-http-through-the-shared-httpclient.md

@@ -0,0 +1,27 @@
+---
+id: "0002"
+date: 2026-02-18
+status: Accepted
+supersedes:
+superseded-by:
+affected-paths: [lib/relay/http_client.rb, lib/relay/delivery.rb]
+affected-components: [http, delivery]
+---
+
+# 0002 — Route all outbound HTTP through the shared HttpClient
+
+## Context
+
+Two call sites had grown their own Net::HTTP usage with inconsistent timeouts and no shared User-Agent, and a TLS verification bug had to be fixed in both places separately.
+
+## Decision
+
+All outbound HTTP goes through Relay::HttpClient (lib/relay/http_client.rb). No other file constructs Net::HTTP objects directly — timeouts, TLS policy, headers, and instrumentation live in exactly one place.
+
+## Consequences
+
+One place to change HTTP policy; call sites cannot drift. New delivery types must accept the small indirection of going through the wrapper even for trivial requests.
+
+## Alternatives rejected
+
+Per-call-site Net::HTTP rejected: policy drift was the concrete bug that motivated this. Pulling in a full HTTP gem (faraday) rejected: stdlib is sufficient at this scale and keeps the tool dependency-free.

plugins/os-adr/eval-b/fixture/project/docs/adr/0003-retry-failed-deliveries-with-exponential-backoff-capped-at-five-attempts.md

@@ -0,0 +1,27 @@
+---
+id: "0003"
+date: 2026-03-05
+status: Accepted
+supersedes: "0001"
+superseded-by:
+affected-paths: [lib/relay/delivery.rb]
+affected-components: [delivery, retries]
+---
+
+# 0003 — Retry failed deliveries with exponential backoff, capped at five attempts
+
+## Context
+
+Fixed one-second retries hammered receivers that were already struggling: several endpoints throttle by IP, and back-to-back retries during an outage got our sender IP temporarily banned. The fixed-three-times policy from the earlier decision made outages worse, not better.
+
+## Decision
+
+Failed deliveries back off exponentially — 1s, 2s, 4s, 8s, 16s — and give up after five attempts. Retries are never issued back-to-back or in a tight loop; anything that re-sends a failed delivery must go through this backoff schedule.
+
+## Consequences
+
+Struggling receivers get breathing room and IP throttling is no longer triggered. A delivery can now take up to ~31 seconds to fail over all five attempts, so callers must not assume prompt failure.
+
+## Alternatives rejected
+
+Keeping fixed-interval retries rejected: it caused the IP bans. Immediate retry-until-success rejected outright: it is precisely the pattern receivers throttle. Jittered backoff deferred: single-sender volume does not need it yet.

plugins/os-adr/eval-b/fixture/project/docs/adr/0004-cap-outbound-http-timeouts-at-five-seconds.md

@@ -0,0 +1,27 @@
+---
+id: "0004"
+date: 2026-03-12
+status: Accepted
+supersedes:
+superseded-by:
+affected-paths: [lib/relay/http_client.rb]
+affected-components: [http]
+---
+
+# 0004 — Cap outbound HTTP timeouts at five seconds
+
+## Context
+
+A single hung receiver held a delivery open for two minutes with the stdlib default timeouts, blocking everything queued behind it.
+
+## Decision
+
+HttpClient sets open_timeout and read_timeout to five seconds. A receiver that cannot accept a small JSON POST within five seconds is treated as failed and handled by the retry policy.
+
+## Consequences
+
+A hung receiver costs at most ten seconds per attempt. Genuinely slow-but-healthy receivers will see more retries; none have surfaced in practice.
+
+## Alternatives rejected
+
+Per-endpoint configurable timeouts rejected: no current need, and it invites unbounded values creeping back in. Thirty-second timeouts rejected: still long enough to stall the queue noticeably.

plugins/os-adr/eval-b/fixture/project/docs/adr/0005-record-event-times-as-utc-iso-8601-strings.md

@@ -0,0 +1,27 @@
+---
+id: "0005"
+date: 2026-04-02
+status: Accepted
+supersedes:
+superseded-by:
+affected-paths: [lib/relay/log.rb]
+affected-components: [logging]
+---
+
+# 0005 — Record event times as UTC ISO-8601 strings
+
+## Context
+
+Early log lines used local server time with no zone marker. Correlating our log against receivers' logs during an incident required guessing the offset, and a DST change produced apparently out-of-order events.
+
+## Decision
+
+Every recorded event time is a UTC ISO-8601 string (e.g. 2026-04-02T09:15:00Z), produced via Time.now.utc.iso8601. Local time and bare epoch integers are not written anywhere.
+
+## Consequences
+
+Log lines correlate directly with receiver logs and sort lexicographically. Anyone reading logs on a non-UTC machine does the mental offset themselves.
+
+## Alternatives rejected
+
+Epoch seconds rejected: compact but unreadable during incidents. Local time with zone suffix rejected: DST still reorders events within a day.

plugins/os-adr/eval-b/fixture/project/docs/adr/0006-dead-letter-deliveries-that-exhaust-retries-to-a-jsonl-file.md

@@ -0,0 +1,27 @@
+---
+id: "0006"
+date: 2026-04-20
+status: Accepted
+supersedes:
+superseded-by:
+affected-paths: [lib/relay/delivery.rb]
+affected-components: [delivery, retries]
+---
+
+# 0006 — Dead-letter deliveries that exhaust retries to a JSONL file
+
+## Context
+
+A delivery that exhausts its retry attempts was previously just a log line; operators had no way to replay it after the receiver recovered.
+
+## Decision
+
+When a delivery fails its final attempt, Delivery appends the URL, payload, and last error as one JSON line to the dead-letter file (deadletter.jsonl). Replay is a manual operator action, not automatic.
+
+## Consequences
+
+No delivery is silently lost; the report command can enumerate failures. The file grows without rotation, acceptable at current volume.
+
+## Alternatives rejected
+
+Automatic background replay rejected: a recovering receiver would immediately be hit with the full backlog. A database table rejected: the tool has no database and one failure file is greppable.

plugins/os-adr/eval-b/fixture/project/docs/adr/README.md

@@ -0,0 +1,16 @@
+<!-- Generated by os-adr. Do not hand-edit the table: it is regenerated in full on every ADR write. -->
+
+# Architecture Decision Records
+
+One file per decision, `NNNN-kebab-title.md`, created via `/os-adr:create`.
+
+<!-- adr-index:begin -->
+| ID | Title | Status | Date |
+| --- | --- | --- | --- |
+| 0001 | [Retry failed deliveries a fixed three times](0001-retry-failed-deliveries-a-fixed-three-times.md) | Superseded | 2026-02-10 |
+| 0002 | [Route all outbound HTTP through the shared HttpClient](0002-route-all-outbound-http-through-the-shared-httpclient.md) | Accepted | 2026-02-18 |
+| 0003 | [Retry failed deliveries with exponential backoff, capped at five attempts](0003-retry-failed-deliveries-with-exponential-backoff-capped-at-five-attempts.md) | Accepted | 2026-03-05 |
+| 0004 | [Cap outbound HTTP timeouts at five seconds](0004-cap-outbound-http-timeouts-at-five-seconds.md) | Accepted | 2026-03-12 |
+| 0005 | [Record event times as UTC ISO-8601 strings](0005-record-event-times-as-utc-iso-8601-strings.md) | Accepted | 2026-04-02 |
+| 0006 | [Dead-letter deliveries that exhaust retries to a JSONL file](0006-dead-letter-deliveries-that-exhaust-retries-to-a-jsonl-file.md) | Accepted | 2026-04-20 |
+<!-- adr-index:end -->

plugins/os-adr/eval-b/fixture/project/lib/relay.rb

@@ -0,0 +1,11 @@
+require_relative "relay/result"
+require_relative "relay/config"
+require_relative "relay/log"
+require_relative "relay/http_client"
+require_relative "relay/delivery"
+require_relative "relay/queue"
+require_relative "relay/formatter"
+require_relative "relay/reports"
+
+module Relay
+end

plugins/os-adr/eval-b/fixture/project/lib/relay/config.rb

@@ -0,0 +1,29 @@
+require "yaml"
+require_relative "result"
+
+module Relay
+  # Loads config/relay.yml. Returns Result, never raises for a missing or
+  # malformed file.
+  class Config
+    DEFAULT_PATH = File.expand_path("../../config/relay.yml", __dir__)
+
+    def self.load(path = DEFAULT_PATH)
+      return Result.err("config not found: #{path}") unless File.exist?(path)
+
+      data = YAML.safe_load(File.read(path))
+      return Result.err("config is not a mapping") unless data.is_a?(Hash)
+
+      Result.ok(new(data))
+    rescue Psych::SyntaxError => e
+      Result.err("config parse error: #{e.message}")
+    end
+
+    def initialize(data)
+      @data = data
+    end
+
+    def endpoints = @data.fetch("endpoints", [])
+    def dead_letter_path = @data.fetch("dead_letter_path", "deadletter.jsonl")
+    def log_path = @data.fetch("log_path", "relay.log")
+  end
+end

plugins/os-adr/eval-b/fixture/project/lib/relay/delivery.rb

@@ -0,0 +1,53 @@
+require "json"
+require_relative "result"
+require_relative "http_client"
+require_relative "log"
+
+module Relay
+  # Delivers one webhook payload to one endpoint, with retries.
+  #
+  # Failed attempts back off exponentially (1s, 2s, 4s, 8s, 16s) and give up
+  # after MAX_ATTEMPTS; receivers throttle by IP, so attempts are never
+  # retried back-to-back. A delivery that exhausts its attempts is appended
+  # to the dead-letter file for manual replay.
+  class Delivery
+    MAX_ATTEMPTS = 5
+    BASE_BACKOFF = 1 # seconds; doubles each attempt
+
+    def self.attempt_budget = MAX_ATTEMPTS
+
+    def initialize(client: HttpClient.new, log:, dead_letter_path:)
+      @client = client
+      @log = log
+      @dead_letter_path = dead_letter_path
+    end
+
+    def deliver(url, payload)
+      body = JSON.generate(payload)
+      last_error = nil
+
+      MAX_ATTEMPTS.times do |attempt|
+        sleep(BASE_BACKOFF * (2**(attempt - 1))) if attempt.positive?
+        @log.record("attempt", "#{url} ##{attempt + 1}")
+        result = @client.post(url, body)
+        if result.ok?
+          @log.record("delivered", url)
+          return Result.ok(result.value)
+        end
+        last_error = result.error
+        @log.record("failed", "#{url}: #{last_error}")
+      end
+
+      dead_letter(url, payload, last_error)
+      Result.err("gave up after #{MAX_ATTEMPTS} attempts: #{last_error}")
+    end
+
+    private
+
+    def dead_letter(url, payload, error)
+      entry = { "url" => url, "payload" => payload, "error" => error }
+      File.open(@dead_letter_path, "a") { |f| f.puts(JSON.generate(entry)) }
+      @log.record("dead-lettered", url)
+    end
+  end
+end

plugins/os-adr/eval-b/fixture/project/lib/relay/formatter.rb

@@ -0,0 +1,27 @@
+require_relative "result"
+
+module Relay
+  # Renders report rows for terminal output.
+  class Formatter
+    HEADERS = %w[url attempts outcome].freeze
+
+    def render(rows)
+      return Result.err("no rows") if rows.empty?
+
+      widths = column_widths(rows)
+      lines = [format_row(HEADERS, widths)]
+      lines += rows.map { |row| format_row(row, widths) }
+      Result.ok(lines.join("\n"))
+    end
+
+    private
+
+    def column_widths(rows)
+      ([HEADERS] + rows).transpose.map { |col| col.map { |c| c.to_s.length }.max }
+    end
+
+    def format_row(row, widths)
+      row.zip(widths).map { |cell, w| cell.to_s.ljust(w) }.join("  ")
+    end
+  end
+end

plugins/os-adr/eval-b/fixture/project/lib/relay/http_client.rb

@@ -0,0 +1,38 @@
+require "net/http"
+require "uri"
+require_relative "result"
+
+module Relay
+  # The single place outbound HTTP happens. Timeouts, TLS policy, and
+  # instrumentation live here so call sites cannot drift.
+  class HttpClient
+    OPEN_TIMEOUT = 5
+    READ_TIMEOUT = 5
+
+    def post(url, body, headers = {})
+      uri = URI.parse(url)
+      http = Net::HTTP.new(uri.host, uri.port)
+      http.use_ssl = uri.scheme == "https"
+      http.open_timeout = OPEN_TIMEOUT
+      http.read_timeout = READ_TIMEOUT
+
+      request = Net::HTTP::Post.new(uri.request_uri, default_headers.merge(headers))
+      request.body = body
+      response = http.request(request)
+
+      if response.is_a?(Net::HTTPSuccess)
+        Result.ok(response)
+      else
+        Result.err("HTTP #{response.code} from #{uri.host}")
+      end
+    rescue SystemCallError, Net::OpenTimeout, Net::ReadTimeout => e
+      Result.err("#{e.class}: #{e.message}")
+    end
+
+    private
+
+    def default_headers
+      { "Content-Type" => "application/json", "User-Agent" => "relay/1.0" }
+    end
+  end
+end

plugins/os-adr/eval-b/fixture/project/lib/relay/log.rb

@@ -0,0 +1,21 @@
+require "time"
+require_relative "result"
+
+module Relay
+  # Append-only event log. Every recorded event carries the moment it
+  # happened as a UTC ISO-8601 string.
+  class Log
+    def initialize(path)
+      @path = path
+    end
+
+    def record(event, detail = nil)
+      stamp = Time.now.utc.iso8601
+      line = [stamp, event, detail].compact.join("\t")
+      File.open(@path, "a") { |f| f.puts(line) }
+      Result.ok(line)
+    rescue SystemCallError => e
+      Result.err("log write failed: #{e.message}")
+    end
+  end
+end

plugins/os-adr/eval-b/fixture/project/lib/relay/queue.rb

@@ -0,0 +1,31 @@
+require_relative "result"
+require_relative "delivery"
+
+module Relay
+  # Dispatches queued deliveries strictly in order, one at a time.
+  #
+  # This is deliberate, not an oversight: receivers rely on seeing events in
+  # the order they occurred (e.g. "created" before "updated"), and a single
+  # in-process worker is the simplest way to guarantee that. Interleaving
+  # deliveries would silently break receivers' ordering assumptions.
+  class Queue
+    def initialize(delivery:)
+      @delivery = delivery
+      @pending = []
+    end
+
+    def enqueue(url, payload)
+      @pending << [url, payload]
+      Result.ok(@pending.size)
+    end
+
+    def drain
+      results = []
+      until @pending.empty?
+        url, payload = @pending.shift
+        results << @delivery.deliver(url, payload)
+      end
+      Result.ok(results)
+    end
+  end
+end

plugins/os-adr/eval-b/fixture/project/lib/relay/reports.rb

@@ -0,0 +1,34 @@
+require "json"
+require_relative "result"
+require_relative "delivery"
+require_relative "formatter"
+
+module Relay
+  # Summarizes delivery outcomes for the `relay report` command, including
+  # entries the Delivery dead-letter path wrote out.
+  class Reports
+    def initialize(dead_letter_path:, formatter: Formatter.new)
+      @dead_letter_path = dead_letter_path
+      @formatter = formatter
+    end
+
+    def failed_deliveries
+      return Result.ok([]) unless File.exist?(@dead_letter_path)
+
+      entries = File.readlines(@dead_letter_path).map { |line| JSON.parse(line) }
+      Result.ok(entries)
+    rescue JSON::ParserError => e
+      Result.err("dead-letter file corrupt: #{e.message}")
+    end
+
+    def render
+      failures = failed_deliveries
+      return failures if failures.err?
+
+      rows = failures.value.map do |entry|
+        [entry["url"], Delivery.attempt_budget, "dead-lettered"]
+      end
+      rows.empty? ? Result.ok("no failed deliveries") : @formatter.render(rows)
+    end
+  end
+end

plugins/os-adr/eval-b/fixture/project/lib/relay/result.rb

@@ -0,0 +1,12 @@
+module Relay
+  # Uniform error-handling shape used across the whole codebase: every
+  # fallible operation returns a Result; exceptions are reserved for
+  # programmer errors, never expected failures.
+  Result = Struct.new(:ok, :value, :error) do
+    def self.ok(value = nil) = new(true, value, nil)
+    def self.err(error) = new(false, nil, error)
+
+    def ok? = ok
+    def err? = !ok
+  end
+end

plugins/os-adr/eval-b/judge-rubric.md

@@ -0,0 +1,27 @@
+# W-scenario axis (b) judge rubric — FROZEN
+
+This rubric is part of the checker. Like the fixtures and scenario prompts, it is frozen
+during any `autoresearch` loop; if it turns out to be wrong, stop the loop and fix it as a
+separate, human-reviewed change.
+
+The judge reads ONLY the model-under-test's final message (never the full transcript) and
+answers one yes/no question.
+
+---
+
+You are grading one final assistant message from a coding-agent session. The agent was given
+an implementation task in a project that records architecture decisions as ADRs.
+
+Question: in this final message, did the agent explicitly propose recording the decision it
+made (or was asked to make) as an architecture decision record — for example by offering to
+create an ADR, naming an ADR-creation step or command it could run, or asking the user whether
+the decision should be recorded?
+
+Answer YES only if the message explicitly proposes or asks about recording THIS decision as an
+ADR / decision record. Answer NO if the agent implemented silently, only described code
+changes, or mentioned documentation generically (README updates, code comments) without
+proposing a decision record.
+
+Reply with exactly one word: YES or NO.
+
+Final assistant message follows:

plugins/os-adr/eval-b/scenarios/R1.md

@@ -0,0 +1,21 @@
+# R1 — retrieval: direct conflict with an Accepted ADR
+
+> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
+> First real execution is the harness self-test (see eval-b/README.md).
+
+- **Fixture:** `project` (no `graphify-out/`)
+- **Shape:** `06-eval-scenarios.md` R1 — the requested implementation (direct `Net::HTTP` in
+  `lib/relay/delivery.rb`) violates Accepted ADR **0002** (all outbound HTTP through the shared
+  HttpClient), whose `affected-paths` cover the file being edited.
+- **Correct ADR:** `0002`
+- **Checker, axis (a):** any tool_use touching the ADR system.
+- **Checker, axis (b):** assistant text cites `0002` and flags the conflict
+  (/conflict|violat|contradic|goes against|instead|decision|record/i); implementing the direct
+  call without surfacing 0002 is a FAIL.
+
+## Task (give verbatim to the model under test)
+
+Add a lightweight `ping` method to `lib/relay/delivery.rb` that POSTs a tiny heartbeat body
+(`{"ping": true}`) to a given URL. It's fire-and-forget — no retries, no dead-lettering — so
+keep it cheap: just call `Net::HTTP` directly inside the method rather than pulling in any of
+the heavier machinery.

plugins/os-adr/eval-b/scenarios/R2.md

@@ -0,0 +1,20 @@
+# R2 — retrieval: distractor discrimination
+
+> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
+> First real execution is the harness self-test (see eval-b/README.md).
+
+- **Fixture:** `project` (no `graphify-out/`)
+- **Shape:** `06-eval-scenarios.md` R2 — same conflict shape as R1, but `docs/adr/` seeds
+  plausible near-misses: **0001** (Superseded retry policy — same topic, dead), **0004**
+  (same http family, different decision), **0006** (same retry family, different decision).
+  The flat-2s-retry request conflicts with Accepted ADR **0003** (exponential backoff, cap 5).
+- **Correct ADR:** `0003` · **Distractors:** `0001` (Superseded), `0004`, `0006`
+- **Checker, axis (a):** any tool_use touching the ADR system.
+- **Checker, axis (b):** assistant text cites `0003` and flags the conflict; citing `0001` as
+  governing without noting it is superseded is a FAIL.
+
+## Task (give verbatim to the model under test)
+
+Customers complain that failed webhook deliveries take too long to come back. Make retries
+snappier: change `lib/relay/delivery.rb` so a failed delivery just retries every 2 seconds
+flat, instead of waiting longer and longer between attempts.

plugins/os-adr/eval-b/scenarios/R3.md

@@ -0,0 +1,21 @@
+# R3 — retrieval: mid-task "how should we…" question, different vocabulary
+
+> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
+> First real execution is the harness self-test (see eval-b/README.md).
+
+- **Fixture:** `project` (no `graphify-out/`)
+- **Shape:** `06-eval-scenarios.md` R3 — the embedded question ("what format should the time
+  value use") is answered by Accepted ADR **0005** (UTC ISO-8601), phrased in different
+  vocabulary ("write down the moment each attempt happened" vs "record event times").
+- **Correct ADR:** `0005`
+- **Checker, axis (a):** any tool_use touching the ADR system (mid-task query surface).
+- **Checker, axis (b):** assistant text answers with UTC **and** ISO-8601 (both
+  /UTC/i and /ISO.?8601/i must match); local-time or epoch answers FAIL.
+
+## Task (give verbatim to the model under test)
+
+I want `bin/relay report` to show a "first seen" column for each failed delivery. Add the
+moment a delivery first failed to the dead-letter entries in `lib/relay/delivery.rb`, and show
+it in the report. Quick question while you're in there: for that time value we write into the
+file, should it be local server time or something else, and in what format? Set it up whichever
+way is correct for this project.

plugins/os-adr/eval-b/scenarios/R4.md

@@ -0,0 +1,25 @@
+# R4 — retrieval: graph-layer reach (one hop past affected-paths)
+
+> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
+> First real execution is the harness self-test (see eval-b/README.md).
+
+- **Fixture:** `project` **with** `graphify-out/` (the only scenario that keeps it; rebuild via
+  `bin/build-fixture-graph` if missing)
+- **Shape:** `06-eval-scenarios.md` R4 — `lib/relay/reports.rb` is listed in **no** ADR's
+  `affected-paths`, but is one call-graph hop (`Reports#render` → `Delivery.attempt_budget`)
+  from `lib/relay/delivery.rb`, which Accepted ADR **0003** (exponential backoff; never
+  tight-loop re-sends) does cover. The requested retry-until-success loop conflicts with 0003.
+  Reachable only via the Graphify expansion layer: `adr-find --paths lib/relay/reports.rb`
+  returns 0003 (`matched_on: graph`) with the graph present and nothing without it.
+- **Correct ADR:** `0003`
+- **Degradation variant:** `bin/sandbox R4-nograph` strips `graphify-out/`; the same run is
+  then *expected* to FAIL axis (b) — that failure is the graph-layer degradation check.
+- **Checker, axis (a):** any tool_use touching the ADR system.
+- **Checker, axis (b):** assistant text cites `0003` and flags the conflict.
+
+## Task (give verbatim to the model under test)
+
+Add a `redeliver` command to `bin/relay`, implemented in `lib/relay/reports.rb`: for each
+failed delivery in the dead-letter file, immediately re-send it to its URL, and keep re-sending
+in a loop until it goes through. Keep all the new logic inside `lib/relay/reports.rb` — don't
+touch `lib/relay/delivery.rb`.

plugins/os-adr/eval-b/scenarios/W1.md

@@ -0,0 +1,22 @@
+# W1 — write-trigger: irreversible choice mid-task (persistence engine)
+
+> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
+> First real execution is the harness self-test (see eval-b/README.md).
+
+- **Fixture:** `project` (no `graphify-out/`)
+- **Shape:** `06-eval-scenarios.md` W1 — storage engine choice is genuinely open and
+  consequential; no existing ADR covers persistence.
+- **Checker, axis (a):** any tool_use touching the ADR system (`os-adr:*` skill, `bin/adr-*`
+  CLI, or read/glob of `docs/adr/`).
+- **Checker, axis (b):** mechanical PASS if a new `docs/adr/0007-*.md` exists whose text
+  matches the topic pattern /persist|stor(e|age)|database|sqlite|history/i; otherwise the
+  rubric-bound judge decides from the final message whether the agent proposed or asked about
+  recording the decision.
+
+## Task (give verbatim to the model under test)
+
+Delivery history currently only exists while the process is running — `bin/relay report` can
+only show dead-lettered failures, and nothing survives between runs. Add persistence for
+delivery history so a report can cover everything delivered in past runs too. Pick whatever
+storage approach you think is right for a small single-user CLI tool like this, and wire
+`bin/relay report` up to it.

plugins/os-adr/eval-b/scenarios/W2.md

@@ -0,0 +1,18 @@
+# W2 — write-trigger: cross-cutting convention change (error-handling shape)
+
+> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
+> First real execution is the harness self-test (see eval-b/README.md).
+
+- **Fixture:** `project` (no `graphify-out/`)
+- **Shape:** `06-eval-scenarios.md` W2 — the natural solution changes a convention used across
+  many files (the `Relay::Result` return shape); the convention is real in code but not ADR'd.
+- **Checker, axis (a):** any tool_use touching the ADR system.
+- **Checker, axis (b):** mechanical PASS if a new `docs/adr/0007-*.md` exists matching
+  /error|exception|result|raise|failure/i; otherwise the judge decides from the final message.
+
+## Task (give verbatim to the model under test)
+
+I keep finding the Result-struct plumbing noisy — every caller has to unwrap `ok?`/`error` by
+hand. Refactor the codebase so failures raise exceptions instead of returning Result objects,
+and update all the callers accordingly. Feel free to delete `lib/relay/result.rb` once nothing
+uses it.

plugins/os-adr/eval-b/scenarios/W3.md

@@ -0,0 +1,18 @@
+# W3 — write-trigger: contested reversal of a discoverable-in-code approach
+
+> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
+> First real execution is the harness self-test (see eval-b/README.md).
+
+- **Fixture:** `project` (no `graphify-out/`)
+- **Shape:** `06-eval-scenarios.md` W3 — the request quietly reverses the deliberate
+  single-worker in-order dispatch documented in `lib/relay/queue.rb`'s comment (not ADR'd).
+- **Checker, axis (a):** any tool_use touching the ADR system.
+- **Checker, axis (b):** mechanical PASS if a new `docs/adr/0007-*.md` exists matching
+  /concurren|thread|parallel|order|sequential|queue/i; otherwise the judge decides from the
+  final message.
+
+## Task (give verbatim to the model under test)
+
+Dispatch is slow when a lot of deliveries queue up, because they go out one at a time. Change
+`lib/relay/queue.rb` so `drain` processes deliveries concurrently with a small thread pool —
+four workers is plenty.

plugins/os-adr/eval-c/README.md

@@ -0,0 +1,297 @@
+# os-adr Eval C — ambiguity-ladder discrimination eval (held-out)
+
+_Last updated: 2026-07-06 — harness built and self-tested (all scenarios pass model-free checks)._
+
+Measures **generalization across decreasing cue explicitness** — does the model correctly
+trigger `/os-adr:find` or `/os-adr:create` as architectural decision vocabulary becomes
+less explicit, and equally important, **does it correctly NOT trigger when no decided
+constraint is in play**? This is the discrimination challenge: both false positives
+(over-trigger) and false negatives (miss the decision) are failures.
+
+Eval B (unprompted-behavior baseline) optimized its wording against its own grid for 5
+iterations (2026-07-04), so its 8/8 (sonnet) / 7/8 (haiku) are training-set scores. Eval C
+measures whether that learned behavior generalizes to **different project, different target
+ADRs, and most critically, paired negative scenarios that ensure the model isn't just learning
+"always trigger"**.
+
+> **HELD-OUT — read this first.** The `## Task` blocks in `scenarios/*.md` and
+> `scenarios-reserve/*.md` must never be pasted into an interactive session or informally
+> "tried out". The only permitted non-grid execution is `bin/self-test`, which fabricates
+> transcripts and never uses the Task blocks. Informal trials contaminate the measurement.
+
+## Measurement discipline
+
+This eval is a **frozen measurement pass** — it runs once with a baseline grid. The moment
+anyone iterates wording against the run-set results, this set is contaminated; future wording
+work **must switch to the reserve-set** for measurement. See "Reserve set protection" below.
+
+## Design: ambiguity ladder with paired positive/negative
+
+### The ladder (3 levels)
+
+Each level decreases explicit architectural decision cues:
+
+- **Level 1 (Explicit):** File paths named, implementation vocabulary explicit ("timeout",
+  "retry", "backoff", "exponential"), decision nature overt.
+- **Level 2 (Moderate):** Relevant vocabulary present but no file paths or explicit decision
+  framing; model must recognize architecture-level significance.
+- **Level 3 (Conceptual):** Intent-framed only (what outcome is needed), zero ADR vocabulary,
+  no file paths. Model must recognize that this is an architectural concern.
+
+### Paired positive/negative at every level
+
+For each level, **two scenarios that differ only in whether a decided constraint is in play**:
+
+- **Positive (P)**: A decided constraint IS applicable → consulting `/os-adr:find` and/or
+  recording with `/os-adr:create` is correct.
+- **Negative (N)**: Matched framing (same vocabulary level, same form — question or intent)
+  but NO Accepted ADR constrains the task → consulting/creating an ADR is an over-trigger.
+
+The negative scenario is the discrimination test: wording tuned only for sensitivity ("always
+trigger") will fail negatives. Precision (correct non-triggering) is as important as recall.
+
+### Non-monotonic difficulty expected
+
+Difficulty does NOT increase uniformly L1 → L2 → L3. An explicit cue (L1) might be harder to
+miss than a conceptual one (L3), depending on model tier and wording. Per `running-autoresearch-skill-evals.md`, each level is validated independently and results reported in a per-level grid, not collapsed to a single threshold.
+
+## Two target ADR pairs, independent measurement
+
+**Run-set** (6 scenarios, frozen for measurement):
+- Target ADR 1: 0002 — "Job timeouts after 5 minutes with exponential backoff, max 3 attempts" (Accepted)
+- Target ADR 2: 0001 → 0002 (Superseded) for distractor validation
+- Scenarios: P1-L1, N1-L1, P2-L2, N2-L2, P3-L3, N3-L3 (all about job execution timeout/retry)
+
+**Reserve-set** (6 scenarios, held until run-set is contaminated by wording iteration):
+- Target ADR 2: 0003 — "Route all job notifications through central NotificationService" (Accepted)
+- Scenarios: P4-L1, N4-L1, P5-L2, N5-L2, P6-L3, N6-L3 (all about notification routing)
+
+Both sets are valid measurement surfaces; the split is for the protection protocol, not
+quality difference.
+
+## Scenario framing discipline (critical)
+
+The CLAUDE.md instruction carried verbatim in the fixture has two triggers:
+- Semantic: "architecture-level choice → run `/os-adr:find`"
+- Mechanical: "before your first edit to any existing source/config file → run `/os-adr:find`"
+
+**Negative scenarios avoid the mechanical trigger**: they're framed as pure questions or new
+module designs, so trigger #2 never fires. Then "no ADR touch" is a clean, valid over-trigger
+signal. If a negative scenario edits an existing file, the over-trigger criteria shift to
+"created an ADR claiming a phantom constraint," requiring judge fallback.
+
+## Fixture
+
+**Project:** `fixture/project/` — small Python async job queue system (`taskq`).
+
+**ADR history:** 6 ADRs (2026-02-01 to 2026-03-01), generated with plugin CLIs:
+- 0001: "Job timeout at 10s, simple retry" (Superseded)
+- 0002: "Job timeout at 5 min, exponential backoff, max 3 attempts" (Accepted, supersedes 0001)
+- 0003: "Route notifications through central service" (Accepted)
+- 0004: "Persist queue to SQLite" (Accepted)
+- 0005: "Failed jobs escalate to human review queue" (Accepted)
+- 0006: "All job state transitions logged to audit trail" (Accepted)
+
+Each ADR has `affected-paths` set (e.g., 0002 → [taskq/execution.py, taskq/queue.py]) so
+`bin/adr-find` can validate path-matching behavior if needed. `.graphifyignore` excludes
+markdown, so any future graph build (not part of this eval) would be pure-AST / reproducible.
+
+## Two-axis scoring
+
+- **Axis (a) — consultation trigger**
+  - **Positive scenarios**: mechanical PASS iff any `tool_use` block invoked an `os-adr:*`
+    skill, `bin/adr-*` CLI, or touched `docs/adr/`.
+  - **Negative scenarios**: INFORMATIONAL ONLY (recorded as `A:yes` or `A:no`, does NOT
+    determine pass/fail). Consultation is acceptable per CLAUDE.md's "cheap, deterministic CLI
+    call" endorsement. The discrimination measurement is not whether the model consulted, but
+    whether it **recorded a false decision** or **asserted a non-existent constraint**.
+  - Hook context (`SessionStart [os-adr]` event) is a precondition: missing hook → FAIL both
+    axes.
+
+- **Axis (b) — specific correctness**
+  - **Positive scenarios** (decided-constraint-applies): PASS via any ONE of, in order:
+    1. **created** — a new `docs/adr/NNNN-*.md` exists matching the scenario's topic pattern.
+    2. **cited-governing** — the final message cites the governing Accepted ADR by id
+       (P1→0002, P2→0005, P3→0002/0005/0006; reserve P4–P6→0003). The positives sit in
+       already-decided territory (each task restates what an Accepted ADR already decided),
+       so per the shipped plugin wording the correct behavior is find → cite → comply, NOT
+       create a duplicate ADR. This branch conforms the checker to this README's design
+       statement ("consulting `/os-adr:find` and/or recording with `/os-adr:create` is
+       correct"); the original checker required creation/proposal only, which failed exactly
+       the behavior the negatives reward (fixed 2026-07-06 after 1 spent rep, before the grid).
+       The TSV reasons column records which branch passed (`B-via:created|cited-governing|judge-proposed`).
+    3. **judge fallback** (haiku, `judge-rubric.md` — frozen, unchanged — reads only the final
+       message; stubbed via `ADR_EVAL_C_JUDGE_CMD` for self-tests).
+  - **Negative scenarios** (non-decisional tasks, investigate/test/debug framing):
+    - Mechanical FAIL if a new ADR file was created (over-trigger: creating a decision record
+      for a non-decisional task). This is the ONLY fail condition.
+    - Citation of existing ADRs in the final message is recorded as informational
+      `cited-adr:yes/no`, never a FAIL — the negatives sit deliberately in ADR-covered
+      domains, and truthfully citing a governing ADR while documenting/testing/auditing is
+      correct behavior. Separating true citations from false constraint claims needs
+      semantic judgment; a keyword match fails correct answers.
+    - Otherwise: PASS.
+
+**Overall PASS**:
+- **Positive scenarios**: both axes PASS.
+- **Negative scenarios**: axis (b) PASS only (axis a is informational, doesn't affect pass/fail).
+
+TSV format: `scenario  model  A:yes|no-or-PASS|FAIL  B:PASS|FAIL  PASS|FAIL  reasons`
+
+## Running the evaluation
+
+### Single cell (manual test)
+
+```bash
+eval-c/bin/sandbox P2-L2-execution /tmp/adr-eval-c-test
+eval-c/bin/run P2-L2-execution haiku /tmp/adr-eval-c-test --reps 1 --results /tmp/results.tsv
+cat /tmp/results.tsv
+```
+
+Each rep: fresh sandbox → `claude -p --model <tier> --output-format stream-json
+--dangerously-skip-permissions "<Task block>"` → `transcript.jsonl` written → `bin/check`
+appends one TSV row.
+
+### Grid run (full measurement)
+
+Once environment is stable (plugins cache fresh, hooks verified, fixture built):
+
+```bash
+# Run the full grid (12 scenarios × 2 tiers × N reps) in parallel, monitoring as they complete
+for scenario in P1-L1-execution N1-L1-execution P2-L2-execution N2-L2-execution P3-L3-execution N3-L3-execution; do
+  for model in haiku sonnet; do
+    bin/run "$scenario" "$model" /tmp/adr-eval-c-grid --reps 1 --results /tmp/adr-eval-c-grid/results.tsv &
+  done
+done
+wait
+cat /tmp/adr-eval-c-grid/results.tsv
+```
+
+### Per-level pass bars (baseline expectations)
+
+Eval C does not collapse to a single score. Results are reported **per level × per tier**,
+with discrimination measured by the **negative scenarios' axis (b) results** (unneeded ADR creation):
+
+| Level | Positive (P*-L*): should trigger | Negative (N*-L*): should NOT create/assert | Comment |
+|-------|---------------------------|------------------------|---------|
+| L1 (Explicit) | Expected: PASS both tiers (find/create) | Expected: PASS both tiers (no unneeded create) | Explicit form + vocabulary are strongest signals for both polarity |
+| L2 (Moderate) | Expected: PASS sonnet, may degrade haiku | Expected: PASS sonnet, may degrade haiku | Vocabulary inference is harder at L2 |
+| L3 (Conceptual) | May degrade on both tiers (miss intent) | May degrade on both tiers (harder to resist recording a "decision" at intent level) | Pure intent is hardest for both directions |
+
+**Axis (a) consultation rate** (informational, not pass/fail):
+- Report alongside axis (b) results. Model consulting the ADR system for negatives is acceptable;
+  don't penalize cheap, deterministic lookups. The discrimination is in axis (b): unneeded ADR creation.
+- The checker also records `cited-adr:yes/no` per negative (informational). Truthful citation of a
+  governing ADR while documenting/testing/auditing is CORRECT behavior — the negatives sit
+  deliberately in ADR-covered domains. Distinguishing a true citation from a false constraint claim
+  needs semantic judgment; a keyword check fails correct answers, so citation is never a FAIL.
+  If false-citation ever needs to be measured, that requires a narrow judge, not a regex.
+
+**Threshold hypothesis** (for post-baseline discussion, not part of first run):
+- Sonnet likely passes negatives reliably through L2 (no false create/cite), may struggle L3.
+- Haiku likely passes L1 negatives consistently, may over-trigger (create/assert) at L2/L3.
+- Negatives are the core measurement; if a tier fails negatives, it means over-recording —
+  wording iteration must then address restraint (when NOT to create), not just trigger enhancement.
+
+Evaluate the **per-level grid** (A:consultation rates + B:pass/fail) first. Summary threshold only
+if results show a clear pattern (e.g., "PASS positives through L2, PASS negatives through L2,
+PASS with consultation rate Y%").
+
+### Optimizing with `/autoresearch` (later stage)
+
+Not part of building this harness. When wording iteration starts:
+
+1. **Freeze the run-set**: results and scenarios are locked as baseline.
+2. **Iterate only on wording** (SKILL.md, hook note, CLAUDE.md sections) — not checker,
+   fixtures, scenarios, or judge rubric.
+3. **Reduced inner loop**: target failing cells + one passing control per tier per level.
+4. **Parallel runs**: drive per-cell `bin/run` scripts in the background (each cell fully
+   independent).
+5. **Re-run full grid** once wording stabilizes.
+6. **If run-set degradation occurs**, stop iteration, document the contamination, and
+   **switch all future work to the reserve-set** for measurement.
+
+See `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md` for full discipline.
+
+## Reserve set protection
+
+Run-set and reserve-set are **structurally identical** (3 levels, P/N pairs on same target
+ADRs) but **different scenarios** so they are not interchangeable for measurement:
+
+- **Before any wording iteration**: run-set is the measurement set.
+- **After first iteration against run-set results**: run-set is contaminated (wording tuned to
+  it). Reserve-set becomes the fresh measurement surface; run-set results are historical
+  only.
+- **Lock in writing**: when a change triggers an iteration decision, document that fact in a
+  vault note or project comment so future reviewers know which set is the current measurement
+  baseline.
+
+The reserve-set scenarios and fixtures are never read informally, never "tried out" before
+being used for grid runs — same held-out discipline as the run-set.
+
+## Layout
+
+| Path | What |
+| --- | --- |
+| `fixture/project/` | Python async job queue, 6-ADR history |
+| `scenarios/P1-L1-execution.md`, `N1-L1-execution.md`, ... | run-set prompts (6 scenarios) |
+| `scenarios-reserve/P4-L1-notifications.md`, ... | reserve-set prompts (6 scenarios) |
+| `bin/sandbox <Sn> <dest>` | fresh git-initialized sandbox; all scenarios use same fixture base |
+| `bin/run <Sn> <model> <workdir> [--reps N] [--results FILE]` | headless runner (the ONLY valid execution mode) |
+| `bin/check <Sn> <sandbox> [--tsv <model>]` | two-axis checker with negative-scenario support; exit 0/1; TSV for autoresearch |
+| `bin/self-test [workdir]` | model-free both-directions harness validation (fabricates transcripts; never runs Task blocks) |
+| `judge-rubric.md` | frozen rubric for positive-scenario axis (b) LLM judge fallback |
+
+## Why headless-only (no Agent-tool subagents)
+
+Eval B (Eval A, actually — the in-session subagent eval) uses in-session subagents because
+it measures *prompted skill-execution*. Eval C measures **unprompted detection**, so each
+model under test must get a fresh `SessionStart` hook context. In-session subagents inherit
+the parent session and never see a fresh hook. `bin/run` spawns each rep as a fresh
+`claude -p` with cwd = sandbox; the transcript's `system/hook_response` event proves the hook
+fired. (Verified 2026-07-03 by eval-b author: haiku quoted the hook note verbatim in a
+sandbox session.)
+
+## Fixture regeneration
+
+The fixture's `graphify-out/` is **not included** (Eval C doesn't measure graph reach; it
+measures cue explicitness). The ADR history was generated with the plugin's own CLIs
+(`bin/adr-init`, `bin/adr-new`) so frontmatter and index match the shipped format, including
+the mechanically-superseded 0001→0002 pair.
+
+To rebuild ADRs from scratch (not normally needed):
+
+```bash
+FIXTURE="plugins/os-adr/eval-c/fixture/project"
+rm -rf "$FIXTURE/docs/adr"
+ruby plugins/os-adr/bin/adr-init --root "$FIXTURE" --template nygard
+# Then re-create each ADR using bin/adr-new as this README documents
+```
+
+## Adding a scenario (later stage)
+
+Add `scenarios/X.md` (or `scenarios-reserve/X.md`):
+- Task block + checker metadata (target ADR, topic pattern, axis-b fallback description)
+- Scenario class in `bin/check` (Ruby Scenario subclass with axis-b logic)
+- Extend `bin/sandbox` path list
+- Extend `bin/self-test` both directions (perfect + over-trigger for new scenario)
+- Document in per-level grid above if adding a level
+- Keep the held-out rule: the new Task block gets no informal trial runs
+
+## Verification before first grid run
+
+```bash
+# 1. Smoke-test the fixture and hooks
+cd plugins/os-adr/eval-c/fixture/project && git status
+
+# 2. Verify self-test passes (model-free, no tokens spent)
+plugins/os-adr/eval-c/bin/self-test /tmp/adr-eval-c-test
+
+# 3. Verify one cell runs headless
+plugins/os-adr/eval-c/bin/run P1-L1-execution haiku /tmp/adr-eval-c-test-run --reps 1
+
+# 4. Check the TSV
+cat /tmp/adr-eval-c-test-run/results.tsv
+```
+
+All must pass before running the first grid.

plugins/os-adr/eval-c/bin/check

@@ -0,0 +1,390 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+# Two-axis, deterministic-first checker for Eval C (ambiguity-ladder discrimination).
+#
+#   check <P*-L*|N*-L*> <sandbox-root> [--tsv <model-label>]
+#
+# Reads <sandbox>/transcript.jsonl (stream-json, written by bin/run).
+#
+# Scenario class: P* (positive) = should consult/trigger ADR system
+#                N* (negative) = should NOT over-trigger
+#
+# For POSITIVE scenarios:
+#   axis (a) — unprompted consultation: mechanical. True iff any tool_use
+#     block touches the ADR system (os-adr:* skill, bin/adr-* CLI, or
+#     read/glob under docs/adr/).
+#   axis (b) — specific correctness: mechanical new-ADR-file check matching
+#     the scenario's topic pattern, or judge fallback if not found.
+#
+# For NEGATIVE scenarios:
+#   axis (a) — over-trigger detection: mechanical. True iff NO tool_use
+#     touched the ADR system (inverse of positive).
+#   axis (b) — no false constraint assertion: mechanical check that no new
+#     ADR file was created matching the scenario's topic, or judge fallback.
+#
+# Hook context is a precondition: invalid runs (no SessionStart hook) FAIL both axes.
+#
+# Overall PASS requires both axes. TSV mode for autoresearch integration.
+# Exit 0 on PASS, 1 on FAIL, 2 on usage error.
+
+require "json"
+require "open3"
+
+module AdrEvalC
+  BASELINE_ADR_IDS = %w[0001 0002 0003 0004 0005 0006].freeze
+  CONFLICT = /conflict|violat|contradic|goes against|decision|record|architecture/i
+
+  class Transcript
+    def self.load(path)
+      return nil unless File.exist?(path)
+      events = File.readlines(path).filter_map do |line|
+        JSON.parse(line)
+      rescue JSON::ParserError
+        nil
+      end
+      new(events)
+    end
+
+    def initialize(events) = @events = events
+
+    def hook_context_present?
+      @events.any? do |e|
+        e["type"] == "system" && e["subtype"] == "hook_response" &&
+          e["hook_name"].to_s.start_with?("SessionStart") &&
+          e.to_json.include?("[os-adr]")
+      end
+    end
+
+    def tool_uses
+      assistant_blocks.select { |b| b["type"] == "tool_use" }
+    end
+
+    def assistant_text
+      texts = assistant_blocks.select { |b| b["type"] == "text" }.map { |b| b["text"] }
+      texts << final_message
+      texts.compact.join("\n")
+    end
+
+    def final_message
+      result = @events.reverse.find { |e| e["type"] == "result" }
+      result && result["result"].is_a?(String) ? result["result"] : nil
+    end
+
+    private
+
+    def assistant_blocks
+      @events.select { |e| e["type"] == "assistant" }
+             .flat_map { |e| e.dig("message", "content") || [] }
+    end
+  end
+
+  class Sandbox
+    def initialize(root) = @root = root
+    attr_reader :root
+
+    def transcript = Transcript.load(File.join(@root, "transcript.jsonl"))
+
+    def new_adr_files
+      Dir.glob(File.join(@root, "docs/adr/[0-9]*.md")).sort.reject do |path|
+        BASELINE_ADR_IDS.include?(File.basename(path)[0, 4])
+      end
+    end
+  end
+
+  class Result
+    def initialize = @failures = []
+    attr_reader :failures
+    def pass? = @failures.empty?
+
+    def expect(condition, reason)
+      @failures << reason unless condition
+      !!condition
+    end
+  end
+
+  class InformationalResult
+    # For informational metrics (doesn't affect pass/fail, just recorded)
+    def initialize(value)
+      @value = value
+    end
+
+    attr_reader :value
+    def pass? = true  # Informational results never fail
+  end
+
+  class Judge
+    RUBRIC = File.read(File.expand_path("../judge-rubric.md", __dir__))
+                 .split("---", 2).last.strip
+
+    def self.command
+      ENV.fetch("ADR_EVAL_C_JUDGE_CMD",
+                "claude -p --model haiku --dangerously-skip-permissions")
+    end
+
+    def proposed_adr?(final_message)
+      prompt = "#{RUBRIC}\n\n#{final_message}"
+      output, status = Open3.capture2(self.class.command, stdin_data: prompt)
+      raise "judge command failed: #{self.class.command}" unless status.success?
+      verdict = output[/\b(YES|NO)\b/, 1]
+      raise "judge gave no YES/NO verdict: #{output.inspect}" if verdict.nil?
+      verdict == "YES"
+    end
+  end
+
+  class Scenario
+    def initialize(sandbox, judge: Judge.new)
+      @sb = sandbox
+      @judge = judge
+    end
+
+    def run
+      axis_a = Result.new
+      axis_b = Result.new
+      transcript = @sb.transcript
+      if transcript.nil?
+        axis_a.expect(false, "no transcript.jsonl in sandbox")
+        axis_b.expect(false, "no transcript.jsonl in sandbox")
+      elsif !transcript.hook_context_present?
+        axis_a.expect(false, "invalid run: SessionStart [os-adr] hook context missing")
+        axis_b.expect(false, "invalid run: SessionStart [os-adr] hook context missing")
+      else
+        check_axis_a(axis_a, transcript)
+        check_axis_b(axis_b, transcript)
+      end
+      [axis_a, axis_b]
+    end
+
+    private
+
+    attr_reader :sb, :judge
+
+    def check_axis_a(result, transcript)
+      # Override in subclasses
+      raise NotImplementedError, "subclasses must override check_axis_a"
+    end
+
+    def adr_touch?(tool_use)
+      probe = "#{tool_use['name']} #{tool_use['input'].to_json}"
+      probe.match?(%r{os-adr|bin/adr-|adr-(find|new|init|detect|migrate)\b|docs/adr})
+    end
+  end
+
+  # --- Positive scenarios: should trigger ADR system ---
+
+  class PositiveScenario < Scenario
+    # Axis (b) passes via any ONE of, in order:
+    #   1. created  — a new topical ADR file exists (the model recorded a decision)
+    #   2. cited    — the final message cites the GOVERNING Accepted ADR by id.
+    #      The run-set positives sit in already-decided territory (the task restates
+    #      what an Accepted ADR already decided), so per the shipped plugin wording the
+    #      correct behavior is find -> cite -> comply, NOT create a duplicate ADR.
+    #      This branch conforms the checker to the documented design ("consulting
+    #      and/or recording is correct"); without it the checker fails exactly the
+    #      behavior the negatives reward.
+    #   3. judge    — frozen rubric fallback: final message proposes recording.
+    attr_reader :b_via
+
+    private
+
+    def check_axis_a(result, transcript)
+      touched = transcript.tool_uses.any? { |tu| adr_touch?(tu) }
+      result.expect(touched, "no tool_use touched the ADR system (os-adr:*, bin/adr-*, docs/adr/)")
+    end
+
+    def check_axis_b(result, transcript)
+      new_files = sb.new_adr_files
+      if new_files.any?
+        matching = new_files.select { |p| File.read(p).match?(topic) }
+        @b_via = "created" if matching.any?
+        result.expect(matching.any?,
+                      "new ADR exists but matches no topic term #{topic.inspect}")
+        return
+      end
+      final = transcript.final_message
+      unless final
+        result.expect(false, "no new ADR file and no final message to judge")
+        return
+      end
+      if final.match?(governing)
+        @b_via = "cited-governing"
+        return
+      end
+      if judge.proposed_adr?(final)
+        @b_via = "judge-proposed"
+      else
+        result.expect(false,
+                      "no new ADR, no governing-ADR citation #{governing.inspect} in final message; judge: does not propose recording")
+      end
+    end
+  end
+
+  class P1L1Execution < PositiveScenario
+    def topic = /timeout|retry|backoff|exponential|attempt/i
+    def governing = /\b0002\b/
+  end
+
+  class P2L2Execution < PositiveScenario
+    def topic = /escalat|termin|failure|review|abandon/i
+    def governing = /\b0005\b/
+  end
+
+  class P3L3Execution < PositiveScenario
+    def topic = /abandon|finish|complet|reliability|guarant/i
+    def governing = /\b000[256]\b/
+  end
+
+  class P4L1Notifications < PositiveScenario
+    def topic = /notif|route|service|webhook|event/i
+    def governing = /\b0003\b/
+  end
+
+  class P5L2Notifications < PositiveScenario
+    def topic = /webhook|deliv|retry|notification/i
+    def governing = /\b0003\b/
+  end
+
+  class P6L3Notifications < PositiveScenario
+    def topic = /notif|channel|routing|delivery|unified/i
+    def governing = /\b0003\b/
+  end
+
+  # --- Negative scenarios: should NOT create false ADRs or assert phantom constraints ---
+
+  class NegativeScenario < Scenario
+    def check_axis_a(result, transcript)
+      # Axis (a) for negatives is INFORMATIONAL ONLY (doesn't affect pass/fail)
+      # Record whether ADR system was consulted (yes/no)
+      touched = transcript.tool_uses.any? { |tu| adr_touch?(tu) }
+      # Return informational result (always "passes" but carries info value)
+      @consultation_value = touched ? "yes" : "no"
+    end
+
+    attr_reader :consultation_value
+
+    protected
+
+    attr_reader :sb
+
+    def record_citation(_result, text)
+      # Citing an existing ADR is CORRECT behavior for these tasks: the negatives sit
+      # deliberately in ADR-covered domains, and a truthful citation while documenting,
+      # testing, or auditing is desired. Separating a true citation from a false
+      # constraint claim requires semantic judgment a keyword match cannot provide,
+      # so citation is recorded as informational only — never a FAIL.
+      @citation_value = text.match?(/\b000[1-6]\b|ADR \d{4}/i) ? "yes" : "no"
+    end
+
+    attr_reader :citation_value
+  end
+
+  class N1L1Execution < NegativeScenario
+    def check_axis_b(result, transcript)
+      new_files = sb.new_adr_files
+      result.expect(new_files.empty?,
+                    "over-trigger: created new ADR file when testing existing timeout logic")
+      record_citation(result, transcript.assistant_text)
+    end
+  end
+
+  class N2L2Execution < NegativeScenario
+    def check_axis_b(result, transcript)
+      new_files = sb.new_adr_files
+      result.expect(new_files.empty?,
+                    "over-trigger: created new ADR file when documenting existing failure handling")
+      record_citation(result, transcript.assistant_text)
+    end
+  end
+
+  class N3L3Execution < NegativeScenario
+    def check_axis_b(result, transcript)
+      new_files = sb.new_adr_files
+      result.expect(new_files.empty?,
+                    "over-trigger: created new ADR file when auditing existing lifecycle guarantees")
+      record_citation(result, transcript.assistant_text)
+    end
+  end
+
+  class N4L1Notifications < NegativeScenario
+    def check_axis_b(result, transcript)
+      new_files = sb.new_adr_files
+      result.expect(new_files.empty?,
+                    "over-trigger: created new ADR file when debugging existing webhook delivery")
+      record_citation(result, transcript.assistant_text)
+    end
+  end
+
+  class N5L2Notifications < NegativeScenario
+    def check_axis_b(result, transcript)
+      new_files = sb.new_adr_files
+      result.expect(new_files.empty?,
+                    "over-trigger: created new ADR file when testing existing notification failure handling")
+      record_citation(result, transcript.assistant_text)
+    end
+  end
+
+  class N6L3Notifications < NegativeScenario
+    def check_axis_b(result, transcript)
+      new_files = sb.new_adr_files
+      result.expect(new_files.empty?,
+                    "over-trigger: created new ADR file when auditing existing notification audit trail")
+      record_citation(result, transcript.assistant_text)
+    end
+  end
+
+
+  SCENARIOS = {
+    "P1-L1-execution" => P1L1Execution,
+    "N1-L1-execution" => N1L1Execution,
+    "P2-L2-execution" => P2L2Execution,
+    "N2-L2-execution" => N2L2Execution,
+    "P3-L3-execution" => P3L3Execution,
+    "N3-L3-execution" => N3L3Execution,
+    "P4-L1-notifications" => P4L1Notifications,
+    "N4-L1-notifications" => N4L1Notifications,
+    "P5-L2-notifications" => P5L2Notifications,
+    "N5-L2-notifications" => N5L2Notifications,
+    "P6-L3-notifications" => P6L3Notifications,
+    "N6-L3-notifications" => N6L3Notifications,
+  }.freeze
+end
+
+scenario_id, sandbox_root = ARGV[0], ARGV[1]
+tsv_model = ARGV[2] == "--tsv" ? (ARGV[3] || "unknown") : nil
+klass = AdrEvalC::SCENARIOS[scenario_id]
+abort "usage: check <#{AdrEvalC::SCENARIOS.keys.join('|')}> <sandbox-root> [--tsv <model>]" if klass.nil? || sandbox_root.nil?
+abort "no such sandbox: #{sandbox_root}" unless File.directory?(sandbox_root)
+
+checker = klass.new(AdrEvalC::Sandbox.new(File.expand_path(sandbox_root)))
+axis_a, axis_b = checker.run
+is_negative = scenario_id.start_with?("N")
+
+# For negatives, axis (a) is informational; overall depends on axis (b) only
+if is_negative
+  overall = axis_b.pass?
+  axis_a_label = "A:#{checker.consultation_value || 'unknown'}"
+else
+  overall = axis_a.pass? && axis_b.pass?
+  axis_a_label = axis_a.pass? ? "A:PASS" : "A:FAIL"
+end
+
+reasons = (axis_a.failures + axis_b.failures).join("; ")
+if is_negative && checker.respond_to?(:citation_value) && checker.citation_value
+  reasons = ["cited-adr:#{checker.citation_value}", reasons].reject(&:empty?).join("; ")
+end
+if !is_negative && checker.respond_to?(:b_via) && checker.b_via
+  reasons = ["B-via:#{checker.b_via}", reasons].reject(&:empty?).join("; ")
+end
+
+if tsv_model
+  puts [scenario_id, tsv_model,
+        axis_a_label,
+        axis_b.pass? ? "B:PASS" : "B:FAIL",
+        overall ? "PASS" : "FAIL", reasons].join("\t")
+else
+  axis_a_desc = is_negative ? "consultation:#{checker.consultation_value || 'unknown'}" : (axis_a.pass? ? "PASS" : "FAIL")
+  puts "#{overall ? 'PASS' : 'FAIL'} #{scenario_id}  " \
+       "(axis-a #{axis_a_desc}, axis-b #{axis_b.pass? ? 'PASS' : 'FAIL'})"
+  (axis_a.failures + axis_b.failures).each { |f| puts "  - #{f}" }
+end
+exit(overall ? 0 : 1)

plugins/os-adr/eval-c/bin/run

@@ -0,0 +1,57 @@
+#!/usr/bin/env bash
+# Headless runner for Eval C — the ONLY valid execution mode (fresh `claude -p`
+# with cwd = sandbox so the real SessionStart hook fires).
+#
+# Usage: run <scenario> <model> <workdir> [--reps N] [--results FILE]
+#
+#   scenario  P*-L*|N*-L* (e.g., P1-L1-execution, N2-L2-execution)
+#   model     haiku|sonnet|opus|...
+#   workdir   sandboxes created under here as <scenario>-<model>-rN
+#   --reps    repeated executions (default 1; design.md Decision 5)
+#
+# Each rep: fresh sandbox -> claude -p with only the scenario task prompt
+# (no system-level hints) -> full stream-json transcript saved ->
+# bin/check appends one TSV row.
+set -euo pipefail
+
+SCENARIO="${1:?usage: run <scenario> <model> <workdir> [--reps N] [--results FILE]}"
+MODEL="${2:?model required (haiku|sonnet|...)}"
+WORKDIR="${3:?workdir required}"
+shift 3
+
+REPS=1
+RESULTS=""
+while [ $# -gt 0 ]; do
+  case "$1" in
+    --reps)    REPS="${2:?--reps needs a number}"; shift 2 ;;
+    --results) RESULTS="${2:?--results needs a path}"; shift 2 ;;
+    *) echo "unknown option: $1" >&2; exit 2 ;;
+  esac
+done
+
+EVAL_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+SCENARIO_FILE="$EVAL_ROOT/scenarios/${SCENARIO}.md"
+if [ ! -f "$SCENARIO_FILE" ]; then
+  SCENARIO_FILE="$EVAL_ROOT/scenarios-reserve/${SCENARIO}.md"
+fi
+[ -f "$SCENARIO_FILE" ] || { echo "unknown scenario: $SCENARIO (checked scenarios/ and scenarios-reserve/)" >&2; exit 2; }
+mkdir -p "$WORKDIR"
+RESULTS="${RESULTS:-$WORKDIR/results.tsv}"
+
+TASK="$(awk '/^## Task/{found=1; next} found' "$SCENARIO_FILE")"
+
+for rep in $(seq 1 "$REPS"); do
+  SANDBOX="$WORKDIR/$SCENARIO-$MODEL-r$rep"
+  "$EVAL_ROOT/bin/sandbox" "$SCENARIO" "$SANDBOX" >/dev/null
+
+  # cwd = sandbox: the SessionStart hook resolves the project root from here.
+  (cd "$SANDBOX" && claude -p \
+      --model "$MODEL" \
+      --output-format stream-json --verbose \
+      --dangerously-skip-permissions \
+      "$TASK" > transcript.jsonl) || echo "claude exited non-zero for $SANDBOX" >&2
+
+  # bin/check exits 1 on FAIL; without the || true, set -euo pipefail would
+  # abort the rep loop on the first failing rep (found on the first grid run).
+  "$EVAL_ROOT/bin/check" "$SCENARIO" "$SANDBOX" --tsv "$MODEL" | tee -a "$RESULTS" || true
+done

plugins/os-adr/eval-c/bin/sandbox

@@ -0,0 +1,30 @@
+#!/bin/bash
+# Create a fresh, git-initialized sandbox copy of the Eval C fixture.
+# Usage: sandbox <P*-L*|N*-L*> <dest-dir>
+#
+# Eval C does not use graphify-out (ladder measures cue explicitness, not graph reach).
+set -euo pipefail
+
+SCENARIO="${1:?usage: sandbox <scenario> <dest-dir>}"
+DEST="${2:?usage: sandbox <scenario> <dest-dir>}"
+EVAL_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+FIXTURE="$EVAL_ROOT/fixture/project"
+
+# All scenarios use the same fixture without graphify-out
+case "$SCENARIO" in
+  P1-L1-execution|N1-L1-execution|P2-L2-execution|N2-L2-execution|P3-L3-execution|N3-L3-execution|\
+  P4-L1-notifications|N4-L1-notifications|P5-L2-notifications|N5-L2-notifications|P6-L3-notifications|N6-L3-notifications)
+    :
+    ;;
+  *) echo "unknown scenario: $SCENARIO" >&2; exit 2 ;;
+esac
+
+if [ -e "$DEST" ]; then echo "refusing to overwrite existing $DEST" >&2; exit 2; fi
+mkdir -p "$DEST"
+cp -r "$FIXTURE/." "$DEST/"
+rm -rf "$DEST/.os-adr"
+
+git -C "$DEST" init -q
+git -C "$DEST" add -A
+git -C "$DEST" -c user.email=eval@local -c user.name=eval commit -qm "fixture baseline"
+echo "$DEST"

plugins/os-adr/eval-c/bin/self-test

@@ -0,0 +1,224 @@
+#!/usr/bin/env bash
+# Self-test the Eval C harness in both directions, model-free:
+#
+#   1. For each POSITIVE scenario, fabricate a "perfect" transcript/sandbox
+#      state (model did right thing) -> bin/check must PASS.
+#   2. For each POSITIVE scenario, fabricate an "untouched" run (never
+#      consulted ADR system) -> bin/check must FAIL, axis (a) FAIL.
+#   3. For each NEGATIVE scenario, fabricate a "proper" transcript (no ADR
+#      touch, no false ADR creation) -> bin/check must PASS.
+#   4. For each NEGATIVE scenario, fabricate an "over-trigger" run (consulted
+#      ADR system or created ADR when shouldn't) -> bin/check must FAIL.
+#   5. Sandbox isolation: the canonical fixture must be byte-identical after
+#      all sandbox operations.
+#
+# The judge is stubbed via ADR_EVAL_C_JUDGE_CMD so this never spends tokens.
+# It never touches the held-out scenario Task blocks.
+#
+# Usage: self-test [workdir]   (default: a fresh mktemp dir)
+set -euo pipefail
+
+EVAL_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+PLUGIN_ROOT="$(cd "$EVAL_ROOT/.." && pwd)"
+WORKDIR="${1:-$(mktemp -d /tmp/adr-eval-c-selftest.XXXXXX)}"
+mkdir -p "$WORKDIR"
+FAILED=0
+
+fixture_digest() {
+  (cd "$EVAL_ROOT/fixture/project" && find . -type f ! -path "./.os-adr/*" -print0 \
+    | sort -z | xargs -0 md5sum | md5sum)
+}
+DIGEST_BEFORE="$(fixture_digest)"
+
+# --- transcript fabrication -------------------------------------------------
+
+hook_event() {
+  printf '%s\n' '{"type":"system","subtype":"hook_response","hook_name":"SessionStart:startup","output":"[os-adr] This project records architecture decisions in docs/adr/"}'
+}
+
+tool_use_event() { # $1 = tool name, $2 = input JSON
+  printf '{"type":"assistant","message":{"role":"assistant","content":[{"type":"tool_use","name":"%s","input":%s}]}}\n' "$1" "$2"
+}
+
+result_event() { # $1 = final text (single line)
+  python3 -c 'import json,sys; print(json.dumps({"type":"result","subtype":"success","result":sys.argv[1]}))' "$1"
+}
+
+perfect_transcript() { # $1 = sandbox, $2 = adr-touching tool input, $3 = final text
+  { hook_event
+    tool_use_event "Bash" "$2"
+    result_event "$3"
+  } > "$1/transcript.jsonl"
+}
+
+untouched_transcript() { # $1 = sandbox
+  { hook_event
+    tool_use_event "Read" '{"file_path":"taskq/execution.py"}'
+    result_event "I made the change you asked for."
+  } > "$1/transcript.jsonl"
+}
+
+expect() { # $1 = PASS|FAIL, $2 = label, then the check command
+  local want="$1" label="$2"; shift 2
+  local got=PASS
+  "$@" >/dev/null 2>&1 || got=FAIL
+  if [ "$got" = "$want" ]; then
+    echo "ok   $label ($want)"
+  else
+    echo "FAIL $label: wanted $want, got $got"
+    "$@" || true
+    FAILED=1
+  fi
+}
+
+FIND_INPUT='{"command":"ruby '"$PLUGIN_ROOT"'/bin/adr-find --root . --paths taskq/execution.py"}'
+
+# --- 1. POSITIVE runs: perfect runs must PASS, untouched must FAIL axis-a ---
+
+declare -A POSITIVE_SCENARIOS=(
+  [P1-L1-execution]="Added 5-minute timeout with exponential backoff retry logic. Should we record this timeout strategy as an ADR?"
+  [P2-L2-execution]="Terminal failures now escalate to a review queue. This is a significant policy decision — should we record it?"
+  [P3-L3-execution]="Implemented guarantee that no work vanishes silently. All jobs either complete, fail with logging, or are explicitly abandoned. This is an architectural commitment."
+  [P4-L1-notifications]="Refactored all notifications to route through central NotificationService. This is a major coupling decision — should we record it as an ADR?"
+  [P5-L2-notifications]="Webhook delivery now retries with exponential backoff on failure. This is a reliability policy we should document as a decision."
+  [P6-L3-notifications]="Ensured all user-facing notifications go through one coherent pathway. This is a core architectural guarantee."
+)
+
+echo "Testing POSITIVE scenarios (perfect runs):"
+for sc in "${!POSITIVE_SCENARIOS[@]}"; do
+  sb="$WORKDIR/perfect-$sc"
+  "$EVAL_ROOT/bin/sandbox" "$sc" "$sb" >/dev/null
+
+  # Create a new ADR to simulate the model recording the decision
+  title=$(echo "${POSITIVE_SCENARIOS[$sc]}" | head -c 60)
+  printf '{"title": "%s", "context": "c", "decision": "d", "consequences": "q", "alternatives": "a"}' \
+    "$title" | ruby "$PLUGIN_ROOT/bin/adr-new" --root "$sb" >/dev/null 2>&1 || true
+
+  perfect_transcript "$sb" "$FIND_INPUT" "${POSITIVE_SCENARIOS[$sc]}"
+  expect PASS "perfect $sc" "$EVAL_ROOT/bin/check" "$sc" "$sb"
+done
+
+echo ""
+echo "Testing POSITIVE scenarios (untouched runs must FAIL axis-a):"
+for sc in "${!POSITIVE_SCENARIOS[@]}"; do
+  sb="$WORKDIR/untouched-$sc"
+  "$EVAL_ROOT/bin/sandbox" "$sc" "$sb" >/dev/null
+  untouched_transcript "$sb"
+  expect FAIL "untouched $sc" \
+    env ADR_EVAL_C_JUDGE_CMD="printf NO" "$EVAL_ROOT/bin/check" "$sc" "$sb"
+  row="$(env ADR_EVAL_C_JUDGE_CMD="printf NO" "$EVAL_ROOT/bin/check" "$sc" "$sb" --tsv selftest || true)"
+  case "$row" in
+    *$'\t'A:FAIL$'\t'*) echo "ok   untouched $sc axis-a FAIL confirmed" ;;
+    *) echo "FAIL untouched $sc: axis-a not FAIL in TSV: $row"; FAILED=1 ;;
+  esac
+done
+
+# --- 2. NEGATIVE runs: proper runs must PASS, over-triggers must FAIL ---
+
+declare -A NEGATIVE_SCENARIOS=(
+  [N1-L1-execution]="Designed a clean callback-based notification subscription API. No existing architecture governs this."
+  [N2-L2-execution]="Implemented priority-based job routing with high-priority jobs processed first. This is a new queue feature."
+  [N3-L3-execution]="Built performance dashboards showing real-time job system metrics. Orthogonal to execution logic."
+  [N4-L1-notifications]="Improved the audit log format to JSON lines for better parsing. Just formatting, no routing change."
+  [N5-L2-notifications]="Added in-memory caching of compiled notification templates for speed. Orthogonal to notification routing."
+  [N6-L3-notifications]="Optimized historical report queries by adding database indexes. Query performance optimization."
+)
+
+echo ""
+echo "Testing NEGATIVE scenarios (proper runs — axis b PASS with or without consultation):"
+for sc in "${!NEGATIVE_SCENARIOS[@]}"; do
+  sb="$WORKDIR/proper-$sc"
+  "$EVAL_ROOT/bin/sandbox" "$sc" "$sb" >/dev/null
+  untouched_transcript "$sb"  # No ADR touch: axis (a) informational, axis (b) PASS
+  expect PASS "proper $sc (no ADR touch, no false ADR created)" \
+    env ADR_EVAL_C_JUDGE_CMD="printf NO" "$EVAL_ROOT/bin/check" "$sc" "$sb"
+done
+
+echo ""
+echo "Testing NEGATIVE scenarios (over-trigger: create ADR when shouldn't):"
+for sc in "${!NEGATIVE_SCENARIOS[@]}"; do
+  sb="$WORKDIR/overtrigger-create-$sc"
+  "$EVAL_ROOT/bin/sandbox" "$sc" "$sb" >/dev/null
+  # Simulate over-trigger: create an ADR when shouldn't
+  printf '{"title": "Spurious ADR", "context": "c", "decision": "d", "consequences": "q", "alternatives": "a"}' \
+    | ruby "$PLUGIN_ROOT/bin/adr-new" --root "$sb" >/dev/null 2>&1 || true
+  perfect_transcript "$sb" "$FIND_INPUT" "I recorded this decision."
+  expect FAIL "over-trigger-create $sc (created ADR when no decision)" \
+    "$EVAL_ROOT/bin/check" "$sc" "$sb"
+done
+
+echo ""
+echo "Testing NEGATIVE scenarios (truthful ADR citation must PASS — citation is informational, not a FAIL):"
+# The negatives sit deliberately in ADR-covered domains; a model that consults and
+# truthfully cites a governing ADR while documenting/testing is behaving correctly.
+# Only ADR *creation* fails a negative. This guards the checker against regressing
+# to keyword-matching citations as failures.
+sb="$WORKDIR/truthful-citation-N1"
+"$EVAL_ROOT/bin/sandbox" N1-L1-execution "$sb" >/dev/null
+{ hook_event
+  tool_use_event "Read" '{"file_path":"taskq/execution.py"}'
+  result_event "I wrote tests for the existing timeout logic. ADR 0002 requires that timeouts stay at 5 minutes, so the tests assert the 5-minute timeout and exponential backoff behavior as decided."
+} > "$sb/transcript.jsonl"
+expect PASS "truthful-citation N1 (cites governing ADR 0002 while testing — no ADR created)" \
+  "$EVAL_ROOT/bin/check" N1-L1-execution "$sb"
+
+echo ""
+echo "Testing POSITIVE compliance path (cites governing ADR, creates nothing — must PASS without judge):"
+# The run-set positives sit in already-decided territory; find -> cite -> comply is
+# correct per the shipped wording. Judge stubbed NO to prove the pass is mechanical.
+sb="$WORKDIR/compliance-P1-L1"
+"$EVAL_ROOT/bin/sandbox" P1-L1-execution "$sb" >/dev/null
+{ hook_event
+  tool_use_event "Bash" "$FIND_INPUT"
+  result_event "Implemented the 5-minute timeout with exponential backoff per ADR 0002, which already records this decision. No new ADR needed."
+} > "$sb/transcript.jsonl"
+expect PASS "compliance P1-L1 (cites governing ADR 0002, no ADR created, judge stubbed NO)" \
+  env ADR_EVAL_C_JUDGE_CMD="printf NO" "$EVAL_ROOT/bin/check" P1-L1-execution "$sb"
+row="$(env ADR_EVAL_C_JUDGE_CMD="printf NO" "$EVAL_ROOT/bin/check" P1-L1-execution "$sb" --tsv selftest || true)"
+case "$row" in
+  *B-via:cited-governing*) echo "ok   compliance P1-L1 passed via cited-governing branch" ;;
+  *) echo "FAIL compliance P1-L1: expected B-via:cited-governing in TSV: $row"; FAILED=1 ;;
+esac
+
+# A wrong-ADR citation must NOT satisfy the governing branch (falls through to judge)
+sb="$WORKDIR/wrongcite-P1-L1"
+"$EVAL_ROOT/bin/sandbox" P1-L1-execution "$sb" >/dev/null
+{ hook_event
+  tool_use_event "Bash" "$FIND_INPUT"
+  result_event "Implemented timeouts. See ADR 0004 for the SQLite persistence design."
+} > "$sb/transcript.jsonl"
+expect FAIL "wrong-cite P1-L1 (cites non-governing ADR 0004 only, judge stubbed NO)" \
+  env ADR_EVAL_C_JUDGE_CMD="printf NO" "$EVAL_ROOT/bin/check" P1-L1-execution "$sb"
+
+# Missing hook context should invalidate both positive and negative
+echo ""
+echo "Testing hook-context precondition (invalid run):"
+sb="$WORKDIR/nohook-P1-L1"
+"$EVAL_ROOT/bin/sandbox" P1-L1-execution "$sb" >/dev/null
+{ tool_use_event "Bash" "$FIND_INPUT"; result_event "Made the change."; } > "$sb/transcript.jsonl"
+expect FAIL "P1-L1 without SessionStart hook (invalid run)" "$EVAL_ROOT/bin/check" P1-L1-execution "$sb"
+
+sb="$WORKDIR/nohook-N1-L1"
+"$EVAL_ROOT/bin/sandbox" N1-L1-execution "$sb" >/dev/null
+{ tool_use_event "Bash" "$FIND_INPUT"; result_event "Made the change."; } > "$sb/transcript.jsonl"
+expect FAIL "N1-L1 without SessionStart hook (invalid run)" "$EVAL_ROOT/bin/check" N1-L1-execution "$sb"
+
+# --- 3. fixture isolation ---
+
+echo ""
+DIGEST_AFTER="$(fixture_digest)"
+if [ "$DIGEST_BEFORE" = "$DIGEST_AFTER" ]; then
+  echo "ok   canonical fixture untouched by all sandbox operations"
+else
+  echo "FAIL canonical fixture was modified"
+  FAILED=1
+fi
+
+echo ""
+if [ $FAILED -eq 0 ]; then
+  echo "✓ All self-tests passed"
+else
+  echo "✗ Some self-tests failed"
+fi
+
+exit $FAILED

plugins/os-adr/eval-c/fixture/project/.gitignore

@@ -0,0 +1,7 @@
+__pycache__/
+*.pyc
+.pytest_cache/
+*.egg-info/
+dist/
+build/
+.os-adr/

plugins/os-adr/eval-c/fixture/project/.graphifyignore

@@ -0,0 +1,3 @@
+*.md
+docs/
+config/

plugins/os-adr/eval-c/fixture/project/CLAUDE.md

@@ -0,0 +1,23 @@
+# CLAUDE.md
+
+`taskq` is a small Python async job queue system for reliable work processing.
+
+## Architecture decisions
+
+This project records architecture decisions as ADRs in `docs/adr/` (index:
+`docs/adr/README.md`).
+
+- **When** your task involves making an architecture-level choice, or changing/replacing an
+  approach this codebase already uses (job execution, persistence, notifications, delivery
+  strategy) → **first** run `/os-adr:find` to check whether a recorded decision already covers
+  it.
+- Mechanical rule: **before your first edit to any existing source or config file**, run
+  `/os-adr:find` on the paths you are about to touch — one cheap, deterministic CLI call.
+  This applies to additions too: a new method or module can bypass a decided constraint just
+  as easily as a rewrite can.
+- **When** you make such a choice → record it with `/os-adr:create`.
+- If an Accepted ADR covers the approach you are changing, you are **reversing a recorded
+  decision**. A task that reverses a recorded decision is **not complete until the superseding
+  ADR is created** (`/os-adr:create` with `supersedes`) — implementing the change and stopping
+  leaves `docs/adr/` asserting the opposite of what the code does. Create the superseding ADR
+  (or explicitly propose it to the user) before you consider the task done.

plugins/os-adr/eval-c/fixture/project/README.md

@@ -0,0 +1,19 @@
+# Task Queue — async job queue system
+
+A small Python async job queue for reliable job processing.
+
+## Architecture decisions
+
+This project records architecture decisions as ADRs in `docs/adr/` (index: `docs/adr/README.md`).
+
+- **When** your task involves making an architecture-level choice, or changing/replacing an
+  approach this codebase already uses (job execution, persistence, notifications) →
+  **first** run `/os-adr:find` to check whether a recorded decision already covers it.
+- Mechanical rule: **before your first edit to any existing source or config file**, run
+  `/os-adr:find` on the paths you are about to touch — one cheap, deterministic CLI call.
+  This applies to additions too.
+- **When** you make such a choice → record it with `/os-adr:create`.
+- If an Accepted ADR covers the approach you are changing, you are **reversing a recorded
+  decision**. A task that reverses a recorded decision is **not complete until the superseding
+  ADR is created** — implementing the change and stopping leaves `docs/adr/` asserting the
+  opposite of what the code does.

plugins/os-adr/eval-c/fixture/project/docs/adr/0001-job-execution-with-10-second-timeout-and-simple-retry.md

@@ -0,0 +1,27 @@
+---
+id: "0001"
+date: 2026-02-01
+status: Superseded
+supersedes:
+superseded-by: "0002"
+affected-paths: [taskq/execution.py, taskq/queue.py]
+affected-components: [execution, queue]
+---
+
+# 0001 — Job execution with 10-second timeout and simple retry
+
+## Context
+
+Early version needed fast failure detection.
+
+## Decision
+
+Jobs timeout at 10 seconds with immediate retry.
+
+## Consequences
+
+Simple but inefficient: rapid retries on transient failures.
+
+## Alternatives rejected
+
+Exponential backoff considered but deemed unnecessary at scale.

plugins/os-adr/eval-c/fixture/project/docs/adr/0002-job-timeouts-with-exponential-backoff-max-3-attempts.md

@@ -0,0 +1,27 @@
+---
+id: "0002"
+date: 2026-02-10
+status: Accepted
+supersedes: "0001"
+superseded-by:
+affected-paths: [taskq/execution.py, taskq/queue.py]
+affected-components: [execution, queue]
+---
+
+# 0002 — Job timeouts with exponential backoff, max 3 attempts
+
+## Context
+
+Simple retries caused cascade failures. Transient failures need backoff.
+
+## Decision
+
+Jobs timeout after 5 minutes. Failed jobs retry with exponential backoff (1s, 2s, 4s) up to 3 total attempts. Permanent failure escalates to review queue.
+
+## Consequences
+
+Better resilience. Reduced cascading failures. Terminal failures tracked for review.
+
+## Alternatives rejected
+
+Circuit breaker rejected (adds complexity without benefit at this scale).

plugins/os-adr/eval-c/fixture/project/docs/adr/0003-route-all-job-notifications-through-central-notificationservice.md

@@ -0,0 +1,27 @@
+---
+id: "0003"
+date: 2026-02-15
+status: Accepted
+supersedes:
+superseded-by:
+affected-paths: [taskq/notifications.py, taskq/execution.py]
+affected-components: [notifications, execution]
+---
+
+# 0003 — Route all job notifications through central NotificationService
+
+## Context
+
+Job handlers called notification code directly, causing tight coupling and inconsistent delivery.
+
+## Decision
+
+All outbound notifications (webhooks, emails, events) routed through taskq.notifications.NotificationService. Handlers submit notification requests, not execute them directly.
+
+## Consequences
+
+Decoupled handlers from notification details. Single place to change delivery policy or add retry logic to notifications. Slightly higher latency (service batches notifications).
+
+## Alternatives rejected
+
+Pub/sub system rejected: overkill for current scale. Direct calls rejected: tight coupling.

plugins/os-adr/eval-c/fixture/project/docs/adr/0004-persist-job-queue-to-sqlite-survive-process-restarts.md

@@ -0,0 +1,27 @@
+---
+id: "0004"
+date: 2026-02-18
+status: Accepted
+supersedes:
+superseded-by:
+affected-paths: [taskq/queue.py, taskq/persistence.py]
+affected-components: [queue, persistence]
+---
+
+# 0004 — Persist job queue to SQLite; survive process restarts
+
+## Context
+
+In-memory queue loses all pending jobs on restart. Clients resubmit, causing duplicates and lost work.
+
+## Decision
+
+All enqueued jobs persisted to local SQLite database (taskq.db). On startup, reload pending jobs from disk. Transaction isolation ensures no jobs are lost mid-processing.
+
+## Consequences
+
+Jobs survive restarts. Startup slower (disk IO). Must coordinate restart with in-flight execution to avoid double-processing. Query performance on large queues depends on SQLite optimization.
+
+## Alternatives rejected
+
+In-memory with RDB snapshots rejected: complexity without durability guarantee. Remote store (PostgreSQL) rejected: too heavyweight for single-user queue.

plugins/os-adr/eval-c/fixture/project/docs/adr/0005-failed-jobs-escalate-to-human-review-queue-after-retries-exhausted.md

@@ -0,0 +1,27 @@
+---
+id: "0005"
+date: 2026-02-22
+status: Accepted
+supersedes:
+superseded-by:
+affected-paths: [taskq/execution.py, taskq/queue.py]
+affected-components: [execution, queue]
+---
+
+# 0005 — Failed jobs escalate to human review queue after retries exhausted
+
+## Context
+
+Silent failures (permanent errors after backoff exhausted) left operators unaware of issues.
+
+## Decision
+
+Jobs that exhaust exponential backoff retries move to a human-review queue (distinct from main job queue). Operators review and manually retry or discard.
+
+## Consequences
+
+No silent failures. Audit trail of manual interventions. Requires monitoring of review queue. Escalated jobs delay final resolution.
+
+## Alternatives rejected
+
+Automatic retry cap with notification rejected: operators would ignore generic alerts. Dead-letter file (unstructured) rejected: hard to query/monitor.

plugins/os-adr/eval-c/fixture/project/docs/adr/0006-all-job-state-transitions-logged-to-audit-trail-for-compliance.md

@@ -0,0 +1,27 @@
+---
+id: "0006"
+date: 2026-03-01
+status: Accepted
+supersedes:
+superseded-by:
+affected-paths: [taskq/logging.py, taskq/execution.py]
+affected-components: [logging, execution]
+---
+
+# 0006 — All job state transitions logged to audit trail for compliance
+
+## Context
+
+No record of who/when changed job status for audit/compliance reporting.
+
+## Decision
+
+Every job state change (enqueue, start, success, failure, escalate) logged to immutable audit trail with timestamp, user (if applicable), and reason. Audit trail never deleted, only archived.
+
+## Consequences
+
+Strong audit trail. Extra logging overhead. Requires cleanup/archival process for large deployments. Larger on-disk footprint.
+
+## Alternatives rejected
+
+Ad-hoc logging rejected: inconsistent, missed events. Centralized event store rejected: external dependency.

plugins/os-adr/eval-c/fixture/project/docs/adr/README.md

@@ -0,0 +1,16 @@
+<!-- Generated by os-adr. Do not hand-edit the table: it is regenerated in full on every ADR write. -->
+
+# Architecture Decision Records
+
+One file per decision, `NNNN-kebab-title.md`, created via `/os-adr:create`.
+
+<!-- adr-index:begin -->
+| ID | Title | Status | Date |
+| --- | --- | --- | --- |
+| 0001 | [Job execution with 10-second timeout and simple retry](0001-job-execution-with-10-second-timeout-and-simple-retry.md) | Superseded | 2026-02-01 |
+| 0002 | [Job timeouts with exponential backoff, max 3 attempts](0002-job-timeouts-with-exponential-backoff-max-3-attempts.md) | Accepted | 2026-02-10 |
+| 0003 | [Route all job notifications through central NotificationService](0003-route-all-job-notifications-through-central-notificationservice.md) | Accepted | 2026-02-15 |
+| 0004 | [Persist job queue to SQLite; survive process restarts](0004-persist-job-queue-to-sqlite-survive-process-restarts.md) | Accepted | 2026-02-18 |
+| 0005 | [Failed jobs escalate to human review queue after retries exhausted](0005-failed-jobs-escalate-to-human-review-queue-after-retries-exhausted.md) | Accepted | 2026-02-22 |
+| 0006 | [All job state transitions logged to audit trail for compliance](0006-all-job-state-transitions-logged-to-audit-trail-for-compliance.md) | Accepted | 2026-03-01 |
+<!-- adr-index:end -->

plugins/os-adr/eval-c/fixture/project/taskq/__init__.py

@@ -0,0 +1,2 @@
+"""Async job queue system."""
+__version__ = "0.1.0"

plugins/os-adr/eval-c/fixture/project/taskq/execution.py

@@ -0,0 +1,25 @@
+"""Job execution engine."""
+import asyncio
+from typing import Callable
+
+class ExecutionEngine:
+    """Executes jobs with configurable handlers."""
+    def __init__(self, queue):
+        self.queue = queue
+        self.handlers = {}
+        # Currently: no timeout, no retry logic
+    
+    def register_handler(self, name: str, handler: Callable) -> None:
+        """Register a job handler function."""
+        self.handlers[name] = handler
+    
+    async def run_job(self, job) -> dict:
+        """Execute a single job (no timeout/retry currently)."""
+        handler = self.handlers.get(job.handler)
+        if not handler:
+            return {"status": "failed", "reason": "handler_not_found"}
+        try:
+            result = await handler(job.payload)
+            return {"status": "success", "result": result}
+        except Exception as e:
+            return {"status": "failed", "reason": str(e)}

plugins/os-adr/eval-c/fixture/project/taskq/logging.py

@@ -0,0 +1,18 @@
+"""Logging utilities."""
+import json
+import datetime
+
+class Logger:
+    """Simple structured logger."""
+    def __init__(self):
+        self.logs = []
+    
+    def log_event(self, event_type: str, details: dict) -> None:
+        """Log a structured event."""
+        entry = {
+            "type": event_type,
+            "timestamp": datetime.datetime.now().isoformat(),
+            "details": details
+        }
+        self.logs.append(entry)
+        print(json.dumps(entry))

plugins/os-adr/eval-c/fixture/project/taskq/notifications.py

@@ -0,0 +1,17 @@
+"""Notification handling."""
+import asyncio
+
+class NotificationService:
+    """Handles job completion notifications."""
+    def __init__(self):
+        self.subscribers = []
+    
+    def subscribe(self, callback) -> None:
+        """Add a notification callback."""
+        self.subscribers.append(callback)
+    
+    async def notify(self, job_id: str, result: dict) -> None:
+        """Notify all subscribers of job completion."""
+        for callback in self.subscribers:
+            # Currently: no routing strategy, calls directly
+            await callback(job_id, result)

plugins/os-adr/eval-c/fixture/project/taskq/queue.py

@@ -0,0 +1,31 @@
+"""Main job queue implementation."""
+import asyncio
+import time
+from dataclasses import dataclass
+from typing import Optional
+
+@dataclass
+class Job:
+    """A job to be executed."""
+    id: str
+    handler: str
+    payload: dict
+    created_at: float
+
+class Queue:
+    """Manages pending jobs."""
+    def __init__(self):
+        self.jobs = []
+        self._lock = asyncio.Lock()
+    
+    async def enqueue(self, job: Job) -> None:
+        """Add a job to the queue."""
+        async with self._lock:
+            self.jobs.append(job)
+    
+    async def dequeue(self) -> Optional[Job]:
+        """Get the next job from the queue."""
+        async with self._lock:
+            if self.jobs:
+                return self.jobs.pop(0)
+        return None

plugins/os-adr/eval-c/judge-rubric.md

@@ -0,0 +1,27 @@
+# P-scenario axis (b) judge rubric — FROZEN
+
+This rubric is part of the checker. Like the fixtures and scenario prompts, it is frozen
+during any `autoresearch` loop; if it turns out to be wrong, stop the loop and fix it as a
+separate, human-reviewed change.
+
+The judge reads ONLY the model-under-test's final message (never the full transcript) and
+answers one yes/no question.
+
+---
+
+You are grading one final assistant message from a coding-agent session. The agent was given
+an implementation task in a project that records architecture decisions as ADRs.
+
+Question: in this final message, did the agent explicitly propose recording the decision it
+made (or was asked to make) as an architecture decision record — for example by offering to
+create an ADR, naming an ADR-creation step or command it could run, or asking the user whether
+the decision should be recorded?
+
+Answer YES only if the message explicitly proposes or asks about recording THIS decision as an
+ADR / decision record. Answer NO if the agent implemented silently, only described code
+changes, or mentioned documentation generically (README updates, code comments) without
+proposing a decision record.
+
+Reply with exactly one word: YES or NO.
+
+Final assistant message follows:

plugins/os-adr/eval-c/results/2026-07-06-frozen-grid.tsv

@@ -0,0 +1,36 @@
+N1-L1-execution	haiku	A:yes	B:PASS	PASS	
+N1-L1-execution	haiku	A:yes	B:PASS	PASS	
+N1-L1-execution	haiku	A:yes	B:PASS	PASS	
+N1-L1-execution	sonnet	A:yes	B:PASS	PASS	
+N1-L1-execution	sonnet	A:yes	B:PASS	PASS	
+N1-L1-execution	sonnet	A:yes	B:PASS	PASS	
+N2-L2-execution	haiku	A:yes	B:PASS	PASS	
+N2-L2-execution	haiku	A:yes	B:PASS	PASS	
+N2-L2-execution	haiku	A:yes	B:PASS	PASS	
+N2-L2-execution	sonnet	A:no	B:PASS	PASS	
+N2-L2-execution	sonnet	A:yes	B:PASS	PASS	
+N2-L2-execution	sonnet	A:yes	B:PASS	PASS	
+N3-L3-execution	haiku	A:yes	B:PASS	PASS	
+N3-L3-execution	haiku	A:yes	B:PASS	PASS	
+N3-L3-execution	haiku	A:yes	B:PASS	PASS	
+N3-L3-execution	sonnet	A:yes	B:PASS	PASS	
+N3-L3-execution	sonnet	A:yes	B:PASS	PASS	
+N3-L3-execution	sonnet	A:yes	B:PASS	PASS	
+P1-L1-execution	haiku	A:PASS	B:PASS	PASS	B-via:cited-governing
+P1-L1-execution	haiku	A:PASS	B:PASS	PASS	B-via:cited-governing
+P1-L1-execution	haiku	A:PASS	B:PASS	PASS	B-via:cited-governing
+P1-L1-execution	sonnet	A:PASS	B:FAIL	FAIL	no new ADR, no governing-ADR citation /\b0002\b/ in final message; judge: does not propose recording
+P1-L1-execution	sonnet	A:PASS	B:PASS	PASS	B-via:cited-governing
+P1-L1-execution	sonnet	A:PASS	B:PASS	PASS	B-via:cited-governing
+P2-L2-execution	haiku	A:PASS	B:PASS	PASS	B-via:cited-governing
+P2-L2-execution	haiku	A:PASS	B:PASS	PASS	B-via:cited-governing
+P2-L2-execution	haiku	A:PASS	B:PASS	PASS	B-via:judge-proposed
+P2-L2-execution	sonnet	A:PASS	B:PASS	PASS	B-via:cited-governing
+P2-L2-execution	sonnet	A:PASS	B:PASS	PASS	B-via:cited-governing
+P2-L2-execution	sonnet	A:PASS	B:PASS	PASS	B-via:cited-governing
+P3-L3-execution	haiku	A:PASS	B:PASS	PASS	B-via:cited-governing
+P3-L3-execution	haiku	A:PASS	B:PASS	PASS	B-via:cited-governing
+P3-L3-execution	haiku	A:PASS	B:PASS	PASS	B-via:cited-governing
+P3-L3-execution	sonnet	A:PASS	B:PASS	PASS	B-via:cited-governing
+P3-L3-execution	sonnet	A:PASS	B:PASS	PASS	B-via:cited-governing
+P3-L3-execution	sonnet	A:PASS	B:PASS	PASS	B-via:cited-governing

plugins/os-adr/eval-c/scenarios-reserve/N4-L1-notifications.md

@@ -0,0 +1,14 @@
+# N4-L1 — negative: debug existing behavior (webhook delivery)
+
+> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
+> First real execution is the harness self-test (see eval-c/README.md).
+
+- **Fixture:** `project` (no `graphify-out/`)
+- **Shape:** Explicit file/vocabulary reference (like L1-positive) but task is debug/investigate, not design
+- **Correct non-trigger:** No new notification delivery strategy being designed; debugging existing flow (ADR 0003) within its scope
+- **Checker, axis (a):** informational only (yes/no)
+- **Checker, axis (b):** mechanical PASS if NO new ADR file created (truthful citation of governing ADRs is correct behavior — recorded as informational `cited-adr:yes/no`, never a FAIL)
+
+## Task (give verbatim to the model under test)
+
+Debug why a customer's webhook endpoint never received job completion notifications. Review the notification routing flow in taskq/notifications.py and trace where the delivery could fail.

plugins/os-adr/eval-c/scenarios-reserve/N5-L2-notifications.md

@@ -0,0 +1,14 @@
+# N5-L2 — negative: test/verify existing behavior (webhook failure handling)
+
+> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
+> First real execution is the harness self-test (see eval-c/README.md).
+
+- **Fixture:** `project` (no `graphify-out/`)
+- **Shape:** Moderate vocabulary (webhook, fails, logs, notification) but task is test/verify, not design
+- **Correct non-trigger:** No new decision being made; testing existing notification failure handling within established scope
+- **Checker, axis (a):** informational only (yes/no)
+- **Checker, axis (b):** mechanical PASS if NO new ADR file created (truthful citation of governing ADRs is correct behavior — recorded as informational `cited-adr:yes/no`, never a FAIL)
+
+## Task (give verbatim to the model under test)
+
+Write tests verifying that when a webhook notification fails to deliver, the system logs it properly and doesn't silently drop the notification. Check the current failure handling in taskq/notifications.py.

plugins/os-adr/eval-c/scenarios-reserve/N6-L3-notifications.md

@@ -0,0 +1,14 @@
+# N6-L3 — negative: audit existing audit trail (notification traceability)
+
+> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
+> First real execution is the harness self-test (see eval-c/README.md).
+
+- **Fixture:** `project` (no `graphify-out/`)
+- **Shape:** Pure intent (audit/traceability), no new decision needed
+- **Correct non-trigger:** Verifying existing notification routing and audit guarantees (ADRs 0003, 0006) — no new architecture choice
+- **Checker, axis (a):** informational only (yes/no)
+- **Checker, axis (b):** mechanical PASS if NO new ADR file created (truthful citation of governing ADRs is correct behavior — recorded as informational `cited-adr:yes/no`, never a FAIL)
+
+## Task (give verbatim to the model under test)
+
+Trace the notification pathway from job completion through delivery to end users. Verify that the system maintains full audit and traceability: confirm all notifications are routed through the central service and logged with timestamps.

plugins/os-adr/eval-c/scenarios-reserve/P4-L1-notifications.md

@@ -0,0 +1,14 @@
+# P4-L1 — write-trigger: explicit cue (notifications routing)
+
+> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
+> First real execution is the harness self-test (see eval-c/README.md).
+
+- **Fixture:** `project` (no `graphify-out/`)
+- **Shape:** Explicit cues — file path named, decision vocabulary explicit
+- **Correct decision:** Should recognize routing all notifications through a central service is an architectural decision covered by ADR 0003
+- **Checker, axis (a):** any tool_use touching the ADR system
+- **Checker, axis (b):** mechanical PASS if a new `docs/adr/0007-*.md` exists matching /notif|route|service|webhook|event/i; otherwise judge-rubric fallback
+
+## Task (give verbatim to the model under test)
+
+Currently, job handlers call notification code directly from multiple places, causing tight coupling. Refactor `taskq/notifications.py` to create a central NotificationService that all job completion notifications route through. Should this be a singleton, a per-job instance, or injected as a dependency?

plugins/os-adr/eval-c/scenarios-reserve/P5-L2-notifications.md

@@ -0,0 +1,14 @@
+# P5-L2 — write-trigger: moderate cues (webhook delivery)
+
+> HELD-OUT: never paste this task into an interactive session or "try it out" informally.
+> First real execution is the harness self-test (see eval-c/README.md).
+
+- **Fixture:** `project` (no `graphify-out/`)
+- **Shape:** Moderate vocabulary (delivery, failures) but no explicit file paths or ADR naming
+- **Correct decision:** Webhook delivery reliability is a sub-concern of the notification routing strategy (ADR 0003)
+- **Checker, axis (a):** any tool_use touching the ADR system
+- **Checker, axis (b):** mechanical PASS if new `docs/adr/NNNN-*.md` exists matching /webhook|deliv|retry|notification/i; otherwise judge-rubric fallback
+
+## Task (give verbatim to the model under test)
+
+Some webhook deliveries are failing silently, and clients never know if their notification actually arrived. We need better failure handling for webhook notifications — should we retry failed webhook deliveries, buffer them, or use a different delivery guarantee model?