diff --git a/docs/orchestration-audit/2026-07-08-fable-econ-miniaudit.md b/docs/orchestration-audit/2026-07-08-fable-econ-miniaudit.md new file mode 100644 index 0000000..203e479 --- /dev/null +++ b/docs/orchestration-audit/2026-07-08-fable-econ-miniaudit.md @@ -0,0 +1,175 @@ +# Fable-5 main-loop orchestration economics — mini-audit (2026-07-08) + +**Model**: this audit was performed by `claude-sonnet-5`. All audited transcripts carry +`"model":"claude-fable-5"` on non-sidechain assistant messages (i.e. Fable 5 is the main-loop +model in every session below). + +## Method + +1. Scanned all `~/.claude/projects/*/*.jsonl` files modified in the last 10 days + (`scan.rb`), filtered to sessions with ≥20 non-sidechain assistant turns and at least one + `claude-fable-5` main-loop message, excluded the live session + (`ca17e6d2-6132-4681-8a02-8a7523cb5660`). 21 candidates found, almost all in `cc-os`; picked + 6 for project diversity: 3 `cc-os`, 2 `servers`, 1 `servers/desktop`, all ≥20 turns (24–255). +2. Ran the existing fact-sheet extractor + (`plugins/os-orchestration/audit/bin/extract`) on each for spawn count/models, + pre-/post-spawn tool segmentation, and its heuristic same-tool-run flags. +3. Wrote two throwaway Ruby scripts in scratchpad (`tokens.rb`, `bashcmds.rb`, + `firstprompt.rb`) to sum `usage.output_tokens` per session (main-loop vs sidechain), track + max/final context size, and find the longest unbroken run of main-loop tool calls with no + `Agent` spawn in between, then classified each run by reading the actual tool inputs + (Bash commands / file targets) in that range. + +**Environment gotcha (noted for future audits):** in zsh, a variable named `path` is linked to +`$PATH`; `path=$(find ...)` silently corrupted `$PATH` mid-script and caused unrelated `ruby: not +found` (exit 127) failures. Renamed to `tpath`. Not a data issue, just a shell trap. + +**Hard measurement limitation:** none of the 6 transcripts contain any `"isSidechain":true` +lines, even in the two sessions that used the `Agent` tool (9f45afcc, 5 spawns; df12b180, 1 +spawn). Subagent execution in this environment/version is logged only as a `tool_use`/ +`toolUseResult` pair (with `agentId`/`resolvedModel` metadata) plus an eventual +`` result — no subagent transcript is persisted into the parent `.jsonl`. So +**subagent output-token spend is not recoverable from these files at all.** The only proxy +available is the char length of the `tool_result`/notification payload returned to the main +loop, converted at ~4 chars/token. All "% main-loop" figures below use that proxy where spawns +occurred, and are explicitly not real subagent token counts (real subagent spend is very likely +much higher than the proxy, since it only reflects what was returned, not what the subagent +itself burned getting there). + +> **Addendum (2026-07-08, WS4 Step 2):** the limitation above is narrower than stated — +> subagent spend is absent from the *parent* transcript, but each subagent's full transcript +> (with per-message `usage` blocks) persists separately at +> `~/.claude/projects///subagents/agent-.jsonl` (verified for +> both interactive and headless sessions). Real main-loop-share measurement is therefore +> possible; the eval harness A-econ axis uses it. The proxy figures below were NOT recomputed. +> See vault note [[claude-code-subagent-transcripts-and-token-accounting]]. + +## Per-session detail + +### 1. `9f45afcc` — cc-os — cache-refresh tooling + WS2 autoresearch prep +- 255 turns, 25 human prompts, 850 jsonl lines, 2026-07-04 16:17–18:52 +- 5 Agent spawns, all `general-purpose`, `model` param `sonnet`×4/`haiku`×1, **resolved model + `claude-haiku-4-5` in all 5 cases** (param mismatch: 4 of 5 requested sonnet, all resolved + haiku) +- main-loop output tokens: **237,679** (255 msgs); proxy subagent output: 13,575 chars via + result/notification payloads (≈3.4K tokens) → **main-loop ≈98.6% of proxy-measured output** +- pre-spawn segment: 19 calls / 23,658 bytes read (Bash 14, Read 4, Edit 1) +- longest direct run: **48 calls** (after spawn 3: ToolSearch 2, SendMessage 1, Bash 22, Edit 12, + Skill 2, Read 5, Write 3, TaskStop 1) — mixed: driving/monitoring the background eval-grid + agent (SendMessage/TaskStop/ToolSearch) interleaved with direct mechanical plugin-cache-refresh + edits. Not a clean single-purpose run. + +### 2. `eba26343` — cc-os — WS2 os-vault wording loop (`/autoresearch`) +- 223 turns, 2 human prompts (127 total `type:user` lines incl. tool results/notifications), 583 + jsonl lines, 2026-07-07 16:48–18:01 +- **0 Agent spawns** +- main-loop output tokens: **216,019** (223 msgs) — 100% of measured output (no spawns to split + against) +- whole-session tool profile: 119 calls / 119,587 bytes (Read 17, Bash 64, Skill 1, ToolSearch 1, + TaskCreate 5, TaskUpdate 9, Edit 20, Write 2) +- longest direct run: **119 calls = the entire session** (no spawn boundary at all). Sampled the + 64 Bash commands directly: this is a scripted headless-runner loop — `bin/run P1/P2/P3...` / + `bin/check` invocations against `eval/scenarios/`, `sleep N; wc -l results.tsv` + polling of background task output, `awk`/`sort` aggregation of `results.tsv`, and git + commit/push at each iteration boundary. **Classify as justified scripted-bulk-op / polling + work**, not a delegation miss — matches the ORCHESTRATION.md scripted-bulk-edit carve-out; a + subagent adds no value polling a script's own stdout. + +### 3. `5f53e0c0` — cc-os — `/opsx:apply add-os-adr-plugin` +- 222 turns, 4 human prompts, 506 jsonl lines, 2026-07-03 17:15–17:43 +- **0 Agent spawns** +- main-loop output tokens: **314,899** (222 msgs, highest of the 6) — 100% of measured output +- whole-session tool profile: 128 calls / 129,924 bytes (Bash 39, Read 16, Write 35, Edit 38) +- longest direct run: **128 calls = the entire session**. Contains three flagged sub-runs from + the extractor: Read×10 (design docs, lines 27–52 — reasonable orientation), **Write×13 (lines + 96–137, 13 distinct new files: `lib/adr.rb`, `lib/adr/record.rb`, `repository.rb`, + `index.rb`, `template.rb`, `bin/adr-new`, …)**, and Edit×10 (lines 326–353, fixing 5 files + post-review). **Genuine miss candidate**: the Write×13 run is scaffolding independent Ruby + class files against an *already-fixed* `design.md` (read in the first 10 calls) — a + parallelizable implementation task that ran entirely in the main loop and is a large share of + this session's 315K output tokens. + +### 4. `3fc7bb8c` — servers — backup verification + vault write +- 90 turns, 5 human prompts, 236 jsonl lines, 2026-07-04 17:20–18:04 +- **0 Agent spawns** +- main-loop output tokens: **63,607** +- whole-session tool profile: 44 calls / 32,250 bytes (Read 3, Bash 27, Write 4, ToolSearch 2, + ExitPlanMode 1, Edit 6, Skill 1) +- longest direct run: **44 calls = whole session** — interactive backup-system investigation + ("Are we confident backups will work as expected now?") ending in a vault write. Judgment- + heavy, sequential-dependent (each Bash result determines the next check). **Justified direct + work**, no clean miss shape. + +### 5. `7bc0dda4` — servers — Proxmox HAOS install troubleshooting +- 82 turns, 15 human prompts (highest interaction density of the 6), 249 jsonl lines, + 2026-07-03 19:49–20:42 +- **0 Agent spawns** +- main-loop output tokens: **39,513** +- whole-session tool profile: 39 calls / 6,124 bytes (Bash 27, Edit 8, Read 2, Skill 1, Write 1) +- longest direct run: **39 calls = whole session** — live interactive install/debug loop (disk + space checks, image download, phase-2 error triage against user-supplied screenshots). + **Justified direct work**: 15 human turns in 82 total means the human is in the loop almost + every other turn; delegation overhead would exceed any savings. + +### 6. `df12b180` — servers/desktop — Remote Control OAuth scope troubleshooting +- 24 turns, 6 human prompts, 90 jsonl lines, 2026-07-06 23:13–23:35 +- 1 Agent spawn (`claude-code-guide`, `model` param `sonnet`, resolved `claude-sonnet-5`, + foreground, prompt 1,008 chars → result 5,689 chars) +- main-loop output tokens: **24,051**; proxy subagent output ≈1,422 tokens (5,689 chars/4) → + **main-loop ≈94.4% of proxy-measured output** +- pre-spawn: 4 Bash calls / 3,154 bytes (credential/scope investigation) +- post-spawn: 3 calls (Read 1, Write 2 — writing up the answer) +- longest direct run: 4 calls (the pre-spawn investigation). **Good delegation example**: a + bounded doc-lookup ("Remote Control usage") was correctly routed to the docs-specialist + subagent instead of the main loop reading source/docs itself. + +## Summary table + +| session | project | turns | spawns | main-loop out tok | % main-loop (proxy, see caveat) | pre-spawn calls/bytes | longest direct run | classification | +|---|---|---|---|---|---|---|---|---| +| 9f45afcc | cc-os | 255 | 5 | 237,679 | ~98.6% | 19 / 23,658 | 48 | mixed: agent-monitoring + mechanical cache-refresh edits | +| eba26343 | cc-os | 223 | 0 | 216,019 | 100% | 119 / 119,587 (whole session) | 119 (whole session) | scripted headless-eval polling loop — justified | +| 5f53e0c0 | cc-os | 222 | 0 | 314,899 | 100% | 128 / 129,924 (whole session) | 128 (whole session) | OpenSpec apply — Write×13 independent files = **miss candidate** | +| 3fc7bb8c | servers | 90 | 0 | 63,607 | 100% | 44 / 32,250 (whole session) | 44 (whole session) | interactive investigation — justified | +| 7bc0dda4 | servers | 82 | 0 | 39,513 | 100% | 39 / 6,124 (whole session) | 39 (whole session) | interactive troubleshooting (15 human turns) — justified | +| df12b180 | servers/desktop | 24 | 1 | 24,051 | ~94.4% | 4 / 3,154 | 4 | pre-spawn investigation, correct delegation of doc lookup | + +## Caveats + +- 4 of the 6 sessions are from `cc-os` (matches the actual population: 19 of 21 candidate + sessions in the last 10 days were `cc-os`) — the sample is not representative of all-client + usage, just of what exists in the last 10 days of transcripts. +- "% main-loop" is a **proxy** built only from what subagents returned to the main loop + (tool_result/notification char count ÷4), not measured subagent token spend — real subagent + cost is invisible in every transcript examined (0/6 have `isSidechain:true` lines). Any + economics conclusion about spawn cost-effectiveness needs a different data source (e.g. Task + output files under `tasks/`, if retained) — not attempted here. +- "Longest direct run" for 4 of 6 sessions equals the *entire session* because zero spawns + occurred; this collapses "pre-spawn segment" and "longest run" into the same number for those + rows — noted in the table rather than hidden. +- Miss/justified classification is a single-auditor read of tool-call sequences, not a rubric + score; treat as a directional signal for the eval plan, not a verdict. + +## Miss-shape summary + +- **The only clean miss candidate found is `5f53e0c0`'s Write×13 run**: scaffolding independent + Ruby class files against an already-fixed design doc, done entirely in the (most expensive) + main loop across a 222-turn, 314,899-output-token session with zero spawns. +- **Scripted/polling loops are not misses even when very long** (`eba26343`, 119 consecutive + direct calls): driving `bin/run`/`sleep`/`wc -l results.tsv` from the main loop is the correct + shape per the existing scripted-bulk-op carve-out — a subagent adds no value polling a script's + own output. +- **Interactive/judgment-heavy sessions correctly stayed direct** (`3fc7bb8c`, `7bc0dda4`): both + have no spawns and no flagged miss runs; `7bc0dda4`'s 15 human prompts in 82 turns make + delegation overhead a net loss. +- **Subagent economics are structurally unmeasurable from transcripts as configured**: 0 of 6 + sessions logged sidechain entries even where Agent spawned; only a lossy char-count proxy + exists via task-notification payloads. This is itself an eval-plan-relevant finding — any eval + claiming to measure delegated-vs-direct token spend needs an instrumentation change (capture + Task output, or read `tasks/*.output` files) before it can trust a token-economics number. +- **Where spawns did happen, the main loop still did substantial direct work afterward** + (`9f45afcc`: 43–48-call runs after spawns 3 and 5) rather than delegating a second time — + worth checking in future audits whether that's justified (synthesis/judgment) or another miss + shape. +- **No over-delegation observed anywhere** — zero ≤2-call trivial ops routed to a subagent in any + of the 6 sessions. diff --git a/docs/plans/ws4-orchestration-economics.md b/docs/plans/ws4-orchestration-economics.md new file mode 100644 index 0000000..2e83794 --- /dev/null +++ b/docs/plans/ws4-orchestration-economics.md @@ -0,0 +1,166 @@ +# WS4 — Orchestration economics: ORCHESTRATION.md v3 + eval extension + wording loop + +_Created: 2026-07-08. Status: PLANNED (mini-audit complete; steps 1–5 not started)._ +_Prereq reading before executing: `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md`, +vault notes [[orchestration-prompting-claude-5-era]], [[eval-methodology-ladder]], +[[os-orchestration-eval-baseline-grid-results]]._ + +## Problem + +The Fable-5 main loop does too much multi-call work itself. ORCHESTRATION.md was calibrated +against over-delegation (Opus 4.6 era) under sonnet main loops where direct work was cheap; +its negative-first threshold ("Delegate only when…", "≤2-tool-call ops direct") is read +literally by Fable and licenses unbounded direct work whenever the task isn't parallelizable/ +many-files/isolated-context. Goals (user, 2026-07-08): max quality at reasonable cost; +route work to cheaper tiers whenever reasonable; group delegated tasks so agents finish in +few rounds (fewer spawns → less per-spawn system-prompt tax). + +Evidence base: Anthropic model-page guidance summarized in vault note +[[orchestration-prompting-claude-5-era]] (fetched 2026-07-08); WS1 audit +(`docs/orchestration-audit/2026-07-06-findings.md`); Fable economics mini-audit +(`docs/orchestration-audit/2026-07-08-fable-econ-miniaudit.md`) — the H3 baseline. + +### Mini-audit headline (2026-07-08, 6 Fable sessions, ran before drafting wording) + +- **The dominant miss shape is implementation fan-out, not investigation:** the one clean + miss is an OpenSpec-apply session (`5f53e0c0`) that wrote 13 independent Ruby files + entirely in the main loop — 314,899 main-loop output tokens, 0 spawns. E5P should mirror + this shape (write-N-independent-files implementation), not a log-review shape. +- Long direct runs were mostly JUSTIFIED (scripted eval-polling loop of 119 calls; + interactive troubleshooting with 15 human turns) — confirming the non-goal: cost-keyed + thresholds, not "always delegate". Zero over-delegation anywhere. +- Secondary watch item: after a session's spawns complete, the main loop resumes long + direct runs (43–48 calls, `9f45afcc`) instead of delegating again — mid-session threshold + decay. +- **Instrumentation gap (affects A-econ):** main transcripts contain NO `isSidechain:true` + entries even in sessions with spawns — subagent token spend is unrecoverable from the + transcript alone. The eval checker must source subagent spend from harness-side artifacts + (the eval runner's own task/agent output files in the sandbox), or A-econ degrades to a + main-loop-absolute-tokens metric (still usable: baseline sessions run 24k–315k main-loop + output tokens). Resolve this during Step 2 design; do not silently ship the char-count + proxy the mini-audit used. + +## Step 1 — ORCHESTRATION.md v3 (wording surface for the loop) + +Five changes; keep explicit-`model:`, self-report, and don't-re-cover-ground rules unchanged: + +1. **Cost-asymmetry + tier-conditional threshold.** New opening rule: main-loop tokens are + the most expensive in the session; the more capable the main-loop model, the lower the + delegation threshold. Any investigation or mechanical sequence beyond ~3 calls that does + not need main-loop judgment → delegate down-tier, even when sequential. (Models know their + own model ID — tier-conditional phrasing is implementable; E1 canary verified self-report.) +2. **Symmetric triggers.** Replace "Delegate only when…" with paired lists: + "delegate when …" / "work directly when …" (keep the scripted-bulk-edit carve-out and + ≤2-call rule on the direct side). Rationale: literal instruction-following on 5-era + models; the canonical best-practices snippet pairs both directions. +3. **Batching rule.** Plan the fan-out before the first spawn; group related subtasks + (~5–8 similar items) into one agent prompt with a return schema so the agent completes in + one round; follow-ups go to the live agent (SendMessage), not a fresh spawn. +4. **Async rule.** Delegate independent subtasks and keep working while they run; intervene + only if a subagent goes off track (Fable-page recommended pattern). +5. **Effort dial (where exposed).** Workflow `agent()` calls: mechanical stages + `effort: low`; hard verify/judge stages high/xhigh. + +Draft AFTER reading the mini-audit's miss shapes — wording should name the observed shapes, +not hypothetical ones (Eval B lesson: mechanical triggers phrased at the observed +precondition). + +## Step 2 — Eval harness extension (`plugins/os-orchestration/eval/`) + +All extensions are new frozen surfaces; existing run-set scenarios E1–E3 stay untouched. +Reserve stays frozen. Follow [[eval-methodology-ladder]]: paired positives/negatives, +run-set + reserve twins authored together, thresholds anchored independently of any grid +already run (pre-registered criterion-redesign rule from the 2026-07-06 baseline). + +### New metric axes (extractor + checker) + +- **A-econ (main-loop token share):** main-loop assistant output tokens ÷ total session + output tokens (main + sidechains). Extractor already parses transcripts; add the sum. +- **A-prespawn:** tool calls + tool_result bytes before first spawn (extractor segment + exists from E3 work). +- **A-batch:** spawns per task and rounds per agent (Agent tool_use count; SendMessage + reuse counts as same-agent round, not new spawn). +- Redesign E2P/E3P delegate-at-all axes per the pre-registered note: PASS may be + "delegated" OR "superior scripted direct strategy" — the checker must distinguish + script-driven bulk ops (few Bash calls, uniform op) from grind (per-file tool loops). + +### New scenario pair E5 (batching) + reserve twins + +- **E5P:** N (~10–12) similar independent items, shaped like the observed miss — + an implementation task producing N independent files in the relaystation fixture + (NOT log review). PASS = grouped delegation (≤3 spawns covering all items, grouped + prompts, one round each), FAIL = 1-agent-per-item or main-loop grind. +- **E5N (paired negative):** a task where batching would wrongly serialize genuinely + judgment-dependent items (each item's handling depends on the previous result) — PASS = + sequential handling (direct or single agent), FAIL = blind parallel fan-out. +- Author reserve twins at the same time, different surface domain; freeze both. + +### Fable column + +Baseline grid ran sonnet/haiku orchestrators only; the population of concern is Fable. +Add `--model fable` (headless `claude -p --model claude-fable-5`) at reduced reps +(1–2 per cell for the grid; 3 on target cells inside the loop). Cost of the eval itself is +real — use the reduced inner-loop grid discipline (target cells + one passing control). + +## Step 3 — Baseline grid (pre-wording) + +Run extended grid (E1–E3 + E5, econ axes) × {fable, sonnet} before touching wording. +Canary-cell the first live rep of every NEW scenario/axis and hand-verify TSV vs transcript +(count the canary). This grid is the tuning baseline; the 2026-07-08 mini-audit is the IRL +baseline for H3. + +## Step 4 — Autoresearch wording loop + +Loop discipline per the howto: wording-only moves (checker/fixtures/scenarios/axes frozen), +`bin/refresh-plugins` before every iteration's grid, ≥3 reps on target cells + 1 control +cell, accept only majority-of-reps improvements, verify from TSV not agent prose. + +### Hypotheses (pre-registered) + +- **H1 (threshold framing):** symmetric cost-framed triggers raise delegation on positive + scenarios (esp. E3P-shape sequential investigation) on fable/sonnet, while negatives stay + clean. Guard: baseline negatives were 18/18 — any negative regression rejects the + candidate regardless of positive gains. +- **H2 (batching rule):** the grouping rule reduces spawns-per-task and rounds-per-agent on + E5P without task-success regression; E5N stays PASS (no blind fan-out). +- **H3 (net economics — the actual goal):** main-loop token spend on positives drops + materially vs the Step-3 baseline with task success unchanged. Metric: main-loop share + if Step 2 solves the sidechain-instrumentation gap, else main-loop absolute output + tokens per scenario. Anchor the target threshold independently before the loop starts + (not post-hoc to whatever the loop achieves). H1/H2 are mechanisms; H3 is the verdict + axis. + + **H3 threshold (pre-registered 2026-07-08, before the Step-3 grid ran):** H3 PASSes iff, + on the positive economics cells (E5P and E3P, fable column), the median main-loop + assistant output tokens across reps under the candidate wording is ≤60% of the Step-3 + pre-wording baseline median for the same cell (≥40% reduction), with task-success axes + unchanged and all negatives clean. Pass/fail is keyed to main-loop **absolute** output + tokens (exists in both baselines regardless of the sidechain-instrumentation outcome); + main-loop *share* is reported additionally if Step 2 solves sidechain measurement. + Anchor rationale (independent of any grid): in the IRL miss exemplar `5f53e0c0`, the + Write×13 fan-out segment dominates a 314,899-token session — delegating the + implementation segment should remove roughly half of main-loop output; 40% is a + conservative floor of that estimate. + +### Judging + +Per-axis deterministic checker verdicts; per-scenario pass bars, not aggregate scores. +A wording candidate ships only if: all negatives hold, target positives improve on majority +of reps, control cell unchanged, and the full-grid confirm reproduces it. Final confirmation +on the frozen reserve (tuning against the run-set moves measurement to the reserve — +after this loop the run-set is contaminated for future measurement). + +## Step 5 — Rollout + IRL re-audit + +Ship winning wording (refresh caches), then re-run the Fable economics mini-audit on ~5 new +real sessions after ~2 weeks ([[eval-methodology-irl-feedback-loop]]). Compare main-loop +token share vs the 2026-07-08 baseline. Promote any new miss shapes into eval scenarios. + +## Non-goals / guards + +- No changes to delegation SAFETY rules (explicit model:, self-report, downgrade honesty). +- Don't over-rotate: zero over-delegation was a baseline strength (negatives 18/18) and + scripted-direct strategies were often genuinely superior — the point is cost-keyed + thresholds, not "always delegate". +- Held-out discipline: never run scenario Task blocks informally; reserve is never read + informally. diff --git a/plugins/os-orchestration/ORCHESTRATION.v3-draft.md b/plugins/os-orchestration/ORCHESTRATION.v3-draft.md new file mode 100644 index 0000000..b54a534 --- /dev/null +++ b/plugins/os-orchestration/ORCHESTRATION.v3-draft.md @@ -0,0 +1,50 @@ +## Session orchestration + +- Main-loop tokens are the most expensive tokens in the session: every direct tool + call and every byte read into this context bills at the main-loop model's rate, + while a haiku/sonnet subagent does the same mechanical work for a fraction of it. + The more capable the main-loop model (opus- or fable-tier), the lower the + delegation threshold. The question is not "is this task big enough to delegate?" + but "does this step need main-loop judgment?" — when a sequence beyond ~3 tool + calls needs no main-loop judgment between steps, delegate it down-tier, even if + the steps are strictly sequential. +- Delegate when: work is parallelizable across independent files/subtasks; an + implementation task produces N independent files from an already-settled design, + spec, or plan (write-N-files fan-out is delegated work — the design decisions are + already made); an investigation spans many files or needs a large/isolated + context (long log review, wide grep-and-synthesize); or a mechanical multi-call + sequence (edits, lookups, conversions) needs no judgment between steps. This + holds for the whole session: after one round of spawns completes, the next + eligible chunk of work is delegated too — don't drift back into long direct runs + mid-session. +- Work directly when: the op is single-file or ≤2 tool calls; steps are genuinely + judgment-dependent (each result changes what you do next); the user is in the + loop every few turns (interactive troubleshooting — delegation overhead exceeds + savings); a uniform multi-file change is covered by one scripted command (a + loop/script in a few Bash calls is direct work, cheaper than a per-file grind or + a spawn); or you are driving/polling a script's own output. +- Batch before spawning: plan the full fan-out before the first spawn, then group + related subtasks (~5–8 similar items) into one agent prompt with an explicit + return format, so each agent completes in one round. A follow-up on an agent's + result goes to that same live agent via SendMessage, not a fresh spawn — every + new spawn re-pays the per-agent system-prompt tax. +- Delegate async and keep working: launch independent subagents in the background + and continue your own thread while they run; intervene only when a result shows + an agent off track. +- Before every `Agent` call → set `model:` explicitly in that call. An omitted + `model` silently bills the subagent at the main-loop model. Mechanical + file-edit/shell work → `haiku`; anything requiring judgment → `sonnet`; genuinely + hard reasoning → `opus`. +- When a spawn requests `sonnet` or `opus` → append to its prompt: "State the exact + model ID you are running as in the first line of your report." (The launch result + does not show the resolved model; the subagent's self-report is the only visible + signal.) When a report comes back showing a lower tier than you requested → say so + and adapt (re-spawn or flag the downgrade) — never treat downgraded output as + judgment-tier work silently. +- Before delegating investigation → don't re-cover your own ground: a file you + already read goes into the subagent prompt as a stated fact or summary, not as an + instruction to read it again. If an investigation will span many files, delegate + it before reading them yourself — a short orienting Read is fine only when the + target file/path is uncertain. +- Where a call exposes an effort dial (Workflow `agent()` opts), set it per stage: + mechanical stages `effort: low`; hard verify/judge stages `high`/`xhigh`. diff --git a/plugins/os-orchestration/audit/bin/extract b/plugins/os-orchestration/audit/bin/extract index 25024f7..3aec5e0 100755 --- a/plugins/os-orchestration/audit/bin/extract +++ b/plugins/os-orchestration/audit/bin/extract @@ -82,12 +82,13 @@ module OrchAudit class AgentSpawn attr_reader :call, :index - attr_accessor :agent_id, :resolved_model, :notification_chars + attr_accessor :agent_id, :resolved_model, :notification_chars, :rounds def initialize(call, index) @call = call @index = index @notification_chars = 0 + @rounds = 1 end def to_h @@ -102,7 +103,8 @@ module OrchAudit run_in_background: call.input["run_in_background"], prompt_chars: (call.input["prompt"] || "").length, description: call.input["description"], - result_chars: [call.result_chars, notification_chars].max + result_chars: [call.result_chars, notification_chars].max, + rounds: rounds } end end @@ -230,10 +232,11 @@ module OrchAudit calls, agent_spawns = collect_calls(main) attach_results(main, calls) attach_agent_metadata(main, calls, agent_spawns) + attach_rounds(main, agent_spawns) orchestrator_calls = calls.values.reject { |c| c.name == "Agent" } { - metadata: metadata(main), + metadata: metadata(main, agent_spawns), agent_spawns: agent_spawns.map(&:to_h), segments: segments(main).map(&:to_h), missed_delegation_candidates: @@ -294,6 +297,31 @@ module OrchAudit end end + # SendMessage tool_use calls address an agent by its raw agentId or by the + # teammate name assigned to it; the transcript only ever carries the + # spawn's description as a name-shaped string, so match on either. + def attach_rounds(main, spawns) + targets = send_message_targets(main) + spawns.each do |spawn| + spawn.rounds = 1 + targets.count { |t| addressed_to?(t, spawn) } + end + end + + def send_message_targets(main) + main.flat_map do |line| + line.tool_uses.select { |b| b["name"] == "SendMessage" } + .map { |b| b.dig("input", "to") } + end.compact + end + + def addressed_to?(target, spawn) + t = target.to_s.strip.downcase + return false if t.empty? + return true if spawn.agent_id && t == spawn.agent_id.to_s.downcase + desc = spawn.call.input["description"].to_s.strip.downcase + !desc.empty? && (t == desc || desc.include?(t) || t.include?(desc)) + end + def result_size(block, line) content = block["content"] size = case content @@ -304,9 +332,12 @@ module OrchAudit size.zero? ? line.raw["toolUseResult"].to_s.length : size end - def metadata(main) + def metadata(main, spawns) assistants = main.select { |l| l.type == "assistant" } stamps = main.map(&:timestamp).compact + main_tok = main_loop_output_tokens(assistants) + side_tok = sidechain_output_tokens + rounds = spawns.map(&:rounds) { cwd: main.map(&:cwd).compact.first, started: stamps.min, @@ -314,10 +345,41 @@ module OrchAudit assistant_turns: assistants.size, human_prompts: main.count(&:human_prompt?), main_loop_models: assistants.map(&:model).compact.tally, - jsonl_lines: main.last&.number + jsonl_lines: main.last&.number, + main_loop_output_tokens: main_tok, + sidechain_output_tokens: side_tok, + main_loop_share: main_loop_share(main_tok, side_tok), + total_spawns: spawns.size, + max_rounds: rounds.max || 0 } end + def main_loop_output_tokens(assistants) + assistants.sum { |l| l.message.dig("usage", "output_tokens").to_i } + end + + # Subagent transcripts persist as sibling files at + # //subagents/agent-*.jsonl (same + # native JSONL shape, own usage blocks); 0 when the dir is absent. + def sidechain_output_tokens + dir = File.dirname(@path) + base = File.basename(@path, ".jsonl") + total = 0 + Dir.glob(File.join(dir, base, "subagents", "*.jsonl")).each do |f| + File.foreach(f) do |raw| + parsed = JSON.parse(raw) rescue next + next unless parsed["type"] == "assistant" + total += parsed.dig("message", "usage", "output_tokens").to_i + end + end + total + end + + def main_loop_share(main_tok, side_tok) + return 1.0 if side_tok.zero? + (main_tok.to_f / (main_tok + side_tok)).round(3) + end + def segments(main) segs = [Segment.new("pre-spawn-1")] spawn_count = 0 diff --git a/plugins/os-orchestration/eval/README.md b/plugins/os-orchestration/eval/README.md index 37dcfa6..3b8a9b2 100644 --- a/plugins/os-orchestration/eval/README.md +++ b/plugins/os-orchestration/eval/README.md @@ -1,4 +1,4 @@ -# os-orchestration eval — E1–E3 (session-orchestration behavior) +# os-orchestration eval — E1–E3, E5 (session-orchestration behavior) _Created: 2026-07-06. Scenarios sourced from the verified misses in `docs/orchestration-audit/2026-07-06-findings.md` (clusters 1–3), per the @@ -34,14 +34,73 @@ The checker scores the REAL session transcript written to |----|---------|-------|--------------------------------------| | E1P | 1 | 3-module judgment review; runner sets `CLAUDE_CODE_SUBAGENT_MODEL=haiku` | A: no spawn / no non-haiku request. B: downgrade not flagged in visible text (narrow haiku-judge fallback on ambiguous haiku mentions only) | | E1N | 1 | identical task, stub absent | FAIL only on a false downgrade claim (judge confirms) | -| E2P | 2 | 9 independent mechanical JSON edits | A: no delegation. B: any spawn without explicit `model:`. Tier tally informational | +| E2P | 2 | 9 independent mechanical JSON edits | A: 0 spawns AND >=9 Edit+Write ops (`no-delegation-and-grind`). B: any spawn without explicit `model:`. C: task-incomplete (see below). Tier tally informational | | E2N | 2 | single-file known typo fix | any Agent spawn | -| E3P | 3 | ~400KB log investigation | A: no delegation. B: pre-spawn-1 bytes > 15KB. C: ≥5KB orchestrator-Read file re-named in a spawn prompt (dual-read) | +| E3P | 3 | ~400KB log investigation | A: whole-session `bytes_read` > 74KB (`overread`). B: fewer than 2 of {worker-death, queue-backup, dropped-events} named in the final message (`root-cause-missing`). C: dual-read, only checked when a spawn happened | | E3N | 3 | uncertain-target doc fix (orienting grep correct) | any Agent spawn | +| E5P | batch delegation | colleague-authored fixture, `## Expected files` coverage | A: 0 spawns AND >=12 Edit+Write ops (`main-loop-grind`). B: >3 spawns (`one-per-item`). C: any expected file missing/empty. D: any spawn without explicit `model:` | +| E5N | batch delegation | task that doesn't warrant fan-out | 2+ spawns (`parallel-fanout`); 0 or 1 spawn are both `sequential` and fine | Harness ERRORs (not behavioral FAILs): missing transcript, E1P stub not applied, E1N env leak. Check reason strings before counting a cell. +**E2P/E3P criterion redesign (pre-registered, 2026-07-06):** neither positive +requires delegation any more — both accept a well-scoped direct/scripted +path, since the WS1 baseline grid showed genuinely superior direct +strategies scoring FAIL under the old delegation-mandatory axes. E2P +PASS iff task-complete (all 9 `services/*.json` differ from the pristine +fixture copies in `eval/fixture/project/services/` and remain valid JSON) +AND (delegated with every spawn `model_explicit`, OR 0 spawns with +< 9 Edit+Write ops — "scripted-direct"). E3P PASS iff whole-session +`bytes_read` <= 74000 (anchor: WS1 audit S7 exemplar) AND the final +assistant message names >=2 of the three planted-incident concepts (see +`fixture/project/bin/gen-logs`) AND (no dual-read, only evaluated when a +spawn happened). + +## E5 family and the `## Expected files` convention + +E5 measures batched-vs-per-item delegation on a task with several +independent outputs. A colleague is authoring the E5 scenario files in +parallel (run-set + reserve, per the same held-out discipline as E1–E3) — +**do not create or edit `scenarios/E5*.md` / `scenarios-reserve/E5*.md` +here.** The fixed convention those files must follow: a `## Expected files` +section listing one repo-relative path per line (bare or `-`/`*` list-item, +optionally backtick-quoted); the checker's coverage axis (E5P axis C) +requires every listed path to exist non-empty in the sandbox. `eval/bin/run` +resolves E5* scenarios through the same `scenarios/` then `scenarios-reserve/` +lookup as every other scenario, and takes the default env arm (`env -u +CLAUDE_CODE_SUBAGENT_MODEL`) — no new case arm was needed. Self-test proves +the checker logic against a fabricated scenario file delivered via +`ORCH_EVAL_SCENARIO_FILE` (self-test-only override, mirrors +`ORCH_EVAL_TRANSCRIPT`) rather than writing into `scenarios/`. + +## Model aliases + +`eval/bin/run ...` accepts `fable` as a model name; it is +passed to `claude -p --model` as `claude-fable-5` while the TSV keeps +recording the short name (`fable`) given on the CLI, matching how `sonnet`/ +`haiku` are recorded verbatim. + +## Token/round instrumentation + +Every scenario row's TSV `info` column now carries econ figures, +informational only (never a verdict axis) on every scenario: +`mltok=;sctok=;mlshare=<0.NNN main-loop share>;prebytes=;spawns=;maxrounds=`. + +`mlshare` is now a real token-based ratio, replacing the old char-count proxy +implicit in `bytes_read`. Subagent transcripts persist as sibling files at +`//subagents/agent-*.jsonl` +(same native JSONL shape, own `usage` blocks per assistant line) — verified +for both in-session and headless (`claude -p`) sessions. `audit/bin/extract` +sums `message.usage.output_tokens` over non-sidechain main-transcript +assistant lines for `main_loop_output_tokens`, and over every assistant line +in the sibling `subagents/*.jsonl` files (0 if the dir is absent) for +`sidechain_output_tokens`; `main_loop_share` is `main / (main + sidechain)` +rounded to 3 places, `1.0` when sidechain is 0. `agent_spawns[].rounds` is +`1 + SendMessage calls addressed to that agent` (by raw `agentId` or by the +spawn's `description` as a name proxy — the transcript carries no other +name field); metadata gains `total_spawns` and `max_rounds`. + ## Conformance dry-run (done at design time, 2026-07-06) Per the ladder anti-pattern list, each cell was checked against "would a model @@ -49,8 +108,10 @@ perfectly following the shipped ORCHESTRATION.md pass?" — yes on all six (E1P: delegate+sonnet+flag; E1N: no false claim; E2P: fan out with explicit models; E2N/E3N: do directly; E3P: delegate the log review after at most an orienting look) — and "would an always-delegate model pass positives while failing negatives?" — it fails -E2N/E3N, so discrimination is real. `bin/self-test` (21 cases, model-free) encodes -these including shipped-compliant transcripts, per the self-test-blind-spot lesson. +E2N/E3N, so discrimination is real. `bin/self-test` (32 cases, model-free) encodes +these including shipped-compliant transcripts, per the self-test-blind-spot lesson; +this now also covers the E2P/E3P criterion redesign, the E5 family, and the +token/round accounting. ## Procedure diff --git a/plugins/os-orchestration/eval/bin/check b/plugins/os-orchestration/eval/bin/check index f76945b..2500d69 100755 --- a/plugins/os-orchestration/eval/bin/check +++ b/plugins/os-orchestration/eval/bin/check @@ -1,7 +1,7 @@ #!/usr/bin/env ruby # frozen_string_literal: true -# Deterministic-first checker for the os-orchestration eval (E1-E3). +# Deterministic-first checker for the os-orchestration eval (E1-E3, E5). # # Usage: check [--tsv []] # @@ -11,8 +11,9 @@ # ORCH_EVAL_TRANSCRIPT (self-test). Judge fallback (E1 language axes only) is # a narrow frozen-rubric haiku call, stubbable via ORCH_EVAL_JUDGE_CMD. # -# FAIL axes map ONLY to the verified 2026-07-06 audit misses (clusters 1-3); -# everything else (tier tally, edit-applied, consultation) is informational. +# FAIL axes map ONLY to verified audit misses / pre-registered criteria; +# everything else (tier tally, edit-applied, consultation, econ figures) is +# informational. # # Exit: 0 PASS, 1 FAIL, 2 harness ERROR. @@ -21,15 +22,38 @@ require "json" load File.expand_path("../../audit/bin/extract", __dir__) module OrchEval - PRE_SPAWN_BYTE_BUDGET = 15_000 # E3P axis B; audit: good sessions ~0, S7 74KB DUAL_READ_MIN_CHARS = 5_000 # smaller reads count as orienting, not dual-read + # E3P axis A; anchor: WS1 audit S7 exemplar (~74KB whole-session bytes_read). + MAIN_LOOP_BYTE_BUDGET = 74_000 + + # E2P axis A scripted-direct branch: fewer Edit+Write ops than the 9 + # services/*.json files means the change was scripted, not a per-file grind. + E2P_GRIND_EDIT_THRESHOLD = 9 + + # E5P axis A scripted-direct branch (pre-registered constant, not tied to + # any one scenario's expected-file count). + E5P_GRIND_EDIT_THRESHOLD = 12 + E5P_SPAWN_BATCH_CEILING = 3 + + FIXTURE_SERVICES_DIR = File.expand_path("../fixture/project/services", __dir__) + + # E3P axis B: at least two of the three planted-incident concepts (worker + # death, queue backup, dropped events — see fixture/project/bin/gen-logs) + # must be named in the final assistant message. + ROOT_CAUSE_CONCEPTS = { + worker: /\bworkers?(?:-3)?\b.{0,60}?\b(died|dead|crash(?:ed)?|down|failed|failure)\b|\b(died|dead|crash(?:ed)?|down|failed|failure)\b.{0,60}?\bworkers?(?:-3)?\b/i, + queue: /\bqueue\b.{0,60}?\b(back ?up|backlog|full|filled|overflow(?:ed)?|saturat\w*|depth)\b|\b(back ?up|backlog|overflow(?:ed)?|saturat\w*)\b.{0,60}?\bqueue\b/i, + dropped: /\b(dropped|drop(?:ping|s)?|lost|loss)\b.{0,60}?\bevents?\b|\bevents?\b.{0,60}?\b(dropped|drop(?:ping|s)?|lost|loss)\b/i + }.freeze + JUDGE_RUBRIC = File.expand_path("../judge-rubric.md", __dir__) class Transcript - attr_reader :path + attr_reader :path, :sandbox_path def initialize(sandbox) + @sandbox_path = sandbox @path = ENV["ORCH_EVAL_TRANSCRIPT"] || discover(sandbox) end @@ -61,6 +85,11 @@ module OrchEval end end + # Text of the last assistant text block, main chain only ("" if none). + def final_assistant_text + assistant_texts.last&.dig(:text).to_s + end + private def each_main_line @@ -126,6 +155,8 @@ module OrchEval def self.error(reason) = new("ERROR", { harness: reason }) + def merge_info(extra) = self.class.new(verdict, axes, info.merge(extra)) + def tsv(scenario, model, rep) ax = axes.map { |k, v| "#{k}:#{v}" }.join(";") inf = info.map { |k, v| "#{k}=#{v}" }.join(";") @@ -143,6 +174,12 @@ module OrchEval def result return Result.error("transcript-missing") unless @t.found? + dispatch.merge_info(econ_info) + end + + private + + def dispatch case @id.split("-").first when "E1P" then e1_positive when "E1N" then e1_negative @@ -150,14 +187,28 @@ module OrchEval when "E2N" then e2_negative when "E3P" then e3_positive when "E3N" then e3_negative + when "E5P" then e5_positive + when "E5N" then e5_negative else Result.error("unknown-scenario:#{@id}") end end - private - def spawns = @t.fact[:agent_spawns] def pre_spawn = @t.fact[:segments].find { |s| s[:label] == "pre-spawn-1" } + def segments = @t.fact[:segments] + + # Informational econ figures appended to EVERY scenario row. + def econ_info + meta = @t.fact[:metadata] + { + mltok: meta[:main_loop_output_tokens].to_i, + sctok: meta[:sidechain_output_tokens].to_i, + mlshare: meta[:main_loop_share], + prebytes: pre_spawn[:bytes_read], + spawns: spawns.size, + maxrounds: meta[:max_rounds].to_i + } + end def tier_tally spawns.map { |s| s[:model_explicit] ? s[:model_param] : "ABSENT" }.tally @@ -214,12 +265,24 @@ module OrchEval Result.new(verdict, { B: claim }, tiers: tier_tally, spawns: spawns.size) end + # PASS iff task-complete AND (delegated-with-explicit-models OR + # scripted-direct: 0 spawns and < E2P_GRIND_EDIT_THRESHOLD Edit+Write ops). def e2_positive - return Result.new("FAIL", { A: "no-delegation" }) if spawns.empty? + changed_n = changed_services_count + + if spawns.empty? + edits = whole_session_edit_write_count + return Result.new("FAIL", { A: "no-delegation-and-grind" }) if edits >= E2P_GRIND_EDIT_THRESHOLD + return Result.new("FAIL", { C: "task-incomplete:#{changed_n}-changed" }) unless task_complete_services? + return Result.new("PASS", { A: "scripted-direct", C: "pass" }, edits: edits) + end + implicit = spawns.reject { |s| s[:model_explicit] } - axes = { A: "pass", B: implicit.empty? ? "pass" : "implicit-model:#{implicit.map { |s| s[:seq] }.join(',')}" } - verdict = implicit.empty? ? "PASS" : "FAIL" - Result.new(verdict, axes, tiers: tier_tally, spawns: spawns.size) + return Result.new("FAIL", { B: "implicit-model:#{implicit.map { |s| s[:seq] }.join(',')}" }, + tiers: tier_tally) unless implicit.empty? + return Result.new("FAIL", { C: "task-incomplete:#{changed_n}-changed" }, tiers: tier_tally) unless task_complete_services? + + Result.new("PASS", { A: "delegated", B: "pass", C: "pass" }, tiers: tier_tally) end def e2_negative @@ -228,14 +291,25 @@ module OrchEval pre_bytes: pre_spawn[:bytes_read]) end + # Verdict no longer requires delegation. A = whole-session bytes_read + # under budget; B = root cause named mechanically; C = the existing + # dual-read check, only evaluated when a spawn happened. def e3_positive - return Result.new("FAIL", { A: "no-delegation" }) if spawns.empty? - bytes = pre_spawn[:bytes_read] - axes = { A: "pass", B: bytes <= PRE_SPAWN_BYTE_BUDGET ? "pass" : "overread:#{bytes}" } - dual = dual_reads - axes[:C] = dual.empty? ? "pass" : "dual-read:#{dual.join(',')}" + bytes = whole_session_bytes_read + axes = { A: bytes <= MAIN_LOOP_BYTE_BUDGET ? "pass" : "overread:#{bytes}" } + + matched = ROOT_CAUSE_CONCEPTS.select { |_, re| @t.final_assistant_text =~ re }.keys + axes[:B] = matched.size >= 2 ? "pass" : "root-cause-missing:#{matched.join(',')}" + + if spawns.empty? + axes[:C] = "pass" + else + dual = dual_reads + axes[:C] = dual.empty? ? "pass" : "dual-read:#{dual.join(',')}" + end + verdict = axes.values.all? { |v| v == "pass" } ? "PASS" : "FAIL" - Result.new(verdict, axes, pre_bytes: bytes, spawns: spawns.size, tiers: tier_tally) + Result.new(verdict, axes, bytes_read: bytes, tiers: tier_tally) end def e3_negative @@ -245,6 +319,35 @@ module OrchEval pre_bytes: pre_spawn[:bytes_read], edited: edited ? "yes" : "no") end + # PASS iff (spawns>=1 OR scripted-direct) AND batching AND coverage AND + # every spawn has an explicit model. + def e5_positive + coverage = expected_files_coverage + + if spawns.empty? + edits = whole_session_edit_write_count + return Result.new("FAIL", { A: "main-loop-grind" }) if edits >= E5P_GRIND_EDIT_THRESHOLD + return Result.new("FAIL", { C: "missing:#{coverage[:missing].size}" }) unless coverage[:complete] + return Result.new("PASS", { A: "scripted-direct", B: "pass", C: "pass", D: "pass" }, edits: edits) + end + + axes = { A: "pass" } + axes[:B] = spawns.size <= E5P_SPAWN_BATCH_CEILING ? "pass" : "one-per-item:#{spawns.size}" + axes[:C] = coverage[:complete] ? "pass" : "missing:#{coverage[:missing].size}" + implicit = spawns.reject { |s| s[:model_explicit] } + axes[:D] = implicit.empty? ? "pass" : "implicit-model:#{implicit.map { |s| s[:seq] }.join(',')}" + verdict = axes.values.all? { |v| v == "pass" } ? "PASS" : "FAIL" + Result.new(verdict, axes, tiers: tier_tally) + end + + # FAIL iff 2+ spawns (parallel fan-out on a task that doesn't warrant it); + # 0 or 1 spawn are both fine ("sequential"). + def e5_negative + verdict = spawns.size >= 2 ? "FAIL" : "PASS" + Result.new(verdict, { A: verdict == "FAIL" ? "parallel-fanout:#{spawns.size}" : "sequential" }, + tiers: tier_tally) + end + # Large pre-spawn Read whose basename reappears in any spawn prompt. def dual_reads prompts = @t.spawn_prompts.map { |p| p[:prompt] }.join("\n") @@ -259,6 +362,74 @@ module OrchEval pre = pre_spawn pre && (pre[:tool_counts]["Edit"].to_i + pre[:tool_counts]["Write"].to_i) > 0 end + + def whole_session_edit_write_count + segments.sum { |s| s[:tool_counts]["Edit"].to_i + s[:tool_counts]["Write"].to_i } + end + + def whole_session_bytes_read + segments.sum { |s| s[:bytes_read].to_i } + end + + # E2P task-complete check: every pristine services/*.json file must have + # a sandbox counterpart that differs from the pristine copy and is still + # valid JSON. `sandbox` is the checker's second positional arg (a real + # sandbox dir in headless runs; a fabricated tmpdir in self-test). + def task_complete_services? + pristine_service_files.all? { |p| changed_and_valid_service?(p) } + end + + def changed_services_count + pristine_service_files.count { |p| changed_and_valid_service?(p) } + end + + def pristine_service_files + Dir.glob(File.join(FIXTURE_SERVICES_DIR, "*.json")).sort + end + + def changed_and_valid_service?(pristine_path) + sandbox_path = File.join(@t.sandbox_path.to_s, "services", File.basename(pristine_path)) + return false unless File.exist?(sandbox_path) + content = File.read(sandbox_path) + return false if content == File.read(pristine_path) + JSON.parse(content) + true + rescue JSON::ParserError + false + end + + # E5P axis C: "## Expected files" section of the scenario file lists one + # repo-relative path per line; every path must exist non-empty in the + # sandbox. + def expected_files_coverage + paths = expected_files + missing = paths.reject { |rel| file_present_nonempty?(rel) } + { complete: missing.empty?, missing: missing } + end + + def expected_files + body = File.read(scenario_file) + section = body[/^## Expected files\n(.*?)(\n## |\z)/m, 1].to_s + section.lines.filter_map do |line| + line.sub(/^[-*]\s*/, "").strip.delete("`") + end.reject(&:empty?) + end + + # ORCH_EVAL_SCENARIO_FILE (self-test only) points at a fabricated + # scenario file so self-test never has to write into scenarios/ or + # scenarios-reserve/ (E5's real scenario files are colleague-authored). + def scenario_file + return ENV["ORCH_EVAL_SCENARIO_FILE"] if ENV["ORCH_EVAL_SCENARIO_FILE"] + eval_root = File.expand_path("..", __dir__) + run_set = File.join(eval_root, "scenarios", "#{@id}.md") + return run_set if File.exist?(run_set) + File.join(eval_root, "scenarios-reserve", "#{@id}.md") + end + + def file_present_nonempty?(rel) + path = File.join(@t.sandbox_path.to_s, rel) + File.exist?(path) && File.size(path).positive? + end end end diff --git a/plugins/os-orchestration/eval/bin/run b/plugins/os-orchestration/eval/bin/run index eb983d1..8975548 100755 --- a/plugins/os-orchestration/eval/bin/run +++ b/plugins/os-orchestration/eval/bin/run @@ -8,6 +8,10 @@ # E1P* scenarios get CLAUDE_CODE_SUBAGENT_MODEL=haiku injected into the child # environment (the cluster-1 downgrade stub). All other scenarios explicitly # strip that var so a leftover global setting can't leak in. +# +# MODEL accepts a short alias on the CLI ("fable" -> claude-fable-5, passed to +# `claude -p --model`); the TSV keeps recording the short name given on the +# CLI, not the resolved alias. set -euo pipefail SCENARIO="${1:?usage: run [--reps N] [--results FILE]}" @@ -39,6 +43,13 @@ ENV_PREFIX=(env -u CLAUDE_CODE_SUBAGENT_MODEL) case "$SCENARIO" in E1P*) ENV_PREFIX=(env CLAUDE_CODE_SUBAGENT_MODEL=haiku) ;; esac +# E5* scenarios take the default env arm above (no downgrade stub) — same +# case statement, no new arm needed. + +case "$MODEL" in + fable) CLI_MODEL="claude-fable-5" ;; + *) CLI_MODEL="$MODEL" ;; +esac for rep in $(seq 1 "$REPS"); do SANDBOX="$WORKDIR/$SCENARIO-$MODEL-r$rep" @@ -46,7 +57,7 @@ for rep in $(seq 1 "$REPS"); do # cwd = sandbox: global plugins' SessionStart hooks fire for real. (cd "$SANDBOX" && timeout 1500 "${ENV_PREFIX[@]}" claude -p \ - --model "$MODEL" \ + --model "$CLI_MODEL" \ --output-format stream-json --verbose \ --dangerously-skip-permissions \ "$TASK" > cli-output.jsonl) || echo "claude exited non-zero for $SANDBOX" >&2 diff --git a/plugins/os-orchestration/eval/bin/self-test b/plugins/os-orchestration/eval/bin/self-test index 002410a..da56502 100755 --- a/plugins/os-orchestration/eval/bin/self-test +++ b/plugins/os-orchestration/eval/bin/self-test @@ -13,19 +13,50 @@ require "json" require "tmpdir" +require "fileutils" CHECK = File.expand_path("check", __dir__) JUDGE_YES = "sh -c 'cat >/dev/null; echo YES'" JUDGE_NO = "sh -c 'cat >/dev/null; echo NO'" +FIXTURE_SERVICES = Dir.glob(File.expand_path("../fixture/project/services/*.json", __dir__)).sort +ALL_SERVICE_NAMES = FIXTURE_SERVICES.map { |p| File.basename(p) } + +# Seeds dir/services/*.json from the pristine fixture, mutating the ones +# named in changed_names so the checker's task-complete comparison sees them +# as done (E2P). +def seed_services(dir, changed_names:) + FileUtils.mkdir_p(File.join(dir, "services")) + FIXTURE_SERVICES.each do |pristine_path| + name = File.basename(pristine_path) + if changed_names.include?(name) + data = JSON.parse(File.read(pristine_path)) + data["retry_policy"] = "exponential" + File.write(File.join(dir, "services", name), JSON.pretty_generate(data)) + else + File.write(File.join(dir, "services", name), File.read(pristine_path)) + end + end +end + +# Writes non-empty placeholder files at the given sandbox-relative paths +# (E5P "## Expected files" coverage check). +def write_files(dir, relative_paths) + relative_paths.each do |rel| + full = File.join(dir, rel) + FileUtils.mkdir_p(File.dirname(full)) + File.write(full, "ok\n") + end +end + class TranscriptBuilder def initialize @lines = [] @seq = 0 end - def assistant_text(text) - push_assistant([{ "type" => "text", "text" => text }]) + def assistant_text(text, output_tokens: nil) + push_assistant([{ "type" => "text", "text" => text }], output_tokens: output_tokens) self end @@ -60,10 +91,11 @@ class TranscriptBuilder def next_id = "t#{@seq += 1}" - def push_assistant(content) + def push_assistant(content, output_tokens: nil) + message = { "role" => "assistant", "model" => "claude-sonnet-4-6", "content" => content } + message["usage"] = { "output_tokens" => output_tokens } if output_tokens @lines << { "type" => "assistant", "isSidechain" => false, "cwd" => "/tmp/fab", - "timestamp" => "2026-07-06T12:00:00Z", - "message" => { "role" => "assistant", "model" => "claude-sonnet-4-6", "content" => content } } + "timestamp" => "2026-07-06T12:00:00Z", "message" => message } end def push_user(content) @@ -79,20 +111,36 @@ class SelfTest @count = 0 end - def case!(name, scenario, builder, expect_verdict, expect_axis: nil, judge: JUDGE_NO) + # seed: proc(dir) called before the transcript is written, for cases that + # need real files under the sandbox (E2P services/, E5P expected files). + # scenario_file: fabricated "## Expected files"-bearing scenario markdown + # (E5*), delivered via ORCH_EVAL_SCENARIO_FILE — self-test never writes + # into scenarios/ or scenarios-reserve/. + # expect_info: substrings that must all appear in the TSV info column. + def case!(name, scenario, builder, expect_verdict, expect_axis: nil, judge: JUDGE_NO, + seed: nil, scenario_file: nil, expect_info: []) @count += 1 Dir.mktmpdir do |dir| + seed&.call(dir) path = builder.write(File.join(dir, "t.jsonl")) env = { "ORCH_EVAL_TRANSCRIPT" => path, "ORCH_EVAL_JUDGE_CMD" => judge } + if scenario_file + sf = File.join(dir, "scenario.md") + File.write(sf, scenario_file) + env["ORCH_EVAL_SCENARIO_FILE"] = sf + end out = IO.popen(env, [CHECK, scenario, dir, "--tsv", "fab", "1"], &:read) cols = out.strip.split("\t") verdict = cols[3] axes = cols[4].to_s - ok = verdict == expect_verdict && (expect_axis.nil? || axes.include?(expect_axis)) + info = cols[5].to_s + ok = verdict == expect_verdict && + (expect_axis.nil? || axes.include?(expect_axis)) && + expect_info.all? { |frag| info.include?(frag) } unless ok - @failures << "#{name}: expected #{expect_verdict}#{" (#{expect_axis})" if expect_axis}, got #{verdict} axes=#{axes}" + @failures << "#{name}: expected #{expect_verdict}#{" (#{expect_axis})" if expect_axis}, got #{verdict} axes=#{axes} info=#{info}" end - puts format("%-42s %s", name, ok ? "ok" : "FAIL <- #{verdict} #{axes}") + puts format("%-42s %s", name, ok ? "ok" : "FAIL <- #{verdict} #{axes} #{info}") end end @@ -156,23 +204,39 @@ leaked = TranscriptBuilder.new .assistant_text("Done.") t.case!("E1N env leak -> harness ERROR", "E1N-pipeline-review", leaked, "ERROR") -# --- E2P (explicit model on fan-out) --- +# --- E2P v2 (task-complete = all 9 services/*.json changed + valid JSON; +# delegated-with-explicit-models OR scripted-direct) --- compliant_e2p = TranscriptBuilder.new .assistant_text("Nine independent mechanical edits - fanning out at haiku.") .spawn(prompt: "Add retry_policy to services/billing.json", model: "haiku", resolved: "claude-haiku-4-5-20251001") .spawn(prompt: "Add retry_policy to services/crm.json", model: "haiku", resolved: "claude-haiku-4-5-20251001") .spawn(prompt: "Add retry_policy to services/email.json", model: "haiku", resolved: "claude-haiku-4-5-20251001") -t.case!("E2P shipped-compliant", "E2P-retry-policy", compliant_e2p, "PASS") +t.case!("E2P shipped-compliant delegated", "E2P-retry-policy", compliant_e2p, "PASS", expect_axis: "A:delegated", + seed: ->(dir) { seed_services(dir, changed_names: ALL_SERVICE_NAMES) }) -implicit = TranscriptBuilder.new +scripted_e2p = TranscriptBuilder.new + .assistant_text("Wrote a one-off script to add retry_policy to every services/*.json file.") + .tool("Bash", { "command" => "ruby bin/add_retry_policy.rb" }, result_content: "updated 9 files") +t.case!("E2P scripted-direct", "E2P-retry-policy", scripted_e2p, "PASS", expect_axis: "A:scripted-direct", + seed: ->(dir) { seed_services(dir, changed_names: ALL_SERVICE_NAMES) }) + +implicit_e2p = TranscriptBuilder.new .spawn(prompt: "Add retry_policy to services/billing.json", model: "haiku", resolved: "claude-haiku-4-5-20251001") .spawn(prompt: "Add retry_policy to services/crm.json", resolved: "claude-sonnet-4-6") -t.case!("E2P one implicit model", "E2P-retry-policy", implicit, "FAIL", expect_axis: "implicit-model:2") +t.case!("E2P implicit model on one spawn", "E2P-retry-policy", implicit_e2p, "FAIL", expect_axis: "implicit-model:2", + seed: ->(dir) { seed_services(dir, changed_names: ALL_SERVICE_NAMES) }) -nine_direct = TranscriptBuilder.new -9.times { |i| nine_direct.tool("Edit", { "file_path" => "/tmp/fab/services/s#{i}.json" }) } -t.case!("E2P did it all directly", "E2P-retry-policy", nine_direct, "FAIL", expect_axis: "A:no-delegation") +grind_e2p = TranscriptBuilder.new +9.times { |i| grind_e2p.tool("Edit", { "file_path" => "/tmp/fab/services/s#{i}.json" }) } +t.case!("E2P per-file-grind", "E2P-retry-policy", grind_e2p, "FAIL", expect_axis: "A:no-delegation-and-grind", + seed: ->(dir) { seed_services(dir, changed_names: ALL_SERVICE_NAMES) }) + +incomplete_e2p = TranscriptBuilder.new + .spawn(prompt: "Add retry_policy to services/billing.json", model: "haiku", resolved: "claude-haiku-4-5-20251001") + .spawn(prompt: "Add retry_policy to services/crm.json", model: "haiku", resolved: "claude-haiku-4-5-20251001") +t.case!("E2P task-incomplete", "E2P-retry-policy", incomplete_e2p, "FAIL", expect_axis: "task-incomplete:2-changed", + seed: ->(dir) { seed_services(dir, changed_names: %w[billing.json crm.json]) }) # --- E2N (trivial single-file) --- @@ -183,28 +247,32 @@ t.case!("E2N delegated trivial fix", "E2N-readme-typo", TranscriptBuilder.new.spawn(prompt: "Fix the typo in README.md", model: "haiku", resolved: "claude-haiku-4-5-20251001"), "FAIL", expect_axis: "spawned-on-trivial") -# --- E3P (delegate the investigation) --- +# --- E3P v2 (delegation no longer required; A = whole-session bytes_read +# budget, B = >=2 root-cause concepts in the final message, C = dual-read +# only checked when a spawn happened) --- -compliant_e3p = TranscriptBuilder.new - .tool("Bash", { "command" => "ls logs/" }, result_content: "relay-2026-07-01.log\nrelay-2026-07-04.log\nworker-2026-07-04.log") - .assistant_text("400KB of logs - delegating the review rather than reading them here.") - .spawn(prompt: "Read all files under logs/ for 2026-07-04, reconstruct the incident timeline, report root cause.", model: "sonnet", resolved: "claude-sonnet-4-6") -t.case!("E3P shipped-compliant (orienting ls only)", "E3P-dropped-events", compliant_e3p, "PASS") +surgical_e3p = TranscriptBuilder.new + .tool("Bash", { "command" => "ls logs/" }, result_content: "relay-2026-07-04.log\nworker-2026-07-04.log") + .tool("Grep", { "pattern" => "ERROR|WARN" }, result_content: "x" * 2_000) + .assistant_text("Root cause: worker-3 died, which caused the queue to back up, and dispatch dropped events once the backlog maxed out.") +t.case!("E3P surgical-direct (0 spawns, root cause named)", "E3P-dropped-events", surgical_e3p, "PASS") -overread = TranscriptBuilder.new - .tool("Read", { "file_path" => "/tmp/fab/logs/relay-2026-07-03.log" }, result_content: "x" * 30_000) - .tool("Read", { "file_path" => "/tmp/fab/logs/worker-2026-07-04.log" }, result_content: "x" * 30_000) - .spawn(prompt: "Summarize the incident from logs/ and report root cause.", model: "sonnet", resolved: "claude-sonnet-4-6") -t.case!("E3P pre-spawn overread", "E3P-dropped-events", overread, "FAIL", expect_axis: "B:overread") +overread_e3p = TranscriptBuilder.new + .tool("Read", { "file_path" => "/tmp/fab/logs/relay-2026-07-03.log" }, result_content: "x" * 40_000) + .tool("Read", { "file_path" => "/tmp/fab/logs/worker-2026-07-04.log" }, result_content: "x" * 40_000) + .assistant_text("Root cause: worker-3 died and the queue backed up, dropping events.") +t.case!("E3P whole-session overread", "E3P-dropped-events", overread_e3p, "FAIL", expect_axis: "A:overread") -dual = TranscriptBuilder.new +missing_rc_e3p = TranscriptBuilder.new + .tool("Bash", { "command" => "ls logs/" }, result_content: "relay-2026-07-04.log") + .assistant_text("Investigated the logs and wrote up a timeline for the team.") +t.case!("E3P root-cause-missing", "E3P-dropped-events", missing_rc_e3p, "FAIL", expect_axis: "B:root-cause-missing") + +dual_e3p = TranscriptBuilder.new .tool("Read", { "file_path" => "/tmp/fab/logs/relay-2026-07-04.log" }, result_content: "x" * 6_000) .spawn(prompt: "Read logs/relay-2026-07-04.log and logs/worker-2026-07-04.log and report the root cause.", model: "sonnet", resolved: "claude-sonnet-4-6") -t.case!("E3P dual-read (within byte budget)", "E3P-dropped-events", dual, "FAIL", expect_axis: "dual-read:relay-2026-07-04.log") - -t.case!("E3P never delegated", "E3P-dropped-events", - TranscriptBuilder.new.tool("Read", { "file_path" => "/tmp/fab/logs/relay-2026-07-04.log" }, result_content: "x" * 40_000).assistant_text("Root cause: worker-3 died."), - "FAIL", expect_axis: "A:no-delegation") + .assistant_text("Root cause: worker-3 died, the queue backed up, and events were dropped.") +t.case!("E3P dual-read (with spawn)", "E3P-dropped-events", dual_e3p, "FAIL", expect_axis: "dual-read:relay-2026-07-04.log") # --- E3N (orienting reads are correct) --- @@ -218,4 +286,93 @@ t.case!("E3N delegated the hunt", "E3N-brand-spelling", TranscriptBuilder.new.spawn(prompt: "Find where RelayStation is misspelled in docs/ and fix it.", model: "haiku", resolved: "claude-haiku-4-5-20251001"), "FAIL", expect_axis: "spawned-orienting-work") +# --- E5P/E5N (fabricated scenario file — colleague authors the real E5 +# scenarios; self-test never touches scenarios/ or scenarios-reserve/) --- +# +# A = delegate-or-script (spawns>=1, OR 0 spawns with <12 Edit+Write and +# coverage complete); B = batching (spawns<=3); C = "## Expected files" +# coverage; D = every spawn has an explicit model. + +e5p_scenario_md = <<~MD + # E5P — fabricated self-test scenario + + ## Task + + Produce out/report-a.md and out/report-b.md. + + ## Expected files + + - out/report-a.md + - out/report-b.md +MD + +grouped_e5p = TranscriptBuilder.new + .assistant_text("Grouping the work into two batched spawns.") + .spawn(prompt: "Produce out/report-a.md", model: "haiku", resolved: "claude-haiku-4-5-20251001") + .spawn(prompt: "Produce out/report-b.md", model: "haiku", resolved: "claude-haiku-4-5-20251001") +t.case!("E5P grouped-delegation", "E5P-fab", grouped_e5p, "PASS", + seed: ->(dir) { write_files(dir, %w[out/report-a.md out/report-b.md]) }, + scenario_file: e5p_scenario_md) + +one_per_item_e5p = TranscriptBuilder.new +12.times { |i| one_per_item_e5p.spawn(prompt: "Handle item #{i}", model: "haiku", resolved: "claude-haiku-4-5-20251001") } +t.case!("E5P one-per-item fan-out", "E5P-fab", one_per_item_e5p, "FAIL", expect_axis: "B:one-per-item:12", + seed: ->(dir) { write_files(dir, %w[out/report-a.md out/report-b.md]) }, + scenario_file: e5p_scenario_md) + +grind_e5p = TranscriptBuilder.new +12.times { |i| grind_e5p.tool("Edit", { "file_path" => "/tmp/fab/out/item#{i}.md" }) } +t.case!("E5P main-loop-grind", "E5P-fab", grind_e5p, "FAIL", expect_axis: "A:main-loop-grind", + seed: ->(dir) { write_files(dir, %w[out/report-a.md out/report-b.md]) }, + scenario_file: e5p_scenario_md) + +coverage_missing_e5p = TranscriptBuilder.new + .spawn(prompt: "Produce out/report-a.md", model: "haiku", resolved: "claude-haiku-4-5-20251001") + .spawn(prompt: "Produce out/report-b.md", model: "haiku", resolved: "claude-haiku-4-5-20251001") +t.case!("E5P coverage-missing", "E5P-fab", coverage_missing_e5p, "FAIL", expect_axis: "C:missing:1", + seed: ->(dir) { write_files(dir, %w[out/report-a.md]) }, + scenario_file: e5p_scenario_md) + +scripted_e5p = TranscriptBuilder.new + .tool("Bash", { "command" => "ruby bin/gen_reports.rb" }, result_content: "wrote 2 reports") +t.case!("E5P scripted-direct", "E5P-fab", scripted_e5p, "PASS", expect_axis: "A:scripted-direct", + seed: ->(dir) { write_files(dir, %w[out/report-a.md out/report-b.md]) }, + scenario_file: e5p_scenario_md) + +t.case!("E5N direct (0 spawns) sequential", "E5N-fab", + TranscriptBuilder.new.tool("Edit", { "file_path" => "/tmp/fab/README.md" }), + "PASS", expect_axis: "A:sequential") +t.case!("E5N single-agent (1 spawn) sequential", "E5N-fab", + TranscriptBuilder.new.spawn(prompt: "Fix the one thing", model: "haiku", resolved: "claude-haiku-4-5-20251001"), + "PASS", expect_axis: "A:sequential") +t.case!("E5N parallel-fanout (2 spawns)", "E5N-fab", + TranscriptBuilder.new + .spawn(prompt: "Do A", model: "haiku", resolved: "claude-haiku-4-5-20251001") + .spawn(prompt: "Do B", model: "haiku", resolved: "claude-haiku-4-5-20251001"), + "FAIL", expect_axis: "A:parallel-fanout:2") + +# --- Token/round accounting unit case --- +# Main loop assistant usage.output_tokens: 100 + 150 = 250. Fabricated +# sibling subagents/*.jsonl next to the transcript: 40 + 60 = 100. Share = +# 250 / (250 + 100) = 0.714. + +token_seed = lambda do |dir| + FileUtils.mkdir_p(File.join(dir, "t", "subagents")) + sub_lines = [ + { "type" => "assistant", "isSidechain" => false, + "message" => { "role" => "assistant", "usage" => { "output_tokens" => 40 } } }, + { "type" => "assistant", "isSidechain" => false, + "message" => { "role" => "assistant", "usage" => { "output_tokens" => 60 } } } + ] + File.write(File.join(dir, "t", "subagents", "agent-1.jsonl"), sub_lines.map(&:to_json).join("\n") + "\n") +end + +token_builder = TranscriptBuilder.new + .assistant_text("Working on it.", output_tokens: 100) + .assistant_text("Done.", output_tokens: 150) + .tool("Read", { "file_path" => "/tmp/fab/README.md" }) +t.case!("token/sidechain/share accounting", "E3N-brand-spelling", token_builder, "PASS", + expect_axis: "A:no-spawn", seed: token_seed, + expect_info: %w[mltok=250 sctok=100 mlshare=0.714]) + t.finish! diff --git a/plugins/os-orchestration/eval/fixture/project/docs/handler-spec.md b/plugins/os-orchestration/eval/fixture/project/docs/handler-spec.md new file mode 100644 index 0000000..82c5a3b --- /dev/null +++ b/plugins/os-orchestration/eval/fixture/project/docs/handler-spec.md @@ -0,0 +1,170 @@ +# Per-service delivery handlers + +Relaystation currently delivers every event through the single generic path in +`src/relay.js` (`deliver` → `fetchLike`, with failures handed to `src/retry.js`). +That path treats every downstream service identically: same payload shape, same +retry policy, same failure handling. In practice each of the nine downstream +integrations in `services/` has different tolerance for retries, different +payload requirements, and different failure semantics, and three cross-cutting +concerns (auth concurrency, per-service latency visibility, and dead-letter +handling) need their own dedicated modules. + +This spec defines twelve new handler modules under `src/handlers/`. Each module +is self-contained and does not require changes to `src/relay.js`, `src/router.js`, +`src/retry.js`, or any file under `services/` — implement each file to the +interface and behavior below. + +## Conventions + +- `'use strict';` at the top, matching the rest of `src/`. +- Each module exports a plain object via `module.exports = { ... }` (same style + as `src/metrics.js` / `src/store.js`), not a class. +- Where a handler needs the shared metrics counters, `require('../metrics')` + and call `increment(name)` — do not create a second counter store. +- Each file should be small: roughly 20–40 lines including requires and + `module.exports`. +- No new npm dependencies. Use only Node core modules already used elsewhere + in `src/` (`crypto`, `fs`, `path`, `http`) plus what a handler needs. +- Tests are not required for this pass — implementation only. + +## Part A — one handler per downstream service (9 files) + +Each service handler lives at `src/handlers/.js` and exports two +functions: + +```js +module.exports = { + prepare(event), // returns the outbound payload object for this service + isRetryable(err), // returns true/false given an error object with a + // `statusCode` field (may be undefined for network errors) +}; +``` + +`err.statusCode` is `undefined` for a connection/network failure (no response +received) and an HTTP status integer when a response was received but was not +2xx. Implement `prepare` and `isRetryable` exactly per the per-service rules +below — the rules differ enough between services that no single template +satisfies more than one of them. + +| Service | `prepare(event)` payload rule | `isRetryable(err)` rule | +| --- | --- | --- | +| `analytics` | Wrap the event: return `{ batch: [event], emitted_at: Date.now() }`. | Retry when `statusCode` is undefined (network) or `>= 500`. Never retry on any 4xx. | +| `audit` | Return the event plus an integrity field: `{ ...event, hash: sha256-hex of JSON.stringify(event) }` (use `crypto.createHash('sha256')`). | Retry **only** on a network failure (`statusCode === undefined`). Any received HTTP status, 4xx or 5xx, is not retryable — audit delivery failures must surface immediately rather than silently retry. | +| `billing` | Return a payload containing **only** an allowlist of fields copied from `event`, dropping everything else: `id`, `topic`, `amount`, `currency`, `receivedAt`. Fields absent on the event are simply omitted from the output, not set to `null`. | Retry only on `statusCode === 429` or `statusCode === 503`. Nothing else (not network failures, not other 5xx) is retryable. | +| `crm` | Return a shallow-transformed payload: rename `topic` to `event_type`, and if `event.payload` has a nested `contact` object, flatten it to top-level `contact_id` and `contact_email` fields (copied from `event.payload.contact.id` / `.email` if present) instead of nesting it. | Retry when `statusCode` is undefined or `>= 500`, same as `analytics` — but cap distinctly (see Part B note below; the cap itself lives in the retry loop, not in this predicate). | +| `email` | If `event.payload` is missing a `to` field, do not build a payload at all — call `metrics.increment('handlers.email.invalid')` and return `null` (the caller is expected to skip sending when `prepare` returns `null`). Otherwise return the event unchanged. | Retry when `statusCode` is undefined, `429`, or `>= 500`. | +| `reports` | Collapse the event into a single-field summary payload: `{ summary: \`${event.topic} at ${event.receivedAt}\` }` (use the actual template literal). | Never retryable — always return `false`, regardless of `err`. Reports tolerates loss; a single attempt is sufficient. | +| `search` | Return the event plus an `index_hint` field computed as the substring of `event.topic` before its first `.` (or the whole topic if there is no `.`). | Retry when `statusCode` is undefined or `>= 500` (same predicate shape as `analytics`/`crm`, but see Part B — `search` gets the longest retry budget of any service). | +| `sms` | Return the event with `payload.message` truncated to 160 characters if present and longer than that (leave other fields untouched). | Retry when `statusCode` is undefined or `statusCode >= 500`, but explicitly **not** on timeouts represented as `err.code === 'ETIMEDOUT'` — treat a timeout as non-retryable (carrier cost control), even though it has no `statusCode`. | +| `webhooks` | Return the event unchanged, with one added field: `relay_version` read from `package.json`'s `version` field (`require('../../package.json').version`). | Retry on any non-2xx status, i.e. `statusCode === undefined || statusCode < 200 || statusCode >= 300`. | + +Note on retry *counts*: `isRetryable` only decides whether a given failure is a +candidate for another attempt at all — it does not encode the attempt cap. +Attempt caps are documented per-service in Part C for reference by any future +caller; `src/handlers/*.js` files themselves do not need to enforce the cap +(that remains `src/retry.js`'s job when it is later wired up — not part of +this pass). + +## Part B — three core handlers (3 files) + +These are structurally different from the service handlers above — each +addresses a different cross-cutting concern, not a per-service payload/retry +rule. + +### `src/handlers/auth.js` — per-tenant concurrency limiter + +Tracks how many deliveries are currently in flight for each tenant and enforces +a maximum concurrency of 4 simultaneous in-flight deliveries per tenant. Export: + +```js +module.exports = { + acquire(tenant), // returns true if the tenant is under its concurrency cap + // (and increments its in-flight count), false if the + // tenant is already at the cap (does not increment) + release(tenant), // decrements the tenant's in-flight count, floored at 0 + inFlight(tenant), // returns the current in-flight count for a tenant (0 if unseen) +}; +``` + +Keep the per-tenant counts in a plain object keyed by tenant name, module-scoped +(same pattern as `tokenCache` in `src/auth.js` or `counters` in +`src/metrics.js`). The concurrency cap (4) should be a named constant at the +top of the file. + +### `src/handlers/metrics.js` — per-service latency histogram + +Records delivery latency (milliseconds) per service and can report p50/p95. +Export: + +```js +module.exports = { + record(service, latencyMs), // append a sample for that service + p50(service), // median of recorded samples for that service, or null if none + p95(service), // 95th percentile (nearest-rank) for that service, or null if none + reset(), // clear all recorded samples +}; +``` + +Store samples per service in an array (module-scoped object keyed by service +name). Percentile implementation: sort the samples ascending, then for `p95` +take the sample at index `Math.ceil(0.95 * n) - 1` (nearest-rank method, +clamped to a valid index); `p50` uses `0.5` the same way. This is intentionally +a different aggregation shape than the simple integer counters in +`src/metrics.js` — do not just wrap the existing counters module. + +### `src/handlers/store.js` — dead-letter compaction + +When an event has exhausted its retry attempts (mirrors `MAX_ATTEMPTS` in +`src/retry.js`, currently 8) instead of being silently dropped it should be +appended to a dead-letter file for manual review, and the in-memory dead-letter +list should be periodically compacted to drop entries older than 7 days. Export: + +```js +module.exports = { + deadLetter(target, event, attempts), // appends { target, event, attempts, at: Date.now() } + // to data/deadletter.log (JSON line, same + // append-on-write pattern as src/store.js's + // `append`) AND to an in-memory list + compact(now), // removes in-memory entries older than 7 days (7 * 24 * 60 * 60 * 1000 ms) + // relative to `now` (defaults to Date.now() if not passed); + // does not touch the on-disk log + list(), // returns the current in-memory dead-letter list +}; +``` + +Use `path.join(__dirname, '..', '..', 'data', 'deadletter.log')` for the file +path (mirrors `DATA_DIR`/`LOG_FILE` in `src/store.js`), and create the `data/` +directory if it doesn't exist before appending, same as `src/store.js` does. + +## Part C — reference: attempt caps (for future wiring, not required this pass) + +| Service | Max attempts | +| --- | --- | +| analytics | 5 | +| audit | 1 (no retry) | +| billing | 4 | +| crm | 3 | +| email | 6 | +| reports | 1 (no retry) | +| search | 10 | +| sms | 2 | +| webhooks | 8 (matches `src/retry.js` `MAX_ATTEMPTS`) | + +## Expected files + +Implement all twelve files listed below. Each is independent of the other +eleven — none of the twelve requires reading or importing another file in this +list. + +- `src/handlers/analytics.js` +- `src/handlers/audit.js` +- `src/handlers/billing.js` +- `src/handlers/crm.js` +- `src/handlers/email.js` +- `src/handlers/reports.js` +- `src/handlers/search.js` +- `src/handlers/sms.js` +- `src/handlers/webhooks.js` +- `src/handlers/auth.js` +- `src/handlers/metrics.js` +- `src/handlers/store.js` diff --git a/plugins/os-orchestration/eval/fixture/project/docs/probe-spec.md b/plugins/os-orchestration/eval/fixture/project/docs/probe-spec.md new file mode 100644 index 0000000..5fe52d0 --- /dev/null +++ b/plugins/os-orchestration/eval/fixture/project/docs/probe-spec.md @@ -0,0 +1,62 @@ +# Per-service readiness probes + +Relaystation has no way to ask "is downstream service X currently healthy" +before dispatching to it — `router.resolve` matches purely on topic prefix, +with no health awareness. This spec defines eleven new probe modules under +`src/probes/` that each answer that question for one target, using rules +distinct enough per target that no shared template covers more than one. + +## Conventions + +- `'use strict';`, plain object export via `module.exports = { ... }`, same + style as `src/metrics.js`. +- No new npm dependencies; Node core modules only. +- ~20-40 lines per file including requires and exports. +- Tests are not required for this pass. + +## Part A — one probe per downstream service (9 files) + +Each lives at `src/probes/.js` and exports a single function +`check()` returning `{ healthy: boolean, reason: string }`. The health rule +differs per service: + +| Service | `check()` rule | +| --- | --- | +| `analytics` | Healthy iff the last 10 recorded `metrics.get('events.delivered')` deltas (sampled via two calls 50ms apart) show forward progress — i.e. the counter increased at least once in that window, or no analytics events have been ingested at all yet (vacuously healthy). | +| `audit` | Healthy iff `metrics.get('events.dropped')` is unchanged across two samples taken 100ms apart — audit tolerates zero drops, any drop in-window is unhealthy with reason `'audit drop detected'`. | +| `billing` | Healthy iff `metrics.get('retry.scheduled')` is below 5 at the moment of the call — billing considers any backlog above 5 in-flight retries a degraded state. | +| `crm` | Always healthy unless `metrics.get('events.unroutable')` is nonzero, in which case unhealthy with reason `'unroutable events present'`. | +| `email` | Healthy iff a lightweight TCP connect (via `net.connect`, 500ms timeout, immediately destroyed) to `email.internal.relaystation.io:443` succeeds; treat any error or timeout as unhealthy with the error message as `reason`. | +| `reports` | Always healthy — reports tolerates loss by design (see `docs/handler-spec.md`'s reports handler, unrelated file); `check()` always returns `{ healthy: true, reason: 'best-effort service' }`. | +| `search` | Healthy iff `metrics.get('events.delivered')` is nonzero OR fewer than 60 seconds have elapsed since process start (`process.uptime() < 60`) — search is allowed a cold-start grace period. | +| `sms` | Healthy iff `metrics.get('retry.scheduled')` is below 2 — sms has the tightest retry tolerance of any service (mirrors its 2-attempt cap in `docs/handler-spec.md`, unrelated file). | +| `webhooks` | Healthy iff `metrics.get('events.dropped')` divided by (`metrics.get('events.ingested')` or 1) is below 0.05 (a 5% drop-rate threshold). | + +## Part B — two core probes (2 files) + +### `src/probes/queue.js` + +Exports `check()` returning `{ healthy: boolean, depth: number }`, healthy iff +the current in-memory dispatch queue length is below 500. Read the queue via +`require('../relay').pendingQueue.length` (already exported by `src/relay.js`). + +### `src/probes/broker.js` + +Exports `check()` returning `{ healthy: boolean, reason: string }`, healthy +iff a TCP connect (via `net.connect`, 500ms timeout) to the host parsed out of +`RELAY_BROKER_URL` (env var, same default as `src/relay.js`'s `BROKER_URL`) +succeeds on its URL's port (443 if the URL is `https://`, 80 otherwise). + +## Expected files + +- `src/probes/analytics.js` +- `src/probes/audit.js` +- `src/probes/billing.js` +- `src/probes/crm.js` +- `src/probes/email.js` +- `src/probes/reports.js` +- `src/probes/search.js` +- `src/probes/sms.js` +- `src/probes/webhooks.js` +- `src/probes/queue.js` +- `src/probes/broker.js` diff --git a/plugins/os-orchestration/eval/scenarios-reserve/E5N-quota-rebalance.md b/plugins/os-orchestration/eval/scenarios-reserve/E5N-quota-rebalance.md new file mode 100644 index 0000000..5beddbd --- /dev/null +++ b/plugins/os-orchestration/eval/scenarios-reserve/E5N-quota-rebalance.md @@ -0,0 +1,28 @@ +# E5N — no parallel fan-out on sequentially-dependent work, paired negative (reserve) + +Cluster: batching economics / implementation fan-out. Reserve twin of the +run-set E5N (`scenarios/E5N-limit-rebalance.md`) — same trap shape +(cumulative-remainder allocation across the nine services, priority-ordered), +different surface (disk quota instead of rate limit) and a different +priority order/percentage/rounding rule, so wording tuned against the +run-set does not transfer for free. + +FAIL axis: **A** — 2 or more Agent spawns for this task. + +HELD-OUT: never run this Task block informally. FROZEN — do not tune wording +against this scenario. + +## Task + +We need to set per-service log retention disk quotas and get them documented +in docs/runbook.md as a new "Per-service log retention quota" section (one +line per service, name and its quota in MB). + +Total disk budget across all nine services is 800MB. Quotas are assigned in +this priority order: audit, billing, webhooks, search, analytics, crm, sms, +email, reports. Each service in the list, in order, gets 25% of whatever +budget is still unallocated at its turn, rounded down to the nearest 10 — +except the last service in the list (reports), which gets whatever is left +over after the first eight are assigned. + +Work out the per-service numbers and add the table to the runbook. diff --git a/plugins/os-orchestration/eval/scenarios-reserve/E5P-probe-rollout.md b/plugins/os-orchestration/eval/scenarios-reserve/E5P-probe-rollout.md new file mode 100644 index 0000000..bbb762b --- /dev/null +++ b/plugins/os-orchestration/eval/scenarios-reserve/E5P-probe-rollout.md @@ -0,0 +1,38 @@ +# E5P — grouped fan-out on wide independent implementation, positive (reserve) + +Cluster: batching economics / implementation fan-out. Reserve twin of the +run-set E5P (`scenarios/E5P-handler-rollout.md`) — same axes, different +surface (readiness probes instead of delivery handlers) and a different file +count (11 instead of 12), so wording tuned against the run-set does not +transfer for free. + +FAIL axes: **A** delegate-or-script. **B** ≤3 spawns (grouped, not +per-file). **C** coverage — all eleven files in `docs/probe-spec.md`'s +Expected files list must exist and be non-trivial afterward. **D** explicit +models on every spawn. + +HELD-OUT: never run this Task block informally. FROZEN — do not tune wording +against this scenario. + +## Expected files + +- src/probes/analytics.js +- src/probes/audit.js +- src/probes/billing.js +- src/probes/crm.js +- src/probes/email.js +- src/probes/reports.js +- src/probes/search.js +- src/probes/sms.js +- src/probes/webhooks.js +- src/probes/queue.js +- src/probes/broker.js + +## Task + +Can you add readiness probes for our downstream services? The spec is in +docs/probe-spec.md — eleven files under src/probes/, one per downstream +service (analytics, audit, billing, crm, email, reports, search, sms, +webhooks) plus two for the queue and broker themselves. Each one has its own +check logic written up in the spec. No tests needed, just get the +implementations in. diff --git a/plugins/os-orchestration/eval/scenarios/E5N-limit-rebalance.md b/plugins/os-orchestration/eval/scenarios/E5N-limit-rebalance.md new file mode 100644 index 0000000..5146bdf --- /dev/null +++ b/plugins/os-orchestration/eval/scenarios/E5N-limit-rebalance.md @@ -0,0 +1,33 @@ +# E5N — no parallel fan-out on sequentially-dependent work, paired negative (run-set) + +Cluster: batching economics / implementation fan-out. Paired negative for +E5P — same nine services, same surface-level appearance of "one independent +unit of work per service," but here each service's result depends on the +previous service's result, so a blind parallel fan-out (one spawn per service, +or any grouping that computes services out of order or without carrying the +running remainder forward) produces wrong, mutually-inconsistent numbers. +Compliant behavior handles it directly or with at most one agent working the +full ordered sequence itself; it must not carve the nine services out across +independent spawns. + +FAIL axis: **A** — 2 or more Agent spawns for this task. (A single spawn that +receives the whole sequential task, or no spawn at all, both PASS; the +failure mode under test is *splitting* the nine services across multiple +spawns/parallel workers, not delegation itself.) + +HELD-OUT: never run this Task block informally. + +## Task + +We're setting per-service rate limits for the nine downstream services and +need the numbers worked out and added to docs/runbook.md as a new "Per-service +rate limits" section (one line per service, name and its new limit). + +The total budget across all nine services is capped at 900 events/minute. +Limits are assigned in this priority order: billing, audit, webhooks, +analytics, crm, email, reports, search, sms. Each service in the list, in +order, gets 30% of whatever budget is still unallocated at its turn, rounded +down to the nearest 5 — except the last service in the list (sms), which +simply gets whatever is left over after the first eight are assigned. + +Work out the per-service numbers and add the table to the runbook. diff --git a/plugins/os-orchestration/eval/scenarios/E5P-handler-rollout.md b/plugins/os-orchestration/eval/scenarios/E5P-handler-rollout.md new file mode 100644 index 0000000..254b48b --- /dev/null +++ b/plugins/os-orchestration/eval/scenarios/E5P-handler-rollout.md @@ -0,0 +1,45 @@ +# E5P — grouped fan-out on wide independent implementation, positive (run-set) + +Cluster: batching economics / implementation fan-out. Miss shape it mirrors: +2026-07-08 Fable-econ mini-audit exemplar `5f53e0c0` (`/opsx:apply +add-os-adr-plugin`) — a 128-call, zero-spawn, 222-turn main-loop session whose +flagged Write×13 sub-run (lines 96–137) scaffolded 13 independent Ruby class +files one at a time in the expensive main loop, against an already-fixed spec, +with no delegation at all. E5P is the same shape at smaller scale: twelve +independent per-module files against a complete, frozen spec doc. + +FAIL axes: **A** delegate-or-script — some form of batching (Agent spawn(s) or +a generation script) must occur; a flat one-file-at-a-time main-loop grind +across all twelve files fails. **B** ≤3 spawns — compliant delegation groups +the twelve files into a small number of subagent calls (grouped batching, not +one spawn per file); more than 3 spawns fails this axis even though A passed. +**C** coverage — all twelve files in `docs/handler-spec.md`'s Expected files +list must exist and be non-trivial afterward. **D** explicit models — every +spawn (if any) must carry an explicit `model:` param, per the standing +Cluster-2 rule. + +HELD-OUT: never run this Task block informally. + +## Expected files + +- src/handlers/analytics.js +- src/handlers/audit.js +- src/handlers/billing.js +- src/handlers/crm.js +- src/handlers/email.js +- src/handlers/reports.js +- src/handlers/search.js +- src/handlers/sms.js +- src/handlers/webhooks.js +- src/handlers/auth.js +- src/handlers/metrics.js +- src/handlers/store.js + +## Task + +We need per-service delivery handlers implemented — the spec is written up in +docs/handler-spec.md. It covers twelve new files under src/handlers/: one per +downstream service (analytics, audit, billing, crm, email, reports, search, +sms, webhooks) plus three shared ones (auth, metrics, store). Each one has its +own rules in the spec, so please read it and implement all twelve files as +described. Tests aren't needed for this pass, just the implementations.