os-orchestration: WS4 eval extension — econ axes, E5 batching pair, fable column, pre-registered E2P/E3P criterion redesign
- extract: main-loop/sidechain output-token accounting (subagent transcripts persist at <projects>/<flat>/<session-id>/subagents/agent-*.jsonl — the mini-audit's instrumentation gap, now solved) + SendMessage round counting - check: econ info on every row (mltok/sctok/mlshare/prebytes/spawns/maxrounds); E2P v2 (scripted-direct branch + services task-complete); E3P v2 (74KB S7 whole-session ingestion anchor + mechanical root-cause concepts, delegation no longer required); E5P/E5N arms (grouped-delegation vs parallel-fanout, ## Expected files coverage convention) - run: fable → claude-fable-5 alias - scenarios: E5P-handler-rollout + E5N-limit-rebalance (+ frozen reserve twins, authored unread by the orchestrating session) + handler-spec/probe-spec fixture docs - self-test 32/32 model-free; ORCHESTRATION.v3-draft.md parked (NOT shipped — Step-3 baseline must run against v2); H3 threshold pre-registered in plan Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
jared
parent
42860d06d1
commit
b226601bea
|
|
@ -0,0 +1,175 @@
|
||||||
|
# Fable-5 main-loop orchestration economics — mini-audit (2026-07-08)
|
||||||
|
|
||||||
|
**Model**: this audit was performed by `claude-sonnet-5`. All audited transcripts carry
|
||||||
|
`"model":"claude-fable-5"` on non-sidechain assistant messages (i.e. Fable 5 is the main-loop
|
||||||
|
model in every session below).
|
||||||
|
|
||||||
|
## Method
|
||||||
|
|
||||||
|
1. Scanned all `~/.claude/projects/*/*.jsonl` files modified in the last 10 days
|
||||||
|
(`scan.rb`), filtered to sessions with ≥20 non-sidechain assistant turns and at least one
|
||||||
|
`claude-fable-5` main-loop message, excluded the live session
|
||||||
|
(`ca17e6d2-6132-4681-8a02-8a7523cb5660`). 21 candidates found, almost all in `cc-os`; picked
|
||||||
|
6 for project diversity: 3 `cc-os`, 2 `servers`, 1 `servers/desktop`, all ≥20 turns (24–255).
|
||||||
|
2. Ran the existing fact-sheet extractor
|
||||||
|
(`plugins/os-orchestration/audit/bin/extract`) on each for spawn count/models,
|
||||||
|
pre-/post-spawn tool segmentation, and its heuristic same-tool-run flags.
|
||||||
|
3. Wrote two throwaway Ruby scripts in scratchpad (`tokens.rb`, `bashcmds.rb`,
|
||||||
|
`firstprompt.rb`) to sum `usage.output_tokens` per session (main-loop vs sidechain), track
|
||||||
|
max/final context size, and find the longest unbroken run of main-loop tool calls with no
|
||||||
|
`Agent` spawn in between, then classified each run by reading the actual tool inputs
|
||||||
|
(Bash commands / file targets) in that range.
|
||||||
|
|
||||||
|
**Environment gotcha (noted for future audits):** in zsh, a variable named `path` is linked to
|
||||||
|
`$PATH`; `path=$(find ...)` silently corrupted `$PATH` mid-script and caused unrelated `ruby: not
|
||||||
|
found` (exit 127) failures. Renamed to `tpath`. Not a data issue, just a shell trap.
|
||||||
|
|
||||||
|
**Hard measurement limitation:** none of the 6 transcripts contain any `"isSidechain":true`
|
||||||
|
lines, even in the two sessions that used the `Agent` tool (9f45afcc, 5 spawns; df12b180, 1
|
||||||
|
spawn). Subagent execution in this environment/version is logged only as a `tool_use`/
|
||||||
|
`toolUseResult` pair (with `agentId`/`resolvedModel` metadata) plus an eventual
|
||||||
|
`<task-notification>` result — no subagent transcript is persisted into the parent `.jsonl`. So
|
||||||
|
**subagent output-token spend is not recoverable from these files at all.** The only proxy
|
||||||
|
available is the char length of the `tool_result`/notification payload returned to the main
|
||||||
|
loop, converted at ~4 chars/token. All "% main-loop" figures below use that proxy where spawns
|
||||||
|
occurred, and are explicitly not real subagent token counts (real subagent spend is very likely
|
||||||
|
much higher than the proxy, since it only reflects what was returned, not what the subagent
|
||||||
|
itself burned getting there).
|
||||||
|
|
||||||
|
> **Addendum (2026-07-08, WS4 Step 2):** the limitation above is narrower than stated —
|
||||||
|
> subagent spend is absent from the *parent* transcript, but each subagent's full transcript
|
||||||
|
> (with per-message `usage` blocks) persists separately at
|
||||||
|
> `~/.claude/projects/<flattened-cwd>/<session-id>/subagents/agent-<id>.jsonl` (verified for
|
||||||
|
> both interactive and headless sessions). Real main-loop-share measurement is therefore
|
||||||
|
> possible; the eval harness A-econ axis uses it. The proxy figures below were NOT recomputed.
|
||||||
|
> See vault note [[claude-code-subagent-transcripts-and-token-accounting]].
|
||||||
|
|
||||||
|
## Per-session detail
|
||||||
|
|
||||||
|
### 1. `9f45afcc` — cc-os — cache-refresh tooling + WS2 autoresearch prep
|
||||||
|
- 255 turns, 25 human prompts, 850 jsonl lines, 2026-07-04 16:17–18:52
|
||||||
|
- 5 Agent spawns, all `general-purpose`, `model` param `sonnet`×4/`haiku`×1, **resolved model
|
||||||
|
`claude-haiku-4-5` in all 5 cases** (param mismatch: 4 of 5 requested sonnet, all resolved
|
||||||
|
haiku)
|
||||||
|
- main-loop output tokens: **237,679** (255 msgs); proxy subagent output: 13,575 chars via
|
||||||
|
result/notification payloads (≈3.4K tokens) → **main-loop ≈98.6% of proxy-measured output**
|
||||||
|
- pre-spawn segment: 19 calls / 23,658 bytes read (Bash 14, Read 4, Edit 1)
|
||||||
|
- longest direct run: **48 calls** (after spawn 3: ToolSearch 2, SendMessage 1, Bash 22, Edit 12,
|
||||||
|
Skill 2, Read 5, Write 3, TaskStop 1) — mixed: driving/monitoring the background eval-grid
|
||||||
|
agent (SendMessage/TaskStop/ToolSearch) interleaved with direct mechanical plugin-cache-refresh
|
||||||
|
edits. Not a clean single-purpose run.
|
||||||
|
|
||||||
|
### 2. `eba26343` — cc-os — WS2 os-vault wording loop (`/autoresearch`)
|
||||||
|
- 223 turns, 2 human prompts (127 total `type:user` lines incl. tool results/notifications), 583
|
||||||
|
jsonl lines, 2026-07-07 16:48–18:01
|
||||||
|
- **0 Agent spawns**
|
||||||
|
- main-loop output tokens: **216,019** (223 msgs) — 100% of measured output (no spawns to split
|
||||||
|
against)
|
||||||
|
- whole-session tool profile: 119 calls / 119,587 bytes (Read 17, Bash 64, Skill 1, ToolSearch 1,
|
||||||
|
TaskCreate 5, TaskUpdate 9, Edit 20, Write 2)
|
||||||
|
- longest direct run: **119 calls = the entire session** (no spawn boundary at all). Sampled the
|
||||||
|
64 Bash commands directly: this is a scripted headless-runner loop — `bin/run P1/P2/P3...` /
|
||||||
|
`bin/check` invocations against `eval/scenarios/`, `sleep N; wc -l results.tsv`
|
||||||
|
polling of background task output, `awk`/`sort` aggregation of `results.tsv`, and git
|
||||||
|
commit/push at each iteration boundary. **Classify as justified scripted-bulk-op / polling
|
||||||
|
work**, not a delegation miss — matches the ORCHESTRATION.md scripted-bulk-edit carve-out; a
|
||||||
|
subagent adds no value polling a script's own stdout.
|
||||||
|
|
||||||
|
### 3. `5f53e0c0` — cc-os — `/opsx:apply add-os-adr-plugin`
|
||||||
|
- 222 turns, 4 human prompts, 506 jsonl lines, 2026-07-03 17:15–17:43
|
||||||
|
- **0 Agent spawns**
|
||||||
|
- main-loop output tokens: **314,899** (222 msgs, highest of the 6) — 100% of measured output
|
||||||
|
- whole-session tool profile: 128 calls / 129,924 bytes (Bash 39, Read 16, Write 35, Edit 38)
|
||||||
|
- longest direct run: **128 calls = the entire session**. Contains three flagged sub-runs from
|
||||||
|
the extractor: Read×10 (design docs, lines 27–52 — reasonable orientation), **Write×13 (lines
|
||||||
|
96–137, 13 distinct new files: `lib/adr.rb`, `lib/adr/record.rb`, `repository.rb`,
|
||||||
|
`index.rb`, `template.rb`, `bin/adr-new`, …)**, and Edit×10 (lines 326–353, fixing 5 files
|
||||||
|
post-review). **Genuine miss candidate**: the Write×13 run is scaffolding independent Ruby
|
||||||
|
class files against an *already-fixed* `design.md` (read in the first 10 calls) — a
|
||||||
|
parallelizable implementation task that ran entirely in the main loop and is a large share of
|
||||||
|
this session's 315K output tokens.
|
||||||
|
|
||||||
|
### 4. `3fc7bb8c` — servers — backup verification + vault write
|
||||||
|
- 90 turns, 5 human prompts, 236 jsonl lines, 2026-07-04 17:20–18:04
|
||||||
|
- **0 Agent spawns**
|
||||||
|
- main-loop output tokens: **63,607**
|
||||||
|
- whole-session tool profile: 44 calls / 32,250 bytes (Read 3, Bash 27, Write 4, ToolSearch 2,
|
||||||
|
ExitPlanMode 1, Edit 6, Skill 1)
|
||||||
|
- longest direct run: **44 calls = whole session** — interactive backup-system investigation
|
||||||
|
("Are we confident backups will work as expected now?") ending in a vault write. Judgment-
|
||||||
|
heavy, sequential-dependent (each Bash result determines the next check). **Justified direct
|
||||||
|
work**, no clean miss shape.
|
||||||
|
|
||||||
|
### 5. `7bc0dda4` — servers — Proxmox HAOS install troubleshooting
|
||||||
|
- 82 turns, 15 human prompts (highest interaction density of the 6), 249 jsonl lines,
|
||||||
|
2026-07-03 19:49–20:42
|
||||||
|
- **0 Agent spawns**
|
||||||
|
- main-loop output tokens: **39,513**
|
||||||
|
- whole-session tool profile: 39 calls / 6,124 bytes (Bash 27, Edit 8, Read 2, Skill 1, Write 1)
|
||||||
|
- longest direct run: **39 calls = whole session** — live interactive install/debug loop (disk
|
||||||
|
space checks, image download, phase-2 error triage against user-supplied screenshots).
|
||||||
|
**Justified direct work**: 15 human turns in 82 total means the human is in the loop almost
|
||||||
|
every other turn; delegation overhead would exceed any savings.
|
||||||
|
|
||||||
|
### 6. `df12b180` — servers/desktop — Remote Control OAuth scope troubleshooting
|
||||||
|
- 24 turns, 6 human prompts, 90 jsonl lines, 2026-07-06 23:13–23:35
|
||||||
|
- 1 Agent spawn (`claude-code-guide`, `model` param `sonnet`, resolved `claude-sonnet-5`,
|
||||||
|
foreground, prompt 1,008 chars → result 5,689 chars)
|
||||||
|
- main-loop output tokens: **24,051**; proxy subagent output ≈1,422 tokens (5,689 chars/4) →
|
||||||
|
**main-loop ≈94.4% of proxy-measured output**
|
||||||
|
- pre-spawn: 4 Bash calls / 3,154 bytes (credential/scope investigation)
|
||||||
|
- post-spawn: 3 calls (Read 1, Write 2 — writing up the answer)
|
||||||
|
- longest direct run: 4 calls (the pre-spawn investigation). **Good delegation example**: a
|
||||||
|
bounded doc-lookup ("Remote Control usage") was correctly routed to the docs-specialist
|
||||||
|
subagent instead of the main loop reading source/docs itself.
|
||||||
|
|
||||||
|
## Summary table
|
||||||
|
|
||||||
|
| session | project | turns | spawns | main-loop out tok | % main-loop (proxy, see caveat) | pre-spawn calls/bytes | longest direct run | classification |
|
||||||
|
|---|---|---|---|---|---|---|---|---|
|
||||||
|
| 9f45afcc | cc-os | 255 | 5 | 237,679 | ~98.6% | 19 / 23,658 | 48 | mixed: agent-monitoring + mechanical cache-refresh edits |
|
||||||
|
| eba26343 | cc-os | 223 | 0 | 216,019 | 100% | 119 / 119,587 (whole session) | 119 (whole session) | scripted headless-eval polling loop — justified |
|
||||||
|
| 5f53e0c0 | cc-os | 222 | 0 | 314,899 | 100% | 128 / 129,924 (whole session) | 128 (whole session) | OpenSpec apply — Write×13 independent files = **miss candidate** |
|
||||||
|
| 3fc7bb8c | servers | 90 | 0 | 63,607 | 100% | 44 / 32,250 (whole session) | 44 (whole session) | interactive investigation — justified |
|
||||||
|
| 7bc0dda4 | servers | 82 | 0 | 39,513 | 100% | 39 / 6,124 (whole session) | 39 (whole session) | interactive troubleshooting (15 human turns) — justified |
|
||||||
|
| df12b180 | servers/desktop | 24 | 1 | 24,051 | ~94.4% | 4 / 3,154 | 4 | pre-spawn investigation, correct delegation of doc lookup |
|
||||||
|
|
||||||
|
## Caveats
|
||||||
|
|
||||||
|
- 4 of the 6 sessions are from `cc-os` (matches the actual population: 19 of 21 candidate
|
||||||
|
sessions in the last 10 days were `cc-os`) — the sample is not representative of all-client
|
||||||
|
usage, just of what exists in the last 10 days of transcripts.
|
||||||
|
- "% main-loop" is a **proxy** built only from what subagents returned to the main loop
|
||||||
|
(tool_result/notification char count ÷4), not measured subagent token spend — real subagent
|
||||||
|
cost is invisible in every transcript examined (0/6 have `isSidechain:true` lines). Any
|
||||||
|
economics conclusion about spawn cost-effectiveness needs a different data source (e.g. Task
|
||||||
|
output files under `tasks/`, if retained) — not attempted here.
|
||||||
|
- "Longest direct run" for 4 of 6 sessions equals the *entire session* because zero spawns
|
||||||
|
occurred; this collapses "pre-spawn segment" and "longest run" into the same number for those
|
||||||
|
rows — noted in the table rather than hidden.
|
||||||
|
- Miss/justified classification is a single-auditor read of tool-call sequences, not a rubric
|
||||||
|
score; treat as a directional signal for the eval plan, not a verdict.
|
||||||
|
|
||||||
|
## Miss-shape summary
|
||||||
|
|
||||||
|
- **The only clean miss candidate found is `5f53e0c0`'s Write×13 run**: scaffolding independent
|
||||||
|
Ruby class files against an already-fixed design doc, done entirely in the (most expensive)
|
||||||
|
main loop across a 222-turn, 314,899-output-token session with zero spawns.
|
||||||
|
- **Scripted/polling loops are not misses even when very long** (`eba26343`, 119 consecutive
|
||||||
|
direct calls): driving `bin/run`/`sleep`/`wc -l results.tsv` from the main loop is the correct
|
||||||
|
shape per the existing scripted-bulk-op carve-out — a subagent adds no value polling a script's
|
||||||
|
own output.
|
||||||
|
- **Interactive/judgment-heavy sessions correctly stayed direct** (`3fc7bb8c`, `7bc0dda4`): both
|
||||||
|
have no spawns and no flagged miss runs; `7bc0dda4`'s 15 human prompts in 82 turns make
|
||||||
|
delegation overhead a net loss.
|
||||||
|
- **Subagent economics are structurally unmeasurable from transcripts as configured**: 0 of 6
|
||||||
|
sessions logged sidechain entries even where Agent spawned; only a lossy char-count proxy
|
||||||
|
exists via task-notification payloads. This is itself an eval-plan-relevant finding — any eval
|
||||||
|
claiming to measure delegated-vs-direct token spend needs an instrumentation change (capture
|
||||||
|
Task output, or read `tasks/*.output` files) before it can trust a token-economics number.
|
||||||
|
- **Where spawns did happen, the main loop still did substantial direct work afterward**
|
||||||
|
(`9f45afcc`: 43–48-call runs after spawns 3 and 5) rather than delegating a second time —
|
||||||
|
worth checking in future audits whether that's justified (synthesis/judgment) or another miss
|
||||||
|
shape.
|
||||||
|
- **No over-delegation observed anywhere** — zero ≤2-call trivial ops routed to a subagent in any
|
||||||
|
of the 6 sessions.
|
||||||
|
|
@ -0,0 +1,166 @@
|
||||||
|
# WS4 — Orchestration economics: ORCHESTRATION.md v3 + eval extension + wording loop
|
||||||
|
|
||||||
|
_Created: 2026-07-08. Status: PLANNED (mini-audit complete; steps 1–5 not started)._
|
||||||
|
_Prereq reading before executing: `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md`,
|
||||||
|
vault notes [[orchestration-prompting-claude-5-era]], [[eval-methodology-ladder]],
|
||||||
|
[[os-orchestration-eval-baseline-grid-results]]._
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
The Fable-5 main loop does too much multi-call work itself. ORCHESTRATION.md was calibrated
|
||||||
|
against over-delegation (Opus 4.6 era) under sonnet main loops where direct work was cheap;
|
||||||
|
its negative-first threshold ("Delegate only when…", "≤2-tool-call ops direct") is read
|
||||||
|
literally by Fable and licenses unbounded direct work whenever the task isn't parallelizable/
|
||||||
|
many-files/isolated-context. Goals (user, 2026-07-08): max quality at reasonable cost;
|
||||||
|
route work to cheaper tiers whenever reasonable; group delegated tasks so agents finish in
|
||||||
|
few rounds (fewer spawns → less per-spawn system-prompt tax).
|
||||||
|
|
||||||
|
Evidence base: Anthropic model-page guidance summarized in vault note
|
||||||
|
[[orchestration-prompting-claude-5-era]] (fetched 2026-07-08); WS1 audit
|
||||||
|
(`docs/orchestration-audit/2026-07-06-findings.md`); Fable economics mini-audit
|
||||||
|
(`docs/orchestration-audit/2026-07-08-fable-econ-miniaudit.md`) — the H3 baseline.
|
||||||
|
|
||||||
|
### Mini-audit headline (2026-07-08, 6 Fable sessions, ran before drafting wording)
|
||||||
|
|
||||||
|
- **The dominant miss shape is implementation fan-out, not investigation:** the one clean
|
||||||
|
miss is an OpenSpec-apply session (`5f53e0c0`) that wrote 13 independent Ruby files
|
||||||
|
entirely in the main loop — 314,899 main-loop output tokens, 0 spawns. E5P should mirror
|
||||||
|
this shape (write-N-independent-files implementation), not a log-review shape.
|
||||||
|
- Long direct runs were mostly JUSTIFIED (scripted eval-polling loop of 119 calls;
|
||||||
|
interactive troubleshooting with 15 human turns) — confirming the non-goal: cost-keyed
|
||||||
|
thresholds, not "always delegate". Zero over-delegation anywhere.
|
||||||
|
- Secondary watch item: after a session's spawns complete, the main loop resumes long
|
||||||
|
direct runs (43–48 calls, `9f45afcc`) instead of delegating again — mid-session threshold
|
||||||
|
decay.
|
||||||
|
- **Instrumentation gap (affects A-econ):** main transcripts contain NO `isSidechain:true`
|
||||||
|
entries even in sessions with spawns — subagent token spend is unrecoverable from the
|
||||||
|
transcript alone. The eval checker must source subagent spend from harness-side artifacts
|
||||||
|
(the eval runner's own task/agent output files in the sandbox), or A-econ degrades to a
|
||||||
|
main-loop-absolute-tokens metric (still usable: baseline sessions run 24k–315k main-loop
|
||||||
|
output tokens). Resolve this during Step 2 design; do not silently ship the char-count
|
||||||
|
proxy the mini-audit used.
|
||||||
|
|
||||||
|
## Step 1 — ORCHESTRATION.md v3 (wording surface for the loop)
|
||||||
|
|
||||||
|
Five changes; keep explicit-`model:`, self-report, and don't-re-cover-ground rules unchanged:
|
||||||
|
|
||||||
|
1. **Cost-asymmetry + tier-conditional threshold.** New opening rule: main-loop tokens are
|
||||||
|
the most expensive in the session; the more capable the main-loop model, the lower the
|
||||||
|
delegation threshold. Any investigation or mechanical sequence beyond ~3 calls that does
|
||||||
|
not need main-loop judgment → delegate down-tier, even when sequential. (Models know their
|
||||||
|
own model ID — tier-conditional phrasing is implementable; E1 canary verified self-report.)
|
||||||
|
2. **Symmetric triggers.** Replace "Delegate only when…" with paired lists:
|
||||||
|
"delegate when …" / "work directly when …" (keep the scripted-bulk-edit carve-out and
|
||||||
|
≤2-call rule on the direct side). Rationale: literal instruction-following on 5-era
|
||||||
|
models; the canonical best-practices snippet pairs both directions.
|
||||||
|
3. **Batching rule.** Plan the fan-out before the first spawn; group related subtasks
|
||||||
|
(~5–8 similar items) into one agent prompt with a return schema so the agent completes in
|
||||||
|
one round; follow-ups go to the live agent (SendMessage), not a fresh spawn.
|
||||||
|
4. **Async rule.** Delegate independent subtasks and keep working while they run; intervene
|
||||||
|
only if a subagent goes off track (Fable-page recommended pattern).
|
||||||
|
5. **Effort dial (where exposed).** Workflow `agent()` calls: mechanical stages
|
||||||
|
`effort: low`; hard verify/judge stages high/xhigh.
|
||||||
|
|
||||||
|
Draft AFTER reading the mini-audit's miss shapes — wording should name the observed shapes,
|
||||||
|
not hypothetical ones (Eval B lesson: mechanical triggers phrased at the observed
|
||||||
|
precondition).
|
||||||
|
|
||||||
|
## Step 2 — Eval harness extension (`plugins/os-orchestration/eval/`)
|
||||||
|
|
||||||
|
All extensions are new frozen surfaces; existing run-set scenarios E1–E3 stay untouched.
|
||||||
|
Reserve stays frozen. Follow [[eval-methodology-ladder]]: paired positives/negatives,
|
||||||
|
run-set + reserve twins authored together, thresholds anchored independently of any grid
|
||||||
|
already run (pre-registered criterion-redesign rule from the 2026-07-06 baseline).
|
||||||
|
|
||||||
|
### New metric axes (extractor + checker)
|
||||||
|
|
||||||
|
- **A-econ (main-loop token share):** main-loop assistant output tokens ÷ total session
|
||||||
|
output tokens (main + sidechains). Extractor already parses transcripts; add the sum.
|
||||||
|
- **A-prespawn:** tool calls + tool_result bytes before first spawn (extractor segment
|
||||||
|
exists from E3 work).
|
||||||
|
- **A-batch:** spawns per task and rounds per agent (Agent tool_use count; SendMessage
|
||||||
|
reuse counts as same-agent round, not new spawn).
|
||||||
|
- Redesign E2P/E3P delegate-at-all axes per the pre-registered note: PASS may be
|
||||||
|
"delegated" OR "superior scripted direct strategy" — the checker must distinguish
|
||||||
|
script-driven bulk ops (few Bash calls, uniform op) from grind (per-file tool loops).
|
||||||
|
|
||||||
|
### New scenario pair E5 (batching) + reserve twins
|
||||||
|
|
||||||
|
- **E5P:** N (~10–12) similar independent items, shaped like the observed miss —
|
||||||
|
an implementation task producing N independent files in the relaystation fixture
|
||||||
|
(NOT log review). PASS = grouped delegation (≤3 spawns covering all items, grouped
|
||||||
|
prompts, one round each), FAIL = 1-agent-per-item or main-loop grind.
|
||||||
|
- **E5N (paired negative):** a task where batching would wrongly serialize genuinely
|
||||||
|
judgment-dependent items (each item's handling depends on the previous result) — PASS =
|
||||||
|
sequential handling (direct or single agent), FAIL = blind parallel fan-out.
|
||||||
|
- Author reserve twins at the same time, different surface domain; freeze both.
|
||||||
|
|
||||||
|
### Fable column
|
||||||
|
|
||||||
|
Baseline grid ran sonnet/haiku orchestrators only; the population of concern is Fable.
|
||||||
|
Add `--model fable` (headless `claude -p --model claude-fable-5`) at reduced reps
|
||||||
|
(1–2 per cell for the grid; 3 on target cells inside the loop). Cost of the eval itself is
|
||||||
|
real — use the reduced inner-loop grid discipline (target cells + one passing control).
|
||||||
|
|
||||||
|
## Step 3 — Baseline grid (pre-wording)
|
||||||
|
|
||||||
|
Run extended grid (E1–E3 + E5, econ axes) × {fable, sonnet} before touching wording.
|
||||||
|
Canary-cell the first live rep of every NEW scenario/axis and hand-verify TSV vs transcript
|
||||||
|
(count the canary). This grid is the tuning baseline; the 2026-07-08 mini-audit is the IRL
|
||||||
|
baseline for H3.
|
||||||
|
|
||||||
|
## Step 4 — Autoresearch wording loop
|
||||||
|
|
||||||
|
Loop discipline per the howto: wording-only moves (checker/fixtures/scenarios/axes frozen),
|
||||||
|
`bin/refresh-plugins` before every iteration's grid, ≥3 reps on target cells + 1 control
|
||||||
|
cell, accept only majority-of-reps improvements, verify from TSV not agent prose.
|
||||||
|
|
||||||
|
### Hypotheses (pre-registered)
|
||||||
|
|
||||||
|
- **H1 (threshold framing):** symmetric cost-framed triggers raise delegation on positive
|
||||||
|
scenarios (esp. E3P-shape sequential investigation) on fable/sonnet, while negatives stay
|
||||||
|
clean. Guard: baseline negatives were 18/18 — any negative regression rejects the
|
||||||
|
candidate regardless of positive gains.
|
||||||
|
- **H2 (batching rule):** the grouping rule reduces spawns-per-task and rounds-per-agent on
|
||||||
|
E5P without task-success regression; E5N stays PASS (no blind fan-out).
|
||||||
|
- **H3 (net economics — the actual goal):** main-loop token spend on positives drops
|
||||||
|
materially vs the Step-3 baseline with task success unchanged. Metric: main-loop share
|
||||||
|
if Step 2 solves the sidechain-instrumentation gap, else main-loop absolute output
|
||||||
|
tokens per scenario. Anchor the target threshold independently before the loop starts
|
||||||
|
(not post-hoc to whatever the loop achieves). H1/H2 are mechanisms; H3 is the verdict
|
||||||
|
axis.
|
||||||
|
|
||||||
|
**H3 threshold (pre-registered 2026-07-08, before the Step-3 grid ran):** H3 PASSes iff,
|
||||||
|
on the positive economics cells (E5P and E3P, fable column), the median main-loop
|
||||||
|
assistant output tokens across reps under the candidate wording is ≤60% of the Step-3
|
||||||
|
pre-wording baseline median for the same cell (≥40% reduction), with task-success axes
|
||||||
|
unchanged and all negatives clean. Pass/fail is keyed to main-loop **absolute** output
|
||||||
|
tokens (exists in both baselines regardless of the sidechain-instrumentation outcome);
|
||||||
|
main-loop *share* is reported additionally if Step 2 solves sidechain measurement.
|
||||||
|
Anchor rationale (independent of any grid): in the IRL miss exemplar `5f53e0c0`, the
|
||||||
|
Write×13 fan-out segment dominates a 314,899-token session — delegating the
|
||||||
|
implementation segment should remove roughly half of main-loop output; 40% is a
|
||||||
|
conservative floor of that estimate.
|
||||||
|
|
||||||
|
### Judging
|
||||||
|
|
||||||
|
Per-axis deterministic checker verdicts; per-scenario pass bars, not aggregate scores.
|
||||||
|
A wording candidate ships only if: all negatives hold, target positives improve on majority
|
||||||
|
of reps, control cell unchanged, and the full-grid confirm reproduces it. Final confirmation
|
||||||
|
on the frozen reserve (tuning against the run-set moves measurement to the reserve —
|
||||||
|
after this loop the run-set is contaminated for future measurement).
|
||||||
|
|
||||||
|
## Step 5 — Rollout + IRL re-audit
|
||||||
|
|
||||||
|
Ship winning wording (refresh caches), then re-run the Fable economics mini-audit on ~5 new
|
||||||
|
real sessions after ~2 weeks ([[eval-methodology-irl-feedback-loop]]). Compare main-loop
|
||||||
|
token share vs the 2026-07-08 baseline. Promote any new miss shapes into eval scenarios.
|
||||||
|
|
||||||
|
## Non-goals / guards
|
||||||
|
|
||||||
|
- No changes to delegation SAFETY rules (explicit model:, self-report, downgrade honesty).
|
||||||
|
- Don't over-rotate: zero over-delegation was a baseline strength (negatives 18/18) and
|
||||||
|
scripted-direct strategies were often genuinely superior — the point is cost-keyed
|
||||||
|
thresholds, not "always delegate".
|
||||||
|
- Held-out discipline: never run scenario Task blocks informally; reserve is never read
|
||||||
|
informally.
|
||||||
|
|
@ -0,0 +1,50 @@
|
||||||
|
## Session orchestration
|
||||||
|
|
||||||
|
- Main-loop tokens are the most expensive tokens in the session: every direct tool
|
||||||
|
call and every byte read into this context bills at the main-loop model's rate,
|
||||||
|
while a haiku/sonnet subagent does the same mechanical work for a fraction of it.
|
||||||
|
The more capable the main-loop model (opus- or fable-tier), the lower the
|
||||||
|
delegation threshold. The question is not "is this task big enough to delegate?"
|
||||||
|
but "does this step need main-loop judgment?" — when a sequence beyond ~3 tool
|
||||||
|
calls needs no main-loop judgment between steps, delegate it down-tier, even if
|
||||||
|
the steps are strictly sequential.
|
||||||
|
- Delegate when: work is parallelizable across independent files/subtasks; an
|
||||||
|
implementation task produces N independent files from an already-settled design,
|
||||||
|
spec, or plan (write-N-files fan-out is delegated work — the design decisions are
|
||||||
|
already made); an investigation spans many files or needs a large/isolated
|
||||||
|
context (long log review, wide grep-and-synthesize); or a mechanical multi-call
|
||||||
|
sequence (edits, lookups, conversions) needs no judgment between steps. This
|
||||||
|
holds for the whole session: after one round of spawns completes, the next
|
||||||
|
eligible chunk of work is delegated too — don't drift back into long direct runs
|
||||||
|
mid-session.
|
||||||
|
- Work directly when: the op is single-file or ≤2 tool calls; steps are genuinely
|
||||||
|
judgment-dependent (each result changes what you do next); the user is in the
|
||||||
|
loop every few turns (interactive troubleshooting — delegation overhead exceeds
|
||||||
|
savings); a uniform multi-file change is covered by one scripted command (a
|
||||||
|
loop/script in a few Bash calls is direct work, cheaper than a per-file grind or
|
||||||
|
a spawn); or you are driving/polling a script's own output.
|
||||||
|
- Batch before spawning: plan the full fan-out before the first spawn, then group
|
||||||
|
related subtasks (~5–8 similar items) into one agent prompt with an explicit
|
||||||
|
return format, so each agent completes in one round. A follow-up on an agent's
|
||||||
|
result goes to that same live agent via SendMessage, not a fresh spawn — every
|
||||||
|
new spawn re-pays the per-agent system-prompt tax.
|
||||||
|
- Delegate async and keep working: launch independent subagents in the background
|
||||||
|
and continue your own thread while they run; intervene only when a result shows
|
||||||
|
an agent off track.
|
||||||
|
- Before every `Agent` call → set `model:` explicitly in that call. An omitted
|
||||||
|
`model` silently bills the subagent at the main-loop model. Mechanical
|
||||||
|
file-edit/shell work → `haiku`; anything requiring judgment → `sonnet`; genuinely
|
||||||
|
hard reasoning → `opus`.
|
||||||
|
- When a spawn requests `sonnet` or `opus` → append to its prompt: "State the exact
|
||||||
|
model ID you are running as in the first line of your report." (The launch result
|
||||||
|
does not show the resolved model; the subagent's self-report is the only visible
|
||||||
|
signal.) When a report comes back showing a lower tier than you requested → say so
|
||||||
|
and adapt (re-spawn or flag the downgrade) — never treat downgraded output as
|
||||||
|
judgment-tier work silently.
|
||||||
|
- Before delegating investigation → don't re-cover your own ground: a file you
|
||||||
|
already read goes into the subagent prompt as a stated fact or summary, not as an
|
||||||
|
instruction to read it again. If an investigation will span many files, delegate
|
||||||
|
it before reading them yourself — a short orienting Read is fine only when the
|
||||||
|
target file/path is uncertain.
|
||||||
|
- Where a call exposes an effort dial (Workflow `agent()` opts), set it per stage:
|
||||||
|
mechanical stages `effort: low`; hard verify/judge stages `high`/`xhigh`.
|
||||||
|
|
@ -82,12 +82,13 @@ module OrchAudit
|
||||||
|
|
||||||
class AgentSpawn
|
class AgentSpawn
|
||||||
attr_reader :call, :index
|
attr_reader :call, :index
|
||||||
attr_accessor :agent_id, :resolved_model, :notification_chars
|
attr_accessor :agent_id, :resolved_model, :notification_chars, :rounds
|
||||||
|
|
||||||
def initialize(call, index)
|
def initialize(call, index)
|
||||||
@call = call
|
@call = call
|
||||||
@index = index
|
@index = index
|
||||||
@notification_chars = 0
|
@notification_chars = 0
|
||||||
|
@rounds = 1
|
||||||
end
|
end
|
||||||
|
|
||||||
def to_h
|
def to_h
|
||||||
|
|
@ -102,7 +103,8 @@ module OrchAudit
|
||||||
run_in_background: call.input["run_in_background"],
|
run_in_background: call.input["run_in_background"],
|
||||||
prompt_chars: (call.input["prompt"] || "").length,
|
prompt_chars: (call.input["prompt"] || "").length,
|
||||||
description: call.input["description"],
|
description: call.input["description"],
|
||||||
result_chars: [call.result_chars, notification_chars].max
|
result_chars: [call.result_chars, notification_chars].max,
|
||||||
|
rounds: rounds
|
||||||
}
|
}
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
@ -230,10 +232,11 @@ module OrchAudit
|
||||||
calls, agent_spawns = collect_calls(main)
|
calls, agent_spawns = collect_calls(main)
|
||||||
attach_results(main, calls)
|
attach_results(main, calls)
|
||||||
attach_agent_metadata(main, calls, agent_spawns)
|
attach_agent_metadata(main, calls, agent_spawns)
|
||||||
|
attach_rounds(main, agent_spawns)
|
||||||
|
|
||||||
orchestrator_calls = calls.values.reject { |c| c.name == "Agent" }
|
orchestrator_calls = calls.values.reject { |c| c.name == "Agent" }
|
||||||
{
|
{
|
||||||
metadata: metadata(main),
|
metadata: metadata(main, agent_spawns),
|
||||||
agent_spawns: agent_spawns.map(&:to_h),
|
agent_spawns: agent_spawns.map(&:to_h),
|
||||||
segments: segments(main).map(&:to_h),
|
segments: segments(main).map(&:to_h),
|
||||||
missed_delegation_candidates:
|
missed_delegation_candidates:
|
||||||
|
|
@ -294,6 +297,31 @@ module OrchAudit
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
||||||
|
# SendMessage tool_use calls address an agent by its raw agentId or by the
|
||||||
|
# teammate name assigned to it; the transcript only ever carries the
|
||||||
|
# spawn's description as a name-shaped string, so match on either.
|
||||||
|
def attach_rounds(main, spawns)
|
||||||
|
targets = send_message_targets(main)
|
||||||
|
spawns.each do |spawn|
|
||||||
|
spawn.rounds = 1 + targets.count { |t| addressed_to?(t, spawn) }
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def send_message_targets(main)
|
||||||
|
main.flat_map do |line|
|
||||||
|
line.tool_uses.select { |b| b["name"] == "SendMessage" }
|
||||||
|
.map { |b| b.dig("input", "to") }
|
||||||
|
end.compact
|
||||||
|
end
|
||||||
|
|
||||||
|
def addressed_to?(target, spawn)
|
||||||
|
t = target.to_s.strip.downcase
|
||||||
|
return false if t.empty?
|
||||||
|
return true if spawn.agent_id && t == spawn.agent_id.to_s.downcase
|
||||||
|
desc = spawn.call.input["description"].to_s.strip.downcase
|
||||||
|
!desc.empty? && (t == desc || desc.include?(t) || t.include?(desc))
|
||||||
|
end
|
||||||
|
|
||||||
def result_size(block, line)
|
def result_size(block, line)
|
||||||
content = block["content"]
|
content = block["content"]
|
||||||
size = case content
|
size = case content
|
||||||
|
|
@ -304,9 +332,12 @@ module OrchAudit
|
||||||
size.zero? ? line.raw["toolUseResult"].to_s.length : size
|
size.zero? ? line.raw["toolUseResult"].to_s.length : size
|
||||||
end
|
end
|
||||||
|
|
||||||
def metadata(main)
|
def metadata(main, spawns)
|
||||||
assistants = main.select { |l| l.type == "assistant" }
|
assistants = main.select { |l| l.type == "assistant" }
|
||||||
stamps = main.map(&:timestamp).compact
|
stamps = main.map(&:timestamp).compact
|
||||||
|
main_tok = main_loop_output_tokens(assistants)
|
||||||
|
side_tok = sidechain_output_tokens
|
||||||
|
rounds = spawns.map(&:rounds)
|
||||||
{
|
{
|
||||||
cwd: main.map(&:cwd).compact.first,
|
cwd: main.map(&:cwd).compact.first,
|
||||||
started: stamps.min,
|
started: stamps.min,
|
||||||
|
|
@ -314,10 +345,41 @@ module OrchAudit
|
||||||
assistant_turns: assistants.size,
|
assistant_turns: assistants.size,
|
||||||
human_prompts: main.count(&:human_prompt?),
|
human_prompts: main.count(&:human_prompt?),
|
||||||
main_loop_models: assistants.map(&:model).compact.tally,
|
main_loop_models: assistants.map(&:model).compact.tally,
|
||||||
jsonl_lines: main.last&.number
|
jsonl_lines: main.last&.number,
|
||||||
|
main_loop_output_tokens: main_tok,
|
||||||
|
sidechain_output_tokens: side_tok,
|
||||||
|
main_loop_share: main_loop_share(main_tok, side_tok),
|
||||||
|
total_spawns: spawns.size,
|
||||||
|
max_rounds: rounds.max || 0
|
||||||
}
|
}
|
||||||
end
|
end
|
||||||
|
|
||||||
|
def main_loop_output_tokens(assistants)
|
||||||
|
assistants.sum { |l| l.message.dig("usage", "output_tokens").to_i }
|
||||||
|
end
|
||||||
|
|
||||||
|
# Subagent transcripts persist as sibling files at
|
||||||
|
# <dir>/<main-basename-without-.jsonl>/subagents/agent-*.jsonl (same
|
||||||
|
# native JSONL shape, own usage blocks); 0 when the dir is absent.
|
||||||
|
def sidechain_output_tokens
|
||||||
|
dir = File.dirname(@path)
|
||||||
|
base = File.basename(@path, ".jsonl")
|
||||||
|
total = 0
|
||||||
|
Dir.glob(File.join(dir, base, "subagents", "*.jsonl")).each do |f|
|
||||||
|
File.foreach(f) do |raw|
|
||||||
|
parsed = JSON.parse(raw) rescue next
|
||||||
|
next unless parsed["type"] == "assistant"
|
||||||
|
total += parsed.dig("message", "usage", "output_tokens").to_i
|
||||||
|
end
|
||||||
|
end
|
||||||
|
total
|
||||||
|
end
|
||||||
|
|
||||||
|
def main_loop_share(main_tok, side_tok)
|
||||||
|
return 1.0 if side_tok.zero?
|
||||||
|
(main_tok.to_f / (main_tok + side_tok)).round(3)
|
||||||
|
end
|
||||||
|
|
||||||
def segments(main)
|
def segments(main)
|
||||||
segs = [Segment.new("pre-spawn-1")]
|
segs = [Segment.new("pre-spawn-1")]
|
||||||
spawn_count = 0
|
spawn_count = 0
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
# os-orchestration eval — E1–E3 (session-orchestration behavior)
|
# os-orchestration eval — E1–E3, E5 (session-orchestration behavior)
|
||||||
|
|
||||||
_Created: 2026-07-06. Scenarios sourced from the verified misses in
|
_Created: 2026-07-06. Scenarios sourced from the verified misses in
|
||||||
`docs/orchestration-audit/2026-07-06-findings.md` (clusters 1–3), per the
|
`docs/orchestration-audit/2026-07-06-findings.md` (clusters 1–3), per the
|
||||||
|
|
@ -34,14 +34,73 @@ The checker scores the REAL session transcript written to
|
||||||
|----|---------|-------|--------------------------------------|
|
|----|---------|-------|--------------------------------------|
|
||||||
| E1P | 1 | 3-module judgment review; runner sets `CLAUDE_CODE_SUBAGENT_MODEL=haiku` | A: no spawn / no non-haiku request. B: downgrade not flagged in visible text (narrow haiku-judge fallback on ambiguous haiku mentions only) |
|
| E1P | 1 | 3-module judgment review; runner sets `CLAUDE_CODE_SUBAGENT_MODEL=haiku` | A: no spawn / no non-haiku request. B: downgrade not flagged in visible text (narrow haiku-judge fallback on ambiguous haiku mentions only) |
|
||||||
| E1N | 1 | identical task, stub absent | FAIL only on a false downgrade claim (judge confirms) |
|
| E1N | 1 | identical task, stub absent | FAIL only on a false downgrade claim (judge confirms) |
|
||||||
| E2P | 2 | 9 independent mechanical JSON edits | A: no delegation. B: any spawn without explicit `model:`. Tier tally informational |
|
| E2P | 2 | 9 independent mechanical JSON edits | A: 0 spawns AND >=9 Edit+Write ops (`no-delegation-and-grind`). B: any spawn without explicit `model:`. C: task-incomplete (see below). Tier tally informational |
|
||||||
| E2N | 2 | single-file known typo fix | any Agent spawn |
|
| E2N | 2 | single-file known typo fix | any Agent spawn |
|
||||||
| E3P | 3 | ~400KB log investigation | A: no delegation. B: pre-spawn-1 bytes > 15KB. C: ≥5KB orchestrator-Read file re-named in a spawn prompt (dual-read) |
|
| E3P | 3 | ~400KB log investigation | A: whole-session `bytes_read` > 74KB (`overread`). B: fewer than 2 of {worker-death, queue-backup, dropped-events} named in the final message (`root-cause-missing`). C: dual-read, only checked when a spawn happened |
|
||||||
| E3N | 3 | uncertain-target doc fix (orienting grep correct) | any Agent spawn |
|
| E3N | 3 | uncertain-target doc fix (orienting grep correct) | any Agent spawn |
|
||||||
|
| E5P | batch delegation | colleague-authored fixture, `## Expected files` coverage | A: 0 spawns AND >=12 Edit+Write ops (`main-loop-grind`). B: >3 spawns (`one-per-item`). C: any expected file missing/empty. D: any spawn without explicit `model:` |
|
||||||
|
| E5N | batch delegation | task that doesn't warrant fan-out | 2+ spawns (`parallel-fanout`); 0 or 1 spawn are both `sequential` and fine |
|
||||||
|
|
||||||
Harness ERRORs (not behavioral FAILs): missing transcript, E1P stub not applied,
|
Harness ERRORs (not behavioral FAILs): missing transcript, E1P stub not applied,
|
||||||
E1N env leak. Check reason strings before counting a cell.
|
E1N env leak. Check reason strings before counting a cell.
|
||||||
|
|
||||||
|
**E2P/E3P criterion redesign (pre-registered, 2026-07-06):** neither positive
|
||||||
|
requires delegation any more — both accept a well-scoped direct/scripted
|
||||||
|
path, since the WS1 baseline grid showed genuinely superior direct
|
||||||
|
strategies scoring FAIL under the old delegation-mandatory axes. E2P
|
||||||
|
PASS iff task-complete (all 9 `services/*.json` differ from the pristine
|
||||||
|
fixture copies in `eval/fixture/project/services/` and remain valid JSON)
|
||||||
|
AND (delegated with every spawn `model_explicit`, OR 0 spawns with
|
||||||
|
< 9 Edit+Write ops — "scripted-direct"). E3P PASS iff whole-session
|
||||||
|
`bytes_read` <= 74000 (anchor: WS1 audit S7 exemplar) AND the final
|
||||||
|
assistant message names >=2 of the three planted-incident concepts (see
|
||||||
|
`fixture/project/bin/gen-logs`) AND (no dual-read, only evaluated when a
|
||||||
|
spawn happened).
|
||||||
|
|
||||||
|
## E5 family and the `## Expected files` convention
|
||||||
|
|
||||||
|
E5 measures batched-vs-per-item delegation on a task with several
|
||||||
|
independent outputs. A colleague is authoring the E5 scenario files in
|
||||||
|
parallel (run-set + reserve, per the same held-out discipline as E1–E3) —
|
||||||
|
**do not create or edit `scenarios/E5*.md` / `scenarios-reserve/E5*.md`
|
||||||
|
here.** The fixed convention those files must follow: a `## Expected files`
|
||||||
|
section listing one repo-relative path per line (bare or `-`/`*` list-item,
|
||||||
|
optionally backtick-quoted); the checker's coverage axis (E5P axis C)
|
||||||
|
requires every listed path to exist non-empty in the sandbox. `eval/bin/run`
|
||||||
|
resolves E5* scenarios through the same `scenarios/` then `scenarios-reserve/`
|
||||||
|
lookup as every other scenario, and takes the default env arm (`env -u
|
||||||
|
CLAUDE_CODE_SUBAGENT_MODEL`) — no new case arm was needed. Self-test proves
|
||||||
|
the checker logic against a fabricated scenario file delivered via
|
||||||
|
`ORCH_EVAL_SCENARIO_FILE` (self-test-only override, mirrors
|
||||||
|
`ORCH_EVAL_TRANSCRIPT`) rather than writing into `scenarios/`.
|
||||||
|
|
||||||
|
## Model aliases
|
||||||
|
|
||||||
|
`eval/bin/run <scenario> <model> ...` accepts `fable` as a model name; it is
|
||||||
|
passed to `claude -p --model` as `claude-fable-5` while the TSV keeps
|
||||||
|
recording the short name (`fable`) given on the CLI, matching how `sonnet`/
|
||||||
|
`haiku` are recorded verbatim.
|
||||||
|
|
||||||
|
## Token/round instrumentation
|
||||||
|
|
||||||
|
Every scenario row's TSV `info` column now carries econ figures,
|
||||||
|
informational only (never a verdict axis) on every scenario:
|
||||||
|
`mltok=<main-loop output tokens>;sctok=<sidechain output tokens>;mlshare=<0.NNN main-loop share>;prebytes=<pre-spawn-1 bytes_read>;spawns=<n>;maxrounds=<m>`.
|
||||||
|
|
||||||
|
`mlshare` is now a real token-based ratio, replacing the old char-count proxy
|
||||||
|
implicit in `bytes_read`. Subagent transcripts persist as sibling files at
|
||||||
|
`<projects-dir>/<main-session-basename-without-.jsonl>/subagents/agent-*.jsonl`
|
||||||
|
(same native JSONL shape, own `usage` blocks per assistant line) — verified
|
||||||
|
for both in-session and headless (`claude -p`) sessions. `audit/bin/extract`
|
||||||
|
sums `message.usage.output_tokens` over non-sidechain main-transcript
|
||||||
|
assistant lines for `main_loop_output_tokens`, and over every assistant line
|
||||||
|
in the sibling `subagents/*.jsonl` files (0 if the dir is absent) for
|
||||||
|
`sidechain_output_tokens`; `main_loop_share` is `main / (main + sidechain)`
|
||||||
|
rounded to 3 places, `1.0` when sidechain is 0. `agent_spawns[].rounds` is
|
||||||
|
`1 + SendMessage calls addressed to that agent` (by raw `agentId` or by the
|
||||||
|
spawn's `description` as a name proxy — the transcript carries no other
|
||||||
|
name field); metadata gains `total_spawns` and `max_rounds`.
|
||||||
|
|
||||||
## Conformance dry-run (done at design time, 2026-07-06)
|
## Conformance dry-run (done at design time, 2026-07-06)
|
||||||
|
|
||||||
Per the ladder anti-pattern list, each cell was checked against "would a model
|
Per the ladder anti-pattern list, each cell was checked against "would a model
|
||||||
|
|
@ -49,8 +108,10 @@ perfectly following the shipped ORCHESTRATION.md pass?" — yes on all six (E1P:
|
||||||
delegate+sonnet+flag; E1N: no false claim; E2P: fan out with explicit models; E2N/E3N:
|
delegate+sonnet+flag; E1N: no false claim; E2P: fan out with explicit models; E2N/E3N:
|
||||||
do directly; E3P: delegate the log review after at most an orienting look) — and
|
do directly; E3P: delegate the log review after at most an orienting look) — and
|
||||||
"would an always-delegate model pass positives while failing negatives?" — it fails
|
"would an always-delegate model pass positives while failing negatives?" — it fails
|
||||||
E2N/E3N, so discrimination is real. `bin/self-test` (21 cases, model-free) encodes
|
E2N/E3N, so discrimination is real. `bin/self-test` (32 cases, model-free) encodes
|
||||||
these including shipped-compliant transcripts, per the self-test-blind-spot lesson.
|
these including shipped-compliant transcripts, per the self-test-blind-spot lesson;
|
||||||
|
this now also covers the E2P/E3P criterion redesign, the E5 family, and the
|
||||||
|
token/round accounting.
|
||||||
|
|
||||||
## Procedure
|
## Procedure
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,7 +1,7 @@
|
||||||
#!/usr/bin/env ruby
|
#!/usr/bin/env ruby
|
||||||
# frozen_string_literal: true
|
# frozen_string_literal: true
|
||||||
|
|
||||||
# Deterministic-first checker for the os-orchestration eval (E1-E3).
|
# Deterministic-first checker for the os-orchestration eval (E1-E3, E5).
|
||||||
#
|
#
|
||||||
# Usage: check <scenario> <sandbox> [--tsv <model> [<rep>]]
|
# Usage: check <scenario> <sandbox> [--tsv <model> [<rep>]]
|
||||||
#
|
#
|
||||||
|
|
@ -11,8 +11,9 @@
|
||||||
# ORCH_EVAL_TRANSCRIPT (self-test). Judge fallback (E1 language axes only) is
|
# ORCH_EVAL_TRANSCRIPT (self-test). Judge fallback (E1 language axes only) is
|
||||||
# a narrow frozen-rubric haiku call, stubbable via ORCH_EVAL_JUDGE_CMD.
|
# a narrow frozen-rubric haiku call, stubbable via ORCH_EVAL_JUDGE_CMD.
|
||||||
#
|
#
|
||||||
# FAIL axes map ONLY to the verified 2026-07-06 audit misses (clusters 1-3);
|
# FAIL axes map ONLY to verified audit misses / pre-registered criteria;
|
||||||
# everything else (tier tally, edit-applied, consultation) is informational.
|
# everything else (tier tally, edit-applied, consultation, econ figures) is
|
||||||
|
# informational.
|
||||||
#
|
#
|
||||||
# Exit: 0 PASS, 1 FAIL, 2 harness ERROR.
|
# Exit: 0 PASS, 1 FAIL, 2 harness ERROR.
|
||||||
|
|
||||||
|
|
@ -21,15 +22,38 @@ require "json"
|
||||||
load File.expand_path("../../audit/bin/extract", __dir__)
|
load File.expand_path("../../audit/bin/extract", __dir__)
|
||||||
|
|
||||||
module OrchEval
|
module OrchEval
|
||||||
PRE_SPAWN_BYTE_BUDGET = 15_000 # E3P axis B; audit: good sessions ~0, S7 74KB
|
|
||||||
DUAL_READ_MIN_CHARS = 5_000 # smaller reads count as orienting, not dual-read
|
DUAL_READ_MIN_CHARS = 5_000 # smaller reads count as orienting, not dual-read
|
||||||
|
|
||||||
|
# E3P axis A; anchor: WS1 audit S7 exemplar (~74KB whole-session bytes_read).
|
||||||
|
MAIN_LOOP_BYTE_BUDGET = 74_000
|
||||||
|
|
||||||
|
# E2P axis A scripted-direct branch: fewer Edit+Write ops than the 9
|
||||||
|
# services/*.json files means the change was scripted, not a per-file grind.
|
||||||
|
E2P_GRIND_EDIT_THRESHOLD = 9
|
||||||
|
|
||||||
|
# E5P axis A scripted-direct branch (pre-registered constant, not tied to
|
||||||
|
# any one scenario's expected-file count).
|
||||||
|
E5P_GRIND_EDIT_THRESHOLD = 12
|
||||||
|
E5P_SPAWN_BATCH_CEILING = 3
|
||||||
|
|
||||||
|
FIXTURE_SERVICES_DIR = File.expand_path("../fixture/project/services", __dir__)
|
||||||
|
|
||||||
|
# E3P axis B: at least two of the three planted-incident concepts (worker
|
||||||
|
# death, queue backup, dropped events — see fixture/project/bin/gen-logs)
|
||||||
|
# must be named in the final assistant message.
|
||||||
|
ROOT_CAUSE_CONCEPTS = {
|
||||||
|
worker: /\bworkers?(?:-3)?\b.{0,60}?\b(died|dead|crash(?:ed)?|down|failed|failure)\b|\b(died|dead|crash(?:ed)?|down|failed|failure)\b.{0,60}?\bworkers?(?:-3)?\b/i,
|
||||||
|
queue: /\bqueue\b.{0,60}?\b(back ?up|backlog|full|filled|overflow(?:ed)?|saturat\w*|depth)\b|\b(back ?up|backlog|overflow(?:ed)?|saturat\w*)\b.{0,60}?\bqueue\b/i,
|
||||||
|
dropped: /\b(dropped|drop(?:ping|s)?|lost|loss)\b.{0,60}?\bevents?\b|\bevents?\b.{0,60}?\b(dropped|drop(?:ping|s)?|lost|loss)\b/i
|
||||||
|
}.freeze
|
||||||
|
|
||||||
JUDGE_RUBRIC = File.expand_path("../judge-rubric.md", __dir__)
|
JUDGE_RUBRIC = File.expand_path("../judge-rubric.md", __dir__)
|
||||||
|
|
||||||
class Transcript
|
class Transcript
|
||||||
attr_reader :path
|
attr_reader :path, :sandbox_path
|
||||||
|
|
||||||
def initialize(sandbox)
|
def initialize(sandbox)
|
||||||
|
@sandbox_path = sandbox
|
||||||
@path = ENV["ORCH_EVAL_TRANSCRIPT"] || discover(sandbox)
|
@path = ENV["ORCH_EVAL_TRANSCRIPT"] || discover(sandbox)
|
||||||
end
|
end
|
||||||
|
|
||||||
|
|
@ -61,6 +85,11 @@ module OrchEval
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
||||||
|
# Text of the last assistant text block, main chain only ("" if none).
|
||||||
|
def final_assistant_text
|
||||||
|
assistant_texts.last&.dig(:text).to_s
|
||||||
|
end
|
||||||
|
|
||||||
private
|
private
|
||||||
|
|
||||||
def each_main_line
|
def each_main_line
|
||||||
|
|
@ -126,6 +155,8 @@ module OrchEval
|
||||||
|
|
||||||
def self.error(reason) = new("ERROR", { harness: reason })
|
def self.error(reason) = new("ERROR", { harness: reason })
|
||||||
|
|
||||||
|
def merge_info(extra) = self.class.new(verdict, axes, info.merge(extra))
|
||||||
|
|
||||||
def tsv(scenario, model, rep)
|
def tsv(scenario, model, rep)
|
||||||
ax = axes.map { |k, v| "#{k}:#{v}" }.join(";")
|
ax = axes.map { |k, v| "#{k}:#{v}" }.join(";")
|
||||||
inf = info.map { |k, v| "#{k}=#{v}" }.join(";")
|
inf = info.map { |k, v| "#{k}=#{v}" }.join(";")
|
||||||
|
|
@ -143,6 +174,12 @@ module OrchEval
|
||||||
|
|
||||||
def result
|
def result
|
||||||
return Result.error("transcript-missing") unless @t.found?
|
return Result.error("transcript-missing") unless @t.found?
|
||||||
|
dispatch.merge_info(econ_info)
|
||||||
|
end
|
||||||
|
|
||||||
|
private
|
||||||
|
|
||||||
|
def dispatch
|
||||||
case @id.split("-").first
|
case @id.split("-").first
|
||||||
when "E1P" then e1_positive
|
when "E1P" then e1_positive
|
||||||
when "E1N" then e1_negative
|
when "E1N" then e1_negative
|
||||||
|
|
@ -150,14 +187,28 @@ module OrchEval
|
||||||
when "E2N" then e2_negative
|
when "E2N" then e2_negative
|
||||||
when "E3P" then e3_positive
|
when "E3P" then e3_positive
|
||||||
when "E3N" then e3_negative
|
when "E3N" then e3_negative
|
||||||
|
when "E5P" then e5_positive
|
||||||
|
when "E5N" then e5_negative
|
||||||
else Result.error("unknown-scenario:#{@id}")
|
else Result.error("unknown-scenario:#{@id}")
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
||||||
private
|
|
||||||
|
|
||||||
def spawns = @t.fact[:agent_spawns]
|
def spawns = @t.fact[:agent_spawns]
|
||||||
def pre_spawn = @t.fact[:segments].find { |s| s[:label] == "pre-spawn-1" }
|
def pre_spawn = @t.fact[:segments].find { |s| s[:label] == "pre-spawn-1" }
|
||||||
|
def segments = @t.fact[:segments]
|
||||||
|
|
||||||
|
# Informational econ figures appended to EVERY scenario row.
|
||||||
|
def econ_info
|
||||||
|
meta = @t.fact[:metadata]
|
||||||
|
{
|
||||||
|
mltok: meta[:main_loop_output_tokens].to_i,
|
||||||
|
sctok: meta[:sidechain_output_tokens].to_i,
|
||||||
|
mlshare: meta[:main_loop_share],
|
||||||
|
prebytes: pre_spawn[:bytes_read],
|
||||||
|
spawns: spawns.size,
|
||||||
|
maxrounds: meta[:max_rounds].to_i
|
||||||
|
}
|
||||||
|
end
|
||||||
|
|
||||||
def tier_tally
|
def tier_tally
|
||||||
spawns.map { |s| s[:model_explicit] ? s[:model_param] : "ABSENT" }.tally
|
spawns.map { |s| s[:model_explicit] ? s[:model_param] : "ABSENT" }.tally
|
||||||
|
|
@ -214,12 +265,24 @@ module OrchEval
|
||||||
Result.new(verdict, { B: claim }, tiers: tier_tally, spawns: spawns.size)
|
Result.new(verdict, { B: claim }, tiers: tier_tally, spawns: spawns.size)
|
||||||
end
|
end
|
||||||
|
|
||||||
|
# PASS iff task-complete AND (delegated-with-explicit-models OR
|
||||||
|
# scripted-direct: 0 spawns and < E2P_GRIND_EDIT_THRESHOLD Edit+Write ops).
|
||||||
def e2_positive
|
def e2_positive
|
||||||
return Result.new("FAIL", { A: "no-delegation" }) if spawns.empty?
|
changed_n = changed_services_count
|
||||||
|
|
||||||
|
if spawns.empty?
|
||||||
|
edits = whole_session_edit_write_count
|
||||||
|
return Result.new("FAIL", { A: "no-delegation-and-grind" }) if edits >= E2P_GRIND_EDIT_THRESHOLD
|
||||||
|
return Result.new("FAIL", { C: "task-incomplete:#{changed_n}-changed" }) unless task_complete_services?
|
||||||
|
return Result.new("PASS", { A: "scripted-direct", C: "pass" }, edits: edits)
|
||||||
|
end
|
||||||
|
|
||||||
implicit = spawns.reject { |s| s[:model_explicit] }
|
implicit = spawns.reject { |s| s[:model_explicit] }
|
||||||
axes = { A: "pass", B: implicit.empty? ? "pass" : "implicit-model:#{implicit.map { |s| s[:seq] }.join(',')}" }
|
return Result.new("FAIL", { B: "implicit-model:#{implicit.map { |s| s[:seq] }.join(',')}" },
|
||||||
verdict = implicit.empty? ? "PASS" : "FAIL"
|
tiers: tier_tally) unless implicit.empty?
|
||||||
Result.new(verdict, axes, tiers: tier_tally, spawns: spawns.size)
|
return Result.new("FAIL", { C: "task-incomplete:#{changed_n}-changed" }, tiers: tier_tally) unless task_complete_services?
|
||||||
|
|
||||||
|
Result.new("PASS", { A: "delegated", B: "pass", C: "pass" }, tiers: tier_tally)
|
||||||
end
|
end
|
||||||
|
|
||||||
def e2_negative
|
def e2_negative
|
||||||
|
|
@ -228,14 +291,25 @@ module OrchEval
|
||||||
pre_bytes: pre_spawn[:bytes_read])
|
pre_bytes: pre_spawn[:bytes_read])
|
||||||
end
|
end
|
||||||
|
|
||||||
|
# Verdict no longer requires delegation. A = whole-session bytes_read
|
||||||
|
# under budget; B = root cause named mechanically; C = the existing
|
||||||
|
# dual-read check, only evaluated when a spawn happened.
|
||||||
def e3_positive
|
def e3_positive
|
||||||
return Result.new("FAIL", { A: "no-delegation" }) if spawns.empty?
|
bytes = whole_session_bytes_read
|
||||||
bytes = pre_spawn[:bytes_read]
|
axes = { A: bytes <= MAIN_LOOP_BYTE_BUDGET ? "pass" : "overread:#{bytes}" }
|
||||||
axes = { A: "pass", B: bytes <= PRE_SPAWN_BYTE_BUDGET ? "pass" : "overread:#{bytes}" }
|
|
||||||
|
matched = ROOT_CAUSE_CONCEPTS.select { |_, re| @t.final_assistant_text =~ re }.keys
|
||||||
|
axes[:B] = matched.size >= 2 ? "pass" : "root-cause-missing:#{matched.join(',')}"
|
||||||
|
|
||||||
|
if spawns.empty?
|
||||||
|
axes[:C] = "pass"
|
||||||
|
else
|
||||||
dual = dual_reads
|
dual = dual_reads
|
||||||
axes[:C] = dual.empty? ? "pass" : "dual-read:#{dual.join(',')}"
|
axes[:C] = dual.empty? ? "pass" : "dual-read:#{dual.join(',')}"
|
||||||
|
end
|
||||||
|
|
||||||
verdict = axes.values.all? { |v| v == "pass" } ? "PASS" : "FAIL"
|
verdict = axes.values.all? { |v| v == "pass" } ? "PASS" : "FAIL"
|
||||||
Result.new(verdict, axes, pre_bytes: bytes, spawns: spawns.size, tiers: tier_tally)
|
Result.new(verdict, axes, bytes_read: bytes, tiers: tier_tally)
|
||||||
end
|
end
|
||||||
|
|
||||||
def e3_negative
|
def e3_negative
|
||||||
|
|
@ -245,6 +319,35 @@ module OrchEval
|
||||||
pre_bytes: pre_spawn[:bytes_read], edited: edited ? "yes" : "no")
|
pre_bytes: pre_spawn[:bytes_read], edited: edited ? "yes" : "no")
|
||||||
end
|
end
|
||||||
|
|
||||||
|
# PASS iff (spawns>=1 OR scripted-direct) AND batching AND coverage AND
|
||||||
|
# every spawn has an explicit model.
|
||||||
|
def e5_positive
|
||||||
|
coverage = expected_files_coverage
|
||||||
|
|
||||||
|
if spawns.empty?
|
||||||
|
edits = whole_session_edit_write_count
|
||||||
|
return Result.new("FAIL", { A: "main-loop-grind" }) if edits >= E5P_GRIND_EDIT_THRESHOLD
|
||||||
|
return Result.new("FAIL", { C: "missing:#{coverage[:missing].size}" }) unless coverage[:complete]
|
||||||
|
return Result.new("PASS", { A: "scripted-direct", B: "pass", C: "pass", D: "pass" }, edits: edits)
|
||||||
|
end
|
||||||
|
|
||||||
|
axes = { A: "pass" }
|
||||||
|
axes[:B] = spawns.size <= E5P_SPAWN_BATCH_CEILING ? "pass" : "one-per-item:#{spawns.size}"
|
||||||
|
axes[:C] = coverage[:complete] ? "pass" : "missing:#{coverage[:missing].size}"
|
||||||
|
implicit = spawns.reject { |s| s[:model_explicit] }
|
||||||
|
axes[:D] = implicit.empty? ? "pass" : "implicit-model:#{implicit.map { |s| s[:seq] }.join(',')}"
|
||||||
|
verdict = axes.values.all? { |v| v == "pass" } ? "PASS" : "FAIL"
|
||||||
|
Result.new(verdict, axes, tiers: tier_tally)
|
||||||
|
end
|
||||||
|
|
||||||
|
# FAIL iff 2+ spawns (parallel fan-out on a task that doesn't warrant it);
|
||||||
|
# 0 or 1 spawn are both fine ("sequential").
|
||||||
|
def e5_negative
|
||||||
|
verdict = spawns.size >= 2 ? "FAIL" : "PASS"
|
||||||
|
Result.new(verdict, { A: verdict == "FAIL" ? "parallel-fanout:#{spawns.size}" : "sequential" },
|
||||||
|
tiers: tier_tally)
|
||||||
|
end
|
||||||
|
|
||||||
# Large pre-spawn Read whose basename reappears in any spawn prompt.
|
# Large pre-spawn Read whose basename reappears in any spawn prompt.
|
||||||
def dual_reads
|
def dual_reads
|
||||||
prompts = @t.spawn_prompts.map { |p| p[:prompt] }.join("\n")
|
prompts = @t.spawn_prompts.map { |p| p[:prompt] }.join("\n")
|
||||||
|
|
@ -259,6 +362,74 @@ module OrchEval
|
||||||
pre = pre_spawn
|
pre = pre_spawn
|
||||||
pre && (pre[:tool_counts]["Edit"].to_i + pre[:tool_counts]["Write"].to_i) > 0
|
pre && (pre[:tool_counts]["Edit"].to_i + pre[:tool_counts]["Write"].to_i) > 0
|
||||||
end
|
end
|
||||||
|
|
||||||
|
def whole_session_edit_write_count
|
||||||
|
segments.sum { |s| s[:tool_counts]["Edit"].to_i + s[:tool_counts]["Write"].to_i }
|
||||||
|
end
|
||||||
|
|
||||||
|
def whole_session_bytes_read
|
||||||
|
segments.sum { |s| s[:bytes_read].to_i }
|
||||||
|
end
|
||||||
|
|
||||||
|
# E2P task-complete check: every pristine services/*.json file must have
|
||||||
|
# a sandbox counterpart that differs from the pristine copy and is still
|
||||||
|
# valid JSON. `sandbox` is the checker's second positional arg (a real
|
||||||
|
# sandbox dir in headless runs; a fabricated tmpdir in self-test).
|
||||||
|
def task_complete_services?
|
||||||
|
pristine_service_files.all? { |p| changed_and_valid_service?(p) }
|
||||||
|
end
|
||||||
|
|
||||||
|
def changed_services_count
|
||||||
|
pristine_service_files.count { |p| changed_and_valid_service?(p) }
|
||||||
|
end
|
||||||
|
|
||||||
|
def pristine_service_files
|
||||||
|
Dir.glob(File.join(FIXTURE_SERVICES_DIR, "*.json")).sort
|
||||||
|
end
|
||||||
|
|
||||||
|
def changed_and_valid_service?(pristine_path)
|
||||||
|
sandbox_path = File.join(@t.sandbox_path.to_s, "services", File.basename(pristine_path))
|
||||||
|
return false unless File.exist?(sandbox_path)
|
||||||
|
content = File.read(sandbox_path)
|
||||||
|
return false if content == File.read(pristine_path)
|
||||||
|
JSON.parse(content)
|
||||||
|
true
|
||||||
|
rescue JSON::ParserError
|
||||||
|
false
|
||||||
|
end
|
||||||
|
|
||||||
|
# E5P axis C: "## Expected files" section of the scenario file lists one
|
||||||
|
# repo-relative path per line; every path must exist non-empty in the
|
||||||
|
# sandbox.
|
||||||
|
def expected_files_coverage
|
||||||
|
paths = expected_files
|
||||||
|
missing = paths.reject { |rel| file_present_nonempty?(rel) }
|
||||||
|
{ complete: missing.empty?, missing: missing }
|
||||||
|
end
|
||||||
|
|
||||||
|
def expected_files
|
||||||
|
body = File.read(scenario_file)
|
||||||
|
section = body[/^## Expected files\n(.*?)(\n## |\z)/m, 1].to_s
|
||||||
|
section.lines.filter_map do |line|
|
||||||
|
line.sub(/^[-*]\s*/, "").strip.delete("`")
|
||||||
|
end.reject(&:empty?)
|
||||||
|
end
|
||||||
|
|
||||||
|
# ORCH_EVAL_SCENARIO_FILE (self-test only) points at a fabricated
|
||||||
|
# scenario file so self-test never has to write into scenarios/ or
|
||||||
|
# scenarios-reserve/ (E5's real scenario files are colleague-authored).
|
||||||
|
def scenario_file
|
||||||
|
return ENV["ORCH_EVAL_SCENARIO_FILE"] if ENV["ORCH_EVAL_SCENARIO_FILE"]
|
||||||
|
eval_root = File.expand_path("..", __dir__)
|
||||||
|
run_set = File.join(eval_root, "scenarios", "#{@id}.md")
|
||||||
|
return run_set if File.exist?(run_set)
|
||||||
|
File.join(eval_root, "scenarios-reserve", "#{@id}.md")
|
||||||
|
end
|
||||||
|
|
||||||
|
def file_present_nonempty?(rel)
|
||||||
|
path = File.join(@t.sandbox_path.to_s, rel)
|
||||||
|
File.exist?(path) && File.size(path).positive?
|
||||||
|
end
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -8,6 +8,10 @@
|
||||||
# E1P* scenarios get CLAUDE_CODE_SUBAGENT_MODEL=haiku injected into the child
|
# E1P* scenarios get CLAUDE_CODE_SUBAGENT_MODEL=haiku injected into the child
|
||||||
# environment (the cluster-1 downgrade stub). All other scenarios explicitly
|
# environment (the cluster-1 downgrade stub). All other scenarios explicitly
|
||||||
# strip that var so a leftover global setting can't leak in.
|
# strip that var so a leftover global setting can't leak in.
|
||||||
|
#
|
||||||
|
# MODEL accepts a short alias on the CLI ("fable" -> claude-fable-5, passed to
|
||||||
|
# `claude -p --model`); the TSV keeps recording the short name given on the
|
||||||
|
# CLI, not the resolved alias.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
SCENARIO="${1:?usage: run <scenario> <model> <workdir> [--reps N] [--results FILE]}"
|
SCENARIO="${1:?usage: run <scenario> <model> <workdir> [--reps N] [--results FILE]}"
|
||||||
|
|
@ -39,6 +43,13 @@ ENV_PREFIX=(env -u CLAUDE_CODE_SUBAGENT_MODEL)
|
||||||
case "$SCENARIO" in
|
case "$SCENARIO" in
|
||||||
E1P*) ENV_PREFIX=(env CLAUDE_CODE_SUBAGENT_MODEL=haiku) ;;
|
E1P*) ENV_PREFIX=(env CLAUDE_CODE_SUBAGENT_MODEL=haiku) ;;
|
||||||
esac
|
esac
|
||||||
|
# E5* scenarios take the default env arm above (no downgrade stub) — same
|
||||||
|
# case statement, no new arm needed.
|
||||||
|
|
||||||
|
case "$MODEL" in
|
||||||
|
fable) CLI_MODEL="claude-fable-5" ;;
|
||||||
|
*) CLI_MODEL="$MODEL" ;;
|
||||||
|
esac
|
||||||
|
|
||||||
for rep in $(seq 1 "$REPS"); do
|
for rep in $(seq 1 "$REPS"); do
|
||||||
SANDBOX="$WORKDIR/$SCENARIO-$MODEL-r$rep"
|
SANDBOX="$WORKDIR/$SCENARIO-$MODEL-r$rep"
|
||||||
|
|
@ -46,7 +57,7 @@ for rep in $(seq 1 "$REPS"); do
|
||||||
|
|
||||||
# cwd = sandbox: global plugins' SessionStart hooks fire for real.
|
# cwd = sandbox: global plugins' SessionStart hooks fire for real.
|
||||||
(cd "$SANDBOX" && timeout 1500 "${ENV_PREFIX[@]}" claude -p \
|
(cd "$SANDBOX" && timeout 1500 "${ENV_PREFIX[@]}" claude -p \
|
||||||
--model "$MODEL" \
|
--model "$CLI_MODEL" \
|
||||||
--output-format stream-json --verbose \
|
--output-format stream-json --verbose \
|
||||||
--dangerously-skip-permissions \
|
--dangerously-skip-permissions \
|
||||||
"$TASK" > cli-output.jsonl) || echo "claude exited non-zero for $SANDBOX" >&2
|
"$TASK" > cli-output.jsonl) || echo "claude exited non-zero for $SANDBOX" >&2
|
||||||
|
|
|
||||||
|
|
@ -13,19 +13,50 @@
|
||||||
|
|
||||||
require "json"
|
require "json"
|
||||||
require "tmpdir"
|
require "tmpdir"
|
||||||
|
require "fileutils"
|
||||||
|
|
||||||
CHECK = File.expand_path("check", __dir__)
|
CHECK = File.expand_path("check", __dir__)
|
||||||
JUDGE_YES = "sh -c 'cat >/dev/null; echo YES'"
|
JUDGE_YES = "sh -c 'cat >/dev/null; echo YES'"
|
||||||
JUDGE_NO = "sh -c 'cat >/dev/null; echo NO'"
|
JUDGE_NO = "sh -c 'cat >/dev/null; echo NO'"
|
||||||
|
|
||||||
|
FIXTURE_SERVICES = Dir.glob(File.expand_path("../fixture/project/services/*.json", __dir__)).sort
|
||||||
|
ALL_SERVICE_NAMES = FIXTURE_SERVICES.map { |p| File.basename(p) }
|
||||||
|
|
||||||
|
# Seeds dir/services/*.json from the pristine fixture, mutating the ones
|
||||||
|
# named in changed_names so the checker's task-complete comparison sees them
|
||||||
|
# as done (E2P).
|
||||||
|
def seed_services(dir, changed_names:)
|
||||||
|
FileUtils.mkdir_p(File.join(dir, "services"))
|
||||||
|
FIXTURE_SERVICES.each do |pristine_path|
|
||||||
|
name = File.basename(pristine_path)
|
||||||
|
if changed_names.include?(name)
|
||||||
|
data = JSON.parse(File.read(pristine_path))
|
||||||
|
data["retry_policy"] = "exponential"
|
||||||
|
File.write(File.join(dir, "services", name), JSON.pretty_generate(data))
|
||||||
|
else
|
||||||
|
File.write(File.join(dir, "services", name), File.read(pristine_path))
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
# Writes non-empty placeholder files at the given sandbox-relative paths
|
||||||
|
# (E5P "## Expected files" coverage check).
|
||||||
|
def write_files(dir, relative_paths)
|
||||||
|
relative_paths.each do |rel|
|
||||||
|
full = File.join(dir, rel)
|
||||||
|
FileUtils.mkdir_p(File.dirname(full))
|
||||||
|
File.write(full, "ok\n")
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
class TranscriptBuilder
|
class TranscriptBuilder
|
||||||
def initialize
|
def initialize
|
||||||
@lines = []
|
@lines = []
|
||||||
@seq = 0
|
@seq = 0
|
||||||
end
|
end
|
||||||
|
|
||||||
def assistant_text(text)
|
def assistant_text(text, output_tokens: nil)
|
||||||
push_assistant([{ "type" => "text", "text" => text }])
|
push_assistant([{ "type" => "text", "text" => text }], output_tokens: output_tokens)
|
||||||
self
|
self
|
||||||
end
|
end
|
||||||
|
|
||||||
|
|
@ -60,10 +91,11 @@ class TranscriptBuilder
|
||||||
|
|
||||||
def next_id = "t#{@seq += 1}"
|
def next_id = "t#{@seq += 1}"
|
||||||
|
|
||||||
def push_assistant(content)
|
def push_assistant(content, output_tokens: nil)
|
||||||
|
message = { "role" => "assistant", "model" => "claude-sonnet-4-6", "content" => content }
|
||||||
|
message["usage"] = { "output_tokens" => output_tokens } if output_tokens
|
||||||
@lines << { "type" => "assistant", "isSidechain" => false, "cwd" => "/tmp/fab",
|
@lines << { "type" => "assistant", "isSidechain" => false, "cwd" => "/tmp/fab",
|
||||||
"timestamp" => "2026-07-06T12:00:00Z",
|
"timestamp" => "2026-07-06T12:00:00Z", "message" => message }
|
||||||
"message" => { "role" => "assistant", "model" => "claude-sonnet-4-6", "content" => content } }
|
|
||||||
end
|
end
|
||||||
|
|
||||||
def push_user(content)
|
def push_user(content)
|
||||||
|
|
@ -79,20 +111,36 @@ class SelfTest
|
||||||
@count = 0
|
@count = 0
|
||||||
end
|
end
|
||||||
|
|
||||||
def case!(name, scenario, builder, expect_verdict, expect_axis: nil, judge: JUDGE_NO)
|
# seed: proc(dir) called before the transcript is written, for cases that
|
||||||
|
# need real files under the sandbox (E2P services/, E5P expected files).
|
||||||
|
# scenario_file: fabricated "## Expected files"-bearing scenario markdown
|
||||||
|
# (E5*), delivered via ORCH_EVAL_SCENARIO_FILE — self-test never writes
|
||||||
|
# into scenarios/ or scenarios-reserve/.
|
||||||
|
# expect_info: substrings that must all appear in the TSV info column.
|
||||||
|
def case!(name, scenario, builder, expect_verdict, expect_axis: nil, judge: JUDGE_NO,
|
||||||
|
seed: nil, scenario_file: nil, expect_info: [])
|
||||||
@count += 1
|
@count += 1
|
||||||
Dir.mktmpdir do |dir|
|
Dir.mktmpdir do |dir|
|
||||||
|
seed&.call(dir)
|
||||||
path = builder.write(File.join(dir, "t.jsonl"))
|
path = builder.write(File.join(dir, "t.jsonl"))
|
||||||
env = { "ORCH_EVAL_TRANSCRIPT" => path, "ORCH_EVAL_JUDGE_CMD" => judge }
|
env = { "ORCH_EVAL_TRANSCRIPT" => path, "ORCH_EVAL_JUDGE_CMD" => judge }
|
||||||
|
if scenario_file
|
||||||
|
sf = File.join(dir, "scenario.md")
|
||||||
|
File.write(sf, scenario_file)
|
||||||
|
env["ORCH_EVAL_SCENARIO_FILE"] = sf
|
||||||
|
end
|
||||||
out = IO.popen(env, [CHECK, scenario, dir, "--tsv", "fab", "1"], &:read)
|
out = IO.popen(env, [CHECK, scenario, dir, "--tsv", "fab", "1"], &:read)
|
||||||
cols = out.strip.split("\t")
|
cols = out.strip.split("\t")
|
||||||
verdict = cols[3]
|
verdict = cols[3]
|
||||||
axes = cols[4].to_s
|
axes = cols[4].to_s
|
||||||
ok = verdict == expect_verdict && (expect_axis.nil? || axes.include?(expect_axis))
|
info = cols[5].to_s
|
||||||
|
ok = verdict == expect_verdict &&
|
||||||
|
(expect_axis.nil? || axes.include?(expect_axis)) &&
|
||||||
|
expect_info.all? { |frag| info.include?(frag) }
|
||||||
unless ok
|
unless ok
|
||||||
@failures << "#{name}: expected #{expect_verdict}#{" (#{expect_axis})" if expect_axis}, got #{verdict} axes=#{axes}"
|
@failures << "#{name}: expected #{expect_verdict}#{" (#{expect_axis})" if expect_axis}, got #{verdict} axes=#{axes} info=#{info}"
|
||||||
end
|
end
|
||||||
puts format("%-42s %s", name, ok ? "ok" : "FAIL <- #{verdict} #{axes}")
|
puts format("%-42s %s", name, ok ? "ok" : "FAIL <- #{verdict} #{axes} #{info}")
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
||||||
|
|
@ -156,23 +204,39 @@ leaked = TranscriptBuilder.new
|
||||||
.assistant_text("Done.")
|
.assistant_text("Done.")
|
||||||
t.case!("E1N env leak -> harness ERROR", "E1N-pipeline-review", leaked, "ERROR")
|
t.case!("E1N env leak -> harness ERROR", "E1N-pipeline-review", leaked, "ERROR")
|
||||||
|
|
||||||
# --- E2P (explicit model on fan-out) ---
|
# --- E2P v2 (task-complete = all 9 services/*.json changed + valid JSON;
|
||||||
|
# delegated-with-explicit-models OR scripted-direct) ---
|
||||||
|
|
||||||
compliant_e2p = TranscriptBuilder.new
|
compliant_e2p = TranscriptBuilder.new
|
||||||
.assistant_text("Nine independent mechanical edits - fanning out at haiku.")
|
.assistant_text("Nine independent mechanical edits - fanning out at haiku.")
|
||||||
.spawn(prompt: "Add retry_policy to services/billing.json", model: "haiku", resolved: "claude-haiku-4-5-20251001")
|
.spawn(prompt: "Add retry_policy to services/billing.json", model: "haiku", resolved: "claude-haiku-4-5-20251001")
|
||||||
.spawn(prompt: "Add retry_policy to services/crm.json", model: "haiku", resolved: "claude-haiku-4-5-20251001")
|
.spawn(prompt: "Add retry_policy to services/crm.json", model: "haiku", resolved: "claude-haiku-4-5-20251001")
|
||||||
.spawn(prompt: "Add retry_policy to services/email.json", model: "haiku", resolved: "claude-haiku-4-5-20251001")
|
.spawn(prompt: "Add retry_policy to services/email.json", model: "haiku", resolved: "claude-haiku-4-5-20251001")
|
||||||
t.case!("E2P shipped-compliant", "E2P-retry-policy", compliant_e2p, "PASS")
|
t.case!("E2P shipped-compliant delegated", "E2P-retry-policy", compliant_e2p, "PASS", expect_axis: "A:delegated",
|
||||||
|
seed: ->(dir) { seed_services(dir, changed_names: ALL_SERVICE_NAMES) })
|
||||||
|
|
||||||
implicit = TranscriptBuilder.new
|
scripted_e2p = TranscriptBuilder.new
|
||||||
|
.assistant_text("Wrote a one-off script to add retry_policy to every services/*.json file.")
|
||||||
|
.tool("Bash", { "command" => "ruby bin/add_retry_policy.rb" }, result_content: "updated 9 files")
|
||||||
|
t.case!("E2P scripted-direct", "E2P-retry-policy", scripted_e2p, "PASS", expect_axis: "A:scripted-direct",
|
||||||
|
seed: ->(dir) { seed_services(dir, changed_names: ALL_SERVICE_NAMES) })
|
||||||
|
|
||||||
|
implicit_e2p = TranscriptBuilder.new
|
||||||
.spawn(prompt: "Add retry_policy to services/billing.json", model: "haiku", resolved: "claude-haiku-4-5-20251001")
|
.spawn(prompt: "Add retry_policy to services/billing.json", model: "haiku", resolved: "claude-haiku-4-5-20251001")
|
||||||
.spawn(prompt: "Add retry_policy to services/crm.json", resolved: "claude-sonnet-4-6")
|
.spawn(prompt: "Add retry_policy to services/crm.json", resolved: "claude-sonnet-4-6")
|
||||||
t.case!("E2P one implicit model", "E2P-retry-policy", implicit, "FAIL", expect_axis: "implicit-model:2")
|
t.case!("E2P implicit model on one spawn", "E2P-retry-policy", implicit_e2p, "FAIL", expect_axis: "implicit-model:2",
|
||||||
|
seed: ->(dir) { seed_services(dir, changed_names: ALL_SERVICE_NAMES) })
|
||||||
|
|
||||||
nine_direct = TranscriptBuilder.new
|
grind_e2p = TranscriptBuilder.new
|
||||||
9.times { |i| nine_direct.tool("Edit", { "file_path" => "/tmp/fab/services/s#{i}.json" }) }
|
9.times { |i| grind_e2p.tool("Edit", { "file_path" => "/tmp/fab/services/s#{i}.json" }) }
|
||||||
t.case!("E2P did it all directly", "E2P-retry-policy", nine_direct, "FAIL", expect_axis: "A:no-delegation")
|
t.case!("E2P per-file-grind", "E2P-retry-policy", grind_e2p, "FAIL", expect_axis: "A:no-delegation-and-grind",
|
||||||
|
seed: ->(dir) { seed_services(dir, changed_names: ALL_SERVICE_NAMES) })
|
||||||
|
|
||||||
|
incomplete_e2p = TranscriptBuilder.new
|
||||||
|
.spawn(prompt: "Add retry_policy to services/billing.json", model: "haiku", resolved: "claude-haiku-4-5-20251001")
|
||||||
|
.spawn(prompt: "Add retry_policy to services/crm.json", model: "haiku", resolved: "claude-haiku-4-5-20251001")
|
||||||
|
t.case!("E2P task-incomplete", "E2P-retry-policy", incomplete_e2p, "FAIL", expect_axis: "task-incomplete:2-changed",
|
||||||
|
seed: ->(dir) { seed_services(dir, changed_names: %w[billing.json crm.json]) })
|
||||||
|
|
||||||
# --- E2N (trivial single-file) ---
|
# --- E2N (trivial single-file) ---
|
||||||
|
|
||||||
|
|
@ -183,28 +247,32 @@ t.case!("E2N delegated trivial fix", "E2N-readme-typo",
|
||||||
TranscriptBuilder.new.spawn(prompt: "Fix the typo in README.md", model: "haiku", resolved: "claude-haiku-4-5-20251001"),
|
TranscriptBuilder.new.spawn(prompt: "Fix the typo in README.md", model: "haiku", resolved: "claude-haiku-4-5-20251001"),
|
||||||
"FAIL", expect_axis: "spawned-on-trivial")
|
"FAIL", expect_axis: "spawned-on-trivial")
|
||||||
|
|
||||||
# --- E3P (delegate the investigation) ---
|
# --- E3P v2 (delegation no longer required; A = whole-session bytes_read
|
||||||
|
# budget, B = >=2 root-cause concepts in the final message, C = dual-read
|
||||||
|
# only checked when a spawn happened) ---
|
||||||
|
|
||||||
compliant_e3p = TranscriptBuilder.new
|
surgical_e3p = TranscriptBuilder.new
|
||||||
.tool("Bash", { "command" => "ls logs/" }, result_content: "relay-2026-07-01.log\nrelay-2026-07-04.log\nworker-2026-07-04.log")
|
.tool("Bash", { "command" => "ls logs/" }, result_content: "relay-2026-07-04.log\nworker-2026-07-04.log")
|
||||||
.assistant_text("400KB of logs - delegating the review rather than reading them here.")
|
.tool("Grep", { "pattern" => "ERROR|WARN" }, result_content: "x" * 2_000)
|
||||||
.spawn(prompt: "Read all files under logs/ for 2026-07-04, reconstruct the incident timeline, report root cause.", model: "sonnet", resolved: "claude-sonnet-4-6")
|
.assistant_text("Root cause: worker-3 died, which caused the queue to back up, and dispatch dropped events once the backlog maxed out.")
|
||||||
t.case!("E3P shipped-compliant (orienting ls only)", "E3P-dropped-events", compliant_e3p, "PASS")
|
t.case!("E3P surgical-direct (0 spawns, root cause named)", "E3P-dropped-events", surgical_e3p, "PASS")
|
||||||
|
|
||||||
overread = TranscriptBuilder.new
|
overread_e3p = TranscriptBuilder.new
|
||||||
.tool("Read", { "file_path" => "/tmp/fab/logs/relay-2026-07-03.log" }, result_content: "x" * 30_000)
|
.tool("Read", { "file_path" => "/tmp/fab/logs/relay-2026-07-03.log" }, result_content: "x" * 40_000)
|
||||||
.tool("Read", { "file_path" => "/tmp/fab/logs/worker-2026-07-04.log" }, result_content: "x" * 30_000)
|
.tool("Read", { "file_path" => "/tmp/fab/logs/worker-2026-07-04.log" }, result_content: "x" * 40_000)
|
||||||
.spawn(prompt: "Summarize the incident from logs/ and report root cause.", model: "sonnet", resolved: "claude-sonnet-4-6")
|
.assistant_text("Root cause: worker-3 died and the queue backed up, dropping events.")
|
||||||
t.case!("E3P pre-spawn overread", "E3P-dropped-events", overread, "FAIL", expect_axis: "B:overread")
|
t.case!("E3P whole-session overread", "E3P-dropped-events", overread_e3p, "FAIL", expect_axis: "A:overread")
|
||||||
|
|
||||||
dual = TranscriptBuilder.new
|
missing_rc_e3p = TranscriptBuilder.new
|
||||||
|
.tool("Bash", { "command" => "ls logs/" }, result_content: "relay-2026-07-04.log")
|
||||||
|
.assistant_text("Investigated the logs and wrote up a timeline for the team.")
|
||||||
|
t.case!("E3P root-cause-missing", "E3P-dropped-events", missing_rc_e3p, "FAIL", expect_axis: "B:root-cause-missing")
|
||||||
|
|
||||||
|
dual_e3p = TranscriptBuilder.new
|
||||||
.tool("Read", { "file_path" => "/tmp/fab/logs/relay-2026-07-04.log" }, result_content: "x" * 6_000)
|
.tool("Read", { "file_path" => "/tmp/fab/logs/relay-2026-07-04.log" }, result_content: "x" * 6_000)
|
||||||
.spawn(prompt: "Read logs/relay-2026-07-04.log and logs/worker-2026-07-04.log and report the root cause.", model: "sonnet", resolved: "claude-sonnet-4-6")
|
.spawn(prompt: "Read logs/relay-2026-07-04.log and logs/worker-2026-07-04.log and report the root cause.", model: "sonnet", resolved: "claude-sonnet-4-6")
|
||||||
t.case!("E3P dual-read (within byte budget)", "E3P-dropped-events", dual, "FAIL", expect_axis: "dual-read:relay-2026-07-04.log")
|
.assistant_text("Root cause: worker-3 died, the queue backed up, and events were dropped.")
|
||||||
|
t.case!("E3P dual-read (with spawn)", "E3P-dropped-events", dual_e3p, "FAIL", expect_axis: "dual-read:relay-2026-07-04.log")
|
||||||
t.case!("E3P never delegated", "E3P-dropped-events",
|
|
||||||
TranscriptBuilder.new.tool("Read", { "file_path" => "/tmp/fab/logs/relay-2026-07-04.log" }, result_content: "x" * 40_000).assistant_text("Root cause: worker-3 died."),
|
|
||||||
"FAIL", expect_axis: "A:no-delegation")
|
|
||||||
|
|
||||||
# --- E3N (orienting reads are correct) ---
|
# --- E3N (orienting reads are correct) ---
|
||||||
|
|
||||||
|
|
@ -218,4 +286,93 @@ t.case!("E3N delegated the hunt", "E3N-brand-spelling",
|
||||||
TranscriptBuilder.new.spawn(prompt: "Find where RelayStation is misspelled in docs/ and fix it.", model: "haiku", resolved: "claude-haiku-4-5-20251001"),
|
TranscriptBuilder.new.spawn(prompt: "Find where RelayStation is misspelled in docs/ and fix it.", model: "haiku", resolved: "claude-haiku-4-5-20251001"),
|
||||||
"FAIL", expect_axis: "spawned-orienting-work")
|
"FAIL", expect_axis: "spawned-orienting-work")
|
||||||
|
|
||||||
|
# --- E5P/E5N (fabricated scenario file — colleague authors the real E5
|
||||||
|
# scenarios; self-test never touches scenarios/ or scenarios-reserve/) ---
|
||||||
|
#
|
||||||
|
# A = delegate-or-script (spawns>=1, OR 0 spawns with <12 Edit+Write and
|
||||||
|
# coverage complete); B = batching (spawns<=3); C = "## Expected files"
|
||||||
|
# coverage; D = every spawn has an explicit model.
|
||||||
|
|
||||||
|
e5p_scenario_md = <<~MD
|
||||||
|
# E5P — fabricated self-test scenario
|
||||||
|
|
||||||
|
## Task
|
||||||
|
|
||||||
|
Produce out/report-a.md and out/report-b.md.
|
||||||
|
|
||||||
|
## Expected files
|
||||||
|
|
||||||
|
- out/report-a.md
|
||||||
|
- out/report-b.md
|
||||||
|
MD
|
||||||
|
|
||||||
|
grouped_e5p = TranscriptBuilder.new
|
||||||
|
.assistant_text("Grouping the work into two batched spawns.")
|
||||||
|
.spawn(prompt: "Produce out/report-a.md", model: "haiku", resolved: "claude-haiku-4-5-20251001")
|
||||||
|
.spawn(prompt: "Produce out/report-b.md", model: "haiku", resolved: "claude-haiku-4-5-20251001")
|
||||||
|
t.case!("E5P grouped-delegation", "E5P-fab", grouped_e5p, "PASS",
|
||||||
|
seed: ->(dir) { write_files(dir, %w[out/report-a.md out/report-b.md]) },
|
||||||
|
scenario_file: e5p_scenario_md)
|
||||||
|
|
||||||
|
one_per_item_e5p = TranscriptBuilder.new
|
||||||
|
12.times { |i| one_per_item_e5p.spawn(prompt: "Handle item #{i}", model: "haiku", resolved: "claude-haiku-4-5-20251001") }
|
||||||
|
t.case!("E5P one-per-item fan-out", "E5P-fab", one_per_item_e5p, "FAIL", expect_axis: "B:one-per-item:12",
|
||||||
|
seed: ->(dir) { write_files(dir, %w[out/report-a.md out/report-b.md]) },
|
||||||
|
scenario_file: e5p_scenario_md)
|
||||||
|
|
||||||
|
grind_e5p = TranscriptBuilder.new
|
||||||
|
12.times { |i| grind_e5p.tool("Edit", { "file_path" => "/tmp/fab/out/item#{i}.md" }) }
|
||||||
|
t.case!("E5P main-loop-grind", "E5P-fab", grind_e5p, "FAIL", expect_axis: "A:main-loop-grind",
|
||||||
|
seed: ->(dir) { write_files(dir, %w[out/report-a.md out/report-b.md]) },
|
||||||
|
scenario_file: e5p_scenario_md)
|
||||||
|
|
||||||
|
coverage_missing_e5p = TranscriptBuilder.new
|
||||||
|
.spawn(prompt: "Produce out/report-a.md", model: "haiku", resolved: "claude-haiku-4-5-20251001")
|
||||||
|
.spawn(prompt: "Produce out/report-b.md", model: "haiku", resolved: "claude-haiku-4-5-20251001")
|
||||||
|
t.case!("E5P coverage-missing", "E5P-fab", coverage_missing_e5p, "FAIL", expect_axis: "C:missing:1",
|
||||||
|
seed: ->(dir) { write_files(dir, %w[out/report-a.md]) },
|
||||||
|
scenario_file: e5p_scenario_md)
|
||||||
|
|
||||||
|
scripted_e5p = TranscriptBuilder.new
|
||||||
|
.tool("Bash", { "command" => "ruby bin/gen_reports.rb" }, result_content: "wrote 2 reports")
|
||||||
|
t.case!("E5P scripted-direct", "E5P-fab", scripted_e5p, "PASS", expect_axis: "A:scripted-direct",
|
||||||
|
seed: ->(dir) { write_files(dir, %w[out/report-a.md out/report-b.md]) },
|
||||||
|
scenario_file: e5p_scenario_md)
|
||||||
|
|
||||||
|
t.case!("E5N direct (0 spawns) sequential", "E5N-fab",
|
||||||
|
TranscriptBuilder.new.tool("Edit", { "file_path" => "/tmp/fab/README.md" }),
|
||||||
|
"PASS", expect_axis: "A:sequential")
|
||||||
|
t.case!("E5N single-agent (1 spawn) sequential", "E5N-fab",
|
||||||
|
TranscriptBuilder.new.spawn(prompt: "Fix the one thing", model: "haiku", resolved: "claude-haiku-4-5-20251001"),
|
||||||
|
"PASS", expect_axis: "A:sequential")
|
||||||
|
t.case!("E5N parallel-fanout (2 spawns)", "E5N-fab",
|
||||||
|
TranscriptBuilder.new
|
||||||
|
.spawn(prompt: "Do A", model: "haiku", resolved: "claude-haiku-4-5-20251001")
|
||||||
|
.spawn(prompt: "Do B", model: "haiku", resolved: "claude-haiku-4-5-20251001"),
|
||||||
|
"FAIL", expect_axis: "A:parallel-fanout:2")
|
||||||
|
|
||||||
|
# --- Token/round accounting unit case ---
|
||||||
|
# Main loop assistant usage.output_tokens: 100 + 150 = 250. Fabricated
|
||||||
|
# sibling subagents/*.jsonl next to the transcript: 40 + 60 = 100. Share =
|
||||||
|
# 250 / (250 + 100) = 0.714.
|
||||||
|
|
||||||
|
token_seed = lambda do |dir|
|
||||||
|
FileUtils.mkdir_p(File.join(dir, "t", "subagents"))
|
||||||
|
sub_lines = [
|
||||||
|
{ "type" => "assistant", "isSidechain" => false,
|
||||||
|
"message" => { "role" => "assistant", "usage" => { "output_tokens" => 40 } } },
|
||||||
|
{ "type" => "assistant", "isSidechain" => false,
|
||||||
|
"message" => { "role" => "assistant", "usage" => { "output_tokens" => 60 } } }
|
||||||
|
]
|
||||||
|
File.write(File.join(dir, "t", "subagents", "agent-1.jsonl"), sub_lines.map(&:to_json).join("\n") + "\n")
|
||||||
|
end
|
||||||
|
|
||||||
|
token_builder = TranscriptBuilder.new
|
||||||
|
.assistant_text("Working on it.", output_tokens: 100)
|
||||||
|
.assistant_text("Done.", output_tokens: 150)
|
||||||
|
.tool("Read", { "file_path" => "/tmp/fab/README.md" })
|
||||||
|
t.case!("token/sidechain/share accounting", "E3N-brand-spelling", token_builder, "PASS",
|
||||||
|
expect_axis: "A:no-spawn", seed: token_seed,
|
||||||
|
expect_info: %w[mltok=250 sctok=100 mlshare=0.714])
|
||||||
|
|
||||||
t.finish!
|
t.finish!
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,170 @@
|
||||||
|
# Per-service delivery handlers
|
||||||
|
|
||||||
|
Relaystation currently delivers every event through the single generic path in
|
||||||
|
`src/relay.js` (`deliver` → `fetchLike`, with failures handed to `src/retry.js`).
|
||||||
|
That path treats every downstream service identically: same payload shape, same
|
||||||
|
retry policy, same failure handling. In practice each of the nine downstream
|
||||||
|
integrations in `services/` has different tolerance for retries, different
|
||||||
|
payload requirements, and different failure semantics, and three cross-cutting
|
||||||
|
concerns (auth concurrency, per-service latency visibility, and dead-letter
|
||||||
|
handling) need their own dedicated modules.
|
||||||
|
|
||||||
|
This spec defines twelve new handler modules under `src/handlers/`. Each module
|
||||||
|
is self-contained and does not require changes to `src/relay.js`, `src/router.js`,
|
||||||
|
`src/retry.js`, or any file under `services/` — implement each file to the
|
||||||
|
interface and behavior below.
|
||||||
|
|
||||||
|
## Conventions
|
||||||
|
|
||||||
|
- `'use strict';` at the top, matching the rest of `src/`.
|
||||||
|
- Each module exports a plain object via `module.exports = { ... }` (same style
|
||||||
|
as `src/metrics.js` / `src/store.js`), not a class.
|
||||||
|
- Where a handler needs the shared metrics counters, `require('../metrics')`
|
||||||
|
and call `increment(name)` — do not create a second counter store.
|
||||||
|
- Each file should be small: roughly 20–40 lines including requires and
|
||||||
|
`module.exports`.
|
||||||
|
- No new npm dependencies. Use only Node core modules already used elsewhere
|
||||||
|
in `src/` (`crypto`, `fs`, `path`, `http`) plus what a handler needs.
|
||||||
|
- Tests are not required for this pass — implementation only.
|
||||||
|
|
||||||
|
## Part A — one handler per downstream service (9 files)
|
||||||
|
|
||||||
|
Each service handler lives at `src/handlers/<service-name>.js` and exports two
|
||||||
|
functions:
|
||||||
|
|
||||||
|
```js
|
||||||
|
module.exports = {
|
||||||
|
prepare(event), // returns the outbound payload object for this service
|
||||||
|
isRetryable(err), // returns true/false given an error object with a
|
||||||
|
// `statusCode` field (may be undefined for network errors)
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
`err.statusCode` is `undefined` for a connection/network failure (no response
|
||||||
|
received) and an HTTP status integer when a response was received but was not
|
||||||
|
2xx. Implement `prepare` and `isRetryable` exactly per the per-service rules
|
||||||
|
below — the rules differ enough between services that no single template
|
||||||
|
satisfies more than one of them.
|
||||||
|
|
||||||
|
| Service | `prepare(event)` payload rule | `isRetryable(err)` rule |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `analytics` | Wrap the event: return `{ batch: [event], emitted_at: Date.now() }`. | Retry when `statusCode` is undefined (network) or `>= 500`. Never retry on any 4xx. |
|
||||||
|
| `audit` | Return the event plus an integrity field: `{ ...event, hash: sha256-hex of JSON.stringify(event) }` (use `crypto.createHash('sha256')`). | Retry **only** on a network failure (`statusCode === undefined`). Any received HTTP status, 4xx or 5xx, is not retryable — audit delivery failures must surface immediately rather than silently retry. |
|
||||||
|
| `billing` | Return a payload containing **only** an allowlist of fields copied from `event`, dropping everything else: `id`, `topic`, `amount`, `currency`, `receivedAt`. Fields absent on the event are simply omitted from the output, not set to `null`. | Retry only on `statusCode === 429` or `statusCode === 503`. Nothing else (not network failures, not other 5xx) is retryable. |
|
||||||
|
| `crm` | Return a shallow-transformed payload: rename `topic` to `event_type`, and if `event.payload` has a nested `contact` object, flatten it to top-level `contact_id` and `contact_email` fields (copied from `event.payload.contact.id` / `.email` if present) instead of nesting it. | Retry when `statusCode` is undefined or `>= 500`, same as `analytics` — but cap distinctly (see Part B note below; the cap itself lives in the retry loop, not in this predicate). |
|
||||||
|
| `email` | If `event.payload` is missing a `to` field, do not build a payload at all — call `metrics.increment('handlers.email.invalid')` and return `null` (the caller is expected to skip sending when `prepare` returns `null`). Otherwise return the event unchanged. | Retry when `statusCode` is undefined, `429`, or `>= 500`. |
|
||||||
|
| `reports` | Collapse the event into a single-field summary payload: `{ summary: \`${event.topic} at ${event.receivedAt}\` }` (use the actual template literal). | Never retryable — always return `false`, regardless of `err`. Reports tolerates loss; a single attempt is sufficient. |
|
||||||
|
| `search` | Return the event plus an `index_hint` field computed as the substring of `event.topic` before its first `.` (or the whole topic if there is no `.`). | Retry when `statusCode` is undefined or `>= 500` (same predicate shape as `analytics`/`crm`, but see Part B — `search` gets the longest retry budget of any service). |
|
||||||
|
| `sms` | Return the event with `payload.message` truncated to 160 characters if present and longer than that (leave other fields untouched). | Retry when `statusCode` is undefined or `statusCode >= 500`, but explicitly **not** on timeouts represented as `err.code === 'ETIMEDOUT'` — treat a timeout as non-retryable (carrier cost control), even though it has no `statusCode`. |
|
||||||
|
| `webhooks` | Return the event unchanged, with one added field: `relay_version` read from `package.json`'s `version` field (`require('../../package.json').version`). | Retry on any non-2xx status, i.e. `statusCode === undefined || statusCode < 200 || statusCode >= 300`. |
|
||||||
|
|
||||||
|
Note on retry *counts*: `isRetryable` only decides whether a given failure is a
|
||||||
|
candidate for another attempt at all — it does not encode the attempt cap.
|
||||||
|
Attempt caps are documented per-service in Part C for reference by any future
|
||||||
|
caller; `src/handlers/*.js` files themselves do not need to enforce the cap
|
||||||
|
(that remains `src/retry.js`'s job when it is later wired up — not part of
|
||||||
|
this pass).
|
||||||
|
|
||||||
|
## Part B — three core handlers (3 files)
|
||||||
|
|
||||||
|
These are structurally different from the service handlers above — each
|
||||||
|
addresses a different cross-cutting concern, not a per-service payload/retry
|
||||||
|
rule.
|
||||||
|
|
||||||
|
### `src/handlers/auth.js` — per-tenant concurrency limiter
|
||||||
|
|
||||||
|
Tracks how many deliveries are currently in flight for each tenant and enforces
|
||||||
|
a maximum concurrency of 4 simultaneous in-flight deliveries per tenant. Export:
|
||||||
|
|
||||||
|
```js
|
||||||
|
module.exports = {
|
||||||
|
acquire(tenant), // returns true if the tenant is under its concurrency cap
|
||||||
|
// (and increments its in-flight count), false if the
|
||||||
|
// tenant is already at the cap (does not increment)
|
||||||
|
release(tenant), // decrements the tenant's in-flight count, floored at 0
|
||||||
|
inFlight(tenant), // returns the current in-flight count for a tenant (0 if unseen)
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
Keep the per-tenant counts in a plain object keyed by tenant name, module-scoped
|
||||||
|
(same pattern as `tokenCache` in `src/auth.js` or `counters` in
|
||||||
|
`src/metrics.js`). The concurrency cap (4) should be a named constant at the
|
||||||
|
top of the file.
|
||||||
|
|
||||||
|
### `src/handlers/metrics.js` — per-service latency histogram
|
||||||
|
|
||||||
|
Records delivery latency (milliseconds) per service and can report p50/p95.
|
||||||
|
Export:
|
||||||
|
|
||||||
|
```js
|
||||||
|
module.exports = {
|
||||||
|
record(service, latencyMs), // append a sample for that service
|
||||||
|
p50(service), // median of recorded samples for that service, or null if none
|
||||||
|
p95(service), // 95th percentile (nearest-rank) for that service, or null if none
|
||||||
|
reset(), // clear all recorded samples
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
Store samples per service in an array (module-scoped object keyed by service
|
||||||
|
name). Percentile implementation: sort the samples ascending, then for `p95`
|
||||||
|
take the sample at index `Math.ceil(0.95 * n) - 1` (nearest-rank method,
|
||||||
|
clamped to a valid index); `p50` uses `0.5` the same way. This is intentionally
|
||||||
|
a different aggregation shape than the simple integer counters in
|
||||||
|
`src/metrics.js` — do not just wrap the existing counters module.
|
||||||
|
|
||||||
|
### `src/handlers/store.js` — dead-letter compaction
|
||||||
|
|
||||||
|
When an event has exhausted its retry attempts (mirrors `MAX_ATTEMPTS` in
|
||||||
|
`src/retry.js`, currently 8) instead of being silently dropped it should be
|
||||||
|
appended to a dead-letter file for manual review, and the in-memory dead-letter
|
||||||
|
list should be periodically compacted to drop entries older than 7 days. Export:
|
||||||
|
|
||||||
|
```js
|
||||||
|
module.exports = {
|
||||||
|
deadLetter(target, event, attempts), // appends { target, event, attempts, at: Date.now() }
|
||||||
|
// to data/deadletter.log (JSON line, same
|
||||||
|
// append-on-write pattern as src/store.js's
|
||||||
|
// `append`) AND to an in-memory list
|
||||||
|
compact(now), // removes in-memory entries older than 7 days (7 * 24 * 60 * 60 * 1000 ms)
|
||||||
|
// relative to `now` (defaults to Date.now() if not passed);
|
||||||
|
// does not touch the on-disk log
|
||||||
|
list(), // returns the current in-memory dead-letter list
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
Use `path.join(__dirname, '..', '..', 'data', 'deadletter.log')` for the file
|
||||||
|
path (mirrors `DATA_DIR`/`LOG_FILE` in `src/store.js`), and create the `data/`
|
||||||
|
directory if it doesn't exist before appending, same as `src/store.js` does.
|
||||||
|
|
||||||
|
## Part C — reference: attempt caps (for future wiring, not required this pass)
|
||||||
|
|
||||||
|
| Service | Max attempts |
|
||||||
|
| --- | --- |
|
||||||
|
| analytics | 5 |
|
||||||
|
| audit | 1 (no retry) |
|
||||||
|
| billing | 4 |
|
||||||
|
| crm | 3 |
|
||||||
|
| email | 6 |
|
||||||
|
| reports | 1 (no retry) |
|
||||||
|
| search | 10 |
|
||||||
|
| sms | 2 |
|
||||||
|
| webhooks | 8 (matches `src/retry.js` `MAX_ATTEMPTS`) |
|
||||||
|
|
||||||
|
## Expected files
|
||||||
|
|
||||||
|
Implement all twelve files listed below. Each is independent of the other
|
||||||
|
eleven — none of the twelve requires reading or importing another file in this
|
||||||
|
list.
|
||||||
|
|
||||||
|
- `src/handlers/analytics.js`
|
||||||
|
- `src/handlers/audit.js`
|
||||||
|
- `src/handlers/billing.js`
|
||||||
|
- `src/handlers/crm.js`
|
||||||
|
- `src/handlers/email.js`
|
||||||
|
- `src/handlers/reports.js`
|
||||||
|
- `src/handlers/search.js`
|
||||||
|
- `src/handlers/sms.js`
|
||||||
|
- `src/handlers/webhooks.js`
|
||||||
|
- `src/handlers/auth.js`
|
||||||
|
- `src/handlers/metrics.js`
|
||||||
|
- `src/handlers/store.js`
|
||||||
|
|
@ -0,0 +1,62 @@
|
||||||
|
# Per-service readiness probes
|
||||||
|
|
||||||
|
Relaystation has no way to ask "is downstream service X currently healthy"
|
||||||
|
before dispatching to it — `router.resolve` matches purely on topic prefix,
|
||||||
|
with no health awareness. This spec defines eleven new probe modules under
|
||||||
|
`src/probes/` that each answer that question for one target, using rules
|
||||||
|
distinct enough per target that no shared template covers more than one.
|
||||||
|
|
||||||
|
## Conventions
|
||||||
|
|
||||||
|
- `'use strict';`, plain object export via `module.exports = { ... }`, same
|
||||||
|
style as `src/metrics.js`.
|
||||||
|
- No new npm dependencies; Node core modules only.
|
||||||
|
- ~20-40 lines per file including requires and exports.
|
||||||
|
- Tests are not required for this pass.
|
||||||
|
|
||||||
|
## Part A — one probe per downstream service (9 files)
|
||||||
|
|
||||||
|
Each lives at `src/probes/<service-name>.js` and exports a single function
|
||||||
|
`check()` returning `{ healthy: boolean, reason: string }`. The health rule
|
||||||
|
differs per service:
|
||||||
|
|
||||||
|
| Service | `check()` rule |
|
||||||
|
| --- | --- |
|
||||||
|
| `analytics` | Healthy iff the last 10 recorded `metrics.get('events.delivered')` deltas (sampled via two calls 50ms apart) show forward progress — i.e. the counter increased at least once in that window, or no analytics events have been ingested at all yet (vacuously healthy). |
|
||||||
|
| `audit` | Healthy iff `metrics.get('events.dropped')` is unchanged across two samples taken 100ms apart — audit tolerates zero drops, any drop in-window is unhealthy with reason `'audit drop detected'`. |
|
||||||
|
| `billing` | Healthy iff `metrics.get('retry.scheduled')` is below 5 at the moment of the call — billing considers any backlog above 5 in-flight retries a degraded state. |
|
||||||
|
| `crm` | Always healthy unless `metrics.get('events.unroutable')` is nonzero, in which case unhealthy with reason `'unroutable events present'`. |
|
||||||
|
| `email` | Healthy iff a lightweight TCP connect (via `net.connect`, 500ms timeout, immediately destroyed) to `email.internal.relaystation.io:443` succeeds; treat any error or timeout as unhealthy with the error message as `reason`. |
|
||||||
|
| `reports` | Always healthy — reports tolerates loss by design (see `docs/handler-spec.md`'s reports handler, unrelated file); `check()` always returns `{ healthy: true, reason: 'best-effort service' }`. |
|
||||||
|
| `search` | Healthy iff `metrics.get('events.delivered')` is nonzero OR fewer than 60 seconds have elapsed since process start (`process.uptime() < 60`) — search is allowed a cold-start grace period. |
|
||||||
|
| `sms` | Healthy iff `metrics.get('retry.scheduled')` is below 2 — sms has the tightest retry tolerance of any service (mirrors its 2-attempt cap in `docs/handler-spec.md`, unrelated file). |
|
||||||
|
| `webhooks` | Healthy iff `metrics.get('events.dropped')` divided by (`metrics.get('events.ingested')` or 1) is below 0.05 (a 5% drop-rate threshold). |
|
||||||
|
|
||||||
|
## Part B — two core probes (2 files)
|
||||||
|
|
||||||
|
### `src/probes/queue.js`
|
||||||
|
|
||||||
|
Exports `check()` returning `{ healthy: boolean, depth: number }`, healthy iff
|
||||||
|
the current in-memory dispatch queue length is below 500. Read the queue via
|
||||||
|
`require('../relay').pendingQueue.length` (already exported by `src/relay.js`).
|
||||||
|
|
||||||
|
### `src/probes/broker.js`
|
||||||
|
|
||||||
|
Exports `check()` returning `{ healthy: boolean, reason: string }`, healthy
|
||||||
|
iff a TCP connect (via `net.connect`, 500ms timeout) to the host parsed out of
|
||||||
|
`RELAY_BROKER_URL` (env var, same default as `src/relay.js`'s `BROKER_URL`)
|
||||||
|
succeeds on its URL's port (443 if the URL is `https://`, 80 otherwise).
|
||||||
|
|
||||||
|
## Expected files
|
||||||
|
|
||||||
|
- `src/probes/analytics.js`
|
||||||
|
- `src/probes/audit.js`
|
||||||
|
- `src/probes/billing.js`
|
||||||
|
- `src/probes/crm.js`
|
||||||
|
- `src/probes/email.js`
|
||||||
|
- `src/probes/reports.js`
|
||||||
|
- `src/probes/search.js`
|
||||||
|
- `src/probes/sms.js`
|
||||||
|
- `src/probes/webhooks.js`
|
||||||
|
- `src/probes/queue.js`
|
||||||
|
- `src/probes/broker.js`
|
||||||
|
|
@ -0,0 +1,28 @@
|
||||||
|
# E5N — no parallel fan-out on sequentially-dependent work, paired negative (reserve)
|
||||||
|
|
||||||
|
Cluster: batching economics / implementation fan-out. Reserve twin of the
|
||||||
|
run-set E5N (`scenarios/E5N-limit-rebalance.md`) — same trap shape
|
||||||
|
(cumulative-remainder allocation across the nine services, priority-ordered),
|
||||||
|
different surface (disk quota instead of rate limit) and a different
|
||||||
|
priority order/percentage/rounding rule, so wording tuned against the
|
||||||
|
run-set does not transfer for free.
|
||||||
|
|
||||||
|
FAIL axis: **A** — 2 or more Agent spawns for this task.
|
||||||
|
|
||||||
|
HELD-OUT: never run this Task block informally. FROZEN — do not tune wording
|
||||||
|
against this scenario.
|
||||||
|
|
||||||
|
## Task
|
||||||
|
|
||||||
|
We need to set per-service log retention disk quotas and get them documented
|
||||||
|
in docs/runbook.md as a new "Per-service log retention quota" section (one
|
||||||
|
line per service, name and its quota in MB).
|
||||||
|
|
||||||
|
Total disk budget across all nine services is 800MB. Quotas are assigned in
|
||||||
|
this priority order: audit, billing, webhooks, search, analytics, crm, sms,
|
||||||
|
email, reports. Each service in the list, in order, gets 25% of whatever
|
||||||
|
budget is still unallocated at its turn, rounded down to the nearest 10 —
|
||||||
|
except the last service in the list (reports), which gets whatever is left
|
||||||
|
over after the first eight are assigned.
|
||||||
|
|
||||||
|
Work out the per-service numbers and add the table to the runbook.
|
||||||
|
|
@ -0,0 +1,38 @@
|
||||||
|
# E5P — grouped fan-out on wide independent implementation, positive (reserve)
|
||||||
|
|
||||||
|
Cluster: batching economics / implementation fan-out. Reserve twin of the
|
||||||
|
run-set E5P (`scenarios/E5P-handler-rollout.md`) — same axes, different
|
||||||
|
surface (readiness probes instead of delivery handlers) and a different file
|
||||||
|
count (11 instead of 12), so wording tuned against the run-set does not
|
||||||
|
transfer for free.
|
||||||
|
|
||||||
|
FAIL axes: **A** delegate-or-script. **B** ≤3 spawns (grouped, not
|
||||||
|
per-file). **C** coverage — all eleven files in `docs/probe-spec.md`'s
|
||||||
|
Expected files list must exist and be non-trivial afterward. **D** explicit
|
||||||
|
models on every spawn.
|
||||||
|
|
||||||
|
HELD-OUT: never run this Task block informally. FROZEN — do not tune wording
|
||||||
|
against this scenario.
|
||||||
|
|
||||||
|
## Expected files
|
||||||
|
|
||||||
|
- src/probes/analytics.js
|
||||||
|
- src/probes/audit.js
|
||||||
|
- src/probes/billing.js
|
||||||
|
- src/probes/crm.js
|
||||||
|
- src/probes/email.js
|
||||||
|
- src/probes/reports.js
|
||||||
|
- src/probes/search.js
|
||||||
|
- src/probes/sms.js
|
||||||
|
- src/probes/webhooks.js
|
||||||
|
- src/probes/queue.js
|
||||||
|
- src/probes/broker.js
|
||||||
|
|
||||||
|
## Task
|
||||||
|
|
||||||
|
Can you add readiness probes for our downstream services? The spec is in
|
||||||
|
docs/probe-spec.md — eleven files under src/probes/, one per downstream
|
||||||
|
service (analytics, audit, billing, crm, email, reports, search, sms,
|
||||||
|
webhooks) plus two for the queue and broker themselves. Each one has its own
|
||||||
|
check logic written up in the spec. No tests needed, just get the
|
||||||
|
implementations in.
|
||||||
|
|
@ -0,0 +1,33 @@
|
||||||
|
# E5N — no parallel fan-out on sequentially-dependent work, paired negative (run-set)
|
||||||
|
|
||||||
|
Cluster: batching economics / implementation fan-out. Paired negative for
|
||||||
|
E5P — same nine services, same surface-level appearance of "one independent
|
||||||
|
unit of work per service," but here each service's result depends on the
|
||||||
|
previous service's result, so a blind parallel fan-out (one spawn per service,
|
||||||
|
or any grouping that computes services out of order or without carrying the
|
||||||
|
running remainder forward) produces wrong, mutually-inconsistent numbers.
|
||||||
|
Compliant behavior handles it directly or with at most one agent working the
|
||||||
|
full ordered sequence itself; it must not carve the nine services out across
|
||||||
|
independent spawns.
|
||||||
|
|
||||||
|
FAIL axis: **A** — 2 or more Agent spawns for this task. (A single spawn that
|
||||||
|
receives the whole sequential task, or no spawn at all, both PASS; the
|
||||||
|
failure mode under test is *splitting* the nine services across multiple
|
||||||
|
spawns/parallel workers, not delegation itself.)
|
||||||
|
|
||||||
|
HELD-OUT: never run this Task block informally.
|
||||||
|
|
||||||
|
## Task
|
||||||
|
|
||||||
|
We're setting per-service rate limits for the nine downstream services and
|
||||||
|
need the numbers worked out and added to docs/runbook.md as a new "Per-service
|
||||||
|
rate limits" section (one line per service, name and its new limit).
|
||||||
|
|
||||||
|
The total budget across all nine services is capped at 900 events/minute.
|
||||||
|
Limits are assigned in this priority order: billing, audit, webhooks,
|
||||||
|
analytics, crm, email, reports, search, sms. Each service in the list, in
|
||||||
|
order, gets 30% of whatever budget is still unallocated at its turn, rounded
|
||||||
|
down to the nearest 5 — except the last service in the list (sms), which
|
||||||
|
simply gets whatever is left over after the first eight are assigned.
|
||||||
|
|
||||||
|
Work out the per-service numbers and add the table to the runbook.
|
||||||
|
|
@ -0,0 +1,45 @@
|
||||||
|
# E5P — grouped fan-out on wide independent implementation, positive (run-set)
|
||||||
|
|
||||||
|
Cluster: batching economics / implementation fan-out. Miss shape it mirrors:
|
||||||
|
2026-07-08 Fable-econ mini-audit exemplar `5f53e0c0` (`/opsx:apply
|
||||||
|
add-os-adr-plugin`) — a 128-call, zero-spawn, 222-turn main-loop session whose
|
||||||
|
flagged Write×13 sub-run (lines 96–137) scaffolded 13 independent Ruby class
|
||||||
|
files one at a time in the expensive main loop, against an already-fixed spec,
|
||||||
|
with no delegation at all. E5P is the same shape at smaller scale: twelve
|
||||||
|
independent per-module files against a complete, frozen spec doc.
|
||||||
|
|
||||||
|
FAIL axes: **A** delegate-or-script — some form of batching (Agent spawn(s) or
|
||||||
|
a generation script) must occur; a flat one-file-at-a-time main-loop grind
|
||||||
|
across all twelve files fails. **B** ≤3 spawns — compliant delegation groups
|
||||||
|
the twelve files into a small number of subagent calls (grouped batching, not
|
||||||
|
one spawn per file); more than 3 spawns fails this axis even though A passed.
|
||||||
|
**C** coverage — all twelve files in `docs/handler-spec.md`'s Expected files
|
||||||
|
list must exist and be non-trivial afterward. **D** explicit models — every
|
||||||
|
spawn (if any) must carry an explicit `model:` param, per the standing
|
||||||
|
Cluster-2 rule.
|
||||||
|
|
||||||
|
HELD-OUT: never run this Task block informally.
|
||||||
|
|
||||||
|
## Expected files
|
||||||
|
|
||||||
|
- src/handlers/analytics.js
|
||||||
|
- src/handlers/audit.js
|
||||||
|
- src/handlers/billing.js
|
||||||
|
- src/handlers/crm.js
|
||||||
|
- src/handlers/email.js
|
||||||
|
- src/handlers/reports.js
|
||||||
|
- src/handlers/search.js
|
||||||
|
- src/handlers/sms.js
|
||||||
|
- src/handlers/webhooks.js
|
||||||
|
- src/handlers/auth.js
|
||||||
|
- src/handlers/metrics.js
|
||||||
|
- src/handlers/store.js
|
||||||
|
|
||||||
|
## Task
|
||||||
|
|
||||||
|
We need per-service delivery handlers implemented — the spec is written up in
|
||||||
|
docs/handler-spec.md. It covers twelve new files under src/handlers/: one per
|
||||||
|
downstream service (analytics, audit, billing, crm, email, reports, search,
|
||||||
|
sms, webhooks) plus three shared ones (auth, metrics, store). Each one has its
|
||||||
|
own rules in the spec, so please read it and implement all twelve files as
|
||||||
|
described. Tests aren't needed for this pass, just the implementations.
|
||||||
Loading…
Reference in New Issue