cc-os/docs/plans/ws4-orchestration-economics.md

WS4 — Orchestration economics: ORCHESTRATION.md v3 + eval extension + wording loop

Created: 2026-07-08. Status: PLANNED (mini-audit complete; steps 1–5 not started). Prereq reading before executing: ~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md, vault notes orchestration-prompting-claude-5-era, eval-methodology-ladder, os-orchestration-eval-baseline-grid-results.

Problem

The Fable-5 main loop does too much multi-call work itself. ORCHESTRATION.md was calibrated against over-delegation (Opus 4.6 era) under sonnet main loops where direct work was cheap; its negative-first threshold ("Delegate only when…", "≤2-tool-call ops direct") is read literally by Fable and licenses unbounded direct work whenever the task isn't parallelizable/ many-files/isolated-context. Goals (user, 2026-07-08): max quality at reasonable cost; route work to cheaper tiers whenever reasonable; group delegated tasks so agents finish in few rounds (fewer spawns → less per-spawn system-prompt tax).

Evidence base: Anthropic model-page guidance summarized in vault note orchestration-prompting-claude-5-era (fetched 2026-07-08); WS1 audit (docs/orchestration-audit/2026-07-06-findings.md); Fable economics mini-audit (docs/orchestration-audit/2026-07-08-fable-econ-miniaudit.md) — the H3 baseline.

Mini-audit headline (2026-07-08, 6 Fable sessions, ran before drafting wording)

• The dominant miss shape is implementation fan-out, not investigation: the one clean miss is an OpenSpec-apply session (5f53e0c0) that wrote 13 independent Ruby files entirely in the main loop — 314,899 main-loop output tokens, 0 spawns. E5P should mirror this shape (write-N-independent-files implementation), not a log-review shape.
• Long direct runs were mostly JUSTIFIED (scripted eval-polling loop of 119 calls; interactive troubleshooting with 15 human turns) — confirming the non-goal: cost-keyed thresholds, not "always delegate". Zero over-delegation anywhere.
• Secondary watch item: after a session's spawns complete, the main loop resumes long direct runs (43–48 calls, 9f45afcc) instead of delegating again — mid-session threshold decay.
• Instrumentation gap (affects A-econ): main transcripts contain NO isSidechain:true entries even in sessions with spawns — subagent token spend is unrecoverable from the transcript alone. The eval checker must source subagent spend from harness-side artifacts (the eval runner's own task/agent output files in the sandbox), or A-econ degrades to a main-loop-absolute-tokens metric (still usable: baseline sessions run 24k–315k main-loop output tokens). Resolve this during Step 2 design; do not silently ship the char-count proxy the mini-audit used.

Step 1 — ORCHESTRATION.md v3 (wording surface for the loop)

Five changes; keep explicit-model:, self-report, and don't-re-cover-ground rules unchanged:

1. Cost-asymmetry + tier-conditional threshold. New opening rule: main-loop tokens are the most expensive in the session; the more capable the main-loop model, the lower the delegation threshold. Any investigation or mechanical sequence beyond ~3 calls that does not need main-loop judgment → delegate down-tier, even when sequential. (Models know their own model ID — tier-conditional phrasing is implementable; E1 canary verified self-report.)
2. Symmetric triggers. Replace "Delegate only when…" with paired lists: "delegate when …" / "work directly when …" (keep the scripted-bulk-edit carve-out and ≤2-call rule on the direct side). Rationale: literal instruction-following on 5-era models; the canonical best-practices snippet pairs both directions.
3. Batching rule. Plan the fan-out before the first spawn; group related subtasks (~5–8 similar items) into one agent prompt with a return schema so the agent completes in one round; follow-ups go to the live agent (SendMessage), not a fresh spawn.
4. Async rule. Delegate independent subtasks and keep working while they run; intervene only if a subagent goes off track (Fable-page recommended pattern).
5. Effort dial (where exposed). Workflow agent() calls: mechanical stages effort: low; hard verify/judge stages high/xhigh.

Draft AFTER reading the mini-audit's miss shapes — wording should name the observed shapes, not hypothetical ones (Eval B lesson: mechanical triggers phrased at the observed precondition).

Step 2 — Eval harness extension (plugins/os-orchestration/eval/)

All extensions are new frozen surfaces; existing run-set scenarios E1–E3 stay untouched. Reserve stays frozen. Follow eval-methodology-ladder: paired positives/negatives, run-set + reserve twins authored together, thresholds anchored independently of any grid already run (pre-registered criterion-redesign rule from the 2026-07-06 baseline).

New metric axes (extractor + checker)

• A-econ (main-loop token share): main-loop assistant output tokens ÷ total session output tokens (main + sidechains). Extractor already parses transcripts; add the sum.
• A-prespawn: tool calls + tool_result bytes before first spawn (extractor segment exists from E3 work).
• A-batch: spawns per task and rounds per agent (Agent tool_use count; SendMessage reuse counts as same-agent round, not new spawn).
• Redesign E2P/E3P delegate-at-all axes per the pre-registered note: PASS may be "delegated" OR "superior scripted direct strategy" — the checker must distinguish script-driven bulk ops (few Bash calls, uniform op) from grind (per-file tool loops).

New scenario pair E5 (batching) + reserve twins

• E5P: N (~10–12) similar independent items, shaped like the observed miss — an implementation task producing N independent files in the relaystation fixture (NOT log review). PASS = grouped delegation (≤3 spawns covering all items, grouped prompts, one round each), FAIL = 1-agent-per-item or main-loop grind.
• E5N (paired negative): a task where batching would wrongly serialize genuinely judgment-dependent items (each item's handling depends on the previous result) — PASS = sequential handling (direct or single agent), FAIL = blind parallel fan-out.
• Author reserve twins at the same time, different surface domain; freeze both.

Fable column

Baseline grid ran sonnet/haiku orchestrators only; the population of concern is Fable. Add --model fable (headless claude -p --model claude-fable-5) at reduced reps (1–2 per cell for the grid; 3 on target cells inside the loop). Cost of the eval itself is real — use the reduced inner-loop grid discipline (target cells + one passing control).

Step 3 — Baseline grid (pre-wording)

Run extended grid (E1–E3 + E5, econ axes) × {fable, sonnet} before touching wording. Canary-cell the first live rep of every NEW scenario/axis and hand-verify TSV vs transcript (count the canary). This grid is the tuning baseline; the 2026-07-08 mini-audit is the IRL baseline for H3.

Step 4 — Autoresearch wording loop

Loop discipline per the howto: wording-only moves (checker/fixtures/scenarios/axes frozen), bin/refresh-plugins before every iteration's grid, ≥3 reps on target cells + 1 control cell, accept only majority-of-reps improvements, verify from TSV not agent prose.

Hypotheses (pre-registered)

• H1 (threshold framing): symmetric cost-framed triggers raise delegation on positive scenarios (esp. E3P-shape sequential investigation) on fable/sonnet, while negatives stay clean. Guard: baseline negatives were 18/18 — any negative regression rejects the candidate regardless of positive gains.

• H2 (batching rule): the grouping rule reduces spawns-per-task and rounds-per-agent on E5P without task-success regression; E5N stays PASS (no blind fan-out).

• H3 (net economics — the actual goal): main-loop token spend on positives drops materially vs the Step-3 baseline with task success unchanged. Metric: main-loop share if Step 2 solves the sidechain-instrumentation gap, else main-loop absolute output tokens per scenario. Anchor the target threshold independently before the loop starts (not post-hoc to whatever the loop achieves). H1/H2 are mechanisms; H3 is the verdict axis.

  H3 threshold (pre-registered 2026-07-08, before the Step-3 grid ran): H3 PASSes iff, on the positive economics cells (E5P and E3P, fable column), the median main-loop assistant output tokens across reps under the candidate wording is ≤60% of the Step-3 pre-wording baseline median for the same cell (≥40% reduction), with task-success axes unchanged and all negatives clean. Pass/fail is keyed to main-loop absolute output tokens (exists in both baselines regardless of the sidechain-instrumentation outcome); main-loop share is reported additionally if Step 2 solves sidechain measurement. Anchor rationale (independent of any grid): in the IRL miss exemplar 5f53e0c0, the Write×13 fan-out segment dominates a 314,899-token session — delegating the implementation segment should remove roughly half of main-loop output; 40% is a conservative floor of that estimate.

Judging

Per-axis deterministic checker verdicts; per-scenario pass bars, not aggregate scores. A wording candidate ships only if: all negatives hold, target positives improve on majority of reps, control cell unchanged, and the full-grid confirm reproduces it. Final confirmation on the frozen reserve (tuning against the run-set moves measurement to the reserve — after this loop the run-set is contaminated for future measurement).

Step 5 — Rollout + IRL re-audit

Ship winning wording (refresh caches), then re-run the Fable economics mini-audit on ~5 new real sessions after ~2 weeks (eval-methodology-irl-feedback-loop). Compare main-loop token share vs the 2026-07-08 baseline. Promote any new miss shapes into eval scenarios.

Non-goals / guards

• No changes to delegation SAFETY rules (explicit model:, self-report, downgrade honesty).
• Don't over-rotate: zero over-delegation was a baseline strength (negatives 18/18) and scripted-direct strategies were often genuinely superior — the point is cost-keyed thresholds, not "always delegate".
• Held-out discipline: never run scenario Task blocks informally; reserve is never read informally.