cc-os/docs/plans/ws4-orchestration-economics.md

# WS4 — Orchestration economics: ORCHESTRATION.md v3 + eval extension + wording loop

_Created: 2026-07-08. Status: **SHIPPED** (steps 1–4 complete 2026-07-08; step 5 IRL
re-audit scheduled ~2026-07-22). Loop log: `autoresearch/classic-260708-1039/loop-log.md`;
results TSVs: `plugins/os-orchestration/eval/results/2026-07-08-*.tsv`; vault note:
[[os-orchestration-ws4-econ-wording-results]]._
_Prereq reading before executing: `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md`,
vault notes [[orchestration-prompting-claude-5-era]], [[eval-methodology-ladder]],
[[os-orchestration-eval-baseline-grid-results]]._

## Problem

The Fable-5 main loop does too much multi-call work itself. ORCHESTRATION.md was calibrated
against over-delegation (Opus 4.6 era) under sonnet main loops where direct work was cheap;
its negative-first threshold ("Delegate only when…", "≤2-tool-call ops direct") is read
literally by Fable and licenses unbounded direct work whenever the task isn't parallelizable/
many-files/isolated-context. Goals (user, 2026-07-08): max quality at reasonable cost;
route work to cheaper tiers whenever reasonable; group delegated tasks so agents finish in
few rounds (fewer spawns → less per-spawn system-prompt tax).

Evidence base: Anthropic model-page guidance summarized in vault note
[[orchestration-prompting-claude-5-era]] (fetched 2026-07-08); WS1 audit
(`docs/orchestration-audit/2026-07-06-findings.md`); Fable economics mini-audit
(`docs/orchestration-audit/2026-07-08-fable-econ-miniaudit.md`) — the H3 baseline.

### Mini-audit headline (2026-07-08, 6 Fable sessions, ran before drafting wording)

- **The dominant miss shape is implementation fan-out, not investigation:** the one clean
  miss is an OpenSpec-apply session (`5f53e0c0`) that wrote 13 independent Ruby files
  entirely in the main loop — 314,899 main-loop output tokens, 0 spawns. E5P should mirror
  this shape (write-N-independent-files implementation), not a log-review shape.
- Long direct runs were mostly JUSTIFIED (scripted eval-polling loop of 119 calls;
  interactive troubleshooting with 15 human turns) — confirming the non-goal: cost-keyed
  thresholds, not "always delegate". Zero over-delegation anywhere.
- Secondary watch item: after a session's spawns complete, the main loop resumes long
  direct runs (43–48 calls, `9f45afcc`) instead of delegating again — mid-session threshold
  decay.
- **Instrumentation gap (affects A-econ):** main transcripts contain NO `isSidechain:true`
  entries even in sessions with spawns — subagent token spend is unrecoverable from the
  transcript alone. The eval checker must source subagent spend from harness-side artifacts
  (the eval runner's own task/agent output files in the sandbox), or A-econ degrades to a
  main-loop-absolute-tokens metric (still usable: baseline sessions run 24k–315k main-loop
  output tokens). Resolve this during Step 2 design; do not silently ship the char-count
  proxy the mini-audit used.

## Step 1 — ORCHESTRATION.md v3 (wording surface for the loop)

Five changes; keep explicit-`model:`, self-report, and don't-re-cover-ground rules unchanged:

1. **Cost-asymmetry + tier-conditional threshold.** New opening rule: main-loop tokens are
   the most expensive in the session; the more capable the main-loop model, the lower the
   delegation threshold. Any investigation or mechanical sequence beyond ~3 calls that does
   not need main-loop judgment → delegate down-tier, even when sequential. (Models know their
   own model ID — tier-conditional phrasing is implementable; E1 canary verified self-report.)
2. **Symmetric triggers.** Replace "Delegate only when…" with paired lists:
   "delegate when …" / "work directly when …" (keep the scripted-bulk-edit carve-out and
   ≤2-call rule on the direct side). Rationale: literal instruction-following on 5-era
   models; the canonical best-practices snippet pairs both directions.
3. **Batching rule.** Plan the fan-out before the first spawn; group related subtasks
   (~5–8 similar items) into one agent prompt with a return schema so the agent completes in
   one round; follow-ups go to the live agent (SendMessage), not a fresh spawn.
4. **Async rule.** Delegate independent subtasks and keep working while they run; intervene
   only if a subagent goes off track (Fable-page recommended pattern).
5. **Effort dial (where exposed).** Workflow `agent()` calls: mechanical stages
   `effort: low`; hard verify/judge stages high/xhigh.

Draft AFTER reading the mini-audit's miss shapes — wording should name the observed shapes,
not hypothetical ones (Eval B lesson: mechanical triggers phrased at the observed
precondition).

## Step 2 — Eval harness extension (`plugins/os-orchestration/eval/`)

All extensions are new frozen surfaces; existing run-set scenarios E1–E3 stay untouched.
Reserve stays frozen. Follow [[eval-methodology-ladder]]: paired positives/negatives,
run-set + reserve twins authored together, thresholds anchored independently of any grid
already run (pre-registered criterion-redesign rule from the 2026-07-06 baseline).

### New metric axes (extractor + checker)

- **A-econ (main-loop token share):** main-loop assistant output tokens ÷ total session
  output tokens (main + sidechains). Extractor already parses transcripts; add the sum.
- **A-prespawn:** tool calls + tool_result bytes before first spawn (extractor segment
  exists from E3 work).
- **A-batch:** spawns per task and rounds per agent (Agent tool_use count; SendMessage
  reuse counts as same-agent round, not new spawn).
- Redesign E2P/E3P delegate-at-all axes per the pre-registered note: PASS may be
  "delegated" OR "superior scripted direct strategy" — the checker must distinguish
  script-driven bulk ops (few Bash calls, uniform op) from grind (per-file tool loops).

### New scenario pair E5 (batching) + reserve twins

- **E5P:** N (~10–12) similar independent items, shaped like the observed miss —
  an implementation task producing N independent files in the relaystation fixture
  (NOT log review). PASS = grouped delegation (≤3 spawns covering all items, grouped
  prompts, one round each), FAIL = 1-agent-per-item or main-loop grind.
- **E5N (paired negative):** a task where batching would wrongly serialize genuinely
  judgment-dependent items (each item's handling depends on the previous result) — PASS =
  sequential handling (direct or single agent), FAIL = blind parallel fan-out.
- Author reserve twins at the same time, different surface domain; freeze both.

### Fable column

Baseline grid ran sonnet/haiku orchestrators only; the population of concern is Fable.
Add `--model fable` (headless `claude -p --model claude-fable-5`) at reduced reps
(1–2 per cell for the grid; 3 on target cells inside the loop). Cost of the eval itself is
real — use the reduced inner-loop grid discipline (target cells + one passing control).

## Step 3 — Baseline grid (pre-wording)

Run extended grid (E1–E3 + E5, econ axes) × {fable, sonnet} before touching wording.
Canary-cell the first live rep of every NEW scenario/axis and hand-verify TSV vs transcript
(count the canary). This grid is the tuning baseline; the 2026-07-08 mini-audit is the IRL
baseline for H3.

## Step 4 — Autoresearch wording loop

Loop discipline per the howto: wording-only moves (checker/fixtures/scenarios/axes frozen),
`bin/refresh-plugins` before every iteration's grid, ≥3 reps on target cells + 1 control
cell, accept only majority-of-reps improvements, verify from TSV not agent prose.

### Hypotheses (pre-registered)

- **H1 (threshold framing):** symmetric cost-framed triggers raise delegation on positive
  scenarios (esp. E3P-shape sequential investigation) on fable/sonnet, while negatives stay
  clean. Guard: baseline negatives were 18/18 — any negative regression rejects the
  candidate regardless of positive gains.
- **H2 (batching rule):** the grouping rule reduces spawns-per-task and rounds-per-agent on
  E5P without task-success regression; E5N stays PASS (no blind fan-out).
- **H3 (net economics — the actual goal):** main-loop token spend on positives drops
  materially vs the Step-3 baseline with task success unchanged. Metric: main-loop share
  if Step 2 solves the sidechain-instrumentation gap, else main-loop absolute output
  tokens per scenario. Anchor the target threshold independently before the loop starts
  (not post-hoc to whatever the loop achieves). H1/H2 are mechanisms; H3 is the verdict
  axis.

  **H3 threshold (pre-registered 2026-07-08, before the Step-3 grid ran):** H3 PASSes iff,
  on the positive economics cells (E5P and E3P, fable column), the median main-loop
  assistant output tokens across reps under the candidate wording is ≤60% of the Step-3
  pre-wording baseline median for the same cell (≥40% reduction), with task-success axes
  unchanged and all negatives clean. Pass/fail is keyed to main-loop **absolute** output
  tokens (exists in both baselines regardless of the sidechain-instrumentation outcome);
  main-loop *share* is reported additionally if Step 2 solves sidechain measurement.
  Anchor rationale (independent of any grid): in the IRL miss exemplar `5f53e0c0`, the
  Write×13 fan-out segment dominates a 314,899-token session — delegating the
  implementation segment should remove roughly half of main-loop output; 40% is a
  conservative floor of that estimate.

  **H3 amendment (2026-07-08, after the Step-3 baseline, BEFORE any wording iteration):**
  the baseline showed E3P positives already PASS cheaply on both tiers via surgical direct
  investigation (fable median mltok 10,273; sonnet 9,420) — a reduction target there would
  reward degrading already-correct behavior. Amended H3: (a) **E5P fable** median mltok ≤
  60% of baseline median 62,971 → **≤ 37,783**, with the E5P verdict PASSing on a majority
  of reps; (b) **E3P guard**: verdict stays PASS and median mltok ≤ 120% of baseline
  (fable ≤ 12,328); (c) **E5P sonnet**: verdict flips to PASS on majority (baseline mltok
  already low at 9,789 — no reduction target, guard ≤ 150%); (d) all negatives stay clean.

## Step-3 baseline results (2026-07-08, grid complete)

TSV: `plugins/os-orchestration/eval/results/2026-07-08-ws4-baseline-grid.tsv` — 44 counted
reps (25 valid first-wave + 15 replacements for session-limit-truncated reps, mechanically
excluded by the limit-banner criterion + 4 counted canaries; excluded rows preserved in the
TSV as comments). Headlines:

- **E5P 0/6 both tiers, all `A:main-loop-grind`** — the mini-audit miss shape reproduces in
  the lab. Fable burns median 62,971 mltok doing it (the H3 target cell); sonnet ~9.8k.
- **Criterion redesign validated**: E2P 6/6 PASS via `A:scripted-direct` (old axis would
  have failed all); E3P 6/6 PASS via surgical direct under the 74KB ingestion anchor.
- **All negatives clean** (E1N/E2N/E3N/E5N, both tiers) — including E5N on fable: no blind
  fan-out on the sequential task.
- **E1P: sonnet 0/3 (never delegates), fable 1/2** — fable delegates readily (3–6 spawns)
  and its one FAIL is axis-B (didn't flag the stubbed downgrade), the first observed
  fable-tier self-report miss.
- Sidechain token accounting works live: fable E1 cells show mlshare 0.36–0.68 with real
  sctok sums.
- Wording-loop target cells: E5P (both tiers, A-axis + fable econ), E1P sonnet
  (delegate-at-all), E1P fable (axis-B flag reliability). Controls: E2P (must stay PASS via
  scripted-direct), E3P (non-regression guard), negatives.

### Judging

Per-axis deterministic checker verdicts; per-scenario pass bars, not aggregate scores.
A wording candidate ships only if: all negatives hold, target positives improve on majority
of reps, control cell unchanged, and the full-grid confirm reproduces it. Final confirmation
on the frozen reserve (tuning against the run-set moves measurement to the reserve —
after this loop the run-set is contaminated for future measurement).

## Step 5 — Rollout + IRL re-audit

Ship winning wording (refresh caches), then re-run the Fable economics mini-audit on ~5 new
real sessions after ~2 weeks ([[eval-methodology-irl-feedback-loop]]). Compare main-loop
token share vs the 2026-07-08 baseline. Promote any new miss shapes into eval scenarios.

## Non-goals / guards

- No changes to delegation SAFETY rules (explicit model:, self-report, downgrade honesty).
- Don't over-rotate: zero over-delegation was a baseline strength (negatives 18/18) and
  scripted-direct strategies were often genuinely superior — the point is cost-keyed
  thresholds, not "always delegate".
- Held-out discipline: never run scenario Task blocks informally; reserve is never read
  informally.