12 KiB

Raw Blame History

WS4 — Orchestration economics: ORCHESTRATION.md v3 + eval extension + wording loop

Created: 2026-07-08. Status: PLANNED (mini-audit complete; steps 1–5 not started). Prereq reading before executing: ~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md, vault notes orchestration-prompting-claude-5-era, eval-methodology-ladder, os-orchestration-eval-baseline-grid-results.

Problem

The Fable-5 main loop does too much multi-call work itself. ORCHESTRATION.md was calibrated against over-delegation (Opus 4.6 era) under sonnet main loops where direct work was cheap; its negative-first threshold ("Delegate only when…", "≤2-tool-call ops direct") is read literally by Fable and licenses unbounded direct work whenever the task isn't parallelizable/ many-files/isolated-context. Goals (user, 2026-07-08): max quality at reasonable cost; route work to cheaper tiers whenever reasonable; group delegated tasks so agents finish in few rounds (fewer spawns → less per-spawn system-prompt tax).

Evidence base: Anthropic model-page guidance summarized in vault note orchestration-prompting-claude-5-era (fetched 2026-07-08); WS1 audit (docs/orchestration-audit/2026-07-06-findings.md); Fable economics mini-audit (docs/orchestration-audit/2026-07-08-fable-econ-miniaudit.md) — the H3 baseline.

Mini-audit headline (2026-07-08, 6 Fable sessions, ran before drafting wording)

The dominant miss shape is implementation fan-out, not investigation: the one clean miss is an OpenSpec-apply session (5f53e0c0) that wrote 13 independent Ruby files entirely in the main loop — 314,899 main-loop output tokens, 0 spawns. E5P should mirror this shape (write-N-independent-files implementation), not a log-review shape.
Long direct runs were mostly JUSTIFIED (scripted eval-polling loop of 119 calls; interactive troubleshooting with 15 human turns) — confirming the non-goal: cost-keyed thresholds, not "always delegate". Zero over-delegation anywhere.
Secondary watch item: after a session's spawns complete, the main loop resumes long direct runs (43–48 calls, 9f45afcc) instead of delegating again — mid-session threshold decay.
Instrumentation gap (affects A-econ): main transcripts contain NO isSidechain:true entries even in sessions with spawns — subagent token spend is unrecoverable from the transcript alone. The eval checker must source subagent spend from harness-side artifacts (the eval runner's own task/agent output files in the sandbox), or A-econ degrades to a main-loop-absolute-tokens metric (still usable: baseline sessions run 24k–315k main-loop output tokens). Resolve this during Step 2 design; do not silently ship the char-count proxy the mini-audit used.

Step 1 — ORCHESTRATION.md v3 (wording surface for the loop)

Five changes; keep explicit-model:, self-report, and don't-re-cover-ground rules unchanged:

Cost-asymmetry + tier-conditional threshold. New opening rule: main-loop tokens are the most expensive in the session; the more capable the main-loop model, the lower the delegation threshold. Any investigation or mechanical sequence beyond ~3 calls that does not need main-loop judgment → delegate down-tier, even when sequential. (Models know their own model ID — tier-conditional phrasing is implementable; E1 canary verified self-report.)
Symmetric triggers. Replace "Delegate only when…" with paired lists: "delegate when …" / "work directly when …" (keep the scripted-bulk-edit carve-out and ≤2-call rule on the direct side). Rationale: literal instruction-following on 5-era models; the canonical best-practices snippet pairs both directions.
Batching rule. Plan the fan-out before the first spawn; group related subtasks (~5–8 similar items) into one agent prompt with a return schema so the agent completes in one round; follow-ups go to the live agent (SendMessage), not a fresh spawn.
Async rule. Delegate independent subtasks and keep working while they run; intervene only if a subagent goes off track (Fable-page recommended pattern).
Effort dial (where exposed). Workflow agent() calls: mechanical stages effort: low; hard verify/judge stages high/xhigh.

Draft AFTER reading the mini-audit's miss shapes — wording should name the observed shapes, not hypothetical ones (Eval B lesson: mechanical triggers phrased at the observed precondition).

Step 2 — Eval harness extension (`plugins/os-orchestration/eval/`)

All extensions are new frozen surfaces; existing run-set scenarios E1–E3 stay untouched. Reserve stays frozen. Follow eval-methodology-ladder: paired positives/negatives, run-set + reserve twins authored together, thresholds anchored independently of any grid already run (pre-registered criterion-redesign rule from the 2026-07-06 baseline).

New metric axes (extractor + checker)

A-econ (main-loop token share): main-loop assistant output tokens ÷ total session output tokens (main + sidechains). Extractor already parses transcripts; add the sum.
A-prespawn: tool calls + tool_result bytes before first spawn (extractor segment exists from E3 work).
A-batch: spawns per task and rounds per agent (Agent tool_use count; SendMessage reuse counts as same-agent round, not new spawn).
Redesign E2P/E3P delegate-at-all axes per the pre-registered note: PASS may be "delegated" OR "superior scripted direct strategy" — the checker must distinguish script-driven bulk ops (few Bash calls, uniform op) from grind (per-file tool loops).

New scenario pair E5 (batching) + reserve twins

E5P: N (~10–12) similar independent items, shaped like the observed miss — an implementation task producing N independent files in the relaystation fixture (NOT log review). PASS = grouped delegation (≤3 spawns covering all items, grouped prompts, one round each), FAIL = 1-agent-per-item or main-loop grind.
E5N (paired negative): a task where batching would wrongly serialize genuinely judgment-dependent items (each item's handling depends on the previous result) — PASS = sequential handling (direct or single agent), FAIL = blind parallel fan-out.
Author reserve twins at the same time, different surface domain; freeze both.

Fable column

Baseline grid ran sonnet/haiku orchestrators only; the population of concern is Fable. Add --model fable (headless claude -p --model claude-fable-5) at reduced reps (1–2 per cell for the grid; 3 on target cells inside the loop). Cost of the eval itself is real — use the reduced inner-loop grid discipline (target cells + one passing control).

Step 3 — Baseline grid (pre-wording)

Run extended grid (E1–E3 + E5, econ axes) × {fable, sonnet} before touching wording. Canary-cell the first live rep of every NEW scenario/axis and hand-verify TSV vs transcript (count the canary). This grid is the tuning baseline; the 2026-07-08 mini-audit is the IRL baseline for H3.

Step 4 — Autoresearch wording loop

Loop discipline per the howto: wording-only moves (checker/fixtures/scenarios/axes frozen), bin/refresh-plugins before every iteration's grid, ≥3 reps on target cells + 1 control cell, accept only majority-of-reps improvements, verify from TSV not agent prose.

Hypotheses (pre-registered)

H1 (threshold framing): symmetric cost-framed triggers raise delegation on positive scenarios (esp. E3P-shape sequential investigation) on fable/sonnet, while negatives stay clean. Guard: baseline negatives were 18/18 — any negative regression rejects the candidate regardless of positive gains.
H2 (batching rule): the grouping rule reduces spawns-per-task and rounds-per-agent on E5P without task-success regression; E5N stays PASS (no blind fan-out).
H3 (net economics — the actual goal): main-loop token spend on positives drops materially vs the Step-3 baseline with task success unchanged. Metric: main-loop share if Step 2 solves the sidechain-instrumentation gap, else main-loop absolute output tokens per scenario. Anchor the target threshold independently before the loop starts (not post-hoc to whatever the loop achieves). H1/H2 are mechanisms; H3 is the verdict axis.

H3 threshold (pre-registered 2026-07-08, before the Step-3 grid ran): H3 PASSes iff, on the positive economics cells (E5P and E3P, fable column), the median main-loop assistant output tokens across reps under the candidate wording is ≤60% of the Step-3 pre-wording baseline median for the same cell (≥40% reduction), with task-success axes unchanged and all negatives clean. Pass/fail is keyed to main-loop absolute output tokens (exists in both baselines regardless of the sidechain-instrumentation outcome); main-loop share is reported additionally if Step 2 solves sidechain measurement. Anchor rationale (independent of any grid): in the IRL miss exemplar 5f53e0c0, the Write×13 fan-out segment dominates a 314,899-token session — delegating the implementation segment should remove roughly half of main-loop output; 40% is a conservative floor of that estimate.

H3 amendment (2026-07-08, after the Step-3 baseline, BEFORE any wording iteration): the baseline showed E3P positives already PASS cheaply on both tiers via surgical direct investigation (fable median mltok 10,273; sonnet 9,420) — a reduction target there would reward degrading already-correct behavior. Amended H3: (a) E5P fable median mltok ≤ 60% of baseline median 62,971 → ≤ 37,783, with the E5P verdict PASSing on a majority of reps; (b) E3P guard: verdict stays PASS and median mltok ≤ 120% of baseline (fable ≤ 12,328); (c) E5P sonnet: verdict flips to PASS on majority (baseline mltok already low at 9,789 — no reduction target, guard ≤ 150%); (d) all negatives stay clean.

Step-3 baseline results (2026-07-08, grid complete)

TSV: plugins/os-orchestration/eval/results/2026-07-08-ws4-baseline-grid.tsv — 44 counted reps (25 valid first-wave + 15 replacements for session-limit-truncated reps, mechanically excluded by the limit-banner criterion + 4 counted canaries; excluded rows preserved in the TSV as comments). Headlines:

E5P 0/6 both tiers, all A:main-loop-grind — the mini-audit miss shape reproduces in the lab. Fable burns median 62,971 mltok doing it (the H3 target cell); sonnet ~9.8k.
Criterion redesign validated: E2P 6/6 PASS via A:scripted-direct (old axis would have failed all); E3P 6/6 PASS via surgical direct under the 74KB ingestion anchor.
All negatives clean (E1N/E2N/E3N/E5N, both tiers) — including E5N on fable: no blind fan-out on the sequential task.
E1P: sonnet 0/3 (never delegates), fable 1/2 — fable delegates readily (3–6 spawns) and its one FAIL is axis-B (didn't flag the stubbed downgrade), the first observed fable-tier self-report miss.
Sidechain token accounting works live: fable E1 cells show mlshare 0.36–0.68 with real sctok sums.
Wording-loop target cells: E5P (both tiers, A-axis + fable econ), E1P sonnet (delegate-at-all), E1P fable (axis-B flag reliability). Controls: E2P (must stay PASS via scripted-direct), E3P (non-regression guard), negatives.

Judging

Per-axis deterministic checker verdicts; per-scenario pass bars, not aggregate scores. A wording candidate ships only if: all negatives hold, target positives improve on majority of reps, control cell unchanged, and the full-grid confirm reproduces it. Final confirmation on the frozen reserve (tuning against the run-set moves measurement to the reserve — after this loop the run-set is contaminated for future measurement).

Step 5 — Rollout + IRL re-audit

Ship winning wording (refresh caches), then re-run the Fable economics mini-audit on ~5 new real sessions after ~2 weeks (eval-methodology-irl-feedback-loop). Compare main-loop token share vs the 2026-07-08 baseline. Promote any new miss shapes into eval scenarios.

Non-goals / guards

No changes to delegation SAFETY rules (explicit model:, self-report, downgrade honesty).
Don't over-rotate: zero over-delegation was a baseline strength (negatives 18/18) and scripted-direct strategies were often genuinely superior — the point is cost-keyed thresholds, not "always delegate".
Held-out discipline: never run scenario Task blocks informally; reserve is never read informally.

12 KiB Raw Blame History Unescape Escape