os-orchestration WS4 economics wording loop

Results Grid/Threshold

Passing criterion: per-cell ≥2/3 (sonnet) / 2/2 (fable) reps PASS on the frozen deterministic checker; H3 econ thresholds pre-registered in docs/plans/ws4-orchestration-economics.md (E5P fable median mltok ≤37,783 = 60% of baseline; E3P non-regression guard; negatives must stay clean).

Pre-wording baseline (2026-07-08, 44 counted reps) → candidate-1 (iter-1 15 reps + confirm 40 reps) → frozen reserve (40 reps). Zero harness errors; every rep swept for session-limit truncation (15 baseline reps excluded+replaced by that mechanical criterion).

Cell	baseline	confirm (candidate)	reserve twin
E5P fable	0/3 (grind, median mltok 62,971)	2/2 (median across 5 candidate reps 27,372 = 43%)	2/2
E5P sonnet	0/3	2/3 (+3/3 iter-1)	3/3
E1P fable	1/2	1/2 (B judge flicker; 3/4 across iter+confirm)	1/2 (A miss)
E1P sonnet	0/3	0/3	0/3
E2P / E3P (redesigned criteria)	6/6 / 6/6	5/5 / 5/5	5/5 / 5/5
all negatives (E1N/E2N/E3N/E5N)	clean	16/16	20/20

Summary: H1 (symmetric cost-framed triggers) and H2 (batching: ≤3 grouped spawns, one round, explicit models — reserve fable rep used 2 explicit-haiku spawns) confirmed; H3 met on the pre-registered E5P-fable axis (43% of baseline mltok). E1P sonnet never delegates at any point (fixture too small to force it — pre-registered limitation); zero over-delegation anywhere, including E5N sequential tasks under the new batching rule.

Measurement Setup

Hypotheses / Scenarios Tested — H1/H2/H3 pre-registered in the WS4 plan; E5 (batching/implementation fan-out, mirrors the 2026-07-08 mini-audit Write×13 exemplar) added with reserve twins authored by a subagent so held-out content never entered the tuning session's context. E2P/E3P criteria redesigned per the 2026-07-06 pre-registration (scripted- direct PASS branch; 74KB S7 ingestion anchor + mechanical root-cause concepts).

Fixture and Sampling — relaystation Node.js fixture + new handler-spec (12 pairwise- distinct handler modules, deliberately non-templatable). sonnet ×3 / fable ×2 reps per cell; fable at reduced reps for cost per plan.

Experimental Control (Frozen Surfaces) — checker/extractor/runner/fixtures/scenarios/ rubric frozen during the loop; ORCHESTRATION.md wording the only tuning surface; bin/refresh-plugins + cache==source diff before every grid. One instrument fix landed between confirm and reserve (E3P concepts keyed by scenario id — the reserve twin targets gen-logs' other planted incident; fixed without reading reserve files, run-set scoring byte-identical, self-test 34/34).

Validity and Limitations

How to Interpret These Results — run-set AND reserve are now both contaminated for future wording tuning (reserve was the final measurement). The reserve pass is the generalization claim: same axes, different surface domain, never read informally. Econ figures are real token sums (subagent transcripts persist under <session-id>/subagents/agent-*.jsonl — see claude-code-subagent-transcripts-and-token-accounting), not the char-count proxy the mini-audit used.

Weaknesses of This Eval — headless -p population may differ from interactive sessions (unresolved since the 2026-07-06 baseline). E1P fixture too small to force delegation on sonnet; its cell says nothing about capability. H3(c) sonnet econ guard (≤150% of grind baseline) was breached at median 168%: delegation carries a real ~+6.6k-token premium on the cheap tier for identical task completion — accepted because the frozen checker requires delegate-or-script on a non-scriptable task and fable is the population of concern. Single fixture. E5P mltok variance is high on fable (15k–65k baseline; 22k–47k candidate).

Transferable Lessons

Econ guards on cheap tiers should be absolute-token or scriptability-conditioned, not ratios of a grind baseline — a ratio guard punishes correct delegation when the baseline behavior is an efficient grind. (Promoted to eval-methodology-ladder.)
Account session limits truncate headless reps into plausible-looking behavior FAILs (and PASSes); sweep every rep's final assistant message for the limit banner before counting. (Promoted to running-autoresearch-skill-evals.)
Reserve twins authored before a criterion redesign can be silently unscoreable by the new checker; audit checker constants against every planted fixture incident (not just run-set) before the reserve run — fixable by id-keyed criteria without reading reserve files.
Delegating reserve-twin authoring to a subagent (report = filenames only) keeps held-out content out of the tuning session's context — cheap insurance when the same session will run the wording loop.

Promotion check: promoted lesson 1 to eval-methodology-ladder and lesson 2 to running-autoresearch-skill-evals (both updated 2026-07-08); lessons 3–4 not yet recurring.

Deployment and Evolution

Good-Enough Gate — shipped 2026-07-08 (ORCHESTRATION.md v3 in cc-os + refreshed cache): negatives clean on both grids, batching generalizes to the reserve twin on both tiers, H3 met on the pre-registered axis. Residual known gaps at ship time: E1P sonnet delegate-at-all (fixture-scale, not wording), fable downgrade-flag ~1/4 judge-boundary flicker (os-adr W3 shape — characterize in production, don't lab-chase).

Proposed Changes (Loop Inputs) — none; candidate-1 shipped verbatim (five changes: cost-asymmetry + tier-conditional threshold opener, symmetric delegate/work-directly trigger lists, batching rule ~5–8 items/spawn + SendMessage reuse, async rule, Workflow effort dial). Diff against v2 is commit-visible in cc-os.

Hardening Path / Next Measurement — production IRL re-audit of ~5 real Fable sessions scheduled ~2026-07-22 per eval-methodology-irl-feedback-loop: compare main-loop token share vs the 2026-07-08 mini-audit baseline (now measurable exactly via subagent transcripts); promote new miss shapes into eval scenarios. E1P needs a larger fixture before its sonnet cell is meaningful.

os-orchestration-eval-baseline-grid-results — the 2026-07-06 E1–E3 baseline this loop builds on
orchestration-prompting-claude-5-era — the model-page guidance the v3 wording encodes
eval-methodology-ladder / running-autoresearch-skill-evals — discipline followed + lessons promoted
claude-code-subagent-transcripts-and-token-accounting — the instrumentation discovery behind the econ axes

7.4 KiB Raw Permalink Blame History Unescape Escape