SecondBrain/os-orchestration-ws4-econ-wording-results.md

---
type: eval-results
title: "os-orchestration WS4 economics wording loop (ORCHESTRATION.md v3, fable+sonnet)"
summary: "The v3 cost-asymmetry/symmetric-trigger/batching wording flips E5P implementation fan-out from 0/6 to majority-PASS on both tiers and cuts fable main-loop output tokens to 43% of baseline, with all negatives clean on run-set AND held-out reserve — shipped after one iteration + confirm + reserve; residual gaps are E1P sonnet non-delegation (fixture-scale) and a ~1/4 fable downgrade-flag judge flicker."
tags:
  - type/eval-results
  - domain/llm-evaluation
  - tool/os-orchestration
  - project/cc-os
scope: global
last_updated: 2026-07-08
date: 2026-07-08
related:
  - os-orchestration-eval-baseline-grid-results
  - orchestration-prompting-claude-5-era
  - eval-methodology-ladder
  - running-autoresearch-skill-evals
  - claude-code-subagent-transcripts-and-token-accounting
source: cc-os
---

# os-orchestration WS4 economics wording loop

## Results Grid/Threshold

**Passing criterion:** per-cell ≥2/3 (sonnet) / 2/2 (fable) reps PASS on the frozen
deterministic checker; H3 econ thresholds pre-registered in
`docs/plans/ws4-orchestration-economics.md` (E5P fable median mltok ≤37,783 = 60% of
baseline; E3P non-regression guard; negatives must stay clean).

Pre-wording baseline (2026-07-08, 44 counted reps) → candidate-1 (iter-1 15 reps + confirm
40 reps) → frozen reserve (40 reps). Zero harness errors; every rep swept for
session-limit truncation (15 baseline reps excluded+replaced by that mechanical criterion).

| Cell | baseline | confirm (candidate) | reserve twin |
|---|---|---|---|
| E5P fable | 0/3 (grind, median mltok 62,971) | 2/2 (median across 5 candidate reps 27,372 = 43%) | 2/2 |
| E5P sonnet | 0/3 | 2/3 (+3/3 iter-1) | 3/3 |
| E1P fable | 1/2 | 1/2 (B judge flicker; 3/4 across iter+confirm) | 1/2 (A miss) |
| E1P sonnet | 0/3 | 0/3 | 0/3 |
| E2P / E3P (redesigned criteria) | 6/6 / 6/6 | 5/5 / 5/5 | 5/5 / 5/5 |
| all negatives (E1N/E2N/E3N/E5N) | clean | 16/16 | 20/20 |

**Summary:** H1 (symmetric cost-framed triggers) and H2 (batching: ≤3 grouped spawns, one
round, explicit models — reserve fable rep used 2 explicit-haiku spawns) confirmed; H3 met
on the pre-registered E5P-fable axis (43% of baseline mltok). E1P sonnet never delegates at
any point (fixture too small to force it — pre-registered limitation); zero over-delegation
anywhere, including E5N sequential tasks under the new batching rule.

## Measurement Setup

**Hypotheses / Scenarios Tested** — H1/H2/H3 pre-registered in the WS4 plan; E5
(batching/implementation fan-out, mirrors the 2026-07-08 mini-audit Write×13 exemplar) added
with reserve twins authored by a subagent so held-out content never entered the tuning
session's context. E2P/E3P criteria redesigned per the 2026-07-06 pre-registration (scripted-
direct PASS branch; 74KB S7 ingestion anchor + mechanical root-cause concepts).

**Fixture and Sampling** — relaystation Node.js fixture + new handler-spec (12 pairwise-
distinct handler modules, deliberately non-templatable). sonnet ×3 / fable ×2 reps per cell;
fable at reduced reps for cost per plan.

**Experimental Control (Frozen Surfaces)** — checker/extractor/runner/fixtures/scenarios/
rubric frozen during the loop; ORCHESTRATION.md wording the only tuning surface;
`bin/refresh-plugins` + cache==source diff before every grid. One instrument fix landed
between confirm and reserve (E3P concepts keyed by scenario id — the reserve twin targets
gen-logs' other planted incident; fixed without reading reserve files, run-set scoring
byte-identical, self-test 34/34).

## Validity and Limitations

**How to Interpret These Results** — run-set AND reserve are now both contaminated for
future wording tuning (reserve was the final measurement). The reserve pass is the
generalization claim: same axes, different surface domain, never read informally. Econ
figures are real token sums (subagent transcripts persist under
`<session-id>/subagents/agent-*.jsonl` — see [[claude-code-subagent-transcripts-and-token-accounting]]),
not the char-count proxy the mini-audit used.

**Weaknesses of This Eval** — headless `-p` population may differ from interactive sessions
(unresolved since the 2026-07-06 baseline). E1P fixture too small to force delegation on
sonnet; its cell says nothing about capability. H3(c) sonnet econ guard (≤150% of grind
baseline) was breached at median 168%: delegation carries a real ~+6.6k-token premium on the
cheap tier for identical task completion — accepted because the frozen checker requires
delegate-or-script on a non-scriptable task and fable is the population of concern. Single
fixture. E5P mltok variance is high on fable (15k–65k baseline; 22k–47k candidate).

## Transferable Lessons

- Econ guards on cheap tiers should be absolute-token or scriptability-conditioned, not
  ratios of a grind baseline — a ratio guard punishes correct delegation when the baseline
  behavior is an efficient grind. (Promoted to [[eval-methodology-ladder]].)
- Account session limits truncate headless reps into plausible-looking behavior FAILs (and
  PASSes); sweep every rep's final assistant message for the limit banner before counting.
  (Promoted to [[running-autoresearch-skill-evals]].)
- Reserve twins authored before a criterion redesign can be silently unscoreable by the new
  checker; audit checker constants against every planted fixture incident (not just run-set)
  before the reserve run — fixable by id-keyed criteria without reading reserve files.
- Delegating reserve-twin authoring to a subagent (report = filenames only) keeps held-out
  content out of the tuning session's context — cheap insurance when the same session will
  run the wording loop.

**Promotion check:** promoted lesson 1 to [[eval-methodology-ladder]] and lesson 2 to
[[running-autoresearch-skill-evals]] (both updated 2026-07-08); lessons 3–4 not yet
recurring.

## Deployment and Evolution

**Good-Enough Gate** — shipped 2026-07-08 (ORCHESTRATION.md v3 in cc-os + refreshed cache):
negatives clean on both grids, batching generalizes to the reserve twin on both tiers, H3
met on the pre-registered axis. Residual known gaps at ship time: E1P sonnet delegate-at-all
(fixture-scale, not wording), fable downgrade-flag ~1/4 judge-boundary flicker (os-adr W3
shape — characterize in production, don't lab-chase).

**Proposed Changes (Loop Inputs)** — none; candidate-1 shipped verbatim (five changes:
cost-asymmetry + tier-conditional threshold opener, symmetric delegate/work-directly trigger
lists, batching rule ~5–8 items/spawn + SendMessage reuse, async rule, Workflow effort
dial). Diff against v2 is commit-visible in cc-os.

**Hardening Path / Next Measurement** — production IRL re-audit of ~5 real Fable sessions
scheduled ~2026-07-22 per [[eval-methodology-irl-feedback-loop]]: compare main-loop token
share vs the 2026-07-08 mini-audit baseline (now measurable exactly via subagent
transcripts); promote new miss shapes into eval scenarios. E1P needs a larger fixture before
its sonnet cell is meaningful.

## Related

- [[os-orchestration-eval-baseline-grid-results]] — the 2026-07-06 E1–E3 baseline this loop builds on
- [[orchestration-prompting-claude-5-era]] — the model-page guidance the v3 wording encodes
- [[eval-methodology-ladder]] / [[running-autoresearch-skill-evals]] — discipline followed + lessons promoted
- [[claude-code-subagent-transcripts-and-token-accounting]] — the instrumentation discovery behind the econ axes