130 lines
7.4 KiB
Markdown
130 lines
7.4 KiB
Markdown
---
|
||
type: eval-results
|
||
title: "os-orchestration WS4 economics wording loop (ORCHESTRATION.md v3, fable+sonnet)"
|
||
summary: "The v3 cost-asymmetry/symmetric-trigger/batching wording flips E5P implementation fan-out from 0/6 to majority-PASS on both tiers and cuts fable main-loop output tokens to 43% of baseline, with all negatives clean on run-set AND held-out reserve — shipped after one iteration + confirm + reserve; residual gaps are E1P sonnet non-delegation (fixture-scale) and a ~1/4 fable downgrade-flag judge flicker."
|
||
tags:
|
||
- type/eval-results
|
||
- domain/llm-evaluation
|
||
- tool/os-orchestration
|
||
- project/cc-os
|
||
scope: global
|
||
last_updated: 2026-07-08
|
||
date: 2026-07-08
|
||
related:
|
||
- os-orchestration-eval-baseline-grid-results
|
||
- orchestration-prompting-claude-5-era
|
||
- eval-methodology-ladder
|
||
- running-autoresearch-skill-evals
|
||
- claude-code-subagent-transcripts-and-token-accounting
|
||
source: cc-os
|
||
---
|
||
|
||
# os-orchestration WS4 economics wording loop
|
||
|
||
## Results Grid/Threshold
|
||
|
||
**Passing criterion:** per-cell ≥2/3 (sonnet) / 2/2 (fable) reps PASS on the frozen
|
||
deterministic checker; H3 econ thresholds pre-registered in
|
||
`docs/plans/ws4-orchestration-economics.md` (E5P fable median mltok ≤37,783 = 60% of
|
||
baseline; E3P non-regression guard; negatives must stay clean).
|
||
|
||
Pre-wording baseline (2026-07-08, 44 counted reps) → candidate-1 (iter-1 15 reps + confirm
|
||
40 reps) → frozen reserve (40 reps). Zero harness errors; every rep swept for
|
||
session-limit truncation (15 baseline reps excluded+replaced by that mechanical criterion).
|
||
|
||
| Cell | baseline | confirm (candidate) | reserve twin |
|
||
|---|---|---|---|
|
||
| E5P fable | 0/3 (grind, median mltok 62,971) | 2/2 (median across 5 candidate reps 27,372 = 43%) | 2/2 |
|
||
| E5P sonnet | 0/3 | 2/3 (+3/3 iter-1) | 3/3 |
|
||
| E1P fable | 1/2 | 1/2 (B judge flicker; 3/4 across iter+confirm) | 1/2 (A miss) |
|
||
| E1P sonnet | 0/3 | 0/3 | 0/3 |
|
||
| E2P / E3P (redesigned criteria) | 6/6 / 6/6 | 5/5 / 5/5 | 5/5 / 5/5 |
|
||
| all negatives (E1N/E2N/E3N/E5N) | clean | 16/16 | 20/20 |
|
||
|
||
**Summary:** H1 (symmetric cost-framed triggers) and H2 (batching: ≤3 grouped spawns, one
|
||
round, explicit models — reserve fable rep used 2 explicit-haiku spawns) confirmed; H3 met
|
||
on the pre-registered E5P-fable axis (43% of baseline mltok). E1P sonnet never delegates at
|
||
any point (fixture too small to force it — pre-registered limitation); zero over-delegation
|
||
anywhere, including E5N sequential tasks under the new batching rule.
|
||
|
||
## Measurement Setup
|
||
|
||
**Hypotheses / Scenarios Tested** — H1/H2/H3 pre-registered in the WS4 plan; E5
|
||
(batching/implementation fan-out, mirrors the 2026-07-08 mini-audit Write×13 exemplar) added
|
||
with reserve twins authored by a subagent so held-out content never entered the tuning
|
||
session's context. E2P/E3P criteria redesigned per the 2026-07-06 pre-registration (scripted-
|
||
direct PASS branch; 74KB S7 ingestion anchor + mechanical root-cause concepts).
|
||
|
||
**Fixture and Sampling** — relaystation Node.js fixture + new handler-spec (12 pairwise-
|
||
distinct handler modules, deliberately non-templatable). sonnet ×3 / fable ×2 reps per cell;
|
||
fable at reduced reps for cost per plan.
|
||
|
||
**Experimental Control (Frozen Surfaces)** — checker/extractor/runner/fixtures/scenarios/
|
||
rubric frozen during the loop; ORCHESTRATION.md wording the only tuning surface;
|
||
`bin/refresh-plugins` + cache==source diff before every grid. One instrument fix landed
|
||
between confirm and reserve (E3P concepts keyed by scenario id — the reserve twin targets
|
||
gen-logs' other planted incident; fixed without reading reserve files, run-set scoring
|
||
byte-identical, self-test 34/34).
|
||
|
||
## Validity and Limitations
|
||
|
||
**How to Interpret These Results** — run-set AND reserve are now both contaminated for
|
||
future wording tuning (reserve was the final measurement). The reserve pass is the
|
||
generalization claim: same axes, different surface domain, never read informally. Econ
|
||
figures are real token sums (subagent transcripts persist under
|
||
`<session-id>/subagents/agent-*.jsonl` — see [[claude-code-subagent-transcripts-and-token-accounting]]),
|
||
not the char-count proxy the mini-audit used.
|
||
|
||
**Weaknesses of This Eval** — headless `-p` population may differ from interactive sessions
|
||
(unresolved since the 2026-07-06 baseline). E1P fixture too small to force delegation on
|
||
sonnet; its cell says nothing about capability. H3(c) sonnet econ guard (≤150% of grind
|
||
baseline) was breached at median 168%: delegation carries a real ~+6.6k-token premium on the
|
||
cheap tier for identical task completion — accepted because the frozen checker requires
|
||
delegate-or-script on a non-scriptable task and fable is the population of concern. Single
|
||
fixture. E5P mltok variance is high on fable (15k–65k baseline; 22k–47k candidate).
|
||
|
||
## Transferable Lessons
|
||
|
||
- Econ guards on cheap tiers should be absolute-token or scriptability-conditioned, not
|
||
ratios of a grind baseline — a ratio guard punishes correct delegation when the baseline
|
||
behavior is an efficient grind. (Promoted to [[eval-methodology-ladder]].)
|
||
- Account session limits truncate headless reps into plausible-looking behavior FAILs (and
|
||
PASSes); sweep every rep's final assistant message for the limit banner before counting.
|
||
(Promoted to [[running-autoresearch-skill-evals]].)
|
||
- Reserve twins authored before a criterion redesign can be silently unscoreable by the new
|
||
checker; audit checker constants against every planted fixture incident (not just run-set)
|
||
before the reserve run — fixable by id-keyed criteria without reading reserve files.
|
||
- Delegating reserve-twin authoring to a subagent (report = filenames only) keeps held-out
|
||
content out of the tuning session's context — cheap insurance when the same session will
|
||
run the wording loop.
|
||
|
||
**Promotion check:** promoted lesson 1 to [[eval-methodology-ladder]] and lesson 2 to
|
||
[[running-autoresearch-skill-evals]] (both updated 2026-07-08); lessons 3–4 not yet
|
||
recurring.
|
||
|
||
## Deployment and Evolution
|
||
|
||
**Good-Enough Gate** — shipped 2026-07-08 (ORCHESTRATION.md v3 in cc-os + refreshed cache):
|
||
negatives clean on both grids, batching generalizes to the reserve twin on both tiers, H3
|
||
met on the pre-registered axis. Residual known gaps at ship time: E1P sonnet delegate-at-all
|
||
(fixture-scale, not wording), fable downgrade-flag ~1/4 judge-boundary flicker (os-adr W3
|
||
shape — characterize in production, don't lab-chase).
|
||
|
||
**Proposed Changes (Loop Inputs)** — none; candidate-1 shipped verbatim (five changes:
|
||
cost-asymmetry + tier-conditional threshold opener, symmetric delegate/work-directly trigger
|
||
lists, batching rule ~5–8 items/spawn + SendMessage reuse, async rule, Workflow effort
|
||
dial). Diff against v2 is commit-visible in cc-os.
|
||
|
||
**Hardening Path / Next Measurement** — production IRL re-audit of ~5 real Fable sessions
|
||
scheduled ~2026-07-22 per [[eval-methodology-irl-feedback-loop]]: compare main-loop token
|
||
share vs the 2026-07-08 mini-audit baseline (now measurable exactly via subagent
|
||
transcripts); promote new miss shapes into eval scenarios. E1P needs a larger fixture before
|
||
its sonnet cell is meaningful.
|
||
|
||
## Related
|
||
|
||
- [[os-orchestration-eval-baseline-grid-results]] — the 2026-07-06 E1–E3 baseline this loop builds on
|
||
- [[orchestration-prompting-claude-5-era]] — the model-page guidance the v3 wording encodes
|
||
- [[eval-methodology-ladder]] / [[running-autoresearch-skill-evals]] — discipline followed + lessons promoted
|
||
- [[claude-code-subagent-transcripts-and-token-accounting]] — the instrumentation discovery behind the econ axes
|