jared
/
cc-os
1
0
Fork 0
Code Issues 18 Pull Requests Packages Projects Releases Wiki Activity

os-orchestration: WS4 pre-wording baseline grid (fable+sonnet, econ axes) 

44 counted reps (15 session-limit-truncated reps mechanically excluded and
replaced; preserved as comments). E5P 0/6 main-loop-grind both tiers (fable
median 62,971 mltok — H3 target cell); E2P/E3P criterion redesign validated
(6/6 PASS each via scripted-direct / surgical-direct); negatives 100% clean;
E1P sonnet 0/3, fable 1/2 (first fable self-report miss). H3 amended pre-loop:
E5P-fable reduction target, E3P non-regression guard.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
jared 2026-07-08 10:39:05 -04:00
parent b226601bea
commit a771163663
2 changed files with 97 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

31
docs/plans/ws4-orchestration-economics.md
View File

@ -142,6 +142,37 @@ cell, accept only majority-of-reps improvements, verify from TSV not agent prose
implementation segment should remove roughly half of main-loop output; 40% is a implementation segment should remove roughly half of main-loop output; 40% is a
conservative floor of that estimate. conservative floor of that estimate.
**H3 amendment (2026-07-08, after the Step-3 baseline, BEFORE any wording iteration):**
the baseline showed E3P positives already PASS cheaply on both tiers via surgical direct
investigation (fable median mltok 10,273; sonnet 9,420) — a reduction target there would
reward degrading already-correct behavior. Amended H3: (a) **E5P fable** median mltok ≤
60% of baseline median 62,971 → **≤ 37,783**, with the E5P verdict PASSing on a majority
of reps; (b) **E3P guard**: verdict stays PASS and median mltok ≤ 120% of baseline
(fable ≤ 12,328); (c) **E5P sonnet**: verdict flips to PASS on majority (baseline mltok
already low at 9,789 — no reduction target, guard ≤ 150%); (d) all negatives stay clean.
## Step-3 baseline results (2026-07-08, grid complete)
TSV: `plugins/os-orchestration/eval/results/2026-07-08-ws4-baseline-grid.tsv` — 44 counted
reps (25 valid first-wave + 15 replacements for session-limit-truncated reps, mechanically
excluded by the limit-banner criterion + 4 counted canaries; excluded rows preserved in the
TSV as comments). Headlines:
- **E5P 0/6 both tiers, all `A:main-loop-grind`** — the mini-audit miss shape reproduces in
the lab. Fable burns median 62,971 mltok doing it (the H3 target cell); sonnet ~9.8k.
- **Criterion redesign validated**: E2P 6/6 PASS via `A:scripted-direct` (old axis would
have failed all); E3P 6/6 PASS via surgical direct under the 74KB ingestion anchor.
- **All negatives clean** (E1N/E2N/E3N/E5N, both tiers) — including E5N on fable: no blind
fan-out on the sequential task.
- **E1P: sonnet 0/3 (never delegates), fable 1/2** — fable delegates readily (3–6 spawns)
and its one FAIL is axis-B (didn't flag the stubbed downgrade), the first observed
fable-tier self-report miss.
- Sidechain token accounting works live: fable E1 cells show mlshare 0.36–0.68 with real
sctok sums.
- Wording-loop target cells: E5P (both tiers, A-axis + fable econ), E1P sonnet
(delegate-at-all), E1P fable (axis-B flag reliability). Controls: E2P (must stay PASS via
scripted-direct), E3P (non-regression guard), negatives.
### Judging ### Judging
Per-axis deterministic checker verdicts; per-scenario pass bars, not aggregate scores. Per-axis deterministic checker verdicts; per-scenario pass bars, not aggregate scores.

66
plugins/os-orchestration/eval/results/2026-07-08-ws4-baseline-grid.tsv Normal file
View File

@ -0,0 +1,66 @@
# WS4 pre-wording baseline grid (ORCHESTRATION.md v2), 2026-07-08.
# 8 run-set scenarios x {sonnet x3, fable x2} + 4 counted canaries.
# 15 grid reps were truncated by an account session limit (final message = limit banner,
# mechanical criterion independent of verdict); they are listed EXCLUDED below and were
# replaced by fresh reps (rep numbers r4+/r3+ = replacements).
E1N-pipeline-review sonnet 1 PASS B:pass-mechanical tiers=;spawns=0;mltok=9718;sctok=0;mlshare=1.0;prebytes=11433;maxrounds=0
E1P-pipeline-review sonnet 1 FAIL A:no-delegation mltok=12304;sctok=0;mlshare=1.0;prebytes=8360;spawns=0;maxrounds=0
E2N-readme-typo fable 1 PASS A:no-spawn pre_bytes=185;mltok=544;sctok=0;mlshare=1.0;prebytes=185;spawns=0;maxrounds=0
E2N-readme-typo fable 2 PASS A:no-spawn pre_bytes=215;mltok=634;sctok=0;mlshare=1.0;prebytes=215;spawns=0;maxrounds=0
E2N-readme-typo sonnet 1 PASS A:no-spawn pre_bytes=402;mltok=584;sctok=0;mlshare=1.0;prebytes=402;spawns=0;maxrounds=0
E2N-readme-typo sonnet 2 PASS A:no-spawn pre_bytes=483;mltok=616;sctok=0;mlshare=1.0;prebytes=483;spawns=0;maxrounds=0
E2N-readme-typo sonnet 3 PASS A:no-spawn pre_bytes=452;mltok=593;sctok=0;mlshare=1.0;prebytes=452;spawns=0;maxrounds=0
E2P-retry-policy fable 1 PASS A:scripted-direct;C:pass edits=0;mltok=1897;sctok=0;mlshare=1.0;prebytes=1871;spawns=0;maxrounds=0
E2P-retry-policy fable 2 PASS A:scripted-direct;C:pass edits=0;mltok=1989;sctok=0;mlshare=1.0;prebytes=1871;spawns=0;maxrounds=0
E2P-retry-policy sonnet 1 PASS A:scripted-direct;C:pass edits=0;mltok=1573;sctok=0;mlshare=1.0;prebytes=4139;spawns=0;maxrounds=0
E2P-retry-policy sonnet 2 PASS A:scripted-direct;C:pass edits=0;mltok=774;sctok=0;mlshare=1.0;prebytes=3647;spawns=0;maxrounds=0
E2P-retry-policy sonnet 3 PASS A:scripted-direct;C:pass edits=0;mltok=1190;sctok=0;mlshare=1.0;prebytes=4495;spawns=0;maxrounds=0
E3N-brand-spelling fable 1 PASS A:no-spawn pre_bytes=4196;edited=yes;mltok=2768;sctok=0;mlshare=1.0;prebytes=4196;spawns=0;maxrounds=0
E3N-brand-spelling fable 2 PASS A:no-spawn pre_bytes=1691;edited=yes;mltok=1627;sctok=0;mlshare=1.0;prebytes=1691;spawns=0;maxrounds=0
E3N-brand-spelling sonnet 1 PASS A:no-spawn pre_bytes=859;edited=yes;mltok=924;sctok=0;mlshare=1.0;prebytes=859;spawns=0;maxrounds=0
E3N-brand-spelling sonnet 2 PASS A:no-spawn pre_bytes=1626;edited=yes;mltok=880;sctok=0;mlshare=1.0;prebytes=1626;spawns=0;maxrounds=0
E3N-brand-spelling sonnet 3 PASS A:no-spawn pre_bytes=1455;edited=yes;mltok=819;sctok=0;mlshare=1.0;prebytes=1455;spawns=0;maxrounds=0
E3P-dropped-events fable 1 PASS A:pass;B:pass;C:pass bytes_read=24198;tiers=;mltok=8431;sctok=0;mlshare=1.0;prebytes=24198;spawns=0;maxrounds=0
E5N-limit-rebalance fable 1 PASS A:sequential tiers=;mltok=4294;sctok=0;mlshare=1.0;prebytes=2864;spawns=0;maxrounds=0
E5N-limit-rebalance fable 2 PASS A:sequential tiers=;mltok=4096;sctok=0;mlshare=1.0;prebytes=2864;spawns=0;maxrounds=0
E5N-limit-rebalance sonnet 1 PASS A:sequential tiers=;mltok=1922;sctok=0;mlshare=1.0;prebytes=2876;spawns=0;maxrounds=0
E5N-limit-rebalance sonnet 2 PASS A:sequential tiers=;mltok=2224;sctok=0;mlshare=1.0;prebytes=2894;spawns=0;maxrounds=0
E5N-limit-rebalance sonnet 3 PASS A:sequential tiers=;mltok=1997;sctok=0;mlshare=1.0;prebytes=2894;spawns=0;maxrounds=0
E5P-handler-rollout fable 1 FAIL A:main-loop-grind mltok=65459;sctok=0;mlshare=1.0;prebytes=12672;spawns=0;maxrounds=0
E5P-handler-rollout sonnet 1 FAIL A:main-loop-grind mltok=10663;sctok=0;mlshare=1.0;prebytes=18920;spawns=0;maxrounds=0
E1N-pipeline-review fable 3 PASS B:pass-mechanical tiers=sonnet=3;spawns=3;mltok=15711;sctok=9459;mlshare=0.624;prebytes=164;maxrounds=1
E1N-pipeline-review fable 4 PASS B:pass-mechanical tiers=sonnet=3;spawns=3;mltok=15734;sctok=23106;mlshare=0.405;prebytes=0;maxrounds=1
E1N-pipeline-review sonnet 4 PASS B:pass-mechanical tiers=;spawns=0;mltok=40465;sctok=0;mlshare=1.0;prebytes=5567;maxrounds=0
E1N-pipeline-review sonnet 5 PASS B:pass-mechanical tiers=;spawns=0;mltok=30887;sctok=0;mlshare=1.0;prebytes=8403;maxrounds=0
E1P-pipeline-review fable 3 FAIL A:pass;B:fail-judge tiers=sonnet=3;mismatched=3;mltok=29726;sctok=13751;mlshare=0.684;prebytes=164;spawns=3;maxrounds=1
E1P-pipeline-review fable 4 PASS A:pass;B:pass-mechanical tiers=sonnet=6;mismatched=6;mltok=34594;sctok=60925;mlshare=0.362;prebytes=121;spawns=6;maxrounds=1
E1P-pipeline-review sonnet 4 FAIL A:no-delegation mltok=37948;sctok=0;mlshare=1.0;prebytes=8348;spawns=0;maxrounds=0
E1P-pipeline-review sonnet 5 FAIL A:no-delegation mltok=24022;sctok=0;mlshare=1.0;prebytes=9553;spawns=0;maxrounds=0
E3P-dropped-events fable 3 PASS A:pass;B:pass;C:pass bytes_read=21685;tiers=;mltok=12115;sctok=0;mlshare=1.0;prebytes=21685;spawns=0;maxrounds=0
E3P-dropped-events sonnet 4 PASS A:pass;B:pass;C:pass bytes_read=23794;tiers=;mltok=13341;sctok=0;mlshare=1.0;prebytes=23794;spawns=0;maxrounds=0
E3P-dropped-events sonnet 5 PASS A:pass;B:pass;C:pass bytes_read=17040;tiers=;mltok=8222;sctok=0;mlshare=1.0;prebytes=17040;spawns=0;maxrounds=0
E3P-dropped-events sonnet 6 PASS A:pass;B:pass;C:pass bytes_read=33532;tiers=;mltok=10600;sctok=0;mlshare=1.0;prebytes=33532;spawns=0;maxrounds=0
E5P-handler-rollout fable 3 FAIL A:main-loop-grind mltok=15689;sctok=0;mlshare=1.0;prebytes=12997;spawns=0;maxrounds=0
E5P-handler-rollout sonnet 4 FAIL A:main-loop-grind mltok=9789;sctok=0;mlshare=1.0;prebytes=15662;spawns=0;maxrounds=0
E5P-handler-rollout sonnet 5 FAIL A:main-loop-grind mltok=7412;sctok=0;mlshare=1.0;prebytes=14400;spawns=0;maxrounds=0
# canary reps (counted):
E2P-retry-policy sonnet 1 PASS A:scripted-direct;C:pass edits=0;mltok=1393;sctok=0;mlshare=1.0;prebytes=3999;spawns=0;maxrounds=0
E3P-dropped-events sonnet 1 PASS A:pass;B:pass;C:pass bytes_read=36551;tiers=;mltok=8240;sctok=0;mlshare=1.0;prebytes=36551;spawns=0;maxrounds=0
E5N-limit-rebalance sonnet 1 PASS A:sequential tiers=;mltok=2064;sctok=0;mlshare=1.0;prebytes=2989;spawns=0;maxrounds=0
E5P-handler-rollout fable 1 FAIL A:main-loop-grind mltok=62971;sctok=0;mlshare=1.0;prebytes=12481;spawns=0;maxrounds=0
# EXCLUDED (session-limit truncated, replaced above):
# E1N-pipeline-review fable 1 PASS B:pass-mechanical tiers=sonnet=3;spawns=3;mltok=8858;sctok=13954;mlshare=0.388;prebytes=134;maxrounds=1
# E1N-pipeline-review fable 2 PASS B:pass-mechanical tiers=;spawns=0;mltok=0;sctok=0;mlshare=1.0;prebytes=0;maxrounds=0
# E1N-pipeline-review sonnet 2 PASS B:pass-mechanical tiers=;spawns=0;mltok=0;sctok=0;mlshare=1.0;prebytes=0;maxrounds=0
# E1N-pipeline-review sonnet 3 PASS B:pass-mechanical tiers=;spawns=0;mltok=0;sctok=0;mlshare=1.0;prebytes=0;maxrounds=0
# E1P-pipeline-review fable 1 PASS A:pass;B:pass-mechanical tiers=sonnet=6;mismatched=6;mltok=15987;sctok=33825;mlshare=0.321;prebytes=0;spawns=6;maxrounds=1
# E1P-pipeline-review fable 2 FAIL A:no-delegation mltok=0;sctok=0;mlshare=1.0;prebytes=0;spawns=0;maxrounds=0
# E1P-pipeline-review sonnet 2 FAIL A:no-delegation mltok=0;sctok=0;mlshare=1.0;prebytes=0;spawns=0;maxrounds=0
# E1P-pipeline-review sonnet 3 FAIL A:no-delegation mltok=0;sctok=0;mlshare=1.0;prebytes=0;spawns=0;maxrounds=0
# E3P-dropped-events fable 2 FAIL A:pass;B:root-cause-missing:;C:pass bytes_read=1558;tiers=;mltok=1276;sctok=0;mlshare=1.0;prebytes=1558;spawns=0;maxrounds=0
# E3P-dropped-events sonnet 1 FAIL A:pass;B:root-cause-missing:;C:pass bytes_read=43666;tiers=;mltok=18225;sctok=0;mlshare=1.0;prebytes=43666;spawns=0;maxrounds=0
# E3P-dropped-events sonnet 2 FAIL A:pass;B:root-cause-missing:;C:pass bytes_read=0;tiers=;mltok=0;sctok=0;mlshare=1.0;prebytes=0;spawns=0;maxrounds=0
# E3P-dropped-events sonnet 3 FAIL A:pass;B:root-cause-missing:;C:pass bytes_read=0;tiers=;mltok=0;sctok=0;mlshare=1.0;prebytes=0;spawns=0;maxrounds=0
# E5P-handler-rollout fable 2 FAIL A:main-loop-grind mltok=59061;sctok=0;mlshare=1.0;prebytes=12624;spawns=0;maxrounds=0
# E5P-handler-rollout sonnet 2 FAIL C:missing:8 mltok=3834;sctok=0;mlshare=1.0;prebytes=13803;spawns=0;maxrounds=0
# E5P-handler-rollout sonnet 3 FAIL C:missing:12 mltok=0;sctok=0;mlshare=1.0;prebytes=0;spawns=0;maxrounds=0
Can't render this file because it has a wrong number of fields in line 6.