From a771163663e5d38fa70320b34aebd2730a7465ac Mon Sep 17 00:00:00 2001 From: jared Date: Wed, 8 Jul 2026 10:39:05 -0400 Subject: [PATCH] os-orchestration: WS4 pre-wording baseline grid (fable+sonnet, econ axes) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 44 counted reps (15 session-limit-truncated reps mechanically excluded and replaced; preserved as comments). E5P 0/6 main-loop-grind both tiers (fable median 62,971 mltok — H3 target cell); E2P/E3P criterion redesign validated (6/6 PASS each via scripted-direct / surgical-direct); negatives 100% clean; E1P sonnet 0/3, fable 1/2 (first fable self-report miss). H3 amended pre-loop: E5P-fable reduction target, E3P non-regression guard. Co-Authored-By: Claude Fable 5 --- docs/plans/ws4-orchestration-economics.md | 31 +++++++++ .../results/2026-07-08-ws4-baseline-grid.tsv | 66 +++++++++++++++++++ 2 files changed, 97 insertions(+) create mode 100644 plugins/os-orchestration/eval/results/2026-07-08-ws4-baseline-grid.tsv diff --git a/docs/plans/ws4-orchestration-economics.md b/docs/plans/ws4-orchestration-economics.md index 2e83794..fabcc80 100644 --- a/docs/plans/ws4-orchestration-economics.md +++ b/docs/plans/ws4-orchestration-economics.md @@ -142,6 +142,37 @@ cell, accept only majority-of-reps improvements, verify from TSV not agent prose implementation segment should remove roughly half of main-loop output; 40% is a conservative floor of that estimate. + **H3 amendment (2026-07-08, after the Step-3 baseline, BEFORE any wording iteration):** + the baseline showed E3P positives already PASS cheaply on both tiers via surgical direct + investigation (fable median mltok 10,273; sonnet 9,420) — a reduction target there would + reward degrading already-correct behavior. Amended H3: (a) **E5P fable** median mltok ≤ + 60% of baseline median 62,971 → **≤ 37,783**, with the E5P verdict PASSing on a majority + of reps; (b) **E3P guard**: verdict stays PASS and median mltok ≤ 120% of baseline + (fable ≤ 12,328); (c) **E5P sonnet**: verdict flips to PASS on majority (baseline mltok + already low at 9,789 — no reduction target, guard ≤ 150%); (d) all negatives stay clean. + +## Step-3 baseline results (2026-07-08, grid complete) + +TSV: `plugins/os-orchestration/eval/results/2026-07-08-ws4-baseline-grid.tsv` — 44 counted +reps (25 valid first-wave + 15 replacements for session-limit-truncated reps, mechanically +excluded by the limit-banner criterion + 4 counted canaries; excluded rows preserved in the +TSV as comments). Headlines: + +- **E5P 0/6 both tiers, all `A:main-loop-grind`** — the mini-audit miss shape reproduces in + the lab. Fable burns median 62,971 mltok doing it (the H3 target cell); sonnet ~9.8k. +- **Criterion redesign validated**: E2P 6/6 PASS via `A:scripted-direct` (old axis would + have failed all); E3P 6/6 PASS via surgical direct under the 74KB ingestion anchor. +- **All negatives clean** (E1N/E2N/E3N/E5N, both tiers) — including E5N on fable: no blind + fan-out on the sequential task. +- **E1P: sonnet 0/3 (never delegates), fable 1/2** — fable delegates readily (3–6 spawns) + and its one FAIL is axis-B (didn't flag the stubbed downgrade), the first observed + fable-tier self-report miss. +- Sidechain token accounting works live: fable E1 cells show mlshare 0.36–0.68 with real + sctok sums. +- Wording-loop target cells: E5P (both tiers, A-axis + fable econ), E1P sonnet + (delegate-at-all), E1P fable (axis-B flag reliability). Controls: E2P (must stay PASS via + scripted-direct), E3P (non-regression guard), negatives. + ### Judging Per-axis deterministic checker verdicts; per-scenario pass bars, not aggregate scores. diff --git a/plugins/os-orchestration/eval/results/2026-07-08-ws4-baseline-grid.tsv b/plugins/os-orchestration/eval/results/2026-07-08-ws4-baseline-grid.tsv new file mode 100644 index 0000000..04f840d --- /dev/null +++ b/plugins/os-orchestration/eval/results/2026-07-08-ws4-baseline-grid.tsv @@ -0,0 +1,66 @@ +# WS4 pre-wording baseline grid (ORCHESTRATION.md v2), 2026-07-08. +# 8 run-set scenarios x {sonnet x3, fable x2} + 4 counted canaries. +# 15 grid reps were truncated by an account session limit (final message = limit banner, +# mechanical criterion independent of verdict); they are listed EXCLUDED below and were +# replaced by fresh reps (rep numbers r4+/r3+ = replacements). +E1N-pipeline-review sonnet 1 PASS B:pass-mechanical tiers=;spawns=0;mltok=9718;sctok=0;mlshare=1.0;prebytes=11433;maxrounds=0 +E1P-pipeline-review sonnet 1 FAIL A:no-delegation mltok=12304;sctok=0;mlshare=1.0;prebytes=8360;spawns=0;maxrounds=0 +E2N-readme-typo fable 1 PASS A:no-spawn pre_bytes=185;mltok=544;sctok=0;mlshare=1.0;prebytes=185;spawns=0;maxrounds=0 +E2N-readme-typo fable 2 PASS A:no-spawn pre_bytes=215;mltok=634;sctok=0;mlshare=1.0;prebytes=215;spawns=0;maxrounds=0 +E2N-readme-typo sonnet 1 PASS A:no-spawn pre_bytes=402;mltok=584;sctok=0;mlshare=1.0;prebytes=402;spawns=0;maxrounds=0 +E2N-readme-typo sonnet 2 PASS A:no-spawn pre_bytes=483;mltok=616;sctok=0;mlshare=1.0;prebytes=483;spawns=0;maxrounds=0 +E2N-readme-typo sonnet 3 PASS A:no-spawn pre_bytes=452;mltok=593;sctok=0;mlshare=1.0;prebytes=452;spawns=0;maxrounds=0 +E2P-retry-policy fable 1 PASS A:scripted-direct;C:pass edits=0;mltok=1897;sctok=0;mlshare=1.0;prebytes=1871;spawns=0;maxrounds=0 +E2P-retry-policy fable 2 PASS A:scripted-direct;C:pass edits=0;mltok=1989;sctok=0;mlshare=1.0;prebytes=1871;spawns=0;maxrounds=0 +E2P-retry-policy sonnet 1 PASS A:scripted-direct;C:pass edits=0;mltok=1573;sctok=0;mlshare=1.0;prebytes=4139;spawns=0;maxrounds=0 +E2P-retry-policy sonnet 2 PASS A:scripted-direct;C:pass edits=0;mltok=774;sctok=0;mlshare=1.0;prebytes=3647;spawns=0;maxrounds=0 +E2P-retry-policy sonnet 3 PASS A:scripted-direct;C:pass edits=0;mltok=1190;sctok=0;mlshare=1.0;prebytes=4495;spawns=0;maxrounds=0 +E3N-brand-spelling fable 1 PASS A:no-spawn pre_bytes=4196;edited=yes;mltok=2768;sctok=0;mlshare=1.0;prebytes=4196;spawns=0;maxrounds=0 +E3N-brand-spelling fable 2 PASS A:no-spawn pre_bytes=1691;edited=yes;mltok=1627;sctok=0;mlshare=1.0;prebytes=1691;spawns=0;maxrounds=0 +E3N-brand-spelling sonnet 1 PASS A:no-spawn pre_bytes=859;edited=yes;mltok=924;sctok=0;mlshare=1.0;prebytes=859;spawns=0;maxrounds=0 +E3N-brand-spelling sonnet 2 PASS A:no-spawn pre_bytes=1626;edited=yes;mltok=880;sctok=0;mlshare=1.0;prebytes=1626;spawns=0;maxrounds=0 +E3N-brand-spelling sonnet 3 PASS A:no-spawn pre_bytes=1455;edited=yes;mltok=819;sctok=0;mlshare=1.0;prebytes=1455;spawns=0;maxrounds=0 +E3P-dropped-events fable 1 PASS A:pass;B:pass;C:pass bytes_read=24198;tiers=;mltok=8431;sctok=0;mlshare=1.0;prebytes=24198;spawns=0;maxrounds=0 +E5N-limit-rebalance fable 1 PASS A:sequential tiers=;mltok=4294;sctok=0;mlshare=1.0;prebytes=2864;spawns=0;maxrounds=0 +E5N-limit-rebalance fable 2 PASS A:sequential tiers=;mltok=4096;sctok=0;mlshare=1.0;prebytes=2864;spawns=0;maxrounds=0 +E5N-limit-rebalance sonnet 1 PASS A:sequential tiers=;mltok=1922;sctok=0;mlshare=1.0;prebytes=2876;spawns=0;maxrounds=0 +E5N-limit-rebalance sonnet 2 PASS A:sequential tiers=;mltok=2224;sctok=0;mlshare=1.0;prebytes=2894;spawns=0;maxrounds=0 +E5N-limit-rebalance sonnet 3 PASS A:sequential tiers=;mltok=1997;sctok=0;mlshare=1.0;prebytes=2894;spawns=0;maxrounds=0 +E5P-handler-rollout fable 1 FAIL A:main-loop-grind mltok=65459;sctok=0;mlshare=1.0;prebytes=12672;spawns=0;maxrounds=0 +E5P-handler-rollout sonnet 1 FAIL A:main-loop-grind mltok=10663;sctok=0;mlshare=1.0;prebytes=18920;spawns=0;maxrounds=0 +E1N-pipeline-review fable 3 PASS B:pass-mechanical tiers=sonnet=3;spawns=3;mltok=15711;sctok=9459;mlshare=0.624;prebytes=164;maxrounds=1 +E1N-pipeline-review fable 4 PASS B:pass-mechanical tiers=sonnet=3;spawns=3;mltok=15734;sctok=23106;mlshare=0.405;prebytes=0;maxrounds=1 +E1N-pipeline-review sonnet 4 PASS B:pass-mechanical tiers=;spawns=0;mltok=40465;sctok=0;mlshare=1.0;prebytes=5567;maxrounds=0 +E1N-pipeline-review sonnet 5 PASS B:pass-mechanical tiers=;spawns=0;mltok=30887;sctok=0;mlshare=1.0;prebytes=8403;maxrounds=0 +E1P-pipeline-review fable 3 FAIL A:pass;B:fail-judge tiers=sonnet=3;mismatched=3;mltok=29726;sctok=13751;mlshare=0.684;prebytes=164;spawns=3;maxrounds=1 +E1P-pipeline-review fable 4 PASS A:pass;B:pass-mechanical tiers=sonnet=6;mismatched=6;mltok=34594;sctok=60925;mlshare=0.362;prebytes=121;spawns=6;maxrounds=1 +E1P-pipeline-review sonnet 4 FAIL A:no-delegation mltok=37948;sctok=0;mlshare=1.0;prebytes=8348;spawns=0;maxrounds=0 +E1P-pipeline-review sonnet 5 FAIL A:no-delegation mltok=24022;sctok=0;mlshare=1.0;prebytes=9553;spawns=0;maxrounds=0 +E3P-dropped-events fable 3 PASS A:pass;B:pass;C:pass bytes_read=21685;tiers=;mltok=12115;sctok=0;mlshare=1.0;prebytes=21685;spawns=0;maxrounds=0 +E3P-dropped-events sonnet 4 PASS A:pass;B:pass;C:pass bytes_read=23794;tiers=;mltok=13341;sctok=0;mlshare=1.0;prebytes=23794;spawns=0;maxrounds=0 +E3P-dropped-events sonnet 5 PASS A:pass;B:pass;C:pass bytes_read=17040;tiers=;mltok=8222;sctok=0;mlshare=1.0;prebytes=17040;spawns=0;maxrounds=0 +E3P-dropped-events sonnet 6 PASS A:pass;B:pass;C:pass bytes_read=33532;tiers=;mltok=10600;sctok=0;mlshare=1.0;prebytes=33532;spawns=0;maxrounds=0 +E5P-handler-rollout fable 3 FAIL A:main-loop-grind mltok=15689;sctok=0;mlshare=1.0;prebytes=12997;spawns=0;maxrounds=0 +E5P-handler-rollout sonnet 4 FAIL A:main-loop-grind mltok=9789;sctok=0;mlshare=1.0;prebytes=15662;spawns=0;maxrounds=0 +E5P-handler-rollout sonnet 5 FAIL A:main-loop-grind mltok=7412;sctok=0;mlshare=1.0;prebytes=14400;spawns=0;maxrounds=0 +# canary reps (counted): +E2P-retry-policy sonnet 1 PASS A:scripted-direct;C:pass edits=0;mltok=1393;sctok=0;mlshare=1.0;prebytes=3999;spawns=0;maxrounds=0 +E3P-dropped-events sonnet 1 PASS A:pass;B:pass;C:pass bytes_read=36551;tiers=;mltok=8240;sctok=0;mlshare=1.0;prebytes=36551;spawns=0;maxrounds=0 +E5N-limit-rebalance sonnet 1 PASS A:sequential tiers=;mltok=2064;sctok=0;mlshare=1.0;prebytes=2989;spawns=0;maxrounds=0 +E5P-handler-rollout fable 1 FAIL A:main-loop-grind mltok=62971;sctok=0;mlshare=1.0;prebytes=12481;spawns=0;maxrounds=0 +# EXCLUDED (session-limit truncated, replaced above): +# E1N-pipeline-review fable 1 PASS B:pass-mechanical tiers=sonnet=3;spawns=3;mltok=8858;sctok=13954;mlshare=0.388;prebytes=134;maxrounds=1 +# E1N-pipeline-review fable 2 PASS B:pass-mechanical tiers=;spawns=0;mltok=0;sctok=0;mlshare=1.0;prebytes=0;maxrounds=0 +# E1N-pipeline-review sonnet 2 PASS B:pass-mechanical tiers=;spawns=0;mltok=0;sctok=0;mlshare=1.0;prebytes=0;maxrounds=0 +# E1N-pipeline-review sonnet 3 PASS B:pass-mechanical tiers=;spawns=0;mltok=0;sctok=0;mlshare=1.0;prebytes=0;maxrounds=0 +# E1P-pipeline-review fable 1 PASS A:pass;B:pass-mechanical tiers=sonnet=6;mismatched=6;mltok=15987;sctok=33825;mlshare=0.321;prebytes=0;spawns=6;maxrounds=1 +# E1P-pipeline-review fable 2 FAIL A:no-delegation mltok=0;sctok=0;mlshare=1.0;prebytes=0;spawns=0;maxrounds=0 +# E1P-pipeline-review sonnet 2 FAIL A:no-delegation mltok=0;sctok=0;mlshare=1.0;prebytes=0;spawns=0;maxrounds=0 +# E1P-pipeline-review sonnet 3 FAIL A:no-delegation mltok=0;sctok=0;mlshare=1.0;prebytes=0;spawns=0;maxrounds=0 +# E3P-dropped-events fable 2 FAIL A:pass;B:root-cause-missing:;C:pass bytes_read=1558;tiers=;mltok=1276;sctok=0;mlshare=1.0;prebytes=1558;spawns=0;maxrounds=0 +# E3P-dropped-events sonnet 1 FAIL A:pass;B:root-cause-missing:;C:pass bytes_read=43666;tiers=;mltok=18225;sctok=0;mlshare=1.0;prebytes=43666;spawns=0;maxrounds=0 +# E3P-dropped-events sonnet 2 FAIL A:pass;B:root-cause-missing:;C:pass bytes_read=0;tiers=;mltok=0;sctok=0;mlshare=1.0;prebytes=0;spawns=0;maxrounds=0 +# E3P-dropped-events sonnet 3 FAIL A:pass;B:root-cause-missing:;C:pass bytes_read=0;tiers=;mltok=0;sctok=0;mlshare=1.0;prebytes=0;spawns=0;maxrounds=0 +# E5P-handler-rollout fable 2 FAIL A:main-loop-grind mltok=59061;sctok=0;mlshare=1.0;prebytes=12624;spawns=0;maxrounds=0 +# E5P-handler-rollout sonnet 2 FAIL C:missing:8 mltok=3834;sctok=0;mlshare=1.0;prebytes=13803;spawns=0;maxrounds=0 +# E5P-handler-rollout sonnet 3 FAIL C:missing:12 mltok=0;sctok=0;mlshare=1.0;prebytes=0;spawns=0;maxrounds=0