jared
/
cc-os
1
0
Fork 0
Code Issues 18 Pull Requests Packages Projects Releases Wiki Activity
cc-os/plugins/os-orchestration/eval/results/2026-07-08-ws4-baseline-gri...

67 lines
7.9 KiB
Plaintext
Raw Normal View History

os-orchestration: WS4 pre-wording baseline grid (fable+sonnet, econ axes) 44 counted reps (15 session-limit-truncated reps mechanically excluded and replaced; preserved as comments). E5P 0/6 main-loop-grind both tiers (fable median 62,971 mltok — H3 target cell); E2P/E3P criterion redesign validated (6/6 PASS each via scripted-direct / surgical-direct); negatives 100% clean; E1P sonnet 0/3, fable 1/2 (first fable self-report miss). H3 amended pre-loop: E5P-fable reduction target, E3P non-regression guard. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 14:39:05 +00:00
# WS4 pre-wording baseline grid (ORCHESTRATION.md v2), 2026-07-08.
# 8 run-set scenarios x {sonnet x3, fable x2} + 4 counted canaries.
# 15 grid reps were truncated by an account session limit (final message = limit banner,
# mechanical criterion independent of verdict); they are listed EXCLUDED below and were
# replaced by fresh reps (rep numbers r4+/r3+ = replacements).
E1N-pipeline-review sonnet 1 PASS B:pass-mechanical tiers=;spawns=0;mltok=9718;sctok=0;mlshare=1.0;prebytes=11433;maxrounds=0
E1P-pipeline-review sonnet 1 FAIL A:no-delegation mltok=12304;sctok=0;mlshare=1.0;prebytes=8360;spawns=0;maxrounds=0
E2N-readme-typo fable 1 PASS A:no-spawn pre_bytes=185;mltok=544;sctok=0;mlshare=1.0;prebytes=185;spawns=0;maxrounds=0
E2N-readme-typo fable 2 PASS A:no-spawn pre_bytes=215;mltok=634;sctok=0;mlshare=1.0;prebytes=215;spawns=0;maxrounds=0
E2N-readme-typo sonnet 1 PASS A:no-spawn pre_bytes=402;mltok=584;sctok=0;mlshare=1.0;prebytes=402;spawns=0;maxrounds=0
E2N-readme-typo sonnet 2 PASS A:no-spawn pre_bytes=483;mltok=616;sctok=0;mlshare=1.0;prebytes=483;spawns=0;maxrounds=0
E2N-readme-typo sonnet 3 PASS A:no-spawn pre_bytes=452;mltok=593;sctok=0;mlshare=1.0;prebytes=452;spawns=0;maxrounds=0
E2P-retry-policy fable 1 PASS A:scripted-direct;C:pass edits=0;mltok=1897;sctok=0;mlshare=1.0;prebytes=1871;spawns=0;maxrounds=0
E2P-retry-policy fable 2 PASS A:scripted-direct;C:pass edits=0;mltok=1989;sctok=0;mlshare=1.0;prebytes=1871;spawns=0;maxrounds=0
E2P-retry-policy sonnet 1 PASS A:scripted-direct;C:pass edits=0;mltok=1573;sctok=0;mlshare=1.0;prebytes=4139;spawns=0;maxrounds=0
E2P-retry-policy sonnet 2 PASS A:scripted-direct;C:pass edits=0;mltok=774;sctok=0;mlshare=1.0;prebytes=3647;spawns=0;maxrounds=0
E2P-retry-policy sonnet 3 PASS A:scripted-direct;C:pass edits=0;mltok=1190;sctok=0;mlshare=1.0;prebytes=4495;spawns=0;maxrounds=0
E3N-brand-spelling fable 1 PASS A:no-spawn pre_bytes=4196;edited=yes;mltok=2768;sctok=0;mlshare=1.0;prebytes=4196;spawns=0;maxrounds=0
E3N-brand-spelling fable 2 PASS A:no-spawn pre_bytes=1691;edited=yes;mltok=1627;sctok=0;mlshare=1.0;prebytes=1691;spawns=0;maxrounds=0
E3N-brand-spelling sonnet 1 PASS A:no-spawn pre_bytes=859;edited=yes;mltok=924;sctok=0;mlshare=1.0;prebytes=859;spawns=0;maxrounds=0
E3N-brand-spelling sonnet 2 PASS A:no-spawn pre_bytes=1626;edited=yes;mltok=880;sctok=0;mlshare=1.0;prebytes=1626;spawns=0;maxrounds=0
E3N-brand-spelling sonnet 3 PASS A:no-spawn pre_bytes=1455;edited=yes;mltok=819;sctok=0;mlshare=1.0;prebytes=1455;spawns=0;maxrounds=0
E3P-dropped-events fable 1 PASS A:pass;B:pass;C:pass bytes_read=24198;tiers=;mltok=8431;sctok=0;mlshare=1.0;prebytes=24198;spawns=0;maxrounds=0
E5N-limit-rebalance fable 1 PASS A:sequential tiers=;mltok=4294;sctok=0;mlshare=1.0;prebytes=2864;spawns=0;maxrounds=0
E5N-limit-rebalance fable 2 PASS A:sequential tiers=;mltok=4096;sctok=0;mlshare=1.0;prebytes=2864;spawns=0;maxrounds=0
E5N-limit-rebalance sonnet 1 PASS A:sequential tiers=;mltok=1922;sctok=0;mlshare=1.0;prebytes=2876;spawns=0;maxrounds=0
E5N-limit-rebalance sonnet 2 PASS A:sequential tiers=;mltok=2224;sctok=0;mlshare=1.0;prebytes=2894;spawns=0;maxrounds=0
E5N-limit-rebalance sonnet 3 PASS A:sequential tiers=;mltok=1997;sctok=0;mlshare=1.0;prebytes=2894;spawns=0;maxrounds=0
E5P-handler-rollout fable 1 FAIL A:main-loop-grind mltok=65459;sctok=0;mlshare=1.0;prebytes=12672;spawns=0;maxrounds=0
E5P-handler-rollout sonnet 1 FAIL A:main-loop-grind mltok=10663;sctok=0;mlshare=1.0;prebytes=18920;spawns=0;maxrounds=0
E1N-pipeline-review fable 3 PASS B:pass-mechanical tiers=sonnet=3;spawns=3;mltok=15711;sctok=9459;mlshare=0.624;prebytes=164;maxrounds=1
E1N-pipeline-review fable 4 PASS B:pass-mechanical tiers=sonnet=3;spawns=3;mltok=15734;sctok=23106;mlshare=0.405;prebytes=0;maxrounds=1
E1N-pipeline-review sonnet 4 PASS B:pass-mechanical tiers=;spawns=0;mltok=40465;sctok=0;mlshare=1.0;prebytes=5567;maxrounds=0
E1N-pipeline-review sonnet 5 PASS B:pass-mechanical tiers=;spawns=0;mltok=30887;sctok=0;mlshare=1.0;prebytes=8403;maxrounds=0
E1P-pipeline-review fable 3 FAIL A:pass;B:fail-judge tiers=sonnet=3;mismatched=3;mltok=29726;sctok=13751;mlshare=0.684;prebytes=164;spawns=3;maxrounds=1
E1P-pipeline-review fable 4 PASS A:pass;B:pass-mechanical tiers=sonnet=6;mismatched=6;mltok=34594;sctok=60925;mlshare=0.362;prebytes=121;spawns=6;maxrounds=1
E1P-pipeline-review sonnet 4 FAIL A:no-delegation mltok=37948;sctok=0;mlshare=1.0;prebytes=8348;spawns=0;maxrounds=0
E1P-pipeline-review sonnet 5 FAIL A:no-delegation mltok=24022;sctok=0;mlshare=1.0;prebytes=9553;spawns=0;maxrounds=0
E3P-dropped-events fable 3 PASS A:pass;B:pass;C:pass bytes_read=21685;tiers=;mltok=12115;sctok=0;mlshare=1.0;prebytes=21685;spawns=0;maxrounds=0
E3P-dropped-events sonnet 4 PASS A:pass;B:pass;C:pass bytes_read=23794;tiers=;mltok=13341;sctok=0;mlshare=1.0;prebytes=23794;spawns=0;maxrounds=0
E3P-dropped-events sonnet 5 PASS A:pass;B:pass;C:pass bytes_read=17040;tiers=;mltok=8222;sctok=0;mlshare=1.0;prebytes=17040;spawns=0;maxrounds=0
E3P-dropped-events sonnet 6 PASS A:pass;B:pass;C:pass bytes_read=33532;tiers=;mltok=10600;sctok=0;mlshare=1.0;prebytes=33532;spawns=0;maxrounds=0
E5P-handler-rollout fable 3 FAIL A:main-loop-grind mltok=15689;sctok=0;mlshare=1.0;prebytes=12997;spawns=0;maxrounds=0
E5P-handler-rollout sonnet 4 FAIL A:main-loop-grind mltok=9789;sctok=0;mlshare=1.0;prebytes=15662;spawns=0;maxrounds=0
E5P-handler-rollout sonnet 5 FAIL A:main-loop-grind mltok=7412;sctok=0;mlshare=1.0;prebytes=14400;spawns=0;maxrounds=0
# canary reps (counted):
E2P-retry-policy sonnet 1 PASS A:scripted-direct;C:pass edits=0;mltok=1393;sctok=0;mlshare=1.0;prebytes=3999;spawns=0;maxrounds=0
E3P-dropped-events sonnet 1 PASS A:pass;B:pass;C:pass bytes_read=36551;tiers=;mltok=8240;sctok=0;mlshare=1.0;prebytes=36551;spawns=0;maxrounds=0
E5N-limit-rebalance sonnet 1 PASS A:sequential tiers=;mltok=2064;sctok=0;mlshare=1.0;prebytes=2989;spawns=0;maxrounds=0
E5P-handler-rollout fable 1 FAIL A:main-loop-grind mltok=62971;sctok=0;mlshare=1.0;prebytes=12481;spawns=0;maxrounds=0
# EXCLUDED (session-limit truncated, replaced above):
# E1N-pipeline-review fable 1 PASS B:pass-mechanical tiers=sonnet=3;spawns=3;mltok=8858;sctok=13954;mlshare=0.388;prebytes=134;maxrounds=1
# E1N-pipeline-review fable 2 PASS B:pass-mechanical tiers=;spawns=0;mltok=0;sctok=0;mlshare=1.0;prebytes=0;maxrounds=0
# E1N-pipeline-review sonnet 2 PASS B:pass-mechanical tiers=;spawns=0;mltok=0;sctok=0;mlshare=1.0;prebytes=0;maxrounds=0
# E1N-pipeline-review sonnet 3 PASS B:pass-mechanical tiers=;spawns=0;mltok=0;sctok=0;mlshare=1.0;prebytes=0;maxrounds=0
# E1P-pipeline-review fable 1 PASS A:pass;B:pass-mechanical tiers=sonnet=6;mismatched=6;mltok=15987;sctok=33825;mlshare=0.321;prebytes=0;spawns=6;maxrounds=1
# E1P-pipeline-review fable 2 FAIL A:no-delegation mltok=0;sctok=0;mlshare=1.0;prebytes=0;spawns=0;maxrounds=0
# E1P-pipeline-review sonnet 2 FAIL A:no-delegation mltok=0;sctok=0;mlshare=1.0;prebytes=0;spawns=0;maxrounds=0
# E1P-pipeline-review sonnet 3 FAIL A:no-delegation mltok=0;sctok=0;mlshare=1.0;prebytes=0;spawns=0;maxrounds=0
# E3P-dropped-events fable 2 FAIL A:pass;B:root-cause-missing:;C:pass bytes_read=1558;tiers=;mltok=1276;sctok=0;mlshare=1.0;prebytes=1558;spawns=0;maxrounds=0
# E3P-dropped-events sonnet 1 FAIL A:pass;B:root-cause-missing:;C:pass bytes_read=43666;tiers=;mltok=18225;sctok=0;mlshare=1.0;prebytes=43666;spawns=0;maxrounds=0
# E3P-dropped-events sonnet 2 FAIL A:pass;B:root-cause-missing:;C:pass bytes_read=0;tiers=;mltok=0;sctok=0;mlshare=1.0;prebytes=0;spawns=0;maxrounds=0
# E3P-dropped-events sonnet 3 FAIL A:pass;B:root-cause-missing:;C:pass bytes_read=0;tiers=;mltok=0;sctok=0;mlshare=1.0;prebytes=0;spawns=0;maxrounds=0
# E5P-handler-rollout fable 2 FAIL A:main-loop-grind mltok=59061;sctok=0;mlshare=1.0;prebytes=12624;spawns=0;maxrounds=0
# E5P-handler-rollout sonnet 2 FAIL C:missing:8 mltok=3834;sctok=0;mlshare=1.0;prebytes=13803;spawns=0;maxrounds=0
# E5P-handler-rollout sonnet 3 FAIL C:missing:12 mltok=0;sctok=0;mlshare=1.0;prebytes=0;spawns=0;maxrounds=0