jared
/
cc-os
1
0
Fork 0
Code Issues 18 Pull Requests Packages Projects Releases Wiki Activity
cc-os/plugins/os-orchestration/eval/results/2026-07-06-baseline-grid.tsv

40 lines
2.2 KiB
Plaintext
Raw Normal View History

os-orchestration: baseline grid results + scripted-bulk-edit carve-out Grid (36 reps + 2 canaries, all counted): negatives 18/18 both tiers, zero over-triggering; positives 1/18 on delegate-at-all, but transcript verification splits those FAILs into superior direct strategies (scripted bulk edit, surgical greps - the audit's own justified-non-delegation shape) vs genuine misses (haiku whole-file ingestion, per-file grind). Conditional rules validated where engaged: 14/14 spawns explicit-model, self-report downgrade flag worked. E2P/E3P criterion redesign pre-registered with independently-anchored thresholds; reserve untouched. Analysis in vault: os-orchestration-eval-baseline-grid-results. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 22:35:40 +00:00
E1N-pipeline-review haiku 1 PASS B:pass-mechanical tiers=sonnet=3;spawns=3
E1N-pipeline-review haiku 2 PASS B:pass-mechanical tiers=sonnet=2;spawns=2
E1N-pipeline-review haiku 3 PASS B:pass-judge tiers=sonnet=3;spawns=3
E1N-pipeline-review sonnet 1 PASS B:pass-mechanical tiers=;spawns=0
E1N-pipeline-review sonnet 2 PASS B:pass-mechanical tiers=;spawns=0
E1N-pipeline-review sonnet 3 PASS B:pass-mechanical tiers=;spawns=0
E1P-pipeline-review haiku 1 FAIL A:no-delegation
E1P-pipeline-review haiku 2 FAIL A:no-delegation
E1P-pipeline-review haiku 3 FAIL A:no-delegation
E1P-pipeline-review sonnet 1 FAIL A:no-delegation
E1P-pipeline-review sonnet 2 PASS A:pass;B:pass-mechanical tiers=sonnet=6;mismatched=6
E1P-pipeline-review sonnet 3 FAIL A:no-delegation
E2N-readme-typo haiku 1 PASS A:no-spawn pre_bytes=1201
E2N-readme-typo haiku 2 PASS A:no-spawn pre_bytes=1201
E2N-readme-typo haiku 3 PASS A:no-spawn pre_bytes=1201
E2N-readme-typo sonnet 1 PASS A:no-spawn pre_bytes=483
E2N-readme-typo sonnet 2 PASS A:no-spawn pre_bytes=334
E2N-readme-typo sonnet 3 PASS A:no-spawn pre_bytes=398
E2P-retry-policy haiku 1 FAIL A:no-delegation
E2P-retry-policy haiku 2 FAIL A:no-delegation
E2P-retry-policy haiku 3 FAIL A:no-delegation
E2P-retry-policy sonnet 1 FAIL A:no-delegation
E2P-retry-policy sonnet 2 FAIL A:no-delegation
E2P-retry-policy sonnet 3 FAIL A:no-delegation
E3N-brand-spelling haiku 1 PASS A:no-spawn pre_bytes=2983;edited=yes
E3N-brand-spelling haiku 2 PASS A:no-spawn pre_bytes=2990;edited=yes
E3N-brand-spelling haiku 3 PASS A:no-spawn pre_bytes=2927;edited=yes
E3N-brand-spelling sonnet 1 PASS A:no-spawn pre_bytes=1967;edited=yes
E3N-brand-spelling sonnet 2 PASS A:no-spawn pre_bytes=790;edited=yes
E3N-brand-spelling sonnet 3 PASS A:no-spawn pre_bytes=1550;edited=yes
E3P-dropped-events haiku 1 FAIL A:no-delegation
E3P-dropped-events haiku 2 FAIL A:no-delegation
E3P-dropped-events haiku 3 FAIL A:no-delegation
E3P-dropped-events sonnet 1 FAIL A:no-delegation
E3P-dropped-events sonnet 2 FAIL A:no-delegation
E3P-dropped-events sonnet 3 FAIL A:no-delegation
# canary reps (counted; wording v1 then v2):
E1P-pipeline-review sonnet 1 FAIL A:pass;B:fail-mechanical tiers=sonnet=3;mismatched=3
E1P-pipeline-review sonnet 1 FAIL A:no-delegation