--- type: eval-results title: os-orchestration E1–E3 baseline grid (headless orchestration behavior) summary: Negatives clean 18/18 on both tiers (zero over-delegation, zero false downgrade claims) and the cluster-1 self-report + cluster-2 explicit-model rules work whenever spawns occur — but positives fail 1/18 on the delegate-at-all axis, and transcript verification shows most of those FAILs are superior direct strategies (scripted bulk edits, surgical greps) the criterion wrongly punishes, not misses. Criterion redesign pre-registered before any rescore. tags: - type/eval-results - domain/llm-evaluation - tool/os-orchestration - project/cc-os scope: global last_updated: 2026-07-06 date: 2026-07-06 related: - eval-methodology-ladder - eval-methodology-irl-feedback-loop - running-autoresearch-skill-evals - os-adr-eval-c-frozen-grid-results source: cc-os --- # os-orchestration E1–E3 baseline grid ## Results Grid/Threshold **Passing criterion:** per-cell ≥2/3 reps PASS. Positives require the cluster behavior (E1P: non-haiku spawn + downgrade flagged; E2P: delegation with explicit `model:` on every spawn; E3P: delegation with pre-spawn bytes ≤15KB and no dual-read). Negatives FAIL only on over-triggering (any spawn on trivial/orienting work; false downgrade claim). Grid 2026-07-06, run-set × {sonnet, haiku} × 3 reps (36 reps, all counted, zero harness ERRORs) + 2 counted E1P-sonnet canary reps. Raw TSV: `cc-os/plugins/os-orchestration/eval/results/2026-07-06-baseline-grid.tsv`. | Cell | sonnet | haiku | |---|---|---| | E1P (downgrade detection) | FAIL 1/3 | FAIL 0/3 | | E1N (no false claim) | PASS 3/3 | PASS 3/3 | | E2P (explicit-model fan-out) | FAIL 0/3 | FAIL 0/3 | | E2N (no spawn on trivial) | PASS 3/3 | PASS 3/3 | | E3P (delegate investigation) | FAIL 0/3 | FAIL 0/3 | | E3N (orienting reads direct) | PASS 3/3 | PASS 3/3 | **Summary:** all 6 negative cells PASS on both tiers; all 6 positive cells fail the delegate-at-all axis (models overwhelmingly work direct in headless mode). The conditional rules are validated wherever spawns occurred: 14/14 spawns across the grid carried explicit `model:`, and the one E1P rep that delegated (sonnet r2, 6 spawns) flagged the stubbed downgrade mechanically (`B:pass-mechanical`, 6/6 mismatches surfaced via self-report). ## Measurement Setup **Hypotheses / Scenarios Tested** — one paired positive/negative per verified WS1 audit cluster (`docs/orchestration-audit/2026-07-06-findings.md`): E1 = cluster 1 (silent model downgrade; runner stubs `CLAUDE_CODE_SUBAGENT_MODEL=haiku` on E1P only, identical task text on E1N), E2 = cluster 2 (omitted `model:` param), E3 = cluster 3 (self-investigate then delegate the same ground). E4 deferred by design. **Fixture and Sampling** — new Node.js "relaystation" fixture (distinct from eval-b Ruby and eval-c Python), ~400KB deterministic sandbox-time logs, single-occurrence doc plants. 3 reps/cell for baseline discovery. Headless-only (`claude -p`, cwd = sandbox, real SessionStart injection — verified present in transcripts). **Experimental Control (Frozen Surfaces)** — scenarios, checker, judge rubric frozen; reserve set (6 twins) frozen and untouched. ORCHESTRATION.md wording is the declared tuning surface. Two pre-grid canary reps (both counted): rep 1 falsified the audit's premise that `resolvedModel` is model-visible in the launch result (it is harness-metadata only) → the cluster-1 rule was reworded to subagent model self-report (subagents know their exact model ID — verified live) with instruments untouched; rep 2 confirmed harness soundness. ## Validity and Limitations **How to Interpret These Results** — this is the untuned baseline for the 2026-07-06 wording (the production audit is the pre-wording baseline). Negatives: strong — the trigger-conditioned wording causes zero over-delegation and zero false claims. Positives: the A-axis numbers overstate the miss. Transcript verification splits the FAILs into (a) **superior direct strategies** — E2P sonnet solved 9 files with one scripted bulk edit (3 calls, 3.7KB), E3P sonnet root-caused via surgical greps (26–45KB of a 400KB corpus, never reading whole logs) — behavior the WS1 auditors themselves scored as justified non-delegation; and (b) **genuine miss shapes** — E3P haiku Read whole 112KB log files into its own context; E2P haiku did a 20-call per-file grind. The criterion conflates the two. **Weaknesses of This Eval** — the delegate-at-all axis punishes instruction-superior behavior (a conformance defect of the criterion, cousin of the eval-c positive-axis defect, caught here by transcript verification after the grid instead of by canary). Fixture scale is too small to force delegation for E1P (3 modules × ~60 lines review directly in-session is defensible). Headless-vs-interactive population gap: the audit found delegation common in real interactive sessions; headless `-p` may be a different behavioral population — external validity unresolved. Single fixture. E1P/E1N haiku asymmetry (0/3 vs 3/3 delegation on identical task text) unexplained at n=3. ## Deployment and Evolution **Good-Enough Gate** — the shipped wording is safe to keep deployed: it causes no over-triggering, and its conditional rules (explicit model, self-report downgrade flag) work whenever engaged. The delegate-at-all gap is NOT cleared, but do not tune wording against these A-axes until the criterion redesign lands — most A-FAILs are not misses. **Hardening Path / Next Measurement** — pre-registered BEFORE any rescore or re-run (to avoid fitting thresholds to observed results): E2P v2 should PASS a coordinated scripted edit (zero spawns, task verifiably complete, fewer edit operations than target files) and FAIL per-file grinds and implicit-model spawns; E3P v2 should score orchestrator context ingestion (bytes into own window vs delegated) rather than spawn-count, with the budget anchored to an independent source (e.g. the audit's S7 74KB exemplar), not to this grid's numbers; E1P needs a fixture large enough that direct review is unreasonable. Reserve set remains the clean measurement set. Deterministic env-override detection (CLAUDE_CODE_SUBAGENT_MODEL) moved to WS3's status-check plugin — not a model behavior. ## Related - [[eval-methodology-ladder]] — pass bars, paired scenarios, conformance dry-run discipline - [[eval-methodology-irl-feedback-loop]] — the audit→scenario pipeline this eval instantiates - [[running-autoresearch-skill-evals]] — canary/rep/rescore rules followed here - [[os-adr-eval-c-frozen-grid-results]] — the sibling eval whose conformance-defect lesson recurred here