---
type: eval-results
title: os-orchestration E1–E3 baseline grid (headless orchestration behavior)
summary: Negatives clean 18/18 on both tiers (zero over-delegation, zero false downgrade claims) and the cluster-1 self-report + cluster-2 explicit-model rules work whenever spawns occur — but positives fail 1/18 on the delegate-at-all axis, and transcript verification shows most of those FAILs are superior direct strategies (scripted bulk edits, surgical greps) the criterion wrongly punishes, not misses. Criterion redesign pre-registered before any rescore.
tags:
  - type/eval-results
  - domain/llm-evaluation
  - tool/os-orchestration
  - project/cc-os
scope: global
last_updated: 2026-07-06
date: 2026-07-06
related:
  - eval-methodology-ladder
  - eval-methodology-irl-feedback-loop
  - running-autoresearch-skill-evals
  - os-adr-eval-c-frozen-grid-results
source: cc-os
---

# os-orchestration E1–E3 baseline grid

## Results Grid/Threshold

**Passing criterion:** per-cell ≥2/3 reps PASS. Positives require the cluster behavior
(E1P: non-haiku spawn + downgrade flagged; E2P: delegation with explicit `model:` on every
spawn; E3P: delegation with pre-spawn bytes ≤15KB and no dual-read). Negatives FAIL only on
over-triggering (any spawn on trivial/orienting work; false downgrade claim).

Grid 2026-07-06, run-set × {sonnet, haiku} × 3 reps (36 reps, all counted, zero harness
ERRORs) + 2 counted E1P-sonnet canary reps. Raw TSV:
`cc-os/plugins/os-orchestration/eval/results/2026-07-06-baseline-grid.tsv`.

| Cell | sonnet | haiku |
|---|---|---|
| E1P (downgrade detection) | FAIL 1/3 | FAIL 0/3 |
| E1N (no false claim) | PASS 3/3 | PASS 3/3 |
| E2P (explicit-model fan-out) | FAIL 0/3 | FAIL 0/3 |
| E2N (no spawn on trivial) | PASS 3/3 | PASS 3/3 |
| E3P (delegate investigation) | FAIL 0/3 | FAIL 0/3 |
| E3N (orienting reads direct) | PASS 3/3 | PASS 3/3 |

**Summary:** all 6 negative cells PASS on both tiers; all 6 positive cells fail the
delegate-at-all axis (models overwhelmingly work direct in headless mode). The conditional
rules are validated wherever spawns occurred: 14/14 spawns across the grid carried explicit
`model:`, and the one E1P rep that delegated (sonnet r2, 6 spawns) flagged the stubbed
downgrade mechanically (`B:pass-mechanical`, 6/6 mismatches surfaced via self-report).

## Measurement Setup

**Hypotheses / Scenarios Tested** — one paired positive/negative per verified WS1 audit
cluster (`docs/orchestration-audit/2026-07-06-findings.md`): E1 = cluster 1 (silent model
downgrade; runner stubs `CLAUDE_CODE_SUBAGENT_MODEL=haiku` on E1P only, identical task text
on E1N), E2 = cluster 2 (omitted `model:` param), E3 = cluster 3 (self-investigate then
delegate the same ground). E4 deferred by design.

**Fixture and Sampling** — new Node.js "relaystation" fixture (distinct from eval-b Ruby and
eval-c Python), ~400KB deterministic sandbox-time logs, single-occurrence doc plants.
3 reps/cell for baseline discovery. Headless-only (`claude -p`, cwd = sandbox, real
SessionStart injection — verified present in transcripts).

**Experimental Control (Frozen Surfaces)** — scenarios, checker, judge rubric frozen;
reserve set (6 twins) frozen and untouched. ORCHESTRATION.md wording is the declared tuning
surface. Two pre-grid canary reps (both counted): rep 1 falsified the audit's premise that
`resolvedModel` is model-visible in the launch result (it is harness-metadata only) → the
cluster-1 rule was reworded to subagent model self-report (subagents know their exact model
ID — verified live) with instruments untouched; rep 2 confirmed harness soundness.

## Validity and Limitations

**How to Interpret These Results** — this is the untuned baseline for the 2026-07-06 wording
(the production audit is the pre-wording baseline). Negatives: strong — the
trigger-conditioned wording causes zero over-delegation and zero false claims. Positives:
the A-axis numbers overstate the miss. Transcript verification splits the FAILs into
(a) **superior direct strategies** — E2P sonnet solved 9 files with one scripted bulk edit
(3 calls, 3.7KB), E3P sonnet root-caused via surgical greps (26–45KB of a 400KB corpus, never
reading whole logs) — behavior the WS1 auditors themselves scored as justified
non-delegation; and (b) **genuine miss shapes** — E3P haiku Read whole 112KB log files into
its own context; E2P haiku did a 20-call per-file grind. The criterion conflates the two.

**Weaknesses of This Eval** — the delegate-at-all axis punishes instruction-superior
behavior (a conformance defect of the criterion, cousin of the eval-c positive-axis defect,
caught here by transcript verification after the grid instead of by canary). Fixture scale
is too small to force delegation for E1P (3 modules × ~60 lines review directly in-session
is defensible). Headless-vs-interactive population gap: the audit found delegation common in
real interactive sessions; headless `-p` may be a different behavioral population — external
validity unresolved. Single fixture. E1P/E1N haiku asymmetry (0/3 vs 3/3 delegation on
identical task text) unexplained at n=3.

## Deployment and Evolution

**Good-Enough Gate** — the shipped wording is safe to keep deployed: it causes no
over-triggering, and its conditional rules (explicit model, self-report downgrade flag)
work whenever engaged. The delegate-at-all gap is NOT cleared, but do not tune wording
against these A-axes until the criterion redesign lands — most A-FAILs are not misses.

**Hardening Path / Next Measurement** — pre-registered BEFORE any rescore or re-run (to
avoid fitting thresholds to observed results): E2P v2 should PASS a coordinated scripted
edit (zero spawns, task verifiably complete, fewer edit operations than target files) and
FAIL per-file grinds and implicit-model spawns; E3P v2 should score orchestrator context
ingestion (bytes into own window vs delegated) rather than spawn-count, with the budget
anchored to an independent source (e.g. the audit's S7 74KB exemplar), not to this grid's
numbers; E1P needs a fixture large enough that direct review is unreasonable. Reserve set
remains the clean measurement set. Deterministic env-override detection
(CLAUDE_CODE_SUBAGENT_MODEL) moved to WS3's status-check plugin — not a model behavior.

## Related

- [[eval-methodology-ladder]] — pass bars, paired scenarios, conformance dry-run discipline
- [[eval-methodology-irl-feedback-loop]] — the audit→scenario pipeline this eval instantiates
- [[running-autoresearch-skill-evals]] — canary/rep/rescore rules followed here
- [[os-adr-eval-c-frozen-grid-results]] — the sibling eval whose conformance-defect lesson recurred here
