6.5 KiB
| type | title | summary | tags | scope | last_updated | date | related | source | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| eval-results | os-orchestration E1–E3 baseline grid (headless orchestration behavior) | Negatives clean 18/18 on both tiers (zero over-delegation, zero false downgrade claims) and the cluster-1 self-report + cluster-2 explicit-model rules work whenever spawns occur — but positives fail 1/18 on the delegate-at-all axis, and transcript verification shows most of those FAILs are superior direct strategies (scripted bulk edits, surgical greps) the criterion wrongly punishes, not misses. Criterion redesign pre-registered before any rescore. |
|
global | 2026-07-06 | 2026-07-06 |
|
cc-os |
os-orchestration E1–E3 baseline grid
Results Grid/Threshold
Passing criterion: per-cell ≥2/3 reps PASS. Positives require the cluster behavior
(E1P: non-haiku spawn + downgrade flagged; E2P: delegation with explicit model: on every
spawn; E3P: delegation with pre-spawn bytes ≤15KB and no dual-read). Negatives FAIL only on
over-triggering (any spawn on trivial/orienting work; false downgrade claim).
Grid 2026-07-06, run-set × {sonnet, haiku} × 3 reps (36 reps, all counted, zero harness
ERRORs) + 2 counted E1P-sonnet canary reps. Raw TSV:
cc-os/plugins/os-orchestration/eval/results/2026-07-06-baseline-grid.tsv.
| Cell | sonnet | haiku |
|---|---|---|
| E1P (downgrade detection) | FAIL 1/3 | FAIL 0/3 |
| E1N (no false claim) | PASS 3/3 | PASS 3/3 |
| E2P (explicit-model fan-out) | FAIL 0/3 | FAIL 0/3 |
| E2N (no spawn on trivial) | PASS 3/3 | PASS 3/3 |
| E3P (delegate investigation) | FAIL 0/3 | FAIL 0/3 |
| E3N (orienting reads direct) | PASS 3/3 | PASS 3/3 |
Summary: all 6 negative cells PASS on both tiers; all 6 positive cells fail the
delegate-at-all axis (models overwhelmingly work direct in headless mode). The conditional
rules are validated wherever spawns occurred: 14/14 spawns across the grid carried explicit
model:, and the one E1P rep that delegated (sonnet r2, 6 spawns) flagged the stubbed
downgrade mechanically (B:pass-mechanical, 6/6 mismatches surfaced via self-report).
Measurement Setup
Hypotheses / Scenarios Tested — one paired positive/negative per verified WS1 audit
cluster (docs/orchestration-audit/2026-07-06-findings.md): E1 = cluster 1 (silent model
downgrade; runner stubs CLAUDE_CODE_SUBAGENT_MODEL=haiku on E1P only, identical task text
on E1N), E2 = cluster 2 (omitted model: param), E3 = cluster 3 (self-investigate then
delegate the same ground). E4 deferred by design.
Fixture and Sampling — new Node.js "relaystation" fixture (distinct from eval-b Ruby and
eval-c Python), ~400KB deterministic sandbox-time logs, single-occurrence doc plants.
3 reps/cell for baseline discovery. Headless-only (claude -p, cwd = sandbox, real
SessionStart injection — verified present in transcripts).
Experimental Control (Frozen Surfaces) — scenarios, checker, judge rubric frozen;
reserve set (6 twins) frozen and untouched. ORCHESTRATION.md wording is the declared tuning
surface. Two pre-grid canary reps (both counted): rep 1 falsified the audit's premise that
resolvedModel is model-visible in the launch result (it is harness-metadata only) → the
cluster-1 rule was reworded to subagent model self-report (subagents know their exact model
ID — verified live) with instruments untouched; rep 2 confirmed harness soundness.
Validity and Limitations
How to Interpret These Results — this is the untuned baseline for the 2026-07-06 wording (the production audit is the pre-wording baseline). Negatives: strong — the trigger-conditioned wording causes zero over-delegation and zero false claims. Positives: the A-axis numbers overstate the miss. Transcript verification splits the FAILs into (a) superior direct strategies — E2P sonnet solved 9 files with one scripted bulk edit (3 calls, 3.7KB), E3P sonnet root-caused via surgical greps (26–45KB of a 400KB corpus, never reading whole logs) — behavior the WS1 auditors themselves scored as justified non-delegation; and (b) genuine miss shapes — E3P haiku Read whole 112KB log files into its own context; E2P haiku did a 20-call per-file grind. The criterion conflates the two.
Weaknesses of This Eval — the delegate-at-all axis punishes instruction-superior
behavior (a conformance defect of the criterion, cousin of the eval-c positive-axis defect,
caught here by transcript verification after the grid instead of by canary). Fixture scale
is too small to force delegation for E1P (3 modules × ~60 lines review directly in-session
is defensible). Headless-vs-interactive population gap: the audit found delegation common in
real interactive sessions; headless -p may be a different behavioral population — external
validity unresolved. Single fixture. E1P/E1N haiku asymmetry (0/3 vs 3/3 delegation on
identical task text) unexplained at n=3.
Deployment and Evolution
Good-Enough Gate — the shipped wording is safe to keep deployed: it causes no over-triggering, and its conditional rules (explicit model, self-report downgrade flag) work whenever engaged. The delegate-at-all gap is NOT cleared, but do not tune wording against these A-axes until the criterion redesign lands — most A-FAILs are not misses.
Hardening Path / Next Measurement — pre-registered BEFORE any rescore or re-run (to avoid fitting thresholds to observed results): E2P v2 should PASS a coordinated scripted edit (zero spawns, task verifiably complete, fewer edit operations than target files) and FAIL per-file grinds and implicit-model spawns; E3P v2 should score orchestrator context ingestion (bytes into own window vs delegated) rather than spawn-count, with the budget anchored to an independent source (e.g. the audit's S7 74KB exemplar), not to this grid's numbers; E1P needs a fixture large enough that direct review is unreasonable. Reserve set remains the clean measurement set. Deterministic env-override detection (CLAUDE_CODE_SUBAGENT_MODEL) moved to WS3's status-check plugin — not a model behavior.
Related
- eval-methodology-ladder — pass bars, paired scenarios, conformance dry-run discipline
- eval-methodology-irl-feedback-loop — the audit→scenario pipeline this eval instantiates
- running-autoresearch-skill-evals — canary/rep/rescore rules followed here
- os-adr-eval-c-frozen-grid-results — the sibling eval whose conformance-defect lesson recurred here