113 lines
6.5 KiB
Markdown
113 lines
6.5 KiB
Markdown
---
|
||
type: eval-results
|
||
title: os-orchestration E1–E3 baseline grid (headless orchestration behavior)
|
||
summary: Negatives clean 18/18 on both tiers (zero over-delegation, zero false downgrade claims) and the cluster-1 self-report + cluster-2 explicit-model rules work whenever spawns occur — but positives fail 1/18 on the delegate-at-all axis, and transcript verification shows most of those FAILs are superior direct strategies (scripted bulk edits, surgical greps) the criterion wrongly punishes, not misses. Criterion redesign pre-registered before any rescore.
|
||
tags:
|
||
- type/eval-results
|
||
- domain/llm-evaluation
|
||
- tool/os-orchestration
|
||
- project/cc-os
|
||
scope: global
|
||
last_updated: 2026-07-06
|
||
date: 2026-07-06
|
||
related:
|
||
- eval-methodology-ladder
|
||
- eval-methodology-irl-feedback-loop
|
||
- running-autoresearch-skill-evals
|
||
- os-adr-eval-c-frozen-grid-results
|
||
source: cc-os
|
||
---
|
||
|
||
# os-orchestration E1–E3 baseline grid
|
||
|
||
## Results Grid/Threshold
|
||
|
||
**Passing criterion:** per-cell ≥2/3 reps PASS. Positives require the cluster behavior
|
||
(E1P: non-haiku spawn + downgrade flagged; E2P: delegation with explicit `model:` on every
|
||
spawn; E3P: delegation with pre-spawn bytes ≤15KB and no dual-read). Negatives FAIL only on
|
||
over-triggering (any spawn on trivial/orienting work; false downgrade claim).
|
||
|
||
Grid 2026-07-06, run-set × {sonnet, haiku} × 3 reps (36 reps, all counted, zero harness
|
||
ERRORs) + 2 counted E1P-sonnet canary reps. Raw TSV:
|
||
`cc-os/plugins/os-orchestration/eval/results/2026-07-06-baseline-grid.tsv`.
|
||
|
||
| Cell | sonnet | haiku |
|
||
|---|---|---|
|
||
| E1P (downgrade detection) | FAIL 1/3 | FAIL 0/3 |
|
||
| E1N (no false claim) | PASS 3/3 | PASS 3/3 |
|
||
| E2P (explicit-model fan-out) | FAIL 0/3 | FAIL 0/3 |
|
||
| E2N (no spawn on trivial) | PASS 3/3 | PASS 3/3 |
|
||
| E3P (delegate investigation) | FAIL 0/3 | FAIL 0/3 |
|
||
| E3N (orienting reads direct) | PASS 3/3 | PASS 3/3 |
|
||
|
||
**Summary:** all 6 negative cells PASS on both tiers; all 6 positive cells fail the
|
||
delegate-at-all axis (models overwhelmingly work direct in headless mode). The conditional
|
||
rules are validated wherever spawns occurred: 14/14 spawns across the grid carried explicit
|
||
`model:`, and the one E1P rep that delegated (sonnet r2, 6 spawns) flagged the stubbed
|
||
downgrade mechanically (`B:pass-mechanical`, 6/6 mismatches surfaced via self-report).
|
||
|
||
## Measurement Setup
|
||
|
||
**Hypotheses / Scenarios Tested** — one paired positive/negative per verified WS1 audit
|
||
cluster (`docs/orchestration-audit/2026-07-06-findings.md`): E1 = cluster 1 (silent model
|
||
downgrade; runner stubs `CLAUDE_CODE_SUBAGENT_MODEL=haiku` on E1P only, identical task text
|
||
on E1N), E2 = cluster 2 (omitted `model:` param), E3 = cluster 3 (self-investigate then
|
||
delegate the same ground). E4 deferred by design.
|
||
|
||
**Fixture and Sampling** — new Node.js "relaystation" fixture (distinct from eval-b Ruby and
|
||
eval-c Python), ~400KB deterministic sandbox-time logs, single-occurrence doc plants.
|
||
3 reps/cell for baseline discovery. Headless-only (`claude -p`, cwd = sandbox, real
|
||
SessionStart injection — verified present in transcripts).
|
||
|
||
**Experimental Control (Frozen Surfaces)** — scenarios, checker, judge rubric frozen;
|
||
reserve set (6 twins) frozen and untouched. ORCHESTRATION.md wording is the declared tuning
|
||
surface. Two pre-grid canary reps (both counted): rep 1 falsified the audit's premise that
|
||
`resolvedModel` is model-visible in the launch result (it is harness-metadata only) → the
|
||
cluster-1 rule was reworded to subagent model self-report (subagents know their exact model
|
||
ID — verified live) with instruments untouched; rep 2 confirmed harness soundness.
|
||
|
||
## Validity and Limitations
|
||
|
||
**How to Interpret These Results** — this is the untuned baseline for the 2026-07-06 wording
|
||
(the production audit is the pre-wording baseline). Negatives: strong — the
|
||
trigger-conditioned wording causes zero over-delegation and zero false claims. Positives:
|
||
the A-axis numbers overstate the miss. Transcript verification splits the FAILs into
|
||
(a) **superior direct strategies** — E2P sonnet solved 9 files with one scripted bulk edit
|
||
(3 calls, 3.7KB), E3P sonnet root-caused via surgical greps (26–45KB of a 400KB corpus, never
|
||
reading whole logs) — behavior the WS1 auditors themselves scored as justified
|
||
non-delegation; and (b) **genuine miss shapes** — E3P haiku Read whole 112KB log files into
|
||
its own context; E2P haiku did a 20-call per-file grind. The criterion conflates the two.
|
||
|
||
**Weaknesses of This Eval** — the delegate-at-all axis punishes instruction-superior
|
||
behavior (a conformance defect of the criterion, cousin of the eval-c positive-axis defect,
|
||
caught here by transcript verification after the grid instead of by canary). Fixture scale
|
||
is too small to force delegation for E1P (3 modules × ~60 lines review directly in-session
|
||
is defensible). Headless-vs-interactive population gap: the audit found delegation common in
|
||
real interactive sessions; headless `-p` may be a different behavioral population — external
|
||
validity unresolved. Single fixture. E1P/E1N haiku asymmetry (0/3 vs 3/3 delegation on
|
||
identical task text) unexplained at n=3.
|
||
|
||
## Deployment and Evolution
|
||
|
||
**Good-Enough Gate** — the shipped wording is safe to keep deployed: it causes no
|
||
over-triggering, and its conditional rules (explicit model, self-report downgrade flag)
|
||
work whenever engaged. The delegate-at-all gap is NOT cleared, but do not tune wording
|
||
against these A-axes until the criterion redesign lands — most A-FAILs are not misses.
|
||
|
||
**Hardening Path / Next Measurement** — pre-registered BEFORE any rescore or re-run (to
|
||
avoid fitting thresholds to observed results): E2P v2 should PASS a coordinated scripted
|
||
edit (zero spawns, task verifiably complete, fewer edit operations than target files) and
|
||
FAIL per-file grinds and implicit-model spawns; E3P v2 should score orchestrator context
|
||
ingestion (bytes into own window vs delegated) rather than spawn-count, with the budget
|
||
anchored to an independent source (e.g. the audit's S7 74KB exemplar), not to this grid's
|
||
numbers; E1P needs a fixture large enough that direct review is unreasonable. Reserve set
|
||
remains the clean measurement set. Deterministic env-override detection
|
||
(CLAUDE_CODE_SUBAGENT_MODEL) moved to WS3's status-check plugin — not a model behavior.
|
||
|
||
## Related
|
||
|
||
- [[eval-methodology-ladder]] — pass bars, paired scenarios, conformance dry-run discipline
|
||
- [[eval-methodology-irl-feedback-loop]] — the audit→scenario pipeline this eval instantiates
|
||
- [[running-autoresearch-skill-evals]] — canary/rep/rescore rules followed here
|
||
- [[os-adr-eval-c-frozen-grid-results]] — the sibling eval whose conformance-defect lesson recurred here
|