os-orchestration WS1 session audit (IRL, 10 stratified sessions)

Not a lab eval — an IRL audit of real session transcripts (the post-rollout feedback-loop rung from eval-methodology-irl-feedback-loop). Full findings: cc-os/docs/orchestration-audit/2026-07-06-findings.md (commit 7965f03).

Results Grid/Threshold

Passing criterion: per-question qualitative PASS across the sample; failures grouped into verified clusters ranked by severity. Every load-bearing auditor claim was re-verified against the primary transcript before being counted.

#	Rubric question (the hypotheses evaluated)	Verdict
1	Are subagents called when they should be?	Mostly PASS (8/10; S3/S7 gray-zone heavy in-session investigation)
2	Correct model tier per subagent?	FAIL — the dominant failure area (clusters 1 & 2)
3	Planning/grouping for context efficiency?	PASS (concurrent fan-out in 6/10; minor reactive late spawns)
4	Avoiding unnecessary orchestrator reads?	Mostly PASS (S4: 0 bytes over 41 spawns; exceptions S7, S9)
5	Avoiding over-sharing context with subagents?	PASS everywhere (prompts 0.4–6.7KB, no dumps)
6	Following ORCHESTRATION.md?	Split: post-rollout cc-os 23/23 on explicit-model; pre-rollout/ops omit on 16/56 spawns
7	Receiving only needed context back?	PASS everywhere (0.9–13KB summaries, no full-dump returns)

Summary: Delegation judgment and context hygiene are healthy; both HIGH failures are about model-tier resolution, not delegation decisions. Sample: 10 sessions stratified pre/post plugin rollout across 5 projects (cc-os sessions contaminated — they were building orchestration tooling).

Verified failure clusters

HIGH (environment): all 23 spawns in recent sessions resolved to haiku regardless of the requested tier — including explicit judgment-grade sonnet/opus requests. Root cause found post-audit: CLAUDE_CODE_SUBAGENT_MODEL=haiku in ~/.claude/settings.json's env block, set by an earlier AI session as a "lowest cost first" measure. The var force-overrides the model param on every Agent spawn, silently. The audit's initial "Fable-5 harness bug" attribution was wrong — coincidental timing (var added between the pre-rollout opus/sonnet sessions and the Fable-5 sessions). Removed 2026-07-06. The exposed policy gap survives the fix: nothing tells the orchestrator to verify resolvedModel, so 23 downgrades went unnoticed.
HIGH (orchestrator): omitting model inherits the main-loop model — a mechanical file-edit ran at sonnet, a character-counting task at opus. Misses cluster pre-rollout and in ops projects; post-rollout cc-os had zero omissions. Now unmasked by the env-var removal: the clamp-to-haiku was hiding the cost of omissions.
MEDIUM: orchestrator self-investigates (up to 74KB of reads) then delegates the same ground; the "short orienting Read" boundary is undefined.
LOW: reactive late spawns instead of pre-planned batches (wall-clock only).

Validity and Limitations

How to interpret: the plugin wording works where present and salient (23/23 explicit-model compliance post-rollout in cc-os); misses cluster where the text wasn't in force. But only 2 truly post-rollout non-cc-os sessions (one violated) — the target population is thin in this sample. Single audit pass; several raw auditor claims did not survive verification (only the verified clusters should be cited). The missed-delegation heuristic (≥4 same-tool runs) produced only false positives — needs retuning before it can score an eval.

No rerun needed: the audit measured real behavior and the measurements stand; only the Cluster 1 attribution changed. The E1–E4 backlog items are eval scenarios, not built evals — nothing exists to rerun.

Orchestrator economics (clarifications worth keeping)

The intuition "the expensive orchestrator should delegate all grunt work to cheap workers" is the right frame for the wrong unit of work:

The expensive part of a frontier orchestrator isn't Edit calls — it's context. A direct edit to an in-context file is a few hundred output tokens. Delegating the same edit costs a detailed prompt (orchestrator output) + the subagent re-reading the file and surrounding context in a fresh window + a result the orchestrator must read and verify. For sequential, coordinated edits the handoff exceeds the edit.
Delegation pays when it absorbs context the orchestrator doesn't want to load: parallelizable independent files, many-file sweeps, large isolated reads (log review, wide grep-and-synthesize). WS1 itself used this correctly — 10 parallel sonnet auditors each digested a full transcript the orchestrator never had to hold.
Mental model: keep work you already hold the context for; delegate work whose context you don't want to pay for. Sequential CRUD on an in-context file is the first kind — the audit's Q1 confirmed this (both "missed delegation" flags on in-session sequential work were false positives).

Deployment and Evolution

Good-Enough Gate: plugin stays deployed as-is; the env-var removal fixes Cluster 1's cause. Cluster 2 is the live risk now (omission = expensive inheritance under a Fable main loop).

Hardening path / next measurement:

ORCHESTRATION.md wording fixes: (a) verify-resolvedModel rule (say so and adapt on mismatch for judgment-critical work); (b) mechanical trigger phrasing for the explicit-model rule ("before every Agent call → include model:" — the Eval B lesson from os-adr-eval-b-wording-experiment-hypotheses); (c) define the orienting-read budget.
Build E1–E3 scripted evals from the verified misses (per eval-methodology-ladder: paired positives/negatives, frozen reserve). E1 = detect the downgrade, E2 = explicit model per spawn, E3 = pre-spawn read budget. E4 deferred.
Re-audit IRL in ~2 weeks (post env-var removal, non-cc-os sessions) — that population was thin this pass.

eval-methodology-irl-feedback-loop — this audit is that loop's first iteration for os-orchestration
eval-methodology-ladder — design gate for the E1–E3 evals
os-adr-eval-b-wording-experiment-hypotheses — the trigger-phrasing lessons the wording fixes will reuse

7.0 KiB Raw Blame History Unescape Escape