SecondBrain/os-orchestration-ws1-session-audit-results.md

---
type: eval-results
title: "os-orchestration WS1 session audit (IRL, 10 stratified sessions)"
summary: "Audit of 10 real sessions against the 7-question orchestration rubric: delegation decisions, context hygiene, and grouping are healthy; the two HIGH failures were model-tier resolution — a user-set CLAUDE_CODE_SUBAGENT_MODEL=haiku env var silently overriding every spawn (root-caused post-audit, removed), and omitted model params inheriting the expensive main-loop model."
tags:
  - type/eval-results
  - domain/llm-evaluation
  - domain/orchestration
  - tool/os-orchestration
  - tool/claude-code
  - project/cc-os
scope: global
last_updated: 2026-07-06
date: 2026-07-06
related:
  - eval-methodology-irl-feedback-loop
  - eval-methodology-ladder
source: cc-os
---

# os-orchestration WS1 session audit (IRL, 10 stratified sessions)

Not a lab eval — an IRL audit of real session transcripts (the post-rollout feedback-loop rung
from [[eval-methodology-irl-feedback-loop]]). Full findings: `cc-os/docs/orchestration-audit/2026-07-06-findings.md` (commit 7965f03).

## Results Grid/Threshold

**Passing criterion:** per-question qualitative PASS across the sample; failures grouped into
verified clusters ranked by severity. Every load-bearing auditor claim was re-verified against
the primary transcript before being counted.

| # | Rubric question (the hypotheses evaluated) | Verdict |
|---|---|---|
| 1 | Are subagents called when they should be? | Mostly PASS (8/10; S3/S7 gray-zone heavy in-session investigation) |
| 2 | Correct model tier per subagent? | **FAIL — the dominant failure area** (clusters 1 & 2) |
| 3 | Planning/grouping for context efficiency? | PASS (concurrent fan-out in 6/10; minor reactive late spawns) |
| 4 | Avoiding unnecessary orchestrator reads? | Mostly PASS (S4: 0 bytes over 41 spawns; exceptions S7, S9) |
| 5 | Avoiding over-sharing context with subagents? | PASS everywhere (prompts 0.4–6.7KB, no dumps) |
| 6 | Following ORCHESTRATION.md? | Split: post-rollout cc-os 23/23 on explicit-model; pre-rollout/ops omit on 16/56 spawns |
| 7 | Receiving only needed context back? | PASS everywhere (0.9–13KB summaries, no full-dump returns) |

**Summary:** Delegation judgment and context hygiene are healthy; both HIGH failures are about
model-tier *resolution*, not delegation decisions. Sample: 10 sessions stratified pre/post plugin
rollout across 5 projects (cc-os sessions contaminated — they were building orchestration tooling).

## Verified failure clusters

1. **HIGH (environment):** all 23 spawns in recent sessions resolved to haiku regardless of the
   requested tier — including explicit judgment-grade sonnet/opus requests. **Root cause found
   post-audit:** `CLAUDE_CODE_SUBAGENT_MODEL=haiku` in `~/.claude/settings.json`'s `env` block,
   set by an earlier AI session as a "lowest cost first" measure. The var force-overrides the
   `model` param on every Agent spawn, silently. The audit's initial "Fable-5 harness bug"
   attribution was wrong — coincidental timing (var added between the pre-rollout opus/sonnet
   sessions and the Fable-5 sessions). **Removed 2026-07-06.** The exposed policy gap survives
   the fix: nothing tells the orchestrator to verify `resolvedModel`, so 23 downgrades went
   unnoticed.
2. **HIGH (orchestrator):** omitting `model` inherits the main-loop model — a mechanical
   file-edit ran at sonnet, a character-counting task at opus. Misses cluster pre-rollout and in
   ops projects; post-rollout cc-os had zero omissions. Now *unmasked* by the env-var removal:
   the clamp-to-haiku was hiding the cost of omissions.
3. **MEDIUM:** orchestrator self-investigates (up to 74KB of reads) then delegates the same
   ground; the "short orienting Read" boundary is undefined.
4. **LOW:** reactive late spawns instead of pre-planned batches (wall-clock only).

## Validity and Limitations

**How to interpret:** the plugin wording works where present and salient (23/23 explicit-model
compliance post-rollout in cc-os); misses cluster where the text wasn't in force. But only 2
truly post-rollout non-cc-os sessions (one violated) — the target population is thin in this
sample. Single audit pass; several raw auditor claims did not survive verification (only the
verified clusters should be cited). The missed-delegation heuristic (≥4 same-tool runs) produced
only false positives — needs retuning before it can score an eval.

**No rerun needed:** the audit measured real behavior and the measurements stand; only the
Cluster 1 *attribution* changed. The E1–E4 backlog items are eval *scenarios*, not built evals —
nothing exists to rerun.

## Orchestrator economics (clarifications worth keeping)

The intuition "the expensive orchestrator should delegate all grunt work to cheap workers" is
the right frame for the wrong unit of work:

- **The expensive part of a frontier orchestrator isn't Edit calls — it's context.** A direct
  edit to an in-context file is a few hundred output tokens. Delegating the same edit costs a
  detailed prompt (orchestrator output) + the subagent re-reading the file and surrounding
  context in a fresh window + a result the orchestrator must read and verify. For sequential,
  coordinated edits the handoff exceeds the edit.
- **Delegation pays when it absorbs context the orchestrator doesn't want to load:**
  parallelizable independent files, many-file sweeps, large isolated reads (log review,
  wide grep-and-synthesize). WS1 itself used this correctly — 10 parallel sonnet auditors each
  digested a full transcript the orchestrator never had to hold.
- Mental model: *keep work you already hold the context for; delegate work whose context you
  don't want to pay for.* Sequential CRUD on an in-context file is the first kind — the audit's
  Q1 confirmed this (both "missed delegation" flags on in-session sequential work were false
  positives).

## Deployment and Evolution

**Good-Enough Gate:** plugin stays deployed as-is; the env-var removal fixes Cluster 1's cause.
Cluster 2 is the live risk now (omission = expensive inheritance under a Fable main loop).

**Hardening path / next measurement:**
1. ORCHESTRATION.md wording fixes: (a) verify-`resolvedModel` rule (say so and adapt on
   mismatch for judgment-critical work); (b) mechanical trigger phrasing for the explicit-model
   rule ("before every Agent call → include `model:`" — the Eval B lesson from
   [[os-adr-eval-b-wording-experiment-hypotheses]]); (c) define the orienting-read budget.
2. Build E1–E3 scripted evals from the verified misses (per [[eval-methodology-ladder]]:
   paired positives/negatives, frozen reserve). E1 = detect the downgrade, E2 = explicit model
   per spawn, E3 = pre-spawn read budget. E4 deferred.
3. Re-audit IRL in ~2 weeks (post env-var removal, non-cc-os sessions) — that population was
   thin this pass.

## Related

- [[eval-methodology-irl-feedback-loop]] — this audit is that loop's first iteration for os-orchestration
- [[eval-methodology-ladder]] — design gate for the E1–E3 evals
- [[os-adr-eval-b-wording-experiment-hypotheses]] — the trigger-phrasing lessons the wording fixes will reuse