# os-orchestration session audit — findings _Audit date: 2026-07-06. Status: complete (WS1 of `docs/plans/ws1-orchestration-audit.md`)._ _Method: Phase 1 extractor (`plugins/os-orchestration/audit/bin/extract`) → 10 stratified sessions → 10 parallel sonnet auditors (7-question rubric) → orchestrator-verified synthesis. Every load-bearing claim below was re-verified against the primary transcript; auditor claims that failed or only partially survived verification are marked._ ## Sample | ID | Project | Date | Population | Turns | Spawns | Main-loop model | |----|---------|------|------------|-------|--------|-----------------| | S1 | cc-os | 2026-07-06 | post-rollout (contaminated¹) | 66 | 10 | fable-5 | | S2 | cc-os | 2026-07-06 | post-rollout (contaminated¹) | 207 | 8 | fable-5 | | S3 | cc-os | 2026-07-04 | post-rollout (contaminated¹) | 255 | 5 | fable-5 | | S4 | llf-schema | 2026-06-30 | pre-rollout | 191 | 41 | opus-4-8 | | S5 | llf-schema | 2026-06-29 | pre-rollout | 36 | 8 | sonnet-4-6 | | S6 | philly-sem | 2026-06-29 | pre-rollout | 21 | 1 | sonnet-4-6 | | S7 | servers | 2026-07-04 | post-rollout | 144 | 3 | sonnet-5 | | S8 | servers | 2026-07-04 | post-rollout | 90 | 0 | sonnet-5 | | S9 | systems-admin | 2026-06-30 | pre-rollout | 117 | 3 | opus-4-8 | | S10 | systems-admin | 2026-07-01 | pre-rollout | 53 | 0 | sonnet-5 | ¹ cc-os sessions were actively building eval/orchestration tooling; delegation-awareness in them overstates the general population. Transcripts: `~/.claude/projects//.jsonl`; fact-sheets and full per-session auditor reports were session artifacts (scratchpad); regenerate fact-sheets any time with the extractor. ## Per-question summary (across 10 sessions) 1. **Subagents called when they should be?** Mostly yes. 8/10 PASS or justified-no-delegation (S8, S10 correctly kept sequential/coordinated work in-session — both heuristic missed-delegation flags there were judged false positives). Genuine gray zone: S3 and S7 did heavy in-session investigation (19–37 calls, 74KB) that could have been one "investigate and report" delegation. 2. **Correct model per subagent?** The dominant failure area, in two distinct modes (see clusters 1 and 2). Explicit-tier choices, when the orchestrator made them, were sensible (haiku for mechanical, sonnet for judgment). 3. **Planning/grouping for context efficiency?** Good. Concurrent fan-out used in S1, S2, S4, S5, S7, S9; serial dependencies respected. Minor: reactive late spawns (S1 agents 8–10) instead of pre-planned batches. 4. **Avoiding unnecessary orchestrator reads?** Mostly PASS (S4: zero orchestrator bytes over 41 spawns; S1, S5, S6 near-zero). Exceptions: S7 (74KB pre-spawn diagnostics, then Explore agents told to re-read files "already read" — partially verified²), S9 (read `settings.json`, then spawn #1 instructed to read it again — confirmed). 5. **Over-sharing context with subagents?** PASS everywhere. Prompts 0.4–6.7KB, instruction-forward; no transcript/CLAUDE.md dumps observed in any session. 6. **Following ORCHESTRATION.md?** Split by population: post-rollout cc-os sessions pass the explicit-model rule 23/23 spawns; pre-rollout and ops sessions omit `model` on 16/56 spawns (S4×9, S5×6, S6×1, S7×3, S9×3 — counting per verified transcript data, 22 omissions total including Explore-type spawns). Everything else (delegation thresholds, orienting reads) broadly followed. 7. **Receiving only needed context back?** PASS everywhere. Agent results 0.9–13KB summaries; no full-dump pattern found. ² Confirmed for one file (`incidents.md`, S7 line 182 → spawn-2 prompt) and for the prompt text "CLAUDE.md (already read, but note key conventions)"; the auditor's other named files could not be confirmed as orchestrator Reads (may have been Bash cats) — treated as plausible. ## Failure-mode clusters (verified) ### Cluster 1 — model param silently overridden to haiku in Fable-5 sessions (HIGH, environment) Verified across S1/S2/S3 (23 spawns): requested `sonnet` (18), `opus` (1), `haiku` (4) — every one resolved `claude-haiku-4-5-20251001`. In S4/S5 (opus/sonnet main loops) explicit params were honored (sonnet→sonnet-4-6, haiku→haiku-4-5). So under a Fable-5 main loop the `model` param is currently a no-op and all subagent work runs at haiku tier, including tasks the orchestrator explicitly graded as judgment work (e.g. S3 line 603: "answer from judgment alone" → haiku). The orchestrator never detected or escalated the downgrade — nothing in the policy tells it to check `resolvedModel`. - **Not an orchestrator wording failure** — compliance with the written rule was perfect in these sessions. It is an ops/environment issue plus a policy gap (no verify-resolution rule). - **ROOT CAUSE FOUND (2026-07-06, post-audit):** `CLAUDE_CODE_SUBAGENT_MODEL=haiku` was set in `~/.claude/settings.json`'s `env` block (added by an earlier AI session as a cost measure). That env var force-overrides every subagent's `model` param. The "Fable-5 main loop" correlation was coincidental timing — the var was added after the pre-rollout opus/sonnet sessions (S4/S5, 06-29/30, params honored) and before the Fable-5 sessions. **Removed 2026-07-06.** E1 remains valid as a detect-the-downgrade eval; the policy gap (no verify-`resolvedModel` rule) still stands regardless of cause. ### Cluster 2 — omitted `model` param inherits the main-loop model (HIGH, orchestrator) Verified: omission resolves to the parent session's model — opus-4-8 in S4 (×7) and S9 (×2), sonnet in S5 (×6) and S6 (×1) — except agent types with their own default (Explore→haiku, S7 ×3, S4 ×2, S9 ×1). Cost consequence is real: S6's single spawn was a mechanical file-edit task billed at sonnet; S9 ran a character-counting task at opus. All these sessions are pre-rollout or ops sessions where the global plugin text either wasn't present or wasn't followed on this rule. Post-rollout cc-os sessions (S1–S3): zero omissions. - The rule exists and works where the plugin text is in force and salient; the misses cluster in the population that ran without it (pre-07-03) or in ops projects (S7 07-04, post-rollout, still omitted on all 3 spawns — the one confirmed post-rollout violation). - Action candidates: mechanical trigger-phrased wording (the Eval B lesson — "before every Agent call → include `model:`"); eval scenario E2. ### Cluster 3 — orchestrator self-investigates then delegates the same ground (MEDIUM) S7: 32 Bash + 4 Reads (74KB) of diagnostics before spawning 3 Explore agents whose prompts overlap that ground (one confirmed file-level dual-read; prompt text acknowledges "already read"). S9: read `settings.json` then instructed spawn #1 to read it (confirmed). S3: 19-call investigate-then-fix run kept in-session where one delegation could have isolated it. - Boundary is genuinely fuzzy — the policy allows "a short orienting Read"; these exceed "short" but the work partly informed direct action too. Judged real but medium-severity. - Action candidate: eval scenario E3 (bytes-read-before-first-spawn budget). ### Cluster 4 — reactive late spawns instead of planned batches (LOW) S1 (agents 8–10 spawned only after synthesis of 4–7), S3 (91 post-eval tool calls that could have been folded into the eval spawns' own prompts). Low priority: wall-clock cost only, no correctness or spend impact demonstrated. ### Non-findings worth keeping - Zero cases of context over-sharing to subagents; zero cases of full-dump returns (Q5/Q7 clean across all 10 — the async task-notification pattern returns summaries by design). - The missed-delegation heuristic (≥4 same-tool runs) produced 4 flags in the sample; auditors + verification judged all of them false positives (coordinated multi-file units or sequential-dependent work). Threshold/shape needs tuning before it can score an eval. ## Eval-scenario backlog (inputs to a future eval design — NOT an eval yet) - **E1 (from cluster 1):** Scripted task forces a judgment-grade delegation; harness stubs resolution to haiku. Criterion: orchestrator notices `resolvedModel` mismatch in the launch stub and says so / adapts. Checker: extractor already captures requested vs resolved. - **E2 (from cluster 2):** Multi-file parallelizable task in a non-cc-os fixture. Criterion: every Agent spawn includes `model:`; mechanical work→haiku, judgment→sonnet. Checker: extractor `model_explicit` field — fully mechanical. Paired negative: single-file ≤2-call task where spawning anything is the failure. - **E3 (from cluster 3):** Task whose natural first move is delegatable investigation (log review across many files). Criterion: orchestrator bytes-read before first spawn under a threshold (extractor segment `pre-spawn-1`), no file both Read by orchestrator and assigned to a subagent prompt. Paired negative: task where orienting reads are correct (uncertain target path). - **E4 (from cluster 4, deferred):** Batch-planning quality — hard to score mechanically; design only if clusters 1–3 get fixed and this remains visible. Per the plan's eval-design note: these run as scripted multi-file headless tasks scored by the Phase 1 extractor + thresholds; mid-session behavior, so the Eval B SessionStart pattern does not transfer. Design gate: follow `~/Documents/SecondBrain/eval-methodology-ladder.md` (paired positives/negatives, frozen reserve) when authoring. ## Caveats - Sample: 10 sessions, single audit pass; cc-os sessions contaminated (delegation-aware work); only 2 truly post-rollout non-cc-os sessions (S7, S8) — the population the plugin targets is thin in this sample. Re-audit in ~2 weeks per the IRL-feedback-loop cadence. - Auditor reliability: several auditor claims did not survive verification as stated (S4 "defaults to opus" was right for general-purpose but auditors variously mis-attributed the mechanism; S7's dual-read file list only partially confirmed). All cluster claims above reflect the verified transcript data, not the raw auditor reports.