# os-orchestration session audit — findings

_Audit date: 2026-07-06. Status: complete (WS1 of `docs/plans/ws1-orchestration-audit.md`)._
_Method: Phase 1 extractor (`plugins/os-orchestration/audit/bin/extract`) → 10 stratified
sessions → 10 parallel sonnet auditors (7-question rubric) → orchestrator-verified synthesis.
Every load-bearing claim below was re-verified against the primary transcript; auditor claims
that failed or only partially survived verification are marked._

## Sample

| ID | Project | Date | Population | Turns | Spawns | Main-loop model |
|----|---------|------|------------|-------|--------|-----------------|
| S1 | cc-os | 2026-07-06 | post-rollout (contaminated¹) | 66 | 10 | fable-5 |
| S2 | cc-os | 2026-07-06 | post-rollout (contaminated¹) | 207 | 8 | fable-5 |
| S3 | cc-os | 2026-07-04 | post-rollout (contaminated¹) | 255 | 5 | fable-5 |
| S4 | llf-schema | 2026-06-30 | pre-rollout | 191 | 41 | opus-4-8 |
| S5 | llf-schema | 2026-06-29 | pre-rollout | 36 | 8 | sonnet-4-6 |
| S6 | philly-sem | 2026-06-29 | pre-rollout | 21 | 1 | sonnet-4-6 |
| S7 | servers | 2026-07-04 | post-rollout | 144 | 3 | sonnet-5 |
| S8 | servers | 2026-07-04 | post-rollout | 90 | 0 | sonnet-5 |
| S9 | systems-admin | 2026-06-30 | pre-rollout | 117 | 3 | opus-4-8 |
| S10 | systems-admin | 2026-07-01 | pre-rollout | 53 | 0 | sonnet-5 |

¹ cc-os sessions were actively building eval/orchestration tooling; delegation-awareness in
them overstates the general population.

Transcripts: `~/.claude/projects/<dir>/<uuid>.jsonl`; fact-sheets and full per-session auditor
reports were session artifacts (scratchpad); regenerate fact-sheets any time with the extractor.

## Per-question summary (across 10 sessions)

1. **Subagents called when they should be?** Mostly yes. 8/10 PASS or justified-no-delegation
   (S8, S10 correctly kept sequential/coordinated work in-session — both heuristic
   missed-delegation flags there were judged false positives). Genuine gray zone: S3 and S7
   did heavy in-session investigation (19–37 calls, 74KB) that could have been one
   "investigate and report" delegation.
2. **Correct model per subagent?** The dominant failure area, in two distinct modes (see
   clusters 1 and 2). Explicit-tier choices, when the orchestrator made them, were sensible
   (haiku for mechanical, sonnet for judgment).
3. **Planning/grouping for context efficiency?** Good. Concurrent fan-out used in S1, S2, S4,
   S5, S7, S9; serial dependencies respected. Minor: reactive late spawns (S1 agents 8–10)
   instead of pre-planned batches.
4. **Avoiding unnecessary orchestrator reads?** Mostly PASS (S4: zero orchestrator bytes over
   41 spawns; S1, S5, S6 near-zero). Exceptions: S7 (74KB pre-spawn diagnostics, then Explore
   agents told to re-read files "already read" — partially verified²), S9 (read
   `settings.json`, then spawn #1 instructed to read it again — confirmed).
5. **Over-sharing context with subagents?** PASS everywhere. Prompts 0.4–6.7KB,
   instruction-forward; no transcript/CLAUDE.md dumps observed in any session.
6. **Following ORCHESTRATION.md?** Split by population: post-rollout cc-os sessions pass the
   explicit-model rule 23/23 spawns; pre-rollout and ops sessions omit `model` on 16/56 spawns
   (S4×9, S5×6, S6×1, S7×3, S9×3 — counting per verified transcript data, 22 omissions total
   including Explore-type spawns). Everything else (delegation thresholds, orienting reads)
   broadly followed.
7. **Receiving only needed context back?** PASS everywhere. Agent results 0.9–13KB summaries;
   no full-dump pattern found.

² Confirmed for one file (`incidents.md`, S7 line 182 → spawn-2 prompt) and for the prompt
text "CLAUDE.md (already read, but note key conventions)"; the auditor's other named files
could not be confirmed as orchestrator Reads (may have been Bash cats) — treated as plausible.

## Failure-mode clusters (verified)

### Cluster 1 — model param silently overridden to haiku in Fable-5 sessions (HIGH, environment)

Verified across S1/S2/S3 (23 spawns): requested `sonnet` (18), `opus` (1), `haiku` (4) — every
one resolved `claude-haiku-4-5-20251001`. In S4/S5 (opus/sonnet main loops) explicit params
were honored (sonnet→sonnet-4-6, haiku→haiku-4-5). So under a Fable-5 main loop the `model`
param is currently a no-op and all subagent work runs at haiku tier, including tasks the
orchestrator explicitly graded as judgment work (e.g. S3 line 603: "answer from judgment
alone" → haiku). The orchestrator never detected or escalated the downgrade — nothing in the
policy tells it to check `resolvedModel`.

- **Not an orchestrator wording failure** — compliance with the written rule was perfect in
  these sessions. It is an ops/environment issue plus a policy gap (no verify-resolution rule).
- **ROOT CAUSE FOUND (2026-07-06, post-audit):** `CLAUDE_CODE_SUBAGENT_MODEL=haiku` was set in
  `~/.claude/settings.json`'s `env` block (added by an earlier AI session as a cost measure).
  That env var force-overrides every subagent's `model` param. The "Fable-5 main loop"
  correlation was coincidental timing — the var was added after the pre-rollout opus/sonnet
  sessions (S4/S5, 06-29/30, params honored) and before the Fable-5 sessions. **Removed
  2026-07-06.** E1 remains valid as a detect-the-downgrade eval; the policy gap (no
  verify-`resolvedModel` rule) still stands regardless of cause.
- **PREMISE CORRECTION (2026-07-06, E1 canary):** the audit's assumption that the orchestrator
  can "notice `resolvedModel` in the launch stub" is FALSE — `resolvedModel` exists only in
  harness-side `toolUseResult` metadata; the model-visible launch result carries no model
  info. Verified on the first live E1P canary rep (sonnet requested ×3, all resolved haiku,
  orchestrator had no observable signal). The only visible channel is subagent self-report
  (subagents know their exact model ID from their system prompt — verified live). The
  ORCHESTRATION.md cluster-1 rule was reworded the same day to the self-report mechanism;
  scenario/checker/rubric stayed frozen. Deterministic protection (warn when
  `CLAUDE_CODE_SUBAGENT_MODEL` is set) belongs in WS3's status-check plugin, not in model
  behavior.

### Cluster 2 — omitted `model` param inherits the main-loop model (HIGH, orchestrator)

Verified: omission resolves to the parent session's model — opus-4-8 in S4 (×7) and S9 (×2),
sonnet in S5 (×6) and S6 (×1) — except agent types with their own default (Explore→haiku,
S7 ×3, S4 ×2, S9 ×1). Cost consequence is real: S6's single spawn was a mechanical file-edit
task billed at sonnet; S9 ran a character-counting task at opus. All these sessions are
pre-rollout or ops sessions where the global plugin text either wasn't present or wasn't
followed on this rule. Post-rollout cc-os sessions (S1–S3): zero omissions.

- The rule exists and works where the plugin text is in force and salient; the misses cluster
  in the population that ran without it (pre-07-03) or in ops projects (S7 07-04, post-rollout,
  still omitted on all 3 spawns — the one confirmed post-rollout violation).
- Action candidates: mechanical trigger-phrased wording (the Eval B lesson — "before every
  Agent call → include `model:`"); eval scenario E2.

### Cluster 3 — orchestrator self-investigates then delegates the same ground (MEDIUM)

S7: 32 Bash + 4 Reads (74KB) of diagnostics before spawning 3 Explore agents whose prompts
overlap that ground (one confirmed file-level dual-read; prompt text acknowledges "already
read"). S9: read `settings.json` then instructed spawn #1 to read it (confirmed). S3: 19-call
investigate-then-fix run kept in-session where one delegation could have isolated it.

- Boundary is genuinely fuzzy — the policy allows "a short orienting Read"; these exceed
  "short" but the work partly informed direct action too. Judged real but medium-severity.
- Action candidate: eval scenario E3 (bytes-read-before-first-spawn budget).

### Cluster 4 — reactive late spawns instead of planned batches (LOW)

S1 (agents 8–10 spawned only after synthesis of 4–7), S3 (91 post-eval tool calls that could
have been folded into the eval spawns' own prompts). Low priority: wall-clock cost only,
no correctness or spend impact demonstrated.

### Non-findings worth keeping

- Zero cases of context over-sharing to subagents; zero cases of full-dump returns (Q5/Q7
  clean across all 10 — the async task-notification pattern returns summaries by design).
- The missed-delegation heuristic (≥4 same-tool runs) produced 4 flags in the sample;
  auditors + verification judged all of them false positives (coordinated multi-file units or
  sequential-dependent work). Threshold/shape needs tuning before it can score an eval.

## Eval-scenario backlog (inputs to a future eval design — NOT an eval yet)

- **E1 (from cluster 1):** Scripted task forces a judgment-grade delegation; harness stubs
  resolution to haiku. Criterion: orchestrator notices `resolvedModel` mismatch in the launch
  stub and says so / adapts. Checker: extractor already captures requested vs resolved.
- **E2 (from cluster 2):** Multi-file parallelizable task in a non-cc-os fixture. Criterion:
  every Agent spawn includes `model:`; mechanical work→haiku, judgment→sonnet. Checker:
  extractor `model_explicit` field — fully mechanical. Paired negative: single-file ≤2-call
  task where spawning anything is the failure.
- **E3 (from cluster 3):** Task whose natural first move is delegatable investigation (log
  review across many files). Criterion: orchestrator bytes-read before first spawn under a
  threshold (extractor segment `pre-spawn-1`), no file both Read by orchestrator and assigned
  to a subagent prompt. Paired negative: task where orienting reads are correct (uncertain
  target path).
- **E4 (from cluster 4, deferred):** Batch-planning quality — hard to score mechanically;
  design only if clusters 1–3 get fixed and this remains visible.

Per the plan's eval-design note: these run as scripted multi-file headless tasks scored by the
Phase 1 extractor + thresholds; mid-session behavior, so the Eval B SessionStart pattern does
not transfer. Design gate: follow `~/Documents/SecondBrain/eval-methodology-ladder.md`
(paired positives/negatives, frozen reserve) when authoring.

## Caveats

- Sample: 10 sessions, single audit pass; cc-os sessions contaminated (delegation-aware work);
  only 2 truly post-rollout non-cc-os sessions (S7, S8) — the population the plugin targets is
  thin in this sample. Re-audit in ~2 weeks per the IRL-feedback-loop cadence.
- Auditor reliability: several auditor claims did not survive verification as stated
  (S4 "defaults to opus" was right for general-purpose but auditors variously mis-attributed
  the mechanism; S7's dual-read file list only partially confirmed). All cluster claims above
  reflect the verified transcript data, not the raw auditor reports.
