cc-os/docs/orchestration-audit/auditor-reports/S2-report.md

# Orchestration Audit Report: Session d68aab1b

**Session**: d68aab1b-3e3d-467c-9b9c-70b89e2576b0  
**Date**: 2026-07-06 14:44–17:58 UTC  
**Project**: `/home/jared/dev/cc-os`  
**Duration**: ~3 hours  
**Assistant**: claude-fable-5  
**Agent spawns**: 8  

---

## Question 1: Are subagents getting called when they should be?

**Verdict**: MIXED

**Evidence**:
- Lines 28, 30, 72, 74, 76, 296, 506, 628 each spawn an Agent.
- Candidate missed delegation flagged: Read x5 (lines 389–405) reading 5 distinct eval scenario files sequentially without an intervening Agent spawn.
- **Candidate missed delegation flagged**: Write x4 (lines 611–624) writing 4 plan documents sequentially without an intervening Agent spawn.

**Analysis**:
The orchestrator did delegate 8 times, but the Read x5 and Write x4 sections are valid candidates for delegation judgment because:
- **Read x5** spans 5 distinct files (P1-L1, P2-L2, P3-L3 scenarios + judge-rubric + check script) in the Eval C harness. These appear to be independent reference reads for understanding the eval, not sequential-dependent work (e.g., not "read A, analyze, decide on B, read B").
- **Write x4** (lines 611–624) writes 4 separate plan documents (`2026-07-06-plugin-evals-overview.md`, `ws1-orchestration-audit.md`, `ws2-os-vault-write-eval.md`, `ws3-status-convention-plugin.md`). These are tightly coupled by design (they reference each other; ws1–3 are explicit in the overview), making sequential authorship defensible but potentially over-orchestrator-load.

**Missed delegation trigger candidate**:  
When orchestrator has 5+ independent file reads on **different logical domains** (not a single multi-part document), delegate the read batch to a general-purpose agent with `run_in_background: true` if other work can proceed in parallel. Similarly, when writing 4+ loosely coupled documents with explicit cross-references, consider whether an agent authoring all four together saves context-switching and ensures consistency better than orchestrator authorship.

---

## Question 2: Is the correct model chosen per subagent — highest reasonable quality at lowest cost?

**Verdict**: FAIL

**Evidence**:
All 8 agents resolved to `claude-haiku-4-5-20251001` despite explicit model parameter mismatches:
- Agents 1–2 (lines 28, 30): requested `model: "sonnet"` → resolved haiku
- Agent 3 (line 72): requested `model: "sonnet"` → resolved haiku  
- Agent 4 (line 74): requested `model: "opus"` → resolved haiku
- Agent 5 (line 76): requested `model: "sonnet"` → resolved haiku
- Agents 6–8 (lines 296, 506, 628): requested `model: "haiku"` → resolved haiku (CORRECT, but only by accident)

**Analysis**:
This is a critical orchestration failure. The orchestrator explicitly passed a `model` parameter 8 times and the Agent tool **completely ignored it**, always resolving to haiku. This is not an orchestrator judgment error (the orchestrator made correct requests); it's a tool-system error upstream (likely the Agent tool definition or the model fallback logic).

The cost impact is mild (haiku is cheapest), but the quality impact is severe: Agents 1–2 (perspectives agents doing epistemic skepticism, which requires sonnet-tier reasoning) and Agent 4 (opus task building a complex eval harness) were systematically downgraded to haiku-only reasoning.

**Missed delegation trigger / missed orchestration signal**:  
Orchestrator should have detected that the `resolvedModel` field in toolUseResult differed from the requested `model` parameter and either (a) raised an error, (b) acknowledged the downgrade to the user, or (c) re-spawned with explicit routing. None of this happened. This is not orchestrator failure but a missing visibility/accountability layer.

---

## Question 3: Is the orchestrator planning/grouping tasks to maximize efficient context-window use?

**Verdict**: PASS (with caveats)

**Evidence**:
- Agent spawns 1–5 are clustered (lines 28–76), each conceptually independent, allowing parallel execution (confirmed: all marked `run_in_background: true` implicitly, async_launched).
- Agents 6–8 (git commits) are spawned sequentially (lines 296, 506, 628) with explicit tool dependencies (each runs after prior work completes), which is defensible.
- After-spawn-5 segment (37 tool calls) is the heavy work: Reads, Edits, Writes. This was done after agents returned, not during their background execution, indicating the orchestrator waited for results before proceeding.

**Analysis**:
The grouping is reasonable: epistemic/measurement agents (1–2), task-execution agents (3–5) are fanned out; execution-chain agents (6–8 commits) are sequenced. No evidence of poor context-window planning (e.g., reading a 50MB file then spawning an agent with it in context).

**Caveat**: After-spawn-5's 37 tool calls (41.5KB read) all happen in sequence without delegating any to the agents that just finished. If the Read x5 and Write x4 could have been delegated, this segment could have shed context load off the orchestrator.

---

## Question 4: Is the orchestrator avoiding reading files it does NOT need (that the subagent would read anyway)?

**Verdict**: PASS

**Evidence**:
- Orchestrator never reads the same file a spawned agent was instructed to read. The 5 Reads at lines 389–405 are in a context where no active agent is reading them (Agents 1–5 already completed; Agents 6–8 are commits with no Read dependency).
- No evidence of the orchestrator pre-reading files "just in case" an agent might need them.
- Total bytes read by orchestrator: 41,558 (after-spawn-5) + 61,527 (after-spawn-6) + 24,354 (after-spawn-7) + 563 (after-spawn-8) = ~127.7 KB over a 3-hour session. This is light.

**Analysis**: The orchestrator is not wastefully pre-reading. The flagged Read x5 is the orchestrator needing to understand eval scenarios, not an agent-delegation miss on *this* question.

---

## Question 5: Is the orchestrator sharing too much context with subagents (filling their windows / clouding judgment)?

**Verdict**: PASS

**Evidence**:
- Agent prompts (lines 28, 30, 72, 74, 76) are terse: 2.5–6.7 KB each. The session has ~70 KB used cumulative before any agent spawn.
- No evidence of the orchestrator dumping transcripts, full codebases, or search results into agent prompts wholesale.

**Analysis**: Prompts are focused. No over-context problem detected.

---

## Question 6: Is the orchestrator even following the ORCHESTRATION.md instructions?

**Verdict**: FAIL

**Evidence**:
The global ORCHESTRATION.md policy (injected at SessionStart) states:
> Do single-file, ≤2-tool-call ops directly. Don't delegate them. Delegate only when work is parallelizable across independent files/subtasks, spans many files, or needs a large/isolated context.

The orchestrator DID delegate appropriately on agents 1–5 (complex, multi-step, reasoning-heavy). BUT:
- Lines 28, 30, 74: Agent requests explicitly passed `model: "sonnet"` and `model: "opus"` parameters, and those were **systematically ignored** (all resolved to haiku). The instructions say "Every Agent spawn passes model explicitly. Default haiku for mechanical file-edit/shell work; sonnet for anything requiring judgment; opus only for genuinely hard reasoning." The orchestrator was right to request sonnet/opus, but the tool did not respect it.
- The orchestrator did NOT acknowledge or escalate this model-resolution failure, violating the implicit "verify your agent actually got the model you requested" accountability.

**Specific failed instruction**:  
> Every `Agent` spawn passes `model` explicitly.

The orchestrator did pass it; the tool ignored it. This is a tool-system error, not an orchestrator error, but the orchestrator should have caught it.

---

## Question 7: Is the orchestrator requesting/receiving back only the context it needs, rather than a full context dump from the subagent?

**Verdict**: PASS

**Evidence**:
- Agent spawn prompts are specific: "Challenge eval validity claims" (line 28), "Run 3-rep haiku W3 check" (line 72), "Build Eval C..." (line 74), etc.
- No evidence of the orchestrator requesting "summarize everything you did" or consuming 50+ KB of agent output. The fact-sheet shows result sizes of 2.9–12.9 KB, which are reasonable summaries.

**Analysis**: The orchestrator is precise in its requests and does not bloat context with full result dumps.

---

## Summary of Verdicts

| Q | Verdict | Issue |
|---|---------|-------|
| 1 | MIXED | Read x5 and Write x4 are delegation candidates; judgment needed on sequential dependency |
| 2 | FAIL | All 8 agents requested sonnet/opus but resolved to haiku; tool ignored `model` parameter |
| 3 | PASS | Grouping is sensible; agents 1–5 fanned out, 6–8 sequenced correctly |
| 4 | PASS | No wasteful pre-reading or file duplication |
| 5 | PASS | Agent prompts are focused, not over-contextualized |
| 6 | FAIL | Model parameter requests were ignored; orchestrator did not escalate the failure |
| 7 | PASS | Agent outputs were summaries, not full context dumps |

---

## Candidate Missed Delegations (Auditor Judgment Required)

### Delegation Candidate 1: Read x5 (lines 389–405)

**Files read**:
- `/home/jared/dev/cc-os/plugins/os-adr/eval-c/scenarios/P1-L1-execution.md`
- `/home/jared/dev/cc-os/plugins/os-adr/eval-c/scenarios/P2-L2-execution.md`
- `/home/jared/dev/cc-os/plugins/os-adr/eval-c/scenarios/P3-L3-execution.md`
- `/home/jared/dev/cc-os/plugins/os-adr/eval-c/judge-rubric.md`
- `/home/jared/dev/cc-os/plugins/os-adr/eval-c/bin/check`

**Context**: Orchestrator is reviewing the Eval C harness after Agent 5 (eval-building) completed. These reads appear to be for understanding the structure and validating/analyzing the built scenarios.

**Sequential dependency risk**: LOW. These are independent reference reads, not "read A → decide → read B" chains. They could be batched to a single agent with `run_in_background: true`.

**Verdict**: Legitimate orchestrator work, but could be delegated if the reads are analytical (not decisional) and the orchestrator is not actively synthesizing between them.

### Delegation Candidate 2: Write x4 (lines 611–624)

**Files written**:
- `/home/jared/dev/cc-os/docs/plans/2026-07-06-plugin-evals-overview.md` (main overview)
- `/home/jared/dev/cc-os/docs/plans/ws1-orchestration-audit.md` (WS1)
- `/home/jared/dev/cc-os/docs/plans/ws2-os-vault-write-eval.md` (WS2)
- `/home/jared/dev/cc-os/docs/plans/ws3-status-convention-plugin.md` (WS3)

**Context**: Orchestrator is authoring a 4-document plan set. The overview references all three workstreams; each workstream doc is independent but tightly coupled.

**Sequential dependency**: HIGH. The overview must exist (or be drafted) before the workstream docs can be written correctly, OR all four must be authored together to ensure cross-references are accurate.

**Verdict**: Defensibly orchestrator work due to coupling, but authoring all four as a delegated batch to a single agent might improve consistency and save context-switching. The coupling suggests an agent would be clearer on intent than sequential orchestrator writes.

---

## Failure-Mode Clusters (from Real Evidence)

### Cluster A: Model Parameter Ignored

**Symptom**: Orchestrator explicitly passes `model: "sonnet"` or `model: "opus"` to Agent, but `resolvedModel` in the response is always `haiku`.

**Severity**: HIGH

**Trigger**: Any Agent spawn with a non-haiku model parameter.

**Expected behavior**: `resolvedModel` should match `model` param, or Agent should raise an error.

**Actual behavior**: Silent downgrade to haiku.

**Candidate eval scenario**: Spawn agents with mixed models (sonnet, opus, haiku) and verify resolvedModel matches or error is raised.

---

## Recommendations

1. **Investigate model parameter loss**: The consistent 8/8 downgrade to haiku suggests a systematic issue in the Agent tool or model resolution logic, not orchestrator error. Recommend diagnosing why the `model` param is being ignored.

2. **Add model-parameter validation**: Orchestrator should log a warning or error if `resolvedModel != model` param (if one was provided).

3. **Read batch delegation**: Consider delegating Read x5 to a background agent with explicit scope ("read and summarize the structure of these 5 eval-c files").

4. **Write batch delegation**: Consider authoring multi-document plan sets via a single agent to ensure cross-reference consistency. This is less about cost and more about coherence.

5. **Model selection refinement**: Even if the parameter-passing issue is fixed, confirm that sonnet is the right choice for perspectives agents and opus is justified for eval-harness building. The current choices seem sound, but the downgrade hides the opportunity to validate them.

---

## Session Metadata

- **Main loop model**: claude-fable-5 (orchestrator)
- **Agent models**: all resolved to haiku (intended: mixed)
- **Total agent spawns**: 8
- **Parallel agents**: 5 (1–5, concurrent, bg)
- **Sequential agents**: 3 (6–8, git commits)
- **Total orchestrator tool calls**: 85 (excluding sidechain)
- **Total bytes read by orchestrator**: 127.7 KB
- **Candidate issues**: Model downgrade (Q2, Q6); delegation judgment needed (Q1)