cc-os/docs/orchestration-audit/auditor-reports/S1-report.md

# Orchestration Audit: S1-df546b88

## Session Metadata
- Transcript: `/home/jared/.claude/projects/-home-jared-dev-cc-os/df546b88-e27d-47e1-888c-648012c24e62.jsonl`
- cwd: /home/jared/dev/cc-os
- Duration: 2026-07-06 15:32:42 to 18:20:38 (2h 48m)
- Assistant turns: 66 (all Fable-5 / Haiku 4.5)
- Agent spawns: 10 (general-purpose ×3, perspectives ×4, Explore ×1, general-purpose ×2)
- Model params: All 10 requested `model: "sonnet"` → all 10 resolved as `claude-haiku-4-5-20251001`

## Critical Finding
**Agent model parameter is not being respected by the Agent framework.** All 10 agents explicitly requested `model: "sonnet"` in their tool_use input but resolved to Haiku. This prevents the orchestrator from implementing the cost-quality tradeoff specified in ORCHESTRATION.md.

Evidence: Line 21 (agent spawn 1) shows `"model": "sonnet"` in the Agent input; line 22 (toolUseResult) shows `"resolvedModel": "claude-haiku-4-5-20251001"`. Pattern holds for all 10 agents (lines 21, 23, 25, 79, 81, 83, 85, 202, 218, 219 as spawns; each followed by a tool result line with resolvedModel=haiku).

---

## Q1: Are subagents getting called when they should be?

**Verdict: PASS (with user's explicit request caveat)**

The user's initial prompt explicitly requests delegation: "Dispatch subagents based on model to intelligence and cost (low cost, high quality) to explore and research options, approaches as well as tools and services."

**Evidence:**
- Line 4 (user prompt): "Dispatch subagents based on model to intelligence and cost (low cost, high quality)"
- Lines 20-21: Orchestrator accepts the delegation request and clarifies reasoning: "Dispatching three sonnet researchers now (this is judgment-heavy evaluation work, not mechanical, so no haiku)"
- Agents 1-3 (lines 21, 23, 25) research distinct tools (Storybloq, self-hosted kanban options, agent-native backlogs) — parallelizable research tasks with no dependencies

**Pattern observation:** Agents 4-7 (perspectives) and 8-10 (deep-dives) appear to be reactive rather than pre-planned, running after earlier results come back. This is defensible (each synthesizes prior findings) but weakens the "pre-planned delegation" story.

**Candidate issue:** Agents 8-10 (Explore Hermes, Planka deep-dive, modern kanban alternatives sweep) lack clear pre-planning signals in the transcript. They appear reactive to synthesized agent findings rather than pre-delegated work.

---

## Q2: Is the correct model chosen per subagent?

**Verdict: FAIL — Model parameter ignored by Agent framework**

The orchestrator correctly specifies model parameters per ORCHESTRATION.md policy:
- All 10 agents requested `model: "sonnet"` (judgment-heavy research and perspective work)
- Policy expectation: Sonnet for judgment work, Haiku for mechanical work
- **Actual outcome:** All 10 resolved to Haiku

**Evidence:**
- Line 21: `"model": "sonnet"` in Agent tool input
- Line 22: `"resolvedModel": "claude-haiku-4-5-20251001"` in toolUseResult
- Line 23: Same pattern; line 24: `"resolvedModel": "claude-haiku-4-5-20251001"`
- Line 25: `"model": "sonnet"`; line 26: `"resolvedModel": "claude-haiku-4-5-20251001"`
- Agents 4-7 (perspectives: devils-advocate, simplifier, implementer, premortem): all requested `model: "sonnet"` but resolved to Haiku
- Agents 8-10: same pattern (lines 202, 218, 219 spawns → Haiku resolves on subsequent lines)

**Root cause:** Unknown — likely Agent framework / Claude Code plugin issue, not orchestrator error. The orchestrator is making the correct specification; the framework is downgrading.

**Impact:** Orchestrator cannot implement the stated policy of "sonnet for judgment work, haiku for mechanical work." All work runs at lower quality/cost than specified.

---

## Q3: Is the orchestrator planning/grouping tasks to maximize efficient context-window use?

**Verdict: MIXED — Good initial batching, then reactive pattern**

**Initial parallelization (agents 1-3):**
- Agents 1-3 dispatched together to research three distinct but related kanban systems
- Independent research tasks with no inter-dependencies
- Well-scoped individual prompts (~1100-2100 chars each)
- No orchestrator reading between spawns 1-3 (fact-sheet shows 0 tool calls pre-spawn-4)
- **This is optimal parallel batching**

**Perspectives batch (agents 4-7):**
- Agents 4-7 (devils-advocate, simplifier, implementer, premortem) dispatched after agents 1-3 complete
- Reviewing a concrete proposal synthesized from agent 1-3 results
- All four perspectives agents dispatched together (lines 79, 81, 83, 85)
- No reading between spawns 4-7
- **Pattern:** Sequential synthesis (agents 1-3 complete → orchestrator synthesizes → agents 4-7 launch against synthesis)
- This is reasonable but not pre-planned; it's reactive to agent results

**Final deep-dives (agents 8-10):**
- Agent 8 (Explore Hermes agent OS): line 202
- Agent 9 (Planka maturity deep-dive): line 218
- Agent 10 (Modern kanban alternatives sweep): line 219
- Agents 9-10 launched together but agent 8 is isolated
- No orchestrator context prep visible between perspectives completion and agent 8 launch
- **Pattern:** Reactive to ongoing synthesis, not pre-planned

**Inefficiency candidate:** The orchestrator could have planned agents 8-10 upfront rather than spawning them reactively. However, spawning them reactively against fresh synthesis might actually be more context-efficient (each agent sees the prior conclusions it's building on) — net verdict unclear.

---

## Q4: Is the orchestrator avoiding reading files it does NOT need?

**Verdict: PASS — Minimal unnecessary reading**

**File reading pattern:**
- Pre-spawn-1 through post-spawn-6: 0 bytes read (fact-sheet segments pre-spawn-1 through after-spawn-6)
- After-spawn-7 (post-perspectives): 970 bytes read via Skill:1, Bash:1, Write:2
- After-spawn-10 (final synthesis): 7106 bytes read via Bash:3, Read:1, Write:1, Edit:1

**Details of after-spawn-7 reading (970 bytes):**
- Likely routine write of agent results or notes; Skill invocation suggests a vault operation

**Details of after-spawn-10 reading (7106 bytes):**
- Line 239: Bash to list `/home/jared/servers/ovh-prod/` and grep for SecondBrain references (practical investigation of user's infrastructure)
- Line 244: Read tool (exact file unknown without parsing, but single Read suggests targeted lookup, not bulk scan)
- Writes: likely capturing findings

**Assessment:** The orchestrator does not pre-load CLAUDE.md, docs/, or large context files before delegating. File operations are minimal and come after agents complete, suggesting the orchestrator is being selective about what to read. This is aligned with ORCHESTRATION.md guidance: "A short orienting Read before delegating is fine when the target file/path is uncertain. Don't delegate the orienting step itself."

---

## Q5: Is the orchestrator sharing too much context with subagents?

**Verdict: PASS — Prompts are focused, not bloated**

**Agent 1 prompt (Storybloq research):** ~1098 chars
- Specific research task (assess GitHub project for kanban/backlog viability)
- Clear criteria (maturity, self-hosting, AI accessibility, visual dashboard)
- No dump of CLAUDE.md, project history, or vault context
- Appropriate scope for isolated research

**Agents 2-3 prompts:** ~1600-2100 chars each
- Agent 2: Specific survey task (self-hosted kanban tools inventory)
- Agent 3: Specific research task (agent-native/markdown backlogs)
- No bloat; instruction-forward, not context-forward

**Agents 4-7 (perspectives) prompts:** ~2100-3300 chars each
- Each perspective receives the concrete proposal being reviewed
- No full CLAUDE.md dump; focused on the proposal and the perspective lens
- Reasonable context load for judgment work

**Agents 8-10 prompts:** Unknown exact sizes but factsheet rows suggest ~1100-2300 chars (row 8: 1120 chars, row 9: 1832 chars, row 10: 2301 chars)
- Proportionate to research scope, not bloated

**Assessment:** No evidence of context waste. Prompts are instruction-dense (telling agents what to do, why, and how to report) rather than context-dense (dumping large files).

---

## Q6: Is the orchestrator following the ORCHESTRATION.md instructions?

**Verdict: MIXED — Orchestrator tries to follow but Agent framework doesn't comply**

**ORCHESTRATION.md policy (stated verbatim):**
1. "Do single-file, ≤2-tool-call ops directly. Don't delegate them."
2. "Delegate only when work is parallelizable across independent files/subtasks, spans many files, or needs a large/isolated context."
3. "Every `Agent` spawn passes `model` explicitly."
4. "Default `haiku` for mechanical file-edit/shell work; `sonnet` for anything requiring judgment; `opus` only for genuinely hard reasoning."

**Orchestrator's behavior:**
1. **Single-file / ≤2-tool ops:** Not violated. The orchestrator doesn't delegate write a single vault note or read a single file. Delegation is reserved for multi-sourced research and judgment work.
2. **Parallelizable / spans-files:** Respected. Agents 1-3 are independent research; agents 4-7 are parallel judgment.
3. **Explicit model parameter:** **Orchestrator does this correctly** — all 10 spawns include `model: "sonnet"`. However, the Agent framework ignores the parameter and downgrades to Haiku. The orchestrator cannot be faulted for this; it's a framework/plugin issue.
4. **Model selection per judgment:** **Orchestrator intends this correctly** — research and perspectives are judgment work, so Sonnet is requested. But the framework downgrade means actual models are all Haiku.

**Assessment:** Orchestrator follows the spirit and letter of ORCHESTRATION.md. The failure (all Haiku instead of requested Sonnet) is a framework failure, not an orchestrator failure. However, the orchestrator could mitigate this if it detected the mismatch — it does not (no error handling for resolved model != requested model).

**Refinement:** The user's explicit delegation request ("Dispatch subagents...") overrides the ORCHESTRATION.md preference for direct work. This is correct per the policy's intent: "Delegate only when..." — the user's explicit need qualifies.

---

## Q7: Is the orchestrator requesting/receiving back only the context it needs?

**Verdict: PASS — Concise communication, no full-context-dump pattern**

**How agents report:**
- Agents complete asynchronously in the background (line 22: "The agent is working in the background. You will be notified automatically when it completes.")
- Task notifications include agent output via the tool result (exact channel uncertain without parsing task output files)
- Orchestrator receives task notifications as user lines, summarizes findings, and moves on

**Orchestrator's synthesis pattern:**
- Lines 77-78: Orchestrator reads the Storybloq/tools/agent-native research and synthesizes key insights (adds recurrence/lifecycle concepts to the proposal) before perspectives
- No evidence of the orchestrator reading 100KB of raw agent output; synthesis is rapid and concise

**Final synthesis:**
- Line 201 shows orchestrator text output after perspectives complete, synthesizing four perspectives into recommendations
- Tight, focused synthesis; no dumping of raw agent transcripts

**Assessment:** The communication pattern is clean. Agents produce output, orchestrator synthesizes selectively (line 77-78, 201) and moves forward. No evidence of "full context dump" or reading files it doesn't need.

---

## Summary of Issues

### Critical Issue
**Model parameter not respected by Agent framework** (lines 22, 24, 26, etc. — all resolvedModel = haiku despite model: sonnet request)
- Blocks orchestrator from implementing cost-quality policy
- Not an orchestrator error; framework/plugin failure
- Orchestrator makes correct parameter choices; execution layer fails

### Secondary Issue
**Agents 8-10 appear reactive rather than pre-planned**
- Could indicate lack of forward planning
- May be acceptable if reactive spawning is more context-efficient (each agent sees latest synthesis)
- Weak signal without seeing the orchestrator's internal reasoning

### N/A Issues
- File reading: minimal and appropriate
- Context sharing: focused, not bloated
- Model selection intent: correct (all judgment work gets Sonnet request)
- ORCHESTRATION.md compliance: followed except for framework failure

---

## Recommendations

1. **Immediate:** Debug why Agent framework downgrades model parameter. Check Claude Code plugin / Agent framework configuration.

2. **Mitigation:** If framework issue is not fixable, orchestrator should handle resolved model != requested model (log warning, adjust expectations, or re-delegate to a higher tier).

3. **Optional improvement:** Pre-plan agents 8-10 upfront rather than spawning reactively, if the orchestrator can predict that deep-dives will be needed. Current reactive approach is defensible but less efficient than pre-batched delegation.

4. **Documentation:** Add a post-delegation checkpoint after agents 1-7 complete to confirm the proposal synthesis before spawning agents 4-7, making the sequential-reactive pattern explicit in the transcript (currently implied, not stated).

---

## Checklist (7 Questions)

| # | Question | Verdict | Key Evidence Line(s) |
|---|----------|---------|----------------------|
| 1 | Subagents called when should be? | PASS | 4, 20, 21 (user request + orchestrator accepts) |
| 2 | Correct model chosen? | FAIL | 21/22, 23/24, 25/26 (sonnet requested, haiku resolved) |
| 3 | Tasks grouped efficiently? | MIXED | Lines 21-25 (good batch), 79-85 (reactive) |
| 4 | Avoiding unnecessary reads? | PASS | Factsheet: 0 reads pre-spawn-7 |
| 5 | Too much context shared? | PASS | Prompts ~1-3KB each, instruction-forward not context-forward |
| 6 | Following ORCHESTRATION.md? | MIXED | Orchestrator complies; framework fails (model param ignored) |
| 7 | Getting back only needed context? | PASS | Concise synthesis, no full-dump pattern |