jared
/
cc-os
1
0
Fork 0
Code Issues 18 Pull Requests Packages Projects Releases Wiki Activity
cc-os/docs/orchestration-audit/auditor-reports/S9-report.md

16 KiB
Raw Permalink Blame History

Orchestration Audit: Session 2525fa2d-1b15-4495-99bf-c093716fab55

Project: systems-admin
Date: 2026-06-30
Main loop model: claude-opus-4-8 (all 117 turns)
Assistant turns / human prompts: 117 / 6

Summary

The session executed a self-audit of Claude Code orchestration/delegation practices across the user's project portfolio. It spawned 3 background agents (2 general-purpose/opus, 1 claude-code-guide/haiku) to parallelize read-only research and documentation verification. The orchestrator correctly avoided reading files before delegating (except for initial orientation), structured prompts precisely without context bloat, and produced a PRD + handoff summary as deliverables.

Critical policy violation: None of the 3 Agent spawns passed a model parameter explicitly, violating the stated ORCHESTRATION.md rule ("Every Agent spawn passes model explicitly").

Findings by Question

1. Are subagents getting called when they should be?

Verdict: PASS — Delegations were appropriate and well-scoped.

Evidence:

  • Line 95: Agent spawn #1 for "Inventory loaded context surface" — reads multiple config/skill directories and sums character counts. Parallelizable across multiple file paths; spans multiple directories (~/.claude/settings.json, ~/.claude/settings.local.json, ~/.claude/skills/, ~/.claude/plugins/). Read-only task that requires no modification. Appropriate delegation.
  • Line 97: Agent spawn #2 for "Pull session examples" — analyze JSONL transcripts across 4 project categories to find representative examples. Explicitly requires reaching across many project directories (~/.claude/projects/*); good candidate for parallelization. Appropriate delegation.
  • Line 212: Agent spawn #3 for "Verify skill loading mechanism" — verify factual information from code.claude.com/docs. Specialist task (claude-code-guide subagent type). Appropriate delegation.

Analysis:
All three delegations follow the spirit of "delegate when work spans many files or needs isolated context." None were single-file operations; all were read-only; none were ≤2-tool-call mechanical ops that should have been done directly.

2. Is the correct model chosen per subagent — highest reasonable quality at lowest cost?

Verdict: FAIL — All spawns lack explicit model parameters; two are overpowered.

Evidence:

  • Line 95: Agent spawn #1 has no model param; resolvedModel = claude-opus-4-8. Task is character-counting and summation (mechanical, read-only). Appropriate model would be haiku (mechanical file-reading) or at most sonnet (if judgment needed on "contextually heavy" categorization). Opus is overpowered. Fact-sheet: prompt chars = 2662; result chars = 7517.
  • Line 97: Agent spawn #2 has no model param; resolvedModel = claude-opus-4-8. Task is example selection and characterization — requires judgment on what makes examples representative. Appropriate model would be sonnet (judgment). Opus is overpowered. Fact-sheet: prompt = 2397 chars; result = 6095 chars.
  • Line 212: Agent spawn #3 has no model param; resolvedModel = claude-haiku-4-5-20251001. Task is documentation lookup/verification (mechanical). Model is correct. Fact-sheet: prompt = 2065 chars; result = 5290 chars.

Analysis:
The policy states: "Default haiku for mechanical file-edit/shell work; sonnet for anything requiring judgment; opus only for genuinely hard reasoning."

  • Spawn #1: Character-counting is mechanical → should be haiku.
  • Spawn #2: Example selection requires judgment → should be sonnet.
  • Spawn #3: Doc lookup is mechanical → haiku is correct.

The fact that spawn #3 resolved to the correct model despite no explicit param suggests either accidental correct default or the haiku-for-docs-lookup heuristic in the subagent type. Spawns #1 and #2 defaulting to opus indicates the main-loop model is the fallback when no param is passed — a risk.

3. Is the orchestrator planning/grouping tasks to maximize efficient context-window use?

Verdict: MIXED — Good parallelization structure; weak on pre-planning.

Evidence:

  • Fact-sheet: Pre-spawn-1 segment: 15 tool calls, 75,718 bytes read. This is heavy read-ahead before delegating.
  • Lines 1–94: The orchestrator read broadly (ToolSearch for deferred tools, WebFetch for docs, 12 Bash calls for inventory, 1 Read). This is exploratory/orientation work.
  • Once orientation was complete, the orchestrator delegated to agents that could parallelize: spawn #1 and #2 are independent (different data sources: settings/skills vs. session transcripts). No coordination between them. Good.
  • After-spawn-2 segment: 15 tool calls, 8100 bytes read. This is synthetic/analysis work between agent results (Bash for intersecting data, measurements, hook timing — not duplicating what agents did).
  • After-spawn-3 segment: 12 tool calls, 5712 bytes read. This is synthesis/application work (editing settings, writing docs).

Analysis:
The orchestrator did substantial pre-work (75KB read) before spawning agents. This is justified for orientation, but the question is: did it read files the agents would need to read anyway?

The agents' prompts specify their sources clearly (e.g., "Look in ~/.claude/settings.json..."), suggesting the orchestrator didn't pre-read and pass file contents as context (which would bloat the subagent prompts). This is good context discipline. However, the pre-spawn read included some files later read by agents (e.g., settings.json appears in both the orchestrator's pre-read and agent prompts), suggesting some redundancy.

Grouped well: Three independent tasks → delegated in parallel (both async spawns at lines 95–97 are background, non-blocking).

4. Is the orchestrator avoiding reading files it does NOT need (that the subagent would read anyway)?

Verdict: MIXED — Mostly good; some redundancy detected.

Evidence:

  • Orchestrator's Reads (4 calls):

    1. /tmp/.../tool-results/toolu_01B9mmZvD4pgvmGYRYBi8QsP.txt — reading its own prior tool output. Necessary.
    2. /home/jared/systems-admin/docs/incidents.md — not directly mentioned in any agent prompt. Supporting context for synthesis.
    3. /home/jared/.claude/settings.json — this file is explicitly requested in agent spawn #1's prompt ("~/.claude/settings.json and ~/.claude/settings.local.json"). Orchestrator read it, agent also reads it. Redundancy.
    4. /home/jared/.claude/projects/-home-jared-systems-admin/memory/MEMORY.md — session's own project memory, used for synthesis.

  • Pre-spawn-1 Bash calls included: "Survey session files by project," "Aggregate tool usage across all sessions," "Full aggregation incl subagents." These are inventory/aggregation, not file-contents reads.

Analysis:
The orchestrator read ~/.claude/settings.json before delegating to spawn #1, which also reads the same file. This suggests the orchestrator was orienting to the data before delegating, not trusting the agent to explore independently. The agents' prompts are exploratory ("Look in ~/.claude/skills/ and in installed plugins") and don't require pre-staging of file contents.

One critical check: None of the agent prompts contain large file contents (e.g., no "here is your settings.json:..."). The prompts are instructions + paths, not data. This is correct. The orchestrator's redundant read of settings.json was for its own synthesis, not for pre-staging the agent.

5. Is the orchestrator sharing too much context with subagents (filling their windows / clouding judgment)?

Verdict: PASS — Prompts are concise and instructional; no unnecessary context dumps.

Evidence:

  • Spawn #1 prompt (Line 95): 2662 chars. Structured as: problem statement (1 sentence), 5 numbered investigation directives (specific file paths, measurement method), deliverable format. No file contents. No system-prompt injections. Clean.
  • Spawn #2 prompt (Line 97): 2397 chars. Structured as: problem (1 sentence), file locations (with technical note about path escaping), 4 concrete example categories, what to report per example, goal statement. No file contents. Clean.
  • Spawn #3 prompt (Line 212): 2065 chars. Structured as: problem (1 sentence), 3 numbered specific questions, instruction to verify via official docs. No file contents. Clean.

Result sizes:

  • Spawn #1 result: 7517 chars (3x the prompt) — agent returned structured data (tables, descriptions). Expected.
  • Spawn #2 result: 6095 chars (2.5x the prompt) — agent returned 4 example session case-studies. Expected.
  • Spawn #3 result: 5290 chars (2.5x the prompt) — agent returned answers to 3 questions. Expected.

Analysis:
The prompts are instruction-heavy and data-light. They guide the agent to read specific paths / sources directly, rather than passing pre-fetched contents. This is efficient and leaves maximum window space in the subagent for reading / reasoning. No bloat detected.

6. Is the orchestrator even following the ORCHESTRATION.md instructions?

Verdict: FAIL — Does not pass model explicitly in any spawn; otherwise follows the spirit.

Evidence:

Explicit policy (from ORCHESTRATION.md as of the session date, 2026-06-30):

- Do single-file, ≤2-tool-call ops directly. Don't delegate them. Delegate only when
  work is parallelizable across independent files/subtasks, spans many files, or
  needs a large/isolated context (long log review, wide grep-and-synthesize).
- Every `Agent` spawn passes `model` explicitly. Default `haiku` for mechanical
  file-edit/shell work; `sonnet` for anything requiring judgment; `opus` only for
  genuinely hard reasoning.
- A short orienting Read before delegating is fine when the target file/path is
  uncertain. Don't delegate the orienting step itself.

Compliance check:

  1. "Do single-file, ≤2-tool-call ops directly."
    ✓ PASS — All three agent-delegated tasks span multiple files or are large research tasks. No small ops were delegated.

  2. "Every Agent spawn passes model explicitly."
    ✗ FAIL — Lines 95, 97, 212: All three spawns omit the model parameter. The policy requires it to be explicit.

    // Line 95 (should include "model": "haiku")
{"name": "Agent", "input": {"description": "Inventory...", "subagent_type": "general-purpose", "prompt": "..."}}

// Line 97 (should include "model": "sonnet")
{"name": "Agent", "input": {"description": "Pull session...", "subagent_type": "general-purpose", "prompt": "..."}}

// Line 212 (should include "model": "haiku")  
{"name": "Agent", "input": {"description": "Verify skill...", "subagent_type": "claude-code-guide", "prompt": "..."}}

  3. "A short orienting Read before delegating is fine."
    ✓ PASS — The 75KB pre-spawn-1 read is orientation (exploring what tools exist, where transcripts live). It does not hand-hold or pre-fetch entire configs for agents to parse. Appropriate.

Analysis:
The orchestrator follows the spirit of ORCHESTRATION.md (delegate well-scoped, multi-file work; avoid bloating prompts; do light orientation) but violates the explicit syntactic requirement that every Agent spawn include a model parameter. This is a clear, fixable defect.

7. Is the orchestrator requesting/receiving back only the context it needs, rather than a full context dump from the subagent?

Verdict: PASS — Return values are structured and focused; no raw dumps.

Evidence:

  • Spawn #1 deliverable instruction: "Deliverable (return as structured markdown, this is the data I need back — be concrete with numbers): A ranked table: contributor category → total chars → est. tokens → # items. Top 15 individual heaviest skill/agent descriptions..."
    Result: Structured markdown with tables and brief descriptions. Not a directory dump or full file contents. ✓

  • Spawn #2 deliverable instruction: "Report: Project + short session id, One-line summary... The DISTINCT set of tools actually used... Which broad capabilities were NOT used at all... Any Skill invocations... Return structured markdown."
    Result: 4 case-study sections with the requested fields. Not a raw JSONL dump. ✓

  • Spawn #3 deliverable instruction: "Return a tight, factual answer to each of the three, flagging anything the docs don't clearly state."
    Result: 3 paragraphs answering the 3 questions. Not a doc-dump. ✓

Analysis:
All three agent results are precisely bounded to what was requested. No agent returned unfiltered file contents, full directory listings, or raw JSONL. The orchestrator specified expected output format (structured markdown, tables, case studies) in each prompt, and agents complied. Efficient.

Detailed Findings

Model Selection Defect

The most actionable finding is the absence of explicit model parameters. This violates the policy and incurs unnecessary cost:

Spawn Task Recommended Model Resolved Model Cost Ratio Impact
#1 Inventory (char-count, read-only) haiku opus ~3.5x Overspent by ~2.5x input cost on a mechanical task
#2 Example selection (judgment) sonnet opus ~1.5x Overspent by ~0.5x on judgment task; haiku would likely fail
#3 Doc verification (lookup) haiku haiku 1x Correct; lucked into right default via subagent type

Estimated cost overrun on this session: ~$0.XX (exact numbers would require token count from subagent transcripts, which this audit cannot access without exceeding scope). For a single session, negligible; systemically across projects, material.

Context Reuse Opportunity

The orchestrator read ~/.claude/settings.json before spawn #1, then spawn #1 reads it again. This is not a bloat issue (settings.json is small, ~5–10KB), but it suggests the orchestrator could rely more on agents to explore independently rather than self-orienting on data the agent is designed to discover.

Candidate rule refinement: "A short orienting Read before delegating is fine when needed for task planning or decision-making, but avoid reading the same data the agent's prompt instructs it to gather."

Recommendations

  1. Fix: Require explicit model parameter in every Agent spawn.

    • Add "model": "haiku" to spawn #1 (inventory/char-count).
    • Add "model": "sonnet" to spawn #2 (example selection/judgment).
    • Add "model": "haiku" to spawn #3 (doc lookup) — already resolved correctly, but make it explicit.

  2. Consider: Clarify the "orienting Read" exception.

    • Current rule allows "short orienting Read before delegating." Refine to avoid reading data the agent's prompt explicitly instructs it to gather.
    • Or: Accept that dual-reading small files (~5–10KB) is acceptable cost for clear, self-contained agent prompts (no hand-staging).

  3. No structural issues detected with task decomposition, prompt design, or result handling. All three delegations were well-scoped and produced focused deliverables.

Appendix: Tool & Context Tracking

Orchestrator tool calls (117 turns, main loop):

  • Pre-spawn-1 (lines 1–94): 15 tool calls, 75,718 bytes read
    • ToolSearch, WebFetch, 12× Bash (exploration), 1× Read
  • Span 1→2 (lines 95–96): 0 tool calls (async agents launching)
  • Between spawn-2 and spawn-3 (lines 97–211): 15 tool calls, 8,100 bytes read
    • 12× Bash (measurement/analysis), 1× ScheduleWakeup, 1× Read, 1× Write
  • After spawn-3 (lines 212 onwards): 12 tool calls, 5,712 bytes read
    • 3× Edit (settings.json, audit doc, project memory)
    • 2× Read (tool results, settings context)
    • 2× Skill (to-prd, handoff)
    • 2× Bash (validation, issue tracker check)
    • 3× Write (final audit docs)

Subagent spawns: 3 (all async background; no sequential coupling)

No missed delegations: Fact-sheet heuristic detected no runs of ≥4 same-tool calls without delegation.

Audit completed 2026-07-06. Report path: /tmp/claude-1000/-home-jared-dev-cc-os/3c15edee-8ac7-4ffe-b6ee-ee6a6ebd670d/scratchpad/audits/S9-report.md