server-desktop-01/docs/orchestration-audit-prd.md

# PRD — Deeper orchestration audit

**Status:** ready to run (new session) · **Created:** 2026-06-30
**Predecessor:** `docs/cc-tool-audit.md` (§ Addendum — orchestration audit)
**Note:** No issue tracker/remote on this repo, so this PRD lives as a doc. The
`/to-prd` "publish + `ready-for-agent` label" step was adapted to a file.

---

## Problem Statement

Sessions feel slow and over-spend on model cost. A prior audit
(`docs/cc-tool-audit.md`) traced this less to loaded tools/skills/hooks (all small)
and more to the **orchestrator-subagent delegation pattern**. A first-pass audit of
1,962 subagent transcripts found that **27% of subagents run on Opus**, the majority
of them doing mechanical file-edit/shell work that Haiku/Sonnet would handle faster
and cheaper. The CLAUDE.md "Session Orchestration" routing table is **advisory** —
when an `Agent` spawn omits `model`, the subagent inherits the parent (Opus). The
rule also mandates delegation with **no minimum complexity threshold**, so trivial
one-line edits pay a full subagent prefill + sequential wait.

An interim fix is already applied (`CLAUDE_CODE_SUBAGENT_MODEL=haiku` in global
settings). What's missing is the **quantified evidence** needed to safely revise the
orchestration *instructions* across many CLAUDE.md files — not just the default model.

## Solution

A deeper, repeatable audit of the existing session transcripts that produces the
numbers to ground a **pilot** revision of the Session Orchestration rules on one
project, before any broad rollout. The audit answers: how often is the wrong model
chosen, is delegation itself wasteful for small tasks, and is work being decomposed
well before hand-off. Output is a short findings report + a proposed pilot rule diff.

## User Stories

1. As the operator, I want the subagent model distribution broken down per project, so that I can see which repos leak Opus most.
2. As the operator, I want the "file-edit/shell" subagent bucket split into *trivial* (1–2 calls, single file) vs *substantial* (multi-file, many calls), so that I can tell genuine Sonnet/Opus work from Haiku-appropriate work instead of relying on a coarse bucket.
3. As the operator, I want each subagent's spawn traced to whether its model was **explicitly passed** by the orchestrator or **inherited** from the parent, so that I can measure the routing-leak rate directly.
4. As the operator, I want a count of subagents that did near-zero work (0–1 tool calls, or returned trivially), so that I can quantify wasted delegations caused by the "no threshold" rule.
5. As the operator, I want to see how many orchestrator turns spawned a single subagent for a single small op (delegation that a direct op would beat on latency), so that I can justify a complexity threshold.
6. As the operator, I want parallelism measured — how often independent subagents were batched concurrently vs. spawned sequentially — so that I can see if decomposition exploits the parallel-batch guidance.
7. As the operator, I want examples of good and bad decomposition (a clean parallel batch vs. a needless single-op delegation), so that the pilot rule change is illustrated, not just asserted.
8. As the operator, I want the audit limited to delegation-heavy projects (cc-os, llf-schema, proxmox, philly-seo, ovh-prod), so that it runs on the relevant sample, not every repo.
9. As the operator, I want the interim `CLAUDE_CODE_SUBAGENT_MODEL=haiku` effect measurable in future sessions (a before/after marker), so that I can confirm the env var actually shifted the distribution.
10. As the operator, I want a proposed pilot diff to one project's "Session Orchestration" section (threshold + enforced routing + allow orienting reads), so that I can try it on a single repo and measure wall-clock/cost before rolling out.
11. As the operator, I want the audit scripted/repeatable (saved under the repo), so that I can re-run it after the pilot to compare.

## Implementation Decisions

- **Data source:** existing JSONL transcripts in `~/.claude/projects/*/` (main) and
  `~/.claude/projects/*/<session-id>/subagents/agent-*.jsonl` (subagents). Model is in
  each subagent message's `model` field; tool calls are `type:"tool_use"` blocks
  (walk recursively). Project dir names start with `-` — use python `glob`, never
  shell `find`/`ls`.
- **Trivial vs substantial classifier:** trivial = ≤2 tool calls and ≤1 distinct file
  touched; substantial = otherwise. Tune threshold against spot-checked examples.
- **Inherited vs explicit model trace:** match parent `Agent` tool_use input (does it
  carry a `model`/`subagent_type`?) to the child transcript via agent id, then compare
  requested vs actual model. Where the parent omitted model and child==parent model,
  classify as "inherited (leak)."
- **Parallelism metric:** within a main session, detect `Agent` tool_use blocks issued
  in the same assistant turn (concurrent) vs separate turns (sequential).
- **Scope:** the 5 delegation-heavy projects above; report per-project and pooled.
- **Output:** append a "Deep audit" section to `docs/cc-tool-audit.md` (or a sibling
  doc) with tables + 3–5 examples, plus a proposed pilot rule diff for ONE project.
- **No production change** beyond the already-applied env var until the pilot is
  reviewed.

## Testing Decisions

- This is an analysis task, not a shipped feature — "tests" = sanity checks on the
  parser: total subagent count matches the prior audit (~1,962 with a model), model
  totals reconcile, and the trivial/substantial split sums to the file-edit bucket.
- Spot-verify the classifier on 5 hand-read transcripts (one per project) before
  trusting the aggregate.
- Re-run the same script post-pilot to confirm the distribution shifted (regression
  baseline = today's numbers).

## Out of Scope

- Editing every CLAUDE.md (this is assessment → single-project pilot only).
- Changing tool/skill/hook config (prior audit showed negligible payoff).
- Building any new tooling, MCP, or harness changes.
- Touching the live orchestration behavior beyond the env var already set.

## Further Notes

- Interim change already live: `env.CLAUDE_CODE_SUBAGENT_MODEL=haiku` in
  `~/.claude/settings.json` (applies to sessions launched after 2026-06-30).
- Prior-audit baseline to beat: Opus 26.6% / Sonnet 36.7% / Haiku 36.5% of subagents;
  Opus subagents 53% file-edit/shell.
- Reusable scratch scripts from the first pass:
  `/tmp/claude-1000/.../scratchpad/{full,cat,orch}.py` (ephemeral — recreate or
  promote into the repo when running the deep audit).
- Open instruction-design questions for the pilot: where to set the complexity
  threshold; whether to allow N orienting reads before delegating; how to phrase
  "always pass model on spawn" given the env var now provides a cheap default.