98 lines
6.7 KiB
Markdown
98 lines
6.7 KiB
Markdown
# PRD — Deeper orchestration audit
|
||
|
||
**Status:** ready to run (new session) · **Created:** 2026-06-30
|
||
**Predecessor:** `docs/cc-tool-audit.md` (§ Addendum — orchestration audit)
|
||
**Note:** No issue tracker/remote on this repo, so this PRD lives as a doc. The
|
||
`/to-prd` "publish + `ready-for-agent` label" step was adapted to a file.
|
||
|
||
---
|
||
|
||
## Problem Statement
|
||
|
||
Sessions feel slow and over-spend on model cost. A prior audit
|
||
(`docs/cc-tool-audit.md`) traced this less to loaded tools/skills/hooks (all small)
|
||
and more to the **orchestrator-subagent delegation pattern**. A first-pass audit of
|
||
1,962 subagent transcripts found that **27% of subagents run on Opus**, the majority
|
||
of them doing mechanical file-edit/shell work that Haiku/Sonnet would handle faster
|
||
and cheaper. The CLAUDE.md "Session Orchestration" routing table is **advisory** —
|
||
when an `Agent` spawn omits `model`, the subagent inherits the parent (Opus). The
|
||
rule also mandates delegation with **no minimum complexity threshold**, so trivial
|
||
one-line edits pay a full subagent prefill + sequential wait.
|
||
|
||
An interim fix is already applied (`CLAUDE_CODE_SUBAGENT_MODEL=haiku` in global
|
||
settings). What's missing is the **quantified evidence** needed to safely revise the
|
||
orchestration *instructions* across many CLAUDE.md files — not just the default model.
|
||
|
||
## Solution
|
||
|
||
A deeper, repeatable audit of the existing session transcripts that produces the
|
||
numbers to ground a **pilot** revision of the Session Orchestration rules on one
|
||
project, before any broad rollout. The audit answers: how often is the wrong model
|
||
chosen, is delegation itself wasteful for small tasks, and is work being decomposed
|
||
well before hand-off. Output is a short findings report + a proposed pilot rule diff.
|
||
|
||
## User Stories
|
||
|
||
1. As the operator, I want the subagent model distribution broken down per project, so that I can see which repos leak Opus most.
|
||
2. As the operator, I want the "file-edit/shell" subagent bucket split into *trivial* (1–2 calls, single file) vs *substantial* (multi-file, many calls), so that I can tell genuine Sonnet/Opus work from Haiku-appropriate work instead of relying on a coarse bucket.
|
||
3. As the operator, I want each subagent's spawn traced to whether its model was **explicitly passed** by the orchestrator or **inherited** from the parent, so that I can measure the routing-leak rate directly.
|
||
4. As the operator, I want a count of subagents that did near-zero work (0–1 tool calls, or returned trivially), so that I can quantify wasted delegations caused by the "no threshold" rule.
|
||
5. As the operator, I want to see how many orchestrator turns spawned a single subagent for a single small op (delegation that a direct op would beat on latency), so that I can justify a complexity threshold.
|
||
6. As the operator, I want parallelism measured — how often independent subagents were batched concurrently vs. spawned sequentially — so that I can see if decomposition exploits the parallel-batch guidance.
|
||
7. As the operator, I want examples of good and bad decomposition (a clean parallel batch vs. a needless single-op delegation), so that the pilot rule change is illustrated, not just asserted.
|
||
8. As the operator, I want the audit limited to delegation-heavy projects (cc-os, llf-schema, proxmox, philly-seo, ovh-prod), so that it runs on the relevant sample, not every repo.
|
||
9. As the operator, I want the interim `CLAUDE_CODE_SUBAGENT_MODEL=haiku` effect measurable in future sessions (a before/after marker), so that I can confirm the env var actually shifted the distribution.
|
||
10. As the operator, I want a proposed pilot diff to one project's "Session Orchestration" section (threshold + enforced routing + allow orienting reads), so that I can try it on a single repo and measure wall-clock/cost before rolling out.
|
||
11. As the operator, I want the audit scripted/repeatable (saved under the repo), so that I can re-run it after the pilot to compare.
|
||
|
||
## Implementation Decisions
|
||
|
||
- **Data source:** existing JSONL transcripts in `~/.claude/projects/*/` (main) and
|
||
`~/.claude/projects/*/<session-id>/subagents/agent-*.jsonl` (subagents). Model is in
|
||
each subagent message's `model` field; tool calls are `type:"tool_use"` blocks
|
||
(walk recursively). Project dir names start with `-` — use python `glob`, never
|
||
shell `find`/`ls`.
|
||
- **Trivial vs substantial classifier:** trivial = ≤2 tool calls and ≤1 distinct file
|
||
touched; substantial = otherwise. Tune threshold against spot-checked examples.
|
||
- **Inherited vs explicit model trace:** match parent `Agent` tool_use input (does it
|
||
carry a `model`/`subagent_type`?) to the child transcript via agent id, then compare
|
||
requested vs actual model. Where the parent omitted model and child==parent model,
|
||
classify as "inherited (leak)."
|
||
- **Parallelism metric:** within a main session, detect `Agent` tool_use blocks issued
|
||
in the same assistant turn (concurrent) vs separate turns (sequential).
|
||
- **Scope:** the 5 delegation-heavy projects above; report per-project and pooled.
|
||
- **Output:** append a "Deep audit" section to `docs/cc-tool-audit.md` (or a sibling
|
||
doc) with tables + 3–5 examples, plus a proposed pilot rule diff for ONE project.
|
||
- **No production change** beyond the already-applied env var until the pilot is
|
||
reviewed.
|
||
|
||
## Testing Decisions
|
||
|
||
- This is an analysis task, not a shipped feature — "tests" = sanity checks on the
|
||
parser: total subagent count matches the prior audit (~1,962 with a model), model
|
||
totals reconcile, and the trivial/substantial split sums to the file-edit bucket.
|
||
- Spot-verify the classifier on 5 hand-read transcripts (one per project) before
|
||
trusting the aggregate.
|
||
- Re-run the same script post-pilot to confirm the distribution shifted (regression
|
||
baseline = today's numbers).
|
||
|
||
## Out of Scope
|
||
|
||
- Editing every CLAUDE.md (this is assessment → single-project pilot only).
|
||
- Changing tool/skill/hook config (prior audit showed negligible payoff).
|
||
- Building any new tooling, MCP, or harness changes.
|
||
- Touching the live orchestration behavior beyond the env var already set.
|
||
|
||
## Further Notes
|
||
|
||
- Interim change already live: `env.CLAUDE_CODE_SUBAGENT_MODEL=haiku` in
|
||
`~/.claude/settings.json` (applies to sessions launched after 2026-06-30).
|
||
- Prior-audit baseline to beat: Opus 26.6% / Sonnet 36.7% / Haiku 36.5% of subagents;
|
||
Opus subagents 53% file-edit/shell.
|
||
- Reusable scratch scripts from the first pass:
|
||
`/tmp/claude-1000/.../scratchpad/{full,cat,orch}.py` (ephemeral — recreate or
|
||
promote into the repo when running the deep audit).
|
||
- Open instruction-design questions for the pilot: where to set the complexity
|
||
threshold; whether to allow N orienting reads before delegating; how to phrase
|
||
"always pass model on spawn" given the env var now provides a cheap default.
|