6.7 KiB

Raw Permalink Blame History

PRD — Deeper orchestration audit

Status: ready to run (new session) · Created: 2026-06-30 Predecessor: docs/cc-tool-audit.md (§ Addendum — orchestration audit) Note: No issue tracker/remote on this repo, so this PRD lives as a doc. The /to-prd "publish + ready-for-agent label" step was adapted to a file.

Problem Statement

Sessions feel slow and over-spend on model cost. A prior audit (docs/cc-tool-audit.md) traced this less to loaded tools/skills/hooks (all small) and more to the orchestrator-subagent delegation pattern. A first-pass audit of 1,962 subagent transcripts found that 27% of subagents run on Opus, the majority of them doing mechanical file-edit/shell work that Haiku/Sonnet would handle faster and cheaper. The CLAUDE.md "Session Orchestration" routing table is advisory — when an Agent spawn omits model, the subagent inherits the parent (Opus). The rule also mandates delegation with no minimum complexity threshold, so trivial one-line edits pay a full subagent prefill + sequential wait.

An interim fix is already applied (CLAUDE_CODE_SUBAGENT_MODEL=haiku in global settings). What's missing is the quantified evidence needed to safely revise the orchestration instructions across many CLAUDE.md files — not just the default model.

Solution

A deeper, repeatable audit of the existing session transcripts that produces the numbers to ground a pilot revision of the Session Orchestration rules on one project, before any broad rollout. The audit answers: how often is the wrong model chosen, is delegation itself wasteful for small tasks, and is work being decomposed well before hand-off. Output is a short findings report + a proposed pilot rule diff.

User Stories

As the operator, I want the subagent model distribution broken down per project, so that I can see which repos leak Opus most.
As the operator, I want the "file-edit/shell" subagent bucket split into trivial (1–2 calls, single file) vs substantial (multi-file, many calls), so that I can tell genuine Sonnet/Opus work from Haiku-appropriate work instead of relying on a coarse bucket.
As the operator, I want each subagent's spawn traced to whether its model was explicitly passed by the orchestrator or inherited from the parent, so that I can measure the routing-leak rate directly.
As the operator, I want a count of subagents that did near-zero work (0–1 tool calls, or returned trivially), so that I can quantify wasted delegations caused by the "no threshold" rule.
As the operator, I want to see how many orchestrator turns spawned a single subagent for a single small op (delegation that a direct op would beat on latency), so that I can justify a complexity threshold.
As the operator, I want parallelism measured — how often independent subagents were batched concurrently vs. spawned sequentially — so that I can see if decomposition exploits the parallel-batch guidance.
As the operator, I want examples of good and bad decomposition (a clean parallel batch vs. a needless single-op delegation), so that the pilot rule change is illustrated, not just asserted.
As the operator, I want the audit limited to delegation-heavy projects (cc-os, llf-schema, proxmox, philly-seo, ovh-prod), so that it runs on the relevant sample, not every repo.
As the operator, I want the interim CLAUDE_CODE_SUBAGENT_MODEL=haiku effect measurable in future sessions (a before/after marker), so that I can confirm the env var actually shifted the distribution.
As the operator, I want a proposed pilot diff to one project's "Session Orchestration" section (threshold + enforced routing + allow orienting reads), so that I can try it on a single repo and measure wall-clock/cost before rolling out.
As the operator, I want the audit scripted/repeatable (saved under the repo), so that I can re-run it after the pilot to compare.

Implementation Decisions

Data source: existing JSONL transcripts in ~/.claude/projects/*/ (main) and ~/.claude/projects/*/<session-id>/subagents/agent-*.jsonl (subagents). Model is in each subagent message's model field; tool calls are type:"tool_use" blocks (walk recursively). Project dir names start with - — use python glob, never shell find/ls.
Trivial vs substantial classifier: trivial = ≤2 tool calls and ≤1 distinct file touched; substantial = otherwise. Tune threshold against spot-checked examples.
Inherited vs explicit model trace: match parent Agent tool_use input (does it carry a model/subagent_type?) to the child transcript via agent id, then compare requested vs actual model. Where the parent omitted model and child==parent model, classify as "inherited (leak)."
Parallelism metric: within a main session, detect Agent tool_use blocks issued in the same assistant turn (concurrent) vs separate turns (sequential).
Scope: the 5 delegation-heavy projects above; report per-project and pooled.
Output: append a "Deep audit" section to docs/cc-tool-audit.md (or a sibling doc) with tables + 3–5 examples, plus a proposed pilot rule diff for ONE project.
No production change beyond the already-applied env var until the pilot is reviewed.

Testing Decisions

This is an analysis task, not a shipped feature — "tests" = sanity checks on the parser: total subagent count matches the prior audit (~1,962 with a model), model totals reconcile, and the trivial/substantial split sums to the file-edit bucket.
Spot-verify the classifier on 5 hand-read transcripts (one per project) before trusting the aggregate.
Re-run the same script post-pilot to confirm the distribution shifted (regression baseline = today's numbers).

Out of Scope

Editing every CLAUDE.md (this is assessment → single-project pilot only).
Changing tool/skill/hook config (prior audit showed negligible payoff).
Building any new tooling, MCP, or harness changes.
Touching the live orchestration behavior beyond the env var already set.

Further Notes

Interim change already live: env.CLAUDE_CODE_SUBAGENT_MODEL=haiku in ~/.claude/settings.json (applies to sessions launched after 2026-06-30).
Prior-audit baseline to beat: Opus 26.6% / Sonnet 36.7% / Haiku 36.5% of subagents; Opus subagents 53% file-edit/shell.
Reusable scratch scripts from the first pass: /tmp/claude-1000/.../scratchpad/{full,cat,orch}.py (ephemeral — recreate or promote into the repo when running the deep audit).
Open instruction-design questions for the pilot: where to set the complexity threshold; whether to allow N orienting reads before delegating; how to phrase "always pass model on spawn" given the env var now provides a cheap default.

6.7 KiB Raw Permalink Blame History Unescape Escape