10 KiB

Raw Blame History

os-orchestration session audit — findings

Audit date: 2026-07-06. Status: complete (WS1 of docs/plans/ws1-orchestration-audit.md). Method: Phase 1 extractor (plugins/os-orchestration/audit/bin/extract) → 10 stratified sessions → 10 parallel sonnet auditors (7-question rubric) → orchestrator-verified synthesis. Every load-bearing claim below was re-verified against the primary transcript; auditor claims that failed or only partially survived verification are marked.

Sample

ID	Project	Date	Population	Turns	Spawns	Main-loop model
S1	cc-os	2026-07-06	post-rollout (contaminated¹)	66	10	fable-5
S2	cc-os	2026-07-06	post-rollout (contaminated¹)	207	8	fable-5
S3	cc-os	2026-07-04	post-rollout (contaminated¹)	255	5	fable-5
S4	llf-schema	2026-06-30	pre-rollout	191	41	opus-4-8
S5	llf-schema	2026-06-29	pre-rollout	36	8	sonnet-4-6
S6	philly-sem	2026-06-29	pre-rollout	21	1	sonnet-4-6
S7	servers	2026-07-04	post-rollout	144	3	sonnet-5
S8	servers	2026-07-04	post-rollout	90	0	sonnet-5
S9	systems-admin	2026-06-30	pre-rollout	117	3	opus-4-8
S10	systems-admin	2026-07-01	pre-rollout	53	0	sonnet-5

¹ cc-os sessions were actively building eval/orchestration tooling; delegation-awareness in them overstates the general population.

Transcripts: ~/.claude/projects/<dir>/<uuid>.jsonl; fact-sheets and full per-session auditor reports were session artifacts (scratchpad); regenerate fact-sheets any time with the extractor.

Per-question summary (across 10 sessions)

Subagents called when they should be? Mostly yes. 8/10 PASS or justified-no-delegation (S8, S10 correctly kept sequential/coordinated work in-session — both heuristic missed-delegation flags there were judged false positives). Genuine gray zone: S3 and S7 did heavy in-session investigation (19–37 calls, 74KB) that could have been one "investigate and report" delegation.
Correct model per subagent? The dominant failure area, in two distinct modes (see clusters 1 and 2). Explicit-tier choices, when the orchestrator made them, were sensible (haiku for mechanical, sonnet for judgment).
Planning/grouping for context efficiency? Good. Concurrent fan-out used in S1, S2, S4, S5, S7, S9; serial dependencies respected. Minor: reactive late spawns (S1 agents 8–10) instead of pre-planned batches.
Avoiding unnecessary orchestrator reads? Mostly PASS (S4: zero orchestrator bytes over 41 spawns; S1, S5, S6 near-zero). Exceptions: S7 (74KB pre-spawn diagnostics, then Explore agents told to re-read files "already read" — partially verified²), S9 (read settings.json, then spawn #1 instructed to read it again — confirmed).
Over-sharing context with subagents? PASS everywhere. Prompts 0.4–6.7KB, instruction-forward; no transcript/CLAUDE.md dumps observed in any session.
Following ORCHESTRATION.md? Split by population: post-rollout cc-os sessions pass the explicit-model rule 23/23 spawns; pre-rollout and ops sessions omit model on 16/56 spawns (S4×9, S5×6, S6×1, S7×3, S9×3 — counting per verified transcript data, 22 omissions total including Explore-type spawns). Everything else (delegation thresholds, orienting reads) broadly followed.
Receiving only needed context back? PASS everywhere. Agent results 0.9–13KB summaries; no full-dump pattern found.

² Confirmed for one file (incidents.md, S7 line 182 → spawn-2 prompt) and for the prompt text "CLAUDE.md (already read, but note key conventions)"; the auditor's other named files could not be confirmed as orchestrator Reads (may have been Bash cats) — treated as plausible.

Failure-mode clusters (verified)

Cluster 1 — model param silently overridden to haiku in Fable-5 sessions (HIGH, environment)

Verified across S1/S2/S3 (23 spawns): requested sonnet (18), opus (1), haiku (4) — every one resolved claude-haiku-4-5-20251001. In S4/S5 (opus/sonnet main loops) explicit params were honored (sonnet→sonnet-4-6, haiku→haiku-4-5). So under a Fable-5 main loop the model param is currently a no-op and all subagent work runs at haiku tier, including tasks the orchestrator explicitly graded as judgment work (e.g. S3 line 603: "answer from judgment alone" → haiku). The orchestrator never detected or escalated the downgrade — nothing in the policy tells it to check resolvedModel.

Not an orchestrator wording failure — compliance with the written rule was perfect in these sessions. It is an ops/environment issue plus a policy gap (no verify-resolution rule).
ROOT CAUSE FOUND (2026-07-06, post-audit): CLAUDE_CODE_SUBAGENT_MODEL=haiku was set in ~/.claude/settings.json's env block (added by an earlier AI session as a cost measure). That env var force-overrides every subagent's model param. The "Fable-5 main loop" correlation was coincidental timing — the var was added after the pre-rollout opus/sonnet sessions (S4/S5, 06-29/30, params honored) and before the Fable-5 sessions. Removed 2026-07-06. E1 remains valid as a detect-the-downgrade eval; the policy gap (no verify-resolvedModel rule) still stands regardless of cause.

Cluster 2 — omitted `model` param inherits the main-loop model (HIGH, orchestrator)

Verified: omission resolves to the parent session's model — opus-4-8 in S4 (×7) and S9 (×2), sonnet in S5 (×6) and S6 (×1) — except agent types with their own default (Explore→haiku, S7 ×3, S4 ×2, S9 ×1). Cost consequence is real: S6's single spawn was a mechanical file-edit task billed at sonnet; S9 ran a character-counting task at opus. All these sessions are pre-rollout or ops sessions where the global plugin text either wasn't present or wasn't followed on this rule. Post-rollout cc-os sessions (S1–S3): zero omissions.

The rule exists and works where the plugin text is in force and salient; the misses cluster in the population that ran without it (pre-07-03) or in ops projects (S7 07-04, post-rollout, still omitted on all 3 spawns — the one confirmed post-rollout violation).
Action candidates: mechanical trigger-phrased wording (the Eval B lesson — "before every Agent call → include model:"); eval scenario E2.

Cluster 3 — orchestrator self-investigates then delegates the same ground (MEDIUM)

S7: 32 Bash + 4 Reads (74KB) of diagnostics before spawning 3 Explore agents whose prompts overlap that ground (one confirmed file-level dual-read; prompt text acknowledges "already read"). S9: read settings.json then instructed spawn #1 to read it (confirmed). S3: 19-call investigate-then-fix run kept in-session where one delegation could have isolated it.

Boundary is genuinely fuzzy — the policy allows "a short orienting Read"; these exceed "short" but the work partly informed direct action too. Judged real but medium-severity.
Action candidate: eval scenario E3 (bytes-read-before-first-spawn budget).

Cluster 4 — reactive late spawns instead of planned batches (LOW)

S1 (agents 8–10 spawned only after synthesis of 4–7), S3 (91 post-eval tool calls that could have been folded into the eval spawns' own prompts). Low priority: wall-clock cost only, no correctness or spend impact demonstrated.

Non-findings worth keeping

Zero cases of context over-sharing to subagents; zero cases of full-dump returns (Q5/Q7 clean across all 10 — the async task-notification pattern returns summaries by design).
The missed-delegation heuristic (≥4 same-tool runs) produced 4 flags in the sample; auditors + verification judged all of them false positives (coordinated multi-file units or sequential-dependent work). Threshold/shape needs tuning before it can score an eval.

Eval-scenario backlog (inputs to a future eval design — NOT an eval yet)

E1 (from cluster 1): Scripted task forces a judgment-grade delegation; harness stubs resolution to haiku. Criterion: orchestrator notices resolvedModel mismatch in the launch stub and says so / adapts. Checker: extractor already captures requested vs resolved.
E2 (from cluster 2): Multi-file parallelizable task in a non-cc-os fixture. Criterion: every Agent spawn includes model:; mechanical work→haiku, judgment→sonnet. Checker: extractor model_explicit field — fully mechanical. Paired negative: single-file ≤2-call task where spawning anything is the failure.
E3 (from cluster 3): Task whose natural first move is delegatable investigation (log review across many files). Criterion: orchestrator bytes-read before first spawn under a threshold (extractor segment pre-spawn-1), no file both Read by orchestrator and assigned to a subagent prompt. Paired negative: task where orienting reads are correct (uncertain target path).
E4 (from cluster 4, deferred): Batch-planning quality — hard to score mechanically; design only if clusters 1–3 get fixed and this remains visible.

Per the plan's eval-design note: these run as scripted multi-file headless tasks scored by the Phase 1 extractor + thresholds; mid-session behavior, so the Eval B SessionStart pattern does not transfer. Design gate: follow ~/Documents/SecondBrain/eval-methodology-ladder.md (paired positives/negatives, frozen reserve) when authoring.

Caveats

Sample: 10 sessions, single audit pass; cc-os sessions contaminated (delegation-aware work); only 2 truly post-rollout non-cc-os sessions (S7, S8) — the population the plugin targets is thin in this sample. Re-audit in ~2 weeks per the IRL-feedback-loop cadence.
Auditor reliability: several auditor claims did not survive verification as stated (S4 "defaults to opus" was right for general-purpose but auditors variously mis-attributed the mechanism; S7's dual-read file list only partially confirmed). All cluster claims above reflect the verified transcript data, not the raw auditor reports.

10 KiB Raw Blame History Unescape Escape