154 lines
10 KiB
Markdown
154 lines
10 KiB
Markdown
# os-orchestration session audit — findings
|
||
|
||
_Audit date: 2026-07-06. Status: complete (WS1 of `docs/plans/ws1-orchestration-audit.md`)._
|
||
_Method: Phase 1 extractor (`plugins/os-orchestration/audit/bin/extract`) → 10 stratified
|
||
sessions → 10 parallel sonnet auditors (7-question rubric) → orchestrator-verified synthesis.
|
||
Every load-bearing claim below was re-verified against the primary transcript; auditor claims
|
||
that failed or only partially survived verification are marked._
|
||
|
||
## Sample
|
||
|
||
| ID | Project | Date | Population | Turns | Spawns | Main-loop model |
|
||
|----|---------|------|------------|-------|--------|-----------------|
|
||
| S1 | cc-os | 2026-07-06 | post-rollout (contaminated¹) | 66 | 10 | fable-5 |
|
||
| S2 | cc-os | 2026-07-06 | post-rollout (contaminated¹) | 207 | 8 | fable-5 |
|
||
| S3 | cc-os | 2026-07-04 | post-rollout (contaminated¹) | 255 | 5 | fable-5 |
|
||
| S4 | llf-schema | 2026-06-30 | pre-rollout | 191 | 41 | opus-4-8 |
|
||
| S5 | llf-schema | 2026-06-29 | pre-rollout | 36 | 8 | sonnet-4-6 |
|
||
| S6 | philly-sem | 2026-06-29 | pre-rollout | 21 | 1 | sonnet-4-6 |
|
||
| S7 | servers | 2026-07-04 | post-rollout | 144 | 3 | sonnet-5 |
|
||
| S8 | servers | 2026-07-04 | post-rollout | 90 | 0 | sonnet-5 |
|
||
| S9 | systems-admin | 2026-06-30 | pre-rollout | 117 | 3 | opus-4-8 |
|
||
| S10 | systems-admin | 2026-07-01 | pre-rollout | 53 | 0 | sonnet-5 |
|
||
|
||
¹ cc-os sessions were actively building eval/orchestration tooling; delegation-awareness in
|
||
them overstates the general population.
|
||
|
||
Transcripts: `~/.claude/projects/<dir>/<uuid>.jsonl`; fact-sheets and full per-session auditor
|
||
reports were session artifacts (scratchpad); regenerate fact-sheets any time with the extractor.
|
||
|
||
## Per-question summary (across 10 sessions)
|
||
|
||
1. **Subagents called when they should be?** Mostly yes. 8/10 PASS or justified-no-delegation
|
||
(S8, S10 correctly kept sequential/coordinated work in-session — both heuristic
|
||
missed-delegation flags there were judged false positives). Genuine gray zone: S3 and S7
|
||
did heavy in-session investigation (19–37 calls, 74KB) that could have been one
|
||
"investigate and report" delegation.
|
||
2. **Correct model per subagent?** The dominant failure area, in two distinct modes (see
|
||
clusters 1 and 2). Explicit-tier choices, when the orchestrator made them, were sensible
|
||
(haiku for mechanical, sonnet for judgment).
|
||
3. **Planning/grouping for context efficiency?** Good. Concurrent fan-out used in S1, S2, S4,
|
||
S5, S7, S9; serial dependencies respected. Minor: reactive late spawns (S1 agents 8–10)
|
||
instead of pre-planned batches.
|
||
4. **Avoiding unnecessary orchestrator reads?** Mostly PASS (S4: zero orchestrator bytes over
|
||
41 spawns; S1, S5, S6 near-zero). Exceptions: S7 (74KB pre-spawn diagnostics, then Explore
|
||
agents told to re-read files "already read" — partially verified²), S9 (read
|
||
`settings.json`, then spawn #1 instructed to read it again — confirmed).
|
||
5. **Over-sharing context with subagents?** PASS everywhere. Prompts 0.4–6.7KB,
|
||
instruction-forward; no transcript/CLAUDE.md dumps observed in any session.
|
||
6. **Following ORCHESTRATION.md?** Split by population: post-rollout cc-os sessions pass the
|
||
explicit-model rule 23/23 spawns; pre-rollout and ops sessions omit `model` on 16/56 spawns
|
||
(S4×9, S5×6, S6×1, S7×3, S9×3 — counting per verified transcript data, 22 omissions total
|
||
including Explore-type spawns). Everything else (delegation thresholds, orienting reads)
|
||
broadly followed.
|
||
7. **Receiving only needed context back?** PASS everywhere. Agent results 0.9–13KB summaries;
|
||
no full-dump pattern found.
|
||
|
||
² Confirmed for one file (`incidents.md`, S7 line 182 → spawn-2 prompt) and for the prompt
|
||
text "CLAUDE.md (already read, but note key conventions)"; the auditor's other named files
|
||
could not be confirmed as orchestrator Reads (may have been Bash cats) — treated as plausible.
|
||
|
||
## Failure-mode clusters (verified)
|
||
|
||
### Cluster 1 — model param silently overridden to haiku in Fable-5 sessions (HIGH, environment)
|
||
|
||
Verified across S1/S2/S3 (23 spawns): requested `sonnet` (18), `opus` (1), `haiku` (4) — every
|
||
one resolved `claude-haiku-4-5-20251001`. In S4/S5 (opus/sonnet main loops) explicit params
|
||
were honored (sonnet→sonnet-4-6, haiku→haiku-4-5). So under a Fable-5 main loop the `model`
|
||
param is currently a no-op and all subagent work runs at haiku tier, including tasks the
|
||
orchestrator explicitly graded as judgment work (e.g. S3 line 603: "answer from judgment
|
||
alone" → haiku). The orchestrator never detected or escalated the downgrade — nothing in the
|
||
policy tells it to check `resolvedModel`.
|
||
|
||
- **Not an orchestrator wording failure** — compliance with the written rule was perfect in
|
||
these sessions. It is an ops/environment issue plus a policy gap (no verify-resolution rule).
|
||
- **ROOT CAUSE FOUND (2026-07-06, post-audit):** `CLAUDE_CODE_SUBAGENT_MODEL=haiku` was set in
|
||
`~/.claude/settings.json`'s `env` block (added by an earlier AI session as a cost measure).
|
||
That env var force-overrides every subagent's `model` param. The "Fable-5 main loop"
|
||
correlation was coincidental timing — the var was added after the pre-rollout opus/sonnet
|
||
sessions (S4/S5, 06-29/30, params honored) and before the Fable-5 sessions. **Removed
|
||
2026-07-06.** E1 remains valid as a detect-the-downgrade eval; the policy gap (no
|
||
verify-`resolvedModel` rule) still stands regardless of cause.
|
||
|
||
### Cluster 2 — omitted `model` param inherits the main-loop model (HIGH, orchestrator)
|
||
|
||
Verified: omission resolves to the parent session's model — opus-4-8 in S4 (×7) and S9 (×2),
|
||
sonnet in S5 (×6) and S6 (×1) — except agent types with their own default (Explore→haiku,
|
||
S7 ×3, S4 ×2, S9 ×1). Cost consequence is real: S6's single spawn was a mechanical file-edit
|
||
task billed at sonnet; S9 ran a character-counting task at opus. All these sessions are
|
||
pre-rollout or ops sessions where the global plugin text either wasn't present or wasn't
|
||
followed on this rule. Post-rollout cc-os sessions (S1–S3): zero omissions.
|
||
|
||
- The rule exists and works where the plugin text is in force and salient; the misses cluster
|
||
in the population that ran without it (pre-07-03) or in ops projects (S7 07-04, post-rollout,
|
||
still omitted on all 3 spawns — the one confirmed post-rollout violation).
|
||
- Action candidates: mechanical trigger-phrased wording (the Eval B lesson — "before every
|
||
Agent call → include `model:`"); eval scenario E2.
|
||
|
||
### Cluster 3 — orchestrator self-investigates then delegates the same ground (MEDIUM)
|
||
|
||
S7: 32 Bash + 4 Reads (74KB) of diagnostics before spawning 3 Explore agents whose prompts
|
||
overlap that ground (one confirmed file-level dual-read; prompt text acknowledges "already
|
||
read"). S9: read `settings.json` then instructed spawn #1 to read it (confirmed). S3: 19-call
|
||
investigate-then-fix run kept in-session where one delegation could have isolated it.
|
||
|
||
- Boundary is genuinely fuzzy — the policy allows "a short orienting Read"; these exceed
|
||
"short" but the work partly informed direct action too. Judged real but medium-severity.
|
||
- Action candidate: eval scenario E3 (bytes-read-before-first-spawn budget).
|
||
|
||
### Cluster 4 — reactive late spawns instead of planned batches (LOW)
|
||
|
||
S1 (agents 8–10 spawned only after synthesis of 4–7), S3 (91 post-eval tool calls that could
|
||
have been folded into the eval spawns' own prompts). Low priority: wall-clock cost only,
|
||
no correctness or spend impact demonstrated.
|
||
|
||
### Non-findings worth keeping
|
||
|
||
- Zero cases of context over-sharing to subagents; zero cases of full-dump returns (Q5/Q7
|
||
clean across all 10 — the async task-notification pattern returns summaries by design).
|
||
- The missed-delegation heuristic (≥4 same-tool runs) produced 4 flags in the sample;
|
||
auditors + verification judged all of them false positives (coordinated multi-file units or
|
||
sequential-dependent work). Threshold/shape needs tuning before it can score an eval.
|
||
|
||
## Eval-scenario backlog (inputs to a future eval design — NOT an eval yet)
|
||
|
||
- **E1 (from cluster 1):** Scripted task forces a judgment-grade delegation; harness stubs
|
||
resolution to haiku. Criterion: orchestrator notices `resolvedModel` mismatch in the launch
|
||
stub and says so / adapts. Checker: extractor already captures requested vs resolved.
|
||
- **E2 (from cluster 2):** Multi-file parallelizable task in a non-cc-os fixture. Criterion:
|
||
every Agent spawn includes `model:`; mechanical work→haiku, judgment→sonnet. Checker:
|
||
extractor `model_explicit` field — fully mechanical. Paired negative: single-file ≤2-call
|
||
task where spawning anything is the failure.
|
||
- **E3 (from cluster 3):** Task whose natural first move is delegatable investigation (log
|
||
review across many files). Criterion: orchestrator bytes-read before first spawn under a
|
||
threshold (extractor segment `pre-spawn-1`), no file both Read by orchestrator and assigned
|
||
to a subagent prompt. Paired negative: task where orienting reads are correct (uncertain
|
||
target path).
|
||
- **E4 (from cluster 4, deferred):** Batch-planning quality — hard to score mechanically;
|
||
design only if clusters 1–3 get fixed and this remains visible.
|
||
|
||
Per the plan's eval-design note: these run as scripted multi-file headless tasks scored by the
|
||
Phase 1 extractor + thresholds; mid-session behavior, so the Eval B SessionStart pattern does
|
||
not transfer. Design gate: follow `~/Documents/SecondBrain/eval-methodology-ladder.md`
|
||
(paired positives/negatives, frozen reserve) when authoring.
|
||
|
||
## Caveats
|
||
|
||
- Sample: 10 sessions, single audit pass; cc-os sessions contaminated (delegation-aware work);
|
||
only 2 truly post-rollout non-cc-os sessions (S7, S8) — the population the plugin targets is
|
||
thin in this sample. Re-audit in ~2 weeks per the IRL-feedback-loop cadence.
|
||
- Auditor reliability: several auditor claims did not survive verification as stated
|
||
(S4 "defaults to opus" was right for general-purpose but auditors variously mis-attributed
|
||
the mechanism; S7's dual-read file list only partially confirmed). All cluster claims above
|
||
reflect the verified transcript data, not the raw auditor reports.
|