cc-os/docs/orchestration-audit/auditor-reports/S4-report.md

# Orchestration Audit Report: S4 (b6b07a91)

**Session**: llf-schema, 2026-06-30, feat/harden-schema-verification  
**Model**: claude-opus-4-8 (main loop)  
**Agent spawns**: 41 total  
**Audit scope**: ORCHESTRATION.md compliance + model selection efficiency  
**Policy under audit**:
- Do single-file, ≤2-tool ops directly. Delegate only when parallelizable across independent files/subtasks, spans many files, or needs large/isolated context.
- **Every `Agent` spawn passes `model` explicitly.** Default haiku for mechanical, sonnet for judgment, opus for hard reasoning.
- A short orienting Read before delegating is fine when target paths are uncertain. Don't delegate the orienting step itself.

---

## Answer to Seven Questions

### 1. Are subagents getting called when they should be?

**Verdict: PASS**

The orchestrator appropriately delegates tasks that span multiple files, require isolated execution (git operations, test runs), or benefit from background parallelization. All spawns are for compound tasks:
- Multi-step OpenSpec task implementation (spawns 2-3, 6-7, 9-10, 14-15, 19, 33-34)
- Isolated environment testing/diagnostics (spawns 17, 20-22, 29-30, 38, 40)
- Background-capable reads (spawns 27-28: Explore agents for wide searches)
- Deterministic file updates (spawns 4-5, 8, 12, 16, 32: checkbox flips, memory writes with haiku)

No evidence of single-file/≤2-tool ops being delegated when they should have been direct.

**Evidence**: 
- Line 21: spawn 1 "Gather openspec change context" — could be viewed as optional given that `/opsx:apply` already provided structured task context, but investigating OpenSpec state is defensible as a compound prep task
- Lines 36-77: spawns 2-7 (task bundle 1.1-3.3 + mark complete) — appropriately split: 1.1 code edit (sonnet), 1.2 golden fixture + profiles (sonnet), 3.1 fixtures (sonnet), 3.2 self-check (sonnet), 3.3 doc (haiku), then mark-complete (haiku)

### 2. Is the correct model chosen per subagent — highest reasonable quality at lowest cost?

**Verdict: FAIL** (policy violation)

**Model param omissions (9 spawns violate explicit-model policy):**
- Spawns 1, 24, 25, 26, 29, 30, 31: general-purpose without model param → **default to opus** (inappropriate — should be explicitly specified as sonnet or haiku)
- Spawns 27-28: Explore agents without model param → default to haiku (acceptable for read-only search, though should still be explicit)

**Where models ARE specified, selection is sound:**
- Haiku (spawns 4, 8, 12, 16, 32, 35, 36, 39, 47): mechanical tasks (checkbox flips, schema sample-verify, post cache purge, memory writes) — cost-optimized
- Sonnet (spawns 2-3, 6-7, 9-11, 13-15, 17-23, 33-34, 37-38, 40-41): code edits, fixture design, verifier logic, multi-file diagnostics, schema verification — appropriate judgment tier
- Opus (none explicitly specified, only defaults via omission): the omissions that defaulted to opus are under-specified

**Critical violation**: Policy states "Every `Agent` spawn passes `model` explicitly." Lines 21, 411, 427, 440, 466, 494, 496, 529, 563 (spawns 1, 24-31) omit the parameter:

- **Line 21** (spawn 1, general-purpose, context gathering, 1275 prompt chars, 14627 result chars): **model field absent** from `input` dict. Should be explicit.
- **Line 411** (spawn 24, "Commit harden-schema-verification", git + task update + commit, 2766 prompt chars): **model field absent**. Resolves to opus; should be explicit (likely sonnet for judgment, not opus default).
- **Line 427** (spawn 25, "Run release.sh patch to v0.2.5", shell operation, 2126 prompt chars): **model field absent**. Opus default is overkill for script execution; should be haiku.
- **Line 440** (spawn 26, "Check live service-page WebPage.description", remote HTTP checks + diagnosis, 2311 prompt chars): **model field absent**. Resolves to opus; appropriate outcome but should be explicit.
- **Lines 466, 494** (spawns 27-28, Explore agents, read-only searches): **no model param**. Explore agents are read-only and lower-cost, so defaulting to haiku is reasonable, but policy requires explicit specification.
- **Lines 496, 529, 563** (spawns 29-31, general-purpose, 2423–5292 prompt chars): **model field absent**. Default to opus; three more instances of the pattern.

**Evidence of appropriate choices where specified**:
- Line 38 (spawn 3, sonnet for profiles/verifiers, 4457 prompt chars): Correct tier for multi-file schema design
- Line 75 (spawn 6, sonnet for golden fixture regeneration, 3684 prompt): Correct tier for oracle-based fixture design
- Line 155 (spawn 13, sonnet for URL fix + test update, 1011 prompt): Appropriate for code fix with test understanding

### 3. Is the orchestrator planning/grouping tasks to maximize efficient context-window use?

**Verdict: PASS**

The orchestrator groups tasks into logical batches that respect dependencies while allowing parallelization:

**Batch structure**:
- Pre-batch 1 (spawns 1): context gathering
- Batch 1 (spawns 2-5): tasks 1.1, 1.2, 3.1, 3.3 + mark complete (spawn 5 is deterministic checkbox-flip, unblocked after parallel executions of spawns 2-4 return)
- Batch 2 (spawns 6-8): task 1.2 regen, 3.2 self-check, mark complete
- Batch 3 (spawns 9-12): tasks 4.1, 4.2, 5.1, reconcile + mark complete
- Batch 4 (spawns 13-16): verify gate, commit, mark complete
- Batch 5 (spawns 17-23): ADR verification, docblock fix, release run, three live diagnostics
- Batch 6 (spawns 24-32): commit, release, live check, spec exploration, post_excerpt check, relax + re-release, push, memory write
- Batch 7 (spawns 33-41): schema verification + finalization

**Evidence**:
- Spawns 2-4 are independent (code, profiles, doc) and can run in parallel; spawn 5 (mark complete) only needs serial ordering after 2-4
- Spawns 27-28 (Explore agents) are parallelizable read-only searches, appropriately spawned together without blocking
- Spawns 33-36 (schema verification for PLD and SDD) are independent and parallelizable

### 4. Is the orchestrator avoiding reading files it does NOT need (that the subagent would read anyway)?

**Verdict: PASS** (with caveat)

**Orchestrator tool profile**: The fact-sheet shows **0 total tool calls** across all 41 segments (post-spawn). The orchestrator never read, wrote, or grepped any files directly — all work was delegated.

This is **not a violation** because:
1. The orchestrator is executing a structured OpenSpec change, where task definitions come from `/opsx:apply harden-schema-verification` — context is pre-loaded
2. File paths and task text are explicit in the change spec, so the orchestrator doesn't need to read files to determine what to delegate
3. Subagents have clear prompts specifying files to read; no redundant pre-delegation reads are necessary

However, a **cautionary note**: Had the task been exploratory rather than spec-driven, the zero reads might indicate missing orientation.

**Evidence**:
- Lines 1-20 (before spawn 1): OpenSpec skill loads tasks and context; main-loop model has this in cache/additionalContext
- Each spawn prompt contains explicit file paths (e.g., spawn 2: `/home/jared/dev/llf-schema/openspec/changes/harden-schema-verification/design.md` D-047)
- No agent result references "file path unclear" or "orchestrator didn't tell me where to look"

### 5. Is the orchestrator sharing too much context with subagents (filling their windows / clouding judgment)?

**Verdict: PASS**

Prompt lengths are appropriately scoped:
- Mechanical tasks (spawns 4, 8, 12, 16, 32): 436–659 chars (tight, single-purpose)
- Code edits (spawns 2, 6, 13, 18): 1011–4457 chars (design context + task instructions)
- Diagnostics (spawns 17, 20-22, 38, 40): 2247–5292 chars (problem statement + investigation steps)

**Evidence**:
- Spawn 4 (coverage checklist, 512 chars): minimal context, haiku-appropriate
- Spawn 17 (ADR verification, 3462 chars with 10703 char result): comprehensive but structured; result size justified by need to return full verification matrix
- Spawn 20 (NJ LegalService diagnosis, 3499 chars): investigation is scoped to a single problem; not bloated with unrelated context

**No instance of context pollution** (e.g., dumping 20 files of unrelated code, full deployment logs, etc.).

### 6. Is the orchestrator following the ORCHESTRATION.md instructions?

**Verdict: FAIL** (explicit policy violation)

The orchestrator **VIOLATES the primary policy requirement**: "Every `Agent` spawn passes `model` explicitly."

**Policy adherence summary**:
- ✅ Single-file/≤2-tool ops: mostly upheld (no evidence of delegating trivial tasks)
- ❌ **Explicit model parameter per spawn: VIOLATED (9 spawns)**
- ✅ Model tier selection (where specified): appropriate (haiku→mechanical, sonnet→judgment, opus reserved)
- ✅ No delegation of orientation reads: correct (context provided by OpenSpec skill)

**The 9 violations** are not hidden — they are systematic. The pattern suggests possible causes:
1. **Unintended omission**: The orchestrator (opus) generated spawns but forgot to include `model:` in 9 of them
2. **Assumption of smart defaults**: The orchestrator may have assumed that omitting `model` on general-purpose agents defaults to sonnet, but it defaults to opus instead
3. **Model-selection logic change**: If the agent-spawning code changed its behavior between when the policy was written and when this session ran, the orchestrator's output may not match the policy

**Evidence of violation**:
- Line 21: `"input": {"description": ..., "prompt": ..., "subagent_type": "general-purpose"}` — **no "model" key**
- Line 36: `"input": {"description": ..., "prompt": ..., "subagent_type": "general-purpose", "model": "sonnet"}` — correct, HAS model key (for comparison)
- Lines 411, 427, 440: same pattern (missing model key) on spawns 24-26
- Lines 466, 494: Explore agents also omit model (though haiku default is less problematic)
- Lines 496, 529, 563: spawns 29-31 continue the pattern

### 7. Is the orchestrator requesting/receiving back only the context it needs, rather than a full context dump from the subagent?

**Verdict: PASS**

Each spawn includes specific instructions for what to return:

**Examples of well-scoped return specs**:
- Spawn 2 (line 36): "Return: a brief summary of exactly what you changed (with before/after of both edits) and the absolute path"
- Spawn 4 (line 40): "Return the file location and absolute path"
- Spawn 13 (line 155): (implicit: the fix is in the file edit, no long report needed)
- Spawn 17 (line 219): "Return: a verification matrix (task / expected result / actual result / pass/fail) — one row per assertion"
- Spawn 20 (line 265): "Return: (1) the exact JSON difference … (2) the code path … (3) the classification … (4) a recommended fix direction"
- Spawn 32 (line 581): "Return: the absolute file path of the created memory file + the memory file's absolute path for indexing" (actually says "just report which files were written")

Result sizes (fact-sheet column 9) show reasonable variance:
- Small (749 chars): spawns 37-38 (issue filing, brief confirmations)
- Medium (1200–3500 chars): spawns 2-3, 6-11, 13-15 (code edits, design decisions)
- Large (5000–10700 chars): spawns 17, 20-22 (detailed diagnostics/comparisons)

**The large results are justified by the task scope.** Spawn 17 (verify ADRs, 10703 result chars) needs to return a full matrix to justify pass/fail. Spawn 20 (diagnose NJ issue, 5041 result chars) needs to return JSON diffs + root-cause analysis.

No evidence of subagents returning full logs or unfiltered dumps when a summary would do.

---

## Candidate Missed Delegations

The fact-sheet heuristic (≥4 same-tool calls, distinct targets, no spawn between) flagged: **None**.

Manual review confirms: The fact-sheet's orchestrator tool profile shows **0 tool calls across all segments**. There are no runs of repeated tool use that could have been parallelized — all work was delegated.

---

## Summary: Key Misses

| Question | Verdict | Top Miss | Line Refs |
|----------|---------|----------|-----------|
| 1. Subagent delegation appropriateness | PASS | None notable | N/A |
| 2. Model choice efficiency | FAIL | 9 spawns omit explicit `model` param, default to opus (inappropriate for some) | 21, 411, 427, 440, 466, 494, 496, 529, 563 |
| 3. Context-window grouping | PASS | Batching is logical; parallelization is efficient | N/A |
| 4. Avoid pre-delegation reads | PASS | Caveat: zero reads is appropriate here due to OpenSpec context | N/A |
| 5. Context bloat to subagents | PASS | Prompt lengths are appropriate; return specs are scoped | N/A |
| 6. Follow ORCHESTRATION.md | FAIL | **Policy violation: "Every Agent spawn passes model explicitly" — 9 spawns are missing** | 21, 411, 427, 440, 466, 494, 496, 529, 563 |
| 7. Return-context scope | PASS | Spawns request specific summaries/matrices; no dump-the-whole-log behavior | N/A |

---

## Recommended Evaluation Triggers

**For future orchestration audits**, flag these scenarios:

1. **Model parameter omission**: When `Agent` tool_use appears in transcript WITHOUT a `model` field in the `input` dict, the spawn violates explicit-model policy. Cite the line number and count occurrences.

2. **Orchestrator pre-delegation reads**: When a spawn targets a file path that the orchestrator never read first, check if the task is spec-driven (OpenSpec, explicit task list) — if yes, zero pre-reads is fine; if exploratory, it's a miss.

3. **Spawn complexity vs. model tier**: Spawns with missing model params that resolve to opus — check whether the prompt requires judgment (sonnet) or is mechanical (haiku). Mark as inefficient if wrong.

4. **Batch dependency chains**: Verify that spawn ordering respects task dependencies (e.g., "mark complete" only after preceding tasks in that batch are done). This session passes this check.

5. **Return-context bloat**: Scan spawn prompts for return specs; if a spawn has none (implicit "return everything"), or result_chars >>prompt_chars with no apparent justification, flag for review.

---

## Conclusion

The orchestrator manages **task grouping, parallelization, and scoping well**. The delegation decisions are sound — work is split appropriately between main session and subagents, and batching respects dependency ordering.

However, the **model-parameter omission on 9 spawns is a clear policy violation**. This appears to be a systematic pattern rather than one-off oversight. The spawns that omit the parameter default to opus, which is overkill for some (e.g., spawn 25: "Run release.sh" could be haiku for script execution), while under-specifying the orchestrator's intent for others (e.g., spawn 24: commit workflow with multi-step git logic, which should explicitly request sonnet).

**Immediate action**: Verify that the Agent tool's `model` parameter is always included in future orchestration sessions. This session's violation does not substantially degrade the session's outcome (the tasks completed successfully), but it represents a breach of explicit policy that should be corrected.