cc-os/docs/plans/ws1-orchestration-audit.md

WS1 — os-orchestration session audit

Created: 2026-07-06. Status: proposed. Prereq: none. See 2026-07-06-plugin-evals-overview.md.

Goal

Audit real Claude Code sessions to assess whether delegation under the global os-orchestration plugin is working efficiently, collect concrete miss examples with triggers/criteria, and distill them into an eval-scenario backlog. This instantiates the IRL-feedback-loop methodology (production evidence first, scenarios from real misses) — NOT a lab eval yet.

The seven audit questions (fixed rubric — verbatim from the user)

1. Are subagents getting called when they should be?
2. Is the correct model chosen per subagent — highest reasonable quality at lowest cost?
3. Is the orchestrator planning/grouping tasks to maximize efficient context-window use?
4. Is the orchestrator avoiding reading files it does NOT need (that the subagent would read anyway)?
5. Is the orchestrator sharing too much context with subagents (filling their windows / clouding judgment)?
6. Is the orchestrator even following the ORCHESTRATION.md instructions?
7. Is the orchestrator requesting/receiving back only the context it needs, rather than a full context dump from the subagent?

Phase 1 — deterministic extractor (orchestrator directly; one script)

Raw transcripts are multi-MB jsonl; feeding them whole to auditors violates the principle being audited. Write one script (Python or Ruby; Ruby preferred per user style if it stays clean) that reduces a session jsonl to a delegation fact-sheet:

• Every Agent tool_use: agent type, model param present? which?, prompt char count, run_in_background, result char count returned to orchestrator.
• Orchestrator's own tool profile: Read/Grep/Glob/Bash counts and total bytes read, bucketed before/after each Agent spawn.
• Candidate missed delegations: runs of ≥N same-type tool calls across independent files with no Agent spawn (mechanical heuristic; auditor judges).
• Session metadata: project dir, date, total turns, model of main loop if recoverable.

Transcripts live under ~/.claude/projects/<flattened-path>/*.jsonl. This extractor is dual-use: it becomes the checker core for any later orchestration eval.

Phase 2 — sample selection (orchestrator, from fact-sheets not vibes)

~8–10 sessions, stratified:

• 3 × cc-os (-home-jared-dev-cc-os, 70 sessions available) — heavy orchestration use, but note contamination: these sessions were actively working ON the orchestration text.
• 3 × client/dev projects (philly-search-engine-marketing 28, llf-schema 26, viking-warrior-training-log 18).
• 2 × ops (-home-jared-servers 19, -home-jared-systems-admin 17).
• Bias recent (post-2026-07-03, when os-orchestration went global — earlier sessions ran under different/per-project orchestration text and are a different population).
• Include sessions WITH delegation and large sessions with NONE despite apparent parallelizable work.

Phase 3 — parallel audits (sonnet subagents, one per session)

Each auditor receives: the fact-sheet, the transcript path (for targeted spot-reads only — instruct them NOT to read the whole file), the ORCHESTRATION.md text (10 lines), and the seven questions. Required output per question: verdict + cited evidence (jsonl line refs) + if a miss, a candidate eval trigger/criterion ("when session state X, orchestrator should Y"). Model: sonnet (judgment work). Spawn all auditors concurrently.

Phase 4 — synthesis (orchestrator as final judge)

Verify each claimed miss against the primary transcript before accepting (standing rule — auditor reports have contradicted primary evidence before). Cluster confirmed misses into failure modes; each cluster → an eval-scenario candidate with a measurable trigger.

Deliverables:

• Findings doc: docs/orchestration-audit/2026-07-XX-findings.md (per-question summary, per-session evidence table, failure-mode clusters).
• Scenario backlog appended to the findings doc — inputs to a future eval design, NOT an eval yet.

Eval-design note (deferred, do not start in this workstream)

The Eval B headless pattern does not transfer directly: orchestration behavior is mid-session and depends on task shape, not a SessionStart trigger. Likely shape: scripted multi-file tasks run headless, scored by the Phase 1 extractor + thresholds (did it delegate? model passed? orchestrator bytes-read under budget?). Design only after Phase 4 findings exist — scenarios must come from real misses, not imagination.