# WS1 — os-orchestration session audit

_Created: 2026-07-06. Status: **complete 2026-07-06** — findings + scenario backlog in
`docs/orchestration-audit/2026-07-06-findings.md`; extractor shipped at
`plugins/os-orchestration/audit/bin/extract`. See `2026-07-06-plugin-evals-overview.md`._

## Goal

Audit real Claude Code sessions to assess whether delegation under the global
`os-orchestration` plugin is working efficiently, collect concrete miss examples with
triggers/criteria, and distill them into an eval-scenario backlog. This instantiates the
IRL-feedback-loop methodology (production evidence first, scenarios from real misses) —
NOT a lab eval yet.

## The seven audit questions (fixed rubric — verbatim from the user)

1. Are subagents getting called when they should be?
2. Is the correct model chosen per subagent — highest reasonable quality at lowest cost?
3. Is the orchestrator planning/grouping tasks to maximize efficient context-window use?
4. Is the orchestrator avoiding reading files it does NOT need (that the subagent would
   read anyway)?
5. Is the orchestrator sharing too much context with subagents (filling their windows /
   clouding judgment)?
6. Is the orchestrator even following the ORCHESTRATION.md instructions?
7. Is the orchestrator requesting/receiving back only the context it needs, rather than a
   full context dump from the subagent?

## Phase 1 — deterministic extractor (orchestrator directly; one script)

Raw transcripts are multi-MB jsonl; feeding them whole to auditors violates the principle
being audited. Write one script (Python or Ruby; Ruby preferred per user style if it stays
clean) that reduces a session jsonl to a delegation fact-sheet:

- Every `Agent` tool_use: agent type, `model` param present? which?, prompt char count,
  run_in_background, result char count returned to orchestrator.
- Orchestrator's own tool profile: Read/Grep/Glob/Bash counts and total bytes read,
  bucketed before/after each Agent spawn.
- Candidate missed delegations: runs of ≥N same-type tool calls across independent files
  with no Agent spawn (mechanical heuristic; auditor judges).
- Session metadata: project dir, date, total turns, model of main loop if recoverable.

Transcripts live under `~/.claude/projects/<flattened-path>/*.jsonl`.
This extractor is dual-use: it becomes the checker core for any later orchestration eval.

## Phase 2 — sample selection (orchestrator, from fact-sheets not vibes)

~8–10 sessions, stratified:
- 3 × cc-os (`-home-jared-dev-cc-os`, 70 sessions available) — heavy orchestration use,
  but note contamination: these sessions were actively working ON the orchestration text.
- 3 × client/dev projects (`philly-search-engine-marketing` 28, `llf-schema` 26,
  `viking-warrior-training-log` 18).
- 2 × ops (`-home-jared-servers` 19, `-home-jared-systems-admin` 17).
- Bias recent (post-2026-07-03, when os-orchestration went global — earlier sessions ran
  under different/per-project orchestration text and are a different population).
- Include sessions WITH delegation and large sessions with NONE despite apparent
  parallelizable work.

## Phase 3 — parallel audits (sonnet subagents, one per session)

Each auditor receives: the fact-sheet, the transcript path (for targeted spot-reads only —
instruct them NOT to read the whole file), the ORCHESTRATION.md text (10 lines), and the
seven questions. Required output per question: verdict + cited evidence (jsonl line refs) +
if a miss, a candidate eval trigger/criterion ("when session state X, orchestrator should
Y"). Model: sonnet (judgment work). Spawn all auditors concurrently.

## Phase 4 — synthesis (orchestrator as final judge)

Verify each claimed miss against the primary transcript before accepting (standing rule —
auditor reports have contradicted primary evidence before). Cluster confirmed misses into
failure modes; each cluster → an eval-scenario candidate with a measurable trigger.

Deliverables:
- Findings doc: `docs/orchestration-audit/2026-07-XX-findings.md` (per-question summary,
  per-session evidence table, failure-mode clusters).
- Scenario backlog appended to the findings doc — inputs to a future eval design, NOT an
  eval yet.

## Eval-design note (deferred, do not start in this workstream)

The Eval B headless pattern does not transfer directly: orchestration behavior is
mid-session and depends on task shape, not a SessionStart trigger. Likely shape: scripted
multi-file tasks run headless, scored by the Phase 1 extractor + thresholds (did it
delegate? model passed? orchestrator bytes-read under budget?). Design only after Phase 4
findings exist — scenarios must come from real misses, not imagination.
