83 lines
4.5 KiB
Markdown
83 lines
4.5 KiB
Markdown
# WS1 — os-orchestration session audit
|
||
|
||
_Created: 2026-07-06. Status: proposed. Prereq: none. See `2026-07-06-plugin-evals-overview.md`._
|
||
|
||
## Goal
|
||
|
||
Audit real Claude Code sessions to assess whether delegation under the global
|
||
`os-orchestration` plugin is working efficiently, collect concrete miss examples with
|
||
triggers/criteria, and distill them into an eval-scenario backlog. This instantiates the
|
||
IRL-feedback-loop methodology (production evidence first, scenarios from real misses) —
|
||
NOT a lab eval yet.
|
||
|
||
## The seven audit questions (fixed rubric — verbatim from the user)
|
||
|
||
1. Are subagents getting called when they should be?
|
||
2. Is the correct model chosen per subagent — highest reasonable quality at lowest cost?
|
||
3. Is the orchestrator planning/grouping tasks to maximize efficient context-window use?
|
||
4. Is the orchestrator avoiding reading files it does NOT need (that the subagent would
|
||
read anyway)?
|
||
5. Is the orchestrator sharing too much context with subagents (filling their windows /
|
||
clouding judgment)?
|
||
6. Is the orchestrator even following the ORCHESTRATION.md instructions?
|
||
7. Is the orchestrator requesting/receiving back only the context it needs, rather than a
|
||
full context dump from the subagent?
|
||
|
||
## Phase 1 — deterministic extractor (orchestrator directly; one script)
|
||
|
||
Raw transcripts are multi-MB jsonl; feeding them whole to auditors violates the principle
|
||
being audited. Write one script (Python or Ruby; Ruby preferred per user style if it stays
|
||
clean) that reduces a session jsonl to a delegation fact-sheet:
|
||
|
||
- Every `Agent` tool_use: agent type, `model` param present? which?, prompt char count,
|
||
run_in_background, result char count returned to orchestrator.
|
||
- Orchestrator's own tool profile: Read/Grep/Glob/Bash counts and total bytes read,
|
||
bucketed before/after each Agent spawn.
|
||
- Candidate missed delegations: runs of ≥N same-type tool calls across independent files
|
||
with no Agent spawn (mechanical heuristic; auditor judges).
|
||
- Session metadata: project dir, date, total turns, model of main loop if recoverable.
|
||
|
||
Transcripts live under `~/.claude/projects/<flattened-path>/*.jsonl`.
|
||
This extractor is dual-use: it becomes the checker core for any later orchestration eval.
|
||
|
||
## Phase 2 — sample selection (orchestrator, from fact-sheets not vibes)
|
||
|
||
~8–10 sessions, stratified:
|
||
- 3 × cc-os (`-home-jared-dev-cc-os`, 70 sessions available) — heavy orchestration use,
|
||
but note contamination: these sessions were actively working ON the orchestration text.
|
||
- 3 × client/dev projects (`philly-search-engine-marketing` 28, `llf-schema` 26,
|
||
`viking-warrior-training-log` 18).
|
||
- 2 × ops (`-home-jared-servers` 19, `-home-jared-systems-admin` 17).
|
||
- Bias recent (post-2026-07-03, when os-orchestration went global — earlier sessions ran
|
||
under different/per-project orchestration text and are a different population).
|
||
- Include sessions WITH delegation and large sessions with NONE despite apparent
|
||
parallelizable work.
|
||
|
||
## Phase 3 — parallel audits (sonnet subagents, one per session)
|
||
|
||
Each auditor receives: the fact-sheet, the transcript path (for targeted spot-reads only —
|
||
instruct them NOT to read the whole file), the ORCHESTRATION.md text (10 lines), and the
|
||
seven questions. Required output per question: verdict + cited evidence (jsonl line refs) +
|
||
if a miss, a candidate eval trigger/criterion ("when session state X, orchestrator should
|
||
Y"). Model: sonnet (judgment work). Spawn all auditors concurrently.
|
||
|
||
## Phase 4 — synthesis (orchestrator as final judge)
|
||
|
||
Verify each claimed miss against the primary transcript before accepting (standing rule —
|
||
auditor reports have contradicted primary evidence before). Cluster confirmed misses into
|
||
failure modes; each cluster → an eval-scenario candidate with a measurable trigger.
|
||
|
||
Deliverables:
|
||
- Findings doc: `docs/orchestration-audit/2026-07-XX-findings.md` (per-question summary,
|
||
per-session evidence table, failure-mode clusters).
|
||
- Scenario backlog appended to the findings doc — inputs to a future eval design, NOT an
|
||
eval yet.
|
||
|
||
## Eval-design note (deferred, do not start in this workstream)
|
||
|
||
The Eval B headless pattern does not transfer directly: orchestration behavior is
|
||
mid-session and depends on task shape, not a SessionStart trigger. Likely shape: scripted
|
||
multi-file tasks run headless, scored by the Phase 1 extractor + thresholds (did it
|
||
delegate? model passed? orchestrator bytes-read under budget?). Design only after Phase 4
|
||
findings exist — scenarios must come from real misses, not imagination.
|