cc-os/plugins/os-orchestration/skills/audit-sessions/SKILL.md

description
Run the biweekly orchestration IRL audit - deterministic transcript stats precompute, then auditor fan-out judging flagged regions against ORCHESTRATION.md rules, then a tiered tune-up report. Works from any directory. Trigger via the os-status due nudge or the Planka recurrence card.

Audit production sessions against the ORCHESTRATION.md rules

You are running the recurring (biweekly) IRL audit of the shipped delegation-economics rules. The signal source is real production session transcripts — NEVER eval grid scenarios or reserve sets (contaminated for wording tuning since 2026-07-08). This skill works from any cwd; nothing here requires being inside the cc-os repo.

Step 1 — Deterministic stats precompute (code, no judgment)

Run the driver at <skill base dir>/../../audit/bin/audit-stats (the skill base directory is <plugin root>/skills/audit-sessions, the driver lives at <plugin root>/audit/bin/audit-stats):

<plugin root>/audit/bin/audit-stats --exclude <CURRENT_SESSION_ID>

(If the current session id is unknown, omit --exclude — the 10-minute active-window guard already defers the live session to the next run.)

• --since defaults to the last ledger row's date. It refuses to run without a prior row; a first-ever run needs an explicit --since YYYY-MM-DD.
• Output: ~/.local/state/os-orchestration/audit/run-<date>/ containing stats.tsv (per-session deterministic columns), flags.md (heuristic missed-delegation regions), and sheets/*.json (full per-session fact-sheets). One row is appended to ~/.local/state/os-orchestration/audit/metrics.tsv — the trend ledger. Never edit the ledger by hand; never re-derive these numbers with a model.

Step 2 — Auditor fan-out (judgment only where code can't judge)

Read stats.tsv and flags.md from the run dir. Sessions with zero spawns AND zero flagged runs AND main_loop_share 1.0 at low turn counts usually need no auditor — include them only in the aggregate. Batch the remaining sessions into ~3 parallel auditor agents (5–8 sessions each), model: "sonnet", background, one round each.

Each auditor prompt must include:

• The rubric: read references/rubric.md (sibling to this SKILL.md) and paste it in.
• The session's fact-sheet JSON path and flagged regions — auditors judge mechanical vs judgment-dependent for each flagged region by reading the transcript selectively around the cited jsonl lines (jq on line ranges, never the raw file).
• The self-report line: "State the exact model ID you are running as in the first line of your report." Cross-check it; flag any pin mismatch in the report.
• The exact per-session report format from the rubric.

Step 3 — Synthesize and compare (main loop)

• Aggregate auditor findings into the tiered format: Tier 1 (HIGH, recurring), Tier 2 (MEDIUM), Tier 3 (solved surfaces — explicitly name what NOT to spend wording budget on).
• Compare this run's ledger row against prior rows in metrics.tsv — the previous run's Tier 1 items are hypotheses this run tests (e.g. did settled-design drift drop after the 2026-07-10 wording patch?). State each verdict explicitly.
• Write report.md into the run dir.

Step 4 — Stop for the human

Report the tiers and trend verdicts, then STOP. Do not edit ORCHESTRATION.md without explicit approval — wording changes are a human decision (injected-token budget; grid validation is unavailable). If approved later: edit ~/dev/cc-os/plugins/os-orchestration/ORCHESTRATION.md, run ~/dev/cc-os/bin/refresh-plugins, and record the change in ~/dev/cc-os/docs/implementation-status.md.

Finally, if a Planka card for this audit exists on the backlog board, move it to Review (never to Done).