69 lines
3.7 KiB
Markdown
69 lines
3.7 KiB
Markdown
|
---
|
|||
|
|
description: Run the biweekly orchestration IRL audit - deterministic transcript stats precompute, then auditor fan-out judging flagged regions against ORCHESTRATION.md rules, then a tiered tune-up report. Works from any directory. Trigger via the os-status due nudge or the Planka recurrence card.
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# Audit production sessions against the ORCHESTRATION.md rules
|
|||
|
|
|
|||
|
|
You are running the recurring (biweekly) IRL audit of the shipped delegation-economics
|
|||
|
|
rules. The signal source is real production session transcripts — NEVER eval grid
|
|||
|
|
scenarios or reserve sets (contaminated for wording tuning since 2026-07-08). This skill
|
|||
|
|
works from any cwd; nothing here requires being inside the cc-os repo.
|
|||
|
|
|
|||
|
|
## Step 1 — Deterministic stats precompute (code, no judgment)
|
|||
|
|
|
|||
|
|
Run the driver at `<skill base dir>/../../audit/bin/audit-stats` (the skill base
|
|||
|
|
directory is `<plugin root>/skills/audit-sessions`, the driver lives at
|
|||
|
|
`<plugin root>/audit/bin/audit-stats`):
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
<plugin root>/audit/bin/audit-stats --exclude <CURRENT_SESSION_ID>
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
(If the current session id is unknown, omit `--exclude` — the 10-minute active-window
|
|||
|
|
guard already defers the live session to the next run.)
|
|||
|
|
|
|||
|
|
- `--since` defaults to the last ledger row's date. It refuses to run without a prior row;
|
|||
|
|
a first-ever run needs an explicit `--since YYYY-MM-DD`.
|
|||
|
|
- Output: `~/.local/state/os-orchestration/audit/run-<date>/` containing `stats.tsv`
|
|||
|
|
(per-session deterministic columns), `flags.md` (heuristic missed-delegation regions),
|
|||
|
|
and `sheets/*.json` (full per-session fact-sheets). One row is appended to
|
|||
|
|
`~/.local/state/os-orchestration/audit/metrics.tsv` — the trend ledger. Never edit the
|
|||
|
|
ledger by hand; never re-derive these numbers with a model.
|
|||
|
|
|
|||
|
|
## Step 2 — Auditor fan-out (judgment only where code can't judge)
|
|||
|
|
|
|||
|
|
Read `stats.tsv` and `flags.md` from the run dir. Sessions with zero spawns AND zero
|
|||
|
|
flagged runs AND main_loop_share 1.0 at low turn counts usually need no auditor —
|
|||
|
|
include them only in the aggregate. Batch the remaining sessions into ~3 parallel
|
|||
|
|
auditor agents (5–8 sessions each), `model: "sonnet"`, background, one round each.
|
|||
|
|
|
|||
|
|
Each auditor prompt must include:
|
|||
|
|
- The rubric: read `references/rubric.md` (sibling to this SKILL.md) and paste it in.
|
|||
|
|
- The session's fact-sheet JSON path and flagged regions — auditors judge
|
|||
|
|
**mechanical vs judgment-dependent** for each flagged region by reading the transcript
|
|||
|
|
selectively around the cited jsonl lines (jq on line ranges, never the raw file).
|
|||
|
|
- The self-report line: "State the exact model ID you are running as in the first line
|
|||
|
|
of your report." Cross-check it; flag any pin mismatch in the report.
|
|||
|
|
- The exact per-session report format from the rubric.
|
|||
|
|
|
|||
|
|
## Step 3 — Synthesize and compare (main loop)
|
|||
|
|
|
|||
|
|
- Aggregate auditor findings into the tiered format: Tier 1 (HIGH, recurring),
|
|||
|
|
Tier 2 (MEDIUM), Tier 3 (solved surfaces — explicitly name what NOT to spend
|
|||
|
|
wording budget on).
|
|||
|
|
- Compare this run's ledger row against prior rows in `metrics.tsv` — the previous
|
|||
|
|
run's Tier 1 items are hypotheses this run tests (e.g. did settled-design drift
|
|||
|
|
drop after the 2026-07-10 wording patch?). State each verdict explicitly.
|
|||
|
|
- Write `report.md` into the run dir.
|
|||
|
|
|
|||
|
|
## Step 4 — Stop for the human
|
|||
|
|
|
|||
|
|
Report the tiers and trend verdicts, then STOP. Do not edit ORCHESTRATION.md without
|
|||
|
|
explicit approval — wording changes are a human decision (injected-token budget; grid
|
|||
|
|
validation is unavailable). If approved later: edit
|
|||
|
|
`~/dev/cc-os/plugins/os-orchestration/ORCHESTRATION.md`, run `~/dev/cc-os/bin/refresh-plugins`,
|
|||
|
|
and record the change in `~/dev/cc-os/docs/implementation-status.md`.
|
|||
|
|
|
|||
|
|
Finally, if a Planka card for this audit exists on the backlog board, move it to
|
|||
|
|
Review (never to Done).
|