71 lines
3.9 KiB
Markdown
71 lines
3.9 KiB
Markdown
---
|
||
description: Run the biweekly orchestration IRL audit - deterministic transcript stats precompute, then auditor fan-out judging flagged regions against the shipped `prompts/session-start/10-orchestration.md` rules, then a tiered tune-up report. Works from any directory. Trigger via the os-status due nudge or the recurring issue jared/ops#1 on Forgejo.
|
||
---
|
||
|
||
# Audit production sessions against the shipped orchestration rules
|
||
|
||
You are running the recurring (biweekly) IRL audit of the shipped delegation-economics
|
||
rules. The signal source is real production session transcripts — NEVER eval grid
|
||
scenarios or reserve sets (contaminated for wording tuning since 2026-07-08). This skill
|
||
works from any cwd; nothing here requires being inside the cc-os repo.
|
||
|
||
## Step 1 — Deterministic stats precompute (code, no judgment)
|
||
|
||
Run the driver at `<skill base dir>/../../audit/bin/audit-stats` (the skill base
|
||
directory is `<plugin root>/skills/audit-sessions`, the driver lives at
|
||
`<plugin root>/audit/bin/audit-stats`):
|
||
|
||
```bash
|
||
<plugin root>/audit/bin/audit-stats --exclude <CURRENT_SESSION_ID>
|
||
```
|
||
|
||
(If the current session id is unknown, omit `--exclude` — the 10-minute active-window
|
||
guard already defers the live session to the next run.)
|
||
|
||
- `--since` defaults to the last ledger row's date. It refuses to run without a prior row;
|
||
a first-ever run needs an explicit `--since YYYY-MM-DD`.
|
||
- Output: `~/.local/state/os-context/audit/run-<date>/` containing `stats.tsv`
|
||
(per-session deterministic columns), `flags.md` (heuristic missed-delegation regions),
|
||
and `sheets/*.json` (full per-session fact-sheets). One row is appended to
|
||
`~/.local/state/os-context/audit/metrics.tsv` — the trend ledger. Never edit the
|
||
ledger by hand; never re-derive these numbers with a model.
|
||
|
||
## Step 2 — Auditor fan-out (judgment only where code can't judge)
|
||
|
||
Read `stats.tsv` and `flags.md` from the run dir. Sessions with zero spawns AND zero
|
||
flagged runs AND main_loop_share 1.0 at low turn counts usually need no auditor —
|
||
include them only in the aggregate. Batch the remaining sessions into ~3 parallel
|
||
auditor agents (5–8 sessions each), `model: "sonnet"`, background, one round each.
|
||
|
||
Each auditor prompt must include:
|
||
- The rubric: read `references/rubric.md` (sibling to this SKILL.md) and paste it in.
|
||
- The session's fact-sheet JSON path and flagged regions — auditors judge
|
||
**mechanical vs judgment-dependent** for each flagged region by reading the transcript
|
||
selectively around the cited jsonl lines (jq on line ranges, never the raw file).
|
||
- The self-report line: "State the exact model ID you are running as in the first line
|
||
of your report." Cross-check it; flag any pin mismatch in the report.
|
||
- The exact per-session report format from the rubric.
|
||
|
||
## Step 3 — Synthesize and compare (main loop)
|
||
|
||
- Aggregate auditor findings into the tiered format: Tier 1 (HIGH, recurring),
|
||
Tier 2 (MEDIUM), Tier 3 (solved surfaces — explicitly name what NOT to spend
|
||
wording budget on).
|
||
- Compare this run's ledger row against prior rows in `metrics.tsv` — the previous
|
||
run's Tier 1 items are hypotheses this run tests (e.g. did settled-design drift
|
||
drop after the 2026-07-10 wording patch?). State each verdict explicitly.
|
||
- Write `report.md` into the run dir.
|
||
|
||
## Step 4 — Stop for the human
|
||
|
||
Report the tiers and trend verdicts, then STOP. Do not edit
|
||
`prompts/session-start/10-orchestration.md` without explicit approval — wording changes
|
||
are a human decision (injected-token budget; grid validation is unavailable). If approved
|
||
later: edit
|
||
`~/dev/cc-os/plugins/os-context/prompts/session-start/10-orchestration.md`, run `~/dev/cc-os/bin/refresh-plugins`,
|
||
and record the change in `~/dev/cc-os/docs/implementation-status.md`.
|
||
|
||
Finally, comment the audit date + report location on the recurring issue
|
||
`jared/ops#1` (Forgejo, via `tea`) and leave it open — recurring-convention
|
||
issues are never closed by the AI (ADR-0042).
|