111 lines
7.4 KiB
Markdown
111 lines
7.4 KiB
Markdown
# IRL audit rubric — shipped prompts/session-start/10-orchestration.md delegation rules (ADR-0043 absolute policy)
|
||
|
||
Audit each assigned session against the shipped ABSOLUTE delegation policy: the main
|
||
loop is the executive (interpret, plan, delegate, verify, report); every main-loop tool
|
||
call outside the enumerated exemptions is a violation. A "finding" is a concrete
|
||
violation, near-miss, or friction point, with evidence (jsonl line number / short
|
||
quote). The deterministic fact-sheet (stats + flagged regions) is precomputed — do NOT
|
||
recount tool calls, spawns, or SendMessage rounds; your job is the judgment layer the
|
||
sheet cannot do: exemption classification, rationale reading, severity.
|
||
|
||
## Exemption decision procedure (apply BEFORE counting any missed-delegation finding)
|
||
|
||
For every non-exempt-looking main-loop tool call, walk these steps in order and record
|
||
in the evidence line which step exempted it or that all failed:
|
||
|
||
- **E-a** — nearest preceding user turn (or same turn) explicitly names the file/path
|
||
touched → exempt. Evidence: quote the naming text.
|
||
- **E-b** — the call reads a diff, test output, or verification result and is followed
|
||
within ~3 tool calls by sign-off action or interpretation to the user (commit,
|
||
"verified", result explanation) → exempt. Evidence: quote the sign-off.
|
||
- **E-c** — a single state-inspection command (status/version/path check), not part of
|
||
a run, whose result is relayed to the user in the same or next turn → exempt.
|
||
- **E-d** — an act only the main loop can perform (user interaction, Agent/SendMessage
|
||
calls, final commits it stands behind) → exempt.
|
||
- **None apply** → count it under `missed-delegation`.
|
||
|
||
A finding whose evidence does not state the failed exemption steps is not falsifiable
|
||
by a second reader — do not report one without them.
|
||
|
||
## Rule surfaces (categories)
|
||
|
||
1. **missed-delegation** — any non-exempt main-loop tool call (per the procedure
|
||
above). The old defenses are GONE as defenses: "the chain was short," "steps were
|
||
judgment-dependent," and "sequential-dependent work" no longer justify direct work.
|
||
Sub-tag `missed-delegation/read-write` for main-loop Read/Write/Edit calls
|
||
scattered non-consecutively across the session (the pattern the fact-sheet's
|
||
run-detector cannot flag) — use the sheet's per-segment tool profiles as the
|
||
denominator and judge each call through the procedure.
|
||
Severity modifier (not a defense): a violation inside a tight interactive loop
|
||
(user replying every few turns) or a ≤2-call op is still counted but reported LOW
|
||
and tagged `accepted-cost` (ADR-0043 accepts this overhead knowingly); a long
|
||
direct grunt run is HIGH. Never mix the two at one severity.
|
||
2. **exemption-chaining** — grunt work laundered through the exemption list: 3+
|
||
exemption-justified reads/commands (esp. E-b/E-c) touching the same file or
|
||
subsystem in a short window, which taken together constitute the delegable
|
||
investigation the policy says to delegate. Flag on the aggregate pattern; do not
|
||
adjudicate each call individually.
|
||
3. **delegation-overhead-mismatch** (narrowed over-delegation) — delegation executed
|
||
so badly it bought nothing: a fresh spawn for a trivial exempt-adjacent errand that
|
||
belonged in the persistent manager's next SendMessage; or delegation theater —
|
||
main-loop re-verification so thorough (re-reading full files, re-running the work)
|
||
that it exceeds diff/output review and re-imports the delegated context.
|
||
4. **model-param** — an Agent spawn with no explicit `model:`; mechanical fully
|
||
specified work sent to sonnet/opus when haiku would do; a sonnet/opus spawn whose
|
||
prompt lacks the self-report line; a downgraded resolved model treated silently as
|
||
judgment-tier; or **forecast-escalation**: opus chosen with no cited prior sonnet
|
||
failure on the same problem AND no judgment-dense-end-to-end rationale (arch
|
||
review, subtle concurrency/correctness, adversarial test design). Evidence must
|
||
pair the fact-sheet spawn row (model, round) with a quote of the delegation
|
||
rationale — the sheet alone cannot distinguish legitimate first-choice opus.
|
||
5. **manager-continuation** — no persistent manager: grunt episodes after the first
|
||
delegable work sent as fresh spawns instead of SendMessage to the live manager
|
||
(evidence: the fresh spawn line + the prior live agent from the sheet's spawns[]
|
||
with matching task domain); or many small spawns where ~5–8 similar items should
|
||
have been one batched prompt with an explicit return format. Fresh spawns for
|
||
genuinely parallel independent work are correct.
|
||
6. **async-usage** — main loop idle-waiting on a synchronous agent when background +
|
||
continue was possible; sleep-polling a background job.
|
||
7. **redundant-context** — delegated investigation re-covering ground the main loop
|
||
already read under an exemption (the prompt should have carried it as stated
|
||
fact/summary), or main-loop orienting reads that should have gone to Explore
|
||
(target uncertain ⇒ the orientation itself was the delegable task).
|
||
8. **drift** — session applies the policy early then decays into direct grunt runs
|
||
mid-session, especially after a design/decision settles. Report as a pattern
|
||
finding over the constituent missed-delegation findings, not a duplicate count.
|
||
9. **tracker-routing** — mid-session follow-up work (a concrete task, bug, or deferred
|
||
item not done this session) left as a TODO comment or chat mention instead of
|
||
captured as an issue on the repo's configured tracker (`.cc-os/config` tracker key,
|
||
`os-backlog:capture`); an AI-authored issue carrying state labels or the
|
||
human-curated `next` label it shouldn't (ADR-0042 — git issues are the single
|
||
tracker, `next` is human-only, the AI never adds or removes it short of an explicit
|
||
direct request); or an issue visibly accreting scope/decisions in comments with no
|
||
promotion flag raised to the user — silent promotion counts too. Convention
|
||
compliance only, not whether the routing choice was wise.
|
||
|
||
## Per-session report format (return exactly this)
|
||
|
||
```
|
||
SESSION: <session id>
|
||
PROJECT/TOPIC: <one line — what the session was doing>
|
||
FINDINGS:
|
||
- [<category>] <severity HIGH/MED/LOW> — <one-sentence finding> | evidence: <jsonl line or short quote> | exemption-steps: <e.g. "E-a no (no path named), E-b no (no sign-off follows), E-c no (part of run)" — required for missed-delegation; "n/a" otherwise>
|
||
...or "none"
|
||
FLAGGED-REGION VERDICTS: <for each precomputed flagged region: violation | exempt (state step) | mixed, one line each>
|
||
POSITIVE: <patterns worth keeping, if notable>
|
||
```
|
||
|
||
## Method notes
|
||
|
||
- The fact-sheet JSON gives spawns (with model params, resolved models, SendMessage
|
||
round counts), per-segment tool profiles, and flagged same-tool runs with jsonl line
|
||
ranges. Read the transcript ONLY selectively around those lines, e.g.
|
||
`sed -n '120,180p' file.jsonl | jq -r 'select(.type=="assistant") | .message.content[]? | select(.type=="text") | .text'`
|
||
- Exemption facts live in the surrounding prose, not the tool call: E-a in the
|
||
preceding user turn, E-b/E-c in what happens next with the result. Always read both
|
||
sides of a flagged call.
|
||
- There are no validity defenses for direct grunt work under ADR-0043 — only the
|
||
severity modifiers in category 1. If a finding feels "technically a violation but
|
||
obviously fine," that is what LOW/`accepted-cost` is for; report it there rather
|
||
than dropping it, so the accepted-cost volume stays visible in the trend ledger.
|