56 lines
3.1 KiB
Markdown
56 lines
3.1 KiB
Markdown
|
# IRL audit rubric — shipped ORCHESTRATION.md delegation-economics rules
|
|||
|
|
|
|||
|
|
Audit each assigned session against these shipped rules. A "finding" is a concrete
|
|||
|
|
violation, near-miss, or friction point, with evidence (jsonl line number / short quote).
|
|||
|
|
The deterministic fact-sheet (stats + flagged regions) is precomputed — do NOT recount
|
|||
|
|
tool calls or spawns; your job is the judgment layer: mechanical vs judgment-dependent,
|
|||
|
|
severity, and category.
|
|||
|
|
|
|||
|
|
## Rule surfaces (categories)
|
|||
|
|
|
|||
|
|
1. **missed-delegation** — main loop ran a mechanical sequence beyond ~3 tool calls with no
|
|||
|
|
judgment between steps (multi-file edits, lookups, conversions, long log review, wide
|
|||
|
|
grep-and-synthesize) instead of delegating down-tier. Includes: write-N-files fan-out
|
|||
|
|
from a settled design done directly; the operator's own eval/benchmark/extraction loops.
|
|||
|
|
2. **model-param** — an Agent spawn with no explicit `model:` param, or mechanical work
|
|||
|
|
sent to sonnet/opus when haiku would do, or a sonnet/opus spawn whose prompt lacks the
|
|||
|
|
self-report line, or a downgraded/mismatched resolved model treated silently as
|
|||
|
|
judgment-tier.
|
|||
|
|
3. **spawn-batching** — many small spawns where ~5–8 similar items should have been grouped
|
|||
|
|
into one agent prompt; or a follow-up on an agent's result sent as a fresh spawn instead
|
|||
|
|
of SendMessage to the live agent.
|
|||
|
|
4. **async-usage** — main loop idle-waiting on a synchronous agent when background +
|
|||
|
|
continue was possible; sleep-polling a background job; or conversely
|
|||
|
|
babysitting/polling that the rules say to do directly.
|
|||
|
|
5. **redundant-context** — delegated investigation re-covering ground the main loop already
|
|||
|
|
read, or the main loop reading many files itself before delegating the investigation
|
|||
|
|
anyway.
|
|||
|
|
6. **over-delegation** — delegation where direct work was correct: ≤2 tool-call ops,
|
|||
|
|
judgment-dependent steps, interactive troubleshooting, or a uniform change coverable by
|
|||
|
|
one scripted Bash loop.
|
|||
|
|
7. **drift** — session starts disciplined then drifts into long direct runs mid-session,
|
|||
|
|
especially after a design/decision settles.
|
|||
|
|
|
|||
|
|
## Per-session report format (return exactly this)
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
SESSION: <session id>
|
|||
|
|
PROJECT/TOPIC: <one line — what the session was doing>
|
|||
|
|
FINDINGS:
|
|||
|
|
- [<category>] <severity HIGH/MED/LOW> — <one-sentence finding> | evidence: <jsonl line or short quote>
|
|||
|
|
...or "none"
|
|||
|
|
FLAGGED-REGION VERDICTS: <for each precomputed flagged region: mechanical | judgment-dependent | mixed, one line each>
|
|||
|
|
POSITIVE: <patterns worth keeping, if notable>
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Method notes
|
|||
|
|
|
|||
|
|
- The fact-sheet JSON gives spawns (with model params and resolved models), per-segment
|
|||
|
|
tool profiles, and flagged same-tool runs with jsonl line ranges. Read the transcript
|
|||
|
|
ONLY selectively around those lines, e.g.
|
|||
|
|
`sed -n '120,180p' file.jsonl | jq -r 'select(.type=="assistant") | .message.content[]? | select(.type=="text") | .text'`
|
|||
|
|
- Judge "mechanical vs judgment" from tool names + inputs + the surrounding
|
|||
|
|
user/assistant text: did each result change what happened next?
|
|||
|
|
- Sequential-dependent work is a valid reason not to delegate; interactive
|
|||
|
|
troubleshooting (user replying every few turns) is a valid reason too.
|