65 lines
3.9 KiB
Markdown
65 lines
3.9 KiB
Markdown
# IRL audit rubric — shipped prompts/session-start/10-orchestration.md delegation-economics rules
|
||
|
||
Audit each assigned session against these shipped rules. A "finding" is a concrete
|
||
violation, near-miss, or friction point, with evidence (jsonl line number / short quote).
|
||
The deterministic fact-sheet (stats + flagged regions) is precomputed — do NOT recount
|
||
tool calls or spawns; your job is the judgment layer: mechanical vs judgment-dependent,
|
||
severity, and category.
|
||
|
||
## Rule surfaces (categories)
|
||
|
||
1. **missed-delegation** — main loop ran a mechanical sequence beyond ~3 tool calls with no
|
||
judgment between steps (multi-file edits, lookups, conversions, long log review, wide
|
||
grep-and-synthesize) instead of delegating down-tier. Includes: write-N-files fan-out
|
||
from a settled design done directly; the operator's own eval/benchmark/extraction loops.
|
||
2. **model-param** — an Agent spawn with no explicit `model:` param, or mechanical work
|
||
sent to sonnet/opus when haiku would do, or a sonnet/opus spawn whose prompt lacks the
|
||
self-report line, or a downgraded/mismatched resolved model treated silently as
|
||
judgment-tier.
|
||
3. **spawn-batching** — many small spawns where ~5–8 similar items should have been grouped
|
||
into one agent prompt; or a follow-up on an agent's result sent as a fresh spawn instead
|
||
of SendMessage to the live agent.
|
||
4. **async-usage** — main loop idle-waiting on a synchronous agent when background +
|
||
continue was possible; sleep-polling a background job; or conversely
|
||
babysitting/polling that the rules say to do directly.
|
||
5. **redundant-context** — delegated investigation re-covering ground the main loop already
|
||
read, or the main loop reading many files itself before delegating the investigation
|
||
anyway.
|
||
6. **over-delegation** — delegation where direct work was correct: ≤2 tool-call ops,
|
||
judgment-dependent steps, interactive troubleshooting, or a uniform change coverable by
|
||
one scripted Bash loop.
|
||
7. **drift** — session starts disciplined then drifts into long direct runs mid-session,
|
||
especially after a design/decision settles.
|
||
8. **tracker-routing** — mid-session follow-up work (a concrete task, bug, or deferred
|
||
item not done this session) left as a TODO comment or chat mention instead of captured
|
||
as an issue on the repo's configured tracker (`.cc-os/config` tracker key,
|
||
`os-backlog:capture`); an AI-authored issue carrying state labels or the human-curated
|
||
`next` label it shouldn't (ADR-0042 — git issues are the single tracker, `next` is
|
||
human-only, the AI never adds or removes it short of an explicit direct request); or an
|
||
issue visibly accreting scope/decisions in comments with no promotion flag raised to the
|
||
user — silent promotion counts too. Convention compliance only, not whether the routing
|
||
choice was wise.
|
||
|
||
## Per-session report format (return exactly this)
|
||
|
||
```
|
||
SESSION: <session id>
|
||
PROJECT/TOPIC: <one line — what the session was doing>
|
||
FINDINGS:
|
||
- [<category>] <severity HIGH/MED/LOW> — <one-sentence finding> | evidence: <jsonl line or short quote>
|
||
...or "none"
|
||
FLAGGED-REGION VERDICTS: <for each precomputed flagged region: mechanical | judgment-dependent | mixed, one line each>
|
||
POSITIVE: <patterns worth keeping, if notable>
|
||
```
|
||
|
||
## Method notes
|
||
|
||
- The fact-sheet JSON gives spawns (with model params and resolved models), per-segment
|
||
tool profiles, and flagged same-tool runs with jsonl line ranges. Read the transcript
|
||
ONLY selectively around those lines, e.g.
|
||
`sed -n '120,180p' file.jsonl | jq -r 'select(.type=="assistant") | .message.content[]? | select(.type=="text") | .text'`
|
||
- Judge "mechanical vs judgment" from tool names + inputs + the surrounding
|
||
user/assistant text: did each result change what happened next?
|
||
- Sequential-dependent work is a valid reason not to delegate; interactive
|
||
troubleshooting (user replying every few turns) is a valid reason too.
|