os-context: absolute delegation policy (ADR-0043)
10-orchestration.md rewritten from the per-task threshold heuristic to an absolute policy: main loop as executive, enumerated E-a..E-d exemptions, one persistent sonnet manager continued via SendMessage, opus escalation on observed failure only. Audit rubric rewritten to match (exemption decision procedure, exemption-chaining, delegation-overhead-mismatch, accepted-cost severity modifiers, read-write sub-tag). WS4 eval retired as a measurement instrument (contaminated + encodes superseded policy), kept as harness template. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
parent
4c71f65313
commit
930ac6e80d
|
|
@ -0,0 +1,27 @@
|
||||||
|
---
|
||||||
|
id: "0043"
|
||||||
|
date: 2026-07-17
|
||||||
|
status: Accepted
|
||||||
|
supersedes:
|
||||||
|
superseded-by:
|
||||||
|
affected-paths: [plugins/os-context/prompts/session-start/10-orchestration.md, plugins/os-context/skills/audit-sessions/references/rubric.md]
|
||||||
|
affected-components: [os-context, audit-sessions]
|
||||||
|
---
|
||||||
|
|
||||||
|
# 0043 — Absolute delegation policy: main loop as executive with enumerated exemptions
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
The shipped os-context orchestration rules (prompts/session-start/10-orchestration.md, wording tuned in the WS4 eval loop) gated delegation on a per-task judgment: delegate when a mechanical sequence exceeds ~3 tool calls, work directly when the chain is short, judgment-dependent, or interactive. IRL transcript review (2026-07-17) showed the failure mode of that frame: the main-loop model must predict chain length and follow-up probability at decision time, both unknowable, and the errors land systematically on the keep-it-in-session side. Each 'short chain' decision is locally defensible, is re-made fresh with no memory of accumulated cost, and one chain reliably leads to another — sessions blow past 100k tokens of carried tool results on a fable/opus-tier main loop. Judgment-shaped rules are also expensive to audit: /os-context:audit-sessions had to judge judgment calls rather than count violations.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
Delegation is the default, not a threshold call. The main loop acts as the session's executive: it interprets the user, plans, delegates, verifies, and reports — it never does grunt work, regardless of how short the coming chain looks. Direct work is limited to an enumerated, mechanical exemption list (user-named file reads; diff/verification review before sign-off; single state-inspection commands; acts only the main loop can perform). Grunt work routes to ONE persistent sonnet manager agent spawned at the first delegable work and continued via SendMessage, not spawn-per-errand; fresh spawns only for genuinely parallel independent work. Tier routing: haiku for fully specified mechanical batches; sonnet as default manager tier; opus entered on observed failure (sonnet wrong or stuck twice on the same problem) rather than forecast, plus judgment-dense end-to-end work. Verification stays executive: the main loop reads the diff, not the report of the diff.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
Easier: main-loop context stays small, extending session lifetime before compaction/clear; the audit rubric becomes largely countable (non-exempt main-loop tool calls are violations) instead of interpretive; the economics stop depending on per-episode forecasts. Harder: trivial errands pay delegation overhead (spawn prompt, round-trip latency, report) that direct work would not; interactive tight-loop troubleshooting pays a wall-clock tax per round-trip since the old interactive carve-out was deliberately dropped as judgment-shaped; a main loop that never touches the repo can drift toward vague direction, mitigated by the exemption allowing grounding reads of user-named artifacts and by diff review. The WS4 eval sets validated the old wording and are contaminated, so behavioral validation of the new wording comes only from future IRL session audits.
|
||||||
|
|
||||||
|
## Alternatives rejected
|
||||||
|
|
||||||
|
1) Keep the per-task threshold heuristic (status quo): rejected — requires predicting chain length, which is unknowable at decision time and biased toward direct work; empirically produced 100k+-token main-loop sessions. 2) Cumulative session-budget trigger (delegate once main-loop tool-result tokens pass a budget): rejected as primary rule — better than per-task forecasting but still lets early episodes accrete and still needs a judged threshold; absorbed instead as the rationale (every byte re-billed per turn) rather than the mechanism. 3) Four-tier standing hierarchy with forecast routing (fable→opus→sonnet→haiku, choosing tier by predicted difficulty): rejected — 'does this need opus?' is the same unfalsifiable forecast the policy bans, and deep chains lose intent fidelity; replaced by failure-triggered escalation, which is observable. 4) Evaluative exemptions ('unless it's quick'): rejected — any judgment-shaped exemption reintroduces the broken prediction; exemptions must be enumerable.
|
||||||
|
|
@ -49,4 +49,5 @@ One file per decision, `NNNN-kebab-title.md`, created via `/os-adr:create`.
|
||||||
| 0040 | [Doc-hygiene rules never write other tools' ignore surfaces (no propagate_ignore)](0040-doc-hygiene-rules-never-write-other-tools-ignore-surfaces-no-propagate-ignore.md) | Accepted | 2026-07-14 |
|
| 0040 | [Doc-hygiene rules never write other tools' ignore surfaces (no propagate_ignore)](0040-doc-hygiene-rules-never-write-other-tools-ignore-surfaces-no-propagate-ignore.md) | Accepted | 2026-07-14 |
|
||||||
| 0041 | [Determinism promotion: hygiene nudges projects toward structurally-obvious completion conventions](0041-determinism-promotion-hygiene-nudges-projects-toward-structurally-obvious-completion-conventions.md) | Accepted | 2026-07-14 |
|
| 0041 | [Determinism promotion: hygiene nudges projects toward structurally-obvious completion conventions](0041-determinism-promotion-hygiene-nudges-projects-toward-structurally-obvious-completion-conventions.md) | Accepted | 2026-07-14 |
|
||||||
| 0042 | [Retire Planka: git issues are the single tracker for state and specs](0042-retire-planka-git-issues-are-the-single-tracker-for-state-and-specs.md) | Accepted | 2026-07-16 |
|
| 0042 | [Retire Planka: git issues are the single tracker for state and specs](0042-retire-planka-git-issues-are-the-single-tracker-for-state-and-specs.md) | Accepted | 2026-07-16 |
|
||||||
|
| 0043 | [Absolute delegation policy: main loop as executive with enumerated exemptions](0043-absolute-delegation-policy-main-loop-as-executive-with-enumerated-exemptions.md) | Accepted | 2026-07-17 |
|
||||||
<!-- adr-index:end -->
|
<!-- adr-index:end -->
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
# cc-os implementation status — index
|
# cc-os implementation status — index
|
||||||
|
|
||||||
_Last updated: 2026-07-16. This file is an **index with progressive disclosure** (see the
|
_Last updated: 2026-07-17. This file is an **index with progressive disclosure** (see the
|
||||||
index+PD convention ADR): headline timeline entries and per-component one-liners live here;
|
index+PD convention ADR): headline timeline entries and per-component one-liners live here;
|
||||||
full per-component detail lives in leaf files under `docs/implementation-status/`, read on
|
full per-component detail lives in leaf files under `docs/implementation-status/`, read on
|
||||||
demand. When a build step completes, add a **one-line** timeline entry here and put the
|
demand. When a build step completes, add a **one-line** timeline entry here and put the
|
||||||
|
|
@ -89,6 +89,10 @@ entry is in the leaf file named.
|
||||||
added (os-shortcuts, os-vault), naming convention folded into cc-architect references,
|
added (os-shortcuts, os-vault), naming convention folded into cc-architect references,
|
||||||
repo-level `bin/test` runner added. Detail:
|
repo-level `bin/test` runner added. Detail:
|
||||||
[implementation-status/standards-audit.md](implementation-status/standards-audit.md).
|
[implementation-status/standards-audit.md](implementation-status/standards-audit.md).
|
||||||
|
- **2026-07-17** — Absolute delegation policy shipped (ADR-0043): `10-orchestration.md`
|
||||||
|
rewritten (executive frame, enumerated exemptions, persistent sonnet manager,
|
||||||
|
failure-triggered opus escalation); audit rubric + `extract` driver updated to match;
|
||||||
|
WS4 eval retired as instrument, kept as template. Detail: os-context leaf.
|
||||||
|
|
||||||
**Remaining optional items:** additional project onboarding (one at a time, per ADR-0013);
|
**Remaining optional items:** additional project onboarding (one at a time, per ADR-0013);
|
||||||
bulk vault migration; os-adr rollout to pilot projects; os-backlog routing rollout (#14);
|
bulk vault migration; os-adr rollout to pilot projects; os-backlog routing rollout (#14);
|
||||||
|
|
|
||||||
|
|
@ -108,3 +108,19 @@ the rename refer to the plugin by its former name.
|
||||||
decision, never a continuation of the main loop's review thread (issue #30). Applied
|
decision, never a continuation of the main loop's review thread (issue #30). Applied
|
||||||
directly (design pre-approved, no wording-loop re-run); `bin/refresh-plugins` run after
|
directly (design pre-approved, no wording-loop re-run); `bin/refresh-plugins` run after
|
||||||
the edit.
|
the edit.
|
||||||
|
- **Absolute delegation policy (2026-07-17, ADR-0043):** IRL transcript review showed the
|
||||||
|
per-task delegation threshold fails by induction — chain length is unknowable at decision
|
||||||
|
time, each "short chain" call is locally defensible, and sessions accrete 100k+ tokens of
|
||||||
|
main-loop tool results. `10-orchestration.md` rewritten (design pre-approved in session):
|
||||||
|
executive frame, enumerated E-a..E-d exemptions (no evaluative carve-outs), one persistent
|
||||||
|
sonnet manager continued via SendMessage, opus escalation on observed failure only.
|
||||||
|
Audit rubric rewritten to match, shaped by a sonnet perspectives fan-out (measurement +
|
||||||
|
devils-advocate): exemption decision procedure with recorded steps per finding, new
|
||||||
|
`exemption-chaining` category, `over-delegation` narrowed to `delegation-overhead-mismatch`
|
||||||
|
(not deleted), old defenses demoted to LOW/`accepted-cost` severity modifiers,
|
||||||
|
`missed-delegation/read-write` sub-tag. `audit/bin/extract` gains `read_write_profile`
|
||||||
|
(total main-loop Read/Write/Edit incl. non-consecutive — largest prior session: 67 total,
|
||||||
|
only 14 visible to the run detector) and drops its stale pro-direct-work trailer. Eval
|
||||||
|
E1–E3/E5 retired as a measurement instrument (README banner): sets contaminated AND
|
||||||
|
scenarios encode the superseded policy; kept as harness template. Validation of the new
|
||||||
|
wording comes only from future IRL audits.
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,15 @@
|
||||||
# os-context eval — E1–E3, E5 (session-orchestration behavior)
|
# os-context eval — E1–E3, E5 (session-orchestration behavior)
|
||||||
|
|
||||||
|
> **STATUS 2026-07-17 — RETIRED AS A MEASUREMENT INSTRUMENT; KEPT AS TEMPLATE.**
|
||||||
|
> Two independent reasons, either sufficient: (1) the WS4 wording loop consumed the
|
||||||
|
> run-set AND the reserves — both are contaminated for any future wording tuning;
|
||||||
|
> (2) ADR-0043 replaced the per-task delegation threshold these scenarios encode
|
||||||
|
> with an absolute delegation policy, so the scenario expectations now judge
|
||||||
|
> against a superseded rule set. Do not run or rebuild expectations in place; a
|
||||||
|
> future eval of the ADR-0043 wording needs fresh scenarios sourced from new IRL
|
||||||
|
> audit findings, using this harness (bin/, fixture/, run mode below) as the
|
||||||
|
> structural template.
|
||||||
|
|
||||||
_Created: 2026-07-06. Scenarios sourced from the verified misses in
|
_Created: 2026-07-06. Scenarios sourced from the verified misses in
|
||||||
`docs/orchestration-audit/2026-07-06-findings.md` (clusters 1–3), per the
|
`docs/orchestration-audit/2026-07-06-findings.md` (clusters 1–3), per the
|
||||||
IRL-feedback-loop methodology. E4 (batch planning) deliberately deferred._
|
IRL-feedback-loop methodology. E4 (batch planning) deliberately deferred._
|
||||||
|
|
|
||||||
|
|
@ -1,64 +1,60 @@
|
||||||
## Session orchestration
|
## Session orchestration
|
||||||
|
|
||||||
- Main-loop tokens are the most expensive tokens in the session: every direct tool
|
- The role split is absolute, not situational: the main loop is the session's
|
||||||
call and every byte read into this context bills at the main-loop model's rate,
|
executive. It interprets the user, plans, delegates, verifies, and reports —
|
||||||
while a haiku/sonnet subagent does the same mechanical work for a fraction of it.
|
it does not do grunt work. Main-loop tokens are the most expensive in the
|
||||||
The more capable the main-loop model (opus- or fable-tier), the lower the
|
session, and every byte read here is re-billed on every later turn: budget
|
||||||
delegation threshold. The question is not "is this task big enough to delegate?"
|
spent on grunt work is stolen from the session's total capacity for
|
||||||
but "does this step need main-loop judgment?" — when a sequence beyond ~3 tool
|
high-level turns. Do NOT decide per task whether the coming chain is "short
|
||||||
calls needs no main-loop judgment between steps, delegate it down-tier, even if
|
enough" to do directly — chain length and follow-up probability are
|
||||||
the steps are strictly sequential.
|
unknowable at decision time, and one short chain reliably leads to another.
|
||||||
- Delegate when: work is parallelizable across independent files/subtasks; an
|
Delegation is the default for everything outside the exemption list below.
|
||||||
implementation task produces N independent files from an already-settled design,
|
- Exempt from delegation (enumerated and mechanical — never "quick" or
|
||||||
spec, or plan (write-N-files fan-out is delegated work — the design decisions are
|
"simple"): (a) reading a file or path the user explicitly named this turn;
|
||||||
already made); an investigation spans many files or needs a large/isolated
|
(b) reviewing diffs, test output, and verification results before sign-off —
|
||||||
context (long log review, wide grep-and-synthesize); or a mechanical multi-call
|
verification IS executive work, and a confident report reads the same whether
|
||||||
sequence (edits, lookups, conversions) needs no judgment between steps — this
|
the work is right or subtly wrong: read the diff, not the report of the
|
||||||
includes your own eval/benchmark/extraction work: repeated near-identical
|
diff; (c) a single state-inspection command needed to answer the user
|
||||||
parsers, headless grid runs, and CLI syntax hunts are mechanical sequences too;
|
accurately (a status, a version, a path check); (d) acts only the main loop
|
||||||
script once or delegate. A red-green TDD loop over an already-specified fix
|
can perform — user interaction, spawning/steering agents, final commits it
|
||||||
list is a single delegable unit — the internal test-driven sequencing is the
|
must stand behind. Everything else — searching, investigating, editing,
|
||||||
implementer's judgment, not the orchestrator's; put the failing-test list and
|
testing, running evals — is delegated work, even when strictly sequential.
|
||||||
fix directions into the spawn prompt. This holds for the whole session: after
|
- Persistent manager, not spawn-per-errand: at the first delegable work of the
|
||||||
one round of spawns completes, the next eligible chunk of work is delegated
|
session, spawn ONE sonnet manager agent; route subsequent grunt episodes to
|
||||||
too — don't drift back into long direct runs mid-session. WHEN a
|
that same agent via SendMessage — it already holds the files, layout, and
|
||||||
design/decision settles mid-session, stop before the first direct Write/Edit
|
diagnosis, while a fresh spawn re-pays the per-agent system-prompt tax and
|
||||||
that implements it — the resulting fan-out is a delegation event, even if the
|
re-covers ground. Spawn additional agents only for genuinely parallel,
|
||||||
session started with disciplined spawns. WHEN one or more review/diagnosis
|
independent work (then batch: group ~5–8 similar items into one prompt with
|
||||||
agents return a confirmed defect list → implementing that list is a NEW
|
an explicit return format, planned before the first spawn).
|
||||||
delegation decision, never a continuation of the main loop's review work.
|
- Tier routing — set `model:` explicitly on every `Agent` call (an omitted
|
||||||
- Work directly when: the op is single-file or ≤2 tool calls; steps are genuinely
|
`model` silently bills at the main-loop rate):
|
||||||
judgment-dependent (each result changes what you do next); the user is in the
|
- `haiku`: near-mechanical, fully specified steps — apply a listed edit set,
|
||||||
loop every few turns (interactive troubleshooting — delegation overhead exceeds
|
run-and-report a command matrix, format/parser conversions, ticket and
|
||||||
savings); a uniform multi-file change is covered by one scripted command (a
|
bookkeeping API sequences, bulk moves from an explicit map. The sonnet
|
||||||
loop/script in a few Bash calls is direct work, cheaper than a per-file grind or
|
manager may hand these down itself.
|
||||||
a spawn); or you are driving/polling a script's own output.
|
- `sonnet` (default manager tier): investigation and diagnosis, implementing
|
||||||
- Batch before spawning: plan the full fan-out before the first spawn, then group
|
from a settled design/spec/defect list, red-green TDD over a specified fix
|
||||||
related subtasks (~5–8 similar items) into one agent prompt with an explicit
|
list, wide grep-and-synthesize, log review, eval extraction runs.
|
||||||
return format, so each agent completes in one round. A follow-up on an agent's
|
- `opus`: entered on observed failure, not forecast — when the sonnet
|
||||||
result goes to that same live agent via SendMessage, not a fresh spawn — every
|
manager is wrong or stuck twice on the same problem, re-delegate that
|
||||||
new spawn re-pays the per-agent system-prompt tax.
|
problem to opus (or pull it up to the main loop). Also first choice for
|
||||||
- Delegate async and keep working: launch independent subagents in the background
|
work that is judgment-dense end-to-end: architectural review, subtle
|
||||||
and continue your own thread while they run; intervene only when a result shows
|
concurrency/correctness diagnosis, adversarial test design.
|
||||||
an agent off track. Never sleep-poll a background job from the main loop —
|
- When a spawn requests `sonnet` or `opus` → append to its prompt: "State the
|
||||||
background it and rely on completion notification (or Monitor); a `sleep` loop is
|
exact model ID you are running as in the first line of your report." (The
|
||||||
main-loop idle time billed at the top rate.
|
launch result does not show the resolved model.) A report showing a lower
|
||||||
- Before every `Agent` call → set `model:` explicitly in that call. An omitted
|
tier than requested → say so and adapt (re-spawn or flag the downgrade);
|
||||||
`model` silently bills the subagent at the main-loop model. Mechanical
|
never treat downgraded output as judgment-tier work silently.
|
||||||
file-edit/shell work → `haiku`; anything requiring judgment → `sonnet`; genuinely
|
- Delegate async and keep working: launch independent agents in the background
|
||||||
hard reasoning → `opus`.
|
and continue the executive thread; intervene only when a result shows an
|
||||||
- When a spawn requests `sonnet` or `opus` → append to its prompt: "State the exact
|
agent off track. Never sleep-poll a background job from the main loop —
|
||||||
model ID you are running as in the first line of your report." (The launch result
|
rely on completion notification (or Monitor); a `sleep` loop is main-loop
|
||||||
does not show the resolved model; the subagent's self-report is the only visible
|
idle time billed at the top rate.
|
||||||
signal.) When a report comes back showing a lower tier than you requested → say so
|
- Don't re-cover your own ground: a file already read under an exemption goes
|
||||||
and adapt (re-spawn or flag the downgrade) — never treat downgraded output as
|
into the agent prompt as a stated fact or summary, not an instruction to
|
||||||
judgment-tier work silently.
|
re-read it. If the target of an investigation is uncertain, that uncertainty
|
||||||
- Before delegating investigation → don't re-cover your own ground: a file you
|
is itself the delegable task — send orientation to an Explore agent rather
|
||||||
already read goes into the subagent prompt as a stated fact or summary, not as an
|
than orienting by hand.
|
||||||
instruction to read it again. If an investigation will span many files, delegate
|
- Where a call exposes an effort dial (Workflow `agent()` opts), set it per
|
||||||
it before reading them yourself — a short orienting Read is fine only when the
|
stage: mechanical stages `effort: low`; hard verify/judge stages
|
||||||
target file/path is uncertain. In an unfamiliar codebase, more than ~5 orienting
|
`high`/`xhigh`.
|
||||||
grep/read calls means the orientation itself is the delegable task — send it to
|
|
||||||
an Explore agent.
|
|
||||||
- Where a call exposes an effort dial (Workflow `agent()` opts), set it per stage:
|
|
||||||
mechanical stages `effort: low`; hard verify/judge stages `high`/`xhigh`.
|
|
||||||
|
|
|
||||||
|
|
@ -1,44 +1,87 @@
|
||||||
# IRL audit rubric — shipped prompts/session-start/10-orchestration.md delegation-economics rules
|
# IRL audit rubric — shipped prompts/session-start/10-orchestration.md delegation rules (ADR-0043 absolute policy)
|
||||||
|
|
||||||
Audit each assigned session against these shipped rules. A "finding" is a concrete
|
Audit each assigned session against the shipped ABSOLUTE delegation policy: the main
|
||||||
violation, near-miss, or friction point, with evidence (jsonl line number / short quote).
|
loop is the executive (interpret, plan, delegate, verify, report); every main-loop tool
|
||||||
The deterministic fact-sheet (stats + flagged regions) is precomputed — do NOT recount
|
call outside the enumerated exemptions is a violation. A "finding" is a concrete
|
||||||
tool calls or spawns; your job is the judgment layer: mechanical vs judgment-dependent,
|
violation, near-miss, or friction point, with evidence (jsonl line number / short
|
||||||
severity, and category.
|
quote). The deterministic fact-sheet (stats + flagged regions) is precomputed — do NOT
|
||||||
|
recount tool calls, spawns, or SendMessage rounds; your job is the judgment layer the
|
||||||
|
sheet cannot do: exemption classification, rationale reading, severity.
|
||||||
|
|
||||||
|
## Exemption decision procedure (apply BEFORE counting any missed-delegation finding)
|
||||||
|
|
||||||
|
For every non-exempt-looking main-loop tool call, walk these steps in order and record
|
||||||
|
in the evidence line which step exempted it or that all failed:
|
||||||
|
|
||||||
|
- **E-a** — nearest preceding user turn (or same turn) explicitly names the file/path
|
||||||
|
touched → exempt. Evidence: quote the naming text.
|
||||||
|
- **E-b** — the call reads a diff, test output, or verification result and is followed
|
||||||
|
within ~3 tool calls by sign-off action or interpretation to the user (commit,
|
||||||
|
"verified", result explanation) → exempt. Evidence: quote the sign-off.
|
||||||
|
- **E-c** — a single state-inspection command (status/version/path check), not part of
|
||||||
|
a run, whose result is relayed to the user in the same or next turn → exempt.
|
||||||
|
- **E-d** — an act only the main loop can perform (user interaction, Agent/SendMessage
|
||||||
|
calls, final commits it stands behind) → exempt.
|
||||||
|
- **None apply** → count it under `missed-delegation`.
|
||||||
|
|
||||||
|
A finding whose evidence does not state the failed exemption steps is not falsifiable
|
||||||
|
by a second reader — do not report one without them.
|
||||||
|
|
||||||
## Rule surfaces (categories)
|
## Rule surfaces (categories)
|
||||||
|
|
||||||
1. **missed-delegation** — main loop ran a mechanical sequence beyond ~3 tool calls with no
|
1. **missed-delegation** — any non-exempt main-loop tool call (per the procedure
|
||||||
judgment between steps (multi-file edits, lookups, conversions, long log review, wide
|
above). The old defenses are GONE as defenses: "the chain was short," "steps were
|
||||||
grep-and-synthesize) instead of delegating down-tier. Includes: write-N-files fan-out
|
judgment-dependent," and "sequential-dependent work" no longer justify direct work.
|
||||||
from a settled design done directly; the operator's own eval/benchmark/extraction loops.
|
Sub-tag `missed-delegation/read-write` for main-loop Read/Write/Edit calls
|
||||||
2. **model-param** — an Agent spawn with no explicit `model:` param, or mechanical work
|
scattered non-consecutively across the session (the pattern the fact-sheet's
|
||||||
sent to sonnet/opus when haiku would do, or a sonnet/opus spawn whose prompt lacks the
|
run-detector cannot flag) — use the sheet's per-segment tool profiles as the
|
||||||
self-report line, or a downgraded/mismatched resolved model treated silently as
|
denominator and judge each call through the procedure.
|
||||||
judgment-tier.
|
Severity modifier (not a defense): a violation inside a tight interactive loop
|
||||||
3. **spawn-batching** — many small spawns where ~5–8 similar items should have been grouped
|
(user replying every few turns) or a ≤2-call op is still counted but reported LOW
|
||||||
into one agent prompt; or a follow-up on an agent's result sent as a fresh spawn instead
|
and tagged `accepted-cost` (ADR-0043 accepts this overhead knowingly); a long
|
||||||
of SendMessage to the live agent.
|
direct grunt run is HIGH. Never mix the two at one severity.
|
||||||
4. **async-usage** — main loop idle-waiting on a synchronous agent when background +
|
2. **exemption-chaining** — grunt work laundered through the exemption list: 3+
|
||||||
continue was possible; sleep-polling a background job; or conversely
|
exemption-justified reads/commands (esp. E-b/E-c) touching the same file or
|
||||||
babysitting/polling that the rules say to do directly.
|
subsystem in a short window, which taken together constitute the delegable
|
||||||
5. **redundant-context** — delegated investigation re-covering ground the main loop already
|
investigation the policy says to delegate. Flag on the aggregate pattern; do not
|
||||||
read, or the main loop reading many files itself before delegating the investigation
|
adjudicate each call individually.
|
||||||
anyway.
|
3. **delegation-overhead-mismatch** (narrowed over-delegation) — delegation executed
|
||||||
6. **over-delegation** — delegation where direct work was correct: ≤2 tool-call ops,
|
so badly it bought nothing: a fresh spawn for a trivial exempt-adjacent errand that
|
||||||
judgment-dependent steps, interactive troubleshooting, or a uniform change coverable by
|
belonged in the persistent manager's next SendMessage; or delegation theater —
|
||||||
one scripted Bash loop.
|
main-loop re-verification so thorough (re-reading full files, re-running the work)
|
||||||
7. **drift** — session starts disciplined then drifts into long direct runs mid-session,
|
that it exceeds diff/output review and re-imports the delegated context.
|
||||||
especially after a design/decision settles.
|
4. **model-param** — an Agent spawn with no explicit `model:`; mechanical fully
|
||||||
8. **tracker-routing** — mid-session follow-up work (a concrete task, bug, or deferred
|
specified work sent to sonnet/opus when haiku would do; a sonnet/opus spawn whose
|
||||||
item not done this session) left as a TODO comment or chat mention instead of captured
|
prompt lacks the self-report line; a downgraded resolved model treated silently as
|
||||||
as an issue on the repo's configured tracker (`.cc-os/config` tracker key,
|
judgment-tier; or **forecast-escalation**: opus chosen with no cited prior sonnet
|
||||||
`os-backlog:capture`); an AI-authored issue carrying state labels or the human-curated
|
failure on the same problem AND no judgment-dense-end-to-end rationale (arch
|
||||||
`next` label it shouldn't (ADR-0042 — git issues are the single tracker, `next` is
|
review, subtle concurrency/correctness, adversarial test design). Evidence must
|
||||||
human-only, the AI never adds or removes it short of an explicit direct request); or an
|
pair the fact-sheet spawn row (model, round) with a quote of the delegation
|
||||||
issue visibly accreting scope/decisions in comments with no promotion flag raised to the
|
rationale — the sheet alone cannot distinguish legitimate first-choice opus.
|
||||||
user — silent promotion counts too. Convention compliance only, not whether the routing
|
5. **manager-continuation** — no persistent manager: grunt episodes after the first
|
||||||
choice was wise.
|
delegable work sent as fresh spawns instead of SendMessage to the live manager
|
||||||
|
(evidence: the fresh spawn line + the prior live agent from the sheet's spawns[]
|
||||||
|
with matching task domain); or many small spawns where ~5–8 similar items should
|
||||||
|
have been one batched prompt with an explicit return format. Fresh spawns for
|
||||||
|
genuinely parallel independent work are correct.
|
||||||
|
6. **async-usage** — main loop idle-waiting on a synchronous agent when background +
|
||||||
|
continue was possible; sleep-polling a background job.
|
||||||
|
7. **redundant-context** — delegated investigation re-covering ground the main loop
|
||||||
|
already read under an exemption (the prompt should have carried it as stated
|
||||||
|
fact/summary), or main-loop orienting reads that should have gone to Explore
|
||||||
|
(target uncertain ⇒ the orientation itself was the delegable task).
|
||||||
|
8. **drift** — session applies the policy early then decays into direct grunt runs
|
||||||
|
mid-session, especially after a design/decision settles. Report as a pattern
|
||||||
|
finding over the constituent missed-delegation findings, not a duplicate count.
|
||||||
|
9. **tracker-routing** — mid-session follow-up work (a concrete task, bug, or deferred
|
||||||
|
item not done this session) left as a TODO comment or chat mention instead of
|
||||||
|
captured as an issue on the repo's configured tracker (`.cc-os/config` tracker key,
|
||||||
|
`os-backlog:capture`); an AI-authored issue carrying state labels or the
|
||||||
|
human-curated `next` label it shouldn't (ADR-0042 — git issues are the single
|
||||||
|
tracker, `next` is human-only, the AI never adds or removes it short of an explicit
|
||||||
|
direct request); or an issue visibly accreting scope/decisions in comments with no
|
||||||
|
promotion flag raised to the user — silent promotion counts too. Convention
|
||||||
|
compliance only, not whether the routing choice was wise.
|
||||||
|
|
||||||
## Per-session report format (return exactly this)
|
## Per-session report format (return exactly this)
|
||||||
|
|
||||||
|
|
@ -46,19 +89,22 @@ severity, and category.
|
||||||
SESSION: <session id>
|
SESSION: <session id>
|
||||||
PROJECT/TOPIC: <one line — what the session was doing>
|
PROJECT/TOPIC: <one line — what the session was doing>
|
||||||
FINDINGS:
|
FINDINGS:
|
||||||
- [<category>] <severity HIGH/MED/LOW> — <one-sentence finding> | evidence: <jsonl line or short quote>
|
- [<category>] <severity HIGH/MED/LOW> — <one-sentence finding> | evidence: <jsonl line or short quote> | exemption-steps: <e.g. "E-a no (no path named), E-b no (no sign-off follows), E-c no (part of run)" — required for missed-delegation; "n/a" otherwise>
|
||||||
...or "none"
|
...or "none"
|
||||||
FLAGGED-REGION VERDICTS: <for each precomputed flagged region: mechanical | judgment-dependent | mixed, one line each>
|
FLAGGED-REGION VERDICTS: <for each precomputed flagged region: violation | exempt (state step) | mixed, one line each>
|
||||||
POSITIVE: <patterns worth keeping, if notable>
|
POSITIVE: <patterns worth keeping, if notable>
|
||||||
```
|
```
|
||||||
|
|
||||||
## Method notes
|
## Method notes
|
||||||
|
|
||||||
- The fact-sheet JSON gives spawns (with model params and resolved models), per-segment
|
- The fact-sheet JSON gives spawns (with model params, resolved models, SendMessage
|
||||||
tool profiles, and flagged same-tool runs with jsonl line ranges. Read the transcript
|
round counts), per-segment tool profiles, and flagged same-tool runs with jsonl line
|
||||||
ONLY selectively around those lines, e.g.
|
ranges. Read the transcript ONLY selectively around those lines, e.g.
|
||||||
`sed -n '120,180p' file.jsonl | jq -r 'select(.type=="assistant") | .message.content[]? | select(.type=="text") | .text'`
|
`sed -n '120,180p' file.jsonl | jq -r 'select(.type=="assistant") | .message.content[]? | select(.type=="text") | .text'`
|
||||||
- Judge "mechanical vs judgment" from tool names + inputs + the surrounding
|
- Exemption facts live in the surrounding prose, not the tool call: E-a in the
|
||||||
user/assistant text: did each result change what happened next?
|
preceding user turn, E-b/E-c in what happens next with the result. Always read both
|
||||||
- Sequential-dependent work is a valid reason not to delegate; interactive
|
sides of a flagged call.
|
||||||
troubleshooting (user replying every few turns) is a valid reason too.
|
- There are no validity defenses for direct grunt work under ADR-0043 — only the
|
||||||
|
severity modifiers in category 1. If a finding feels "technically a violation but
|
||||||
|
obviously fine," that is what LOW/`accepted-cost` is for; report it there rather
|
||||||
|
than dropping it, so the accepted-cost volume stays visible in the trend ledger.
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue