os-orchestration: WS4 wording loop shipped — ORCHESTRATION.md v3
Candidate-1 (v3 draft) accepted on iteration 1, confirmed on full grid, validated on frozen reserve (15/16 cells, negatives 20/20, E5P reserve twin 5/5 both tiers). H3 met: E5P fable median mltok 27,372 = 43% of 62,971 baseline. Results TSVs + loop log + CLAUDE.md/plan status; eval-results vault note written; limit-banner sweep + econ-guard lessons promoted to methodology notes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
jared
parent
ab258d5ba1
commit
397145b27b
|
|
@ -95,6 +95,8 @@ to those two and fix the stale doc.
|
||||||
- Eval harness E1–E3 (2026-07-06): `plugins/os-orchestration/eval/` — scripted headless orchestration-behavior eval scored by the audit extractor (extended with per-segment read targets). 6 run-set scenarios (E1P/N downgrade detection with `CLAUDE_CODE_SUBAGENT_MODEL=haiku` runner stub; E2P/N explicit-model fan-out vs no-delegation-on-trivial; E3P/N delegate-investigation vs orienting-reads-correct) + 6 frozen reserve twins; new Node.js "relaystation" fixture with deterministic sandbox-time log generation; deterministic-first checker (`eval/bin/check`, narrow frozen haiku-judge fallback for E1 language axes only, stubbable via `ORCH_EVAL_JUDGE_CMD`); model-free `bin/self-test` 21/21 green incl. shipped-instruction-compliant conformance transcripts. FAIL axes map only to verified audit misses; tier choice informational. **Scenario Task blocks are held-out — never run informally.** Procedure: `plugins/os-orchestration/eval/README.md`.
|
- Eval harness E1–E3 (2026-07-06): `plugins/os-orchestration/eval/` — scripted headless orchestration-behavior eval scored by the audit extractor (extended with per-segment read targets). 6 run-set scenarios (E1P/N downgrade detection with `CLAUDE_CODE_SUBAGENT_MODEL=haiku` runner stub; E2P/N explicit-model fan-out vs no-delegation-on-trivial; E3P/N delegate-investigation vs orienting-reads-correct) + 6 frozen reserve twins; new Node.js "relaystation" fixture with deterministic sandbox-time log generation; deterministic-first checker (`eval/bin/check`, narrow frozen haiku-judge fallback for E1 language axes only, stubbable via `ORCH_EVAL_JUDGE_CMD`); model-free `bin/self-test` 21/21 green incl. shipped-instruction-compliant conformance transcripts. FAIL axes map only to verified audit misses; tier choice informational. **Scenario Task blocks are held-out — never run informally.** Procedure: `plugins/os-orchestration/eval/README.md`.
|
||||||
- Baseline grid run (2026-07-06, run-set × sonnet/haiku × 3 reps + 2 counted canaries, all counted, zero harness errors): negatives 18/18 PASS both tiers (zero over-delegation, zero false downgrade claims); positives 1/18 on the delegate-at-all axis — but transcript verification shows most positive FAILs are *superior direct strategies* (E2P sonnet: one scripted bulk edit; E3P sonnet: surgical greps over 400KB logs), which the WS1 audit itself scored as justified non-delegation, vs genuine miss shapes (E3P haiku: whole 112KB logs into own context; E2P haiku: 20-call per-file grind). Conditional rules validated where engaged: 14/14 spawns explicit-model; the one delegating E1P rep flagged the stubbed downgrade via self-report. Criterion redesign for E2P/E3P A-axes is pre-registered (anchor thresholds independently, NOT to this grid) before any rescore/tuning; reserve stays frozen. ORCHESTRATION.md gained a scripted-bulk-edit carve-out (a uniform multi-file change one script covers is direct work). TSV: `eval/results/2026-07-06-baseline-grid.tsv`; full analysis: vault note [[os-orchestration-eval-baseline-grid-results]].
|
- Baseline grid run (2026-07-06, run-set × sonnet/haiku × 3 reps + 2 counted canaries, all counted, zero harness errors): negatives 18/18 PASS both tiers (zero over-delegation, zero false downgrade claims); positives 1/18 on the delegate-at-all axis — but transcript verification shows most positive FAILs are *superior direct strategies* (E2P sonnet: one scripted bulk edit; E3P sonnet: surgical greps over 400KB logs), which the WS1 audit itself scored as justified non-delegation, vs genuine miss shapes (E3P haiku: whole 112KB logs into own context; E2P haiku: 20-call per-file grind). Conditional rules validated where engaged: 14/14 spawns explicit-model; the one delegating E1P rep flagged the stubbed downgrade via self-report. Criterion redesign for E2P/E3P A-axes is pre-registered (anchor thresholds independently, NOT to this grid) before any rescore/tuning; reserve stays frozen. ORCHESTRATION.md gained a scripted-bulk-edit carve-out (a uniform multi-file change one script covers is direct work). TSV: `eval/results/2026-07-06-baseline-grid.tsv`; full analysis: vault note [[os-orchestration-eval-baseline-grid-results]].
|
||||||
|
|
||||||
|
- WS4 economics loop COMPLETE + SHIPPED (2026-07-08; run-set AND reserve now contaminated for future wording tuning): ORCHESTRATION.md v3 shipped — cost-asymmetry + tier-conditional threshold opener, symmetric delegate-when/work-directly-when trigger lists (replaces "delegate only when"), batching rule (~5–8 items per grouped spawn, SendMessage reuse), async rule, Workflow effort dial; explicit-model/self-report/don't-re-cover-ground rules unchanged. Harness extended first (commit b226601): real token econ axes — subagent transcripts persist at `<projects>/<flat-cwd>/<session-id>/subagents/agent-*.jsonl` with per-message usage blocks (solves the mini-audit's instrumentation gap; vault note [[claude-code-subagent-transcripts-and-token-accounting]]) — E5 batching pair + reserve twins (authored by subagent, held-out content never entered the tuning session), fable column, pre-registered E2P/E3P criterion redesign landed (scripted-direct PASS branch; 74KB ingestion anchor + id-keyed root-cause concepts). Results: E5P 0/6 → majority-PASS both tiers incl. reserve twin 5/5 (fable median mltok 43% of baseline, H3 met); negatives clean everywhere (36/36 across confirm+reserve). Residual gaps: E1P sonnet delegate-at-all (fixture too small — needs bigger fixture before meaningful), fable downgrade-flag ~1/4 judge flicker. 15 baseline reps were session-limit-truncated and mechanically excluded+replaced (limit-banner sweep now a standard step, promoted to the autoresearch howto). TSVs: `eval/results/2026-07-08-*.tsv`; loop log: `autoresearch/classic-260708-1039/`; vault note [[os-orchestration-ws4-econ-wording-results]]. IRL re-audit of ~5 real Fable sessions scheduled ~2026-07-22.
|
||||||
|
|
||||||
**Global os-status plugin** — `cc-os/plugins/os-status/` (git-tracked, 2026-07-06); symlinked into `~/.claude/plugins/os-status`
|
**Global os-status plugin** — `cc-os/plugins/os-status/` (git-tracked, 2026-07-06); symlinked into `~/.claude/plugins/os-status`
|
||||||
- Purpose: aggregated deterministic SessionStart status checks for the cc-os plugin family — per-project artifacts (ADR system, vault hub note) and environment hazards (subagent model env override). Built via OpenSpec change `add-os-status-plugin` (supersedes the WS3 plan doc); decision record ADR-022.
|
- Purpose: aggregated deterministic SessionStart status checks for the cc-os plugin family — per-project artifacts (ADR system, vault hub note) and environment hazards (subagent model env override). Built via OpenSpec change `add-os-status-plugin` (supersedes the WS3 plan doc); decision record ADR-022.
|
||||||
- Architecture: checks are **in-process** plain Python functions in `hooks/checks.py` with the uniform signature `check(ctx) -> CheckResult(status, message)` plus one registry entry (no per-plugin subprocess protocol, no per-project code copies — both rejected in ADR-022 after perspective review). Three states: `ok` (silent), `note` (near-zero-token additionalContext line, never snoozed/suppressed), `warn` (aggregated into at most one banner; once-per-day snooze + permanent `suppress-<check>` markers in gitignored per-project `.cc-os/` — state only, never code; `.cc-os/config` holds optional `hub`/`vault_path` key=value overrides).
|
- Architecture: checks are **in-process** plain Python functions in `hooks/checks.py` with the uniform signature `check(ctx) -> CheckResult(status, message)` plus one registry entry (no per-plugin subprocess protocol, no per-project code copies — both rejected in ADR-022 after perspective review). Three states: `ok` (silent), `note` (near-zero-token additionalContext line, never snoozed/suppressed), `warn` (aggregated into at most one banner; once-per-day snooze + permanent `suppress-<check>` markers in gitignored per-project `.cc-os/` — state only, never code; `.cc-os/config` holds optional `hub`/`vault_path` key=value overrides).
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,117 @@
|
||||||
|
# System-prompt profiles: assessment & recommended approach
|
||||||
|
|
||||||
|
_Last updated: 2026-07-08 · Status: assessment only, no build started_
|
||||||
|
|
||||||
|
Companion to the WS4 orchestration-economics work. Goal: reduce per-session context cost and
|
||||||
|
improve focus by controlling what reaches the model's context — the built-in system prompt,
|
||||||
|
and (more importantly) the layers we add on top of it.
|
||||||
|
|
||||||
|
## What is mechanically possible today
|
||||||
|
|
||||||
|
Verified against `claude --help` (current install) and
|
||||||
|
https://code.claude.com/docs/en/agent-sdk/modifying-system-prompts.md (fetched 2026-07-08):
|
||||||
|
|
||||||
|
| Mechanism | What it does | Risk |
|
||||||
|
|---|---|---|
|
||||||
|
| `--append-system-prompt[-file]` | Adds instructions after the full default prompt | None — nothing removed |
|
||||||
|
| `--system-prompt[-file]` | **Fully replaces** the built-in prompt for the session | High — loses tool guidance, safety rules, env context; you must re-provide them |
|
||||||
|
| `--exclude-dynamic-system-prompt-sections` | Moves per-machine sections (cwd, git status, env, memory paths) into first user message → cross-session cache reuse | Low; env context carries slightly less weight |
|
||||||
|
| `--tools <list>` / `--disallowedTools` | Restricts the built-in toolset (removes tool schemas from context) | Low |
|
||||||
|
| `--settings` / `--setting-sources user,project,local` | Controls which settings layers (and therefore CLAUDE.md files, enabled plugins, hooks) load | Low |
|
||||||
|
| Output styles (`~/.claude/output-styles/*.md`, `keep-coding-instructions: true|false`) | Replace or extend the *coding-instruction section* of the default prompt; persistent, reusable across projects | Medium — partial replacement, officially supported |
|
||||||
|
| Plugin enable/disable per session (`--settings` overlay or profile-specific settings file) | Each enabled plugin's skills/agents/hooks descriptions are injected every session | Low — biggest immediate lever |
|
||||||
|
| Agent SDK (`systemPrompt: {type:'preset', append, excludeDynamicSections}` or custom string) | Same controls programmatically; SDK default is a *minimal* prompt unless the preset is requested | — |
|
||||||
|
|
||||||
|
Deferred-tool loading (ToolSearch) already ships in current CC — most tool schemas are no
|
||||||
|
longer paid up front. That reduced the value of "gut the system prompt" since much of the old
|
||||||
|
bloat was tool definitions.
|
||||||
|
|
||||||
|
## Measured baseline (2026-07-08, headless haiku `-p` runs, `--output-format json` usage)
|
||||||
|
|
||||||
|
| Configuration | Context tokens | Delta |
|
||||||
|
|---|---|---|
|
||||||
|
| `--setting-sources "" --tools ""` (system prompt text only) | **6,347** | — |
|
||||||
|
| `--setting-sources ""` (prompt + full built-in tool schemas) | **24,277** | +17.9k = tool schemas |
|
||||||
|
| Default settings, empty dir (user stack: 17 plugins, global CLAUDE.md, hooks) | **29,042** | +4.8k = user stack |
|
||||||
|
| Full stack, cc-os dir | **41,765** | +12.7k = project layer (CLAUDE.md ~11k dominates) |
|
||||||
|
|
||||||
|
Interpretation:
|
||||||
|
- **The system prompt *text* is only ~6.3k tokens.** The ~24k harness floor is dominated by
|
||||||
|
**tool schemas (~18k)** — so the bottom-up lever is `--tools` restriction per profile, not
|
||||||
|
prompt rewriting. Full `--system-prompt` replacement can save at most ~6k and carries the
|
||||||
|
fork-maintenance risk.
|
||||||
|
- The 17-plugin user stack costs ~4.8k, not tens of k — cheaper than estimated.
|
||||||
|
- The single largest single item we control is the cc-os CLAUDE.md (~11k).
|
||||||
|
- An interactive session that reads ~80k after one exchange is ~42k fixed overhead plus
|
||||||
|
~38k of *turn-1 work product* (web fetches, file reads) — conversation content, not
|
||||||
|
harness bloat. Caveat: these baselines are headless haiku; an interactive Fable session's
|
||||||
|
skill/agent listings and model-specific sections may differ somewhat — worth one `/context`
|
||||||
|
check interactively.
|
||||||
|
|
||||||
|
## Key finding: our bloat is mostly self-inflicted, not Anthropic's
|
||||||
|
|
||||||
|
Measured 2026-07-08 for a cc-os session:
|
||||||
|
|
||||||
|
- **cc-os `CLAUDE.md`: ~43 KB ≈ ~11k tokens**, injected into every session. Much of it is an
|
||||||
|
append-only implementation-status changelog, not orientation the session needs.
|
||||||
|
- **17 enabled plugins** — every skill description (~80 skills), agent type, and hook usage
|
||||||
|
note is injected into every session regardless of relevance (`creative-team`,
|
||||||
|
`invoice-ninja`, `rails-ui-component`, `api-wrapper`… in a cc-os design session).
|
||||||
|
- **memsearch SessionStart recall injection**: ~13 KB this session.
|
||||||
|
- Multiple SessionStart hooks (orchestration, os-vault, os-status) each add context.
|
||||||
|
|
||||||
|
The built-in system prompt is a static, cached prefix (cache reads are ~10% of input price
|
||||||
|
after the first turn within the 5-min TTL). The layers above are the same cost class *and*
|
||||||
|
they're the ones we control. The "bloat → hallucination" link is context rot: attention
|
||||||
|
degrades as context grows, whatever the source. Cutting 20k tokens of our own always-on
|
||||||
|
injection beats fighting Anthropic's prompt.
|
||||||
|
|
||||||
|
Pi reality check: pi.dev makes **no** speed/cost/hallucination claims on its site. Its
|
||||||
|
lean-ness comes from being a *different harness* (no MCP, no subagents, no permission
|
||||||
|
system, minimal prompt + extensions) — not from stripping Claude Code. Fully replacing CC's
|
||||||
|
system prompt while keeping CC's harness means maintaining a fork of prompt content the
|
||||||
|
model was tuned around, re-broken by every CC update.
|
||||||
|
|
||||||
|
## Recommended approach: subtract-first modular profiles
|
||||||
|
|
||||||
|
A `cyolo <flags>` shell wrapper that composes **existing** mechanisms per objective/project.
|
||||||
|
Profiles mostly *subtract* (plugins, tools, setting sources) and lightly *add*
|
||||||
|
(one append-file per objective). Full system-prompt replacement is a later, eval-gated
|
||||||
|
experiment on headless sessions only.
|
||||||
|
|
||||||
|
1. **Phase 0 — measure.** Run `/context` in 3–4 representative session types (cc-os design,
|
||||||
|
client dev, brainstorm) and record the actual breakdown. No optimizing before baselining.
|
||||||
|
2. **Phase 1 — diet (no new machinery).**
|
||||||
|
- Split cc-os CLAUDE.md: keep orientation (~2–3k tokens), move the implementation-status
|
||||||
|
changelog to a linked doc the AI reads on demand (progressive disclosure).
|
||||||
|
- Per-profile plugin enablement: a settings overlay per profile listing `enabledPlugins`;
|
||||||
|
a brainstorm profile doesn't load rails-ui-component/api-wrapper/invoice-ninja.
|
||||||
|
3. **Phase 2 — `cyolo` profile wrapper.**
|
||||||
|
- Central evergreen component prompts in one source dir (e.g. `~/.claude/profiles/`):
|
||||||
|
`dev.md`, `ruby.md`, `planning.md` (objective-focused: starting point → process →
|
||||||
|
definition of done), `brainstorm.md`.
|
||||||
|
- Wrapper resolves profile from flag (`--dev`, `--planning`) + project detection (pwd,
|
||||||
|
Gemfile, etc.), then launches
|
||||||
|
`claude --append-system-prompt-file <composed> --settings <profile-settings> [--tools …]`.
|
||||||
|
- Objective profiles state the session's endpoint explicitly ("this session ends when the
|
||||||
|
plan is written and shared; implementation belongs to a later session").
|
||||||
|
4. **Phase 3 — eval-gated replacement experiment (optional).** Arize-style loop, matching the
|
||||||
|
autoresearch playbook: scenario set × {default prompt, dieted stack, profile stack,
|
||||||
|
full-replacement lean prompt} × model tiers; score task success + tokens/turn +
|
||||||
|
wrong-tool/hallucination proxies from transcripts. Only promote full replacement if it
|
||||||
|
wins on outcome, not just token count. Read
|
||||||
|
`~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md` first, per standing rule.
|
||||||
|
|
||||||
|
Bottom-up track note: to audit the real shipped prompt, capture a live request body via a
|
||||||
|
local logging proxy (`ANTHROPIC_BASE_URL` pointed at a one-file pass-through that dumps the
|
||||||
|
`system` and `tools` arrays) — don't rely on the model to self-report its prompt. Given the
|
||||||
|
6.3k measurement, audit it to find what a lean profile can safely drop, but expect the
|
||||||
|
`--tools` list to be where the real savings are.
|
||||||
|
|
||||||
|
## Open questions
|
||||||
|
|
||||||
|
- Per-model profile variants: deferred (user agrees it's too granular now); the wording-loop
|
||||||
|
evals already show tier-dependent phrasing needs, so revisit after profiles exist.
|
||||||
|
- Whether `--setting-sources` interacts cleanly with symlinked local plugins + hook absolute
|
||||||
|
paths in `settings.json` (hooks are wired by absolute path and would still fire).
|
||||||
|
- Where profile settings overlays live so they don't fight `bin/refresh-plugins`.
|
||||||
|
|
@ -1,6 +1,9 @@
|
||||||
# WS4 — Orchestration economics: ORCHESTRATION.md v3 + eval extension + wording loop
|
# WS4 — Orchestration economics: ORCHESTRATION.md v3 + eval extension + wording loop
|
||||||
|
|
||||||
_Created: 2026-07-08. Status: PLANNED (mini-audit complete; steps 1–5 not started)._
|
_Created: 2026-07-08. Status: **SHIPPED** (steps 1–4 complete 2026-07-08; step 5 IRL
|
||||||
|
re-audit scheduled ~2026-07-22). Loop log: `autoresearch/classic-260708-1039/loop-log.md`;
|
||||||
|
results TSVs: `plugins/os-orchestration/eval/results/2026-07-08-*.tsv`; vault note:
|
||||||
|
[[os-orchestration-ws4-econ-wording-results]]._
|
||||||
_Prereq reading before executing: `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md`,
|
_Prereq reading before executing: `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md`,
|
||||||
vault notes [[orchestration-prompting-claude-5-era]], [[eval-methodology-ladder]],
|
vault notes [[orchestration-prompting-claude-5-era]], [[eval-methodology-ladder]],
|
||||||
[[os-orchestration-eval-baseline-grid-results]]._
|
[[os-orchestration-eval-baseline-grid-results]]._
|
||||||
|
|
|
||||||
|
|
@ -1,10 +1,36 @@
|
||||||
## Session orchestration
|
## Session orchestration
|
||||||
|
|
||||||
- Do single-file, ≤2-tool-call ops directly. Don't delegate them. Delegate only when
|
- Main-loop tokens are the most expensive tokens in the session: every direct tool
|
||||||
work is parallelizable across independent files/subtasks, spans many files, or
|
call and every byte read into this context bills at the main-loop model's rate,
|
||||||
needs a large/isolated context (long log review, wide grep-and-synthesize).
|
while a haiku/sonnet subagent does the same mechanical work for a fraction of it.
|
||||||
A uniform multi-file change that one scripted command covers is direct work too —
|
The more capable the main-loop model (opus- or fable-tier), the lower the
|
||||||
don't spawn agents for what a loop or script does in one coordinated operation.
|
delegation threshold. The question is not "is this task big enough to delegate?"
|
||||||
|
but "does this step need main-loop judgment?" — when a sequence beyond ~3 tool
|
||||||
|
calls needs no main-loop judgment between steps, delegate it down-tier, even if
|
||||||
|
the steps are strictly sequential.
|
||||||
|
- Delegate when: work is parallelizable across independent files/subtasks; an
|
||||||
|
implementation task produces N independent files from an already-settled design,
|
||||||
|
spec, or plan (write-N-files fan-out is delegated work — the design decisions are
|
||||||
|
already made); an investigation spans many files or needs a large/isolated
|
||||||
|
context (long log review, wide grep-and-synthesize); or a mechanical multi-call
|
||||||
|
sequence (edits, lookups, conversions) needs no judgment between steps. This
|
||||||
|
holds for the whole session: after one round of spawns completes, the next
|
||||||
|
eligible chunk of work is delegated too — don't drift back into long direct runs
|
||||||
|
mid-session.
|
||||||
|
- Work directly when: the op is single-file or ≤2 tool calls; steps are genuinely
|
||||||
|
judgment-dependent (each result changes what you do next); the user is in the
|
||||||
|
loop every few turns (interactive troubleshooting — delegation overhead exceeds
|
||||||
|
savings); a uniform multi-file change is covered by one scripted command (a
|
||||||
|
loop/script in a few Bash calls is direct work, cheaper than a per-file grind or
|
||||||
|
a spawn); or you are driving/polling a script's own output.
|
||||||
|
- Batch before spawning: plan the full fan-out before the first spawn, then group
|
||||||
|
related subtasks (~5–8 similar items) into one agent prompt with an explicit
|
||||||
|
return format, so each agent completes in one round. A follow-up on an agent's
|
||||||
|
result goes to that same live agent via SendMessage, not a fresh spawn — every
|
||||||
|
new spawn re-pays the per-agent system-prompt tax.
|
||||||
|
- Delegate async and keep working: launch independent subagents in the background
|
||||||
|
and continue your own thread while they run; intervene only when a result shows
|
||||||
|
an agent off track.
|
||||||
- Before every `Agent` call → set `model:` explicitly in that call. An omitted
|
- Before every `Agent` call → set `model:` explicitly in that call. An omitted
|
||||||
`model` silently bills the subagent at the main-loop model. Mechanical
|
`model` silently bills the subagent at the main-loop model. Mechanical
|
||||||
file-edit/shell work → `haiku`; anything requiring judgment → `sonnet`; genuinely
|
file-edit/shell work → `haiku`; anything requiring judgment → `sonnet`; genuinely
|
||||||
|
|
@ -20,3 +46,5 @@
|
||||||
instruction to read it again. If an investigation will span many files, delegate
|
instruction to read it again. If an investigation will span many files, delegate
|
||||||
it before reading them yourself — a short orienting Read is fine only when the
|
it before reading them yourself — a short orienting Read is fine only when the
|
||||||
target file/path is uncertain.
|
target file/path is uncertain.
|
||||||
|
- Where a call exposes an effort dial (Workflow `agent()` opts), set it per stage:
|
||||||
|
mechanical stages `effort: low`; hard verify/judge stages `high`/`xhigh`.
|
||||||
|
|
|
||||||
|
|
@ -1,50 +0,0 @@
|
||||||
## Session orchestration
|
|
||||||
|
|
||||||
- Main-loop tokens are the most expensive tokens in the session: every direct tool
|
|
||||||
call and every byte read into this context bills at the main-loop model's rate,
|
|
||||||
while a haiku/sonnet subagent does the same mechanical work for a fraction of it.
|
|
||||||
The more capable the main-loop model (opus- or fable-tier), the lower the
|
|
||||||
delegation threshold. The question is not "is this task big enough to delegate?"
|
|
||||||
but "does this step need main-loop judgment?" — when a sequence beyond ~3 tool
|
|
||||||
calls needs no main-loop judgment between steps, delegate it down-tier, even if
|
|
||||||
the steps are strictly sequential.
|
|
||||||
- Delegate when: work is parallelizable across independent files/subtasks; an
|
|
||||||
implementation task produces N independent files from an already-settled design,
|
|
||||||
spec, or plan (write-N-files fan-out is delegated work — the design decisions are
|
|
||||||
already made); an investigation spans many files or needs a large/isolated
|
|
||||||
context (long log review, wide grep-and-synthesize); or a mechanical multi-call
|
|
||||||
sequence (edits, lookups, conversions) needs no judgment between steps. This
|
|
||||||
holds for the whole session: after one round of spawns completes, the next
|
|
||||||
eligible chunk of work is delegated too — don't drift back into long direct runs
|
|
||||||
mid-session.
|
|
||||||
- Work directly when: the op is single-file or ≤2 tool calls; steps are genuinely
|
|
||||||
judgment-dependent (each result changes what you do next); the user is in the
|
|
||||||
loop every few turns (interactive troubleshooting — delegation overhead exceeds
|
|
||||||
savings); a uniform multi-file change is covered by one scripted command (a
|
|
||||||
loop/script in a few Bash calls is direct work, cheaper than a per-file grind or
|
|
||||||
a spawn); or you are driving/polling a script's own output.
|
|
||||||
- Batch before spawning: plan the full fan-out before the first spawn, then group
|
|
||||||
related subtasks (~5–8 similar items) into one agent prompt with an explicit
|
|
||||||
return format, so each agent completes in one round. A follow-up on an agent's
|
|
||||||
result goes to that same live agent via SendMessage, not a fresh spawn — every
|
|
||||||
new spawn re-pays the per-agent system-prompt tax.
|
|
||||||
- Delegate async and keep working: launch independent subagents in the background
|
|
||||||
and continue your own thread while they run; intervene only when a result shows
|
|
||||||
an agent off track.
|
|
||||||
- Before every `Agent` call → set `model:` explicitly in that call. An omitted
|
|
||||||
`model` silently bills the subagent at the main-loop model. Mechanical
|
|
||||||
file-edit/shell work → `haiku`; anything requiring judgment → `sonnet`; genuinely
|
|
||||||
hard reasoning → `opus`.
|
|
||||||
- When a spawn requests `sonnet` or `opus` → append to its prompt: "State the exact
|
|
||||||
model ID you are running as in the first line of your report." (The launch result
|
|
||||||
does not show the resolved model; the subagent's self-report is the only visible
|
|
||||||
signal.) When a report comes back showing a lower tier than you requested → say so
|
|
||||||
and adapt (re-spawn or flag the downgrade) — never treat downgraded output as
|
|
||||||
judgment-tier work silently.
|
|
||||||
- Before delegating investigation → don't re-cover your own ground: a file you
|
|
||||||
already read goes into the subagent prompt as a stated fact or summary, not as an
|
|
||||||
instruction to read it again. If an investigation will span many files, delegate
|
|
||||||
it before reading them yourself — a short orienting Read is fine only when the
|
|
||||||
target file/path is uncertain.
|
|
||||||
- Where a call exposes an effort dial (Workflow `agent()` opts), set it per stage:
|
|
||||||
mechanical stages `effort: low`; hard verify/judge stages `high`/`xhigh`.
|
|
||||||
|
|
@ -0,0 +1,41 @@
|
||||||
|
# WS4 wording loop, candidate-1 (shipped ORCHESTRATION.md v3), source /tmp/orch-ws4-confirm
|
||||||
|
E1N-pipeline-review fable 1 PASS B:pass-mechanical tiers=sonnet=3;spawns=3;mltok=19879;sctok=13163;mlshare=0.602;prebytes=42;maxrounds=1
|
||||||
|
E1N-pipeline-review fable 2 PASS B:pass-mechanical tiers=sonnet=3;spawns=3;mltok=18260;sctok=7041;mlshare=0.722;prebytes=201;maxrounds=1
|
||||||
|
E1N-pipeline-review sonnet 1 PASS B:pass-mechanical tiers=;spawns=0;mltok=16312;sctok=0;mlshare=1.0;prebytes=5567;maxrounds=0
|
||||||
|
E1N-pipeline-review sonnet 2 PASS B:pass-mechanical tiers=;spawns=0;mltok=18422;sctok=0;mlshare=1.0;prebytes=5494;maxrounds=0
|
||||||
|
E1N-pipeline-review sonnet 3 PASS B:pass-mechanical tiers=sonnet=3;spawns=3;mltok=13646;sctok=3984;mlshare=0.774;prebytes=121;maxrounds=1
|
||||||
|
E1P-pipeline-review fable 1 FAIL A:pass;B:fail-judge tiers=sonnet=3;mismatched=3;mltok=25823;sctok=14526;mlshare=0.64;prebytes=42;spawns=3;maxrounds=1
|
||||||
|
E1P-pipeline-review fable 2 PASS A:pass;B:pass-judge tiers=sonnet=3;mismatched=3;mltok=31965;sctok=14691;mlshare=0.685;prebytes=42;spawns=3;maxrounds=1
|
||||||
|
E1P-pipeline-review sonnet 1 FAIL A:no-delegation mltok=14909;sctok=0;mlshare=1.0;prebytes=8377;spawns=0;maxrounds=0
|
||||||
|
E1P-pipeline-review sonnet 2 FAIL A:no-delegation mltok=7211;sctok=0;mlshare=1.0;prebytes=9264;spawns=0;maxrounds=0
|
||||||
|
E1P-pipeline-review sonnet 3 FAIL A:no-delegation mltok=16724;sctok=0;mlshare=1.0;prebytes=8861;spawns=0;maxrounds=0
|
||||||
|
E2N-readme-typo fable 1 PASS A:no-spawn pre_bytes=278;mltok=767;sctok=0;mlshare=1.0;prebytes=278;spawns=0;maxrounds=0
|
||||||
|
E2N-readme-typo fable 2 PASS A:no-spawn pre_bytes=278;mltok=553;sctok=0;mlshare=1.0;prebytes=278;spawns=0;maxrounds=0
|
||||||
|
E2N-readme-typo sonnet 1 PASS A:no-spawn pre_bytes=466;mltok=639;sctok=0;mlshare=1.0;prebytes=466;spawns=0;maxrounds=0
|
||||||
|
E2N-readme-typo sonnet 2 PASS A:no-spawn pre_bytes=402;mltok=593;sctok=0;mlshare=1.0;prebytes=402;spawns=0;maxrounds=0
|
||||||
|
E2N-readme-typo sonnet 3 PASS A:no-spawn pre_bytes=398;mltok=584;sctok=0;mlshare=1.0;prebytes=398;spawns=0;maxrounds=0
|
||||||
|
E2P-retry-policy fable 1 PASS A:scripted-direct;C:pass edits=0;mltok=2217;sctok=0;mlshare=1.0;prebytes=1737;spawns=0;maxrounds=0
|
||||||
|
E2P-retry-policy fable 2 PASS A:scripted-direct;C:pass edits=0;mltok=1918;sctok=0;mlshare=1.0;prebytes=1871;spawns=0;maxrounds=0
|
||||||
|
E2P-retry-policy sonnet 1 PASS A:scripted-direct;C:pass edits=0;mltok=729;sctok=0;mlshare=1.0;prebytes=3647;spawns=0;maxrounds=0
|
||||||
|
E2P-retry-policy sonnet 2 PASS A:scripted-direct;C:pass edits=0;mltok=1485;sctok=0;mlshare=1.0;prebytes=3885;spawns=0;maxrounds=0
|
||||||
|
E2P-retry-policy sonnet 3 PASS A:scripted-direct;C:pass edits=0;mltok=1079;sctok=0;mlshare=1.0;prebytes=3553;spawns=0;maxrounds=0
|
||||||
|
E3N-brand-spelling fable 1 PASS A:no-spawn pre_bytes=2038;edited=yes;mltok=2413;sctok=0;mlshare=1.0;prebytes=2038;spawns=0;maxrounds=0
|
||||||
|
E3N-brand-spelling fable 2 PASS A:no-spawn pre_bytes=2224;edited=yes;mltok=2828;sctok=0;mlshare=1.0;prebytes=2224;spawns=0;maxrounds=0
|
||||||
|
E3N-brand-spelling sonnet 1 PASS A:no-spawn pre_bytes=641;edited=yes;mltok=920;sctok=0;mlshare=1.0;prebytes=641;spawns=0;maxrounds=0
|
||||||
|
E3N-brand-spelling sonnet 2 PASS A:no-spawn pre_bytes=6307;edited=yes;mltok=1002;sctok=0;mlshare=1.0;prebytes=6307;spawns=0;maxrounds=0
|
||||||
|
E3N-brand-spelling sonnet 3 PASS A:no-spawn pre_bytes=582;edited=yes;mltok=610;sctok=0;mlshare=1.0;prebytes=582;spawns=0;maxrounds=0
|
||||||
|
E3P-dropped-events fable 1 PASS A:pass;B:pass;C:pass bytes_read=18798;tiers=;mltok=15029;sctok=0;mlshare=1.0;prebytes=18798;spawns=0;maxrounds=0
|
||||||
|
E3P-dropped-events fable 2 PASS A:pass;B:pass;C:pass bytes_read=29738;tiers=;mltok=12766;sctok=0;mlshare=1.0;prebytes=29738;spawns=0;maxrounds=0
|
||||||
|
E3P-dropped-events sonnet 1 PASS A:pass;B:pass;C:pass bytes_read=25301;tiers=;mltok=10572;sctok=0;mlshare=1.0;prebytes=25301;spawns=0;maxrounds=0
|
||||||
|
E3P-dropped-events sonnet 2 PASS A:pass;B:pass;C:pass bytes_read=483;tiers=sonnet=1;mltok=4234;sctok=7149;mlshare=0.372;prebytes=483;spawns=1;maxrounds=1
|
||||||
|
E3P-dropped-events sonnet 3 PASS A:pass;B:pass;C:pass bytes_read=1412;tiers=sonnet=1;mltok=4689;sctok=7073;mlshare=0.399;prebytes=267;spawns=1;maxrounds=1
|
||||||
|
E5N-limit-rebalance fable 1 PASS A:sequential tiers=;mltok=3742;sctok=0;mlshare=1.0;prebytes=2773;spawns=0;maxrounds=0
|
||||||
|
E5N-limit-rebalance fable 2 PASS A:sequential tiers=;mltok=2928;sctok=0;mlshare=1.0;prebytes=2773;spawns=0;maxrounds=0
|
||||||
|
E5N-limit-rebalance sonnet 1 PASS A:sequential tiers=;mltok=2111;sctok=0;mlshare=1.0;prebytes=2876;spawns=0;maxrounds=0
|
||||||
|
E5N-limit-rebalance sonnet 2 PASS A:sequential tiers=;mltok=1878;sctok=0;mlshare=1.0;prebytes=2801;spawns=0;maxrounds=0
|
||||||
|
E5N-limit-rebalance sonnet 3 PASS A:sequential tiers=;mltok=2108;sctok=0;mlshare=1.0;prebytes=2797;spawns=0;maxrounds=0
|
||||||
|
E5P-handler-rollout fable 1 PASS A:pass;B:pass;C:pass;D:pass tiers=haiku=3;mltok=47345;sctok=11426;mlshare=0.806;prebytes=10457;spawns=3;maxrounds=1
|
||||||
|
E5P-handler-rollout fable 2 PASS A:pass;B:pass;C:pass;D:pass tiers=haiku=2,sonnet=1;mltok=34543;sctok=8252;mlshare=0.807;prebytes=11533;spawns=3;maxrounds=1
|
||||||
|
E5P-handler-rollout sonnet 1 PASS A:pass;B:pass;C:pass;D:pass tiers=haiku=2,sonnet=1;mltok=13904;sctok=12725;mlshare=0.522;prebytes=16277;spawns=3;maxrounds=1
|
||||||
|
E5P-handler-rollout sonnet 2 FAIL A:main-loop-grind mltok=9375;sctok=0;mlshare=1.0;prebytes=15389;spawns=0;maxrounds=0
|
||||||
|
E5P-handler-rollout sonnet 3 PASS A:pass;B:pass;C:pass;D:pass tiers=sonnet=2;mltok=7363;sctok=6071;mlshare=0.548;prebytes=13825;spawns=2;maxrounds=1
|
||||||
|
Can't render this file because it has a wrong number of fields in line 2.
|
|
|
@ -0,0 +1,16 @@
|
||||||
|
# WS4 wording loop, candidate-1 (shipped ORCHESTRATION.md v3), source /tmp/orch-ws4-iter1
|
||||||
|
E1P-pipeline-review fable 1 PASS A:pass;B:pass-judge tiers=sonnet=3;mismatched=3;mltok=31386;sctok=6642;mlshare=0.825;prebytes=201;spawns=3;maxrounds=1
|
||||||
|
E1P-pipeline-review fable 2 PASS A:pass;B:pass-mechanical tiers=sonnet=3;mismatched=3;mltok=24815;sctok=69554;mlshare=0.263;prebytes=201;spawns=3;maxrounds=1
|
||||||
|
E1P-pipeline-review sonnet 1 FAIL A:no-delegation mltok=16778;sctok=0;mlshare=1.0;prebytes=5452;spawns=0;maxrounds=0
|
||||||
|
E1P-pipeline-review sonnet 2 FAIL A:no-delegation mltok=38903;sctok=0;mlshare=1.0;prebytes=5536;spawns=0;maxrounds=0
|
||||||
|
E1P-pipeline-review sonnet 3 FAIL A:no-delegation mltok=31682;sctok=0;mlshare=1.0;prebytes=6866;spawns=0;maxrounds=0
|
||||||
|
E2N-readme-typo sonnet 1 PASS A:no-spawn pre_bytes=398;mltok=568;sctok=0;mlshare=1.0;prebytes=398;spawns=0;maxrounds=0
|
||||||
|
E2P-retry-policy sonnet 1 PASS A:scripted-direct;C:pass edits=0;mltok=1187;sctok=0;mlshare=1.0;prebytes=5002;spawns=0;maxrounds=0
|
||||||
|
E5N-limit-rebalance fable 1 PASS A:sequential tiers=;mltok=4304;sctok=0;mlshare=1.0;prebytes=2864;spawns=0;maxrounds=0
|
||||||
|
E5N-limit-rebalance fable 2 PASS A:sequential tiers=;mltok=3713;sctok=0;mlshare=1.0;prebytes=2773;spawns=0;maxrounds=0
|
||||||
|
E5P-handler-rollout fable 1 PASS A:pass;B:pass;C:pass;D:pass tiers=sonnet=2;mltok=27372;sctok=9016;mlshare=0.752;prebytes=10457;spawns=2;maxrounds=1
|
||||||
|
E5P-handler-rollout fable 2 PASS A:pass;B:pass;C:pass;D:pass tiers=haiku=2;mltok=22003;sctok=13938;mlshare=0.612;prebytes=10457;spawns=2;maxrounds=1
|
||||||
|
E5P-handler-rollout fable 3 PASS A:pass;B:pass;C:pass;D:pass tiers=haiku=1,sonnet=1;mltok=25501;sctok=14908;mlshare=0.631;prebytes=12341;spawns=2;maxrounds=1
|
||||||
|
E5P-handler-rollout sonnet 1 PASS A:pass;B:pass;C:pass;D:pass tiers=sonnet=2;mltok=22345;sctok=5309;mlshare=0.808;prebytes=23164;spawns=2;maxrounds=1
|
||||||
|
E5P-handler-rollout sonnet 2 PASS A:pass;B:pass;C:pass;D:pass tiers=sonnet=2;mltok=16438;sctok=6015;mlshare=0.732;prebytes=15300;spawns=2;maxrounds=1
|
||||||
|
E5P-handler-rollout sonnet 3 PASS A:pass;B:pass;C:pass;D:pass tiers=sonnet=2;mltok=5459;sctok=5927;mlshare=0.479;prebytes=13741;spawns=2;maxrounds=1
|
||||||
|
Can't render this file because it has a wrong number of fields in line 2.
|
|
|
@ -0,0 +1,41 @@
|
||||||
|
# WS4 wording loop, candidate-1 (shipped ORCHESTRATION.md v3), source /tmp/orch-ws4-reserve
|
||||||
|
E1N-datalayer-review fable 1 PASS B:pass-mechanical tiers=sonnet=3;spawns=3;mltok=19274;sctok=9251;mlshare=0.676;prebytes=0;maxrounds=1
|
||||||
|
E1N-datalayer-review fable 2 PASS B:pass-mechanical tiers=sonnet=3;spawns=3;mltok=17063;sctok=17611;mlshare=0.492;prebytes=0;maxrounds=1
|
||||||
|
E1N-datalayer-review sonnet 1 PASS B:pass-mechanical tiers=;spawns=0;mltok=10674;sctok=0;mlshare=1.0;prebytes=4243;maxrounds=0
|
||||||
|
E1N-datalayer-review sonnet 2 PASS B:pass-mechanical tiers=;spawns=0;mltok=13964;sctok=0;mlshare=1.0;prebytes=3816;maxrounds=0
|
||||||
|
E1N-datalayer-review sonnet 3 PASS B:pass-mechanical tiers=;spawns=0;mltok=4488;sctok=0;mlshare=1.0;prebytes=4341;maxrounds=0
|
||||||
|
E1P-datalayer-review fable 1 FAIL A:no-delegation mltok=16120;sctok=0;mlshare=1.0;prebytes=8515;spawns=0;maxrounds=0
|
||||||
|
E1P-datalayer-review fable 2 PASS A:pass;B:pass-mechanical tiers=sonnet=3;mismatched=3;mltok=24644;sctok=24397;mlshare=0.503;prebytes=0;spawns=3;maxrounds=1
|
||||||
|
E1P-datalayer-review sonnet 1 FAIL A:no-delegation mltok=28945;sctok=0;mlshare=1.0;prebytes=3678;spawns=0;maxrounds=0
|
||||||
|
E1P-datalayer-review sonnet 2 FAIL A:no-delegation mltok=21996;sctok=0;mlshare=1.0;prebytes=3815;spawns=0;maxrounds=0
|
||||||
|
E1P-datalayer-review sonnet 3 FAIL A:no-delegation mltok=10270;sctok=0;mlshare=1.0;prebytes=4745;spawns=0;maxrounds=0
|
||||||
|
E2N-port-fix fable 1 PASS A:no-spawn pre_bytes=952;mltok=630;sctok=0;mlshare=1.0;prebytes=952;spawns=0;maxrounds=0
|
||||||
|
E2N-port-fix fable 2 PASS A:no-spawn pre_bytes=730;mltok=632;sctok=0;mlshare=1.0;prebytes=730;spawns=0;maxrounds=0
|
||||||
|
E2N-port-fix sonnet 1 PASS A:no-spawn pre_bytes=631;mltok=550;sctok=0;mlshare=1.0;prebytes=631;spawns=0;maxrounds=0
|
||||||
|
E2N-port-fix sonnet 2 PASS A:no-spawn pre_bytes=598;mltok=385;sctok=0;mlshare=1.0;prebytes=598;spawns=0;maxrounds=0
|
||||||
|
E2N-port-fix sonnet 3 PASS A:no-spawn pre_bytes=636;mltok=473;sctok=0;mlshare=1.0;prebytes=636;spawns=0;maxrounds=0
|
||||||
|
E2P-timeout-bump fable 1 PASS A:scripted-direct;C:pass edits=0;mltok=1528;sctok=0;mlshare=1.0;prebytes=1369;spawns=0;maxrounds=0
|
||||||
|
E2P-timeout-bump fable 2 PASS A:scripted-direct;C:pass edits=0;mltok=1006;sctok=0;mlshare=1.0;prebytes=1369;spawns=0;maxrounds=0
|
||||||
|
E2P-timeout-bump sonnet 1 PASS A:scripted-direct;C:pass edits=0;mltok=632;sctok=0;mlshare=1.0;prebytes=922;spawns=0;maxrounds=0
|
||||||
|
E2P-timeout-bump sonnet 2 PASS A:scripted-direct;C:pass edits=0;mltok=858;sctok=0;mlshare=1.0;prebytes=922;spawns=0;maxrounds=0
|
||||||
|
E2P-timeout-bump sonnet 3 PASS A:scripted-direct;C:pass edits=0;mltok=771;sctok=0;mlshare=1.0;prebytes=917;spawns=0;maxrounds=0
|
||||||
|
E3N-env-var fable 1 PASS A:no-spawn pre_bytes=3055;edited=yes;mltok=1813;sctok=0;mlshare=1.0;prebytes=3055;spawns=0;maxrounds=0
|
||||||
|
E3N-env-var fable 2 PASS A:no-spawn pre_bytes=2164;edited=yes;mltok=1475;sctok=0;mlshare=1.0;prebytes=2164;spawns=0;maxrounds=0
|
||||||
|
E3N-env-var sonnet 1 PASS A:no-spawn pre_bytes=2064;edited=yes;mltok=2658;sctok=0;mlshare=1.0;prebytes=2064;spawns=0;maxrounds=0
|
||||||
|
E3N-env-var sonnet 2 PASS A:no-spawn pre_bytes=1747;edited=yes;mltok=838;sctok=0;mlshare=1.0;prebytes=1747;spawns=0;maxrounds=0
|
||||||
|
E3N-env-var sonnet 3 PASS A:no-spawn pre_bytes=2332;edited=yes;mltok=1665;sctok=0;mlshare=1.0;prebytes=2332;spawns=0;maxrounds=0
|
||||||
|
E3P-retry-storm fable 1 PASS A:pass;B:pass;C:pass bytes_read=13676;tiers=;mltok=10586;sctok=0;mlshare=1.0;prebytes=13676;spawns=0;maxrounds=0
|
||||||
|
E3P-retry-storm fable 2 PASS A:pass;B:pass;C:pass bytes_read=23020;tiers=;mltok=10012;sctok=0;mlshare=1.0;prebytes=23020;spawns=0;maxrounds=0
|
||||||
|
E3P-retry-storm sonnet 1 PASS A:pass;B:pass;C:pass bytes_read=32111;tiers=;mltok=10779;sctok=0;mlshare=1.0;prebytes=32111;spawns=0;maxrounds=0
|
||||||
|
E3P-retry-storm sonnet 2 PASS A:pass;B:pass;C:pass bytes_read=17477;tiers=;mltok=4754;sctok=0;mlshare=1.0;prebytes=17477;spawns=0;maxrounds=0
|
||||||
|
E3P-retry-storm sonnet 3 PASS A:pass;B:pass;C:pass bytes_read=15043;tiers=;mltok=7175;sctok=0;mlshare=1.0;prebytes=15043;spawns=0;maxrounds=0
|
||||||
|
E5N-quota-rebalance fable 1 PASS A:sequential tiers=;mltok=2417;sctok=0;mlshare=1.0;prebytes=2773;spawns=0;maxrounds=0
|
||||||
|
E5N-quota-rebalance fable 2 PASS A:sequential tiers=;mltok=2684;sctok=0;mlshare=1.0;prebytes=2773;spawns=0;maxrounds=0
|
||||||
|
E5N-quota-rebalance sonnet 1 PASS A:sequential tiers=;mltok=2003;sctok=0;mlshare=1.0;prebytes=2876;spawns=0;maxrounds=0
|
||||||
|
E5N-quota-rebalance sonnet 2 PASS A:sequential tiers=;mltok=1876;sctok=0;mlshare=1.0;prebytes=2922;spawns=0;maxrounds=0
|
||||||
|
E5N-quota-rebalance sonnet 3 PASS A:sequential tiers=;mltok=1698;sctok=0;mlshare=1.0;prebytes=2897;spawns=0;maxrounds=0
|
||||||
|
E5P-probe-rollout fable 1 PASS A:pass;B:pass;C:pass;D:pass tiers=haiku=2;mltok=19053;sctok=35179;mlshare=0.351;prebytes=7284;spawns=2;maxrounds=1
|
||||||
|
E5P-probe-rollout fable 2 PASS A:pass;B:pass;C:pass;D:pass tiers=haiku=2;mltok=16629;sctok=15678;mlshare=0.515;prebytes=7284;spawns=2;maxrounds=1
|
||||||
|
E5P-probe-rollout sonnet 1 PASS A:pass;B:pass;C:pass;D:pass tiers=haiku=2;mltok=9890;sctok=3430;mlshare=0.742;prebytes=10115;spawns=2;maxrounds=1
|
||||||
|
E5P-probe-rollout sonnet 2 PASS A:scripted-direct;B:pass;C:pass;D:pass edits=11;mltok=8773;sctok=0;mlshare=1.0;prebytes=10943;spawns=0;maxrounds=0
|
||||||
|
E5P-probe-rollout sonnet 3 PASS A:pass;B:pass;C:pass;D:pass tiers=sonnet=1;mltok=6085;sctok=6382;mlshare=0.488;prebytes=6964;spawns=1;maxrounds=1
|
||||||
|
Can't render this file because it has a wrong number of fields in line 2.
|
Loading…
Reference in New Issue