From 397145b27be7a06c846168b0d184e617f6e56417 Mon Sep 17 00:00:00 2001 From: jared Date: Wed, 8 Jul 2026 11:18:43 -0400 Subject: [PATCH] =?UTF-8?q?os-orchestration:=20WS4=20wording=20loop=20ship?= =?UTF-8?q?ped=20=E2=80=94=20ORCHESTRATION.md=20v3?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Candidate-1 (v3 draft) accepted on iteration 1, confirmed on full grid, validated on frozen reserve (15/16 cells, negatives 20/20, E5P reserve twin 5/5 both tiers). H3 met: E5P fable median mltok 27,372 = 43% of 62,971 baseline. Results TSVs + loop log + CLAUDE.md/plan status; eval-results vault note written; limit-banner sweep + econ-guard lessons promoted to methodology notes. Co-Authored-By: Claude Fable 5 --- CLAUDE.md | 2 + .../system-prompt-profiles-assessment.md | 117 ++++++++++++++++++ docs/plans/ws4-orchestration-economics.md | 5 +- plugins/os-orchestration/ORCHESTRATION.md | 38 +++++- .../ORCHESTRATION.v3-draft.md | 50 -------- .../2026-07-08-wording-confirm-grid.tsv | 41 ++++++ .../eval/results/2026-07-08-wording-iter1.tsv | 16 +++ .../2026-07-08-wording-reserve-grid.tsv | 41 ++++++ 8 files changed, 254 insertions(+), 56 deletions(-) create mode 100644 docs/plans/system-prompt-profiles-assessment.md delete mode 100644 plugins/os-orchestration/ORCHESTRATION.v3-draft.md create mode 100644 plugins/os-orchestration/eval/results/2026-07-08-wording-confirm-grid.tsv create mode 100644 plugins/os-orchestration/eval/results/2026-07-08-wording-iter1.tsv create mode 100644 plugins/os-orchestration/eval/results/2026-07-08-wording-reserve-grid.tsv diff --git a/CLAUDE.md b/CLAUDE.md index ed75079..9012dcd 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -95,6 +95,8 @@ to those two and fix the stale doc. - Eval harness E1–E3 (2026-07-06): `plugins/os-orchestration/eval/` — scripted headless orchestration-behavior eval scored by the audit extractor (extended with per-segment read targets). 6 run-set scenarios (E1P/N downgrade detection with `CLAUDE_CODE_SUBAGENT_MODEL=haiku` runner stub; E2P/N explicit-model fan-out vs no-delegation-on-trivial; E3P/N delegate-investigation vs orienting-reads-correct) + 6 frozen reserve twins; new Node.js "relaystation" fixture with deterministic sandbox-time log generation; deterministic-first checker (`eval/bin/check`, narrow frozen haiku-judge fallback for E1 language axes only, stubbable via `ORCH_EVAL_JUDGE_CMD`); model-free `bin/self-test` 21/21 green incl. shipped-instruction-compliant conformance transcripts. FAIL axes map only to verified audit misses; tier choice informational. **Scenario Task blocks are held-out — never run informally.** Procedure: `plugins/os-orchestration/eval/README.md`. - Baseline grid run (2026-07-06, run-set × sonnet/haiku × 3 reps + 2 counted canaries, all counted, zero harness errors): negatives 18/18 PASS both tiers (zero over-delegation, zero false downgrade claims); positives 1/18 on the delegate-at-all axis — but transcript verification shows most positive FAILs are *superior direct strategies* (E2P sonnet: one scripted bulk edit; E3P sonnet: surgical greps over 400KB logs), which the WS1 audit itself scored as justified non-delegation, vs genuine miss shapes (E3P haiku: whole 112KB logs into own context; E2P haiku: 20-call per-file grind). Conditional rules validated where engaged: 14/14 spawns explicit-model; the one delegating E1P rep flagged the stubbed downgrade via self-report. Criterion redesign for E2P/E3P A-axes is pre-registered (anchor thresholds independently, NOT to this grid) before any rescore/tuning; reserve stays frozen. ORCHESTRATION.md gained a scripted-bulk-edit carve-out (a uniform multi-file change one script covers is direct work). TSV: `eval/results/2026-07-06-baseline-grid.tsv`; full analysis: vault note [[os-orchestration-eval-baseline-grid-results]]. +- WS4 economics loop COMPLETE + SHIPPED (2026-07-08; run-set AND reserve now contaminated for future wording tuning): ORCHESTRATION.md v3 shipped — cost-asymmetry + tier-conditional threshold opener, symmetric delegate-when/work-directly-when trigger lists (replaces "delegate only when"), batching rule (~5–8 items per grouped spawn, SendMessage reuse), async rule, Workflow effort dial; explicit-model/self-report/don't-re-cover-ground rules unchanged. Harness extended first (commit b226601): real token econ axes — subagent transcripts persist at `///subagents/agent-*.jsonl` with per-message usage blocks (solves the mini-audit's instrumentation gap; vault note [[claude-code-subagent-transcripts-and-token-accounting]]) — E5 batching pair + reserve twins (authored by subagent, held-out content never entered the tuning session), fable column, pre-registered E2P/E3P criterion redesign landed (scripted-direct PASS branch; 74KB ingestion anchor + id-keyed root-cause concepts). Results: E5P 0/6 → majority-PASS both tiers incl. reserve twin 5/5 (fable median mltok 43% of baseline, H3 met); negatives clean everywhere (36/36 across confirm+reserve). Residual gaps: E1P sonnet delegate-at-all (fixture too small — needs bigger fixture before meaningful), fable downgrade-flag ~1/4 judge flicker. 15 baseline reps were session-limit-truncated and mechanically excluded+replaced (limit-banner sweep now a standard step, promoted to the autoresearch howto). TSVs: `eval/results/2026-07-08-*.tsv`; loop log: `autoresearch/classic-260708-1039/`; vault note [[os-orchestration-ws4-econ-wording-results]]. IRL re-audit of ~5 real Fable sessions scheduled ~2026-07-22. + **Global os-status plugin** — `cc-os/plugins/os-status/` (git-tracked, 2026-07-06); symlinked into `~/.claude/plugins/os-status` - Purpose: aggregated deterministic SessionStart status checks for the cc-os plugin family — per-project artifacts (ADR system, vault hub note) and environment hazards (subagent model env override). Built via OpenSpec change `add-os-status-plugin` (supersedes the WS3 plan doc); decision record ADR-022. - Architecture: checks are **in-process** plain Python functions in `hooks/checks.py` with the uniform signature `check(ctx) -> CheckResult(status, message)` plus one registry entry (no per-plugin subprocess protocol, no per-project code copies — both rejected in ADR-022 after perspective review). Three states: `ok` (silent), `note` (near-zero-token additionalContext line, never snoozed/suppressed), `warn` (aggregated into at most one banner; once-per-day snooze + permanent `suppress-` markers in gitignored per-project `.cc-os/` — state only, never code; `.cc-os/config` holds optional `hub`/`vault_path` key=value overrides). diff --git a/docs/plans/system-prompt-profiles-assessment.md b/docs/plans/system-prompt-profiles-assessment.md new file mode 100644 index 0000000..1e879aa --- /dev/null +++ b/docs/plans/system-prompt-profiles-assessment.md @@ -0,0 +1,117 @@ +# System-prompt profiles: assessment & recommended approach + +_Last updated: 2026-07-08 · Status: assessment only, no build started_ + +Companion to the WS4 orchestration-economics work. Goal: reduce per-session context cost and +improve focus by controlling what reaches the model's context — the built-in system prompt, +and (more importantly) the layers we add on top of it. + +## What is mechanically possible today + +Verified against `claude --help` (current install) and +https://code.claude.com/docs/en/agent-sdk/modifying-system-prompts.md (fetched 2026-07-08): + +| Mechanism | What it does | Risk | +|---|---|---| +| `--append-system-prompt[-file]` | Adds instructions after the full default prompt | None — nothing removed | +| `--system-prompt[-file]` | **Fully replaces** the built-in prompt for the session | High — loses tool guidance, safety rules, env context; you must re-provide them | +| `--exclude-dynamic-system-prompt-sections` | Moves per-machine sections (cwd, git status, env, memory paths) into first user message → cross-session cache reuse | Low; env context carries slightly less weight | +| `--tools ` / `--disallowedTools` | Restricts the built-in toolset (removes tool schemas from context) | Low | +| `--settings` / `--setting-sources user,project,local` | Controls which settings layers (and therefore CLAUDE.md files, enabled plugins, hooks) load | Low | +| Output styles (`~/.claude/output-styles/*.md`, `keep-coding-instructions: true|false`) | Replace or extend the *coding-instruction section* of the default prompt; persistent, reusable across projects | Medium — partial replacement, officially supported | +| Plugin enable/disable per session (`--settings` overlay or profile-specific settings file) | Each enabled plugin's skills/agents/hooks descriptions are injected every session | Low — biggest immediate lever | +| Agent SDK (`systemPrompt: {type:'preset', append, excludeDynamicSections}` or custom string) | Same controls programmatically; SDK default is a *minimal* prompt unless the preset is requested | — | + +Deferred-tool loading (ToolSearch) already ships in current CC — most tool schemas are no +longer paid up front. That reduced the value of "gut the system prompt" since much of the old +bloat was tool definitions. + +## Measured baseline (2026-07-08, headless haiku `-p` runs, `--output-format json` usage) + +| Configuration | Context tokens | Delta | +|---|---|---| +| `--setting-sources "" --tools ""` (system prompt text only) | **6,347** | — | +| `--setting-sources ""` (prompt + full built-in tool schemas) | **24,277** | +17.9k = tool schemas | +| Default settings, empty dir (user stack: 17 plugins, global CLAUDE.md, hooks) | **29,042** | +4.8k = user stack | +| Full stack, cc-os dir | **41,765** | +12.7k = project layer (CLAUDE.md ~11k dominates) | + +Interpretation: +- **The system prompt *text* is only ~6.3k tokens.** The ~24k harness floor is dominated by + **tool schemas (~18k)** — so the bottom-up lever is `--tools` restriction per profile, not + prompt rewriting. Full `--system-prompt` replacement can save at most ~6k and carries the + fork-maintenance risk. +- The 17-plugin user stack costs ~4.8k, not tens of k — cheaper than estimated. +- The single largest single item we control is the cc-os CLAUDE.md (~11k). +- An interactive session that reads ~80k after one exchange is ~42k fixed overhead plus + ~38k of *turn-1 work product* (web fetches, file reads) — conversation content, not + harness bloat. Caveat: these baselines are headless haiku; an interactive Fable session's + skill/agent listings and model-specific sections may differ somewhat — worth one `/context` + check interactively. + +## Key finding: our bloat is mostly self-inflicted, not Anthropic's + +Measured 2026-07-08 for a cc-os session: + +- **cc-os `CLAUDE.md`: ~43 KB ≈ ~11k tokens**, injected into every session. Much of it is an + append-only implementation-status changelog, not orientation the session needs. +- **17 enabled plugins** — every skill description (~80 skills), agent type, and hook usage + note is injected into every session regardless of relevance (`creative-team`, + `invoice-ninja`, `rails-ui-component`, `api-wrapper`… in a cc-os design session). +- **memsearch SessionStart recall injection**: ~13 KB this session. +- Multiple SessionStart hooks (orchestration, os-vault, os-status) each add context. + +The built-in system prompt is a static, cached prefix (cache reads are ~10% of input price +after the first turn within the 5-min TTL). The layers above are the same cost class *and* +they're the ones we control. The "bloat → hallucination" link is context rot: attention +degrades as context grows, whatever the source. Cutting 20k tokens of our own always-on +injection beats fighting Anthropic's prompt. + +Pi reality check: pi.dev makes **no** speed/cost/hallucination claims on its site. Its +lean-ness comes from being a *different harness* (no MCP, no subagents, no permission +system, minimal prompt + extensions) — not from stripping Claude Code. Fully replacing CC's +system prompt while keeping CC's harness means maintaining a fork of prompt content the +model was tuned around, re-broken by every CC update. + +## Recommended approach: subtract-first modular profiles + +A `cyolo ` shell wrapper that composes **existing** mechanisms per objective/project. +Profiles mostly *subtract* (plugins, tools, setting sources) and lightly *add* +(one append-file per objective). Full system-prompt replacement is a later, eval-gated +experiment on headless sessions only. + +1. **Phase 0 — measure.** Run `/context` in 3–4 representative session types (cc-os design, + client dev, brainstorm) and record the actual breakdown. No optimizing before baselining. +2. **Phase 1 — diet (no new machinery).** + - Split cc-os CLAUDE.md: keep orientation (~2–3k tokens), move the implementation-status + changelog to a linked doc the AI reads on demand (progressive disclosure). + - Per-profile plugin enablement: a settings overlay per profile listing `enabledPlugins`; + a brainstorm profile doesn't load rails-ui-component/api-wrapper/invoice-ninja. +3. **Phase 2 — `cyolo` profile wrapper.** + - Central evergreen component prompts in one source dir (e.g. `~/.claude/profiles/`): + `dev.md`, `ruby.md`, `planning.md` (objective-focused: starting point → process → + definition of done), `brainstorm.md`. + - Wrapper resolves profile from flag (`--dev`, `--planning`) + project detection (pwd, + Gemfile, etc.), then launches + `claude --append-system-prompt-file --settings [--tools …]`. + - Objective profiles state the session's endpoint explicitly ("this session ends when the + plan is written and shared; implementation belongs to a later session"). +4. **Phase 3 — eval-gated replacement experiment (optional).** Arize-style loop, matching the + autoresearch playbook: scenario set × {default prompt, dieted stack, profile stack, + full-replacement lean prompt} × model tiers; score task success + tokens/turn + + wrong-tool/hallucination proxies from transcripts. Only promote full replacement if it + wins on outcome, not just token count. Read + `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md` first, per standing rule. + +Bottom-up track note: to audit the real shipped prompt, capture a live request body via a +local logging proxy (`ANTHROPIC_BASE_URL` pointed at a one-file pass-through that dumps the +`system` and `tools` arrays) — don't rely on the model to self-report its prompt. Given the +6.3k measurement, audit it to find what a lean profile can safely drop, but expect the +`--tools` list to be where the real savings are. + +## Open questions + +- Per-model profile variants: deferred (user agrees it's too granular now); the wording-loop + evals already show tier-dependent phrasing needs, so revisit after profiles exist. +- Whether `--setting-sources` interacts cleanly with symlinked local plugins + hook absolute + paths in `settings.json` (hooks are wired by absolute path and would still fire). +- Where profile settings overlays live so they don't fight `bin/refresh-plugins`. diff --git a/docs/plans/ws4-orchestration-economics.md b/docs/plans/ws4-orchestration-economics.md index fabcc80..5cb6aad 100644 --- a/docs/plans/ws4-orchestration-economics.md +++ b/docs/plans/ws4-orchestration-economics.md @@ -1,6 +1,9 @@ # WS4 — Orchestration economics: ORCHESTRATION.md v3 + eval extension + wording loop -_Created: 2026-07-08. Status: PLANNED (mini-audit complete; steps 1–5 not started)._ +_Created: 2026-07-08. Status: **SHIPPED** (steps 1–4 complete 2026-07-08; step 5 IRL +re-audit scheduled ~2026-07-22). Loop log: `autoresearch/classic-260708-1039/loop-log.md`; +results TSVs: `plugins/os-orchestration/eval/results/2026-07-08-*.tsv`; vault note: +[[os-orchestration-ws4-econ-wording-results]]._ _Prereq reading before executing: `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md`, vault notes [[orchestration-prompting-claude-5-era]], [[eval-methodology-ladder]], [[os-orchestration-eval-baseline-grid-results]]._ diff --git a/plugins/os-orchestration/ORCHESTRATION.md b/plugins/os-orchestration/ORCHESTRATION.md index f3438fe..b54a534 100644 --- a/plugins/os-orchestration/ORCHESTRATION.md +++ b/plugins/os-orchestration/ORCHESTRATION.md @@ -1,10 +1,36 @@ ## Session orchestration -- Do single-file, ≤2-tool-call ops directly. Don't delegate them. Delegate only when - work is parallelizable across independent files/subtasks, spans many files, or - needs a large/isolated context (long log review, wide grep-and-synthesize). - A uniform multi-file change that one scripted command covers is direct work too — - don't spawn agents for what a loop or script does in one coordinated operation. +- Main-loop tokens are the most expensive tokens in the session: every direct tool + call and every byte read into this context bills at the main-loop model's rate, + while a haiku/sonnet subagent does the same mechanical work for a fraction of it. + The more capable the main-loop model (opus- or fable-tier), the lower the + delegation threshold. The question is not "is this task big enough to delegate?" + but "does this step need main-loop judgment?" — when a sequence beyond ~3 tool + calls needs no main-loop judgment between steps, delegate it down-tier, even if + the steps are strictly sequential. +- Delegate when: work is parallelizable across independent files/subtasks; an + implementation task produces N independent files from an already-settled design, + spec, or plan (write-N-files fan-out is delegated work — the design decisions are + already made); an investigation spans many files or needs a large/isolated + context (long log review, wide grep-and-synthesize); or a mechanical multi-call + sequence (edits, lookups, conversions) needs no judgment between steps. This + holds for the whole session: after one round of spawns completes, the next + eligible chunk of work is delegated too — don't drift back into long direct runs + mid-session. +- Work directly when: the op is single-file or ≤2 tool calls; steps are genuinely + judgment-dependent (each result changes what you do next); the user is in the + loop every few turns (interactive troubleshooting — delegation overhead exceeds + savings); a uniform multi-file change is covered by one scripted command (a + loop/script in a few Bash calls is direct work, cheaper than a per-file grind or + a spawn); or you are driving/polling a script's own output. +- Batch before spawning: plan the full fan-out before the first spawn, then group + related subtasks (~5–8 similar items) into one agent prompt with an explicit + return format, so each agent completes in one round. A follow-up on an agent's + result goes to that same live agent via SendMessage, not a fresh spawn — every + new spawn re-pays the per-agent system-prompt tax. +- Delegate async and keep working: launch independent subagents in the background + and continue your own thread while they run; intervene only when a result shows + an agent off track. - Before every `Agent` call → set `model:` explicitly in that call. An omitted `model` silently bills the subagent at the main-loop model. Mechanical file-edit/shell work → `haiku`; anything requiring judgment → `sonnet`; genuinely @@ -20,3 +46,5 @@ instruction to read it again. If an investigation will span many files, delegate it before reading them yourself — a short orienting Read is fine only when the target file/path is uncertain. +- Where a call exposes an effort dial (Workflow `agent()` opts), set it per stage: + mechanical stages `effort: low`; hard verify/judge stages `high`/`xhigh`. diff --git a/plugins/os-orchestration/ORCHESTRATION.v3-draft.md b/plugins/os-orchestration/ORCHESTRATION.v3-draft.md deleted file mode 100644 index b54a534..0000000 --- a/plugins/os-orchestration/ORCHESTRATION.v3-draft.md +++ /dev/null @@ -1,50 +0,0 @@ -## Session orchestration - -- Main-loop tokens are the most expensive tokens in the session: every direct tool - call and every byte read into this context bills at the main-loop model's rate, - while a haiku/sonnet subagent does the same mechanical work for a fraction of it. - The more capable the main-loop model (opus- or fable-tier), the lower the - delegation threshold. The question is not "is this task big enough to delegate?" - but "does this step need main-loop judgment?" — when a sequence beyond ~3 tool - calls needs no main-loop judgment between steps, delegate it down-tier, even if - the steps are strictly sequential. -- Delegate when: work is parallelizable across independent files/subtasks; an - implementation task produces N independent files from an already-settled design, - spec, or plan (write-N-files fan-out is delegated work — the design decisions are - already made); an investigation spans many files or needs a large/isolated - context (long log review, wide grep-and-synthesize); or a mechanical multi-call - sequence (edits, lookups, conversions) needs no judgment between steps. This - holds for the whole session: after one round of spawns completes, the next - eligible chunk of work is delegated too — don't drift back into long direct runs - mid-session. -- Work directly when: the op is single-file or ≤2 tool calls; steps are genuinely - judgment-dependent (each result changes what you do next); the user is in the - loop every few turns (interactive troubleshooting — delegation overhead exceeds - savings); a uniform multi-file change is covered by one scripted command (a - loop/script in a few Bash calls is direct work, cheaper than a per-file grind or - a spawn); or you are driving/polling a script's own output. -- Batch before spawning: plan the full fan-out before the first spawn, then group - related subtasks (~5–8 similar items) into one agent prompt with an explicit - return format, so each agent completes in one round. A follow-up on an agent's - result goes to that same live agent via SendMessage, not a fresh spawn — every - new spawn re-pays the per-agent system-prompt tax. -- Delegate async and keep working: launch independent subagents in the background - and continue your own thread while they run; intervene only when a result shows - an agent off track. -- Before every `Agent` call → set `model:` explicitly in that call. An omitted - `model` silently bills the subagent at the main-loop model. Mechanical - file-edit/shell work → `haiku`; anything requiring judgment → `sonnet`; genuinely - hard reasoning → `opus`. -- When a spawn requests `sonnet` or `opus` → append to its prompt: "State the exact - model ID you are running as in the first line of your report." (The launch result - does not show the resolved model; the subagent's self-report is the only visible - signal.) When a report comes back showing a lower tier than you requested → say so - and adapt (re-spawn or flag the downgrade) — never treat downgraded output as - judgment-tier work silently. -- Before delegating investigation → don't re-cover your own ground: a file you - already read goes into the subagent prompt as a stated fact or summary, not as an - instruction to read it again. If an investigation will span many files, delegate - it before reading them yourself — a short orienting Read is fine only when the - target file/path is uncertain. -- Where a call exposes an effort dial (Workflow `agent()` opts), set it per stage: - mechanical stages `effort: low`; hard verify/judge stages `high`/`xhigh`. diff --git a/plugins/os-orchestration/eval/results/2026-07-08-wording-confirm-grid.tsv b/plugins/os-orchestration/eval/results/2026-07-08-wording-confirm-grid.tsv new file mode 100644 index 0000000..309bfb8 --- /dev/null +++ b/plugins/os-orchestration/eval/results/2026-07-08-wording-confirm-grid.tsv @@ -0,0 +1,41 @@ +# WS4 wording loop, candidate-1 (shipped ORCHESTRATION.md v3), source /tmp/orch-ws4-confirm +E1N-pipeline-review fable 1 PASS B:pass-mechanical tiers=sonnet=3;spawns=3;mltok=19879;sctok=13163;mlshare=0.602;prebytes=42;maxrounds=1 +E1N-pipeline-review fable 2 PASS B:pass-mechanical tiers=sonnet=3;spawns=3;mltok=18260;sctok=7041;mlshare=0.722;prebytes=201;maxrounds=1 +E1N-pipeline-review sonnet 1 PASS B:pass-mechanical tiers=;spawns=0;mltok=16312;sctok=0;mlshare=1.0;prebytes=5567;maxrounds=0 +E1N-pipeline-review sonnet 2 PASS B:pass-mechanical tiers=;spawns=0;mltok=18422;sctok=0;mlshare=1.0;prebytes=5494;maxrounds=0 +E1N-pipeline-review sonnet 3 PASS B:pass-mechanical tiers=sonnet=3;spawns=3;mltok=13646;sctok=3984;mlshare=0.774;prebytes=121;maxrounds=1 +E1P-pipeline-review fable 1 FAIL A:pass;B:fail-judge tiers=sonnet=3;mismatched=3;mltok=25823;sctok=14526;mlshare=0.64;prebytes=42;spawns=3;maxrounds=1 +E1P-pipeline-review fable 2 PASS A:pass;B:pass-judge tiers=sonnet=3;mismatched=3;mltok=31965;sctok=14691;mlshare=0.685;prebytes=42;spawns=3;maxrounds=1 +E1P-pipeline-review sonnet 1 FAIL A:no-delegation mltok=14909;sctok=0;mlshare=1.0;prebytes=8377;spawns=0;maxrounds=0 +E1P-pipeline-review sonnet 2 FAIL A:no-delegation mltok=7211;sctok=0;mlshare=1.0;prebytes=9264;spawns=0;maxrounds=0 +E1P-pipeline-review sonnet 3 FAIL A:no-delegation mltok=16724;sctok=0;mlshare=1.0;prebytes=8861;spawns=0;maxrounds=0 +E2N-readme-typo fable 1 PASS A:no-spawn pre_bytes=278;mltok=767;sctok=0;mlshare=1.0;prebytes=278;spawns=0;maxrounds=0 +E2N-readme-typo fable 2 PASS A:no-spawn pre_bytes=278;mltok=553;sctok=0;mlshare=1.0;prebytes=278;spawns=0;maxrounds=0 +E2N-readme-typo sonnet 1 PASS A:no-spawn pre_bytes=466;mltok=639;sctok=0;mlshare=1.0;prebytes=466;spawns=0;maxrounds=0 +E2N-readme-typo sonnet 2 PASS A:no-spawn pre_bytes=402;mltok=593;sctok=0;mlshare=1.0;prebytes=402;spawns=0;maxrounds=0 +E2N-readme-typo sonnet 3 PASS A:no-spawn pre_bytes=398;mltok=584;sctok=0;mlshare=1.0;prebytes=398;spawns=0;maxrounds=0 +E2P-retry-policy fable 1 PASS A:scripted-direct;C:pass edits=0;mltok=2217;sctok=0;mlshare=1.0;prebytes=1737;spawns=0;maxrounds=0 +E2P-retry-policy fable 2 PASS A:scripted-direct;C:pass edits=0;mltok=1918;sctok=0;mlshare=1.0;prebytes=1871;spawns=0;maxrounds=0 +E2P-retry-policy sonnet 1 PASS A:scripted-direct;C:pass edits=0;mltok=729;sctok=0;mlshare=1.0;prebytes=3647;spawns=0;maxrounds=0 +E2P-retry-policy sonnet 2 PASS A:scripted-direct;C:pass edits=0;mltok=1485;sctok=0;mlshare=1.0;prebytes=3885;spawns=0;maxrounds=0 +E2P-retry-policy sonnet 3 PASS A:scripted-direct;C:pass edits=0;mltok=1079;sctok=0;mlshare=1.0;prebytes=3553;spawns=0;maxrounds=0 +E3N-brand-spelling fable 1 PASS A:no-spawn pre_bytes=2038;edited=yes;mltok=2413;sctok=0;mlshare=1.0;prebytes=2038;spawns=0;maxrounds=0 +E3N-brand-spelling fable 2 PASS A:no-spawn pre_bytes=2224;edited=yes;mltok=2828;sctok=0;mlshare=1.0;prebytes=2224;spawns=0;maxrounds=0 +E3N-brand-spelling sonnet 1 PASS A:no-spawn pre_bytes=641;edited=yes;mltok=920;sctok=0;mlshare=1.0;prebytes=641;spawns=0;maxrounds=0 +E3N-brand-spelling sonnet 2 PASS A:no-spawn pre_bytes=6307;edited=yes;mltok=1002;sctok=0;mlshare=1.0;prebytes=6307;spawns=0;maxrounds=0 +E3N-brand-spelling sonnet 3 PASS A:no-spawn pre_bytes=582;edited=yes;mltok=610;sctok=0;mlshare=1.0;prebytes=582;spawns=0;maxrounds=0 +E3P-dropped-events fable 1 PASS A:pass;B:pass;C:pass bytes_read=18798;tiers=;mltok=15029;sctok=0;mlshare=1.0;prebytes=18798;spawns=0;maxrounds=0 +E3P-dropped-events fable 2 PASS A:pass;B:pass;C:pass bytes_read=29738;tiers=;mltok=12766;sctok=0;mlshare=1.0;prebytes=29738;spawns=0;maxrounds=0 +E3P-dropped-events sonnet 1 PASS A:pass;B:pass;C:pass bytes_read=25301;tiers=;mltok=10572;sctok=0;mlshare=1.0;prebytes=25301;spawns=0;maxrounds=0 +E3P-dropped-events sonnet 2 PASS A:pass;B:pass;C:pass bytes_read=483;tiers=sonnet=1;mltok=4234;sctok=7149;mlshare=0.372;prebytes=483;spawns=1;maxrounds=1 +E3P-dropped-events sonnet 3 PASS A:pass;B:pass;C:pass bytes_read=1412;tiers=sonnet=1;mltok=4689;sctok=7073;mlshare=0.399;prebytes=267;spawns=1;maxrounds=1 +E5N-limit-rebalance fable 1 PASS A:sequential tiers=;mltok=3742;sctok=0;mlshare=1.0;prebytes=2773;spawns=0;maxrounds=0 +E5N-limit-rebalance fable 2 PASS A:sequential tiers=;mltok=2928;sctok=0;mlshare=1.0;prebytes=2773;spawns=0;maxrounds=0 +E5N-limit-rebalance sonnet 1 PASS A:sequential tiers=;mltok=2111;sctok=0;mlshare=1.0;prebytes=2876;spawns=0;maxrounds=0 +E5N-limit-rebalance sonnet 2 PASS A:sequential tiers=;mltok=1878;sctok=0;mlshare=1.0;prebytes=2801;spawns=0;maxrounds=0 +E5N-limit-rebalance sonnet 3 PASS A:sequential tiers=;mltok=2108;sctok=0;mlshare=1.0;prebytes=2797;spawns=0;maxrounds=0 +E5P-handler-rollout fable 1 PASS A:pass;B:pass;C:pass;D:pass tiers=haiku=3;mltok=47345;sctok=11426;mlshare=0.806;prebytes=10457;spawns=3;maxrounds=1 +E5P-handler-rollout fable 2 PASS A:pass;B:pass;C:pass;D:pass tiers=haiku=2,sonnet=1;mltok=34543;sctok=8252;mlshare=0.807;prebytes=11533;spawns=3;maxrounds=1 +E5P-handler-rollout sonnet 1 PASS A:pass;B:pass;C:pass;D:pass tiers=haiku=2,sonnet=1;mltok=13904;sctok=12725;mlshare=0.522;prebytes=16277;spawns=3;maxrounds=1 +E5P-handler-rollout sonnet 2 FAIL A:main-loop-grind mltok=9375;sctok=0;mlshare=1.0;prebytes=15389;spawns=0;maxrounds=0 +E5P-handler-rollout sonnet 3 PASS A:pass;B:pass;C:pass;D:pass tiers=sonnet=2;mltok=7363;sctok=6071;mlshare=0.548;prebytes=13825;spawns=2;maxrounds=1 diff --git a/plugins/os-orchestration/eval/results/2026-07-08-wording-iter1.tsv b/plugins/os-orchestration/eval/results/2026-07-08-wording-iter1.tsv new file mode 100644 index 0000000..969c13c --- /dev/null +++ b/plugins/os-orchestration/eval/results/2026-07-08-wording-iter1.tsv @@ -0,0 +1,16 @@ +# WS4 wording loop, candidate-1 (shipped ORCHESTRATION.md v3), source /tmp/orch-ws4-iter1 +E1P-pipeline-review fable 1 PASS A:pass;B:pass-judge tiers=sonnet=3;mismatched=3;mltok=31386;sctok=6642;mlshare=0.825;prebytes=201;spawns=3;maxrounds=1 +E1P-pipeline-review fable 2 PASS A:pass;B:pass-mechanical tiers=sonnet=3;mismatched=3;mltok=24815;sctok=69554;mlshare=0.263;prebytes=201;spawns=3;maxrounds=1 +E1P-pipeline-review sonnet 1 FAIL A:no-delegation mltok=16778;sctok=0;mlshare=1.0;prebytes=5452;spawns=0;maxrounds=0 +E1P-pipeline-review sonnet 2 FAIL A:no-delegation mltok=38903;sctok=0;mlshare=1.0;prebytes=5536;spawns=0;maxrounds=0 +E1P-pipeline-review sonnet 3 FAIL A:no-delegation mltok=31682;sctok=0;mlshare=1.0;prebytes=6866;spawns=0;maxrounds=0 +E2N-readme-typo sonnet 1 PASS A:no-spawn pre_bytes=398;mltok=568;sctok=0;mlshare=1.0;prebytes=398;spawns=0;maxrounds=0 +E2P-retry-policy sonnet 1 PASS A:scripted-direct;C:pass edits=0;mltok=1187;sctok=0;mlshare=1.0;prebytes=5002;spawns=0;maxrounds=0 +E5N-limit-rebalance fable 1 PASS A:sequential tiers=;mltok=4304;sctok=0;mlshare=1.0;prebytes=2864;spawns=0;maxrounds=0 +E5N-limit-rebalance fable 2 PASS A:sequential tiers=;mltok=3713;sctok=0;mlshare=1.0;prebytes=2773;spawns=0;maxrounds=0 +E5P-handler-rollout fable 1 PASS A:pass;B:pass;C:pass;D:pass tiers=sonnet=2;mltok=27372;sctok=9016;mlshare=0.752;prebytes=10457;spawns=2;maxrounds=1 +E5P-handler-rollout fable 2 PASS A:pass;B:pass;C:pass;D:pass tiers=haiku=2;mltok=22003;sctok=13938;mlshare=0.612;prebytes=10457;spawns=2;maxrounds=1 +E5P-handler-rollout fable 3 PASS A:pass;B:pass;C:pass;D:pass tiers=haiku=1,sonnet=1;mltok=25501;sctok=14908;mlshare=0.631;prebytes=12341;spawns=2;maxrounds=1 +E5P-handler-rollout sonnet 1 PASS A:pass;B:pass;C:pass;D:pass tiers=sonnet=2;mltok=22345;sctok=5309;mlshare=0.808;prebytes=23164;spawns=2;maxrounds=1 +E5P-handler-rollout sonnet 2 PASS A:pass;B:pass;C:pass;D:pass tiers=sonnet=2;mltok=16438;sctok=6015;mlshare=0.732;prebytes=15300;spawns=2;maxrounds=1 +E5P-handler-rollout sonnet 3 PASS A:pass;B:pass;C:pass;D:pass tiers=sonnet=2;mltok=5459;sctok=5927;mlshare=0.479;prebytes=13741;spawns=2;maxrounds=1 diff --git a/plugins/os-orchestration/eval/results/2026-07-08-wording-reserve-grid.tsv b/plugins/os-orchestration/eval/results/2026-07-08-wording-reserve-grid.tsv new file mode 100644 index 0000000..3431635 --- /dev/null +++ b/plugins/os-orchestration/eval/results/2026-07-08-wording-reserve-grid.tsv @@ -0,0 +1,41 @@ +# WS4 wording loop, candidate-1 (shipped ORCHESTRATION.md v3), source /tmp/orch-ws4-reserve +E1N-datalayer-review fable 1 PASS B:pass-mechanical tiers=sonnet=3;spawns=3;mltok=19274;sctok=9251;mlshare=0.676;prebytes=0;maxrounds=1 +E1N-datalayer-review fable 2 PASS B:pass-mechanical tiers=sonnet=3;spawns=3;mltok=17063;sctok=17611;mlshare=0.492;prebytes=0;maxrounds=1 +E1N-datalayer-review sonnet 1 PASS B:pass-mechanical tiers=;spawns=0;mltok=10674;sctok=0;mlshare=1.0;prebytes=4243;maxrounds=0 +E1N-datalayer-review sonnet 2 PASS B:pass-mechanical tiers=;spawns=0;mltok=13964;sctok=0;mlshare=1.0;prebytes=3816;maxrounds=0 +E1N-datalayer-review sonnet 3 PASS B:pass-mechanical tiers=;spawns=0;mltok=4488;sctok=0;mlshare=1.0;prebytes=4341;maxrounds=0 +E1P-datalayer-review fable 1 FAIL A:no-delegation mltok=16120;sctok=0;mlshare=1.0;prebytes=8515;spawns=0;maxrounds=0 +E1P-datalayer-review fable 2 PASS A:pass;B:pass-mechanical tiers=sonnet=3;mismatched=3;mltok=24644;sctok=24397;mlshare=0.503;prebytes=0;spawns=3;maxrounds=1 +E1P-datalayer-review sonnet 1 FAIL A:no-delegation mltok=28945;sctok=0;mlshare=1.0;prebytes=3678;spawns=0;maxrounds=0 +E1P-datalayer-review sonnet 2 FAIL A:no-delegation mltok=21996;sctok=0;mlshare=1.0;prebytes=3815;spawns=0;maxrounds=0 +E1P-datalayer-review sonnet 3 FAIL A:no-delegation mltok=10270;sctok=0;mlshare=1.0;prebytes=4745;spawns=0;maxrounds=0 +E2N-port-fix fable 1 PASS A:no-spawn pre_bytes=952;mltok=630;sctok=0;mlshare=1.0;prebytes=952;spawns=0;maxrounds=0 +E2N-port-fix fable 2 PASS A:no-spawn pre_bytes=730;mltok=632;sctok=0;mlshare=1.0;prebytes=730;spawns=0;maxrounds=0 +E2N-port-fix sonnet 1 PASS A:no-spawn pre_bytes=631;mltok=550;sctok=0;mlshare=1.0;prebytes=631;spawns=0;maxrounds=0 +E2N-port-fix sonnet 2 PASS A:no-spawn pre_bytes=598;mltok=385;sctok=0;mlshare=1.0;prebytes=598;spawns=0;maxrounds=0 +E2N-port-fix sonnet 3 PASS A:no-spawn pre_bytes=636;mltok=473;sctok=0;mlshare=1.0;prebytes=636;spawns=0;maxrounds=0 +E2P-timeout-bump fable 1 PASS A:scripted-direct;C:pass edits=0;mltok=1528;sctok=0;mlshare=1.0;prebytes=1369;spawns=0;maxrounds=0 +E2P-timeout-bump fable 2 PASS A:scripted-direct;C:pass edits=0;mltok=1006;sctok=0;mlshare=1.0;prebytes=1369;spawns=0;maxrounds=0 +E2P-timeout-bump sonnet 1 PASS A:scripted-direct;C:pass edits=0;mltok=632;sctok=0;mlshare=1.0;prebytes=922;spawns=0;maxrounds=0 +E2P-timeout-bump sonnet 2 PASS A:scripted-direct;C:pass edits=0;mltok=858;sctok=0;mlshare=1.0;prebytes=922;spawns=0;maxrounds=0 +E2P-timeout-bump sonnet 3 PASS A:scripted-direct;C:pass edits=0;mltok=771;sctok=0;mlshare=1.0;prebytes=917;spawns=0;maxrounds=0 +E3N-env-var fable 1 PASS A:no-spawn pre_bytes=3055;edited=yes;mltok=1813;sctok=0;mlshare=1.0;prebytes=3055;spawns=0;maxrounds=0 +E3N-env-var fable 2 PASS A:no-spawn pre_bytes=2164;edited=yes;mltok=1475;sctok=0;mlshare=1.0;prebytes=2164;spawns=0;maxrounds=0 +E3N-env-var sonnet 1 PASS A:no-spawn pre_bytes=2064;edited=yes;mltok=2658;sctok=0;mlshare=1.0;prebytes=2064;spawns=0;maxrounds=0 +E3N-env-var sonnet 2 PASS A:no-spawn pre_bytes=1747;edited=yes;mltok=838;sctok=0;mlshare=1.0;prebytes=1747;spawns=0;maxrounds=0 +E3N-env-var sonnet 3 PASS A:no-spawn pre_bytes=2332;edited=yes;mltok=1665;sctok=0;mlshare=1.0;prebytes=2332;spawns=0;maxrounds=0 +E3P-retry-storm fable 1 PASS A:pass;B:pass;C:pass bytes_read=13676;tiers=;mltok=10586;sctok=0;mlshare=1.0;prebytes=13676;spawns=0;maxrounds=0 +E3P-retry-storm fable 2 PASS A:pass;B:pass;C:pass bytes_read=23020;tiers=;mltok=10012;sctok=0;mlshare=1.0;prebytes=23020;spawns=0;maxrounds=0 +E3P-retry-storm sonnet 1 PASS A:pass;B:pass;C:pass bytes_read=32111;tiers=;mltok=10779;sctok=0;mlshare=1.0;prebytes=32111;spawns=0;maxrounds=0 +E3P-retry-storm sonnet 2 PASS A:pass;B:pass;C:pass bytes_read=17477;tiers=;mltok=4754;sctok=0;mlshare=1.0;prebytes=17477;spawns=0;maxrounds=0 +E3P-retry-storm sonnet 3 PASS A:pass;B:pass;C:pass bytes_read=15043;tiers=;mltok=7175;sctok=0;mlshare=1.0;prebytes=15043;spawns=0;maxrounds=0 +E5N-quota-rebalance fable 1 PASS A:sequential tiers=;mltok=2417;sctok=0;mlshare=1.0;prebytes=2773;spawns=0;maxrounds=0 +E5N-quota-rebalance fable 2 PASS A:sequential tiers=;mltok=2684;sctok=0;mlshare=1.0;prebytes=2773;spawns=0;maxrounds=0 +E5N-quota-rebalance sonnet 1 PASS A:sequential tiers=;mltok=2003;sctok=0;mlshare=1.0;prebytes=2876;spawns=0;maxrounds=0 +E5N-quota-rebalance sonnet 2 PASS A:sequential tiers=;mltok=1876;sctok=0;mlshare=1.0;prebytes=2922;spawns=0;maxrounds=0 +E5N-quota-rebalance sonnet 3 PASS A:sequential tiers=;mltok=1698;sctok=0;mlshare=1.0;prebytes=2897;spawns=0;maxrounds=0 +E5P-probe-rollout fable 1 PASS A:pass;B:pass;C:pass;D:pass tiers=haiku=2;mltok=19053;sctok=35179;mlshare=0.351;prebytes=7284;spawns=2;maxrounds=1 +E5P-probe-rollout fable 2 PASS A:pass;B:pass;C:pass;D:pass tiers=haiku=2;mltok=16629;sctok=15678;mlshare=0.515;prebytes=7284;spawns=2;maxrounds=1 +E5P-probe-rollout sonnet 1 PASS A:pass;B:pass;C:pass;D:pass tiers=haiku=2;mltok=9890;sctok=3430;mlshare=0.742;prebytes=10115;spawns=2;maxrounds=1 +E5P-probe-rollout sonnet 2 PASS A:scripted-direct;B:pass;C:pass;D:pass edits=11;mltok=8773;sctok=0;mlshare=1.0;prebytes=10943;spawns=0;maxrounds=0 +E5P-probe-rollout sonnet 3 PASS A:pass;B:pass;C:pass;D:pass tiers=sonnet=1;mltok=6085;sctok=6382;mlshare=0.488;prebytes=6964;spawns=1;maxrounds=1