From c2098525fee8d680cabb4ebe958c1934e12690cb Mon Sep 17 00:00:00 2001 From: jared Date: Mon, 6 Jul 2026 18:35:40 -0400 Subject: [PATCH] os-orchestration: baseline grid results + scripted-bulk-edit carve-out Grid (36 reps + 2 canaries, all counted): negatives 18/18 both tiers, zero over-triggering; positives 1/18 on delegate-at-all, but transcript verification splits those FAILs into superior direct strategies (scripted bulk edit, surgical greps - the audit's own justified-non-delegation shape) vs genuine misses (haiku whole-file ingestion, per-file grind). Conditional rules validated where engaged: 14/14 spawns explicit-model, self-report downgrade flag worked. E2P/E3P criterion redesign pre-registered with independently-anchored thresholds; reserve untouched. Analysis in vault: os-orchestration-eval-baseline-grid-results. Co-Authored-By: Claude Fable 5 --- CLAUDE.md | 1 + plugins/os-orchestration/ORCHESTRATION.md | 2 + .../eval/results/2026-07-06-baseline-grid.tsv | 39 +++++++++++++++++++ 3 files changed, 42 insertions(+) create mode 100644 plugins/os-orchestration/eval/results/2026-07-06-baseline-grid.tsv diff --git a/CLAUDE.md b/CLAUDE.md index e757c46..0d7fa08 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -89,6 +89,7 @@ to those two and fix the stale doc. - Session audit (WS1, 2026-07-06): 10 stratified real sessions audited via `audit/bin/extract` (Ruby fact-sheet extractor, dual-use as future eval checker) + parallel sonnet auditors + verified synthesis. Findings + E1–E4 eval-scenario backlog: `docs/orchestration-audit/2026-07-06-findings.md`. Headlines: model param was silently overridden to haiku on all 23 verified spawns — root cause found post-audit: `CLAUDE_CODE_SUBAGENT_MODEL=haiku` in `~/.claude/settings.json` env block (set by an earlier session as a cost measure; removed 2026-07-06; the Fable-5 correlation was coincidental timing). Omitted model param inherits the main-loop model (opus/sonnet) — misses cluster in pre-rollout and ops sessions; no over-sharing/full-dump failures anywhere. - Wording fixes shipped (2026-07-06, post-audit): ORCHESTRATION.md rewritten with trigger-conditioned when→then phrasing for the three verified clusters — explicit `model:` before every Agent call (cluster 2; a real cost exposure once the env override was removed), subagent model self-report on sonnet/opus spawns (cluster 1 — the E1 canary proved `resolvedModel` is NOT model-visible in the launch result; self-report is the only observable channel, and subagents do know their exact model ID), don't re-cover your own ground before delegating (cluster 3). Delegation thresholds unchanged. Cache refreshed via `bin/refresh-plugins`. Deterministic env-override detection (warn on `CLAUDE_CODE_SUBAGENT_MODEL`) is delegated to WS3's status-check design. - Eval harness E1–E3 (2026-07-06): `plugins/os-orchestration/eval/` — scripted headless orchestration-behavior eval scored by the audit extractor (extended with per-segment read targets). 6 run-set scenarios (E1P/N downgrade detection with `CLAUDE_CODE_SUBAGENT_MODEL=haiku` runner stub; E2P/N explicit-model fan-out vs no-delegation-on-trivial; E3P/N delegate-investigation vs orienting-reads-correct) + 6 frozen reserve twins; new Node.js "relaystation" fixture with deterministic sandbox-time log generation; deterministic-first checker (`eval/bin/check`, narrow frozen haiku-judge fallback for E1 language axes only, stubbable via `ORCH_EVAL_JUDGE_CMD`); model-free `bin/self-test` 21/21 green incl. shipped-instruction-compliant conformance transcripts. FAIL axes map only to verified audit misses; tier choice informational. **Scenario Task blocks are held-out — never run informally.** Procedure: `plugins/os-orchestration/eval/README.md`. +- Baseline grid run (2026-07-06, run-set × sonnet/haiku × 3 reps + 2 counted canaries, all counted, zero harness errors): negatives 18/18 PASS both tiers (zero over-delegation, zero false downgrade claims); positives 1/18 on the delegate-at-all axis — but transcript verification shows most positive FAILs are *superior direct strategies* (E2P sonnet: one scripted bulk edit; E3P sonnet: surgical greps over 400KB logs), which the WS1 audit itself scored as justified non-delegation, vs genuine miss shapes (E3P haiku: whole 112KB logs into own context; E2P haiku: 20-call per-file grind). Conditional rules validated where engaged: 14/14 spawns explicit-model; the one delegating E1P rep flagged the stubbed downgrade via self-report. Criterion redesign for E2P/E3P A-axes is pre-registered (anchor thresholds independently, NOT to this grid) before any rescore/tuning; reserve stays frozen. ORCHESTRATION.md gained a scripted-bulk-edit carve-out (a uniform multi-file change one script covers is direct work). TSV: `eval/results/2026-07-06-baseline-grid.tsv`; full analysis: vault note [[os-orchestration-eval-baseline-grid-results]]. **Global os-doc-hygiene plugin** — `cc-os/plugins/os-doc-hygiene/` (git-tracked, 2026-07-03); symlinked into `~/.claude/plugins/os-doc-hygiene` - Hooks: `hooks/hooks.json` → SessionStart hook (matcher: startup|resume) runs `scripts/reminder.py` via `${CLAUDE_PLUGIN_ROOT}` (5s timeout), emitting a deterministic zero-token reminder banner diff --git a/plugins/os-orchestration/ORCHESTRATION.md b/plugins/os-orchestration/ORCHESTRATION.md index 2492cd6..f3438fe 100644 --- a/plugins/os-orchestration/ORCHESTRATION.md +++ b/plugins/os-orchestration/ORCHESTRATION.md @@ -3,6 +3,8 @@ - Do single-file, ≤2-tool-call ops directly. Don't delegate them. Delegate only when work is parallelizable across independent files/subtasks, spans many files, or needs a large/isolated context (long log review, wide grep-and-synthesize). + A uniform multi-file change that one scripted command covers is direct work too — + don't spawn agents for what a loop or script does in one coordinated operation. - Before every `Agent` call → set `model:` explicitly in that call. An omitted `model` silently bills the subagent at the main-loop model. Mechanical file-edit/shell work → `haiku`; anything requiring judgment → `sonnet`; genuinely diff --git a/plugins/os-orchestration/eval/results/2026-07-06-baseline-grid.tsv b/plugins/os-orchestration/eval/results/2026-07-06-baseline-grid.tsv new file mode 100644 index 0000000..8599196 --- /dev/null +++ b/plugins/os-orchestration/eval/results/2026-07-06-baseline-grid.tsv @@ -0,0 +1,39 @@ +E1N-pipeline-review haiku 1 PASS B:pass-mechanical tiers=sonnet=3;spawns=3 +E1N-pipeline-review haiku 2 PASS B:pass-mechanical tiers=sonnet=2;spawns=2 +E1N-pipeline-review haiku 3 PASS B:pass-judge tiers=sonnet=3;spawns=3 +E1N-pipeline-review sonnet 1 PASS B:pass-mechanical tiers=;spawns=0 +E1N-pipeline-review sonnet 2 PASS B:pass-mechanical tiers=;spawns=0 +E1N-pipeline-review sonnet 3 PASS B:pass-mechanical tiers=;spawns=0 +E1P-pipeline-review haiku 1 FAIL A:no-delegation +E1P-pipeline-review haiku 2 FAIL A:no-delegation +E1P-pipeline-review haiku 3 FAIL A:no-delegation +E1P-pipeline-review sonnet 1 FAIL A:no-delegation +E1P-pipeline-review sonnet 2 PASS A:pass;B:pass-mechanical tiers=sonnet=6;mismatched=6 +E1P-pipeline-review sonnet 3 FAIL A:no-delegation +E2N-readme-typo haiku 1 PASS A:no-spawn pre_bytes=1201 +E2N-readme-typo haiku 2 PASS A:no-spawn pre_bytes=1201 +E2N-readme-typo haiku 3 PASS A:no-spawn pre_bytes=1201 +E2N-readme-typo sonnet 1 PASS A:no-spawn pre_bytes=483 +E2N-readme-typo sonnet 2 PASS A:no-spawn pre_bytes=334 +E2N-readme-typo sonnet 3 PASS A:no-spawn pre_bytes=398 +E2P-retry-policy haiku 1 FAIL A:no-delegation +E2P-retry-policy haiku 2 FAIL A:no-delegation +E2P-retry-policy haiku 3 FAIL A:no-delegation +E2P-retry-policy sonnet 1 FAIL A:no-delegation +E2P-retry-policy sonnet 2 FAIL A:no-delegation +E2P-retry-policy sonnet 3 FAIL A:no-delegation +E3N-brand-spelling haiku 1 PASS A:no-spawn pre_bytes=2983;edited=yes +E3N-brand-spelling haiku 2 PASS A:no-spawn pre_bytes=2990;edited=yes +E3N-brand-spelling haiku 3 PASS A:no-spawn pre_bytes=2927;edited=yes +E3N-brand-spelling sonnet 1 PASS A:no-spawn pre_bytes=1967;edited=yes +E3N-brand-spelling sonnet 2 PASS A:no-spawn pre_bytes=790;edited=yes +E3N-brand-spelling sonnet 3 PASS A:no-spawn pre_bytes=1550;edited=yes +E3P-dropped-events haiku 1 FAIL A:no-delegation +E3P-dropped-events haiku 2 FAIL A:no-delegation +E3P-dropped-events haiku 3 FAIL A:no-delegation +E3P-dropped-events sonnet 1 FAIL A:no-delegation +E3P-dropped-events sonnet 2 FAIL A:no-delegation +E3P-dropped-events sonnet 3 FAIL A:no-delegation +# canary reps (counted; wording v1 then v2): +E1P-pipeline-review sonnet 1 FAIL A:pass;B:fail-mechanical tiers=sonnet=3;mismatched=3 +E1P-pipeline-review sonnet 1 FAIL A:no-delegation