diff --git a/journal/2026-07-06.md b/journal/2026-07-06.md index 3569114..f66ad27 100644 --- a/journal/2026-07-06.md +++ b/journal/2026-07-06.md @@ -421,3 +421,10 @@ tags: [scope/global, type/log] /home/jared/Documents/SecondBrain/vault-backlog-pilot-plan.md /home/jared/Documents/SecondBrain/vault-backlog-pilot-plan.md /home/jared/Documents/SecondBrain/backlog-system-options-research.md + +## Session — 2026-07-06T21:32:53Z + +**Project:** /home/jared/dev/cc-os +**Reason:** prompt_input_exit +**Vault notes touched:** +(none) diff --git a/os-orchestration-ws1-session-audit-results.md b/os-orchestration-ws1-session-audit-results.md new file mode 100644 index 0000000..42dc637 --- /dev/null +++ b/os-orchestration-ws1-session-audit-results.md @@ -0,0 +1,117 @@ +--- +type: eval-results +title: "os-orchestration WS1 session audit (IRL, 10 stratified sessions)" +summary: "Audit of 10 real sessions against the 7-question orchestration rubric: delegation decisions, context hygiene, and grouping are healthy; the two HIGH failures were model-tier resolution — a user-set CLAUDE_CODE_SUBAGENT_MODEL=haiku env var silently overriding every spawn (root-caused post-audit, removed), and omitted model params inheriting the expensive main-loop model." +tags: + - type/eval-results + - domain/llm-evaluation + - domain/orchestration + - tool/os-orchestration + - tool/claude-code + - project/cc-os +scope: global +last_updated: 2026-07-06 +date: 2026-07-06 +related: + - eval-methodology-irl-feedback-loop + - eval-methodology-ladder +source: cc-os +--- + +# os-orchestration WS1 session audit (IRL, 10 stratified sessions) + +Not a lab eval — an IRL audit of real session transcripts (the post-rollout feedback-loop rung +from [[eval-methodology-irl-feedback-loop]]). Full findings: `cc-os/docs/orchestration-audit/2026-07-06-findings.md` (commit 7965f03). + +## Results Grid/Threshold + +**Passing criterion:** per-question qualitative PASS across the sample; failures grouped into +verified clusters ranked by severity. Every load-bearing auditor claim was re-verified against +the primary transcript before being counted. + +| # | Rubric question (the hypotheses evaluated) | Verdict | +|---|---|---| +| 1 | Are subagents called when they should be? | Mostly PASS (8/10; S3/S7 gray-zone heavy in-session investigation) | +| 2 | Correct model tier per subagent? | **FAIL — the dominant failure area** (clusters 1 & 2) | +| 3 | Planning/grouping for context efficiency? | PASS (concurrent fan-out in 6/10; minor reactive late spawns) | +| 4 | Avoiding unnecessary orchestrator reads? | Mostly PASS (S4: 0 bytes over 41 spawns; exceptions S7, S9) | +| 5 | Avoiding over-sharing context with subagents? | PASS everywhere (prompts 0.4–6.7KB, no dumps) | +| 6 | Following ORCHESTRATION.md? | Split: post-rollout cc-os 23/23 on explicit-model; pre-rollout/ops omit on 16/56 spawns | +| 7 | Receiving only needed context back? | PASS everywhere (0.9–13KB summaries, no full-dump returns) | + +**Summary:** Delegation judgment and context hygiene are healthy; both HIGH failures are about +model-tier *resolution*, not delegation decisions. Sample: 10 sessions stratified pre/post plugin +rollout across 5 projects (cc-os sessions contaminated — they were building orchestration tooling). + +## Verified failure clusters + +1. **HIGH (environment):** all 23 spawns in recent sessions resolved to haiku regardless of the + requested tier — including explicit judgment-grade sonnet/opus requests. **Root cause found + post-audit:** `CLAUDE_CODE_SUBAGENT_MODEL=haiku` in `~/.claude/settings.json`'s `env` block, + set by an earlier AI session as a "lowest cost first" measure. The var force-overrides the + `model` param on every Agent spawn, silently. The audit's initial "Fable-5 harness bug" + attribution was wrong — coincidental timing (var added between the pre-rollout opus/sonnet + sessions and the Fable-5 sessions). **Removed 2026-07-06.** The exposed policy gap survives + the fix: nothing tells the orchestrator to verify `resolvedModel`, so 23 downgrades went + unnoticed. +2. **HIGH (orchestrator):** omitting `model` inherits the main-loop model — a mechanical + file-edit ran at sonnet, a character-counting task at opus. Misses cluster pre-rollout and in + ops projects; post-rollout cc-os had zero omissions. Now *unmasked* by the env-var removal: + the clamp-to-haiku was hiding the cost of omissions. +3. **MEDIUM:** orchestrator self-investigates (up to 74KB of reads) then delegates the same + ground; the "short orienting Read" boundary is undefined. +4. **LOW:** reactive late spawns instead of pre-planned batches (wall-clock only). + +## Validity and Limitations + +**How to interpret:** the plugin wording works where present and salient (23/23 explicit-model +compliance post-rollout in cc-os); misses cluster where the text wasn't in force. But only 2 +truly post-rollout non-cc-os sessions (one violated) — the target population is thin in this +sample. Single audit pass; several raw auditor claims did not survive verification (only the +verified clusters should be cited). The missed-delegation heuristic (≥4 same-tool runs) produced +only false positives — needs retuning before it can score an eval. + +**No rerun needed:** the audit measured real behavior and the measurements stand; only the +Cluster 1 *attribution* changed. The E1–E4 backlog items are eval *scenarios*, not built evals — +nothing exists to rerun. + +## Orchestrator economics (clarifications worth keeping) + +The intuition "the expensive orchestrator should delegate all grunt work to cheap workers" is +the right frame for the wrong unit of work: + +- **The expensive part of a frontier orchestrator isn't Edit calls — it's context.** A direct + edit to an in-context file is a few hundred output tokens. Delegating the same edit costs a + detailed prompt (orchestrator output) + the subagent re-reading the file and surrounding + context in a fresh window + a result the orchestrator must read and verify. For sequential, + coordinated edits the handoff exceeds the edit. +- **Delegation pays when it absorbs context the orchestrator doesn't want to load:** + parallelizable independent files, many-file sweeps, large isolated reads (log review, + wide grep-and-synthesize). WS1 itself used this correctly — 10 parallel sonnet auditors each + digested a full transcript the orchestrator never had to hold. +- Mental model: *keep work you already hold the context for; delegate work whose context you + don't want to pay for.* Sequential CRUD on an in-context file is the first kind — the audit's + Q1 confirmed this (both "missed delegation" flags on in-session sequential work were false + positives). + +## Deployment and Evolution + +**Good-Enough Gate:** plugin stays deployed as-is; the env-var removal fixes Cluster 1's cause. +Cluster 2 is the live risk now (omission = expensive inheritance under a Fable main loop). + +**Hardening path / next measurement:** +1. ORCHESTRATION.md wording fixes: (a) verify-`resolvedModel` rule (say so and adapt on + mismatch for judgment-critical work); (b) mechanical trigger phrasing for the explicit-model + rule ("before every Agent call → include `model:`" — the Eval B lesson from + [[os-adr-eval-b-wording-experiment-hypotheses]]); (c) define the orienting-read budget. +2. Build E1–E3 scripted evals from the verified misses (per [[eval-methodology-ladder]]: + paired positives/negatives, frozen reserve). E1 = detect the downgrade, E2 = explicit model + per spawn, E3 = pre-spawn read budget. E4 deferred. +3. Re-audit IRL in ~2 weeks (post env-var removal, non-cc-os sessions) — that population was + thin this pass. + +## Related + +- [[eval-methodology-irl-feedback-loop]] — this audit is that loop's first iteration for os-orchestration +- [[eval-methodology-ladder]] — design gate for the E1–E3 evals +- [[os-adr-eval-b-wording-experiment-hypotheses]] — the trigger-phrasing lessons the wording fixes will reuse