vault: session notes 2026-07-06

journal/2026-07-06.md

@@ -421,3 +421,10 @@ tags: [scope/global, type/log]
 /home/jared/Documents/SecondBrain/vault-backlog-pilot-plan.md
 /home/jared/Documents/SecondBrain/vault-backlog-pilot-plan.md
 /home/jared/Documents/SecondBrain/backlog-system-options-research.md
+
+## Session — 2026-07-06T21:32:53Z
+
+**Project:** /home/jared/dev/cc-os
+**Reason:** prompt_input_exit
+**Vault notes touched:**
+(none)

os-orchestration-ws1-session-audit-results.md

@@ -0,0 +1,117 @@
+---
+type: eval-results
+title: "os-orchestration WS1 session audit (IRL, 10 stratified sessions)"
+summary: "Audit of 10 real sessions against the 7-question orchestration rubric: delegation decisions, context hygiene, and grouping are healthy; the two HIGH failures were model-tier resolution — a user-set CLAUDE_CODE_SUBAGENT_MODEL=haiku env var silently overriding every spawn (root-caused post-audit, removed), and omitted model params inheriting the expensive main-loop model."
+tags:
+  - type/eval-results
+  - domain/llm-evaluation
+  - domain/orchestration
+  - tool/os-orchestration
+  - tool/claude-code
+  - project/cc-os
+scope: global
+last_updated: 2026-07-06
+date: 2026-07-06
+related:
+  - eval-methodology-irl-feedback-loop
+  - eval-methodology-ladder
+source: cc-os
+---
+
+# os-orchestration WS1 session audit (IRL, 10 stratified sessions)
+
+Not a lab eval — an IRL audit of real session transcripts (the post-rollout feedback-loop rung
+from [[eval-methodology-irl-feedback-loop]]). Full findings: `cc-os/docs/orchestration-audit/2026-07-06-findings.md` (commit 7965f03).
+
+## Results Grid/Threshold
+
+**Passing criterion:** per-question qualitative PASS across the sample; failures grouped into
+verified clusters ranked by severity. Every load-bearing auditor claim was re-verified against
+the primary transcript before being counted.
+
+| # | Rubric question (the hypotheses evaluated) | Verdict |
+|---|---|---|
+| 1 | Are subagents called when they should be? | Mostly PASS (8/10; S3/S7 gray-zone heavy in-session investigation) |
+| 2 | Correct model tier per subagent? | **FAIL — the dominant failure area** (clusters 1 & 2) |
+| 3 | Planning/grouping for context efficiency? | PASS (concurrent fan-out in 6/10; minor reactive late spawns) |
+| 4 | Avoiding unnecessary orchestrator reads? | Mostly PASS (S4: 0 bytes over 41 spawns; exceptions S7, S9) |
+| 5 | Avoiding over-sharing context with subagents? | PASS everywhere (prompts 0.4–6.7KB, no dumps) |
+| 6 | Following ORCHESTRATION.md? | Split: post-rollout cc-os 23/23 on explicit-model; pre-rollout/ops omit on 16/56 spawns |
+| 7 | Receiving only needed context back? | PASS everywhere (0.9–13KB summaries, no full-dump returns) |
+
+**Summary:** Delegation judgment and context hygiene are healthy; both HIGH failures are about
+model-tier *resolution*, not delegation decisions. Sample: 10 sessions stratified pre/post plugin
+rollout across 5 projects (cc-os sessions contaminated — they were building orchestration tooling).
+
+## Verified failure clusters
+
+1. **HIGH (environment):** all 23 spawns in recent sessions resolved to haiku regardless of the
+   requested tier — including explicit judgment-grade sonnet/opus requests. **Root cause found
+   post-audit:** `CLAUDE_CODE_SUBAGENT_MODEL=haiku` in `~/.claude/settings.json`'s `env` block,
+   set by an earlier AI session as a "lowest cost first" measure. The var force-overrides the
+   `model` param on every Agent spawn, silently. The audit's initial "Fable-5 harness bug"
+   attribution was wrong — coincidental timing (var added between the pre-rollout opus/sonnet
+   sessions and the Fable-5 sessions). **Removed 2026-07-06.** The exposed policy gap survives
+   the fix: nothing tells the orchestrator to verify `resolvedModel`, so 23 downgrades went
+   unnoticed.
+2. **HIGH (orchestrator):** omitting `model` inherits the main-loop model — a mechanical
+   file-edit ran at sonnet, a character-counting task at opus. Misses cluster pre-rollout and in
+   ops projects; post-rollout cc-os had zero omissions. Now *unmasked* by the env-var removal:
+   the clamp-to-haiku was hiding the cost of omissions.
+3. **MEDIUM:** orchestrator self-investigates (up to 74KB of reads) then delegates the same
+   ground; the "short orienting Read" boundary is undefined.
+4. **LOW:** reactive late spawns instead of pre-planned batches (wall-clock only).
+
+## Validity and Limitations
+
+**How to interpret:** the plugin wording works where present and salient (23/23 explicit-model
+compliance post-rollout in cc-os); misses cluster where the text wasn't in force. But only 2
+truly post-rollout non-cc-os sessions (one violated) — the target population is thin in this
+sample. Single audit pass; several raw auditor claims did not survive verification (only the
+verified clusters should be cited). The missed-delegation heuristic (≥4 same-tool runs) produced
+only false positives — needs retuning before it can score an eval.
+
+**No rerun needed:** the audit measured real behavior and the measurements stand; only the
+Cluster 1 *attribution* changed. The E1–E4 backlog items are eval *scenarios*, not built evals —
+nothing exists to rerun.
+
+## Orchestrator economics (clarifications worth keeping)
+
+The intuition "the expensive orchestrator should delegate all grunt work to cheap workers" is
+the right frame for the wrong unit of work:
+
+- **The expensive part of a frontier orchestrator isn't Edit calls — it's context.** A direct
+  edit to an in-context file is a few hundred output tokens. Delegating the same edit costs a
+  detailed prompt (orchestrator output) + the subagent re-reading the file and surrounding
+  context in a fresh window + a result the orchestrator must read and verify. For sequential,
+  coordinated edits the handoff exceeds the edit.
+- **Delegation pays when it absorbs context the orchestrator doesn't want to load:**
+  parallelizable independent files, many-file sweeps, large isolated reads (log review,
+  wide grep-and-synthesize). WS1 itself used this correctly — 10 parallel sonnet auditors each
+  digested a full transcript the orchestrator never had to hold.
+- Mental model: *keep work you already hold the context for; delegate work whose context you
+  don't want to pay for.* Sequential CRUD on an in-context file is the first kind — the audit's
+  Q1 confirmed this (both "missed delegation" flags on in-session sequential work were false
+  positives).
+
+## Deployment and Evolution
+
+**Good-Enough Gate:** plugin stays deployed as-is; the env-var removal fixes Cluster 1's cause.
+Cluster 2 is the live risk now (omission = expensive inheritance under a Fable main loop).
+
+**Hardening path / next measurement:**
+1. ORCHESTRATION.md wording fixes: (a) verify-`resolvedModel` rule (say so and adapt on
+   mismatch for judgment-critical work); (b) mechanical trigger phrasing for the explicit-model
+   rule ("before every Agent call → include `model:`" — the Eval B lesson from
+   [[os-adr-eval-b-wording-experiment-hypotheses]]); (c) define the orienting-read budget.
+2. Build E1–E3 scripted evals from the verified misses (per [[eval-methodology-ladder]]:
+   paired positives/negatives, frozen reserve). E1 = detect the downgrade, E2 = explicit model
+   per spawn, E3 = pre-spawn read budget. E4 deferred.
+3. Re-audit IRL in ~2 weeks (post env-var removal, non-cc-os sessions) — that population was
+   thin this pass.
+
+## Related
+
+- [[eval-methodology-irl-feedback-loop]] — this audit is that loop's first iteration for os-orchestration
+- [[eval-methodology-ladder]] — design gate for the E1–E3 evals
+- [[os-adr-eval-b-wording-experiment-hypotheses]] — the trigger-phrasing lessons the wording fixes will reuse