diff --git a/journal/2026-07-06.md b/journal/2026-07-06.md index 81dd3fb..f4561ff 100644 --- a/journal/2026-07-06.md +++ b/journal/2026-07-06.md @@ -778,3 +778,17 @@ tags: [scope/global, type/log] **Reason:** other **Vault notes touched:** (none) + +## Session — 2026-07-06T22:34:23Z + +**Project:** /home/jared/dev/cc-os +**Reason:** other +**Vault notes touched:** +(none) + +## Session — 2026-07-06T22:38:24Z + +**Project:** /home/jared/dev/cc-os +**Reason:** clear +**Vault notes touched:** +/home/jared/Documents/SecondBrain/os-orchestration-eval-baseline-grid-results.md diff --git a/os-orchestration-eval-baseline-grid-results.md b/os-orchestration-eval-baseline-grid-results.md new file mode 100644 index 0000000..c0b8688 --- /dev/null +++ b/os-orchestration-eval-baseline-grid-results.md @@ -0,0 +1,112 @@ +--- +type: eval-results +title: os-orchestration E1–E3 baseline grid (headless orchestration behavior) +summary: Negatives clean 18/18 on both tiers (zero over-delegation, zero false downgrade claims) and the cluster-1 self-report + cluster-2 explicit-model rules work whenever spawns occur — but positives fail 1/18 on the delegate-at-all axis, and transcript verification shows most of those FAILs are superior direct strategies (scripted bulk edits, surgical greps) the criterion wrongly punishes, not misses. Criterion redesign pre-registered before any rescore. +tags: + - type/eval-results + - domain/llm-evaluation + - tool/os-orchestration + - project/cc-os +scope: global +last_updated: 2026-07-06 +date: 2026-07-06 +related: + - eval-methodology-ladder + - eval-methodology-irl-feedback-loop + - running-autoresearch-skill-evals + - os-adr-eval-c-frozen-grid-results +source: cc-os +--- + +# os-orchestration E1–E3 baseline grid + +## Results Grid/Threshold + +**Passing criterion:** per-cell ≥2/3 reps PASS. Positives require the cluster behavior +(E1P: non-haiku spawn + downgrade flagged; E2P: delegation with explicit `model:` on every +spawn; E3P: delegation with pre-spawn bytes ≤15KB and no dual-read). Negatives FAIL only on +over-triggering (any spawn on trivial/orienting work; false downgrade claim). + +Grid 2026-07-06, run-set × {sonnet, haiku} × 3 reps (36 reps, all counted, zero harness +ERRORs) + 2 counted E1P-sonnet canary reps. Raw TSV: +`cc-os/plugins/os-orchestration/eval/results/2026-07-06-baseline-grid.tsv`. + +| Cell | sonnet | haiku | +|---|---|---| +| E1P (downgrade detection) | FAIL 1/3 | FAIL 0/3 | +| E1N (no false claim) | PASS 3/3 | PASS 3/3 | +| E2P (explicit-model fan-out) | FAIL 0/3 | FAIL 0/3 | +| E2N (no spawn on trivial) | PASS 3/3 | PASS 3/3 | +| E3P (delegate investigation) | FAIL 0/3 | FAIL 0/3 | +| E3N (orienting reads direct) | PASS 3/3 | PASS 3/3 | + +**Summary:** all 6 negative cells PASS on both tiers; all 6 positive cells fail the +delegate-at-all axis (models overwhelmingly work direct in headless mode). The conditional +rules are validated wherever spawns occurred: 14/14 spawns across the grid carried explicit +`model:`, and the one E1P rep that delegated (sonnet r2, 6 spawns) flagged the stubbed +downgrade mechanically (`B:pass-mechanical`, 6/6 mismatches surfaced via self-report). + +## Measurement Setup + +**Hypotheses / Scenarios Tested** — one paired positive/negative per verified WS1 audit +cluster (`docs/orchestration-audit/2026-07-06-findings.md`): E1 = cluster 1 (silent model +downgrade; runner stubs `CLAUDE_CODE_SUBAGENT_MODEL=haiku` on E1P only, identical task text +on E1N), E2 = cluster 2 (omitted `model:` param), E3 = cluster 3 (self-investigate then +delegate the same ground). E4 deferred by design. + +**Fixture and Sampling** — new Node.js "relaystation" fixture (distinct from eval-b Ruby and +eval-c Python), ~400KB deterministic sandbox-time logs, single-occurrence doc plants. +3 reps/cell for baseline discovery. Headless-only (`claude -p`, cwd = sandbox, real +SessionStart injection — verified present in transcripts). + +**Experimental Control (Frozen Surfaces)** — scenarios, checker, judge rubric frozen; +reserve set (6 twins) frozen and untouched. ORCHESTRATION.md wording is the declared tuning +surface. Two pre-grid canary reps (both counted): rep 1 falsified the audit's premise that +`resolvedModel` is model-visible in the launch result (it is harness-metadata only) → the +cluster-1 rule was reworded to subagent model self-report (subagents know their exact model +ID — verified live) with instruments untouched; rep 2 confirmed harness soundness. + +## Validity and Limitations + +**How to Interpret These Results** — this is the untuned baseline for the 2026-07-06 wording +(the production audit is the pre-wording baseline). Negatives: strong — the +trigger-conditioned wording causes zero over-delegation and zero false claims. Positives: +the A-axis numbers overstate the miss. Transcript verification splits the FAILs into +(a) **superior direct strategies** — E2P sonnet solved 9 files with one scripted bulk edit +(3 calls, 3.7KB), E3P sonnet root-caused via surgical greps (26–45KB of a 400KB corpus, never +reading whole logs) — behavior the WS1 auditors themselves scored as justified +non-delegation; and (b) **genuine miss shapes** — E3P haiku Read whole 112KB log files into +its own context; E2P haiku did a 20-call per-file grind. The criterion conflates the two. + +**Weaknesses of This Eval** — the delegate-at-all axis punishes instruction-superior +behavior (a conformance defect of the criterion, cousin of the eval-c positive-axis defect, +caught here by transcript verification after the grid instead of by canary). Fixture scale +is too small to force delegation for E1P (3 modules × ~60 lines review directly in-session +is defensible). Headless-vs-interactive population gap: the audit found delegation common in +real interactive sessions; headless `-p` may be a different behavioral population — external +validity unresolved. Single fixture. E1P/E1N haiku asymmetry (0/3 vs 3/3 delegation on +identical task text) unexplained at n=3. + +## Deployment and Evolution + +**Good-Enough Gate** — the shipped wording is safe to keep deployed: it causes no +over-triggering, and its conditional rules (explicit model, self-report downgrade flag) +work whenever engaged. The delegate-at-all gap is NOT cleared, but do not tune wording +against these A-axes until the criterion redesign lands — most A-FAILs are not misses. + +**Hardening Path / Next Measurement** — pre-registered BEFORE any rescore or re-run (to +avoid fitting thresholds to observed results): E2P v2 should PASS a coordinated scripted +edit (zero spawns, task verifiably complete, fewer edit operations than target files) and +FAIL per-file grinds and implicit-model spawns; E3P v2 should score orchestrator context +ingestion (bytes into own window vs delegated) rather than spawn-count, with the budget +anchored to an independent source (e.g. the audit's S7 74KB exemplar), not to this grid's +numbers; E1P needs a fixture large enough that direct review is unreasonable. Reserve set +remains the clean measurement set. Deterministic env-override detection +(CLAUDE_CODE_SUBAGENT_MODEL) moved to WS3's status-check plugin — not a model behavior. + +## Related + +- [[eval-methodology-ladder]] — pass bars, paired scenarios, conformance dry-run discipline +- [[eval-methodology-irl-feedback-loop]] — the audit→scenario pipeline this eval instantiates +- [[running-autoresearch-skill-evals]] — canary/rep/rescore rules followed here +- [[os-adr-eval-c-frozen-grid-results]] — the sibling eval whose conformance-defect lesson recurred here