cc-os/plugins/os-context/eval

os-context eval — E1–E3, E5 (session-orchestration behavior)

STATUS 2026-07-17 — RETIRED AS A MEASUREMENT INSTRUMENT; KEPT AS TEMPLATE. Two independent reasons, either sufficient: (1) the WS4 wording loop consumed the run-set AND the reserves — both are contaminated for any future wording tuning; (2) ADR-0043 replaced the per-task delegation threshold these scenarios encode with an absolute delegation policy, so the scenario expectations now judge against a superseded rule set. Do not run or rebuild expectations in place; a future eval of the ADR-0043 wording needs fresh scenarios sourced from new IRL audit findings, using this harness (bin/, fixture/, run mode below) as the structural template.

Created: 2026-07-06. Scenarios sourced from the verified misses in docs/orchestration-audit/2026-07-06-findings.md (clusters 1–3), per the IRL-feedback-loop methodology. E4 (batch planning) deliberately deferred.

Measures whether an orchestrating session follows the shipped prompts/session-start/10-orchestration.md (the 2026-07-06 trigger-conditioned wording) mid-task: explicit model: per spawn, resolved-model verification, delegation thresholds, and not re-covering its own ground. This validates the wording fixes the way Eval B/C validated os-adr's — the untuned baseline is the production audit itself.

HELD-OUT DISCIPLINE: never run scenario Task blocks informally. scenarios-reserve/ is FROZEN — it becomes the measurement set only if wording is ever tuned against the run-set. All reps are counted; no discards (rescore, don't re-run, if the instrument was wrong).

Run mode (the only valid one)

Headless: fresh claude -p per rep, cwd = a sandbox copy of fixture/project/, so the global os-context SessionStart hook injects the orchestration prompt for real. In-session subagents are invalid here (they inherit the parent session). The behavior under test is mid-session, so unlike Eval B there is no SessionStart trigger to protect — but the injection channel is the SessionStart hook, so the constraint is the same.

The checker scores the REAL session transcript written to ~/.claude/projects/<flattened-sandbox-path>/*.jsonl (native format — carries resolvedModel per spawn), using audit/bin/extract as the checker core.

Scenarios (run-set; reserve twins in scenarios-reserve/)

ID	Cluster	Shape	FAIL axes (mechanical unless noted)
E1P	1	3-module judgment review; runner sets CLAUDE_CODE_SUBAGENT_MODEL=haiku	A: no spawn / no non-haiku request. B: downgrade not flagged in visible text (narrow haiku-judge fallback on ambiguous haiku mentions only)
E1N	1	identical task, stub absent	FAIL only on a false downgrade claim (judge confirms)
E2P	2	9 independent mechanical JSON edits	A: 0 spawns AND >=9 Edit+Write ops (no-delegation-and-grind). B: any spawn without explicit model:. C: task-incomplete (see below). Tier tally informational
E2N	2	single-file known typo fix	any Agent spawn
E3P	3	~400KB log investigation	A: whole-session bytes_read > 74KB (overread). B: fewer than 2 of {worker-death, queue-backup, dropped-events} named in the final message (root-cause-missing). C: dual-read, only checked when a spawn happened
E3N	3	uncertain-target doc fix (orienting grep correct)	any Agent spawn
E5P	batch delegation	colleague-authored fixture, ## Expected files coverage	A: 0 spawns AND >=12 Edit+Write ops (main-loop-grind). B: >3 spawns (one-per-item). C: any expected file missing/empty. D: any spawn without explicit model:
E5N	batch delegation	task that doesn't warrant fan-out	2+ spawns (parallel-fanout); 0 or 1 spawn are both sequential and fine

Harness ERRORs (not behavioral FAILs): missing transcript, E1P stub not applied, E1N env leak. Check reason strings before counting a cell.

E2P/E3P criterion redesign (pre-registered, 2026-07-06): neither positive requires delegation any more — both accept a well-scoped direct/scripted path, since the WS1 baseline grid showed genuinely superior direct strategies scoring FAIL under the old delegation-mandatory axes. E2P PASS iff task-complete (all 9 services/*.json differ from the pristine fixture copies in eval/fixture/project/services/ and remain valid JSON) AND (delegated with every spawn model_explicit, OR 0 spawns with < 9 Edit+Write ops — "scripted-direct"). E3P PASS iff whole-session bytes_read <= 74000 (anchor: WS1 audit S7 exemplar) AND the final assistant message names >=2 of the three planted-incident concepts (see fixture/project/bin/gen-logs) AND (no dual-read, only evaluated when a spawn happened).

E5 family and the ## Expected files convention

E5 measures batched-vs-per-item delegation on a task with several independent outputs. A colleague is authoring the E5 scenario files in parallel (run-set + reserve, per the same held-out discipline as E1–E3) — do not create or edit scenarios/E5*.md / scenarios-reserve/E5*.md here. The fixed convention those files must follow: a ## Expected files section listing one repo-relative path per line (bare or -/* list-item, optionally backtick-quoted); the checker's coverage axis (E5P axis C) requires every listed path to exist non-empty in the sandbox. eval/bin/run resolves E5* scenarios through the same scenarios/ then scenarios-reserve/ lookup as every other scenario, and takes the default env arm (env -u CLAUDE_CODE_SUBAGENT_MODEL) — no new case arm was needed. Self-test proves the checker logic against a fabricated scenario file delivered via ORCH_EVAL_SCENARIO_FILE (self-test-only override, mirrors ORCH_EVAL_TRANSCRIPT) rather than writing into scenarios/.

Model aliases

eval/bin/run <scenario> <model> ... accepts fable as a model name; it is passed to claude -p --model as claude-fable-5 while the TSV keeps recording the short name (fable) given on the CLI, matching how sonnet/ haiku are recorded verbatim.

Token/round instrumentation

Every scenario row's TSV info column now carries econ figures, informational only (never a verdict axis) on every scenario: mltok=<main-loop output tokens>;sctok=<sidechain output tokens>;mlshare=<0.NNN main-loop share>;prebytes=<pre-spawn-1 bytes_read>;spawns=<n>;maxrounds=<m>.

mlshare is now a real token-based ratio, replacing the old char-count proxy implicit in bytes_read. Subagent transcripts persist as sibling files at <projects-dir>/<main-session-basename-without-.jsonl>/subagents/agent-*.jsonl (same native JSONL shape, own usage blocks per assistant line) — verified for both in-session and headless (claude -p) sessions. audit/bin/extract sums message.usage.output_tokens over non-sidechain main-transcript assistant lines for main_loop_output_tokens, and over every assistant line in the sibling subagents/*.jsonl files (0 if the dir is absent) for sidechain_output_tokens; main_loop_share is main / (main + sidechain) rounded to 3 places, 1.0 when sidechain is 0. agent_spawns[].rounds is 1 + SendMessage calls addressed to that agent (by raw agentId or by the spawn's description as a name proxy — the transcript carries no other name field); metadata gains total_spawns and max_rounds.

Conformance dry-run (done at design time, 2026-07-06)

Per the ladder anti-pattern list, each cell was checked against "would a model perfectly following the shipped prompts/session-start/10-orchestration.md pass?" — yes on all six (E1P: delegate+sonnet+flag; E1N: no false claim; E2P: fan out with explicit models; E2N/E3N: do directly; E3P: delegate the log review after at most an orienting look) — and "would an always-delegate model pass positives while failing negatives?" — it fails E2N/E3N, so discrimination is real. bin/self-test (32 cases, model-free) encodes these including shipped-compliant transcripts, per the self-test-blind-spot lesson; this now also covers the E2P/E3P criterion redesign, the E5 family, and the token/round accounting.

Procedure

bin/self-test                                   # model-free; must be green
bin/run E1P-pipeline-review sonnet /tmp/orch-eval --reps 1   # CANARY first (verify TSV row vs raw transcript by hand; count it)
# full run-set grid: 6 scenarios x {sonnet,haiku} x 3 reps, parallel via background bash
for s in E1P-pipeline-review E1N-pipeline-review E2P-retry-policy E2N-readme-typo E3P-dropped-events E3N-brand-spelling; do
  for m in sonnet haiku; do
    bin/run "$s" "$m" /tmp/orch-eval --reps 3 --results /tmp/orch-eval/results.tsv &
  done
done; wait

• Sandboxes must live under a dot-free path (/tmp/orch-eval): the checker flattens the sandbox realpath to find the project transcript dir.
• Run cc-os/bin/refresh-plugins after ANY prompts/session-start/10-orchestration.md edit, before any grid.
• TSV columns: scenario, model, rep, verdict, axes, info. The TSV is the truth — verify from it, not from prose reports.
• Judge (E1 language axes only): frozen rubric in judge-rubric.md; stub with ORCH_EVAL_JUDGE_CMD. Rubric is frozen during any wording loop.
• Pass bar (level 1, per ladder): every run-set cell at ≥2/3 reps, per tier.

Canary finding (2026-07-06, before any grid)

The first live E1P rep (sonnet, counted: FAIL, B:fail-mechanical) falsified the audit's premise that the orchestrator can see resolvedModel in the launch stub — the model-visible launch result carries no model info, so the first shipped wording ("compare the resolved model reported in the launch result") was unactionable and E1 was unwinnable by a compliant model. Fixed in prompts/session-start/10-orchestration.md wording ONLY (subagent model self-report mechanism — subagents know their exact model ID; verified live). Scenarios, checker, and rubric were NOT touched.

Fixture

fixture/project/ — "Relaystation", a Node.js webhook relay (distinct language/domain from eval-b Ruby and eval-c Python fixtures). logs/ is generated at sandbox time by fixture/project/bin/gen-logs (deterministic Ruby) and never committed. Planted content: 07-04 worker-death→queue-full→dropped-events incident (E3P), 07-02 auth-token 401 retry storm (E3P-reserve), single-occurrence doc plants for E2N/E3N and their reserves.