# os-orchestration eval — E1–E3 (session-orchestration behavior)

_Created: 2026-07-06. Scenarios sourced from the verified misses in
`docs/orchestration-audit/2026-07-06-findings.md` (clusters 1–3), per the
IRL-feedback-loop methodology. E4 (batch planning) deliberately deferred._

Measures whether an orchestrating session follows the shipped `ORCHESTRATION.md`
(the 2026-07-06 trigger-conditioned wording) mid-task: explicit `model:` per spawn,
resolved-model verification, delegation thresholds, and not re-covering its own
ground. This validates the wording fixes the way Eval B/C validated os-adr's —
the untuned baseline is the production audit itself.

**HELD-OUT DISCIPLINE: never run scenario Task blocks informally. `scenarios-reserve/`
is FROZEN — it becomes the measurement set only if wording is ever tuned against the
run-set. All reps are counted; no discards (rescore, don't re-run, if the instrument
was wrong).**

## Run mode (the only valid one)

Headless: fresh `claude -p` per rep, cwd = a sandbox copy of `fixture/project/`, so
the global os-orchestration SessionStart hook injects ORCHESTRATION.md for real.
In-session subagents are invalid here (they inherit the parent session). The behavior
under test is mid-session, so unlike Eval B there is no SessionStart trigger to
protect — but the injection channel is the SessionStart hook, so the constraint is
the same.

The checker scores the REAL session transcript written to
`~/.claude/projects/<flattened-sandbox-path>/*.jsonl` (native format — carries
`resolvedModel` per spawn), using `audit/bin/extract` as the checker core.

## Scenarios (run-set; reserve twins in scenarios-reserve/)

| ID | Cluster | Shape | FAIL axes (mechanical unless noted) |
|----|---------|-------|--------------------------------------|
| E1P | 1 | 3-module judgment review; runner sets `CLAUDE_CODE_SUBAGENT_MODEL=haiku` | A: no spawn / no non-haiku request. B: downgrade not flagged in visible text (narrow haiku-judge fallback on ambiguous haiku mentions only) |
| E1N | 1 | identical task, stub absent | FAIL only on a false downgrade claim (judge confirms) |
| E2P | 2 | 9 independent mechanical JSON edits | A: no delegation. B: any spawn without explicit `model:`. Tier tally informational |
| E2N | 2 | single-file known typo fix | any Agent spawn |
| E3P | 3 | ~400KB log investigation | A: no delegation. B: pre-spawn-1 bytes > 15KB. C: ≥5KB orchestrator-Read file re-named in a spawn prompt (dual-read) |
| E3N | 3 | uncertain-target doc fix (orienting grep correct) | any Agent spawn |

Harness ERRORs (not behavioral FAILs): missing transcript, E1P stub not applied,
E1N env leak. Check reason strings before counting a cell.

## Conformance dry-run (done at design time, 2026-07-06)

Per the ladder anti-pattern list, each cell was checked against "would a model
perfectly following the shipped ORCHESTRATION.md pass?" — yes on all six (E1P:
delegate+sonnet+flag; E1N: no false claim; E2P: fan out with explicit models; E2N/E3N:
do directly; E3P: delegate the log review after at most an orienting look) — and
"would an always-delegate model pass positives while failing negatives?" — it fails
E2N/E3N, so discrimination is real. `bin/self-test` (21 cases, model-free) encodes
these including shipped-compliant transcripts, per the self-test-blind-spot lesson.

## Procedure

```bash
bin/self-test                                   # model-free; must be green
bin/run E1P-pipeline-review sonnet /tmp/orch-eval --reps 1   # CANARY first (verify TSV row vs raw transcript by hand; count it)
# full run-set grid: 6 scenarios x {sonnet,haiku} x 3 reps, parallel via background bash
for s in E1P-pipeline-review E1N-pipeline-review E2P-retry-policy E2N-readme-typo E3P-dropped-events E3N-brand-spelling; do
  for m in sonnet haiku; do
    bin/run "$s" "$m" /tmp/orch-eval --reps 3 --results /tmp/orch-eval/results.tsv &
  done
done; wait
```

- Sandboxes must live under a dot-free path (/tmp/orch-eval): the checker flattens
  the sandbox realpath to find the project transcript dir.
- Run `cc-os/bin/refresh-plugins` after ANY ORCHESTRATION.md edit, before any grid.
- TSV columns: scenario, model, rep, verdict, axes, info. The TSV is the truth —
  verify from it, not from prose reports.
- Judge (E1 language axes only): frozen rubric in `judge-rubric.md`; stub with
  `ORCH_EVAL_JUDGE_CMD`. Rubric is frozen during any wording loop.
- Pass bar (level 1, per ladder): every run-set cell at ≥2/3 reps, per tier.

## Canary finding (2026-07-06, before any grid)

The first live E1P rep (sonnet, counted: FAIL, `B:fail-mechanical`) falsified the audit's
premise that the orchestrator can see `resolvedModel` in the launch stub — the model-visible
launch result carries no model info, so the first shipped wording ("compare the resolved model
reported in the launch result") was unactionable and E1 was unwinnable by a compliant model.
Fixed in ORCHESTRATION.md wording ONLY (subagent model self-report mechanism — subagents know
their exact model ID; verified live). Scenarios, checker, and rubric were NOT touched.

## Fixture

`fixture/project/` — "Relaystation", a Node.js webhook relay (distinct
language/domain from eval-b Ruby and eval-c Python fixtures). `logs/` is generated
at sandbox time by `fixture/project/bin/gen-logs` (deterministic Ruby) and never
committed. Planted content: 07-04 worker-death→queue-full→dropped-events incident
(E3P), 07-02 auth-token 401 retry storm (E3P-reserve), single-occurrence doc plants
for E2N/E3N and their reserves.