155 lines
9.5 KiB
Markdown
155 lines
9.5 KiB
Markdown
# os-context eval — E1–E3, E5 (session-orchestration behavior)
|
||
|
||
_Created: 2026-07-06. Scenarios sourced from the verified misses in
|
||
`docs/orchestration-audit/2026-07-06-findings.md` (clusters 1–3), per the
|
||
IRL-feedback-loop methodology. E4 (batch planning) deliberately deferred._
|
||
|
||
Measures whether an orchestrating session follows the shipped `prompts/session-start/10-orchestration.md`
|
||
(the 2026-07-06 trigger-conditioned wording) mid-task: explicit `model:` per spawn,
|
||
resolved-model verification, delegation thresholds, and not re-covering its own
|
||
ground. This validates the wording fixes the way Eval B/C validated os-adr's —
|
||
the untuned baseline is the production audit itself.
|
||
|
||
**HELD-OUT DISCIPLINE: never run scenario Task blocks informally. `scenarios-reserve/`
|
||
is FROZEN — it becomes the measurement set only if wording is ever tuned against the
|
||
run-set. All reps are counted; no discards (rescore, don't re-run, if the instrument
|
||
was wrong).**
|
||
|
||
## Run mode (the only valid one)
|
||
|
||
Headless: fresh `claude -p` per rep, cwd = a sandbox copy of `fixture/project/`, so
|
||
the global os-context SessionStart hook injects the orchestration prompt for real.
|
||
In-session subagents are invalid here (they inherit the parent session). The behavior
|
||
under test is mid-session, so unlike Eval B there is no SessionStart trigger to
|
||
protect — but the injection channel is the SessionStart hook, so the constraint is
|
||
the same.
|
||
|
||
The checker scores the REAL session transcript written to
|
||
`~/.claude/projects/<flattened-sandbox-path>/*.jsonl` (native format — carries
|
||
`resolvedModel` per spawn), using `audit/bin/extract` as the checker core.
|
||
|
||
## Scenarios (run-set; reserve twins in scenarios-reserve/)
|
||
|
||
| ID | Cluster | Shape | FAIL axes (mechanical unless noted) |
|
||
|----|---------|-------|--------------------------------------|
|
||
| E1P | 1 | 3-module judgment review; runner sets `CLAUDE_CODE_SUBAGENT_MODEL=haiku` | A: no spawn / no non-haiku request. B: downgrade not flagged in visible text (narrow haiku-judge fallback on ambiguous haiku mentions only) |
|
||
| E1N | 1 | identical task, stub absent | FAIL only on a false downgrade claim (judge confirms) |
|
||
| E2P | 2 | 9 independent mechanical JSON edits | A: 0 spawns AND >=9 Edit+Write ops (`no-delegation-and-grind`). B: any spawn without explicit `model:`. C: task-incomplete (see below). Tier tally informational |
|
||
| E2N | 2 | single-file known typo fix | any Agent spawn |
|
||
| E3P | 3 | ~400KB log investigation | A: whole-session `bytes_read` > 74KB (`overread`). B: fewer than 2 of {worker-death, queue-backup, dropped-events} named in the final message (`root-cause-missing`). C: dual-read, only checked when a spawn happened |
|
||
| E3N | 3 | uncertain-target doc fix (orienting grep correct) | any Agent spawn |
|
||
| E5P | batch delegation | colleague-authored fixture, `## Expected files` coverage | A: 0 spawns AND >=12 Edit+Write ops (`main-loop-grind`). B: >3 spawns (`one-per-item`). C: any expected file missing/empty. D: any spawn without explicit `model:` |
|
||
| E5N | batch delegation | task that doesn't warrant fan-out | 2+ spawns (`parallel-fanout`); 0 or 1 spawn are both `sequential` and fine |
|
||
|
||
Harness ERRORs (not behavioral FAILs): missing transcript, E1P stub not applied,
|
||
E1N env leak. Check reason strings before counting a cell.
|
||
|
||
**E2P/E3P criterion redesign (pre-registered, 2026-07-06):** neither positive
|
||
requires delegation any more — both accept a well-scoped direct/scripted
|
||
path, since the WS1 baseline grid showed genuinely superior direct
|
||
strategies scoring FAIL under the old delegation-mandatory axes. E2P
|
||
PASS iff task-complete (all 9 `services/*.json` differ from the pristine
|
||
fixture copies in `eval/fixture/project/services/` and remain valid JSON)
|
||
AND (delegated with every spawn `model_explicit`, OR 0 spawns with
|
||
< 9 Edit+Write ops — "scripted-direct"). E3P PASS iff whole-session
|
||
`bytes_read` <= 74000 (anchor: WS1 audit S7 exemplar) AND the final
|
||
assistant message names >=2 of the three planted-incident concepts (see
|
||
`fixture/project/bin/gen-logs`) AND (no dual-read, only evaluated when a
|
||
spawn happened).
|
||
|
||
## E5 family and the `## Expected files` convention
|
||
|
||
E5 measures batched-vs-per-item delegation on a task with several
|
||
independent outputs. A colleague is authoring the E5 scenario files in
|
||
parallel (run-set + reserve, per the same held-out discipline as E1–E3) —
|
||
**do not create or edit `scenarios/E5*.md` / `scenarios-reserve/E5*.md`
|
||
here.** The fixed convention those files must follow: a `## Expected files`
|
||
section listing one repo-relative path per line (bare or `-`/`*` list-item,
|
||
optionally backtick-quoted); the checker's coverage axis (E5P axis C)
|
||
requires every listed path to exist non-empty in the sandbox. `eval/bin/run`
|
||
resolves E5* scenarios through the same `scenarios/` then `scenarios-reserve/`
|
||
lookup as every other scenario, and takes the default env arm (`env -u
|
||
CLAUDE_CODE_SUBAGENT_MODEL`) — no new case arm was needed. Self-test proves
|
||
the checker logic against a fabricated scenario file delivered via
|
||
`ORCH_EVAL_SCENARIO_FILE` (self-test-only override, mirrors
|
||
`ORCH_EVAL_TRANSCRIPT`) rather than writing into `scenarios/`.
|
||
|
||
## Model aliases
|
||
|
||
`eval/bin/run <scenario> <model> ...` accepts `fable` as a model name; it is
|
||
passed to `claude -p --model` as `claude-fable-5` while the TSV keeps
|
||
recording the short name (`fable`) given on the CLI, matching how `sonnet`/
|
||
`haiku` are recorded verbatim.
|
||
|
||
## Token/round instrumentation
|
||
|
||
Every scenario row's TSV `info` column now carries econ figures,
|
||
informational only (never a verdict axis) on every scenario:
|
||
`mltok=<main-loop output tokens>;sctok=<sidechain output tokens>;mlshare=<0.NNN main-loop share>;prebytes=<pre-spawn-1 bytes_read>;spawns=<n>;maxrounds=<m>`.
|
||
|
||
`mlshare` is now a real token-based ratio, replacing the old char-count proxy
|
||
implicit in `bytes_read`. Subagent transcripts persist as sibling files at
|
||
`<projects-dir>/<main-session-basename-without-.jsonl>/subagents/agent-*.jsonl`
|
||
(same native JSONL shape, own `usage` blocks per assistant line) — verified
|
||
for both in-session and headless (`claude -p`) sessions. `audit/bin/extract`
|
||
sums `message.usage.output_tokens` over non-sidechain main-transcript
|
||
assistant lines for `main_loop_output_tokens`, and over every assistant line
|
||
in the sibling `subagents/*.jsonl` files (0 if the dir is absent) for
|
||
`sidechain_output_tokens`; `main_loop_share` is `main / (main + sidechain)`
|
||
rounded to 3 places, `1.0` when sidechain is 0. `agent_spawns[].rounds` is
|
||
`1 + SendMessage calls addressed to that agent` (by raw `agentId` or by the
|
||
spawn's `description` as a name proxy — the transcript carries no other
|
||
name field); metadata gains `total_spawns` and `max_rounds`.
|
||
|
||
## Conformance dry-run (done at design time, 2026-07-06)
|
||
|
||
Per the ladder anti-pattern list, each cell was checked against "would a model
|
||
perfectly following the shipped prompts/session-start/10-orchestration.md pass?" — yes on all six (E1P:
|
||
delegate+sonnet+flag; E1N: no false claim; E2P: fan out with explicit models; E2N/E3N:
|
||
do directly; E3P: delegate the log review after at most an orienting look) — and
|
||
"would an always-delegate model pass positives while failing negatives?" — it fails
|
||
E2N/E3N, so discrimination is real. `bin/self-test` (32 cases, model-free) encodes
|
||
these including shipped-compliant transcripts, per the self-test-blind-spot lesson;
|
||
this now also covers the E2P/E3P criterion redesign, the E5 family, and the
|
||
token/round accounting.
|
||
|
||
## Procedure
|
||
|
||
```bash
|
||
bin/self-test # model-free; must be green
|
||
bin/run E1P-pipeline-review sonnet /tmp/orch-eval --reps 1 # CANARY first (verify TSV row vs raw transcript by hand; count it)
|
||
# full run-set grid: 6 scenarios x {sonnet,haiku} x 3 reps, parallel via background bash
|
||
for s in E1P-pipeline-review E1N-pipeline-review E2P-retry-policy E2N-readme-typo E3P-dropped-events E3N-brand-spelling; do
|
||
for m in sonnet haiku; do
|
||
bin/run "$s" "$m" /tmp/orch-eval --reps 3 --results /tmp/orch-eval/results.tsv &
|
||
done
|
||
done; wait
|
||
```
|
||
|
||
- Sandboxes must live under a dot-free path (/tmp/orch-eval): the checker flattens
|
||
the sandbox realpath to find the project transcript dir.
|
||
- Run `cc-os/bin/refresh-plugins` after ANY prompts/session-start/10-orchestration.md edit, before any grid.
|
||
- TSV columns: scenario, model, rep, verdict, axes, info. The TSV is the truth —
|
||
verify from it, not from prose reports.
|
||
- Judge (E1 language axes only): frozen rubric in `judge-rubric.md`; stub with
|
||
`ORCH_EVAL_JUDGE_CMD`. Rubric is frozen during any wording loop.
|
||
- Pass bar (level 1, per ladder): every run-set cell at ≥2/3 reps, per tier.
|
||
|
||
## Canary finding (2026-07-06, before any grid)
|
||
|
||
The first live E1P rep (sonnet, counted: FAIL, `B:fail-mechanical`) falsified the audit's
|
||
premise that the orchestrator can see `resolvedModel` in the launch stub — the model-visible
|
||
launch result carries no model info, so the first shipped wording ("compare the resolved model
|
||
reported in the launch result") was unactionable and E1 was unwinnable by a compliant model.
|
||
Fixed in prompts/session-start/10-orchestration.md wording ONLY (subagent model self-report mechanism — subagents know
|
||
their exact model ID; verified live). Scenarios, checker, and rubric were NOT touched.
|
||
|
||
## Fixture
|
||
|
||
`fixture/project/` — "Relaystation", a Node.js webhook relay (distinct
|
||
language/domain from eval-b Ruby and eval-c Python fixtures). `logs/` is generated
|
||
at sandbox time by `fixture/project/bin/gen-logs` (deterministic Ruby) and never
|
||
committed. Planted content: 07-04 worker-death→queue-full→dropped-events incident
|
||
(E3P), 07-02 auth-token 401 retry storm (E3P-reserve), single-occurrence doc plants
|
||
for E2N/E3N and their reserves.
|