Running autoresearch skill evals

Opening

Reach for this before designing or running any /autoresearch loop that optimizes Claude Code skill/hook wording against an eval grid (the os-adr Eval A / Eval B pattern, or any successor). It encodes what two full grid campaigns (2026-07-03 baseline, 2026-07-04 confirmation) taught about where the time and the false conclusions actually come from. The loop discipline assumed throughout: only wording moves — checker, fixtures, scenarios, and judge rubric are frozen for the duration of a loop.

Prerequisites

A deterministic checker with per-axis output (e.g. axis a = triggered, axis b = correct outcome) — axis-level results locate WHICH wording surface to iterate.
A committed baseline grid with per-cell results, written to a durable note (not just /tmp TSVs).
Plugin caches verified fresh: run cc-os/bin/refresh-plugins — installs COPY plugin files into ~/.claude/plugins/cache/, so SKILL.md/hook edits do NOT reach headless sessions until refreshed.
Environment frozen: no plugin renames, command-name changes, or hook rewires mid-loop — registered command names are part of what the model sees, so changing them mid-loop is a confound (see cc-os-plugin-skill-naming-convention).

Steps

Step 1: Pick the valid run mode for what you're measuring

Prompted skill-execution evals (Eval A shape): in-session Agent-tool subagents with pinned model: are valid and much cheaper than headless runs.
Unprompted-behavior evals (Eval B shape): headless-only — fresh claude -p per rep with cwd = sandbox so the real SessionStart hook fires. In-session subagents inherit the parent session and never get a fresh hook; they are invalid here. Do not conflate the two shapes.

Step 2: Refresh caches after EVERY wording edit

Each loop iteration edits SKILL.md / hook-note wording in the plugin source. Run bin/refresh-plugins before the grid run of every iteration, or the grid silently measures the previous iteration's wording. This is the single most likely way a loop produces garbage.

Step 3: Use a reduced inner-loop grid; save the full grid for confirmation

Iterate only on the target cells (the failing scenarios) plus one passing control cell (to catch regressions). Run the full grid only to confirm a winning candidate before locking it in. Cells that aren't moving are pure cost inside the loop.

Step 4: Parallelize cells; drive scripts directly

Each headless claude -p cell is fully independent — run them concurrently (background the per-cell bin/run invocations). Sandbox setup is a sub-second fixture copy; the live model session is the irreducible unit (~30s–5min per cell). A sequential 16-cell grid takes ~25–30 min; parallel, it's bounded by the slowest cell (~5 min). Drive the grid script directly from the session with background Bash — do NOT wrap it in a babysitting subagent (agents self-pause, need resuming, and re-runs collide with existing sandboxes: "refusing to overwrite").

Step 5: Use enough reps to beat the noise

1 rep/cell is demonstrably noisy: across the two os-adr campaigns, cells flipped between attempts (haiku W2 axis-a, sonnet W1 overall). Inside the loop use ~3 reps on target cells and accept a wording change only if it moves the majority of reps. Re-run the full grid with more reps once wording is stable, to measure variance explicitly.

Step 6: Read failures at the axis level before writing new wording

Different axes point at different wording surfaces. Example from the os-adr baseline: sonnet W3 fails axis b only (it consults the ADR system, then doesn't propose recording — iterate the create-skill's "when to record" guidance), while R1 fails axis a (never looks — iterate trigger salience in the hook note / find-skill description). One "failure" label, two different fixes.

Verification

Baseline reproduces before you start: re-run the grid once post-any-environment-change; expect ≥90% cell agreement with the recorded baseline before trusting deltas.
After a claimed improvement: full grid + the degradation checks pass in the expected pattern.
Verify results from the primary TSV, not from an orchestrating agent's prose report — an agent report has contradicted the TSV on a cell before; the TSV is the truth.

Gotchas

Stale plugin cache — symptoms: wording edits have zero effect across iterations, or hooks silently absent from transcripts. Recover: bin/refresh-plugins, re-run the iteration.
Degradation-check cells (e.g. R4-nograph) are only meaningful paired with a PASS on their non-degraded twin at the same tier — and at 1 rep they can pass "unexpectedly" (sonnet found the right ADR without the graph on 2026-07-04), which weakens the layer-value evidence rather than proving anything. Don't cite degradation cells as proof at 1 rep.
Held-out scenarios stay held out — never run scenario Task blocks informally/by hand; that contaminates the measurement.
Broken cells look like model failures — a missing transcript.jsonl scores FAIL on both axes. Check reasons strings for harness errors before counting a cell as a behavioral result.
Model-tier gaps can be total — haiku 0/8 on Eval B means wording iteration may not reach the lower tier at all; keep one lower-tier canary cell (haiku W2, the one axis-a flicker) in the loop to detect whether wording changes reach it.

os-adr-eval-b-grid-results-and-observations — the concrete baseline numbers and prompting-issue hypothesis any os-adr follow-up loop must compare against.
cc-os-plugin-skill-naming-convention — naming/registration mechanics; changing registered command names mid-loop is a confound.

6.2 KiB Raw Blame History Unescape Escape