summary: How to set up and run an /autoresearch loop over Claude Code skill/hook wording against a model-tier eval grid — which run mode is valid, how to keep iterations fast, and how to keep results trustworthy.
Reach for this before designing or running any `/autoresearch` loop that optimizes Claude Code
skill/hook wording against an eval grid (the os-adr Eval A / Eval B pattern, or any successor).
It encodes what two full grid campaigns (2026-07-03 baseline, 2026-07-04 confirmation) taught
about where the time and the false conclusions actually come from. The loop discipline assumed
throughout: **only wording moves** — checker, fixtures, scenarios, and judge rubric are frozen
for the duration of a loop.
## Prerequisites
- [ ] A deterministic checker with per-axis output (e.g. axis a = triggered, axis b = correct outcome) — axis-level results locate WHICH wording surface to iterate.
- [ ] A committed baseline grid with per-cell results, written to a durable note (not just /tmp TSVs).
- [ ] Plugin caches verified fresh: run `cc-os/bin/refresh-plugins` — installs COPY plugin files into `~/.claude/plugins/cache/`, so SKILL.md/hook edits do NOT reach headless sessions until refreshed.
- [ ] Environment frozen: no plugin renames, command-name changes, or hook rewires mid-loop — registered command names are part of what the model sees, so changing them mid-loop is a confound (see [[cc-os-plugin-skill-naming-convention]]).
## Steps
### Step 1: Pick the valid run mode for what you're measuring
- **Prompted skill-execution evals (Eval A shape):** in-session Agent-tool subagents with pinned `model:` are valid and much cheaper than headless runs.
- **Unprompted-behavior evals (Eval B shape):** headless-only — fresh `claude -p` per rep with cwd = sandbox so the real SessionStart hook fires. In-session subagents inherit the parent session and never get a fresh hook; they are invalid here. Do not conflate the two shapes.
### Step 2: Refresh caches after EVERY wording edit
Each loop iteration edits SKILL.md / hook-note wording in the plugin source. Run
`bin/refresh-plugins` before the grid run of every iteration, or the grid silently measures the
previous iteration's wording. This is the single most likely way a loop produces garbage.
### Step 3: Use a reduced inner-loop grid; save the full grid for confirmation
Iterate only on the target cells (the failing scenarios) plus one passing control cell (to catch
regressions). Run the full grid only to confirm a winning candidate before locking it in. Cells
1 rep/cell is demonstrably noisy: across the two os-adr campaigns, cells flipped between
attempts (haiku W2 axis-a, sonnet W1 overall). Inside the loop use ~3 reps on target cells and
accept a wording change only if it moves the majority of reps. Re-run the full grid with more
reps once wording is stable, to measure variance explicitly.
### Step 6: Read failures at the axis level before writing new wording
Different axes point at different wording surfaces. Example from the os-adr baseline: sonnet W3
fails axis b only (it consults the ADR system, then doesn't propose recording — iterate the
create-skill's "when to record" guidance), while R1 fails axis a (never looks — iterate trigger
salience in the hook note / find-skill description). One "failure" label, two different fixes.
## Verification
- Baseline reproduces before you start: re-run the grid once post-any-environment-change; expect ≥90% cell agreement with the recorded baseline before trusting deltas.
- After a claimed improvement: full grid + the degradation checks pass in the expected pattern.
- Verify results from the primary TSV, not from an orchestrating agent's prose report — an agent report has contradicted the TSV on a cell before; the TSV is the truth.
## Gotchas
- **Stale plugin cache** — symptoms: wording edits have zero effect across iterations, or hooks silently absent from transcripts. Recover: `bin/refresh-plugins`, re-run the iteration.
- **Degradation-check cells (e.g. R4-nograph) are only meaningful paired with a PASS on their non-degraded twin at the same tier** — and at 1 rep they can pass "unexpectedly" (sonnet found the right ADR without the graph on 2026-07-04), which weakens the layer-value evidence rather than proving anything. Don't cite degradation cells as proof at 1 rep.
- **Held-out scenarios stay held out** — never run scenario Task blocks informally/by hand; that contaminates the measurement.
- **Broken cells look like model failures** — a missing `transcript.jsonl` scores FAIL on both axes. Check reasons strings for harness errors before counting a cell as a behavioral result.
- **Account session limits truncate headless reps into plausible verdicts** — a `claude -p` rep cut by a usage/session limit still yields a scoreable transcript: positives FAIL with real-looking axes (task half done) and *negatives can wrongly PASS* (session died before the forbidden action). Sweep EVERY rep's final assistant message for the limit banner ("You've hit your session limit") before counting; excluded reps are infra errors, mechanically identified independent of verdict, so replacing them with fresh reps is not peeking (WS4, 2026-07-08: 15/40 grid reps excluded this way).
- **Model-tier gaps can be total** — haiku 0/8 on Eval B means wording iteration may not reach the lower tier at all; keep one lower-tier canary cell (haiku W2, the one axis-a flicker) in the loop to detect whether wording changes reach it.
- **`set -euo pipefail` + `checker | tee` silently truncates rep loops** — a checker that exits 1 on FAIL, piped into `tee` inside a runner with pipefail, aborts the remaining reps of that cell on the first FAIL, and the outer shell may still report exit 0. Symptom: `--reps 3` produces one TSV row. Guard the pipeline (`|| true`) and check row counts against expected reps before reading results (Eval C, 2026-07-06).
- **When the instrument was wrong, rescore — don't re-run or discard** — if a checker fix changes scoring but the transcript is valid evidence of behavior, re-run `bin/check` on the existing sandbox and replace the row. Re-running spends a rep on a new behavior sample (different question); discarding is peeking bias.
- **Checker semantics need a conformance dry-run, not just a self-test** — model-free self-tests fabricate the transcripts they validate against, so they inherit the designer's assumptions. Before the first grid: cross-check each scenario's expected behavior against what the shipped wording actually instructs given the fixture state ("would a perfectly compliant model pass this cell?"). See the anti-patterns in [[eval-methodology-ladder]] for the full failure shape.
- [[os-adr-eval-b-grid-results-and-observations]] — the concrete baseline numbers and prompting-issue hypothesis any os-adr follow-up loop must compare against.
- [[cc-os-plugin-skill-naming-convention]] — naming/registration mechanics; changing registered command names mid-loop is a confound.
