SecondBrain/howto/running-autoresearch-skill-evals.md

---
type: howto
title: Running autoresearch skill evals (setup, efficiency, quality)
summary: How to set up and run an /autoresearch loop over Claude Code skill/hook wording against a model-tier eval grid — which run mode is valid, how to keep iterations fast, and how to keep results trustworthy.
tags:
  - type/howto
  - domain/llm-evaluation
  - tool/claude-code
  - tool/autoresearch
  - project/cc-os
scope: global
last_updated: 2026-07-08
date: 2026-07-04
related:
  - os-adr-eval-b-grid-results-and-observations
  - cc-os-plugin-skill-naming-convention
source: cc-os
---

# Running autoresearch skill evals

## Opening

Reach for this before designing or running any `/autoresearch` loop that optimizes Claude Code
skill/hook wording against an eval grid (the os-adr Eval A / Eval B pattern, or any successor).
It encodes what two full grid campaigns (2026-07-03 baseline, 2026-07-04 confirmation) taught
about where the time and the false conclusions actually come from. The loop discipline assumed
throughout: **only wording moves** — checker, fixtures, scenarios, and judge rubric are frozen
for the duration of a loop.

## Prerequisites

- [ ] A deterministic checker with per-axis output (e.g. axis a = triggered, axis b = correct outcome) — axis-level results locate WHICH wording surface to iterate.
- [ ] A committed baseline grid with per-cell results, written to a durable note (not just /tmp TSVs).
- [ ] Plugin caches verified fresh: run `cc-os/bin/refresh-plugins` — installs COPY plugin files into `~/.claude/plugins/cache/`, so SKILL.md/hook edits do NOT reach headless sessions until refreshed.
- [ ] Environment frozen: no plugin renames, command-name changes, or hook rewires mid-loop — registered command names are part of what the model sees, so changing them mid-loop is a confound (see [[cc-os-plugin-skill-naming-convention]]).

## Steps

### Step 1: Pick the valid run mode for what you're measuring

- **Prompted skill-execution evals (Eval A shape):** in-session Agent-tool subagents with pinned `model:` are valid and much cheaper than headless runs.
- **Unprompted-behavior evals (Eval B shape):** headless-only — fresh `claude -p` per rep with cwd = sandbox so the real SessionStart hook fires. In-session subagents inherit the parent session and never get a fresh hook; they are invalid here. Do not conflate the two shapes.

### Step 2: Refresh caches after EVERY wording edit

Each loop iteration edits SKILL.md / hook-note wording in the plugin source. Run
`bin/refresh-plugins` before the grid run of every iteration, or the grid silently measures the
previous iteration's wording. This is the single most likely way a loop produces garbage.

### Step 3: Use a reduced inner-loop grid; save the full grid for confirmation

Iterate only on the target cells (the failing scenarios) plus one passing control cell (to catch
regressions). Run the full grid only to confirm a winning candidate before locking it in. Cells
that aren't moving are pure cost inside the loop.

### Step 4: Parallelize cells; drive scripts directly

Each headless `claude -p` cell is fully independent — run them concurrently (background the
per-cell `bin/run` invocations). Sandbox setup is a sub-second fixture copy; the live model
session is the irreducible unit (~30s–5min per cell). A sequential 16-cell grid takes ~25–30
min; parallel, it's bounded by the slowest cell (~5 min). Drive the grid script directly from
the session with background Bash — do NOT wrap it in a babysitting subagent (agents self-pause,
need resuming, and re-runs collide with existing sandboxes: "refusing to overwrite").

### Step 4b: Canary-cell the first live run of any new harness

Before launching a full grid on a harness that has never run live, run ONE cell first and
verify its TSV row against the raw transcript by hand (did the hook fire? does the reason
string match what the model actually did?). Count the canary's result in the measurement —
never discard it (peeking bias). The os-adr Eval C first grid (2026-07-06) caught two harness
defects this way at a cost of 1 rep instead of 36: a runner rep-loop bug and a checker that
failed instruction-compliant behavior.

### Step 5: Use enough reps to beat the noise

1 rep/cell is demonstrably noisy: across the two os-adr campaigns, cells flipped between
attempts (haiku W2 axis-a, sonnet W1 overall). Inside the loop use ~3 reps on target cells and
accept a wording change only if it moves the majority of reps. Re-run the full grid with more
reps once wording is stable, to measure variance explicitly.

### Step 6: Read failures at the axis level before writing new wording

Different axes point at different wording surfaces. Example from the os-adr baseline: sonnet W3
fails axis b only (it consults the ADR system, then doesn't propose recording — iterate the
create-skill's "when to record" guidance), while R1 fails axis a (never looks — iterate trigger
salience in the hook note / find-skill description). One "failure" label, two different fixes.

## Verification

- Baseline reproduces before you start: re-run the grid once post-any-environment-change; expect ≥90% cell agreement with the recorded baseline before trusting deltas.
- After a claimed improvement: full grid + the degradation checks pass in the expected pattern.
- Verify results from the primary TSV, not from an orchestrating agent's prose report — an agent report has contradicted the TSV on a cell before; the TSV is the truth.

## Gotchas

- **Stale plugin cache** — symptoms: wording edits have zero effect across iterations, or hooks silently absent from transcripts. Recover: `bin/refresh-plugins`, re-run the iteration.
- **Degradation-check cells (e.g. R4-nograph) are only meaningful paired with a PASS on their non-degraded twin at the same tier** — and at 1 rep they can pass "unexpectedly" (sonnet found the right ADR without the graph on 2026-07-04), which weakens the layer-value evidence rather than proving anything. Don't cite degradation cells as proof at 1 rep.
- **Held-out scenarios stay held out** — never run scenario Task blocks informally/by hand; that contaminates the measurement.
- **Broken cells look like model failures** — a missing `transcript.jsonl` scores FAIL on both axes. Check reasons strings for harness errors before counting a cell as a behavioral result.
- **Account session limits truncate headless reps into plausible verdicts** — a `claude -p` rep cut by a usage/session limit still yields a scoreable transcript: positives FAIL with real-looking axes (task half done) and *negatives can wrongly PASS* (session died before the forbidden action). Sweep EVERY rep's final assistant message for the limit banner ("You've hit your session limit") before counting; excluded reps are infra errors, mechanically identified independent of verdict, so replacing them with fresh reps is not peeking (WS4, 2026-07-08: 15/40 grid reps excluded this way).
- **Model-tier gaps can be total** — haiku 0/8 on Eval B means wording iteration may not reach the lower tier at all; keep one lower-tier canary cell (haiku W2, the one axis-a flicker) in the loop to detect whether wording changes reach it.
- **`set -euo pipefail` + `checker | tee` silently truncates rep loops** — a checker that exits 1 on FAIL, piped into `tee` inside a runner with pipefail, aborts the remaining reps of that cell on the first FAIL, and the outer shell may still report exit 0. Symptom: `--reps 3` produces one TSV row. Guard the pipeline (`|| true`) and check row counts against expected reps before reading results (Eval C, 2026-07-06).
- **When the instrument was wrong, rescore — don't re-run or discard** — if a checker fix changes scoring but the transcript is valid evidence of behavior, re-run `bin/check` on the existing sandbox and replace the row. Re-running spends a rep on a new behavior sample (different question); discarding is peeking bias.
- **Checker semantics need a conformance dry-run, not just a self-test** — model-free self-tests fabricate the transcripts they validate against, so they inherit the designer's assumptions. Before the first grid: cross-check each scenario's expected behavior against what the shipped wording actually instructs given the fixture state ("would a perfectly compliant model pass this cell?"). See the anti-patterns in [[eval-methodology-ladder]] for the full failure shape.

## Related

- [[os-adr-eval-b-grid-results-and-observations]] — the concrete baseline numbers and prompting-issue hypothesis any os-adr follow-up loop must compare against.
- [[cc-os-plugin-skill-naming-convention]] — naming/registration mechanics; changing registered command names mid-loop is a confound.