SecondBrain/howto/running-autoresearch-skill-evals.md

---
type: howto
title: Running autoresearch skill evals (setup, efficiency, quality)
summary: How to set up and run an /autoresearch loop over Claude Code skill/hook wording against a model-tier eval grid — which run mode is valid, how to keep iterations fast, and how to keep results trustworthy.
tags:
  - type/howto
  - domain/llm-evaluation
  - tool/claude-code
  - tool/autoresearch
  - project/cc-os
scope: global
last_updated: 2026-07-04
date: 2026-07-04
related:
  - os-adr-eval-b-grid-results-and-observations
  - cc-os-plugin-skill-naming-convention
source: cc-os
---

# Running autoresearch skill evals

## Opening

Reach for this before designing or running any `/autoresearch` loop that optimizes Claude Code
skill/hook wording against an eval grid (the os-adr Eval A / Eval B pattern, or any successor).
It encodes what two full grid campaigns (2026-07-03 baseline, 2026-07-04 confirmation) taught
about where the time and the false conclusions actually come from. The loop discipline assumed
throughout: **only wording moves** — checker, fixtures, scenarios, and judge rubric are frozen
for the duration of a loop.

## Prerequisites

- [ ] A deterministic checker with per-axis output (e.g. axis a = triggered, axis b = correct outcome) — axis-level results locate WHICH wording surface to iterate.
- [ ] A committed baseline grid with per-cell results, written to a durable note (not just /tmp TSVs).
- [ ] Plugin caches verified fresh: run `cc-os/bin/refresh-plugins` — installs COPY plugin files into `~/.claude/plugins/cache/`, so SKILL.md/hook edits do NOT reach headless sessions until refreshed.
- [ ] Environment frozen: no plugin renames, command-name changes, or hook rewires mid-loop — registered command names are part of what the model sees, so changing them mid-loop is a confound (see [[cc-os-plugin-skill-naming-convention]]).

## Steps

### Step 1: Pick the valid run mode for what you're measuring

- **Prompted skill-execution evals (Eval A shape):** in-session Agent-tool subagents with pinned `model:` are valid and much cheaper than headless runs.
- **Unprompted-behavior evals (Eval B shape):** headless-only — fresh `claude -p` per rep with cwd = sandbox so the real SessionStart hook fires. In-session subagents inherit the parent session and never get a fresh hook; they are invalid here. Do not conflate the two shapes.

### Step 2: Refresh caches after EVERY wording edit

Each loop iteration edits SKILL.md / hook-note wording in the plugin source. Run
`bin/refresh-plugins` before the grid run of every iteration, or the grid silently measures the
previous iteration's wording. This is the single most likely way a loop produces garbage.

### Step 3: Use a reduced inner-loop grid; save the full grid for confirmation

Iterate only on the target cells (the failing scenarios) plus one passing control cell (to catch
regressions). Run the full grid only to confirm a winning candidate before locking it in. Cells
that aren't moving are pure cost inside the loop.

### Step 4: Parallelize cells; drive scripts directly

Each headless `claude -p` cell is fully independent — run them concurrently (background the
per-cell `bin/run` invocations). Sandbox setup is a sub-second fixture copy; the live model
session is the irreducible unit (~30s–5min per cell). A sequential 16-cell grid takes ~25–30
min; parallel, it's bounded by the slowest cell (~5 min). Drive the grid script directly from
the session with background Bash — do NOT wrap it in a babysitting subagent (agents self-pause,
need resuming, and re-runs collide with existing sandboxes: "refusing to overwrite").

### Step 5: Use enough reps to beat the noise

1 rep/cell is demonstrably noisy: across the two os-adr campaigns, cells flipped between
attempts (haiku W2 axis-a, sonnet W1 overall). Inside the loop use ~3 reps on target cells and
accept a wording change only if it moves the majority of reps. Re-run the full grid with more
reps once wording is stable, to measure variance explicitly.

### Step 6: Read failures at the axis level before writing new wording

Different axes point at different wording surfaces. Example from the os-adr baseline: sonnet W3
fails axis b only (it consults the ADR system, then doesn't propose recording — iterate the
create-skill's "when to record" guidance), while R1 fails axis a (never looks — iterate trigger
salience in the hook note / find-skill description). One "failure" label, two different fixes.

## Verification

- Baseline reproduces before you start: re-run the grid once post-any-environment-change; expect ≥90% cell agreement with the recorded baseline before trusting deltas.
- After a claimed improvement: full grid + the degradation checks pass in the expected pattern.
- Verify results from the primary TSV, not from an orchestrating agent's prose report — an agent report has contradicted the TSV on a cell before; the TSV is the truth.

## Gotchas

- **Stale plugin cache** — symptoms: wording edits have zero effect across iterations, or hooks silently absent from transcripts. Recover: `bin/refresh-plugins`, re-run the iteration.
- **Degradation-check cells (e.g. R4-nograph) are only meaningful paired with a PASS on their non-degraded twin at the same tier** — and at 1 rep they can pass "unexpectedly" (sonnet found the right ADR without the graph on 2026-07-04), which weakens the layer-value evidence rather than proving anything. Don't cite degradation cells as proof at 1 rep.
- **Held-out scenarios stay held out** — never run scenario Task blocks informally/by hand; that contaminates the measurement.
- **Broken cells look like model failures** — a missing `transcript.jsonl` scores FAIL on both axes. Check reasons strings for harness errors before counting a cell as a behavioral result.
- **Model-tier gaps can be total** — haiku 0/8 on Eval B means wording iteration may not reach the lower tier at all; keep one lower-tier canary cell (haiku W2, the one axis-a flicker) in the loop to detect whether wording changes reach it.

## Related

- [[os-adr-eval-b-grid-results-and-observations]] — the concrete baseline numbers and prompting-issue hypothesis any os-adr follow-up loop must compare against.
- [[cc-os-plugin-skill-naming-convention]] — naming/registration mechanics; changing registered command names mid-loop is a confound.