vault: session notes 2026-07-04

2026-07-03-os-adr-eval-b-grid-results-and-observations.md

@@ -65,6 +65,29 @@ failed for haiku.
   this note** for the baseline numbers and the prompting-issue hypothesis, so the loop has a
   concrete before/after to compare against.
 
+## Confirmation run (2026-07-04)
+
+Full grid re-run after fixing stale plugin caches (os-adr's installed cache had drifted — missing
+SessionStart hook + 3 CLIs) and switching skill registration to namespaced commands
+(`/os-adr:find`, not bare `/find`). **Baseline confirmed: haiku 0/8, sonnet 5/8 with the same two
+behavioral failures (W3, R1).** 15/16 cells reproduced; the 16th (W2/sonnet) was a harness error
+(no transcript), re-run separately.
+
+New information beyond confirmation:
+
+- **Sonnet W3 is an axis-b failure**: A:PASS / B:FAIL — it consults the ADR system unprompted,
+  then doesn't propose recording the reversal. The wording target is the create-skill's
+  "when to record" guidance, NOT trigger salience. R1 remains a pure axis-a (trigger) failure.
+- **R4-nograph/sonnet PASSed** (expected FAIL) — sonnet found the correct ADR without the graph
+  layer this rep. At 1 rep this weakens (doesn't refute) the graph-layer-value evidence.
+- **Haiku's first axis-a pass** (W2): one flicker of unprompted consultation — keep as a
+  lower-tier canary cell in any wording loop.
+- **Variance is real at 1 rep/cell** (haiku W2 axis-a and sonnet W1 flipped between attempts) —
+  wording loops need ~3 reps on target cells.
+
+Procedure and efficiency/quality lessons for the follow-up loop:
+[[running-autoresearch-skill-evals]].
+
 ## Source
 
 - cc-os repo, `plugins/os-adr/eval-b/` (harness) and `plugins/os-adr/eval-b/README.md` (status)

howto/running-autoresearch-skill-evals.md

@@ -0,0 +1,97 @@
+---
+type: howto
+title: Running autoresearch skill evals (setup, efficiency, quality)
+summary: How to set up and run an /autoresearch loop over Claude Code skill/hook wording against a model-tier eval grid — which run mode is valid, how to keep iterations fast, and how to keep results trustworthy.
+tags:
+  - type/howto
+  - domain/llm-evaluation
+  - tool/claude-code
+  - tool/autoresearch
+  - project/cc-os
+scope: global
+last_updated: 2026-07-04
+date: 2026-07-04
+related:
+  - os-adr-eval-b-grid-results-and-observations
+  - cc-os-plugin-skill-naming-convention
+source: cc-os
+---
+
+# Running autoresearch skill evals
+
+## Opening
+
+Reach for this before designing or running any `/autoresearch` loop that optimizes Claude Code
+skill/hook wording against an eval grid (the os-adr Eval A / Eval B pattern, or any successor).
+It encodes what two full grid campaigns (2026-07-03 baseline, 2026-07-04 confirmation) taught
+about where the time and the false conclusions actually come from. The loop discipline assumed
+throughout: **only wording moves** — checker, fixtures, scenarios, and judge rubric are frozen
+for the duration of a loop.
+
+## Prerequisites
+
+- [ ] A deterministic checker with per-axis output (e.g. axis a = triggered, axis b = correct outcome) — axis-level results locate WHICH wording surface to iterate.
+- [ ] A committed baseline grid with per-cell results, written to a durable note (not just /tmp TSVs).
+- [ ] Plugin caches verified fresh: run `cc-os/bin/refresh-plugins` — installs COPY plugin files into `~/.claude/plugins/cache/`, so SKILL.md/hook edits do NOT reach headless sessions until refreshed.
+- [ ] Environment frozen: no plugin renames, command-name changes, or hook rewires mid-loop — registered command names are part of what the model sees, so changing them mid-loop is a confound (see [[cc-os-plugin-skill-naming-convention]]).
+
+## Steps
+
+### Step 1: Pick the valid run mode for what you're measuring
+
+- **Prompted skill-execution evals (Eval A shape):** in-session Agent-tool subagents with pinned `model:` are valid and much cheaper than headless runs.
+- **Unprompted-behavior evals (Eval B shape):** headless-only — fresh `claude -p` per rep with cwd = sandbox so the real SessionStart hook fires. In-session subagents inherit the parent session and never get a fresh hook; they are invalid here. Do not conflate the two shapes.
+
+### Step 2: Refresh caches after EVERY wording edit
+
+Each loop iteration edits SKILL.md / hook-note wording in the plugin source. Run
+`bin/refresh-plugins` before the grid run of every iteration, or the grid silently measures the
+previous iteration's wording. This is the single most likely way a loop produces garbage.
+
+### Step 3: Use a reduced inner-loop grid; save the full grid for confirmation
+
+Iterate only on the target cells (the failing scenarios) plus one passing control cell (to catch
+regressions). Run the full grid only to confirm a winning candidate before locking it in. Cells
+that aren't moving are pure cost inside the loop.
+
+### Step 4: Parallelize cells; drive scripts directly
+
+Each headless `claude -p` cell is fully independent — run them concurrently (background the
+per-cell `bin/run` invocations). Sandbox setup is a sub-second fixture copy; the live model
+session is the irreducible unit (~30s–5min per cell). A sequential 16-cell grid takes ~25–30
+min; parallel, it's bounded by the slowest cell (~5 min). Drive the grid script directly from
+the session with background Bash — do NOT wrap it in a babysitting subagent (agents self-pause,
+need resuming, and re-runs collide with existing sandboxes: "refusing to overwrite").
+
+### Step 5: Use enough reps to beat the noise
+
+1 rep/cell is demonstrably noisy: across the two os-adr campaigns, cells flipped between
+attempts (haiku W2 axis-a, sonnet W1 overall). Inside the loop use ~3 reps on target cells and
+accept a wording change only if it moves the majority of reps. Re-run the full grid with more
+reps once wording is stable, to measure variance explicitly.
+
+### Step 6: Read failures at the axis level before writing new wording
+
+Different axes point at different wording surfaces. Example from the os-adr baseline: sonnet W3
+fails axis b only (it consults the ADR system, then doesn't propose recording — iterate the
+create-skill's "when to record" guidance), while R1 fails axis a (never looks — iterate trigger
+salience in the hook note / find-skill description). One "failure" label, two different fixes.
+
+## Verification
+
+- Baseline reproduces before you start: re-run the grid once post-any-environment-change; expect ≥90% cell agreement with the recorded baseline before trusting deltas.
+- After a claimed improvement: full grid + the degradation checks pass in the expected pattern.
+- Verify results from the primary TSV, not from an orchestrating agent's prose report — an agent report has contradicted the TSV on a cell before; the TSV is the truth.
+
+## Gotchas
+
+- **Stale plugin cache** — symptoms: wording edits have zero effect across iterations, or hooks silently absent from transcripts. Recover: `bin/refresh-plugins`, re-run the iteration.
+- **Degradation-check cells (e.g. R4-nograph) are only meaningful paired with a PASS on their non-degraded twin at the same tier** — and at 1 rep they can pass "unexpectedly" (sonnet found the right ADR without the graph on 2026-07-04), which weakens the layer-value evidence rather than proving anything. Don't cite degradation cells as proof at 1 rep.
+- **Held-out scenarios stay held out** — never run scenario Task blocks informally/by hand; that contaminates the measurement.
+- **Broken cells look like model failures** — a missing `transcript.jsonl` scores FAIL on both axes. Check reasons strings for harness errors before counting a cell as a behavioral result.
+- **Model-tier gaps can be total** — haiku 0/8 on Eval B means wording iteration may not reach the lower tier at all; keep one lower-tier canary cell (haiku W2, the one axis-a flicker) in the loop to detect whether wording changes reach it.
+
+## Related
+
+- [[os-adr-eval-b-grid-results-and-observations]] — the concrete baseline numbers and prompting-issue hypothesis any os-adr follow-up loop must compare against.
+- [[cc-os-plugin-skill-naming-convention]] — naming/registration mechanics; changing registered command names mid-loop is a confound.

journal/2026-07-04.md

@@ -450,3 +450,10 @@ tags: [scope/global, type/log]
 **Reason:** other
 **Vault notes touched:**
 (none)
+
+## Session — 2026-07-04T17:19:16Z
+
+**Project:** /home/jared/servers/ovh-prod
+**Reason:** prompt_input_exit
+**Vault notes touched:**
+(none)