From 9290c20febbcb8d597ac80ece26430350f8c5e20 Mon Sep 17 00:00:00 2001 From: jared Date: Sat, 4 Jul 2026 13:20:34 -0400 Subject: [PATCH] vault: session notes 2026-07-04 --- ...dr-eval-b-grid-results-and-observations.md | 23 +++++ howto/running-autoresearch-skill-evals.md | 97 +++++++++++++++++++ journal/2026-07-04.md | 7 ++ 3 files changed, 127 insertions(+) create mode 100644 howto/running-autoresearch-skill-evals.md diff --git a/2026-07-03-os-adr-eval-b-grid-results-and-observations.md b/2026-07-03-os-adr-eval-b-grid-results-and-observations.md index ad904bb..7ba35ea 100644 --- a/2026-07-03-os-adr-eval-b-grid-results-and-observations.md +++ b/2026-07-03-os-adr-eval-b-grid-results-and-observations.md @@ -65,6 +65,29 @@ failed for haiku. this note** for the baseline numbers and the prompting-issue hypothesis, so the loop has a concrete before/after to compare against. +## Confirmation run (2026-07-04) + +Full grid re-run after fixing stale plugin caches (os-adr's installed cache had drifted — missing +SessionStart hook + 3 CLIs) and switching skill registration to namespaced commands +(`/os-adr:find`, not bare `/find`). **Baseline confirmed: haiku 0/8, sonnet 5/8 with the same two +behavioral failures (W3, R1).** 15/16 cells reproduced; the 16th (W2/sonnet) was a harness error +(no transcript), re-run separately. + +New information beyond confirmation: + +- **Sonnet W3 is an axis-b failure**: A:PASS / B:FAIL — it consults the ADR system unprompted, + then doesn't propose recording the reversal. The wording target is the create-skill's + "when to record" guidance, NOT trigger salience. R1 remains a pure axis-a (trigger) failure. +- **R4-nograph/sonnet PASSed** (expected FAIL) — sonnet found the correct ADR without the graph + layer this rep. At 1 rep this weakens (doesn't refute) the graph-layer-value evidence. +- **Haiku's first axis-a pass** (W2): one flicker of unprompted consultation — keep as a + lower-tier canary cell in any wording loop. +- **Variance is real at 1 rep/cell** (haiku W2 axis-a and sonnet W1 flipped between attempts) — + wording loops need ~3 reps on target cells. + +Procedure and efficiency/quality lessons for the follow-up loop: +[[running-autoresearch-skill-evals]]. + ## Source - cc-os repo, `plugins/os-adr/eval-b/` (harness) and `plugins/os-adr/eval-b/README.md` (status) diff --git a/howto/running-autoresearch-skill-evals.md b/howto/running-autoresearch-skill-evals.md new file mode 100644 index 0000000..88a5fee --- /dev/null +++ b/howto/running-autoresearch-skill-evals.md @@ -0,0 +1,97 @@ +--- +type: howto +title: Running autoresearch skill evals (setup, efficiency, quality) +summary: How to set up and run an /autoresearch loop over Claude Code skill/hook wording against a model-tier eval grid — which run mode is valid, how to keep iterations fast, and how to keep results trustworthy. +tags: + - type/howto + - domain/llm-evaluation + - tool/claude-code + - tool/autoresearch + - project/cc-os +scope: global +last_updated: 2026-07-04 +date: 2026-07-04 +related: + - os-adr-eval-b-grid-results-and-observations + - cc-os-plugin-skill-naming-convention +source: cc-os +--- + +# Running autoresearch skill evals + +## Opening + +Reach for this before designing or running any `/autoresearch` loop that optimizes Claude Code +skill/hook wording against an eval grid (the os-adr Eval A / Eval B pattern, or any successor). +It encodes what two full grid campaigns (2026-07-03 baseline, 2026-07-04 confirmation) taught +about where the time and the false conclusions actually come from. The loop discipline assumed +throughout: **only wording moves** — checker, fixtures, scenarios, and judge rubric are frozen +for the duration of a loop. + +## Prerequisites + +- [ ] A deterministic checker with per-axis output (e.g. axis a = triggered, axis b = correct outcome) — axis-level results locate WHICH wording surface to iterate. +- [ ] A committed baseline grid with per-cell results, written to a durable note (not just /tmp TSVs). +- [ ] Plugin caches verified fresh: run `cc-os/bin/refresh-plugins` — installs COPY plugin files into `~/.claude/plugins/cache/`, so SKILL.md/hook edits do NOT reach headless sessions until refreshed. +- [ ] Environment frozen: no plugin renames, command-name changes, or hook rewires mid-loop — registered command names are part of what the model sees, so changing them mid-loop is a confound (see [[cc-os-plugin-skill-naming-convention]]). + +## Steps + +### Step 1: Pick the valid run mode for what you're measuring + +- **Prompted skill-execution evals (Eval A shape):** in-session Agent-tool subagents with pinned `model:` are valid and much cheaper than headless runs. +- **Unprompted-behavior evals (Eval B shape):** headless-only — fresh `claude -p` per rep with cwd = sandbox so the real SessionStart hook fires. In-session subagents inherit the parent session and never get a fresh hook; they are invalid here. Do not conflate the two shapes. + +### Step 2: Refresh caches after EVERY wording edit + +Each loop iteration edits SKILL.md / hook-note wording in the plugin source. Run +`bin/refresh-plugins` before the grid run of every iteration, or the grid silently measures the +previous iteration's wording. This is the single most likely way a loop produces garbage. + +### Step 3: Use a reduced inner-loop grid; save the full grid for confirmation + +Iterate only on the target cells (the failing scenarios) plus one passing control cell (to catch +regressions). Run the full grid only to confirm a winning candidate before locking it in. Cells +that aren't moving are pure cost inside the loop. + +### Step 4: Parallelize cells; drive scripts directly + +Each headless `claude -p` cell is fully independent — run them concurrently (background the +per-cell `bin/run` invocations). Sandbox setup is a sub-second fixture copy; the live model +session is the irreducible unit (~30s–5min per cell). A sequential 16-cell grid takes ~25–30 +min; parallel, it's bounded by the slowest cell (~5 min). Drive the grid script directly from +the session with background Bash — do NOT wrap it in a babysitting subagent (agents self-pause, +need resuming, and re-runs collide with existing sandboxes: "refusing to overwrite"). + +### Step 5: Use enough reps to beat the noise + +1 rep/cell is demonstrably noisy: across the two os-adr campaigns, cells flipped between +attempts (haiku W2 axis-a, sonnet W1 overall). Inside the loop use ~3 reps on target cells and +accept a wording change only if it moves the majority of reps. Re-run the full grid with more +reps once wording is stable, to measure variance explicitly. + +### Step 6: Read failures at the axis level before writing new wording + +Different axes point at different wording surfaces. Example from the os-adr baseline: sonnet W3 +fails axis b only (it consults the ADR system, then doesn't propose recording — iterate the +create-skill's "when to record" guidance), while R1 fails axis a (never looks — iterate trigger +salience in the hook note / find-skill description). One "failure" label, two different fixes. + +## Verification + +- Baseline reproduces before you start: re-run the grid once post-any-environment-change; expect ≥90% cell agreement with the recorded baseline before trusting deltas. +- After a claimed improvement: full grid + the degradation checks pass in the expected pattern. +- Verify results from the primary TSV, not from an orchestrating agent's prose report — an agent report has contradicted the TSV on a cell before; the TSV is the truth. + +## Gotchas + +- **Stale plugin cache** — symptoms: wording edits have zero effect across iterations, or hooks silently absent from transcripts. Recover: `bin/refresh-plugins`, re-run the iteration. +- **Degradation-check cells (e.g. R4-nograph) are only meaningful paired with a PASS on their non-degraded twin at the same tier** — and at 1 rep they can pass "unexpectedly" (sonnet found the right ADR without the graph on 2026-07-04), which weakens the layer-value evidence rather than proving anything. Don't cite degradation cells as proof at 1 rep. +- **Held-out scenarios stay held out** — never run scenario Task blocks informally/by hand; that contaminates the measurement. +- **Broken cells look like model failures** — a missing `transcript.jsonl` scores FAIL on both axes. Check reasons strings for harness errors before counting a cell as a behavioral result. +- **Model-tier gaps can be total** — haiku 0/8 on Eval B means wording iteration may not reach the lower tier at all; keep one lower-tier canary cell (haiku W2, the one axis-a flicker) in the loop to detect whether wording changes reach it. + +## Related + +- [[os-adr-eval-b-grid-results-and-observations]] — the concrete baseline numbers and prompting-issue hypothesis any os-adr follow-up loop must compare against. +- [[cc-os-plugin-skill-naming-convention]] — naming/registration mechanics; changing registered command names mid-loop is a confound. diff --git a/journal/2026-07-04.md b/journal/2026-07-04.md index 2b31cfb..2b6cb98 100644 --- a/journal/2026-07-04.md +++ b/journal/2026-07-04.md @@ -450,3 +450,10 @@ tags: [scope/global, type/log] **Reason:** other **Vault notes touched:** (none) + +## Session — 2026-07-04T17:19:16Z + +**Project:** /home/jared/servers/ovh-prod +**Reason:** prompt_input_exit +**Vault notes touched:** +(none)