Evaluation methodology for LLM skill behavior

Navigation hub for the full evaluation pipeline: from harness design through wording tuning to production rollout and feedback loops. Each note covers a distinct phase or pattern; read the one that matches your current task.

I want to...

I want to...	Read this
Understand how to design eval harnesses and iterate wording safely	running-autoresearch-skill-evals
See the os-adr Eval B baseline (unprompted triggering test)	os-adr-eval-b-grid-results-and-observations
See the os-adr Eval B wording experiment results (5-iteration improvement)	os-adr-eval-b-wording-experiment-hypotheses
Learn the ladder approach for progressively harder evals	eval-methodology-ladder
Set up production auditing and close the feedback loop	eval-methodology-irl-feedback-loop

The Full Progression

Phase 1: Initial eval design and baseline measurement

running-autoresearch-skill-evals — Procedure for running evaluations and wording loops. When to use in-session vs headless runners, how to refresh plugin caches so wording edits take effect, how to parallelize runs and interpret results at the axis level.

os-adr-eval-b-grid-results-and-observations — Concrete baseline eval for os-adr unprompted-triggering behavior (haiku 0/8, sonnet 5/8). Shows what a held-out measurement looks like and why the results point to a prompting/wording issue rather than a capability gap.

Phase 2: Controlled wording optimization

os-adr-eval-b-wording-experiment-hypotheses — Five-iteration wording loop that improved the baseline (sonnet 8/8, haiku 7/8). Demonstrates the hypothesis→verdict tracking pattern, per-iteration results, and how to close gaps via targeted wording in specific channels (hook note, CLAUDE.md, skill description).

Key result: Trigger-conditioned phrasing ("when you encounter X, do Y") outperforms inventory statements; each rule must live where its precondition is visible (step-2 wording in skill bodies, not hook notes).

Phase 3: Ladder progression and generalization testing

eval-methodology-ladder — Design strategy for successive evals at increasing difficulty: Level 1 (clear-cue baseline), Level 2 (ambiguous-cue discrimination), Level 3 (edge-case over-trigger risks). Pairs positive/negative scenarios at every level. Freeze evaluation surfaces (checker, fixture, scenarios) so wording can move independently. Run-set vs held-out reserve discipline to maintain measurement validity after tuning.

Key pattern: Per-level pass bars, not aggregate scores. Once a level is clear, move to the next. Non-monotonic difficulty (passing hard does not imply passing easy), so anchor at easy.

Phase 4: Production validation and feedback closure

eval-methodology-irl-feedback-loop — Audit real sessions in onboarded projects on a recurring schedule (1–2 weeks post-rollout, then per-cycle). Judge each session: "should-have-triggered?" Log miss patterns. Promote recurring misses into new eval scenarios. Run follow-up audits after wording changes to confirm the fix. This is how silent failures (undetected decision misses) surface and feed back into evals.

Key insight: Post-rollout observation is only meaningful if deliberate and instrumented. Without auditing, silent misses go undetected for months.

The Notes at a Glance

Note	Type	What it covers	Read if...
running-autoresearch-skill-evals	howto	Skill-wording eval loops: valid run modes, cache refresh, reduced grids, parallelization, rep counts	You're about to design or run an eval loop
os-adr-eval-b-grid-results-and-observations	eval-results	os-adr Eval B baseline (haiku 0/8, sonnet 5/8, 1 rep/cell), confirmation run, observations	You're designing a follow-up eval and need the baseline context
os-adr-eval-b-wording-experiment-hypotheses	eval-results	os-adr Eval B wording tuning (5 iterations, sonnet 8/8, haiku 7/8), hypothesis tracking, deployment gate	You want to see how a trained-up eval goes and what the next gate looks like
eval-methodology-ladder	reference / pattern-framework	Evaluation ladder design: clear→ambiguous→edge-case progressions, paired scenarios, per-level pass bars, run/reserve splits	You're designing a hardened eval and want to avoid common pitfalls
eval-methodology-irl-feedback-loop	reference / pattern-framework	Production auditing: session sampling, miss pattern logging, scenario authoring from audit findings, close-the-loop pattern	You want to set up deliberate post-rollout observation and close the feedback loop

Cross-project application

The ladder approach and audit pattern are generalizable beyond os-adr:

Ladder approach: Any skill/feature with unprompted-behavior evaluation (should the model notice it's relevant without being told?) can use Level 1 baseline → wording tuning → Level 2 ambiguity discrimination → Level 3 over-trigger safeguards.
Audit pattern: Any feature rolled out to multiple projects benefits from recurring session audits. The pattern is universal; only the audit criterion changes (e.g., "should consult X", "should format as Y", "should refuse Z").

cc-os-plugin-skill-naming-convention — naming and registration mechanics (used in wording placement decisions)
running-autoresearch-skill-evals (howto) — also has a "Related" section with detailed eval-specific references

5.9 KiB Raw Blame History Unescape Escape