SecondBrain/eval-methodology-hub.md

---
type: hub
title: Evaluation methodology for LLM skill behavior (cc-os)
summary: Navigation hub for evaluation design, harness setup, wording optimization, and production validation patterns — the full progression from isolated eval design to real-world rollout and feedback.
tags:
  - type/hub
  - domain/llm-evaluation
  - tool/autoresearch
  - tool/os-adr
  - convention/eval-design
  - project/cc-os
scope: global
last_updated: 2026-07-06
date: 2026-07-06
source: cc-os
---

# Evaluation methodology for LLM skill behavior

Navigation hub for the full evaluation pipeline: from harness design through wording tuning to production rollout and feedback loops. Each note covers a distinct phase or pattern; read the one that matches your current task.

## Quick Navigation

**I want to...**

| I want to... | Read this |
|---|---|
| Understand how to design eval harnesses and iterate wording safely | [[running-autoresearch-skill-evals]] |
| See the os-adr Eval B baseline (unprompted triggering test) | [[os-adr-eval-b-grid-results-and-observations]] |
| See the os-adr Eval B wording experiment results (5-iteration improvement) | [[os-adr-eval-b-wording-experiment-hypotheses]] |
| Learn the ladder approach for progressively harder evals | [[eval-methodology-ladder]] |
| Set up production auditing and close the feedback loop | [[eval-methodology-irl-feedback-loop]] |

## The Full Progression

### Phase 1: Initial eval design and baseline measurement

**[[running-autoresearch-skill-evals]]** — Procedure for running evaluations and wording loops. When to use in-session vs headless runners, how to refresh plugin caches so wording edits take effect, how to parallelize runs and interpret results at the axis level.

**[[os-adr-eval-b-grid-results-and-observations]]** — Concrete baseline eval for os-adr unprompted-triggering behavior (haiku 0/8, sonnet 5/8). Shows what a held-out measurement looks like and why the results point to a prompting/wording issue rather than a capability gap.

### Phase 2: Controlled wording optimization

**[[os-adr-eval-b-wording-experiment-hypotheses]]** — Five-iteration wording loop that improved the baseline (sonnet 8/8, haiku 7/8). Demonstrates the hypothesis→verdict tracking pattern, per-iteration results, and how to close gaps via targeted wording in specific channels (hook note, CLAUDE.md, skill description).

**Key result:** Trigger-conditioned phrasing ("**when** you encounter X, do Y") outperforms inventory statements; each rule must live where its precondition is visible (step-2 wording in skill bodies, not hook notes).

### Phase 3: Ladder progression and generalization testing

**[[eval-methodology-ladder]]** — Design strategy for successive evals at increasing difficulty: Level 1 (clear-cue baseline), Level 2 (ambiguous-cue discrimination), Level 3 (edge-case over-trigger risks). Pairs positive/negative scenarios at every level. Freeze evaluation surfaces (checker, fixture, scenarios) so wording can move independently. Run-set vs held-out reserve discipline to maintain measurement validity after tuning.

**Key pattern:** Per-level pass bars, not aggregate scores. Once a level is clear, move to the next. Non-monotonic difficulty (passing hard does not imply passing easy), so anchor at easy.

### Phase 4: Production validation and feedback closure

**[[eval-methodology-irl-feedback-loop]]** — Audit real sessions in onboarded projects on a recurring schedule (1–2 weeks post-rollout, then per-cycle). Judge each session: "should-have-triggered?" Log miss patterns. Promote recurring misses into new eval scenarios. Run follow-up audits after wording changes to confirm the fix. This is how silent failures (undetected decision misses) surface and feed back into evals.

**Key insight:** Post-rollout observation is only meaningful if deliberate and instrumented. Without auditing, silent misses go undetected for months.

## The Notes at a Glance

| Note | Type | What it covers | Read if... |
|---|---|---|---|
| [[running-autoresearch-skill-evals]] | howto | Skill-wording eval loops: valid run modes, cache refresh, reduced grids, parallelization, rep counts | You're about to design or run an eval loop |
| [[os-adr-eval-b-grid-results-and-observations]] | eval-results | os-adr Eval B baseline (haiku 0/8, sonnet 5/8, 1 rep/cell), confirmation run, observations | You're designing a follow-up eval and need the baseline context |
| [[os-adr-eval-b-wording-experiment-hypotheses]] | eval-results | os-adr Eval B wording tuning (5 iterations, sonnet 8/8, haiku 7/8), hypothesis tracking, deployment gate | You want to see how a trained-up eval goes and what the next gate looks like |
| [[eval-methodology-ladder]] | reference / pattern-framework | Evaluation ladder design: clear→ambiguous→edge-case progressions, paired scenarios, per-level pass bars, run/reserve splits | You're designing a hardened eval and want to avoid common pitfalls |
| [[eval-methodology-irl-feedback-loop]] | reference / pattern-framework | Production auditing: session sampling, miss pattern logging, scenario authoring from audit findings, close-the-loop pattern | You want to set up deliberate post-rollout observation and close the feedback loop |

## Cross-project application

The ladder approach and audit pattern are generalizable beyond os-adr:

- **Ladder approach:** Any skill/feature with unprompted-behavior evaluation (should the model notice it's relevant without being told?) can use Level 1 baseline → wording tuning → Level 2 ambiguity discrimination → Level 3 over-trigger safeguards.
- **Audit pattern:** Any feature rolled out to multiple projects benefits from recurring session audits. The pattern is universal; only the audit criterion changes (e.g., "should consult X", "should format as Y", "should refuse Z").

## Related

- [[cc-os-plugin-skill-naming-convention]] — naming and registration mechanics (used in wording placement decisions)
- [[running-autoresearch-skill-evals]] (howto) — also has a "Related" section with detailed eval-specific references