---
type: reference
subtype: pattern/framework
title: Evaluation ladder approach for LLM behavior measurement
summary: A methodology for progressively hardening evaluations from clear-cue baselines to ambiguity-discriminating tests, with per-level pass bars and held-out reserve discipline to maintain measurement validity.
tags:
  - type/reference
  - domain/llm-evaluation
  - convention/eval-design
  - tool/autoresearch
scope: global
last_updated: 2026-07-08  # econ-guard anti-pattern (WS4)
date: 2026-07-06
related:
  - running-autoresearch-skill-evals
  - os-adr-eval-b-grid-results-and-observations
  - os-adr-eval-b-wording-experiment-hypotheses
  - eval-methodology-irl-feedback-loop
source: cc-os
---

# Evaluation ladder approach

Establish confidence in LLM behavior through a ladder of increasing difficulty and realism, not through a single high-bar eval. Each rung tests a distinct capability or confound; pass bars are per-level, not aggregate. The ladder is a strategy for designing evals that remain valid measurement instruments as tuning occurs.

## Purpose

Evaluations drift from measurement toward training artifacts the moment you start optimizing against them (wording tuning, architecture changes, prompt engineering). The ladder approach maintains validity by:

1. Anchoring at the easy end (clear-cue boundary cases) because difficulty is non-monotonic — passing hard does not imply passing easy, and a single failure at an easy level can invalidate a claimed strength.
2. Building pairs of scenarios at every level: positive (correct action under intended conditions) and negative (equally-subtle non-action or over-action under absence/false-premise conditions). The trivial way to ace a boundary-case eval is to always trigger; paired scenarios catch this.
3. Freezing evaluation surfaces (checker, fixtures, scenarios) so wording can move independently.
4. Declaring per-level pass bars so you can retire a rung once it's clear (e.g., "level 1 clear, stop optimizing for clear-cue; level 2 open").
5. Splitting scenarios into run-set and held-out reserve upfront (~2× authoring), so if wording is ever tuned against the run-set, the reserve becomes a fresh measurement set for the next evaluation cycle.

The ladder is not a single evaluation — it's a **progression strategy** for successive evals that test different aspects of the same behavior.

## Core Principles

**1. Ladder levels represent confound progression, not scoring tiers.** Level 1 tests a behavior with maximum clarity (clear cues, no ambiguity). Level 2 adds one confound (ambiguous cues, but the positive/negative cases are still fully distinct). Level 3 adds another (edge cases where the action *could* apply but shouldn't, requiring boundary judgment). The "success" of level 1 does not imply level 2 success; each level has its own pass bar.

**2. Paired scenarios at every level catch both under-triggering and over-triggering.** A positive scenario tests "does the model take the correct action when conditions are met?" A negative scenario tests "does the model correctly *not* take the action when conditions are absent or when a plausible false-premise tempts over-triggering?" A model that always triggers will ace positive scenarios and fail negatives; a model that under-triggers will fail positives. Both signal real problems that single-axis evals miss.

**3. Difficulty is non-monotonic — anchoring easy catches breaks that hard scenarios miss.** A model might pass a single hard boundary-case scenario but fail the basic case. Starting at easy ensures the foundation is solid before claiming strength at harder levels. Historically, many eval designs began at medium difficulty and missed that the model couldn't handle the basics.

**4. Run-set vs held-out reserve discipline prevents training-set creep.** Scenarios are authored at roughly 2× the quantity needed for the run-set; the reserve is held out and never used for tuning feedback. If a wording optimization loop targets the run-set, the reserve becomes the fresh measurement set for the next cycle. This discipline costs upfront scenario authoring but buys back measurement validity after tuning.

**5. Per-level pass bars, not aggregate scores.** Each level has a passing criterion (e.g., "level 1: all run-set scenarios, 3+ reps per scenario, majority rep threshold"). Once level 1 is clear (reliably PASS over 2–3 eval runs), investment moves to level 2. An aggregate score like "75% pass" conflates clear and ambiguous scenarios, masking which capability gaps remain.

## Decision Framework

| If you're asking... | Use this level structure |
|---|---|
| "Does the model understand what it's supposed to do?" | Level 1 (clear-cue): instructed actions with maximum salience, no ambiguity. Positive = correct action. Negative = do nothing or non-action. Does not test triggering discrimination. |
| "Can the model distinguish between conditions-met and conditions-absent?" | Level 2 (ambiguous-cue): scenarios where the positive case *looks* similar to the negative case, but one has a critical distinguishing feature. Positive = correct action. Negative = similar setup but the trigger condition is actually absent or a false-premise tempts action. Tests whether the model mistakes close-calls. |
| "Can the model handle edge cases and over-trigger risks?" | Level 3 (edge-case discrimination): scenarios where action *could* apply based on a loose reading, but shouldn't based on the actual requirement. Positive = narrow legitimate case. Negative = over-trigger case (model takes the action when it shouldn't). Tests boundary judgment. |
| "Does the behavior generalize beyond the fixture?" | Separate fixture in a different language/domain. Repeats level 1–2 on new fixture to test fixture-generalization. Same ladder structure, different codebase. |

**Start your eval at the level that matches your current confidence.** Prompt-only (instructed) behavior starts at Level 1 (you're not claiming triggering discrimination yet). Unprompted self-triggering starts at Level 1 if it's novel (does the model notice at all?), then climbs to Level 2 (does it notice reliably?) and Level 3 (does it avoid over-triggering?).

## Patterns

### Pattern: Run-set + reserve split

Author scenarios at 2× the volume needed for a single run-set. Assign ~50% to the run-set (tuning-eligible), ~50% to reserve (held-out). If running a wording loop or architecture tuning, only run the run-set; reserve stays frozen. Once tuning is complete, run the reserve as a fresh measurement. If any surprises emerge on reserve (e.g., reserve-only over-triggering not visible in run-set), those become new Level 3 scenarios for the next ladder iteration.

### Pattern: Per-scenario rep count for confidence

Run each scenario multiple times (reps) at a given level. Reps catch variance and model non-determinism. E.g., Level 1: 3 reps per scenario. Level 2: 2–3 reps. Level 3: 1–2 reps (harder scenarios are often noisier, but you're willing to accept less confidence at the edge). Track which scenarios have high variance and mark them as canaries for follow-up evals.

### Pattern: Tier-specific laddering

Lower model tiers (e.g., haiku vs sonnet) may not clear the same rung at the same pass bar. E.g., "sonnet clears level 1 at 3-rep majority (all scenarios PASS), so move to level 2; haiku still at 2/3 on some scenarios, so continue level 1 with more reps." The ladder is flexible per-tier.

### Pattern: Lesson promotion from eval-results notes

Eval-results notes are episodic records of one measurement; methodology notes (this one and
its siblings) are where compounding knowledge accretes. The `eval-results` template ends with
a mandatory promotion check: any transferable lesson that recurs (or names a new confound)
gets written into the relevant methodology note at the moment the eval-results note is
finished — not left stranded per-eval. The eval-results note keeps a one-line bullet pointing
at the methodology note. Eval-results notes also carry a **Proposed Changes (Loop Inputs)**
section holding the *concrete candidate wording* a tuning loop will test — the handoff
artifact that lets a later reader diff proposal against what shipped.

### Pattern: Stop criterion

**"Good enough for now → test IRL"** is the gate between ladder evals and production validation. When the current level is reliably PASS and you're confident in the behavior, stop optimizing for that level and prepare for production rollout. That gate is only meaningful if **deliberate, instrumented observation** of real sessions (see [[eval-methodology-irl-feedback-loop]]) will surface silent misses and feed them back as new scenarios.

## Anti-Patterns

- **Single high-bar scenario as the eval.** A model can pass one hard scenario while failing basics. Use levels, not singular difficulty.
- **Aggregate scoring that hides per-level variance.** Avoid "model X passes 70% of all scenarios." Instead: "level 1: 100%, level 2: 60%, level 3: 30%." The breakdown is where the insight is.
- **Scenarios that are too ambiguous to score cleanly.** If a negative scenario *looks* correct and you have to debate whether the model should have triggered, the scenario isn't at the right level yet. Move it up to a higher level or rewrite it.
- **Forgetting that paired scenarios are *equally* subtle.** If your negative scenarios are obviously non-triggers (e.g., "add logging, no decision involved"), you're not testing discrimination. Negative scenarios should be as close to the positive case as possible without crossing the trigger threshold.
- **Tuning against held-out reserve.** Once the reserve is defined, it becomes Schrodinger's eval — useful as a fresh measurement only if it stays frozen. If you peek at reserve results and change wording based on them, it's now training-set.
- **A checker that contradicts the shipped instructions.** Before spending reps, dry-run the instrument on paper: "would a model that follows the shipped wording perfectly pass every cell?" and "would an always-trigger model pass the positives while failing the negatives?" If yes to the second or no to the first, the eval measures instruction-*disobedience*. Concrete instance (os-adr Eval C, 2026-07-06): positives sat in already-decided ADR territory, where the shipped wording says find→cite→comply — but the checker demanded ADR creation, so a perfectly-behaving model FAILed and an ADR-spamming model would have aced positives. Caught on the first live rep only because of a canary-cell run.
- **Econ guards phrased as ratios of a grind baseline.** When a positive scenario's baseline behavior is an *efficient but wrong-shaped* strategy (e.g. a cheap main-loop grind), a "candidate must stay ≤N% of baseline cost" guard punishes the correct behavior the checker itself demands — delegation legitimately costs more than grinding on a cheap tier. Set cheap-tier econ guards in absolute tokens, or condition them on whether the task is scriptable; reduction targets belong only on the expensive-tier cells whose baseline IS the miss (os-orchestration WS4, 2026-07-08: sonnet guard breached at 168% for identical task completion, accepted with rationale).
- **Trusting model-free self-tests to validate checker *semantics*.** Fabricated "perfect" transcripts encode the same assumptions as the checker — if the designer's idea of correct behavior is wrong, both agree and the self-test stays green. Self-tests catch mechanical bugs, not conformance bugs. Fabricate at least one transcript of *shipped-instruction-compliant* behavior (what the model is actually told to do, given the fixture state), not just the designer's imagined ideal.

## Known Limitations

The ladder approach assumes you can enumerate confounds (ambiguity, edge cases) without hitting a combinatorial explosion. For highly context-dependent behaviors (e.g., "write helpful error messages" — helpful in what context?), the confound space is large and the ladder can be hard to design. In these cases, production auditing (see [[eval-methodology-irl-feedback-loop]]) becomes more critical earlier.

The approach also assumes that scenarios can be authored at 2× desired run-set size without exhausting designer time. This is true for discrete, bounded tasks (e.g., "consult ADRs") but may be expensive for open-ended behaviors.

## Related

- [[eval-methodology-hub]] — navigation hub for all evaluation-methodology notes
- [[running-autoresearch-skill-evals]] — how to run wording loops safely while keeping evals valid
- [[os-adr-eval-b-grid-results-and-observations]] — Level 1 baseline for os-adr unprompted triggering
- [[os-adr-eval-b-wording-experiment-hypotheses]] — Level 1 tuning run against a wording loop
- [[eval-methodology-irl-feedback-loop]] — production validation and how to close the loop: misses become new scenarios
