Evaluation ladder approach

Establish confidence in LLM behavior through a ladder of increasing difficulty and realism, not through a single high-bar eval. Each rung tests a distinct capability or confound; pass bars are per-level, not aggregate. The ladder is a strategy for designing evals that remain valid measurement instruments as tuning occurs.

Purpose

Evaluations drift from measurement toward training artifacts the moment you start optimizing against them (wording tuning, architecture changes, prompt engineering). The ladder approach maintains validity by:

Anchoring at the easy end (clear-cue boundary cases) because difficulty is non-monotonic — passing hard does not imply passing easy, and a single failure at an easy level can invalidate a claimed strength.
Building pairs of scenarios at every level: positive (correct action under intended conditions) and negative (equally-subtle non-action or over-action under absence/false-premise conditions). The trivial way to ace a boundary-case eval is to always trigger; paired scenarios catch this.
Freezing evaluation surfaces (checker, fixtures, scenarios) so wording can move independently.
Declaring per-level pass bars so you can retire a rung once it's clear (e.g., "level 1 clear, stop optimizing for clear-cue; level 2 open").
Splitting scenarios into run-set and held-out reserve upfront (~2× authoring), so if wording is ever tuned against the run-set, the reserve becomes a fresh measurement set for the next evaluation cycle.

The ladder is not a single evaluation — it's a progression strategy for successive evals that test different aspects of the same behavior.

Core Principles

1. Ladder levels represent confound progression, not scoring tiers. Level 1 tests a behavior with maximum clarity (clear cues, no ambiguity). Level 2 adds one confound (ambiguous cues, but the positive/negative cases are still fully distinct). Level 3 adds another (edge cases where the action could apply but shouldn't, requiring boundary judgment). The "success" of level 1 does not imply level 2 success; each level has its own pass bar.

2. Paired scenarios at every level catch both under-triggering and over-triggering. A positive scenario tests "does the model take the correct action when conditions are met?" A negative scenario tests "does the model correctly not take the action when conditions are absent or when a plausible false-premise tempts over-triggering?" A model that always triggers will ace positive scenarios and fail negatives; a model that under-triggers will fail positives. Both signal real problems that single-axis evals miss.

3. Difficulty is non-monotonic — anchoring easy catches breaks that hard scenarios miss. A model might pass a single hard boundary-case scenario but fail the basic case. Starting at easy ensures the foundation is solid before claiming strength at harder levels. Historically, many eval designs began at medium difficulty and missed that the model couldn't handle the basics.

4. Run-set vs held-out reserve discipline prevents training-set creep. Scenarios are authored at roughly 2× the quantity needed for the run-set; the reserve is held out and never used for tuning feedback. If a wording optimization loop targets the run-set, the reserve becomes the fresh measurement set for the next cycle. This discipline costs upfront scenario authoring but buys back measurement validity after tuning.

5. Per-level pass bars, not aggregate scores. Each level has a passing criterion (e.g., "level 1: all run-set scenarios, 3+ reps per scenario, majority rep threshold"). Once level 1 is clear (reliably PASS over 2–3 eval runs), investment moves to level 2. An aggregate score like "75% pass" conflates clear and ambiguous scenarios, masking which capability gaps remain.

Decision Framework

If you're asking...	Use this level structure
"Does the model understand what it's supposed to do?"	Level 1 (clear-cue): instructed actions with maximum salience, no ambiguity. Positive = correct action. Negative = do nothing or non-action. Does not test triggering discrimination.
"Can the model distinguish between conditions-met and conditions-absent?"	Level 2 (ambiguous-cue): scenarios where the positive case looks similar to the negative case, but one has a critical distinguishing feature. Positive = correct action. Negative = similar setup but the trigger condition is actually absent or a false-premise tempts action. Tests whether the model mistakes close-calls.
"Can the model handle edge cases and over-trigger risks?"	Level 3 (edge-case discrimination): scenarios where action could apply based on a loose reading, but shouldn't based on the actual requirement. Positive = narrow legitimate case. Negative = over-trigger case (model takes the action when it shouldn't). Tests boundary judgment.
"Does the behavior generalize beyond the fixture?"	Separate fixture in a different language/domain. Repeats level 1–2 on new fixture to test fixture-generalization. Same ladder structure, different codebase.

Start your eval at the level that matches your current confidence. Prompt-only (instructed) behavior starts at Level 1 (you're not claiming triggering discrimination yet). Unprompted self-triggering starts at Level 1 if it's novel (does the model notice at all?), then climbs to Level 2 (does it notice reliably?) and Level 3 (does it avoid over-triggering?).

Patterns

Pattern: Run-set + reserve split

Author scenarios at 2× the volume needed for a single run-set. Assign ~50% to the run-set (tuning-eligible), ~50% to reserve (held-out). If running a wording loop or architecture tuning, only run the run-set; reserve stays frozen. Once tuning is complete, run the reserve as a fresh measurement. If any surprises emerge on reserve (e.g., reserve-only over-triggering not visible in run-set), those become new Level 3 scenarios for the next ladder iteration.

Pattern: Per-scenario rep count for confidence

Run each scenario multiple times (reps) at a given level. Reps catch variance and model non-determinism. E.g., Level 1: 3 reps per scenario. Level 2: 2–3 reps. Level 3: 1–2 reps (harder scenarios are often noisier, but you're willing to accept less confidence at the edge). Track which scenarios have high variance and mark them as canaries for follow-up evals.

Pattern: Tier-specific laddering

Lower model tiers (e.g., haiku vs sonnet) may not clear the same rung at the same pass bar. E.g., "sonnet clears level 1 at 3-rep majority (all scenarios PASS), so move to level 2; haiku still at 2/3 on some scenarios, so continue level 1 with more reps." The ladder is flexible per-tier.

Pattern: Stop criterion

"Good enough for now → test IRL" is the gate between ladder evals and production validation. When the current level is reliably PASS and you're confident in the behavior, stop optimizing for that level and prepare for production rollout. That gate is only meaningful if deliberate, instrumented observation of real sessions (see eval-methodology-irl-feedback-loop) will surface silent misses and feed them back as new scenarios.

Anti-Patterns

Single high-bar scenario as the eval. A model can pass one hard scenario while failing basics. Use levels, not singular difficulty.
Aggregate scoring that hides per-level variance. Avoid "model X passes 70% of all scenarios." Instead: "level 1: 100%, level 2: 60%, level 3: 30%." The breakdown is where the insight is.
Scenarios that are too ambiguous to score cleanly. If a negative scenario looks correct and you have to debate whether the model should have triggered, the scenario isn't at the right level yet. Move it up to a higher level or rewrite it.
Forgetting that paired scenarios are equally subtle. If your negative scenarios are obviously non-triggers (e.g., "add logging, no decision involved"), you're not testing discrimination. Negative scenarios should be as close to the positive case as possible without crossing the trigger threshold.
Tuning against held-out reserve. Once the reserve is defined, it becomes Schrodinger's eval — useful as a fresh measurement only if it stays frozen. If you peek at reserve results and change wording based on them, it's now training-set.
A checker that contradicts the shipped instructions. Before spending reps, dry-run the instrument on paper: "would a model that follows the shipped wording perfectly pass every cell?" and "would an always-trigger model pass the positives while failing the negatives?" If yes to the second or no to the first, the eval measures instruction-disobedience. Concrete instance (os-adr Eval C, 2026-07-06): positives sat in already-decided ADR territory, where the shipped wording says find→cite→comply — but the checker demanded ADR creation, so a perfectly-behaving model FAILed and an ADR-spamming model would have aced positives. Caught on the first live rep only because of a canary-cell run.
Trusting model-free self-tests to validate checker semantics. Fabricated "perfect" transcripts encode the same assumptions as the checker — if the designer's idea of correct behavior is wrong, both agree and the self-test stays green. Self-tests catch mechanical bugs, not conformance bugs. Fabricate at least one transcript of shipped-instruction-compliant behavior (what the model is actually told to do, given the fixture state), not just the designer's imagined ideal.

Known Limitations

The ladder approach assumes you can enumerate confounds (ambiguity, edge cases) without hitting a combinatorial explosion. For highly context-dependent behaviors (e.g., "write helpful error messages" — helpful in what context?), the confound space is large and the ladder can be hard to design. In these cases, production auditing (see eval-methodology-irl-feedback-loop) becomes more critical earlier.

The approach also assumes that scenarios can be authored at 2× desired run-set size without exhausting designer time. This is true for discrete, bounded tasks (e.g., "consult ADRs") but may be expensive for open-ended behaviors.

eval-methodology-hub — navigation hub for all evaluation-methodology notes
running-autoresearch-skill-evals — how to run wording loops safely while keeping evals valid
os-adr-eval-b-grid-results-and-observations — Level 1 baseline for os-adr unprompted triggering
os-adr-eval-b-wording-experiment-hypotheses — Level 1 tuning run against a wording loop
eval-methodology-irl-feedback-loop — production validation and how to close the loop: misses become new scenarios

11 KiB Raw Blame History Unescape Escape