100 lines
10 KiB
Markdown
100 lines
10 KiB
Markdown
---
|
||
type: reference
|
||
subtype: pattern/framework
|
||
title: Evaluation ladder approach for LLM behavior measurement
|
||
summary: A methodology for progressively hardening evaluations from clear-cue baselines to ambiguity-discriminating tests, with per-level pass bars and held-out reserve discipline to maintain measurement validity.
|
||
tags:
|
||
- type/reference
|
||
- domain/llm-evaluation
|
||
- convention/eval-design
|
||
- tool/autoresearch
|
||
scope: global
|
||
last_updated: 2026-07-06
|
||
date: 2026-07-06
|
||
related:
|
||
- running-autoresearch-skill-evals
|
||
- os-adr-eval-b-grid-results-and-observations
|
||
- os-adr-eval-b-wording-experiment-hypotheses
|
||
- eval-methodology-irl-feedback-loop
|
||
source: cc-os
|
||
---
|
||
|
||
# Evaluation ladder approach
|
||
|
||
Establish confidence in LLM behavior through a ladder of increasing difficulty and realism, not through a single high-bar eval. Each rung tests a distinct capability or confound; pass bars are per-level, not aggregate. The ladder is a strategy for designing evals that remain valid measurement instruments as tuning occurs.
|
||
|
||
## Purpose
|
||
|
||
Evaluations drift from measurement toward training artifacts the moment you start optimizing against them (wording tuning, architecture changes, prompt engineering). The ladder approach maintains validity by:
|
||
|
||
1. Anchoring at the easy end (clear-cue boundary cases) because difficulty is non-monotonic — passing hard does not imply passing easy, and a single failure at an easy level can invalidate a claimed strength.
|
||
2. Building pairs of scenarios at every level: positive (correct action under intended conditions) and negative (equally-subtle non-action or over-action under absence/false-premise conditions). The trivial way to ace a boundary-case eval is to always trigger; paired scenarios catch this.
|
||
3. Freezing evaluation surfaces (checker, fixtures, scenarios) so wording can move independently.
|
||
4. Declaring per-level pass bars so you can retire a rung once it's clear (e.g., "level 1 clear, stop optimizing for clear-cue; level 2 open").
|
||
5. Splitting scenarios into run-set and held-out reserve upfront (~2× authoring), so if wording is ever tuned against the run-set, the reserve becomes a fresh measurement set for the next evaluation cycle.
|
||
|
||
The ladder is not a single evaluation — it's a **progression strategy** for successive evals that test different aspects of the same behavior.
|
||
|
||
## Core Principles
|
||
|
||
**1. Ladder levels represent confound progression, not scoring tiers.** Level 1 tests a behavior with maximum clarity (clear cues, no ambiguity). Level 2 adds one confound (ambiguous cues, but the positive/negative cases are still fully distinct). Level 3 adds another (edge cases where the action *could* apply but shouldn't, requiring boundary judgment). The "success" of level 1 does not imply level 2 success; each level has its own pass bar.
|
||
|
||
**2. Paired scenarios at every level catch both under-triggering and over-triggering.** A positive scenario tests "does the model take the correct action when conditions are met?" A negative scenario tests "does the model correctly *not* take the action when conditions are absent or when a plausible false-premise tempts over-triggering?" A model that always triggers will ace positive scenarios and fail negatives; a model that under-triggers will fail positives. Both signal real problems that single-axis evals miss.
|
||
|
||
**3. Difficulty is non-monotonic — anchoring easy catches breaks that hard scenarios miss.** A model might pass a single hard boundary-case scenario but fail the basic case. Starting at easy ensures the foundation is solid before claiming strength at harder levels. Historically, many eval designs began at medium difficulty and missed that the model couldn't handle the basics.
|
||
|
||
**4. Run-set vs held-out reserve discipline prevents training-set creep.** Scenarios are authored at roughly 2× the quantity needed for the run-set; the reserve is held out and never used for tuning feedback. If a wording optimization loop targets the run-set, the reserve becomes the fresh measurement set for the next cycle. This discipline costs upfront scenario authoring but buys back measurement validity after tuning.
|
||
|
||
**5. Per-level pass bars, not aggregate scores.** Each level has a passing criterion (e.g., "level 1: all run-set scenarios, 3+ reps per scenario, majority rep threshold"). Once level 1 is clear (reliably PASS over 2–3 eval runs), investment moves to level 2. An aggregate score like "75% pass" conflates clear and ambiguous scenarios, masking which capability gaps remain.
|
||
|
||
## Decision Framework
|
||
|
||
| If you're asking... | Use this level structure |
|
||
|---|---|
|
||
| "Does the model understand what it's supposed to do?" | Level 1 (clear-cue): instructed actions with maximum salience, no ambiguity. Positive = correct action. Negative = do nothing or non-action. Does not test triggering discrimination. |
|
||
| "Can the model distinguish between conditions-met and conditions-absent?" | Level 2 (ambiguous-cue): scenarios where the positive case *looks* similar to the negative case, but one has a critical distinguishing feature. Positive = correct action. Negative = similar setup but the trigger condition is actually absent or a false-premise tempts action. Tests whether the model mistakes close-calls. |
|
||
| "Can the model handle edge cases and over-trigger risks?" | Level 3 (edge-case discrimination): scenarios where action *could* apply based on a loose reading, but shouldn't based on the actual requirement. Positive = narrow legitimate case. Negative = over-trigger case (model takes the action when it shouldn't). Tests boundary judgment. |
|
||
| "Does the behavior generalize beyond the fixture?" | Separate fixture in a different language/domain. Repeats level 1–2 on new fixture to test fixture-generalization. Same ladder structure, different codebase. |
|
||
|
||
**Start your eval at the level that matches your current confidence.** Prompt-only (instructed) behavior starts at Level 1 (you're not claiming triggering discrimination yet). Unprompted self-triggering starts at Level 1 if it's novel (does the model notice at all?), then climbs to Level 2 (does it notice reliably?) and Level 3 (does it avoid over-triggering?).
|
||
|
||
## Patterns
|
||
|
||
### Pattern: Run-set + reserve split
|
||
|
||
Author scenarios at 2× the volume needed for a single run-set. Assign ~50% to the run-set (tuning-eligible), ~50% to reserve (held-out). If running a wording loop or architecture tuning, only run the run-set; reserve stays frozen. Once tuning is complete, run the reserve as a fresh measurement. If any surprises emerge on reserve (e.g., reserve-only over-triggering not visible in run-set), those become new Level 3 scenarios for the next ladder iteration.
|
||
|
||
### Pattern: Per-scenario rep count for confidence
|
||
|
||
Run each scenario multiple times (reps) at a given level. Reps catch variance and model non-determinism. E.g., Level 1: 3 reps per scenario. Level 2: 2–3 reps. Level 3: 1–2 reps (harder scenarios are often noisier, but you're willing to accept less confidence at the edge). Track which scenarios have high variance and mark them as canaries for follow-up evals.
|
||
|
||
### Pattern: Tier-specific laddering
|
||
|
||
Lower model tiers (e.g., haiku vs sonnet) may not clear the same rung at the same pass bar. E.g., "sonnet clears level 1 at 3-rep majority (all scenarios PASS), so move to level 2; haiku still at 2/3 on some scenarios, so continue level 1 with more reps." The ladder is flexible per-tier.
|
||
|
||
### Pattern: Stop criterion
|
||
|
||
**"Good enough for now → test IRL"** is the gate between ladder evals and production validation. When the current level is reliably PASS and you're confident in the behavior, stop optimizing for that level and prepare for production rollout. That gate is only meaningful if **deliberate, instrumented observation** of real sessions (see [[eval-methodology-irl-feedback-loop]]) will surface silent misses and feed them back as new scenarios.
|
||
|
||
## Anti-Patterns
|
||
|
||
- **Single high-bar scenario as the eval.** A model can pass one hard scenario while failing basics. Use levels, not singular difficulty.
|
||
- **Aggregate scoring that hides per-level variance.** Avoid "model X passes 70% of all scenarios." Instead: "level 1: 100%, level 2: 60%, level 3: 30%." The breakdown is where the insight is.
|
||
- **Scenarios that are too ambiguous to score cleanly.** If a negative scenario *looks* correct and you have to debate whether the model should have triggered, the scenario isn't at the right level yet. Move it up to a higher level or rewrite it.
|
||
- **Forgetting that paired scenarios are *equally* subtle.** If your negative scenarios are obviously non-triggers (e.g., "add logging, no decision involved"), you're not testing discrimination. Negative scenarios should be as close to the positive case as possible without crossing the trigger threshold.
|
||
- **Tuning against held-out reserve.** Once the reserve is defined, it becomes Schrodinger's eval — useful as a fresh measurement only if it stays frozen. If you peek at reserve results and change wording based on them, it's now training-set.
|
||
|
||
## Known Limitations
|
||
|
||
The ladder approach assumes you can enumerate confounds (ambiguity, edge cases) without hitting a combinatorial explosion. For highly context-dependent behaviors (e.g., "write helpful error messages" — helpful in what context?), the confound space is large and the ladder can be hard to design. In these cases, production auditing (see [[eval-methodology-irl-feedback-loop]]) becomes more critical earlier.
|
||
|
||
The approach also assumes that scenarios can be authored at 2× desired run-set size without exhausting designer time. This is true for discrete, bounded tasks (e.g., "consult ADRs") but may be expensive for open-ended behaviors.
|
||
|
||
## Related
|
||
|
||
- [[eval-methodology-hub]] — navigation hub for all evaluation-methodology notes
|
||
- [[running-autoresearch-skill-evals]] — how to run wording loops safely while keeping evals valid
|
||
- [[os-adr-eval-b-grid-results-and-observations]] — Level 1 baseline for os-adr unprompted triggering
|
||
- [[os-adr-eval-b-wording-experiment-hypotheses]] — Level 1 tuning run against a wording loop
|
||
- [[eval-methodology-irl-feedback-loop]] — production validation and how to close the loop: misses become new scenarios
|