64 lines
5.0 KiB
Markdown
64 lines
5.0 KiB
Markdown
|
---
|
|||
|
|
type: eval-results
|
|||
|
|
title: [Evaluation name, e.g. "os-adr Eval B (unprompted write-trigger & retrieval)"]
|
|||
|
|
summary: [What was measured and the key finding — 1-2 sentences, e.g. "Haiku never self-triggers the ADR system without explicit prompting; sonnet passes 5/8 with two near-miss failures."]
|
|||
|
|
tags:
|
|||
|
|
- type/eval-results
|
|||
|
|
- domain/llm-evaluation
|
|||
|
|
- tool/[tool-under-test] # the skill/plugin/feature being measured
|
|||
|
|
- project/[project] # project that contains the eval harness
|
|||
|
|
scope: global
|
|||
|
|
last_updated: YYYY-MM-DD
|
|||
|
|
date: YYYY-MM-DD # creation date — set once, never updated
|
|||
|
|
related:
|
|||
|
|
- [note-slug] # cross-links to follow-up evals, methodology notes, or design docs
|
|||
|
|
source: [project name] # project that contains the eval harness
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# [Evaluation Title]
|
|||
|
|
|
|||
|
|
## Results Grid/Threshold
|
|||
|
|
|
|||
|
|
<!-- The raw measurement: grid table with pass/fail per cell, OR hypothesis→verdict mapping, or threshold scores. Include rep counts per cell and evaluation date. Include a brief summary line above the table/mapping explaining what passing looks like. -->
|
|||
|
|
|
|||
|
|
**Passing criterion:** [Define what PASS means for this eval — e.g., all scenarios across both tiers, or per-tier threshold, or axis-level aggregate.]
|
|||
|
|
|
|||
|
|
| [Column] | [Column] |
|
|||
|
|
|---|---|
|
|||
|
|
| [Cell] | [Cell] |
|
|||
|
|
|
|||
|
|
**Summary:** [1–2 sentences recapping which tiers/cells passed/failed and the baseline numbers. This is what the reader scans first to know whether results are in the expected ballpark.]
|
|||
|
|
|
|||
|
|
## Measurement Setup
|
|||
|
|
|
|||
|
|
**Hypotheses / Scenarios Tested**
|
|||
|
|
<!-- If hypothesis-driven: name each hypothesis and its test scenarios. If scenario-driven: name each scenario group (e.g., W1–W3 for write-triggers, R1–R4 for retrieval). One paragraph or brief list per hypothesis/group, clarifying what correctness looks like. -->
|
|||
|
|
|
|||
|
|
**Fixture and Sampling**
|
|||
|
|
<!-- Which fixture(s) were used (real project vs synthetic, language/domain, size, ADR/decision history if relevant). Fixture generalization risk. Reps per cell. Rationale for rep count (e.g., 1 rep/cell for baseline discovery, 3 reps for wording-loop stability). -->
|
|||
|
|
|
|||
|
|
**Experimental Control (Frozen Surfaces)**
|
|||
|
|
<!-- What was held constant during this eval and why. For skill-wording loops: checker, fixtures, scenarios, rubric frozen — only wording moves. For harness-design evals: [list what was frozen]. Explicitly name anything intentionally NOT frozen, and why (e.g., model tiers vary to establish tier-specific performance; rep count chosen for discovery vs confirmation). -->
|
|||
|
|
|
|||
|
|
## Validity and Limitations
|
|||
|
|
|
|||
|
|
**How to Interpret These Results**
|
|||
|
|
<!-- Training-set vs held-out framing — if this eval was optimized against (wording loop, rubric tuning), state that explicitly. What the numbers do and do not support (e.g., "8/8 on a training-set grid means wording direction is sound, not that the behavior generalizes"). Confidence caveats: variance at low reps, fixture-specific behavior, scenarios-only measurement (not live observability). When to read a result as significant (e.g., majority of reps, control cells hold). When to read a result as noise (e.g., single-rep flips between cells, below the rep threshold for stability). -->
|
|||
|
|
|
|||
|
|
**Weaknesses of This Eval (Its Ladder Level)**
|
|||
|
|
<!-- What this eval can't see or didn't test. Examples: single fixture generalization (would a second fixture in a different language / domain change the results?), 1 rep/cell variance (high noise floor), no ambiguity axis (scenarios are clear-cue vs no clear boundary-case testing), limited distractor count, no longer held-out (wording was tuned against this grid — it's now training-set), model-specific failure modes, ablation surfaces that were never tested (e.g., "channel ablation not run — don't assume hook redundancy"). Open questions the eval can't resolve. Open questions the eval raises. -->
|
|||
|
|
|
|||
|
|
## Deployment and Evolution
|
|||
|
|
|
|||
|
|
**Good-Enough Gate**
|
|||
|
|
<!-- Explicit criterion under which these results justify deployment, adoption, or real-world rollout. Examples: "Sonnet 8/8 passes for pilot rollout on projects with Rust / Go codebases; haiku pending 3-rep confirmation on the W3 edge case (running today, 2026-07-06)." Or: "5/8 passes the 'prompting issue, not capability gap' threshold; spin up a wording loop before production rollout." Tier-specific status. Conditions that would change the call. -->
|
|||
|
|
|
|||
|
|
**Hardening Path / Next Measurement**
|
|||
|
|
<!-- If there is a follow-up eval harness (e.g., Eval C at a higher ladder level), point to it. Otherwise, point to the ladder-approach methodology note explaining the progression. One paragraph or brief list of what gets tested next and why (e.g., "Eval C will add ambiguity-ladder discrimination scenarios to test whether the model mistakes clear-cue for ambiguous-cue; if it does, wording tuning stops and the feature is capability-limited"). -->
|
|||
|
|
|
|||
|
|
## Related
|
|||
|
|
|
|||
|
|
<!-- Wikilinks to: prior baselines or follow-up evals, the methodology note(s) (ladder approach, autoresearch procedure), the skills/design docs being evaluated, and hub notes if any. -->
|
|||
|
|
|
|||
|
|
- [[note-slug]] — why it is relevant
|