diff --git a/_templates/eval-results.md b/_templates/eval-results.md index 4d45b89..e841ffd 100644 --- a/_templates/eval-results.md +++ b/_templates/eval-results.md @@ -48,11 +48,22 @@ source: [project name] # project that contains the eval harness **Weaknesses of This Eval (Its Ladder Level)** +## Transferable Lessons + + + +- [Generalizable rule — why it matters] + +**Promotion check:** + ## Deployment and Evolution **Good-Enough Gate** +**Proposed Changes (Loop Inputs)** + + **Hardening Path / Next Measurement** diff --git a/eval-methodology-ladder.md b/eval-methodology-ladder.md index 1b82bdd..9dfc7ab 100644 --- a/eval-methodology-ladder.md +++ b/eval-methodology-ladder.md @@ -9,7 +9,7 @@ tags: - convention/eval-design - tool/autoresearch scope: global -last_updated: 2026-07-06 # hardened same day with Eval C first-grid lessons (checker-conformance, self-test blind spot) +last_updated: 2026-07-07 # added lesson-promotion pattern (eval-results → methodology notes, Loop Inputs handoff) date: 2026-07-06 related: - running-autoresearch-skill-evals @@ -72,6 +72,17 @@ Run each scenario multiple times (reps) at a given level. Reps catch variance an Lower model tiers (e.g., haiku vs sonnet) may not clear the same rung at the same pass bar. E.g., "sonnet clears level 1 at 3-rep majority (all scenarios PASS), so move to level 2; haiku still at 2/3 on some scenarios, so continue level 1 with more reps." The ladder is flexible per-tier. +### Pattern: Lesson promotion from eval-results notes + +Eval-results notes are episodic records of one measurement; methodology notes (this one and +its siblings) are where compounding knowledge accretes. The `eval-results` template ends with +a mandatory promotion check: any transferable lesson that recurs (or names a new confound) +gets written into the relevant methodology note at the moment the eval-results note is +finished — not left stranded per-eval. The eval-results note keeps a one-line bullet pointing +at the methodology note. Eval-results notes also carry a **Proposed Changes (Loop Inputs)** +section holding the *concrete candidate wording* a tuning loop will test — the handoff +artifact that lets a later reader diff proposal against what shipped. + ### Pattern: Stop criterion **"Good enough for now → test IRL"** is the gate between ladder evals and production validation. When the current level is reliably PASS and you're confident in the behavior, stop optimizing for that level and prepare for production rollout. That gate is only meaningful if **deliberate, instrumented observation** of real sessions (see [[eval-methodology-irl-feedback-loop]]) will surface silent misses and feed them back as new scenarios. diff --git a/os-vault-write-eval-baseline-grid-results.md b/os-vault-write-eval-baseline-grid-results.md index 687bbca..94a0d19 100644 --- a/os-vault-write-eval-baseline-grid-results.md +++ b/os-vault-write-eval-baseline-grid-results.md @@ -9,7 +9,7 @@ tags: - tool/claude-code - project/cc-os scope: global -last_updated: 2026-07-06 +last_updated: 2026-07-07 # added Transferable Lessons + Proposed Changes (WS2 Loop Inputs) sections date: 2026-07-06 related: - eval-methodology-ladder @@ -85,12 +85,71 @@ only via the informational reasons trail, not a dedicated axis; no measurement o `/os-vault:write` skill when explicitly invoked (that's an Eval-A-shaped question, out of scope here). +## Transferable Lessons + +- **Persistence evals need a destination axis, not just a trigger axis.** When the harness (or any host tool) has a competing built-in persistence channel, "did the model save it?" and "did it save it to the right place?" are separate failure modes — this grid would have read as a flat trigger gap without the reasons-trail showing every L1 rep wrote to auto-memory. +- **"Remember this" wording loses to the default channel.** Trigger wording that names the action but not the destination drains to whatever persistence path is cheapest/built-in. Candidate wording must name the destination explicitly. +- **Count canary reps.** The one vault-conforming pass was a canary; had it been excluded, the ceiling-exists evidence would have been lost (and including it avoids peeking bias in the denominator). +- **A clean-negative baseline converts to a hard regression gate.** 18/18 negatives means the tuning loop gets a free, unambiguous stop condition: any negative FAIL during wording iteration is a hard stop, no judgment needed. + +**Promotion check:** the destination-axis lesson (first bullet) is a new eval-design confound not yet covered by [[eval-methodology-ladder]] — promote it there if a second eval hits a competing-channel shape; nothing else recurring yet. "Count canary reps" is already practiced in [[running-autoresearch-skill-evals]] discipline. + ## Deployment and Evolution **Good-Enough Gate** — Not applicable yet; this grid establishes the gap the WS2 wording loop must close. Negatives at 18/18 mean restraint needs no work — wording iteration should optimize trigger + routing only, and any future negative regression is a hard stop. +**Proposed Changes (WS2 Loop Inputs)** — concrete candidate wording drafted 2026-07-07, +NOT yet applied to any os-vault surface (applying it starts the wording loop and converts the +run-set to training-set; measurement then moves to the frozen reserve). Diff these against +what actually ships. + +*Candidate 1 — destination ladder* (targets: write SKILL.md description/body + an os-vault +SessionStart usage note + candidate trigger-phrased CLAUDE.md section; addresses both measured +gaps — L1 mis-routing and L2/L3 under-trigger): + +> **Where knowledge goes — ask in order:** +> 1. Does the repo already record it (code, ADR, CLAUDE.md, git history)? → write it there or +> nowhere; never duplicate into memory. +> 2. Would this change how you act in a **different** repo, next year? → the **SecondBrain +> vault** via `/os-vault:write` — NOT auto-memory. Facts about the world: tool behavior, +> client conventions, API quirks, methodology that worked. The repo is where you *learned* +> it, not what it's *about*. +> 3. Only useful in future sessions of **this** repo (where things live here, what we tried +> here)? → auto-memory. +> 4. Task status, session state, in-flight progress → nothing. +> +> Tiebreaker: what is the fact **about**? About a tool/client/API → vault. About this repo → +> auto-memory. When a durable fact surfaces mid-task — even with no one saying "remember" — +> run the ladder before moving on. + +*Candidate 2 — query-side triggers* (targets: query SKILL.md description + SessionStart usage +note; NO eval coverage yet — a query-side eval is a separate future harness, don't score it +against this one): + +> - Before designing or setting up an eval → `/os-vault:query` for `type/eval-results` + +> `domain/llm-evaluation` (and the project facet): what was already run, what the +> hypotheses/results were, what mistakes not to repeat. +> - Before first use this session of a tool you have vault notes for → query `tool/`. +> - Before starting work involving a client → query `client/`. +> - Don't re-query a facet already queried this session — the note content is in context. + +*Candidate 3 — update-vs-create rule* (target: write SKILL.md procedure): + +> Write is query-first: before creating a note, query for existing notes on the same subject. +> New fact contradicts/extends/refines an existing note → update it and bump `last_updated`. +> Genuinely new subject → new note. + +*Candidate 4 — type-misfit routing* (target: write SKILL.md; conservative, human-gated +taxonomy growth): + +> Content that fits no existing note type: do NOT invent a type or force a bad fit silently. +> Write it under the closest type with a visible `## Type-fit note` line saying what didn't +> fit, or — if the shape is clearly recurring — propose a new type via +> `/os-vault:design-template`. Misfits get reviewed in `/os-vault:reorganize`; three misfits +> of the same shape justify a new type. + **Hardening Path / Next Measurement** — Per `ws2-os-vault-write-eval.md` step 4: an `/autoresearch` wording loop (checker/fixtures/scenarios/rubric frozen; surfaces = write SKILL.md description/body, an os-vault SessionStart usage note, possibly a trigger-phrased