vault: session notes 2026-07-06
This commit is contained in:
parent
ce4f572137
commit
2e21631043
|
|
@ -0,0 +1,80 @@
|
|||
---
|
||||
type: hub
|
||||
title: Evaluation methodology for LLM skill behavior (cc-os)
|
||||
summary: Navigation hub for evaluation design, harness setup, wording optimization, and production validation patterns — the full progression from isolated eval design to real-world rollout and feedback.
|
||||
tags:
|
||||
- type/hub
|
||||
- domain/llm-evaluation
|
||||
- tool/autoresearch
|
||||
- tool/os-adr
|
||||
- convention/eval-design
|
||||
- project/cc-os
|
||||
scope: global
|
||||
last_updated: 2026-07-06
|
||||
date: 2026-07-06
|
||||
source: cc-os
|
||||
---
|
||||
|
||||
# Evaluation methodology for LLM skill behavior
|
||||
|
||||
Navigation hub for the full evaluation pipeline: from harness design through wording tuning to production rollout and feedback loops. Each note covers a distinct phase or pattern; read the one that matches your current task.
|
||||
|
||||
## Quick Navigation
|
||||
|
||||
**I want to...**
|
||||
|
||||
| I want to... | Read this |
|
||||
|---|---|
|
||||
| Understand how to design eval harnesses and iterate wording safely | [[running-autoresearch-skill-evals]] |
|
||||
| See the os-adr Eval B baseline (unprompted triggering test) | [[os-adr-eval-b-grid-results-and-observations]] |
|
||||
| See the os-adr Eval B wording experiment results (5-iteration improvement) | [[os-adr-eval-b-wording-experiment-hypotheses]] |
|
||||
| Learn the ladder approach for progressively harder evals | [[eval-methodology-ladder]] |
|
||||
| Set up production auditing and close the feedback loop | [[eval-methodology-irl-feedback-loop]] |
|
||||
|
||||
## The Full Progression
|
||||
|
||||
### Phase 1: Initial eval design and baseline measurement
|
||||
|
||||
**[[running-autoresearch-skill-evals]]** — Procedure for running evaluations and wording loops. When to use in-session vs headless runners, how to refresh plugin caches so wording edits take effect, how to parallelize runs and interpret results at the axis level.
|
||||
|
||||
**[[os-adr-eval-b-grid-results-and-observations]]** — Concrete baseline eval for os-adr unprompted-triggering behavior (haiku 0/8, sonnet 5/8). Shows what a held-out measurement looks like and why the results point to a prompting/wording issue rather than a capability gap.
|
||||
|
||||
### Phase 2: Controlled wording optimization
|
||||
|
||||
**[[os-adr-eval-b-wording-experiment-hypotheses]]** — Five-iteration wording loop that improved the baseline (sonnet 8/8, haiku 7/8). Demonstrates the hypothesis→verdict tracking pattern, per-iteration results, and how to close gaps via targeted wording in specific channels (hook note, CLAUDE.md, skill description).
|
||||
|
||||
**Key result:** Trigger-conditioned phrasing ("**when** you encounter X, do Y") outperforms inventory statements; each rule must live where its precondition is visible (step-2 wording in skill bodies, not hook notes).
|
||||
|
||||
### Phase 3: Ladder progression and generalization testing
|
||||
|
||||
**[[eval-methodology-ladder]]** — Design strategy for successive evals at increasing difficulty: Level 1 (clear-cue baseline), Level 2 (ambiguous-cue discrimination), Level 3 (edge-case over-trigger risks). Pairs positive/negative scenarios at every level. Freeze evaluation surfaces (checker, fixture, scenarios) so wording can move independently. Run-set vs held-out reserve discipline to maintain measurement validity after tuning.
|
||||
|
||||
**Key pattern:** Per-level pass bars, not aggregate scores. Once a level is clear, move to the next. Non-monotonic difficulty (passing hard does not imply passing easy), so anchor at easy.
|
||||
|
||||
### Phase 4: Production validation and feedback closure
|
||||
|
||||
**[[eval-methodology-irl-feedback-loop]]** — Audit real sessions in onboarded projects on a recurring schedule (1–2 weeks post-rollout, then per-cycle). Judge each session: "should-have-triggered?" Log miss patterns. Promote recurring misses into new eval scenarios. Run follow-up audits after wording changes to confirm the fix. This is how silent failures (undetected decision misses) surface and feed back into evals.
|
||||
|
||||
**Key insight:** Post-rollout observation is only meaningful if deliberate and instrumented. Without auditing, silent misses go undetected for months.
|
||||
|
||||
## The Notes at a Glance
|
||||
|
||||
| Note | Type | What it covers | Read if... |
|
||||
|---|---|---|---|
|
||||
| [[running-autoresearch-skill-evals]] | howto | Skill-wording eval loops: valid run modes, cache refresh, reduced grids, parallelization, rep counts | You're about to design or run an eval loop |
|
||||
| [[os-adr-eval-b-grid-results-and-observations]] | eval-results | os-adr Eval B baseline (haiku 0/8, sonnet 5/8, 1 rep/cell), confirmation run, observations | You're designing a follow-up eval and need the baseline context |
|
||||
| [[os-adr-eval-b-wording-experiment-hypotheses]] | eval-results | os-adr Eval B wording tuning (5 iterations, sonnet 8/8, haiku 7/8), hypothesis tracking, deployment gate | You want to see how a trained-up eval goes and what the next gate looks like |
|
||||
| [[eval-methodology-ladder]] | reference / pattern-framework | Evaluation ladder design: clear→ambiguous→edge-case progressions, paired scenarios, per-level pass bars, run/reserve splits | You're designing a hardened eval and want to avoid common pitfalls |
|
||||
| [[eval-methodology-irl-feedback-loop]] | reference / pattern-framework | Production auditing: session sampling, miss pattern logging, scenario authoring from audit findings, close-the-loop pattern | You want to set up deliberate post-rollout observation and close the feedback loop |
|
||||
|
||||
## Cross-project application
|
||||
|
||||
The ladder approach and audit pattern are generalizable beyond os-adr:
|
||||
|
||||
- **Ladder approach:** Any skill/feature with unprompted-behavior evaluation (should the model notice it's relevant without being told?) can use Level 1 baseline → wording tuning → Level 2 ambiguity discrimination → Level 3 over-trigger safeguards.
|
||||
- **Audit pattern:** Any feature rolled out to multiple projects benefits from recurring session audits. The pattern is universal; only the audit criterion changes (e.g., "should consult X", "should format as Y", "should refuse Z").
|
||||
|
||||
## Related
|
||||
|
||||
- [[cc-os-plugin-skill-naming-convention]] — naming and registration mechanics (used in wording placement decisions)
|
||||
- [[running-autoresearch-skill-evals]] (howto) — also has a "Related" section with detailed eval-specific references
|
||||
|
|
@ -0,0 +1,136 @@
|
|||
---
|
||||
type: reference
|
||||
subtype: pattern/framework
|
||||
title: Production validation via session auditing — turning silent misses into eval scenarios
|
||||
summary: A pattern for treating post-rollout observation as a deliberate, recurring backlog item: periodically audit real sessions in onboarded projects, log should-have-triggered vs did-trigger patterns, and promote recurring misses into new eval scenarios.
|
||||
tags:
|
||||
- type/reference
|
||||
- domain/llm-evaluation
|
||||
- convention/eval-design
|
||||
scope: global
|
||||
last_updated: 2026-07-06
|
||||
date: 2026-07-06
|
||||
related:
|
||||
- eval-methodology-ladder
|
||||
- running-autoresearch-skill-evals
|
||||
- os-adr-eval-b-grid-results-and-observations
|
||||
source: cc-os
|
||||
---
|
||||
|
||||
# Production validation via session auditing
|
||||
|
||||
## Purpose
|
||||
|
||||
Evaluations measure behavior in controlled scenarios. Production measures behavior in messy reality — different codebases, different decision patterns, different user workflows. Evals can pass while production silently fails. This pattern establishes a deliberate backlog item for discovering and closing the gap.
|
||||
|
||||
Why is "observation" not passive? Because **misses are silent.** If a model fails to record an architecture decision, nothing visibly breaks — decision history drifts, but the project works. A broken error message fails loud (user complains); a missed decision record fails silent (history corrupts, unnoticed for months). Passive hope ("users will tell us") doesn't work for silent failures. You must audit deliberately.
|
||||
|
||||
## Core Principles
|
||||
|
||||
**1. Silent misses require active instrumentation.** A feature that fails loud (raises an error) gives feedback immediately. A feature that fails silent (omits a step, forgets to record) requires sampling and analysis to surface. Post-rollout observation must be structured: pick a sample of sessions in real projects, judge each against a clear criterion ("should-have-triggered, did-trigger?"), and log the patterns.
|
||||
|
||||
**2. The audit is not about blame — it's about scenario authoring.** When a session shows "should have triggered, didn't," the task is not "why did the model fail?" (blame) but "what aspect of this scenario isn't covered in our current eval?" (design). Each recurring miss shape becomes a new scenario for a follow-up eval.
|
||||
|
||||
**3. The feedback loop closes when new scenarios enter the eval.** A miss discovered in audit that never makes it into a future eval is a missed opportunity. The pattern succeeds when production findings become new Eval Level 2 / 3 scenarios (see [[eval-methodology-ladder]]); the loop closes when improved wording / architecture based on those scenarios gets deployed back to the same onboarded projects.
|
||||
|
||||
**4. Cadence matters.** Auditing is not one-time; it's recurring. After the first 1–2 weeks in a onboarded project (when user activity is highest and model interactions are freshest), run the first audit. Then audit every 2 weeks or per-project-phase depending on decision velocity.
|
||||
|
||||
## Decision Framework
|
||||
|
||||
| When | What to audit |
|
||||
|---|---|
|
||||
| First 1–2 weeks after rollout to a new project | Sample 10–20 sessions (or all sessions if the project has <20 in the period). Look for false negatives ("should have triggered, didn't") and false positives ("over-triggered, shouldn't have"). Establish baseline miss patterns. |
|
||||
| After wording/architecture tuning from a prior audit | Sample 5–10 sessions to check whether the changes reduced the miss rate. If misses persist in new shapes, recurse to the "analyze" step. |
|
||||
| Steady-state (3+ projects live, stable wording) | Audit one project per cycle (round-robin) with sample size 5. Track whether miss patterns are consistent across projects (fixture-specific vs universal). |
|
||||
|
||||
**Session sampling strategy:**
|
||||
- **Random or systematic:** Pick every Nth session from the project's audit log, or randomly sample a date range. Avoid cherry-picking (audit shows surprises; cherry-pick hides them).
|
||||
- **Scope:** A "session" is one Claude Code context window. Audit the entire conversation (full tool_use transcript), not just successes.
|
||||
- **Sample size:** 10–20 sessions for the first audit (establishes baseline and catches ~70% of miss patterns with reasonable rarity filtering). 5 sessions for follow-ups (faster iteration, assuming earlier audits already surfaced the common patterns).
|
||||
|
||||
**Judging should-have-triggered:**
|
||||
- Be explicit about the criterion. E.g., "should consult `/os-adr:find` if: (a) user is making or modifying an architecture-level choice, (b) the choice isn't totally novel to the codebase, and (c) the choice is documented anywhere in the ADR system." Write this down before auditing so raters don't drift.
|
||||
- A "should-have-triggered" miss is when criterion (a)–(c) are all met, but the model didn't consult.
|
||||
- An "over-triggered" miss is when the model consulted when only (a) was true but (b)–(c) were false (e.g., a decision so novel that no prior ADR applies).
|
||||
- A "did-trigger, correct outcome" is not a miss — it's a hit.
|
||||
|
||||
## Patterns
|
||||
|
||||
### Pattern: Audit report template
|
||||
|
||||
```
|
||||
**Project:** [name]
|
||||
**Audit date:** [YYYY-MM-DD]
|
||||
**Sessions sampled:** [count and date range]
|
||||
**Auditor:** [who]
|
||||
|
||||
**Baseline metrics:**
|
||||
- Total sessions sampled: [N]
|
||||
- Sessions with relevant decisions: [N] (filtered: only count sessions where the criterion applies)
|
||||
- Correct consultation rate: [N/M]
|
||||
- Should-have-triggered, didn't: [N] (list scenarios)
|
||||
- Over-triggered (false positive): [N] (list scenarios)
|
||||
- Judge call / ambiguous: [N] (borderline cases where the criterion is unclear)
|
||||
|
||||
**Miss patterns identified:**
|
||||
1. [Pattern name]: [description, example]
|
||||
2. [Pattern name]: [description, example]
|
||||
...
|
||||
|
||||
**New scenarios for next eval:**
|
||||
- [Miss pattern 1] → Level [2/3] scenario: positive case [X], negative case [Y]
|
||||
- [Miss pattern 2] → Level [2/3] scenario: positive case [X], negative case [Y]
|
||||
|
||||
**Recommendation:**
|
||||
- Severity: [low/medium/high] (how often does this miss occur? how bad is the consequence?)
|
||||
- Next step: [add to Eval Level N backlog / reword in CLAUDE.md / architecture change needed / defer]
|
||||
```
|
||||
|
||||
### Pattern: Audit backlog and prioritization
|
||||
|
||||
Create a simple backlog of audits as a recurring task:
|
||||
1. Audit newly onboarded projects after 1–2 weeks.
|
||||
2. Track which projects have been audited and when.
|
||||
3. Round-robin steady-state audits (one project per cycle).
|
||||
4. Prioritize re-auditing projects that showed high miss rates in prior cycles.
|
||||
|
||||
Prioritize misses by:
|
||||
- **Severity:** How often does the miss occur? (rare fluke vs systematic gap)
|
||||
- **Consequence:** What breaks if the miss goes undetected? (silent corruption vs visible error)
|
||||
- **Design feasibility:** Can this be fixed by wording, or does it require architecture change?
|
||||
|
||||
High-severity, high-consequence misses with feasible fixes become new Eval Level 2 scenarios immediately.
|
||||
|
||||
### Pattern: Closing the loop — from audit to eval to deployment
|
||||
|
||||
1. **Audit surfaced a miss pattern** → log it in the audit report.
|
||||
2. **New scenarios authored** → author Level 2 or 3 scenarios that reproduce the miss (positive: the correct case; negative: the false-premise case that was missed).
|
||||
3. **Eval run against the scenarios** → confirm the miss (model fails the scenario in the same way it failed in production).
|
||||
4. **Wording/architecture tuning** → iterate based on the new scenarios (in a controlled eval, not directly in production).
|
||||
5. **Deploy the tuning** → ship the improved wording to the onboarded projects.
|
||||
6. **Follow-up audit** → run a quick re-audit on the project where the miss was found to confirm the fix worked.
|
||||
|
||||
This cycle is the **feedback loop closure**. Projects become co-authors of the eval through their production experiences.
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
- **Auditing without a clear criterion.** "Judge whether the model should have consulted the ADR system" is too vague. Write the exact conditions and stick to them across audits.
|
||||
- **One-time audit and moving on.** Post-rollout observation that doesn't recur is passive hope wearing a label. Recurring audits catch drift over time and provide feedback for the next eval cycle.
|
||||
- **Auditing successes only.** If you only sample sessions that went well, you'll miss the silent failures. Audit randomly, or weight toward sessions with *decisions* (the feature is only relevant in decision-heavy work).
|
||||
- **Misses that never become scenarios.** If a production miss is identified but never authoring as an eval scenario, it's context-specific feedback with no leverage. High-severity misses must enter the eval backlog; otherwise, the feature never improves.
|
||||
- **Conflating audit findings with blame.** "The model failed because the wording is bad" is defensible. "The model failed because it's stupid" is not actionable and demoralizes. Audits are design feedback, not model critiques.
|
||||
|
||||
## Known Limitations
|
||||
|
||||
Auditing requires access to real-session transcripts, which may have privacy implications if the projects involve clients or sensitive work. Ensure audit access is governed by the same policies as the vault (e.g., local-only, encrypted).
|
||||
|
||||
Auditing is labor-intensive (10–20 sessions × ~5–10 min per session judgment = 50–200 min per audit). Scaling to many projects requires either automation (e.g., heuristic pre-filtering for decision-relevant sessions) or delegating audit work to multiple raters (and managing inter-rater consistency). A third path is to run audits less frequently (per-quarter instead of bi-weekly) for mature deployments.
|
||||
|
||||
The pattern assumes the model's behavior is the variable under test. If the "miss" is actually a user workflow issue (user didn't provide enough context for a decision to be visible), that's not a model failure — it's a feature-discovery or onboarding gap. Keep that distinction clear in audit reports.
|
||||
|
||||
## Related
|
||||
|
||||
- [[eval-methodology-hub]] — navigation hub for all evaluation-methodology notes
|
||||
- [[eval-methodology-ladder]] — how audits feed into next-level scenarios for the eval progression
|
||||
- [[running-autoresearch-skill-evals]] — how to run evals that incorporate audit-sourced scenarios
|
||||
- [[os-adr-eval-b-grid-results-and-observations]] — the baseline eval before production rollout
|
||||
|
|
@ -0,0 +1,99 @@
|
|||
---
|
||||
type: reference
|
||||
subtype: pattern/framework
|
||||
title: Evaluation ladder approach for LLM behavior measurement
|
||||
summary: A methodology for progressively hardening evaluations from clear-cue baselines to ambiguity-discriminating tests, with per-level pass bars and held-out reserve discipline to maintain measurement validity.
|
||||
tags:
|
||||
- type/reference
|
||||
- domain/llm-evaluation
|
||||
- convention/eval-design
|
||||
- tool/autoresearch
|
||||
scope: global
|
||||
last_updated: 2026-07-06
|
||||
date: 2026-07-06
|
||||
related:
|
||||
- running-autoresearch-skill-evals
|
||||
- os-adr-eval-b-grid-results-and-observations
|
||||
- os-adr-eval-b-wording-experiment-hypotheses
|
||||
- eval-methodology-irl-feedback-loop
|
||||
source: cc-os
|
||||
---
|
||||
|
||||
# Evaluation ladder approach
|
||||
|
||||
Establish confidence in LLM behavior through a ladder of increasing difficulty and realism, not through a single high-bar eval. Each rung tests a distinct capability or confound; pass bars are per-level, not aggregate. The ladder is a strategy for designing evals that remain valid measurement instruments as tuning occurs.
|
||||
|
||||
## Purpose
|
||||
|
||||
Evaluations drift from measurement toward training artifacts the moment you start optimizing against them (wording tuning, architecture changes, prompt engineering). The ladder approach maintains validity by:
|
||||
|
||||
1. Anchoring at the easy end (clear-cue boundary cases) because difficulty is non-monotonic — passing hard does not imply passing easy, and a single failure at an easy level can invalidate a claimed strength.
|
||||
2. Building pairs of scenarios at every level: positive (correct action under intended conditions) and negative (equally-subtle non-action or over-action under absence/false-premise conditions). The trivial way to ace a boundary-case eval is to always trigger; paired scenarios catch this.
|
||||
3. Freezing evaluation surfaces (checker, fixtures, scenarios) so wording can move independently.
|
||||
4. Declaring per-level pass bars so you can retire a rung once it's clear (e.g., "level 1 clear, stop optimizing for clear-cue; level 2 open").
|
||||
5. Splitting scenarios into run-set and held-out reserve upfront (~2× authoring), so if wording is ever tuned against the run-set, the reserve becomes a fresh measurement set for the next evaluation cycle.
|
||||
|
||||
The ladder is not a single evaluation — it's a **progression strategy** for successive evals that test different aspects of the same behavior.
|
||||
|
||||
## Core Principles
|
||||
|
||||
**1. Ladder levels represent confound progression, not scoring tiers.** Level 1 tests a behavior with maximum clarity (clear cues, no ambiguity). Level 2 adds one confound (ambiguous cues, but the positive/negative cases are still fully distinct). Level 3 adds another (edge cases where the action *could* apply but shouldn't, requiring boundary judgment). The "success" of level 1 does not imply level 2 success; each level has its own pass bar.
|
||||
|
||||
**2. Paired scenarios at every level catch both under-triggering and over-triggering.** A positive scenario tests "does the model take the correct action when conditions are met?" A negative scenario tests "does the model correctly *not* take the action when conditions are absent or when a plausible false-premise tempts over-triggering?" A model that always triggers will ace positive scenarios and fail negatives; a model that under-triggers will fail positives. Both signal real problems that single-axis evals miss.
|
||||
|
||||
**3. Difficulty is non-monotonic — anchoring easy catches breaks that hard scenarios miss.** A model might pass a single hard boundary-case scenario but fail the basic case. Starting at easy ensures the foundation is solid before claiming strength at harder levels. Historically, many eval designs began at medium difficulty and missed that the model couldn't handle the basics.
|
||||
|
||||
**4. Run-set vs held-out reserve discipline prevents training-set creep.** Scenarios are authored at roughly 2× the quantity needed for the run-set; the reserve is held out and never used for tuning feedback. If a wording optimization loop targets the run-set, the reserve becomes the fresh measurement set for the next cycle. This discipline costs upfront scenario authoring but buys back measurement validity after tuning.
|
||||
|
||||
**5. Per-level pass bars, not aggregate scores.** Each level has a passing criterion (e.g., "level 1: all run-set scenarios, 3+ reps per scenario, majority rep threshold"). Once level 1 is clear (reliably PASS over 2–3 eval runs), investment moves to level 2. An aggregate score like "75% pass" conflates clear and ambiguous scenarios, masking which capability gaps remain.
|
||||
|
||||
## Decision Framework
|
||||
|
||||
| If you're asking... | Use this level structure |
|
||||
|---|---|
|
||||
| "Does the model understand what it's supposed to do?" | Level 1 (clear-cue): instructed actions with maximum salience, no ambiguity. Positive = correct action. Negative = do nothing or non-action. Does not test triggering discrimination. |
|
||||
| "Can the model distinguish between conditions-met and conditions-absent?" | Level 2 (ambiguous-cue): scenarios where the positive case *looks* similar to the negative case, but one has a critical distinguishing feature. Positive = correct action. Negative = similar setup but the trigger condition is actually absent or a false-premise tempts action. Tests whether the model mistakes close-calls. |
|
||||
| "Can the model handle edge cases and over-trigger risks?" | Level 3 (edge-case discrimination): scenarios where action *could* apply based on a loose reading, but shouldn't based on the actual requirement. Positive = narrow legitimate case. Negative = over-trigger case (model takes the action when it shouldn't). Tests boundary judgment. |
|
||||
| "Does the behavior generalize beyond the fixture?" | Separate fixture in a different language/domain. Repeats level 1–2 on new fixture to test fixture-generalization. Same ladder structure, different codebase. |
|
||||
|
||||
**Start your eval at the level that matches your current confidence.** Prompt-only (instructed) behavior starts at Level 1 (you're not claiming triggering discrimination yet). Unprompted self-triggering starts at Level 1 if it's novel (does the model notice at all?), then climbs to Level 2 (does it notice reliably?) and Level 3 (does it avoid over-triggering?).
|
||||
|
||||
## Patterns
|
||||
|
||||
### Pattern: Run-set + reserve split
|
||||
|
||||
Author scenarios at 2× the volume needed for a single run-set. Assign ~50% to the run-set (tuning-eligible), ~50% to reserve (held-out). If running a wording loop or architecture tuning, only run the run-set; reserve stays frozen. Once tuning is complete, run the reserve as a fresh measurement. If any surprises emerge on reserve (e.g., reserve-only over-triggering not visible in run-set), those become new Level 3 scenarios for the next ladder iteration.
|
||||
|
||||
### Pattern: Per-scenario rep count for confidence
|
||||
|
||||
Run each scenario multiple times (reps) at a given level. Reps catch variance and model non-determinism. E.g., Level 1: 3 reps per scenario. Level 2: 2–3 reps. Level 3: 1–2 reps (harder scenarios are often noisier, but you're willing to accept less confidence at the edge). Track which scenarios have high variance and mark them as canaries for follow-up evals.
|
||||
|
||||
### Pattern: Tier-specific laddering
|
||||
|
||||
Lower model tiers (e.g., haiku vs sonnet) may not clear the same rung at the same pass bar. E.g., "sonnet clears level 1 at 3-rep majority (all scenarios PASS), so move to level 2; haiku still at 2/3 on some scenarios, so continue level 1 with more reps." The ladder is flexible per-tier.
|
||||
|
||||
### Pattern: Stop criterion
|
||||
|
||||
**"Good enough for now → test IRL"** is the gate between ladder evals and production validation. When the current level is reliably PASS and you're confident in the behavior, stop optimizing for that level and prepare for production rollout. That gate is only meaningful if **deliberate, instrumented observation** of real sessions (see [[eval-methodology-irl-feedback-loop]]) will surface silent misses and feed them back as new scenarios.
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
- **Single high-bar scenario as the eval.** A model can pass one hard scenario while failing basics. Use levels, not singular difficulty.
|
||||
- **Aggregate scoring that hides per-level variance.** Avoid "model X passes 70% of all scenarios." Instead: "level 1: 100%, level 2: 60%, level 3: 30%." The breakdown is where the insight is.
|
||||
- **Scenarios that are too ambiguous to score cleanly.** If a negative scenario *looks* correct and you have to debate whether the model should have triggered, the scenario isn't at the right level yet. Move it up to a higher level or rewrite it.
|
||||
- **Forgetting that paired scenarios are *equally* subtle.** If your negative scenarios are obviously non-triggers (e.g., "add logging, no decision involved"), you're not testing discrimination. Negative scenarios should be as close to the positive case as possible without crossing the trigger threshold.
|
||||
- **Tuning against held-out reserve.** Once the reserve is defined, it becomes Schrodinger's eval — useful as a fresh measurement only if it stays frozen. If you peek at reserve results and change wording based on them, it's now training-set.
|
||||
|
||||
## Known Limitations
|
||||
|
||||
The ladder approach assumes you can enumerate confounds (ambiguity, edge cases) without hitting a combinatorial explosion. For highly context-dependent behaviors (e.g., "write helpful error messages" — helpful in what context?), the confound space is large and the ladder can be hard to design. In these cases, production auditing (see [[eval-methodology-irl-feedback-loop]]) becomes more critical earlier.
|
||||
|
||||
The approach also assumes that scenarios can be authored at 2× desired run-set size without exhausting designer time. This is true for discrete, bounded tasks (e.g., "consult ADRs") but may be expensive for open-ended behaviors.
|
||||
|
||||
## Related
|
||||
|
||||
- [[eval-methodology-hub]] — navigation hub for all evaluation-methodology notes
|
||||
- [[running-autoresearch-skill-evals]] — how to run wording loops safely while keeping evals valid
|
||||
- [[os-adr-eval-b-grid-results-and-observations]] — Level 1 baseline for os-adr unprompted triggering
|
||||
- [[os-adr-eval-b-wording-experiment-hypotheses]] — Level 1 tuning run against a wording loop
|
||||
- [[eval-methodology-irl-feedback-loop]] — production validation and how to close the loop: misses become new scenarios
|
||||
|
|
@ -24,3 +24,17 @@ tags: [scope/global, type/log]
|
|||
/home/jared/Documents/SecondBrain/os-adr-eval-b-wording-experiment-hypotheses.md
|
||||
/home/jared/Documents/SecondBrain/os-adr-eval-b-wording-experiment-hypotheses.md
|
||||
/home/jared/Documents/SecondBrain/2026-07-03-os-adr-eval-b-grid-results-and-observations.md
|
||||
|
||||
## Session — 2026-07-06T15:35:30Z
|
||||
|
||||
**Project:** /home/jared/dev/cc-os
|
||||
**Reason:** other
|
||||
**Vault notes touched:**
|
||||
(none)
|
||||
|
||||
## Session — 2026-07-06T16:19:54Z
|
||||
|
||||
**Project:** /home/jared/dev/cc-os
|
||||
**Reason:** other
|
||||
**Vault notes touched:**
|
||||
(none)
|
||||
|
|
|
|||
|
|
@ -1,115 +1,107 @@
|
|||
---
|
||||
summary: Hypotheses for the os-adr Eval B wording experiment (trigger-conditioned phrasing, packaging channel, two-step trigger chain) and the explicit map from grid outcomes to hypothesis verdicts — the tracking pattern any skill-wording eval should copy.
|
||||
type: eval-results
|
||||
title: os-adr Eval B wording experiment (2026-07-04) — hypotheses and results
|
||||
summary: Five-iteration wording loop optimizing trigger-phrasing closed the Eval B gap from sonnet 5/8 / haiku 0/8 to sonnet 8/8 / haiku 7/8 — confirms prompting/wording issue; defines next ladder level.
|
||||
tags:
|
||||
- scope/global
|
||||
- type/reference
|
||||
- project/cc-os
|
||||
- type/eval-results
|
||||
- domain/llm-evaluation
|
||||
- tool/os-adr
|
||||
- tool/autoresearch
|
||||
- domain/llm-evaluation
|
||||
source: cc-os
|
||||
- project/cc-os
|
||||
scope: global
|
||||
last_updated: 2026-07-06
|
||||
date: 2026-07-04
|
||||
last_updated: 2026-07-04
|
||||
related:
|
||||
- os-adr-eval-b-grid-results-and-observations
|
||||
- running-autoresearch-skill-evals
|
||||
- eval-methodology-ladder
|
||||
source: cc-os
|
||||
---
|
||||
|
||||
# os-adr Eval B wording experiment — hypotheses and result mapping
|
||||
# os-adr Eval B wording experiment (2026-07-04) — hypotheses and results
|
||||
|
||||
Design note for the 2026-07-04 `/autoresearch` wording loop against the Eval B baseline
|
||||
(haiku 0/8, sonnet 5/8 failing W3 + R1 — see
|
||||
[[os-adr-eval-b-grid-results-and-observations]]). Procedure discipline lives in
|
||||
[[running-autoresearch-skill-evals]]; this note holds *what we believe and how we'll know*.
|
||||
## Results Grid/Threshold
|
||||
|
||||
## Framing
|
||||
**Passing criterion:** Majority pass (≥2 of 3 reps) on all target cells (W1–W3 write-triggers, R1–R4 retrieval) across both tiers; control cells (W1/sonnet, R2 distractor) hold for regression detection.
|
||||
|
||||
Eval A (prompted) passes 100% on both tiers → the instructions are **followable**. Eval B
|
||||
(unprompted) fails → the problem is **situation recognition**: the model is told *what*
|
||||
exists (inventory: "this project has ADRs, `/os-adr:find` and `/os-adr:create` exist") but
|
||||
never classifies its *current moment* as ADR-relevant. Analogy that framed it: telling
|
||||
someone "the car is in the driveway" vs "when you're ready to go home, walk out to the car."
|
||||
**Final full grid (5-iteration wording loop, 2026-07-04):**
|
||||
|
||||
## Hypotheses
|
||||
**Sonnet 8/8** — all scenarios pass (was 5/8). **Haiku 7/8** — all but W3 axis-b pass (was 0/8; the one miss is a W3 axis-b judge-boundary flicker — model consulted, passed same cell mid-experiment confirmation, suggesting marginal rubric sensitivity).
|
||||
|
||||
**H1 — Phrasing type (primary):** Trigger-conditioned instructions ("**when** you are about
|
||||
to make an architecture-level choice, or change an approach that already exists in the
|
||||
codebase → run `/os-adr:find`") outperform declarative inventory statements, in whatever
|
||||
channel they appear. The model needs recognition cues to pattern-match its current task
|
||||
against, not facts to spontaneously recall.
|
||||
|
||||
**H2 — Channel (packaging):** Where the trigger phrasing lives determines whether it's
|
||||
present at decision time. Candidate channels: SessionStart hook note (seen once, may fade),
|
||||
project CLAUDE.md line (in context every session), skill `description` frontmatter
|
||||
(ambiently visible in the skill list). Sub-hypothesis: the SessionStart hook may be
|
||||
redundant once a trigger-phrased CLAUDE.md line exists — testable by ablation.
|
||||
|
||||
**H3 — Two-step trigger chain (instruction placement):** Each condition must reference only
|
||||
what the model can observe *at that point*. "Reversing a prior decision" is unknowable
|
||||
before consulting — so step 1 triggers on locally observable conditions (making/changing an
|
||||
architecture-level choice → `find`), and step 2 conditions on find's *output* (an Accepted
|
||||
ADR covers the approach you're changing → record a supersession). Step-2 wording belongs in
|
||||
the find/create skill bodies (visible when find's results are on screen), not the
|
||||
session-start note (long out of attention by then).
|
||||
|
||||
## How grid outcomes map to hypotheses
|
||||
**Hypothesis→verdict mapping (the results interpretation):**
|
||||
|
||||
| Outcome | Verdict |
|
||||
|---|---|
|
||||
| Sonnet R1 flips to PASS (axis a) after trigger-phrased entry surfaces | Confirms H1 for the consultation trigger |
|
||||
| Sonnet R1 flips to PASS (axis a) after trigger-phrased entry surfaces | Confirms H1 for consultation trigger |
|
||||
| Sonnet W3 flips to PASS (axis b) after step-2 wording in find/create bodies | Confirms H3 (placement) — W3's axis-a already passed, so entry-surface changes can't explain it |
|
||||
| Ablation: CLAUDE.md-only arm holds sonnet 8/8 without the hook note | Confirms H2 sub-hypothesis (hook redundant); keep hook only if its removal regresses |
|
||||
| Ablation: hook-only arm regresses vs CLAUDE.md-only | Channel strength ordering: persistent-context > session-start injection |
|
||||
| Haiku moves off 0/8 with the winning sonnet packaging | H1/H2 generalize down-tier; W2 (haiku's one axis-a flicker) is the expected first mover |
|
||||
| Haiku stays 0/8 after ~3 iterations of the sonnet-winning wording | Capability floor for unprompted triggering at haiku tier — record it and stop iterating |
|
||||
| W1/sonnet control cell regresses in any arm | Wording change rejected regardless of target-cell gains |
|
||||
| Any target cell moves on <2 of 3 reps | Noise, not signal — do not accept the change (variance demonstrated 2026-07-03/04) |
|
||||
| Ablation: CLAUDE.md-only arm hypothetically holds sonnet 8/8 without hook note | Would confirm H2 sub-hypothesis (hook redundant) — **never tested** |
|
||||
| Ablation: hook-only arm would regress vs CLAUDE.md-only | Would show channel strength ordering: persistent-context > session-start injection — **never tested** |
|
||||
| Haiku moves off 0/8 with sonnet-winning packaging | Confirms H1/H2 generalize down-tier; W2 (haiku's one axis-a flicker) is the expected first mover — **confirmed** |
|
||||
| Haiku stays low after 3+ iterations of the sonnet-winning wording | Would indicate capability floor for unprompted triggering at haiku tier — **not reached; haiku reached 7/8** |
|
||||
| W1/sonnet control cell regresses in any arm | Wording change rejected regardless of target-cell gains — **held across all iterations** |
|
||||
| Any target cell moves on <2 of 3 reps | Noise, not signal — wording change not accepted — **enforced; variance demonstrated** |
|
||||
|
||||
## Meta-lesson (why this note exists)
|
||||
Final **verdicts:** H1 CONFIRMED (trigger-conditioned phrasing ≫ inventory phrasing, both tiers). H3 CONFIRMED (each rule lives where its precondition is visible). H2 NOT TESTED (channels never ablated; SessionStart hook redundancy remains open). **Tier-generalization lesson:** sonnet acts on "architecture-level choice" (semantic trigger); haiku needs "before editing any existing file, run find — additions count" (mechanical trigger). The mechanical rule subsumes the semantic one.
|
||||
|
||||
Track hypothesis → expected-outcome mapping **before** running a wording loop. It forces
|
||||
each grid cell to have a declared meaning, prevents post-hoc rationalization of noisy
|
||||
flips, and lets later evals zero in faster by reusing confirmed mechanisms (e.g., if H1
|
||||
confirms here, start the *next* plugin's unprompted-behavior wording at trigger-conditioned
|
||||
phrasing instead of rediscovering it). Pair this note's pattern with the procedure in
|
||||
[[running-autoresearch-skill-evals]].
|
||||
---
|
||||
|
||||
## Results (2026-07-04 — experiment complete, 5 iterations)
|
||||
## Measurement Setup
|
||||
|
||||
Final full grid: **sonnet 8/8** (was 5/8), **haiku 7/8** (was 0/8; the one miss is a W3
|
||||
axis-b judge-boundary flicker — it consulted, and passed the same cell in the mid-experiment confirmation). Per-iteration:
|
||||
**Hypotheses (Three Framing Questions)**
|
||||
|
||||
1. **Iter 1 — trigger-conditioned rewrite, all channels** (hook note, find/create
|
||||
descriptions + bodies, fixture CLAUDE.md): R1/sonnet FAIL→3/3 PASS; haiku W2 canary
|
||||
0→3/3 PASS; W3/sonnet 1/3 (axis-b persists); W1 control holds. **H1 CONFIRMED** — and it reaches haiku, so haiku's 0/8 was packaging, not a capability floor.
|
||||
2. **Iter 2 — "task not complete until superseding ADR exists"**: W3 still 1/3. Transcript
|
||||
analysis showed the model behaving *well* (catches the conflict, offers options with the
|
||||
ADR attached to the reversal path) but phrasing the offer conditionally inside an option
|
||||
list — judge flicker at the rubric boundary, not forgetting.
|
||||
3. **Iter 3 — explicit-unconditional recording offer** ("state plainly: this reverses
|
||||
ADR-NNNN; I'll record a superseding ADR — never a parenthetical in an option list"):
|
||||
W3/sonnet 3/3 PASS. **H3 CONFIRMED** (placement in the find skill's act-on-findings step,
|
||||
where find's output is on screen).
|
||||
4. **Iter 4 — lower threshold "modifying existing behavior → find first"**: R4/haiku 0→3/3;
|
||||
R1/haiku still 0/3. Transcript: haiku classified R1's task as a *pure addition* (new
|
||||
bypass method), so any semantic trigger missed it.
|
||||
5. **Iter 5 — fully mechanical trigger** ("before your first edit to any existing file →
|
||||
run find on those paths; additions count — a new method can bypass a decided
|
||||
constraint"): R1/haiku axis-a 3/3, full 2/3; R2 distractor control holds (no
|
||||
over-triggering); sonnet unaffected.
|
||||
**H1 — Phrasing type (primary):** Trigger-conditioned instructions ("**when** you are about to make an architecture-level choice, or change an approach that already exists in the codebase → run `/os-adr:find`") outperform declarative inventory statements, in whatever channel they appear. The model needs recognition cues to pattern-match its current task against, not facts to spontaneously recall.
|
||||
|
||||
**Verdicts:** H1 confirmed (trigger-conditioned ≫ inventory phrasing, both tiers). H3
|
||||
confirmed (each rule lives where its precondition is visible; step-2 wording in the find
|
||||
skill fixed W3). **H2 NOT tested** — channels were never ablated (all-on throughout); whether
|
||||
the SessionStart hook is redundant next to a trigger-phrased CLAUDE.md remains open. Also
|
||||
open: R4-nograph now PASSES on both tiers — the strengthened wording made the graph layer
|
||||
unnecessary for that scenario, so the degradation check no longer differentiates.
|
||||
**H2 — Channel (packaging):** Where the trigger phrasing lives determines whether it's present at decision time. Candidate channels: SessionStart hook note (seen once, may fade), project CLAUDE.md line (in context every session), skill `description` frontmatter (ambiently visible in the skill list). Sub-hypothesis: the SessionStart hook may be redundant once a trigger-phrased CLAUDE.md line exists — testable by ablation (never run).
|
||||
|
||||
**Tier-generalization lesson:** each tier down needs a more *mechanical* trigger. Sonnet
|
||||
acts on "architecture-level choice"; haiku needs "before editing any existing file, run
|
||||
find — additions count." The mechanical rule subsumes the semantic one and costs one cheap
|
||||
deterministic CLI call per task — lead with it in future packaging.
|
||||
**H3 — Two-step trigger chain (instruction placement):** Each condition must reference only what the model can observe *at that point*. "Reversing a prior decision" is unknowable before consulting — so step 1 triggers on locally observable conditions (making/changing an architecture-level choice → `find`), and step 2 conditions on find's *output* (an Accepted ADR covers the approach you're changing → record a supersession). Step-2 wording belongs in the find/create skill bodies (visible when find's results are on screen), not the session-start note (long out of attention by then).
|
||||
|
||||
**Deployment note:** the fixture CLAUDE.md section is the template for real-project
|
||||
adoption (candidate: emit it from `/os-adr:init`/`migrate`); winning wording committed to
|
||||
cc-os (hook `session_start.py`, find/create SKILL.mds, eval-b fixture CLAUDE.md).
|
||||
**Baseline and Target Scenarios**
|
||||
|
||||
Baseline (from [[os-adr-eval-b-grid-results-and-observations]]): haiku 0/8, sonnet 5/8 failing W3 + R1. Problem: **situation recognition** — the model is told *what* exists (inventory: "this project has ADRs, `/os-adr:find` and `/os-adr:create` exist") but never classifies its *current moment* as ADR-relevant. Framing: telling someone "the car is in the driveway" vs "when you're ready to go home, walk out to the car."
|
||||
|
||||
**Per-iteration results (5 iterations, 2026-07-04):**
|
||||
|
||||
1. **Iter 1** — trigger-conditioned rewrite, all channels (hook note, find/create descriptions + bodies, fixture CLAUDE.md): R1/sonnet FAIL→3/3 PASS; haiku W2 canary 0→3/3 PASS; W3/sonnet 1/3 (axis-b persists); W1 control holds. **H1 CONFIRMED**; haiku's 0/8 was packaging, not capability floor.
|
||||
2. **Iter 2** — "task not complete until superseding ADR exists": W3 still 1/3. Transcript: model behaves well (catches conflict), but phrases offer conditionally inside option list — judge flicker at rubric boundary.
|
||||
3. **Iter 3** — explicit-unconditional recording offer ("state plainly: this reverses ADR-NNNN; I'll record a superseding ADR — never parenthetical in option list"): W3/sonnet 3/3 PASS. **H3 CONFIRMED** (placement in find skill's act-on-findings step).
|
||||
4. **Iter 4** — lower threshold "modifying existing behavior → find first": R4/haiku 0→3/3; R1/haiku still 0/3. Transcript: haiku classified R1 as *pure addition* (semantic trigger missed it).
|
||||
5. **Iter 5** — fully mechanical trigger ("before your first edit to any existing file → run find on those paths; additions count — a new method can bypass a decided constraint"): R1/haiku axis-a 3/3, full 2/3; R2 distractor control holds (no over-triggering); sonnet unaffected.
|
||||
|
||||
---
|
||||
|
||||
## Validity and Limitations
|
||||
|
||||
**How to Interpret These Results**
|
||||
|
||||
This eval **was optimized against** (wording loop running 5 iterations over the same grid) — the sonnet 8/8 / haiku 7/8 are **training-set scores**, not held-out. What these numbers support: "the wording direction is sound, and haiku is reachable via more-mechanical phrasing." What they do NOT support: "the wording generalizes to other codebases, languages, or unprompted scenarios not in this fixture." The grid-results baseline (haiku 0/8, sonnet 5/8) was held-out; once wording tuning started, this note's measurements became training-set. **Confidence caveats:** Sonnet 8/8 is clean (all 3 reps PASS on all cells). Haiku 7/8 includes one axis-b judge-boundary case (W3) where the model passed mid-experiment but flickered back to FAIL at the end; if the judge rubric sensitivity is the blocker rather than the model behavior, this is a measurement artifact, not a real gap. R4-nograph now PASSES on both tiers — the strengthened wording made the graph layer unnecessary for that scenario, so the degradation check no longer differentiates (open question: does the improved wording help in cases where the graph layer remains necessary elsewhere).
|
||||
|
||||
**Weaknesses of This Eval (Its Ladder Level)**
|
||||
|
||||
Single fixture (Ruby webhook-relay) in one language — generalization to other codebases unknown. 3 reps on target cells, 1 rep on controls — variance is smaller but not eliminated (W3 haiku axis-b flicker evident). No ambiguity axis: all scenarios are clear-cue boundary cases; does not test whether the model mistakes clear-cue for ambiguous-cue or over-triggers on low-confidence edges. Distractor testing minimal (R2 only) — cannot assess false-positive rate under heavy decision-clutter. Axis-b evaluation of W1–W3 relies on frozen rubric (`judge-rubric.md`); judge-boundary sensitivity suspected. Ablation surfaces never tested: channel redundancy (hook vs CLAUDE.md), whether in-session subagents could ever validly substitute for part of this (suspect invalid, but not definitively tested). **This is a training-set measurement** — if results were used to update SKILL.md / hook-note wording (they were), the grid becomes invalid as a held-out measurement. The next eval (Eval C, ambiguity-ladder discrimination) will test generalization via the ladder mechanism.
|
||||
|
||||
**Hardening Path / Next Measurement**
|
||||
|
||||
Eval C (in design, not yet built) will run an ambiguity-ladder discrimination eval on a second fixture (different language/domain to test fixture generalization) with paired positive/negative scenarios at each ladder level and per-level pass bars. See [[eval-methodology-ladder]] for the ladder approach and [[eval-methodology-irl-feedback-loop]] for production validation via session auditing.
|
||||
|
||||
---
|
||||
|
||||
## Deployment and Evolution
|
||||
|
||||
**Good-Enough Gate**
|
||||
|
||||
**Sonnet:** Cleared for real-project pilot rollout. The wording shipped in cc-os (hook `session_start.py`, find/create SKILL.mds, eval-b fixture CLAUDE.md). Projects should adopt via the trigger-phrased "Architecture decisions" CLAUDE.md section (copy from `eval-b/fixture/project/CLAUDE.md`; candidate: emit from `/os-adr:init`/`migrate`).
|
||||
|
||||
**Haiku:** Resolved 2026-07-06 — cleared for pilot, with a characterized caveat. The stability check ran 7 reps total (not the planned 3; the runner agent's first batches included failures it later excluded, so all reps are counted here to avoid peeking bias): axis (a) consultation 7/7 PASS — rock solid; axis (b) recording-offer ~5/7 PASS (~75–85%). One FAIL was plausibly infra (session died, no final message to judge); one was a genuine behavioral miss: haiku consulted, then asked a clarifying question instead of unconditionally proposing the superseding ADR — the same conditional-phrasing failure mode iteration 3 fixed on sonnet, improved but not eliminated on haiku. Verdict: intermittent flicker, not a hard gap. The grid claim stays **7/8 with W3 as an ~1-in-4/5 axis-b miss**, not 8/8. Practical impact is soft: haiku always consults; occasionally the user must say "yes, record it" rather than receiving an unconditional offer. Acceptable for daily use given production auditing (see [[eval-methodology-irl-feedback-loop]]); Eval C's per-level pass bars should anticipate exactly this borderline-behavior flicker at 1 rep.
|
||||
|
||||
**Next steps:** Real-project migration/adoption one at a time via `/os-adr:migrate` (pilot projects first, then cc-os retrofit, then wider). For each onboarded project, add the trigger-phrased "Architecture decisions" CLAUDE.md section. Establish a session-audit backlog (see [[eval-methodology-irl-feedback-loop]]) to surface silent misses and feed them back as new eval scenarios for Eval C.
|
||||
|
||||
## Meta-lesson
|
||||
|
||||
Track hypothesis → expected-outcome mapping **before** running a wording loop. Forces each grid cell to have declared meaning; prevents post-hoc rationalization of noisy flips; lets later evals reuse confirmed mechanisms (e.g., if H1 confirms, start the *next* plugin's unprompted-behavior wording at trigger-conditioned phrasing). Pair this note's pattern with [[running-autoresearch-skill-evals]] procedure.
|
||||
|
||||
## Related
|
||||
|
||||
- [[os-adr-eval-b-grid-results-and-observations]] — baseline held-out grid (2026-07-03) before wording optimization
|
||||
- [[running-autoresearch-skill-evals]] — skill-wording eval loop procedure and efficiency lessons
|
||||
- [[eval-methodology-ladder]] — evaluation ladder approach and progression logic
|
||||
- [[eval-methodology-irl-feedback-loop]] — production validation via session auditing
|
||||
|
|
|
|||
|
|
@ -55,6 +55,7 @@ Use exactly one `type/` tag per note.
|
|||
| `type/adr` | Architecture decision records |
|
||||
| `type/howto` | Process docs, step-by-step guides |
|
||||
| `type/hub` | Hub notes: index/navigation notes linking 3+ related notes (see Hub Notes) |
|
||||
| `type/eval-results` | Evaluation results: measurement data, confidence assessment, interpretation, deployment gates for skill/feature behavior testing |
|
||||
| `type/clip` | Web clips, saved articles |
|
||||
| `type/project-config` | Per-project tag inference rules and conventions |
|
||||
| `type/meta` | Vault governance (this file, CLAUDE.md) |
|
||||
|
|
@ -130,6 +131,13 @@ This section defines the three primary note types used for durable, evergreen kn
|
|||
**Template:** (not yet created)
|
||||
**Sub-templates:** none
|
||||
|
||||
### eval-results
|
||||
**Question this answers:** "What did we measure, how far do I trust it, and what does it justify?" An eval-results note records measurement data (grid/threshold results), the experimental setup, confidence/limitations, and the deployment gate or next-step criterion.
|
||||
**Value gate:** Longevity (findings remain interpretable in 6-12 months?) + Reusability (methodology and ladder-progression patterns generalize beyond the specific skill?)
|
||||
**Mutability:** stable knowledge (results do not change; framework is frozen once run), with supersession (later evals may render this one training-set; see Related links)
|
||||
**Template:** eval-results.md
|
||||
**Sub-templates:** none
|
||||
|
||||
**Note on directories:** The standard directories for these types are `convention/`, `reference/`, and `howto/` at the vault root. These directories do not yet exist; notes of these types currently live at the vault root until directory structure is established.
|
||||
|
||||
## Standard Frontmatter Schema
|
||||
|
|
@ -138,12 +146,12 @@ All notes use this frontmatter block. Type-specific additions are noted inline.
|
|||
|
||||
```yaml
|
||||
---
|
||||
type: [convention|reference|howto]
|
||||
type: [convention|reference|howto|eval-results]
|
||||
subtype: [pattern/framework|api-integration|role-definitions|design-rules] # reference only
|
||||
title: [Human-readable title]
|
||||
summary: [1-2 sentences answering "what question does this note answer?"]
|
||||
tags:
|
||||
- type/[convention|reference|howto]
|
||||
- type/[convention|reference|howto|eval-results]
|
||||
- domain/[field]
|
||||
- tool/[tool] # if tool-specific
|
||||
- client/[client] # if client-specific
|
||||
|
|
|
|||
Loading…
Reference in New Issue