eval-results: os-vault WS2 wording loop + reserve grid
This commit is contained in:
parent
a71399db7c
commit
2bbc528246
|
|
@ -0,0 +1,108 @@
|
||||||
|
---
|
||||||
|
type: eval-results
|
||||||
|
title: os-vault write eval — WS2 wording loop (4 iterations) + held-out reserve grid
|
||||||
|
summary: Four wording iterations took unprompted vault-write positives from 1/19 (baseline) to run-set confirm sonnet 18/18 / haiku 16/18, and held-out reserve 25/36 overall with trigger-axis 13/18 — the residual gaps are YAML-colon frontmatter breakage, haiku L2 no-trigger, and one generalization-laundering over-trigger; the reserve is now contaminated for re-measurement.
|
||||||
|
tags:
|
||||||
|
- type/eval-results
|
||||||
|
- domain/llm-evaluation
|
||||||
|
- tool/os-vault
|
||||||
|
- tool/claude-code
|
||||||
|
- project/cc-os
|
||||||
|
scope: global
|
||||||
|
last_updated: 2026-07-07
|
||||||
|
date: 2026-07-07
|
||||||
|
related:
|
||||||
|
- os-vault-write-eval-baseline-grid-results
|
||||||
|
- running-autoresearch-skill-evals
|
||||||
|
- os-adr-eval-b-wording-experiment-hypotheses
|
||||||
|
- eval-methodology-ladder
|
||||||
|
source: cc-os
|
||||||
|
---
|
||||||
|
|
||||||
|
# os-vault write eval — WS2 wording loop (2026-07-07)
|
||||||
|
|
||||||
|
## Results Grid/Threshold
|
||||||
|
|
||||||
|
Baseline (untuned): positives 1/19, negatives 18/18. After 4 wording iterations
|
||||||
|
(TSVs: `plugins/os-vault/eval/results/2026-07-07-wording-*.tsv`; loop log:
|
||||||
|
`cc-os/autoresearch/classic-260707-1253/loop-log.md`):
|
||||||
|
|
||||||
|
| Stage | sonnet | haiku |
|
||||||
|
|---|---|---|
|
||||||
|
| Confirm, full run-set ×3 reps | 18/18 (one N3 "FAIL" is a checker-coarseness boundary case, see below) | 16/18 (P3-L3 flicker 1/3) |
|
||||||
|
| **Reserve (held-out)** ×3 reps | P4 0/3 (trigger 3/3, YAML-colon title), P5 2/3, P6 2/3; negatives 8/9 | P4 2/3, P5 0/3 (no trigger), P6 2/3; negatives 9/9 |
|
||||||
|
|
||||||
|
Reserve overall: positives 8/18 strict PASS but **trigger axis 13/18**; negatives
|
||||||
|
17/18. Zero real-vault breaches after iteration 2; sandbox isolation held.
|
||||||
|
|
||||||
|
## What shipped (vs the baseline note's Proposed Changes)
|
||||||
|
|
||||||
|
1. **Candidate 1 (destination ladder)** — write SKILL.md description rewritten
|
||||||
|
trigger-conditioned + ladder in body; new SessionStart `USAGE_NOTE`
|
||||||
|
additionalContext in `hooks/session_start.py` with mechanical triggers
|
||||||
|
(docs-vs-reality discrepancy → vault; client requirement → vault scope:
|
||||||
|
client; BEFORE-final-reply completeness checkpoint).
|
||||||
|
2. **Vault-location made mechanical** (not in the candidates; iteration-2 fix for
|
||||||
|
real-vault breaches): resolve `${OS_VAULT_PATH:-$HOME/Documents/SecondBrain}`
|
||||||
|
first, never hardcode.
|
||||||
|
3. **CLAUDE.md "Cross-project memory" section** (fixture copy is the
|
||||||
|
real-project adoption template, mirroring os-adr) — required to reach haiku
|
||||||
|
at L2/L3; includes the ADR-vs-vault disambiguation line (haiku's dominant
|
||||||
|
L3 miss was recording the client fact as a repo ADR instead).
|
||||||
|
4. **Candidates 3+4** (update-before-create; type-misfit routing) shipped
|
||||||
|
unscored. **Candidate 2** (query-side triggers) shipped unscored in query
|
||||||
|
SKILL.md description + body.
|
||||||
|
5. **Post-reserve contract fix**: quote YAML scalars containing colons (from the
|
||||||
|
reserve's dominant conformance failure).
|
||||||
|
|
||||||
|
## Transferable Lessons
|
||||||
|
|
||||||
|
- **Each rule lives where its precondition is visible, confirmed again on a new
|
||||||
|
plugin:** hook-note triggers moved sonnet to 9/9 but haiku 0/6 at L2/L3 even
|
||||||
|
with the note verified in its context; the project-CLAUDE.md section fixed
|
||||||
|
haiku L2 instantly. Same mechanism as os-adr Eval B.
|
||||||
|
- **Channel competition generalizes beyond auto-memory:** haiku's L3 miss shape
|
||||||
|
was routing the durable fact to a repo ADR (the os-adr CLAUDE.md section
|
||||||
|
captured the impulse). Destination wording must disambiguate against EVERY
|
||||||
|
competing persistence channel, not just the default one.
|
||||||
|
- **Model-compliance-dependent isolation seams leak:** three reps wrote to the
|
||||||
|
real vault because honoring `OS_VAULT_PATH` required the model to check an
|
||||||
|
env var. Making the resolution a mechanical one-liner in the skill fixed it
|
||||||
|
(and is production-harmless). Eval isolation should not depend on the model
|
||||||
|
following instructions — or the instructions must make compliance mechanical.
|
||||||
|
- **Held-out sets find content-shaped conformance bugs the run-set can't:** the
|
||||||
|
reserve's chartkit fact contains `streaming: true`; models put it unquoted in
|
||||||
|
the YAML title, breaking the frontmatter (4 reps). No run-set fact contained
|
||||||
|
a colon. Fixture facts should deliberately include YAML-hostile strings.
|
||||||
|
- **Generalization-laundering is the real over-trigger shape:** the single
|
||||||
|
negative miss rewrote a project-local quirk as a "general pattern" note.
|
||||||
|
Restraint wording may need an explicit "generalizing a local incident does
|
||||||
|
not make it vault knowledge" line (future, needs fresh scenarios).
|
||||||
|
|
||||||
|
## Validity and Limitations
|
||||||
|
|
||||||
|
- Run-set is training-set (4 iterations tuned against it). The reserve was the
|
||||||
|
one-shot held-out measurement; **it is now contaminated too** — the YAML-quote
|
||||||
|
fix and this analysis derive from its failures. Any future measurement needs
|
||||||
|
new scenarios (third domain).
|
||||||
|
- The N3-sonnet confirm "FAIL" and N5-sonnet reserve FAIL sit on the checker's
|
||||||
|
"any vault write fails a negative" line: N3's write was a genuinely-new
|
||||||
|
conforming Ruby stdlib discovery (instruction-compliant, checker coarser than
|
||||||
|
scenario intent — criterion-redesign question pre-registered: scope negative
|
||||||
|
FAILs to the scenario's tempting-knowledge topic regexes; anchor thresholds
|
||||||
|
independently, not to these grids); N5's was real laundering. Same line, one
|
||||||
|
defensible, one not.
|
||||||
|
- One checker defect found+fixed mid-loop (breach regex matched `2>/dev/null` +
|
||||||
|
read-only vault mention); rescored per the instrument-fix rule, transcripts
|
||||||
|
untouched; two self-test guards added.
|
||||||
|
- Haiku P3/P5 flicker: L3 ~4/6 run-set, L2 0/3 reserve — semantic triggers still
|
||||||
|
under-reach haiku; consistent with os-adr's accepted W3 flicker.
|
||||||
|
|
||||||
|
## Deployment and Evolution
|
||||||
|
|
||||||
|
Shipped wording is live in the plugin (caches refreshed). Real-project adoption:
|
||||||
|
copy the fixture CLAUDE.md "Cross-project memory" section (candidate: emit from
|
||||||
|
onboarding, mirroring os-adr's init/migrate plan). Next signal is production
|
||||||
|
(IRL session audits per [[eval-methodology-irl-feedback-loop]]), not another lab
|
||||||
|
rung — except the pre-registered negative-criterion redesign if the harness is
|
||||||
|
revisited.
|
||||||
Loading…
Reference in New Issue