SecondBrain/os-vault-write-eval-wording-loop-results.md

type	title	summary	tags	scope	last_updated	date	related	source
eval-results	os-vault write eval — WS2 wording loop (4 iterations) + held-out reserve grid	Four wording iterations took unprompted vault-write positives from 1/19 (baseline) to run-set confirm sonnet 18/18 / haiku 16/18, and held-out reserve 25/36 overall with trigger-axis 13/18 — the residual gaps are YAML-colon frontmatter breakage, haiku L2 no-trigger, and one generalization-laundering over-trigger; the reserve is now contaminated for re-measurement.	type/eval-results domain/llm-evaluation tool/os-vault tool/claude-code project/cc-os	global	2026-07-07	2026-07-07	os-vault-write-eval-baseline-grid-results running-autoresearch-skill-evals os-adr-eval-b-wording-experiment-hypotheses eval-methodology-ladder	cc-os

os-vault write eval — WS2 wording loop (2026-07-07)

Results Grid/Threshold

Baseline (untuned): positives 1/19, negatives 18/18. After 4 wording iterations (TSVs: plugins/os-vault/eval/results/2026-07-07-wording-*.tsv; loop log: cc-os/autoresearch/classic-260707-1253/loop-log.md):

Stage	sonnet	haiku
Confirm, full run-set ×3 reps	18/18 (one N3 "FAIL" is a checker-coarseness boundary case, see below)	16/18 (P3-L3 flicker 1/3)
Reserve (held-out) ×3 reps	P4 0/3 (trigger 3/3, YAML-colon title), P5 2/3, P6 2/3; negatives 8/9	P4 2/3, P5 0/3 (no trigger), P6 2/3; negatives 9/9

Reserve overall: positives 8/18 strict PASS but trigger axis 13/18; negatives 17/18. Zero real-vault breaches after iteration 2; sandbox isolation held.

What shipped (vs the baseline note's Proposed Changes)

1. Candidate 1 (destination ladder) — write SKILL.md description rewritten trigger-conditioned + ladder in body; new SessionStart USAGE_NOTE additionalContext in hooks/session_start.py with mechanical triggers (docs-vs-reality discrepancy → vault; client requirement → vault scope: client; BEFORE-final-reply completeness checkpoint).
2. Vault-location made mechanical (not in the candidates; iteration-2 fix for real-vault breaches): resolve ${OS_VAULT_PATH:-$HOME/Documents/SecondBrain} first, never hardcode.
3. CLAUDE.md "Cross-project memory" section (fixture copy is the real-project adoption template, mirroring os-adr) — required to reach haiku at L2/L3; includes the ADR-vs-vault disambiguation line (haiku's dominant L3 miss was recording the client fact as a repo ADR instead).
4. Candidates 3+4 (update-before-create; type-misfit routing) shipped unscored. Candidate 2 (query-side triggers) shipped unscored in query SKILL.md description + body.
5. Post-reserve contract fix: quote YAML scalars containing colons (from the reserve's dominant conformance failure).

Transferable Lessons

• Each rule lives where its precondition is visible, confirmed again on a new plugin: hook-note triggers moved sonnet to 9/9 but haiku 0/6 at L2/L3 even with the note verified in its context; the project-CLAUDE.md section fixed haiku L2 instantly. Same mechanism as os-adr Eval B.
• Channel competition generalizes beyond auto-memory: haiku's L3 miss shape was routing the durable fact to a repo ADR (the os-adr CLAUDE.md section captured the impulse). Destination wording must disambiguate against EVERY competing persistence channel, not just the default one.
• Model-compliance-dependent isolation seams leak: three reps wrote to the real vault because honoring OS_VAULT_PATH required the model to check an env var. Making the resolution a mechanical one-liner in the skill fixed it (and is production-harmless). Eval isolation should not depend on the model following instructions — or the instructions must make compliance mechanical.
• Held-out sets find content-shaped conformance bugs the run-set can't: the reserve's chartkit fact contains streaming: true; models put it unquoted in the YAML title, breaking the frontmatter (4 reps). No run-set fact contained a colon. Fixture facts should deliberately include YAML-hostile strings.
• Generalization-laundering is the real over-trigger shape: the single negative miss rewrote a project-local quirk as a "general pattern" note. Restraint wording may need an explicit "generalizing a local incident does not make it vault knowledge" line (future, needs fresh scenarios).

Validity and Limitations

• Run-set is training-set (4 iterations tuned against it). The reserve was the one-shot held-out measurement; it is now contaminated too — the YAML-quote fix and this analysis derive from its failures. Any future measurement needs new scenarios (third domain).
• The N3-sonnet confirm "FAIL" and N5-sonnet reserve FAIL sit on the checker's "any vault write fails a negative" line: N3's write was a genuinely-new conforming Ruby stdlib discovery (instruction-compliant, checker coarser than scenario intent — criterion-redesign question pre-registered: scope negative FAILs to the scenario's tempting-knowledge topic regexes; anchor thresholds independently, not to these grids); N5's was real laundering. Same line, one defensible, one not.
• One checker defect found+fixed mid-loop (breach regex matched 2>/dev/null + read-only vault mention); rescored per the instrument-fix rule, transcripts untouched; two self-test guards added.
• Haiku P3/P5 flicker: L3 ~4/6 run-set, L2 0/3 reserve — semantic triggers still under-reach haiku; consistent with os-adr's accepted W3 flicker.

Deployment and Evolution

Shipped wording is live in the plugin (caches refreshed). Real-project adoption: copy the fixture CLAUDE.md "Cross-project memory" section (candidate: emit from onboarding, mirroring os-adr's init/migrate plan). Next signal is production (IRL session audits per eval-methodology-irl-feedback-loop), not another lab rung — except the pre-registered negative-criterion redesign if the harness is revisited.