SecondBrain/os-vault-write-eval-wording-loop-results.md

---
type: eval-results
title: os-vault write eval — WS2 wording loop (4 iterations) + held-out reserve grid
summary: Four wording iterations took unprompted vault-write positives from 1/19 (baseline) to run-set confirm sonnet 18/18 / haiku 16/18, and held-out reserve 25/36 overall with trigger-axis 13/18 — the residual gaps are YAML-colon frontmatter breakage, haiku L2 no-trigger, and one generalization-laundering over-trigger; the reserve is now contaminated for re-measurement.
tags:
  - type/eval-results
  - domain/llm-evaluation
  - tool/os-vault
  - tool/claude-code
  - project/cc-os
scope: global
last_updated: 2026-07-07
date: 2026-07-07
related:
  - os-vault-write-eval-baseline-grid-results
  - running-autoresearch-skill-evals
  - os-adr-eval-b-wording-experiment-hypotheses
  - eval-methodology-ladder
source: cc-os
---

# os-vault write eval — WS2 wording loop (2026-07-07)

## Results Grid/Threshold

Baseline (untuned): positives 1/19, negatives 18/18. After 4 wording iterations
(TSVs: `plugins/os-vault/eval/results/2026-07-07-wording-*.tsv`; loop log:
`cc-os/autoresearch/classic-260707-1253/loop-log.md`):

| Stage | sonnet | haiku |
|---|---|---|
| Confirm, full run-set ×3 reps | 18/18 (one N3 "FAIL" is a checker-coarseness boundary case, see below) | 16/18 (P3-L3 flicker 1/3) |
| **Reserve (held-out)** ×3 reps | P4 0/3 (trigger 3/3, YAML-colon title), P5 2/3, P6 2/3; negatives 8/9 | P4 2/3, P5 0/3 (no trigger), P6 2/3; negatives 9/9 |

Reserve overall: positives 8/18 strict PASS but **trigger axis 13/18**; negatives
17/18. Zero real-vault breaches after iteration 2; sandbox isolation held.

## What shipped (vs the baseline note's Proposed Changes)

1. **Candidate 1 (destination ladder)** — write SKILL.md description rewritten
   trigger-conditioned + ladder in body; new SessionStart `USAGE_NOTE`
   additionalContext in `hooks/session_start.py` with mechanical triggers
   (docs-vs-reality discrepancy → vault; client requirement → vault scope:
   client; BEFORE-final-reply completeness checkpoint).
2. **Vault-location made mechanical** (not in the candidates; iteration-2 fix for
   real-vault breaches): resolve `${OS_VAULT_PATH:-$HOME/Documents/SecondBrain}`
   first, never hardcode.
3. **CLAUDE.md "Cross-project memory" section** (fixture copy is the
   real-project adoption template, mirroring os-adr) — required to reach haiku
   at L2/L3; includes the ADR-vs-vault disambiguation line (haiku's dominant
   L3 miss was recording the client fact as a repo ADR instead).
4. **Candidates 3+4** (update-before-create; type-misfit routing) shipped
   unscored. **Candidate 2** (query-side triggers) shipped unscored in query
   SKILL.md description + body.
5. **Post-reserve contract fix**: quote YAML scalars containing colons (from the
   reserve's dominant conformance failure).

## Transferable Lessons

- **Each rule lives where its precondition is visible, confirmed again on a new
  plugin:** hook-note triggers moved sonnet to 9/9 but haiku 0/6 at L2/L3 even
  with the note verified in its context; the project-CLAUDE.md section fixed
  haiku L2 instantly. Same mechanism as os-adr Eval B.
- **Channel competition generalizes beyond auto-memory:** haiku's L3 miss shape
  was routing the durable fact to a repo ADR (the os-adr CLAUDE.md section
  captured the impulse). Destination wording must disambiguate against EVERY
  competing persistence channel, not just the default one.
- **Model-compliance-dependent isolation seams leak:** three reps wrote to the
  real vault because honoring `OS_VAULT_PATH` required the model to check an
  env var. Making the resolution a mechanical one-liner in the skill fixed it
  (and is production-harmless). Eval isolation should not depend on the model
  following instructions — or the instructions must make compliance mechanical.
- **Held-out sets find content-shaped conformance bugs the run-set can't:** the
  reserve's chartkit fact contains `streaming: true`; models put it unquoted in
  the YAML title, breaking the frontmatter (4 reps). No run-set fact contained
  a colon. Fixture facts should deliberately include YAML-hostile strings.
- **Generalization-laundering is the real over-trigger shape:** the single
  negative miss rewrote a project-local quirk as a "general pattern" note.
  Restraint wording may need an explicit "generalizing a local incident does
  not make it vault knowledge" line (future, needs fresh scenarios).

## Validity and Limitations

- Run-set is training-set (4 iterations tuned against it). The reserve was the
  one-shot held-out measurement; **it is now contaminated too** — the YAML-quote
  fix and this analysis derive from its failures. Any future measurement needs
  new scenarios (third domain).
- The N3-sonnet confirm "FAIL" and N5-sonnet reserve FAIL sit on the checker's
  "any vault write fails a negative" line: N3's write was a genuinely-new
  conforming Ruby stdlib discovery (instruction-compliant, checker coarser than
  scenario intent — criterion-redesign question pre-registered: scope negative
  FAILs to the scenario's tempting-knowledge topic regexes; anchor thresholds
  independently, not to these grids); N5's was real laundering. Same line, one
  defensible, one not.
- One checker defect found+fixed mid-loop (breach regex matched `2>/dev/null` +
  read-only vault mention); rescored per the instrument-fix rule, transcripts
  untouched; two self-test guards added.
- Haiku P3/P5 flicker: L3 ~4/6 run-set, L2 0/3 reserve — semantic triggers still
  under-reach haiku; consistent with os-adr's accepted W3 flicker.

## Deployment and Evolution

Shipped wording is live in the plugin (caches refreshed). Real-project adoption:
copy the fixture CLAUDE.md "Cross-project memory" section (candidate: emit from
onboarding, mirroring os-adr's init/migrate plan). Next signal is production
(IRL session audits per [[eval-methodology-irl-feedback-loop]]), not another lab
rung — except the pre-registered negative-criterion redesign if the harness is
revisited.