From 2bbc528246224a43a06488a77073288b2090b581 Mon Sep 17 00:00:00 2001 From: jared Date: Tue, 7 Jul 2026 14:01:15 -0400 Subject: [PATCH] eval-results: os-vault WS2 wording loop + reserve grid --- os-vault-write-eval-wording-loop-results.md | 108 ++++++++++++++++++++ 1 file changed, 108 insertions(+) create mode 100644 os-vault-write-eval-wording-loop-results.md diff --git a/os-vault-write-eval-wording-loop-results.md b/os-vault-write-eval-wording-loop-results.md new file mode 100644 index 0000000..12ffe75 --- /dev/null +++ b/os-vault-write-eval-wording-loop-results.md @@ -0,0 +1,108 @@ +--- +type: eval-results +title: os-vault write eval — WS2 wording loop (4 iterations) + held-out reserve grid +summary: Four wording iterations took unprompted vault-write positives from 1/19 (baseline) to run-set confirm sonnet 18/18 / haiku 16/18, and held-out reserve 25/36 overall with trigger-axis 13/18 — the residual gaps are YAML-colon frontmatter breakage, haiku L2 no-trigger, and one generalization-laundering over-trigger; the reserve is now contaminated for re-measurement. +tags: + - type/eval-results + - domain/llm-evaluation + - tool/os-vault + - tool/claude-code + - project/cc-os +scope: global +last_updated: 2026-07-07 +date: 2026-07-07 +related: + - os-vault-write-eval-baseline-grid-results + - running-autoresearch-skill-evals + - os-adr-eval-b-wording-experiment-hypotheses + - eval-methodology-ladder +source: cc-os +--- + +# os-vault write eval — WS2 wording loop (2026-07-07) + +## Results Grid/Threshold + +Baseline (untuned): positives 1/19, negatives 18/18. After 4 wording iterations +(TSVs: `plugins/os-vault/eval/results/2026-07-07-wording-*.tsv`; loop log: +`cc-os/autoresearch/classic-260707-1253/loop-log.md`): + +| Stage | sonnet | haiku | +|---|---|---| +| Confirm, full run-set ×3 reps | 18/18 (one N3 "FAIL" is a checker-coarseness boundary case, see below) | 16/18 (P3-L3 flicker 1/3) | +| **Reserve (held-out)** ×3 reps | P4 0/3 (trigger 3/3, YAML-colon title), P5 2/3, P6 2/3; negatives 8/9 | P4 2/3, P5 0/3 (no trigger), P6 2/3; negatives 9/9 | + +Reserve overall: positives 8/18 strict PASS but **trigger axis 13/18**; negatives +17/18. Zero real-vault breaches after iteration 2; sandbox isolation held. + +## What shipped (vs the baseline note's Proposed Changes) + +1. **Candidate 1 (destination ladder)** — write SKILL.md description rewritten + trigger-conditioned + ladder in body; new SessionStart `USAGE_NOTE` + additionalContext in `hooks/session_start.py` with mechanical triggers + (docs-vs-reality discrepancy → vault; client requirement → vault scope: + client; BEFORE-final-reply completeness checkpoint). +2. **Vault-location made mechanical** (not in the candidates; iteration-2 fix for + real-vault breaches): resolve `${OS_VAULT_PATH:-$HOME/Documents/SecondBrain}` + first, never hardcode. +3. **CLAUDE.md "Cross-project memory" section** (fixture copy is the + real-project adoption template, mirroring os-adr) — required to reach haiku + at L2/L3; includes the ADR-vs-vault disambiguation line (haiku's dominant + L3 miss was recording the client fact as a repo ADR instead). +4. **Candidates 3+4** (update-before-create; type-misfit routing) shipped + unscored. **Candidate 2** (query-side triggers) shipped unscored in query + SKILL.md description + body. +5. **Post-reserve contract fix**: quote YAML scalars containing colons (from the + reserve's dominant conformance failure). + +## Transferable Lessons + +- **Each rule lives where its precondition is visible, confirmed again on a new + plugin:** hook-note triggers moved sonnet to 9/9 but haiku 0/6 at L2/L3 even + with the note verified in its context; the project-CLAUDE.md section fixed + haiku L2 instantly. Same mechanism as os-adr Eval B. +- **Channel competition generalizes beyond auto-memory:** haiku's L3 miss shape + was routing the durable fact to a repo ADR (the os-adr CLAUDE.md section + captured the impulse). Destination wording must disambiguate against EVERY + competing persistence channel, not just the default one. +- **Model-compliance-dependent isolation seams leak:** three reps wrote to the + real vault because honoring `OS_VAULT_PATH` required the model to check an + env var. Making the resolution a mechanical one-liner in the skill fixed it + (and is production-harmless). Eval isolation should not depend on the model + following instructions — or the instructions must make compliance mechanical. +- **Held-out sets find content-shaped conformance bugs the run-set can't:** the + reserve's chartkit fact contains `streaming: true`; models put it unquoted in + the YAML title, breaking the frontmatter (4 reps). No run-set fact contained + a colon. Fixture facts should deliberately include YAML-hostile strings. +- **Generalization-laundering is the real over-trigger shape:** the single + negative miss rewrote a project-local quirk as a "general pattern" note. + Restraint wording may need an explicit "generalizing a local incident does + not make it vault knowledge" line (future, needs fresh scenarios). + +## Validity and Limitations + +- Run-set is training-set (4 iterations tuned against it). The reserve was the + one-shot held-out measurement; **it is now contaminated too** — the YAML-quote + fix and this analysis derive from its failures. Any future measurement needs + new scenarios (third domain). +- The N3-sonnet confirm "FAIL" and N5-sonnet reserve FAIL sit on the checker's + "any vault write fails a negative" line: N3's write was a genuinely-new + conforming Ruby stdlib discovery (instruction-compliant, checker coarser than + scenario intent — criterion-redesign question pre-registered: scope negative + FAILs to the scenario's tempting-knowledge topic regexes; anchor thresholds + independently, not to these grids); N5's was real laundering. Same line, one + defensible, one not. +- One checker defect found+fixed mid-loop (breach regex matched `2>/dev/null` + + read-only vault mention); rescored per the instrument-fix rule, transcripts + untouched; two self-test guards added. +- Haiku P3/P5 flicker: L3 ~4/6 run-set, L2 0/3 reserve — semantic triggers still + under-reach haiku; consistent with os-adr's accepted W3 flicker. + +## Deployment and Evolution + +Shipped wording is live in the plugin (caches refreshed). Real-project adoption: +copy the fixture CLAUDE.md "Cross-project memory" section (candidate: emit from +onboarding, mirroring os-adr's init/migrate plan). Next signal is production +(IRL session audits per [[eval-methodology-irl-feedback-loop]]), not another lab +rung — except the pre-registered negative-criterion redesign if the harness is +revisited.