SecondBrain/os-vault-write-eval-baseline-grid-results.md

---
type: eval-results
title: os-vault write eval — untuned baseline grid (unprompted vault-write discrimination)
summary: At baseline (no trigger wording anywhere in os-vault), positives are 1/19 across both tiers while negatives are 18/18 clean — and the dominant failure shape is not "no persistence" but mis-routing to Claude Code's built-in auto-memory instead of the vault at L1; L2/L3 mostly don't persist at all.
tags:
  - type/eval-results
  - domain/llm-evaluation
  - tool/os-vault
  - tool/claude-code
  - project/cc-os
scope: global
last_updated: 2026-07-07  # added Transferable Lessons + Proposed Changes (WS2 Loop Inputs) sections
date: 2026-07-06
related:
  - eval-methodology-ladder
  - running-autoresearch-skill-evals
  - os-adr-eval-b-wording-experiment-hypotheses
  - os-adr-eval-c-frozen-grid-results
source: cc-os
---

# os-vault write eval — untuned baseline grid (2026-07-06)

## Results Grid/Threshold

**Passing criterion:** positives — unprompted vault write with contract-conforming
frontmatter (axis a: written or offered; axis b: mechanical frontmatter conformance);
negatives — no vault write (the sole fail line). Run-set × {sonnet, haiku} × 3 reps + 1
counted sonnet canary rep on P1 (37 counted reps total, zero harness errors, all rows
verified against expected rep counts).

| Cell | sonnet | haiku | Dominant behavior |
|---|---|---|---|
| P1-L1 explicit cue (Finch 500-cap) | 1/4 (canary only) | 0/3 | Persists EVERY rep — but to auto-memory (`~/.claude/projects/<slug>/memory/`), not the vault |
| P2-L2 no cue (Finch expires_in) | 0/3 | 0/3 | Fixes the code; no persistence anywhere |
| P3-L3 conceptual (Acme convention) | 0/3 | 0/3 | sonnet 1/3 wrote auto-memory; otherwise nothing |
| N1-L1 explicit cue, ephemeral | 3/3 | 3/3 | Status written to project file (`project-write:yes`), vault untouched |
| N2-L2 project-local quirk | 3/3 | 3/3 | Clean fix, no vault write |
| N3-L3 repo-recorded (ADR 0002) | 3/3 | 3/3 | Complies with the ADR, no vault duplicate |

**Summary:** positives 1/19 (the one pass: sonnet canary P1, which wrote a fully conforming
`reference/finch-api-page-size-silent-cap.md` — proof the ceiling exists); negatives 18/18
with zero over-trigger at any level on either tier. The measured gap is real and one-sided:
under-triggering plus destination mis-routing, no restraint problem.

## Measurement Setup

**Hypotheses / Scenarios Tested** — Ambiguity ladder L1 explicit-cue → L2 moderate → L3
conceptual, paired positive/negative per level (run-set: Finch API facts + Acme client
convention vs ephemeral status / project-local quirk / repo-recorded ADR). Reserve set
(chartkit + Globex + SFTP ADR) authored upfront, frozen, never read informally. Harness:
`cc-os/plugins/os-vault/eval/` (README documents scoring); TSV:
`eval/results/2026-07-06-baseline-grid.tsv`.

**Fixture and Sampling** — New `reportgen` Ruby fixture (Finch API reports CLI, 3-ADR
history via os-adr's own CLIs, deliberately plain CLAUDE.md with zero trigger wording) +
isolated sandbox vault (frozen copy of real vault-conventions.md + 4 topically-disjoint
seeded notes). Headless-only `claude -p` per rep with `OS_VAULT_PATH`/`OS_VAULT_SKIP_REBUILD`
isolation (verified: real vault untouched across all 37 reps; a canary transcript shows the
model writing into the sandbox vault path). 3 reps/cell + counted canary.

**Experimental Control (Frozen Surfaces)** — This is the pre-tuning baseline: os-vault
carried NO trigger-conditioned wording anywhere (WS2 defect 2 deliberately left in place).
Checker, fixtures, scenarios, judge rubric frozen. The only pre-baseline os-vault changes
were the WS2-sanctioned contract fix (write SKILL.md defers to vault-conventions.md) and the
isolation seam (OS_VAULT_PATH override + "Vault location" skill section).

## Validity and Limitations

**How to Interpret These Results** — Held-out baseline, nothing tuned against it yet; the
run-set remains the measurement set until a wording loop starts. The headline finding is the
**auto-memory competition channel**: on every L1 rep (both tiers) the model recognized the
knowledge as durable and persisted it — into Claude Code's built-in per-project auto-memory,
narrating it as "cross-repo memory" — so the trigger-recognition capability exists at L1 and
the failure is destination routing. (The user's global `~/.claude/CLAUDE.md` memory-routing
section loads in these sandbox sessions exactly as in production, so this competition is
production-faithful, not an artifact.) At L2/L3 the model mostly doesn't persist at all —
that part is a genuine trigger gap. P1 sonnet flipped between canary (vault) and grid
(auto-memory ×3): treat single-rep vault hits as variance, not capability proof.

**Weaknesses of This Eval (Its Ladder Level)** — Single fixture/domain for the run-set;
offer-detection judge never fired live (every positive either wrote somewhere or stayed
silent — the judge path is only self-test-validated so far); auto-memory routing is scored
only via the informational reasons trail, not a dedicated axis; no measurement of the
`/os-vault:write` skill when explicitly invoked (that's an Eval-A-shaped question, out of
scope here).

## Transferable Lessons

- **Persistence evals need a destination axis, not just a trigger axis.** When the harness (or any host tool) has a competing built-in persistence channel, "did the model save it?" and "did it save it to the right place?" are separate failure modes — this grid would have read as a flat trigger gap without the reasons-trail showing every L1 rep wrote to auto-memory.
- **"Remember this" wording loses to the default channel.** Trigger wording that names the action but not the destination drains to whatever persistence path is cheapest/built-in. Candidate wording must name the destination explicitly.
- **Count canary reps.** The one vault-conforming pass was a canary; had it been excluded, the ceiling-exists evidence would have been lost (and including it avoids peeking bias in the denominator).
- **A clean-negative baseline converts to a hard regression gate.** 18/18 negatives means the tuning loop gets a free, unambiguous stop condition: any negative FAIL during wording iteration is a hard stop, no judgment needed.

**Promotion check:** the destination-axis lesson (first bullet) is a new eval-design confound not yet covered by [[eval-methodology-ladder]] — promote it there if a second eval hits a competing-channel shape; nothing else recurring yet. "Count canary reps" is already practiced in [[running-autoresearch-skill-evals]] discipline.

## Deployment and Evolution

**Good-Enough Gate** — Not applicable yet; this grid establishes the gap the WS2 wording
loop must close. Negatives at 18/18 mean restraint needs no work — wording iteration should
optimize trigger + routing only, and any future negative regression is a hard stop.

**Proposed Changes (WS2 Loop Inputs)** — concrete candidate wording drafted 2026-07-07,
NOT yet applied to any os-vault surface (applying it starts the wording loop and converts the
run-set to training-set; measurement then moves to the frozen reserve). Diff these against
what actually ships.

*Candidate 1 — destination ladder* (targets: write SKILL.md description/body + an os-vault
SessionStart usage note + candidate trigger-phrased CLAUDE.md section; addresses both measured
gaps — L1 mis-routing and L2/L3 under-trigger):

> **Where knowledge goes — ask in order:**
> 1. Does the repo already record it (code, ADR, CLAUDE.md, git history)? → write it there or
>    nowhere; never duplicate into memory.
> 2. Would this change how you act in a **different** repo, next year? → the **SecondBrain
>    vault** via `/os-vault:write` — NOT auto-memory. Facts about the world: tool behavior,
>    client conventions, API quirks, methodology that worked. The repo is where you *learned*
>    it, not what it's *about*.
> 3. Only useful in future sessions of **this** repo (where things live here, what we tried
>    here)? → auto-memory.
> 4. Task status, session state, in-flight progress → nothing.
>
> Tiebreaker: what is the fact **about**? About a tool/client/API → vault. About this repo →
> auto-memory. When a durable fact surfaces mid-task — even with no one saying "remember" —
> run the ladder before moving on.

*Candidate 2 — query-side triggers* (targets: query SKILL.md description + SessionStart usage
note; NO eval coverage yet — a query-side eval is a separate future harness, don't score it
against this one):

> - Before designing or setting up an eval → `/os-vault:query` for `type/eval-results` +
>   `domain/llm-evaluation` (and the project facet): what was already run, what the
>   hypotheses/results were, what mistakes not to repeat.
> - Before first use this session of a tool you have vault notes for → query `tool/<name>`.
> - Before starting work involving a client → query `client/<name>`.
> - Don't re-query a facet already queried this session — the note content is in context.

*Candidate 3 — update-vs-create rule* (target: write SKILL.md procedure):

> Write is query-first: before creating a note, query for existing notes on the same subject.
> New fact contradicts/extends/refines an existing note → update it and bump `last_updated`.
> Genuinely new subject → new note.

*Candidate 4 — type-misfit routing* (target: write SKILL.md; conservative, human-gated
taxonomy growth):

> Content that fits no existing note type: do NOT invent a type or force a bad fit silently.
> Write it under the closest type with a visible `## Type-fit note` line saying what didn't
> fit, or — if the shape is clearly recurring — propose a new type via
> `/os-vault:design-template`. Misfits get reviewed in `/os-vault:reorganize`; three misfits
> of the same shape justify a new type.

**Hardening Path / Next Measurement** — Per `ws2-os-vault-write-eval.md` step 4: an
`/autoresearch` wording loop (checker/fixtures/scenarios/rubric frozen; surfaces = write
SKILL.md description/body, an os-vault SessionStart usage note, possibly a trigger-phrased
CLAUDE.md section mirroring os-adr's). Two distinct wording jobs, per the axis-level
reading: (1) trigger at L2/L3, (2) route to the VAULT over the harness's built-in
auto-memory at L1 — wording that only says "remember this" will keep losing to auto-memory;
it must name the vault as the destination for evergreen cross-project knowledge. The moment
the loop starts, this run-set is training-set; the frozen reserve
(chartkit/Globex/delivery domain) becomes the measurement set.

## Related

- [[eval-methodology-ladder]] — ladder design + reserve discipline this harness follows
- [[running-autoresearch-skill-evals]] — loop discipline for the wording iteration ahead
- [[os-adr-eval-c-frozen-grid-results]] — the sibling discrimination eval whose shape this reuses