cc-os/docs/plans/ws2-os-vault-write-eval.md

# WS2 — os-vault "when and how to write" eval

_Created: 2026-07-06. Status: **steps 1–3 done 2026-07-06** — quick wins shipped (contract
fix + cc-os hub note), harness built at `plugins/os-vault/eval/` (12 scenarios authored
directly, fixture + Ruby checker via sonnet subagents, self-test 21/21, isolation via new
`OS_VAULT_PATH`/`OS_VAULT_SKIP_REBUILD` seam), untuned baseline grid run (37 counted reps:
positives 1/19, negatives 18/18; headline finding: at L1 the model persists to Claude Code's
built-in auto-memory instead of the vault — routing, not just triggering, is the wording
target). Results: `eval/results/2026-07-06-baseline-grid.tsv` + vault note
[[os-vault-write-eval-baseline-grid-results]]. Step 4 (wording loop) not started — the
run-set is still uncontaminated. See `2026-07-06-plugin-evals-overview.md`._

## Goal

Make os-vault reliably know WHEN to write a vault note unprompted and HOW to write it
correctly (frontmatter contract), measured by a held-out discrimination eval — applying the
full os-adr first-batch playbook.

## Known defects found 2026-07-06 (fix before baselining)

1. **`skills/write/SKILL.md` contradicts the reconciled vault-conventions.md.** The skill
   still mandates `scope/global` as a TAG and a type list of
   `procedure|reference|log|hub|concept|decision`; vault-conventions (Phase 1 SB content
   plan, 2026-06-30) made `scope:` a frontmatter FIELD and the live vault uses types like
   `howto` and `eval-results`. Fix: make vault-conventions.md the single source of truth —
   the skill summarizes + points, never duplicates the schema. Then `bin/refresh-plugins`.
2. **No trigger-conditioned wording anywhere in os-vault.** The core Eval B lesson
   (when→then phrasing; mechanical triggers for lower tiers; each rule lives where its
   precondition is visible) was never applied. Nothing tells the model when to invoke
   `/os-vault:write` unprompted. Do NOT fix this before the baseline grid — it is the
   tuning surface the eval measures.
3. **cc-os has no project hub note** despite hub notes being the connectivity mechanism
   (ADR-014). Quick win: author it now (one note). AUTOMATIC hub creation is designed into
   `onboard-project` + the WS3 status check — not hacked in here.

Quick wins = items 1 and 3. Both are small; do directly or via one haiku subagent each.

## Behavior under test

Unprompted vault-write discrimination during a normal working session:

- **Positives:** session surfaces evergreen cross-project knowledge (tool/API behavior
  discovered, client-specific fact, a convention established with the user) → model writes
  or offers a vault note with contract-conforming frontmatter (summary at creation, scope
  field, type/ + ≥1 facet tag).
- **Negatives (equally tempting):** ephemeral material — in-progress task state,
  project-local debugging detail, decisions already recorded in repo docs/ADRs → no vault
  write. Over-trigger fail line: writing ephemera to the vault (or duplicating
  repo-recorded knowledge).

Ambiguity ladder: L1 explicit cue ("worth remembering") → L2 moderate → L3 conceptual
(knowledge emerges implicitly from debugging). Paired positive/negative per level.

## Harness (Eval B/C shape)

- Location: `plugins/os-vault/eval/`. Run-set + frozen reserve authored upfront (2×), in
  different knowledge domains. **Task blocks held-out — never run informally; reserve
  never read informally.**
- **Headless-only** (`claude -p`, cwd = sandbox fixture project, real SessionStart hook
  fires). In-session subagents are invalid for unprompted-behavior measurement.
- Sandbox must use an ISOLATED vault path (env-pointed test vault), never the real
  `~/Documents/SecondBrain` — the checker inspects the sandbox vault for written notes.
  Verify the plugin's vault path is env-overridable (config.yaml); if not, that's a small
  pre-harness plugin change.
- Checker: Ruby (Sandi Metz style), deterministic-first. Axis (a): did it write/offer a
  vault note (mechanical from transcript tool_use + sandbox vault diff). Axis (b):
  frontmatter contract conformance — almost fully mechanical (summary present, scope
  field, required tag groups); narrow frozen haiku judge only for "offered but did not
  write" phrasing, stubbable via env var. Negatives: sole fail line = unneeded vault
  write; consultation informational. TSV mode for grids.

## Discipline gates (all mandatory before the first grid)

1. Conformance dry-run on paper: would a perfectly write-skill-compliant model pass every
   cell? Would an always-write model pass positives but fail negatives?
2. Model-free self-test including at least one fabricated SHIPPED-INSTRUCTION-COMPLIANT
   transcript (not just the designer's imagined ideal).
3. Canary cell first (one cell, hand-verify TSV vs raw transcript, count the result).
4. `bin/refresh-plugins` before every grid; check TSV row counts == expected reps
   (pipefail/tee gotcha); rescore-don't-discard on instrument fixes.

## Sequence

1. Quick wins (contract fix + hub note) → refresh caches.
2. Build harness (fixture project + isolated vault + 6-ADR-equivalent seeded vault history
   for negatives to cite): fixture generation → sonnet subagent; checker → sonnet subagent
   (Ruby); scenario authoring + conformance dry-run → ORCHESTRATOR DIRECTLY (Eval C's
   defects came from exactly this step; do not delegate it).
3. **Baseline grid untuned** (post-bugfix, pre-trigger-wording): run-set × {sonnet, haiku}
   × 3 reps. This measures the actual gap.
4. If gapped: `/autoresearch` wording loop per the howto note (checker/fixtures/scenarios
   frozen; only wording moves; reduced inner grid; full grid to confirm). Expected surfaces:
   session_start hook note, write SKILL.md description + body, possibly a CLAUDE.md
   trigger-phrased section template (mirror os-adr's).
5. Results → vault `eval-results` note (template `_templates/eval-results.md`); update
   cc-os CLAUDE.md pointers.

## Models

Sonnet for fixture/checker construction and any judging; haiku for mechanical edits;
orchestrator authors scenarios and acts as final judge over all grid claims.