cc-os/docs/plans/ws2-os-vault-write-eval.md

WS2 — os-vault "when and how to write" eval

Created: 2026-07-06. Status: steps 1–3 done 2026-07-06 — quick wins shipped (contract fix + cc-os hub note), harness built at plugins/os-vault/eval/ (12 scenarios authored directly, fixture + Ruby checker via sonnet subagents, self-test 21/21, isolation via new OS_VAULT_PATH/OS_VAULT_SKIP_REBUILD seam), untuned baseline grid run (37 counted reps: positives 1/19, negatives 18/18; headline finding: at L1 the model persists to Claude Code's built-in auto-memory instead of the vault — routing, not just triggering, is the wording target). Results: eval/results/2026-07-06-baseline-grid.tsv + vault note os-vault-write-eval-baseline-grid-results. Step 4 (wording loop) not started — the run-set is still uncontaminated. See 2026-07-06-plugin-evals-overview.md.

Goal

Make os-vault reliably know WHEN to write a vault note unprompted and HOW to write it correctly (frontmatter contract), measured by a held-out discrimination eval — applying the full os-adr first-batch playbook.

Known defects found 2026-07-06 (fix before baselining)

1. skills/write/SKILL.md contradicts the reconciled vault-conventions.md. The skill still mandates scope/global as a TAG and a type list of procedure|reference|log|hub|concept|decision; vault-conventions (Phase 1 SB content plan, 2026-06-30) made scope: a frontmatter FIELD and the live vault uses types like howto and eval-results. Fix: make vault-conventions.md the single source of truth — the skill summarizes + points, never duplicates the schema. Then bin/refresh-plugins.
2. No trigger-conditioned wording anywhere in os-vault. The core Eval B lesson (when→then phrasing; mechanical triggers for lower tiers; each rule lives where its precondition is visible) was never applied. Nothing tells the model when to invoke /os-vault:write unprompted. Do NOT fix this before the baseline grid — it is the tuning surface the eval measures.
3. cc-os has no project hub note despite hub notes being the connectivity mechanism (ADR-014). Quick win: author it now (one note). AUTOMATIC hub creation is designed into onboard-project + the WS3 status check — not hacked in here.

Quick wins = items 1 and 3. Both are small; do directly or via one haiku subagent each.

Behavior under test

Unprompted vault-write discrimination during a normal working session:

• Positives: session surfaces evergreen cross-project knowledge (tool/API behavior discovered, client-specific fact, a convention established with the user) → model writes or offers a vault note with contract-conforming frontmatter (summary at creation, scope field, type/ + ≥1 facet tag).
• Negatives (equally tempting): ephemeral material — in-progress task state, project-local debugging detail, decisions already recorded in repo docs/ADRs → no vault write. Over-trigger fail line: writing ephemera to the vault (or duplicating repo-recorded knowledge).

Ambiguity ladder: L1 explicit cue ("worth remembering") → L2 moderate → L3 conceptual (knowledge emerges implicitly from debugging). Paired positive/negative per level.

Harness (Eval B/C shape)

• Location: plugins/os-vault/eval/. Run-set + frozen reserve authored upfront (2×), in different knowledge domains. Task blocks held-out — never run informally; reserve never read informally.
• Headless-only (claude -p, cwd = sandbox fixture project, real SessionStart hook fires). In-session subagents are invalid for unprompted-behavior measurement.
• Sandbox must use an ISOLATED vault path (env-pointed test vault), never the real ~/Documents/SecondBrain — the checker inspects the sandbox vault for written notes. Verify the plugin's vault path is env-overridable (config.yaml); if not, that's a small pre-harness plugin change.
• Checker: Ruby (Sandi Metz style), deterministic-first. Axis (a): did it write/offer a vault note (mechanical from transcript tool_use + sandbox vault diff). Axis (b): frontmatter contract conformance — almost fully mechanical (summary present, scope field, required tag groups); narrow frozen haiku judge only for "offered but did not write" phrasing, stubbable via env var. Negatives: sole fail line = unneeded vault write; consultation informational. TSV mode for grids.

Discipline gates (all mandatory before the first grid)

1. Conformance dry-run on paper: would a perfectly write-skill-compliant model pass every cell? Would an always-write model pass positives but fail negatives?
2. Model-free self-test including at least one fabricated SHIPPED-INSTRUCTION-COMPLIANT transcript (not just the designer's imagined ideal).
3. Canary cell first (one cell, hand-verify TSV vs raw transcript, count the result).
4. bin/refresh-plugins before every grid; check TSV row counts == expected reps (pipefail/tee gotcha); rescore-don't-discard on instrument fixes.

Sequence

1. Quick wins (contract fix + hub note) → refresh caches.
2. Build harness (fixture project + isolated vault + 6-ADR-equivalent seeded vault history for negatives to cite): fixture generation → sonnet subagent; checker → sonnet subagent (Ruby); scenario authoring + conformance dry-run → ORCHESTRATOR DIRECTLY (Eval C's defects came from exactly this step; do not delegate it).
3. Baseline grid untuned (post-bugfix, pre-trigger-wording): run-set × {sonnet, haiku} × 3 reps. This measures the actual gap.
4. If gapped: /autoresearch wording loop per the howto note (checker/fixtures/scenarios frozen; only wording moves; reduced inner grid; full grid to confirm). Expected surfaces: session_start hook note, write SKILL.md description + body, possibly a CLAUDE.md trigger-phrased section template (mirror os-adr's).
5. Results → vault eval-results note (template _templates/eval-results.md); update cc-os CLAUDE.md pointers.

Models

Sonnet for fixture/checker construction and any judging; haiku for mechanical edits; orchestrator authors scenarios and acts as final judge over all grid claims.