cc-os/plugins/os-vault/eval

os-vault write-behavior eval — unprompted vault-write discrimination (held-out)

Last updated: 2026-07-06 — harness built (WS2); baseline grid NOT yet run.

Measures whether the model unprompted (a) writes evergreen cross-project knowledge to the SecondBrain vault when it surfaces during normal work, and (b) writes it with contract-conforming frontmatter — and equally, whether it correctly does NOT write ephemera, project-local detail, or repo-recorded decisions to the vault. Applies the os-adr Eval B/C playbook (ambiguity ladder, paired positives/negatives, run-set + frozen reserve). Plan: docs/plans/ws2-os-vault-write-eval.md.

HELD-OUT — read this first. The ## Task blocks in scenarios/*.md and scenarios-reserve/*.md must never be pasted into an interactive session or informally "tried out". The only permitted non-grid execution is bin/self-test, which fabricates transcripts and never uses the Task blocks. The reserve set is additionally never read informally — it becomes the measurement set the moment wording is tuned against the run-set.

What the baseline measures

At baseline the shipped os-vault wording has no trigger-conditioned "when to write" phrasing anywhere (WS2 defect 2, deliberately left unfixed until after the baseline). Expected shape: positives likely FAIL axis (a) at both tiers (that gap IS the measurement); negatives should PASS (nothing currently pushes the model to write). The post-baseline wording loop then tunes trigger wording (skill description/body, hook note, optionally a CLAUDE.md section template mirroring os-adr's) against the run-set only.

Design: ambiguity ladder with paired positive/negative

Level	Positive (should write/offer)	Negative (should NOT write)
L1 explicit	P1: durable Finch API fact + explicit "worth remembering" cue	N1: explicit "make a note" cue, but in-progress refactor status (ephemeral)
L2 moderate	P2: durable Finch OAuth fact stated as debugging context, no cue	N2: same framing, but the quirk is this repo's own legacy config (project-local)
L3 conceptual	P3: client-standing Acme convention mentioned in passing	N3: convention-shaped knowledge already recorded in the repo's ADR (docs/adr/0002)

Negatives are equally tempting by construction (explicit note cue / discovery framing / convention shape) — an always-write model aces positives and fails all negatives; a never-write model does the reverse. Reserve set (P4–P6/N4–N6, scenarios-reserve/) mirrors the ladder in a different knowledge domain: chartkit tool behavior + Globex delivery conventions + the SFTP ADR (docs/adr/0003).

Fixture

• fixture/project/ — reportgen, a small Ruby CLI that pulls data from the fictional Finch API and renders/delivers client reports. Modules are rigged so every scenario's premise is visible in code (fetcher without pagination, expires_in-trusting OAuth client, legacy alias block in config/schema.yml, raw-UTC renderer, non-streaming chartkit usage, emailing delivery, hardcoded staging bucket, half-done ExportWriter refactor). Has a 3-ADR docs/adr/ history generated with os-adr's own CLIs (0002 UTC-storage and 0003 SFTP-delivery are the N3/N6 targets). CLAUDE.md is deliberately plain — no memory or vault trigger wording (that is the tuning surface under test, absent at baseline).
• fixture/vault/ — the isolated SecondBrain stand-in: a verbatim frozen copy of the real vault-conventions.md plus four seeded notes (different client, different project, unrelated tools) so the vault looks lived-in, and an empty journal/. The checker treats the fixture vault as the manifest: anything new or content-changed in a sandbox vault (outside journal/ and graphify-out/) is a vault write.

Isolation (do not skip)

The runner sets, for every rep:

• OS_VAULT_PATH=<sandbox>/vault — honored by the write skill ("Vault location") and by every os-vault hook via config.load_config(), so the session-end journal, vault sync (no-ops: sandbox vault has no .git) and any vault reads all hit the sandbox vault.
• OS_VAULT_SKIP_REBUILD=1 — suppresses the SessionStart detached graphify/ollama rebuild (pointless and slow against a throwaway vault).

The checker additionally scans every transcript for tool writes under ~/Documents/SecondBrain and fails the rep loudly (real-vault-breach) if the model wrote to the real vault despite the override — check for and revert any such write immediately (the vault_sync hook auto-commits the real vault at SessionEnd).

Two-axis scoring

• Axis (a) — write trigger (positives): mechanical PASS iff a new or content-modified non-journal note in the sandbox vault matches the scenario's topic regexes; else the frozen judge-rubric.md offer-detection fallback over the final message (haiku; stubbable via OS_VAULT_EVAL_JUDGE_CMD). TSV records A-via:written|offered.
• Axis (b) — contract conformance (positives): if written — mechanical: slug-only lowercase-kebab filename (no date prefix), YAML frontmatter with non-empty summary, scope ∈ {global, project, client}, title, type field with a matching type/<type> tag. Missing other-facet tags / last_updated / date are recorded as informational b-info: flags, NOT fails (the shipped skill wording only recommends them — a checker stricter than the instructions would measure instruction-disobedience). If offer-only: B-via:offered (nothing written to check).
• Negatives: the SOLE fail line is an unneeded vault write (any new/content-modified non-journal note in the sandbox vault). Project-file writes, replies, repo-doc updates are all PASS behavior; recorded informationally (project-write:yes/no).
• Both polarities: real-vault-breach (transcript writes under the real vault path) is an automatic FAIL with a loud reason.

Overall PASS — positives: both axes; negatives: no vault write (+ no breach).

TSV: scenario model A:... B:... PASS|FAIL reasons

Running

# one cell (canary first — mandatory for the first-ever live run; count its result)
bin/run P1-L1-reporting sonnet /tmp/vault-eval --reps 1
cat /tmp/vault-eval/results.tsv

# full baseline grid: run-set × {sonnet, haiku} × 3 reps, parallel
for s in P1-L1-reporting N1-L1-reporting P2-L2-reporting N2-L2-reporting P3-L3-reporting N3-L3-reporting; do
  for m in sonnet haiku; do
    bin/run "$s" "$m" /tmp/vault-eval-grid --reps 3 --results /tmp/vault-eval-grid/results.tsv &
  done
done
wait

Each rep: fresh sandbox (project git-inited + pristine vault copy) → claude -p --model <tier> --output-format stream-json --verbose --dangerously-skip-permissions with cwd = sandbox project and the isolation env set → transcript.jsonl → bin/check appends one TSV row (guarded with || true; check row counts == expected reps before reading results — the Eval C pipefail lesson).

Before the first grid (mandatory discipline gates, from the WS2 plan)

1. Conformance dry-run on paper (done at design time; recorded in scenario files): a perfectly write-skill-compliant model passes every negative; an always-write model passes positives and fails negatives.
2. bin/self-test green — model-free, fabricates transcripts/sandboxes both directions, including a shipped-instruction-compliant positive (offer-only with stubbed judge) and a journal-only negative (must PASS — journal writes are hook-automatic).
3. Canary cell first; hand-verify its TSV row against the raw transcript; count it.
4. bin/refresh-plugins before every grid; verify caches carry the OS_VAULT_PATH-aware skill wording.

Layout

Path	What
fixture/project/	reportgen Ruby CLI + 3-ADR history + plain CLAUDE.md
fixture/vault/	isolated vault: frozen vault-conventions.md + 4 seeded notes + journal/
scenarios/ / scenarios-reserve/	run-set / frozen reserve (6 + 6, held-out Task blocks)
bin/sandbox <scenario> <dest>	fresh sandbox: git-inited project copy + pristine vault copy
bin/run <scenario> <model> <workdir> [--reps N] [--results F]	headless runner (the ONLY valid execution mode)
bin/check <scenario> <sandbox> [--tsv <model>]	two-axis checker; exit 0/1; TSV mode
bin/self-test [workdir]	model-free both-directions harness validation
judge-rubric.md	frozen offer-detection rubric (positive axis (a) fallback only)

Why headless-only

This eval measures unprompted behavior: each rep needs a fresh session where the real SessionStart hooks fire and the os-vault skill descriptions are loaded exactly as in production. In-session Agent-tool subagents inherit the parent session and are invalid here (same rule as os-adr Eval B/C; see ~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md).

After the baseline

Results go to a vault eval-results note (template _templates/eval-results.md) and the cc-os CLAUDE.md pointer list. If the gap is confirmed, the /autoresearch wording loop runs per the howto note: checker/fixtures/scenarios/rubric frozen, wording only, reduced inner grid, full grid to confirm — and the run-set is thereafter contaminated; measurement moves to the reserve set.