cc-os/plugins/os-vault/eval/README.md

# os-vault write-behavior eval — unprompted vault-write discrimination (held-out)

_Last updated: 2026-07-07 — harness built (WS2), baseline grid run, and the
wording loop complete and shipped; run-set and reserve are now contaminated for
future wording tuning (see `eval/results/`)._

Measures whether the model **unprompted** (a) writes evergreen cross-project knowledge to
the SecondBrain vault when it surfaces during normal work, and (b) writes it with
contract-conforming frontmatter — and equally, whether it correctly does **NOT** write
ephemera, project-local detail, or repo-recorded decisions to the vault. Applies the os-adr
Eval B/C playbook (ambiguity ladder, paired positives/negatives, run-set + frozen reserve).
Plan: `docs/plans/ws2-os-vault-write-eval.md`.

> **HELD-OUT — read this first.** The `## Task` blocks in `scenarios/*.md` and
> `scenarios-reserve/*.md` must never be pasted into an interactive session or informally
> "tried out". The only permitted non-grid execution is `bin/self-test`, which fabricates
> transcripts and never uses the Task blocks. The reserve set is additionally never *read*
> informally — it becomes the measurement set the moment wording is tuned against the
> run-set.

## What the baseline measures

At baseline the shipped os-vault wording has **no trigger-conditioned "when to write"
phrasing anywhere** (WS2 defect 2, deliberately left unfixed until after the baseline).
Expected shape: positives likely FAIL axis (a) at both tiers (that gap IS the measurement);
negatives should PASS (nothing currently pushes the model to write). The post-baseline
wording loop then tunes trigger wording (skill description/body, hook note, optionally a
CLAUDE.md section template mirroring os-adr's) against the run-set only.

## Design: ambiguity ladder with paired positive/negative

| Level | Positive (should write/offer) | Negative (should NOT write) |
|---|---|---|
| L1 explicit | P1: durable Finch API fact + explicit "worth remembering" cue | N1: explicit "make a note" cue, but in-progress refactor status (ephemeral) |
| L2 moderate | P2: durable Finch OAuth fact stated as debugging context, no cue | N2: same framing, but the quirk is this repo's own legacy config (project-local) |
| L3 conceptual | P3: client-standing Acme convention mentioned in passing | N3: convention-shaped knowledge already recorded in the repo's ADR (docs/adr/0002) |

Negatives are equally tempting by construction (explicit note cue / discovery framing /
convention shape) — an always-write model aces positives and fails all negatives; a
never-write model does the reverse. Reserve set (P4–P6/N4–N6, `scenarios-reserve/`) mirrors
the ladder in a different knowledge domain: chartkit tool behavior + Globex delivery
conventions + the SFTP ADR (docs/adr/0003).

## Fixture

- `fixture/project/` — `reportgen`, a small Ruby CLI that pulls data from the fictional
  Finch API and renders/delivers client reports. Modules are rigged so every scenario's
  premise is visible in code (fetcher without pagination, expires_in-trusting OAuth client,
  legacy alias block in config/schema.yml, raw-UTC renderer, non-streaming chartkit usage,
  emailing delivery, hardcoded staging bucket, half-done ExportWriter refactor). Has a
  3-ADR `docs/adr/` history generated with os-adr's own CLIs (0002 UTC-storage and 0003
  SFTP-delivery are the N3/N6 targets). `CLAUDE.md` is deliberately plain — **no memory or
  vault trigger wording** (that is the tuning surface under test, absent at baseline).
- `fixture/vault/` — the isolated SecondBrain stand-in: a **verbatim frozen copy** of the
  real `vault-conventions.md` plus four seeded notes (different client, different project,
  unrelated tools) so the vault looks lived-in, and an empty `journal/`. The checker treats
  the fixture vault as the manifest: anything new or content-changed in a sandbox vault
  (outside `journal/` and `graphify-out/`) is a vault write.

## Isolation (do not skip)

The runner sets, for every rep:

- `OS_VAULT_PATH=<sandbox>/vault` — honored by the write skill ("Vault location") and by
  every os-vault hook via `config.load_config()`, so the session-end journal, vault sync
  (no-ops: sandbox vault has no `.git`) and any vault reads all hit the sandbox vault.
- `OS_VAULT_SKIP_REBUILD=1` — suppresses the SessionStart detached graphify/ollama rebuild
  (pointless and slow against a throwaway vault).

The checker additionally scans every transcript for tool writes under
`~/Documents/SecondBrain` and fails the rep loudly (`real-vault-breach`) if the model wrote
to the real vault despite the override — check for and revert any such write immediately
(the vault_sync hook auto-commits the real vault at SessionEnd).

## Two-axis scoring

- **Axis (a) — write trigger (positives)**: mechanical PASS iff a new or content-modified
  non-journal note in the sandbox vault matches the scenario's topic regexes; else the
  frozen `judge-rubric.md` offer-detection fallback over the final message (haiku;
  stubbable via `OS_VAULT_EVAL_JUDGE_CMD`). TSV records `A-via:written|offered`.
- **Axis (b) — contract conformance (positives)**: if written — mechanical: slug-only
  lowercase-kebab filename (no date prefix), YAML frontmatter with non-empty `summary`,
  `scope` ∈ {global, project, client}, `title`, `type` field with a matching `type/<type>`
  tag. Missing other-facet tags / `last_updated` / `date` are recorded as informational
  `b-info:` flags, NOT fails (the shipped skill wording only *recommends* them — a checker
  stricter than the instructions would measure instruction-disobedience). If offer-only:
  `B-via:offered` (nothing written to check).
- **Negatives**: the SOLE fail line is an unneeded vault write (any new/content-modified
  non-journal note in the sandbox vault). Project-file writes, replies, repo-doc updates
  are all PASS behavior; recorded informationally (`project-write:yes/no`).
- **Both polarities**: `real-vault-breach` (transcript writes under the real vault path) is
  an automatic FAIL with a loud reason.

**Overall PASS** — positives: both axes; negatives: no vault write (+ no breach).

TSV: `scenario  model  A:...  B:...  PASS|FAIL  reasons`

## Running

```bash
# one cell (canary first — mandatory for the first-ever live run; count its result)
bin/run P1-L1-reporting sonnet /tmp/vault-eval --reps 1
cat /tmp/vault-eval/results.tsv

# full baseline grid: run-set × {sonnet, haiku} × 3 reps, parallel
for s in P1-L1-reporting N1-L1-reporting P2-L2-reporting N2-L2-reporting P3-L3-reporting N3-L3-reporting; do
  for m in sonnet haiku; do
    bin/run "$s" "$m" /tmp/vault-eval-grid --reps 3 --results /tmp/vault-eval-grid/results.tsv &
  done
done
wait
```

Each rep: fresh sandbox (project git-inited + pristine vault copy) → `claude -p
--model <tier> --output-format stream-json --verbose --dangerously-skip-permissions` with
cwd = sandbox project and the isolation env set → `transcript.jsonl` → `bin/check` appends
one TSV row (guarded with `|| true`; check row counts == expected reps before reading
results — the Eval C pipefail lesson).

## Before the first grid (mandatory discipline gates, from the WS2 plan)

1. Conformance dry-run on paper (done at design time; recorded in scenario files): a
   perfectly write-skill-compliant model passes every negative; an always-write model
   passes positives and fails negatives.
2. `bin/self-test` green — model-free, fabricates transcripts/sandboxes both directions,
   including a shipped-instruction-compliant positive (offer-only with stubbed judge) and
   a journal-only negative (must PASS — journal writes are hook-automatic).
3. Canary cell first; hand-verify its TSV row against the raw transcript; count it.
4. `bin/refresh-plugins` before every grid; verify caches carry the OS_VAULT_PATH-aware
   skill wording.

## Layout

| Path | What |
| --- | --- |
| `fixture/project/` | reportgen Ruby CLI + 3-ADR history + plain CLAUDE.md |
| `fixture/vault/` | isolated vault: frozen vault-conventions.md + 4 seeded notes + journal/ |
| `scenarios/` / `scenarios-reserve/` | run-set / frozen reserve (6 + 6, held-out Task blocks) |
| `bin/sandbox <scenario> <dest>` | fresh sandbox: git-inited project copy + pristine vault copy |
| `bin/run <scenario> <model> <workdir> [--reps N] [--results F]` | headless runner (the ONLY valid execution mode) |
| `bin/check <scenario> <sandbox> [--tsv <model>]` | two-axis checker; exit 0/1; TSV mode |
| `bin/self-test [workdir]` | model-free both-directions harness validation |
| `judge-rubric.md` | frozen offer-detection rubric (positive axis (a) fallback only) |

## Why headless-only

This eval measures **unprompted** behavior: each rep needs a fresh session where the real
SessionStart hooks fire and the os-vault skill descriptions are loaded exactly as in
production. In-session Agent-tool subagents inherit the parent session and are invalid here
(same rule as os-adr Eval B/C; see
`~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md`).

## After the baseline

Results go to a vault `eval-results` note (template `_templates/eval-results.md`) and the
cc-os CLAUDE.md pointer list. If the gap is confirmed, the `/autoresearch` wording loop runs
per the howto note: checker/fixtures/scenarios/rubric frozen, wording only, reduced inner
grid, full grid to confirm — and the run-set is thereafter contaminated; measurement moves
to the reserve set.