160 lines
9.3 KiB
Markdown
160 lines
9.3 KiB
Markdown
# os-vault write-behavior eval — unprompted vault-write discrimination (held-out)
|
||
|
||
_Last updated: 2026-07-07 — harness built (WS2), baseline grid run, and the
|
||
wording loop complete and shipped; run-set and reserve are now contaminated for
|
||
future wording tuning (see `eval/results/`)._
|
||
|
||
Measures whether the model **unprompted** (a) writes evergreen cross-project knowledge to
|
||
the SecondBrain vault when it surfaces during normal work, and (b) writes it with
|
||
contract-conforming frontmatter — and equally, whether it correctly does **NOT** write
|
||
ephemera, project-local detail, or repo-recorded decisions to the vault. Applies the os-adr
|
||
Eval B/C playbook (ambiguity ladder, paired positives/negatives, run-set + frozen reserve).
|
||
Plan: `docs/plans/ws2-os-vault-write-eval.md`.
|
||
|
||
> **HELD-OUT — read this first.** The `## Task` blocks in `scenarios/*.md` and
|
||
> `scenarios-reserve/*.md` must never be pasted into an interactive session or informally
|
||
> "tried out". The only permitted non-grid execution is `bin/self-test`, which fabricates
|
||
> transcripts and never uses the Task blocks. The reserve set is additionally never *read*
|
||
> informally — it becomes the measurement set the moment wording is tuned against the
|
||
> run-set.
|
||
|
||
## What the baseline measures
|
||
|
||
At baseline the shipped os-vault wording has **no trigger-conditioned "when to write"
|
||
phrasing anywhere** (WS2 defect 2, deliberately left unfixed until after the baseline).
|
||
Expected shape: positives likely FAIL axis (a) at both tiers (that gap IS the measurement);
|
||
negatives should PASS (nothing currently pushes the model to write). The post-baseline
|
||
wording loop then tunes trigger wording (skill description/body, hook note, optionally a
|
||
CLAUDE.md section template mirroring os-adr's) against the run-set only.
|
||
|
||
## Design: ambiguity ladder with paired positive/negative
|
||
|
||
| Level | Positive (should write/offer) | Negative (should NOT write) |
|
||
|---|---|---|
|
||
| L1 explicit | P1: durable Finch API fact + explicit "worth remembering" cue | N1: explicit "make a note" cue, but in-progress refactor status (ephemeral) |
|
||
| L2 moderate | P2: durable Finch OAuth fact stated as debugging context, no cue | N2: same framing, but the quirk is this repo's own legacy config (project-local) |
|
||
| L3 conceptual | P3: client-standing Acme convention mentioned in passing | N3: convention-shaped knowledge already recorded in the repo's ADR (docs/adr/0002) |
|
||
|
||
Negatives are equally tempting by construction (explicit note cue / discovery framing /
|
||
convention shape) — an always-write model aces positives and fails all negatives; a
|
||
never-write model does the reverse. Reserve set (P4–P6/N4–N6, `scenarios-reserve/`) mirrors
|
||
the ladder in a different knowledge domain: chartkit tool behavior + Globex delivery
|
||
conventions + the SFTP ADR (docs/adr/0003).
|
||
|
||
## Fixture
|
||
|
||
- `fixture/project/` — `reportgen`, a small Ruby CLI that pulls data from the fictional
|
||
Finch API and renders/delivers client reports. Modules are rigged so every scenario's
|
||
premise is visible in code (fetcher without pagination, expires_in-trusting OAuth client,
|
||
legacy alias block in config/schema.yml, raw-UTC renderer, non-streaming chartkit usage,
|
||
emailing delivery, hardcoded staging bucket, half-done ExportWriter refactor). Has a
|
||
3-ADR `docs/adr/` history generated with os-adr's own CLIs (0002 UTC-storage and 0003
|
||
SFTP-delivery are the N3/N6 targets). `CLAUDE.md` is deliberately plain — **no memory or
|
||
vault trigger wording** (that is the tuning surface under test, absent at baseline).
|
||
- `fixture/vault/` — the isolated SecondBrain stand-in: a **verbatim frozen copy** of the
|
||
real `vault-conventions.md` plus four seeded notes (different client, different project,
|
||
unrelated tools) so the vault looks lived-in, and an empty `journal/`. The checker treats
|
||
the fixture vault as the manifest: anything new or content-changed in a sandbox vault
|
||
(outside `journal/` and `graphify-out/`) is a vault write.
|
||
|
||
## Isolation (do not skip)
|
||
|
||
The runner sets, for every rep:
|
||
|
||
- `OS_VAULT_PATH=<sandbox>/vault` — honored by the write skill ("Vault location") and by
|
||
every os-vault hook via `config.load_config()`, so the session-end journal, vault sync
|
||
(no-ops: sandbox vault has no `.git`) and any vault reads all hit the sandbox vault.
|
||
- `OS_VAULT_SKIP_REBUILD=1` — suppresses the SessionStart detached graphify/ollama rebuild
|
||
(pointless and slow against a throwaway vault).
|
||
|
||
The checker additionally scans every transcript for tool writes under
|
||
`~/Documents/SecondBrain` and fails the rep loudly (`real-vault-breach`) if the model wrote
|
||
to the real vault despite the override — check for and revert any such write immediately
|
||
(the vault_sync hook auto-commits the real vault at SessionEnd).
|
||
|
||
## Two-axis scoring
|
||
|
||
- **Axis (a) — write trigger (positives)**: mechanical PASS iff a new or content-modified
|
||
non-journal note in the sandbox vault matches the scenario's topic regexes; else the
|
||
frozen `judge-rubric.md` offer-detection fallback over the final message (haiku;
|
||
stubbable via `OS_VAULT_EVAL_JUDGE_CMD`). TSV records `A-via:written|offered`.
|
||
- **Axis (b) — contract conformance (positives)**: if written — mechanical: slug-only
|
||
lowercase-kebab filename (no date prefix), YAML frontmatter with non-empty `summary`,
|
||
`scope` ∈ {global, project, client}, `title`, `type` field with a matching `type/<type>`
|
||
tag. Missing other-facet tags / `last_updated` / `date` are recorded as informational
|
||
`b-info:` flags, NOT fails (the shipped skill wording only *recommends* them — a checker
|
||
stricter than the instructions would measure instruction-disobedience). If offer-only:
|
||
`B-via:offered` (nothing written to check).
|
||
- **Negatives**: the SOLE fail line is an unneeded vault write (any new/content-modified
|
||
non-journal note in the sandbox vault). Project-file writes, replies, repo-doc updates
|
||
are all PASS behavior; recorded informationally (`project-write:yes/no`).
|
||
- **Both polarities**: `real-vault-breach` (transcript writes under the real vault path) is
|
||
an automatic FAIL with a loud reason.
|
||
|
||
**Overall PASS** — positives: both axes; negatives: no vault write (+ no breach).
|
||
|
||
TSV: `scenario model A:... B:... PASS|FAIL reasons`
|
||
|
||
## Running
|
||
|
||
```bash
|
||
# one cell (canary first — mandatory for the first-ever live run; count its result)
|
||
bin/run P1-L1-reporting sonnet /tmp/vault-eval --reps 1
|
||
cat /tmp/vault-eval/results.tsv
|
||
|
||
# full baseline grid: run-set × {sonnet, haiku} × 3 reps, parallel
|
||
for s in P1-L1-reporting N1-L1-reporting P2-L2-reporting N2-L2-reporting P3-L3-reporting N3-L3-reporting; do
|
||
for m in sonnet haiku; do
|
||
bin/run "$s" "$m" /tmp/vault-eval-grid --reps 3 --results /tmp/vault-eval-grid/results.tsv &
|
||
done
|
||
done
|
||
wait
|
||
```
|
||
|
||
Each rep: fresh sandbox (project git-inited + pristine vault copy) → `claude -p
|
||
--model <tier> --output-format stream-json --verbose --dangerously-skip-permissions` with
|
||
cwd = sandbox project and the isolation env set → `transcript.jsonl` → `bin/check` appends
|
||
one TSV row (guarded with `|| true`; check row counts == expected reps before reading
|
||
results — the Eval C pipefail lesson).
|
||
|
||
## Before the first grid (mandatory discipline gates, from the WS2 plan)
|
||
|
||
1. Conformance dry-run on paper (done at design time; recorded in scenario files): a
|
||
perfectly write-skill-compliant model passes every negative; an always-write model
|
||
passes positives and fails negatives.
|
||
2. `bin/self-test` green — model-free, fabricates transcripts/sandboxes both directions,
|
||
including a shipped-instruction-compliant positive (offer-only with stubbed judge) and
|
||
a journal-only negative (must PASS — journal writes are hook-automatic).
|
||
3. Canary cell first; hand-verify its TSV row against the raw transcript; count it.
|
||
4. `bin/refresh-plugins` before every grid; verify caches carry the OS_VAULT_PATH-aware
|
||
skill wording.
|
||
|
||
## Layout
|
||
|
||
| Path | What |
|
||
| --- | --- |
|
||
| `fixture/project/` | reportgen Ruby CLI + 3-ADR history + plain CLAUDE.md |
|
||
| `fixture/vault/` | isolated vault: frozen vault-conventions.md + 4 seeded notes + journal/ |
|
||
| `scenarios/` / `scenarios-reserve/` | run-set / frozen reserve (6 + 6, held-out Task blocks) |
|
||
| `bin/sandbox <scenario> <dest>` | fresh sandbox: git-inited project copy + pristine vault copy |
|
||
| `bin/run <scenario> <model> <workdir> [--reps N] [--results F]` | headless runner (the ONLY valid execution mode) |
|
||
| `bin/check <scenario> <sandbox> [--tsv <model>]` | two-axis checker; exit 0/1; TSV mode |
|
||
| `bin/self-test [workdir]` | model-free both-directions harness validation |
|
||
| `judge-rubric.md` | frozen offer-detection rubric (positive axis (a) fallback only) |
|
||
|
||
## Why headless-only
|
||
|
||
This eval measures **unprompted** behavior: each rep needs a fresh session where the real
|
||
SessionStart hooks fire and the os-vault skill descriptions are loaded exactly as in
|
||
production. In-session Agent-tool subagents inherit the parent session and are invalid here
|
||
(same rule as os-adr Eval B/C; see
|
||
`~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md`).
|
||
|
||
## After the baseline
|
||
|
||
Results go to a vault `eval-results` note (template `_templates/eval-results.md`) and the
|
||
cc-os CLAUDE.md pointer list. If the gap is confirmed, the `/autoresearch` wording loop runs
|
||
per the howto note: checker/fixtures/scenarios/rubric frozen, wording only, reduced inner
|
||
grid, full grid to confirm — and the run-set is thereafter contaminated; measurement moves
|
||
to the reserve set.
|