os-vault: WS2 wording loop — destination ladder shipped, reserve grid measured

4 iterations (destination ladder -> mechanical vault-root -> CLAUDE.md
cross-project-memory section -> ADR-vs-vault disambiguation) + candidates 2-4
unscored + post-reserve YAML-colon quoting rule. Confirm 18/18 sonnet, 16/18
haiku; held-out reserve trigger-axis 13/18, negatives 17/18 (baseline 1/19,
18/18). Checker breach-regex false positive fixed + rescored, 2 self-test
guards added. Run-set and reserve both contaminated now.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
jared 2026-07-07 14:01:15 -04:00
parent 638c6d03ed
commit 42860d06d1
13 changed files with 294 additions and 11 deletions

View File

@ -84,6 +84,7 @@ to those two and fix the stale doc.
- Isolation seam (2026-07-06, WS2): `OS_VAULT_PATH` env var overrides `vault_path` for the write skill and every hook via `config.load_config()`; `OS_VAULT_SKIP_REBUILD` suppresses the SessionStart graphify rebuild. Added for the eval harness; zero production impact when unset. - Isolation seam (2026-07-06, WS2): `OS_VAULT_PATH` env var overrides `vault_path` for the write skill and every hook via `config.load_config()`; `OS_VAULT_SKIP_REBUILD` suppresses the SessionStart graphify rebuild. Added for the eval harness; zero production impact when unset.
- Write-behavior eval (2026-07-06, WS2): `plugins/os-vault/eval/` — held-out unprompted vault-write discrimination eval (Eval B/C playbook: ambiguity ladder L1 explicit → L3 conceptual, paired positives/negatives, 6 run-set scenarios on a new `reportgen` Ruby fixture + 6 frozen reserve twins in a different knowledge domain, isolated sandbox vault, headless-only runner, deterministic-first Ruby checker with narrow frozen offer-detection judge fallback stubbable via `OS_VAULT_EVAL_JUDGE_CMD`, model-free `bin/self-test` 21/21). **Scenario Task blocks are held-out — never run informally; the reserve set is never even read informally.** Procedure: `plugins/os-vault/eval/README.md`. - Write-behavior eval (2026-07-06, WS2): `plugins/os-vault/eval/` — held-out unprompted vault-write discrimination eval (Eval B/C playbook: ambiguity ladder L1 explicit → L3 conceptual, paired positives/negatives, 6 run-set scenarios on a new `reportgen` Ruby fixture + 6 frozen reserve twins in a different knowledge domain, isolated sandbox vault, headless-only runner, deterministic-first Ruby checker with narrow frozen offer-detection judge fallback stubbable via `OS_VAULT_EVAL_JUDGE_CMD`, model-free `bin/self-test` 21/21). **Scenario Task blocks are held-out — never run informally; the reserve set is never even read informally.** Procedure: `plugins/os-vault/eval/README.md`.
- Untuned baseline grid (2026-07-06, run-set × sonnet/haiku × 3 reps + 1 counted canary, 37 reps, zero harness errors): **positives 1/19, negatives 18/18**. Headline: at L1 (explicit cue) both tiers persist the knowledge every rep but route it to Claude Code's built-in auto-memory (`~/.claude/projects/<slug>/memory/`) instead of the vault; L2/L3 mostly don't persist at all; zero over-triggering anywhere. The WS2 wording loop (step 4, NOT started; run-set still uncontaminated) therefore has two targets: trigger at L2/L3, and route to the vault over auto-memory at L1. TSV: `eval/results/2026-07-06-baseline-grid.tsv`; full analysis: vault note [[os-vault-write-eval-baseline-grid-results]]. - Untuned baseline grid (2026-07-06, run-set × sonnet/haiku × 3 reps + 1 counted canary, 37 reps, zero harness errors): **positives 1/19, negatives 18/18**. Headline: at L1 (explicit cue) both tiers persist the knowledge every rep but route it to Claude Code's built-in auto-memory (`~/.claude/projects/<slug>/memory/`) instead of the vault; L2/L3 mostly don't persist at all; zero over-triggering anywhere. The WS2 wording loop (step 4, NOT started; run-set still uncontaminated) therefore has two targets: trigger at L2/L3, and route to the vault over auto-memory at L1. TSV: `eval/results/2026-07-06-baseline-grid.tsv`; full analysis: vault note [[os-vault-write-eval-baseline-grid-results]].
- WS2 wording loop COMPLETE (2026-07-07, 4 iterations + confirm + held-out reserve; run-set AND reserve now contaminated): shipped destination-ladder write SKILL.md + SessionStart USAGE_NOTE (mechanical triggers + before-final-reply checkpoint), mechanical vault-root resolution (`${OS_VAULT_PATH:-$HOME/Documents/SecondBrain}` — fixed real-vault breaches found on iteration 1), fixture CLAUDE.md "Cross-project memory" section (**the real-project adoption template**, required to reach haiku at L2/L3; includes ADR-vs-vault disambiguation), candidates 24 unscored, post-reserve YAML-colon quoting rule. Confirm grid sonnet 18/18 / haiku 16/18; reserve (held-out) positives 8/18 strict / trigger-axis 13/18, negatives 17/18 (vs baseline 1/19, 18/18). Residual gaps: YAML-hostile titles, haiku L2 no-trigger, generalization-laundering over-trigger. One checker false-positive fixed+rescored mid-loop (breach regex vs `2>/dev/null`); negative-criterion redesign pre-registered. TSVs: `eval/results/2026-07-07-wording-*.tsv`; loop log: `autoresearch/classic-260707-1253/loop-log.md`; analysis: vault note [[os-vault-write-eval-wording-loop-results]]. Next signal is production IRL audits.
**Global os-orchestration plugin** — `cc-os/plugins/os-orchestration/` (git-tracked, 2026-07-03); symlinked into `~/.claude/plugins/os-orchestration` **Global os-orchestration plugin** — `cc-os/plugins/os-orchestration/` (git-tracked, 2026-07-03); symlinked into `~/.claude/plugins/os-orchestration`
- Hooks: `hooks/``inject.py` (injects `ORCHESTRATION.md` as additionalContext to all sessions) - Hooks: `hooks/``inject.py` (injects `ORCHESTRATION.md` as additionalContext to all sessions)

View File

@ -135,8 +135,12 @@ module OsVaultEval
return false unless command.match?(REAL_VAULT_PATTERN) return false unless command.match?(REAL_VAULT_PATTERN)
# Plain reads (cat/ls/grep/find) of the real vault are not a breach — # Plain reads (cat/ls/grep/find) of the real vault are not a breach —
# only flag when the command also writes INTO that path. # only flag when the command also writes INTO that path. The redirect
command.match?(/>>?[^&]*(?:SecondBrain)/) || # TARGET itself must be under SecondBrain (a `2>/dev/null` followed by a
# later read-only mention of the vault path is not a write — false
# positive found on iteration 1, 2026-07-07; rescored per the
# instrument-fix rule, transcripts untouched).
command.match?(/>>?\s*["']?[^\s"'|;&<>]*SecondBrain/) ||
command.match?(/\btee\b[^|]*SecondBrain/) || command.match?(/\btee\b[^|]*SecondBrain/) ||
command.match?(/\b(?:cp|mv|rsync)\b.*SecondBrain/) command.match?(/\b(?:cp|mv|rsync)\b.*SecondBrain/)
end end

View File

@ -288,6 +288,43 @@ expect FAIL "breach (Write to real vault path)" \
env OS_VAULT_EVAL_JUDGE_CMD="printf PASS" "$EVAL_ROOT/bin/check" P1-L1-reporting "$sb" env OS_VAULT_EVAL_JUDGE_CMD="printf PASS" "$EVAL_ROOT/bin/check" P1-L1-reporting "$sb"
expect_contains "$row" "real-vault-breach" "breach records real-vault-breach" expect_contains "$row" "real-vault-breach" "breach records real-vault-breach"
# --- 9b. NOT a breach: read-only Bash mentioning the real vault path with an
# unrelated redirect (2>/dev/null) must not trip the redirect heuristic, and a
# sandbox-vault write alongside it still PASSes (iteration-1 false positive,
# 2026-07-07). A Bash redirect whose TARGET is under the real vault stays FAIL. ---
sb="$WORKDIR/breach-fp"
make_sandbox "$sb"
mkdir -p "$sb/vault/reference"
cat > "$sb/vault/reference/finch-api-page-size-silent-cap.md" <<'NOTE'
---
type: reference
title: Finch API silently caps page_size at 500
summary: Finch API caps every response at 500 records unless page_size is passed explicitly.
scope: global
tags:
- type/reference
- tool/finch-api
---
Finch caps responses at 500 records (pagination required).
NOTE
write_transcript "$sb" <<EOF
$(tool_use_event "Bash" '{"command":"echo \"OS_VAULT_PATH=$OS_VAULT_PATH\"; ls ~/Documents/SecondBrain 2>/dev/null | head"}')
$(tool_use_event "Write" '{"file_path":"SANDBOX/vault/reference/finch-api-page-size-silent-cap.md","content":"..."}')
$(result_event "Recorded the Finch page-size cap in the vault.")
EOF
expect PASS "read-only real-vault mention with 2>/dev/null is not a breach" \
env OS_VAULT_EVAL_JUDGE_CMD="printf FAIL" "$EVAL_ROOT/bin/check" P1-L1-reporting "$sb"
sb="$WORKDIR/breach-redirect"
make_sandbox "$sb"
write_transcript "$sb" <<EOF
$(tool_use_event "Bash" '{"command":"echo note > ~/Documents/SecondBrain/x.md"}')
$(result_event "Saved a note.")
EOF
expect FAIL "Bash redirect INTO real vault is still a breach" \
env OS_VAULT_EVAL_JUDGE_CMD="printf PASS" "$EVAL_ROOT/bin/check" P1-L1-reporting "$sb"
# --- 10. Harness error: sandbox without transcript.jsonl -> FAIL --- # --- 10. Harness error: sandbox without transcript.jsonl -> FAIL ---
sb="$WORKDIR/no-transcript" sb="$WORKDIR/no-transcript"

View File

@ -33,3 +33,20 @@ every file, one class per file under `lib/reportgen/`.
## Architecture decisions ## Architecture decisions
Architecture decisions for this project are recorded in `docs/adr/`. Architecture decisions for this project are recorded in `docs/adr/`.
## Cross-project memory
Durable knowledge that is not about this repo goes to the SecondBrain vault via
`/os-vault:write` — not into this repo, and not into auto-memory.
- **When** you discover during a task that a tool, API, or service behaves differently from
what its docs or config claim (advertised X, actually Y) → that discrepancy is evergreen
vault knowledge. Run `/os-vault:write` before you consider the task complete.
- **When** the user states a client-specific requirement, standard, or standing preference →
record it with `/os-vault:write` (scope: client). An ADR records how *this repo's code*
implements it; the client's requirement itself is client knowledge and an ADR does **not**
satisfy this rule — the vault note is still required.
- Mechanical rule: **before your final reply on any task**, check whether either of the above
occurred this session; if yes, the task is not complete until the vault note is written.
- Ephemeral material stays out of the vault: task status, in-progress refactor state,
project-local quirks, and anything this repo's docs or ADRs already record.

View File

@ -0,0 +1,36 @@
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
N2-L2-reporting haiku W:no B:PASS PASS project-write:yes
P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
P2-L2-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
N2-L2-reporting sonnet W:no B:PASS PASS project-write:yes
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
N1-L1-reporting sonnet W:no B:PASS PASS project-write:no
P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
N2-L2-reporting haiku W:no B:PASS PASS project-write:yes
P2-L2-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
P3-L3-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
N2-L2-reporting sonnet W:no B:PASS PASS project-write:yes
N2-L2-reporting haiku W:no B:PASS PASS project-write:yes
P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
P3-L3-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
N2-L2-reporting sonnet W:no B:PASS PASS project-write:yes
N3-L3-reporting haiku W:no B:PASS PASS project-write:yes
P2-L2-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
N3-L3-reporting sonnet W:no B:PASS PASS project-write:yes
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
P3-L3-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
N3-L3-reporting haiku W:no B:PASS PASS project-write:yes
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
N3-L3-reporting haiku W:no B:PASS PASS project-write:yes
N3-L3-reporting sonnet W:yes B:FAIL FAIL wrote:ruby-time-getlocal-offset-only.md; project-write:yes
P3-L3-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
N3-L3-reporting sonnet W:no B:PASS PASS project-write:yes
1 N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
2 N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
3 N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
4 P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
5 N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
6 P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
7 N2-L2-reporting haiku W:no B:PASS PASS project-write:yes
8 P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
9 P2-L2-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
10 N2-L2-reporting sonnet W:no B:PASS PASS project-write:yes
11 N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
12 N1-L1-reporting sonnet W:no B:PASS PASS project-write:no
13 P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
14 P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
15 N2-L2-reporting haiku W:no B:PASS PASS project-write:yes
16 P2-L2-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
17 P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
18 P3-L3-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
19 N2-L2-reporting sonnet W:no B:PASS PASS project-write:yes
20 N2-L2-reporting haiku W:no B:PASS PASS project-write:yes
21 P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
22 P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
23 P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
24 P3-L3-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
25 N2-L2-reporting sonnet W:no B:PASS PASS project-write:yes
26 N3-L3-reporting haiku W:no B:PASS PASS project-write:yes
27 P2-L2-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
28 N3-L3-reporting sonnet W:no B:PASS PASS project-write:yes
29 P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
30 P3-L3-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
31 N3-L3-reporting haiku W:no B:PASS PASS project-write:yes
32 P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
33 N3-L3-reporting haiku W:no B:PASS PASS project-write:yes
34 N3-L3-reporting sonnet W:yes B:FAIL FAIL wrote:ruby-time-getlocal-offset-only.md; project-write:yes
35 P3-L3-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
36 N3-L3-reporting sonnet W:no B:PASS PASS project-write:yes

View File

@ -0,0 +1,24 @@
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
P1-L1-reporting haiku A:FAIL B:FAIL FAIL real-vault-breach; real-vault-breach
P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming; rescored:breach-regex-false-positive
P2-L2-reporting sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
P1-L1-reporting haiku A:PASS B:PASS PASS A-via:offered; B-via:offered
P3-L3-reporting sonnet A:FAIL B:FAIL FAIL real-vault-breach; real-vault-breach
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
P1-L1-reporting haiku A:FAIL B:FAIL FAIL real-vault-breach; real-vault-breach
P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
P2-L2-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
P2-L2-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
P2-L2-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
P3-L3-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
P3-L3-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
1 N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
2 N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
3 N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
4 P1-L1-reporting haiku A:FAIL B:FAIL FAIL real-vault-breach; real-vault-breach
5 P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming; rescored:breach-regex-false-positive
6 P2-L2-reporting sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
7 N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
8 N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
9 P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
10 P1-L1-reporting haiku A:PASS B:PASS PASS A-via:offered; B-via:offered
11 P3-L3-reporting sonnet A:FAIL B:FAIL FAIL real-vault-breach; real-vault-breach
12 P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
13 P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
14 N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
15 P1-L1-reporting haiku A:FAIL B:FAIL FAIL real-vault-breach; real-vault-breach
16 P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
17 P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
18 P2-L2-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
19 P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
20 P2-L2-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
21 P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
22 P2-L2-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
23 P3-L3-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
24 P3-L3-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming

View File

@ -0,0 +1,24 @@
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
P2-L2-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
N1-L1-reporting haiku W:no B:PASS PASS project-write:no
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
P2-L2-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
P3-L3-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
P2-L2-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
P3-L3-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
P3-L3-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
1 N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
2 N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
3 P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
4 N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
5 P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
6 P2-L2-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
7 P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
8 N1-L1-reporting haiku W:no B:PASS PASS project-write:no
9 P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
10 N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
11 P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
12 P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
13 P2-L2-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
14 P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
15 N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
16 P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
17 P3-L3-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
18 P2-L2-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
19 P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
20 P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
21 P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
22 P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
23 P3-L3-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
24 P3-L3-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming

View File

@ -0,0 +1,18 @@
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
N1-L1-reporting haiku W:no B:PASS PASS project-write:no
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
N1-L1-reporting haiku W:no B:PASS PASS project-write:no
P2-L2-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
P2-L2-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
P2-L2-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
P3-L3-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
1 N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
2 N1-L1-reporting haiku W:no B:PASS PASS project-write:no
3 N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
4 P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
5 P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
6 N1-L1-reporting haiku W:no B:PASS PASS project-write:no
7 P2-L2-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
8 N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
9 P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
10 P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
11 P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
12 N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
13 P2-L2-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
14 P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
15 P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
16 P2-L2-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
17 P3-L3-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
18 P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance

View File

@ -0,0 +1,6 @@
P3-L3-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
N3-L3-reporting haiku W:no B:PASS PASS project-write:yes
P3-L3-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
P3-L3-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
N3-L3-reporting haiku W:no B:PASS PASS project-write:yes
N3-L3-reporting haiku W:no B:PASS PASS project-write:no
1 P3-L3-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
2 N3-L3-reporting haiku W:no B:PASS PASS project-write:yes
3 P3-L3-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
4 P3-L3-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
5 N3-L3-reporting haiku W:no B:PASS PASS project-write:yes
6 N3-L3-reporting haiku W:no B:PASS PASS project-write:no

View File

@ -0,0 +1,36 @@
N4-L1-delivery haiku W:no B:PASS PASS project-write:no
N5-L2-delivery haiku W:no B:PASS PASS project-write:yes
N4-L1-delivery sonnet W:no B:PASS PASS project-write:yes
P4-L1-delivery sonnet A:PASS B:FAIL FAIL A-via:written; no parseable YAML frontmatter block
N5-L2-delivery haiku W:no B:PASS PASS project-write:yes
N4-L1-delivery haiku W:no B:PASS PASS project-write:no
P4-L1-delivery haiku A:PASS B:FAIL FAIL A-via:written; no parseable YAML frontmatter block
P5-L2-delivery haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
N6-L3-delivery haiku W:no B:PASS PASS project-write:yes
N4-L1-delivery haiku W:no B:PASS PASS project-write:yes
N6-L3-delivery sonnet W:no B:PASS PASS project-write:yes
N5-L2-delivery haiku W:no B:PASS PASS project-write:yes
N4-L1-delivery sonnet W:no B:PASS PASS project-write:yes
P4-L1-delivery sonnet A:PASS B:FAIL FAIL A-via:written; no parseable YAML frontmatter block
P4-L1-delivery haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
P5-L2-delivery haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
N4-L1-delivery sonnet W:no B:PASS PASS project-write:yes
P4-L1-delivery sonnet A:PASS B:FAIL FAIL A-via:written; no parseable YAML frontmatter block
N5-L2-delivery sonnet W:no B:PASS PASS project-write:yes
N6-L3-delivery sonnet W:no B:PASS PASS project-write:yes
P4-L1-delivery haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
N6-L3-delivery haiku W:no B:PASS PASS project-write:yes
N5-L2-delivery sonnet W:yes B:FAIL FAIL wrote:convention/hardcoded-env-literals-survive-environment-splits.md; project-write:yes
P5-L2-delivery haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
N6-L3-delivery sonnet W:no B:PASS PASS project-write:yes
P6-L3-delivery haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
P5-L2-delivery sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
N6-L3-delivery haiku W:no B:PASS PASS project-write:yes
P6-L3-delivery sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
P5-L2-delivery sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
N5-L2-delivery sonnet W:no B:PASS PASS project-write:yes
P6-L3-delivery haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
P5-L2-delivery sonnet A:PASS B:FAIL FAIL A-via:written; missing or empty title; b-info:missing-last_updated
P6-L3-delivery haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
P6-L3-delivery sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
P6-L3-delivery sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
1 N4-L1-delivery haiku W:no B:PASS PASS project-write:no
2 N5-L2-delivery haiku W:no B:PASS PASS project-write:yes
3 N4-L1-delivery sonnet W:no B:PASS PASS project-write:yes
4 P4-L1-delivery sonnet A:PASS B:FAIL FAIL A-via:written; no parseable YAML frontmatter block
5 N5-L2-delivery haiku W:no B:PASS PASS project-write:yes
6 N4-L1-delivery haiku W:no B:PASS PASS project-write:no
7 P4-L1-delivery haiku A:PASS B:FAIL FAIL A-via:written; no parseable YAML frontmatter block
8 P5-L2-delivery haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
9 N6-L3-delivery haiku W:no B:PASS PASS project-write:yes
10 N4-L1-delivery haiku W:no B:PASS PASS project-write:yes
11 N6-L3-delivery sonnet W:no B:PASS PASS project-write:yes
12 N5-L2-delivery haiku W:no B:PASS PASS project-write:yes
13 N4-L1-delivery sonnet W:no B:PASS PASS project-write:yes
14 P4-L1-delivery sonnet A:PASS B:FAIL FAIL A-via:written; no parseable YAML frontmatter block
15 P4-L1-delivery haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
16 P5-L2-delivery haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
17 N4-L1-delivery sonnet W:no B:PASS PASS project-write:yes
18 P4-L1-delivery sonnet A:PASS B:FAIL FAIL A-via:written; no parseable YAML frontmatter block
19 N5-L2-delivery sonnet W:no B:PASS PASS project-write:yes
20 N6-L3-delivery sonnet W:no B:PASS PASS project-write:yes
21 P4-L1-delivery haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
22 N6-L3-delivery haiku W:no B:PASS PASS project-write:yes
23 N5-L2-delivery sonnet W:yes B:FAIL FAIL wrote:convention/hardcoded-env-literals-survive-environment-splits.md; project-write:yes
24 P5-L2-delivery haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
25 N6-L3-delivery sonnet W:no B:PASS PASS project-write:yes
26 P6-L3-delivery haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
27 P5-L2-delivery sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
28 N6-L3-delivery haiku W:no B:PASS PASS project-write:yes
29 P6-L3-delivery sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
30 P5-L2-delivery sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
31 N5-L2-delivery sonnet W:no B:PASS PASS project-write:yes
32 P6-L3-delivery haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
33 P5-L2-delivery sonnet A:PASS B:FAIL FAIL A-via:written; missing or empty title; b-info:missing-last_updated
34 P6-L3-delivery haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
35 P6-L3-delivery sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
36 P6-L3-delivery sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance

View File

@ -4,6 +4,7 @@
Must return sub-second; heavy work is detached in the background. Context Must return sub-second; heavy work is detached in the background. Context
injection is handled by session_context.py (UserPromptSubmit hook). injection is handled by session_context.py (UserPromptSubmit hook).
""" """
import json
import os import os
import sys import sys
import time import time
@ -16,6 +17,39 @@ import hook_io
HOOK_LOG = "/tmp/memory-hook.log" HOOK_LOG = "/tmp/memory-hook.log"
# Trigger-conditioned destination ladder (WS2 wording loop, 2026-07-07).
# Injected as additionalContext every session; keep it short — it is a
# per-session token cost in every project.
USAGE_NOTE = (
"[os-vault] Where knowledge goes — ask in order: "
"(1) the repo already records it (code, ADR, CLAUDE.md)? -> write it there or "
"nowhere; never duplicate into memory. "
"(2) would it change how you act in a DIFFERENT repo next year (tool/API "
"behavior discovered, client conventions, methodology that worked)? -> the "
"SecondBrain vault via /os-vault:write — NOT auto-memory; the repo is where "
"you learned it, not what it's about. "
"(3) only useful in future sessions of THIS repo? -> auto-memory. "
"(4) task status / in-flight state -> nothing. "
"Mechanical triggers — treat these as part of the task itself: "
"WHEN you discover a tool/API/service behaves differently from what its "
"docs or config claim (advertised X, actually Y) -> that discrepancy is "
"vault knowledge, /os-vault:write it. "
"WHEN the user states a client-specific requirement or standing preference "
"-> vault note, scope: client. "
"BEFORE your final reply on any task -> check: did either occur this "
"session? If yes, the task is NOT COMPLETE until /os-vault:write has "
"captured it. Ephemeral status and in-progress state never go to the vault."
)
def _emit_usage_note():
print(json.dumps({
"hookSpecificOutput": {
"hookEventName": "SessionStart",
"additionalContext": USAGE_NOTE,
}
}))
def _utc(): def _utc():
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
@ -46,6 +80,8 @@ def main():
cfg = config_mod.load_config() cfg = config_mod.load_config()
_emit_usage_note()
if os.environ.get("OS_VAULT_SKIP_REBUILD"): if os.environ.get("OS_VAULT_SKIP_REBUILD"):
_log(f"[{_utc()}] session-start: rebuild skipped (OS_VAULT_SKIP_REBUILD set)") _log(f"[{_utc()}] session-start: rebuild skipped (OS_VAULT_SKIP_REBUILD set)")
return return

View File

@ -1,9 +1,18 @@
--- ---
description: Query the SecondBrain vault knowledge graph for cross-project evergreen knowledge — conventions, tool behavior, client facts, patterns that generalize beyond one repo description: Query the SecondBrain vault knowledge graph for cross-project evergreen knowledge — conventions, tool behavior, client facts, patterns that generalize beyond one repo. Use unprompted WHEN starting work you may have vault notes for — before designing or setting up an eval, query type/eval-results + domain/llm-evaluation; before first use this session of a tool you have notes on, query tool/<name>; before starting work involving a client, query client/<name>. Don't re-query a facet already queried this session.
--- ---
**Scope:** vault at `~/Documents/SecondBrain` — cross-project evergreen knowledge, not codebase structure. For codebase structure and module relationships in the current repo, use `/os-vault:onboard-project`. **Scope:** vault at `~/Documents/SecondBrain` — cross-project evergreen knowledge, not codebase structure. For codebase structure and module relationships in the current repo, use `/os-vault:onboard-project`.
## When to query unprompted
- Before designing or setting up an eval → query `type/eval-results` + `domain/llm-evaluation`
(and the project facet): what was already run, what the hypotheses/results were, what
mistakes not to repeat.
- Before first use this session of a tool you have vault notes for → query `tool/<name>`.
- Before starting work involving a client → query `client/<name>`.
- Don't re-query a facet already queried this session — the note content is in context.
Use this skill when you need to retrieve evergreen knowledge from the SecondBrain vault (conventions, tool/API behavior, client facts, cross-project patterns). For **episodic questions** ("what did we do last week", "when did X happen"), use memsearch instead. Use this skill when you need to retrieve evergreen knowledge from the SecondBrain vault (conventions, tool/API behavior, client facts, cross-project patterns). For **episodic questions** ("what did we do last week", "when did X happen"), use memsearch instead.
## Query commands ## Query commands

View File

@ -1,20 +1,38 @@
--- ---
description: Write evergreen knowledge to the SecondBrain vault with correct frontmatter and scope description: Write evergreen cross-project knowledge to the SecondBrain vault with correct frontmatter and scope. Use unprompted WHEN a durable fact surfaces mid-task — tool/API behavior discovered, a client convention, a methodology that worked — anything that would change how you act in a DIFFERENT repo. The vault, NOT auto-memory, is the destination for knowledge that generalizes beyond this repo; auto-memory is only for facts about this repo itself.
--- ---
Use this skill when you have knowledge that should persist across projects and sessions. Use this skill when you have knowledge that should persist across projects and sessions.
## Evergreen vs. ephemeral test ## Where knowledge goes — ask in order
**Write to vault if:** the knowledge is reusable across projects (tool/API behavior, client-specific facts, conventions, cross-project patterns). Vault is for knowledge that survives the current session. 1. Does the repo already record it (code, ADR, CLAUDE.md, git history)? → write it there or
nowhere; never duplicate into memory.
2. Would this change how you act in a **different** repo, next year? → the **SecondBrain
vault** via this skill — NOT auto-memory. Facts about the world: tool behavior, client
conventions, API quirks, methodology that worked. The repo is where you *learned* it, not
what it's *about*.
3. Only useful in future sessions of **this** repo (where things live here, what we tried
here)? → auto-memory.
4. Task status, session state, in-flight progress → nothing (the session journal is handled
automatically by the SessionEnd hook).
**Do NOT write to vault:** project-ephemeral state (current task, in-progress decisions, temporary context). These belong in project files or the episodic layer (session journal — handled automatically by SessionEnd hook). Tiebreaker: what is the fact **about**? About a tool/client/API → vault. About this repo →
auto-memory. When a durable fact surfaces mid-task — even with no one saying "remember" —
run the ladder before moving on.
## Vault location ## Vault location — resolve it FIRST, mechanically
The vault root is `~/Documents/SecondBrain` — unless the environment variable `OS_VAULT_PATH` Before writing anything, resolve the vault root with one command and use ONLY the path it
is set, in which case use that path as the vault root instead (rare; tests and evals use it to prints for every read and write in this skill:
point at an isolated vault).
```bash
echo "${OS_VAULT_PATH:-$HOME/Documents/SecondBrain}"
```
Do not assume `~/Documents/SecondBrain``OS_VAULT_PATH` overrides it (tests and evals point
it at an isolated vault, and writing to the default path there contaminates the user's real
vault). Never write a vault note to a hardcoded path.
## Frontmatter contract — read vault-conventions.md first ## Frontmatter contract — read vault-conventions.md first
@ -36,9 +54,26 @@ vault-conventions.md, not this list, if anything looks off):
- Filenames are slug-only lowercase-kebab-case (no date prefix). - Filenames are slug-only lowercase-kebab-case (no date prefix).
- Write `summary` at creation time — it is the primary router hint for graph traversal and - Write `summary` at creation time — it is the primary router hint for graph traversal and
memsearch. Never defer it. memsearch. Never defer it.
- Frontmatter must be parseable YAML: quote any scalar value that contains a colon
(`title: "Chartkit segfaults without streaming: true"` — unquoted, that line breaks the
whole frontmatter block and the note drops out of retrieval).
**Write `summary` now.** Deferring it breaks retrieval. **Write `summary` now.** Deferring it breaks retrieval.
## Update before create
Write is query-first: before creating a note, check for existing notes on the same subject
(grep the vault or use `/os-vault:query`). If the new fact contradicts, extends, or refines
an existing note → update that note and bump `last_updated`. Only a genuinely new subject
gets a new note.
## Content that fits no existing type
Do NOT invent a note type, and do not force a bad fit silently. Write the note under the
closest existing type with a visible `## Type-fit note` line saying what didn't fit — or, if
the shape is clearly recurring, propose a new type via `/os-vault:design-template`. Misfits
get reviewed in `/os-vault:reorganize`; three misfits of the same shape justify a new type.
## Vault-not-repo rule ## Vault-not-repo rule
Write ONLY to the vault root (see Vault location above). Never silently write to a project repository what belongs in the vault. If in doubt, write to the vault. Write ONLY to the vault root (see Vault location above). Never silently write to a project repository what belongs in the vault. If in doubt, write to the vault.