os-vault: WS2 wording loop — destination ladder shipped, reserve grid measured
4 iterations (destination ladder -> mechanical vault-root -> CLAUDE.md cross-project-memory section -> ADR-vs-vault disambiguation) + candidates 2-4 unscored + post-reserve YAML-colon quoting rule. Confirm 18/18 sonnet, 16/18 haiku; held-out reserve trigger-axis 13/18, negatives 17/18 (baseline 1/19, 18/18). Checker breach-regex false positive fixed + rescored, 2 self-test guards added. Run-set and reserve both contaminated now. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
jared
parent
638c6d03ed
commit
42860d06d1
|
|
@ -84,6 +84,7 @@ to those two and fix the stale doc.
|
|||
- Isolation seam (2026-07-06, WS2): `OS_VAULT_PATH` env var overrides `vault_path` for the write skill and every hook via `config.load_config()`; `OS_VAULT_SKIP_REBUILD` suppresses the SessionStart graphify rebuild. Added for the eval harness; zero production impact when unset.
|
||||
- Write-behavior eval (2026-07-06, WS2): `plugins/os-vault/eval/` — held-out unprompted vault-write discrimination eval (Eval B/C playbook: ambiguity ladder L1 explicit → L3 conceptual, paired positives/negatives, 6 run-set scenarios on a new `reportgen` Ruby fixture + 6 frozen reserve twins in a different knowledge domain, isolated sandbox vault, headless-only runner, deterministic-first Ruby checker with narrow frozen offer-detection judge fallback stubbable via `OS_VAULT_EVAL_JUDGE_CMD`, model-free `bin/self-test` 21/21). **Scenario Task blocks are held-out — never run informally; the reserve set is never even read informally.** Procedure: `plugins/os-vault/eval/README.md`.
|
||||
- Untuned baseline grid (2026-07-06, run-set × sonnet/haiku × 3 reps + 1 counted canary, 37 reps, zero harness errors): **positives 1/19, negatives 18/18**. Headline: at L1 (explicit cue) both tiers persist the knowledge every rep but route it to Claude Code's built-in auto-memory (`~/.claude/projects/<slug>/memory/`) instead of the vault; L2/L3 mostly don't persist at all; zero over-triggering anywhere. The WS2 wording loop (step 4, NOT started; run-set still uncontaminated) therefore has two targets: trigger at L2/L3, and route to the vault over auto-memory at L1. TSV: `eval/results/2026-07-06-baseline-grid.tsv`; full analysis: vault note [[os-vault-write-eval-baseline-grid-results]].
|
||||
- WS2 wording loop COMPLETE (2026-07-07, 4 iterations + confirm + held-out reserve; run-set AND reserve now contaminated): shipped destination-ladder write SKILL.md + SessionStart USAGE_NOTE (mechanical triggers + before-final-reply checkpoint), mechanical vault-root resolution (`${OS_VAULT_PATH:-$HOME/Documents/SecondBrain}` — fixed real-vault breaches found on iteration 1), fixture CLAUDE.md "Cross-project memory" section (**the real-project adoption template**, required to reach haiku at L2/L3; includes ADR-vs-vault disambiguation), candidates 2–4 unscored, post-reserve YAML-colon quoting rule. Confirm grid sonnet 18/18 / haiku 16/18; reserve (held-out) positives 8/18 strict / trigger-axis 13/18, negatives 17/18 (vs baseline 1/19, 18/18). Residual gaps: YAML-hostile titles, haiku L2 no-trigger, generalization-laundering over-trigger. One checker false-positive fixed+rescored mid-loop (breach regex vs `2>/dev/null`); negative-criterion redesign pre-registered. TSVs: `eval/results/2026-07-07-wording-*.tsv`; loop log: `autoresearch/classic-260707-1253/loop-log.md`; analysis: vault note [[os-vault-write-eval-wording-loop-results]]. Next signal is production IRL audits.
|
||||
|
||||
**Global os-orchestration plugin** — `cc-os/plugins/os-orchestration/` (git-tracked, 2026-07-03); symlinked into `~/.claude/plugins/os-orchestration`
|
||||
- Hooks: `hooks/` — `inject.py` (injects `ORCHESTRATION.md` as additionalContext to all sessions)
|
||||
|
|
|
|||
|
|
@ -135,8 +135,12 @@ module OsVaultEval
|
|||
return false unless command.match?(REAL_VAULT_PATTERN)
|
||||
|
||||
# Plain reads (cat/ls/grep/find) of the real vault are not a breach —
|
||||
# only flag when the command also writes INTO that path.
|
||||
command.match?(/>>?[^&]*(?:SecondBrain)/) ||
|
||||
# only flag when the command also writes INTO that path. The redirect
|
||||
# TARGET itself must be under SecondBrain (a `2>/dev/null` followed by a
|
||||
# later read-only mention of the vault path is not a write — false
|
||||
# positive found on iteration 1, 2026-07-07; rescored per the
|
||||
# instrument-fix rule, transcripts untouched).
|
||||
command.match?(/>>?\s*["']?[^\s"'|;&<>]*SecondBrain/) ||
|
||||
command.match?(/\btee\b[^|]*SecondBrain/) ||
|
||||
command.match?(/\b(?:cp|mv|rsync)\b.*SecondBrain/)
|
||||
end
|
||||
|
|
|
|||
|
|
@ -288,6 +288,43 @@ expect FAIL "breach (Write to real vault path)" \
|
|||
env OS_VAULT_EVAL_JUDGE_CMD="printf PASS" "$EVAL_ROOT/bin/check" P1-L1-reporting "$sb"
|
||||
expect_contains "$row" "real-vault-breach" "breach records real-vault-breach"
|
||||
|
||||
# --- 9b. NOT a breach: read-only Bash mentioning the real vault path with an
|
||||
# unrelated redirect (2>/dev/null) must not trip the redirect heuristic, and a
|
||||
# sandbox-vault write alongside it still PASSes (iteration-1 false positive,
|
||||
# 2026-07-07). A Bash redirect whose TARGET is under the real vault stays FAIL. ---
|
||||
|
||||
sb="$WORKDIR/breach-fp"
|
||||
make_sandbox "$sb"
|
||||
mkdir -p "$sb/vault/reference"
|
||||
cat > "$sb/vault/reference/finch-api-page-size-silent-cap.md" <<'NOTE'
|
||||
---
|
||||
type: reference
|
||||
title: Finch API silently caps page_size at 500
|
||||
summary: Finch API caps every response at 500 records unless page_size is passed explicitly.
|
||||
scope: global
|
||||
tags:
|
||||
- type/reference
|
||||
- tool/finch-api
|
||||
---
|
||||
Finch caps responses at 500 records (pagination required).
|
||||
NOTE
|
||||
write_transcript "$sb" <<EOF
|
||||
$(tool_use_event "Bash" '{"command":"echo \"OS_VAULT_PATH=$OS_VAULT_PATH\"; ls ~/Documents/SecondBrain 2>/dev/null | head"}')
|
||||
$(tool_use_event "Write" '{"file_path":"SANDBOX/vault/reference/finch-api-page-size-silent-cap.md","content":"..."}')
|
||||
$(result_event "Recorded the Finch page-size cap in the vault.")
|
||||
EOF
|
||||
expect PASS "read-only real-vault mention with 2>/dev/null is not a breach" \
|
||||
env OS_VAULT_EVAL_JUDGE_CMD="printf FAIL" "$EVAL_ROOT/bin/check" P1-L1-reporting "$sb"
|
||||
|
||||
sb="$WORKDIR/breach-redirect"
|
||||
make_sandbox "$sb"
|
||||
write_transcript "$sb" <<EOF
|
||||
$(tool_use_event "Bash" '{"command":"echo note > ~/Documents/SecondBrain/x.md"}')
|
||||
$(result_event "Saved a note.")
|
||||
EOF
|
||||
expect FAIL "Bash redirect INTO real vault is still a breach" \
|
||||
env OS_VAULT_EVAL_JUDGE_CMD="printf PASS" "$EVAL_ROOT/bin/check" P1-L1-reporting "$sb"
|
||||
|
||||
# --- 10. Harness error: sandbox without transcript.jsonl -> FAIL ---
|
||||
|
||||
sb="$WORKDIR/no-transcript"
|
||||
|
|
|
|||
|
|
@ -33,3 +33,20 @@ every file, one class per file under `lib/reportgen/`.
|
|||
## Architecture decisions
|
||||
|
||||
Architecture decisions for this project are recorded in `docs/adr/`.
|
||||
|
||||
## Cross-project memory
|
||||
|
||||
Durable knowledge that is not about this repo goes to the SecondBrain vault via
|
||||
`/os-vault:write` — not into this repo, and not into auto-memory.
|
||||
|
||||
- **When** you discover during a task that a tool, API, or service behaves differently from
|
||||
what its docs or config claim (advertised X, actually Y) → that discrepancy is evergreen
|
||||
vault knowledge. Run `/os-vault:write` before you consider the task complete.
|
||||
- **When** the user states a client-specific requirement, standard, or standing preference →
|
||||
record it with `/os-vault:write` (scope: client). An ADR records how *this repo's code*
|
||||
implements it; the client's requirement itself is client knowledge and an ADR does **not**
|
||||
satisfy this rule — the vault note is still required.
|
||||
- Mechanical rule: **before your final reply on any task**, check whether either of the above
|
||||
occurred this session; if yes, the task is not complete until the vault note is written.
|
||||
- Ephemeral material stays out of the vault: task status, in-progress refactor state,
|
||||
project-local quirks, and anything this repo's docs or ADRs already record.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,36 @@
|
|||
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N2-L2-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P2-L2-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N2-L2-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
N1-L1-reporting sonnet W:no B:PASS PASS project-write:no
|
||||
P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N2-L2-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
P2-L2-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P3-L3-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N2-L2-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
N2-L2-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P3-L3-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N2-L2-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
N3-L3-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
P2-L2-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N3-L3-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P3-L3-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N3-L3-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
N3-L3-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
N3-L3-reporting sonnet W:yes B:FAIL FAIL wrote:ruby-time-getlocal-offset-only.md; project-write:yes
|
||||
P3-L3-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N3-L3-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
|
|
|
@ -0,0 +1,24 @@
|
|||
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
P1-L1-reporting haiku A:FAIL B:FAIL FAIL real-vault-breach; real-vault-breach
|
||||
P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming; rescored:breach-regex-false-positive
|
||||
P2-L2-reporting sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P1-L1-reporting haiku A:PASS B:PASS PASS A-via:offered; B-via:offered
|
||||
P3-L3-reporting sonnet A:FAIL B:FAIL FAIL real-vault-breach; real-vault-breach
|
||||
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
P1-L1-reporting haiku A:FAIL B:FAIL FAIL real-vault-breach; real-vault-breach
|
||||
P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P2-L2-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P2-L2-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P2-L2-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P3-L3-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P3-L3-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
|
|
|
@ -0,0 +1,24 @@
|
|||
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P2-L2-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N1-L1-reporting haiku W:no B:PASS PASS project-write:no
|
||||
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P2-L2-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
P1-L1-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P3-L3-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P2-L2-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P3-L3-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P3-L3-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
|
|
|
@ -0,0 +1,18 @@
|
|||
N1-L1-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
N1-L1-reporting haiku W:no B:PASS PASS project-write:no
|
||||
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N1-L1-reporting haiku W:no B:PASS PASS project-write:no
|
||||
P2-L2-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N1-L1-reporting sonnet W:no B:PASS PASS project-write:yes
|
||||
P2-L2-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P1-L1-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P2-L2-reporting sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P2-L2-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P3-L3-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P3-L3-reporting haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
|
|
|
@ -0,0 +1,6 @@
|
|||
P3-L3-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N3-L3-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
P3-L3-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P3-L3-reporting haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N3-L3-reporting haiku W:no B:PASS PASS project-write:yes
|
||||
N3-L3-reporting haiku W:no B:PASS PASS project-write:no
|
||||
|
|
|
@ -0,0 +1,36 @@
|
|||
N4-L1-delivery haiku W:no B:PASS PASS project-write:no
|
||||
N5-L2-delivery haiku W:no B:PASS PASS project-write:yes
|
||||
N4-L1-delivery sonnet W:no B:PASS PASS project-write:yes
|
||||
P4-L1-delivery sonnet A:PASS B:FAIL FAIL A-via:written; no parseable YAML frontmatter block
|
||||
N5-L2-delivery haiku W:no B:PASS PASS project-write:yes
|
||||
N4-L1-delivery haiku W:no B:PASS PASS project-write:no
|
||||
P4-L1-delivery haiku A:PASS B:FAIL FAIL A-via:written; no parseable YAML frontmatter block
|
||||
P5-L2-delivery haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
N6-L3-delivery haiku W:no B:PASS PASS project-write:yes
|
||||
N4-L1-delivery haiku W:no B:PASS PASS project-write:yes
|
||||
N6-L3-delivery sonnet W:no B:PASS PASS project-write:yes
|
||||
N5-L2-delivery haiku W:no B:PASS PASS project-write:yes
|
||||
N4-L1-delivery sonnet W:no B:PASS PASS project-write:yes
|
||||
P4-L1-delivery sonnet A:PASS B:FAIL FAIL A-via:written; no parseable YAML frontmatter block
|
||||
P4-L1-delivery haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P5-L2-delivery haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
N4-L1-delivery sonnet W:no B:PASS PASS project-write:yes
|
||||
P4-L1-delivery sonnet A:PASS B:FAIL FAIL A-via:written; no parseable YAML frontmatter block
|
||||
N5-L2-delivery sonnet W:no B:PASS PASS project-write:yes
|
||||
N6-L3-delivery sonnet W:no B:PASS PASS project-write:yes
|
||||
P4-L1-delivery haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N6-L3-delivery haiku W:no B:PASS PASS project-write:yes
|
||||
N5-L2-delivery sonnet W:yes B:FAIL FAIL wrote:convention/hardcoded-env-literals-survive-environment-splits.md; project-write:yes
|
||||
P5-L2-delivery haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
N6-L3-delivery sonnet W:no B:PASS PASS project-write:yes
|
||||
P6-L3-delivery haiku A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
P5-L2-delivery sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N6-L3-delivery haiku W:no B:PASS PASS project-write:yes
|
||||
P6-L3-delivery sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P5-L2-delivery sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
N5-L2-delivery sonnet W:no B:PASS PASS project-write:yes
|
||||
P6-L3-delivery haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P5-L2-delivery sonnet A:PASS B:FAIL FAIL A-via:written; missing or empty title; b-info:missing-last_updated
|
||||
P6-L3-delivery haiku A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P6-L3-delivery sonnet A:PASS B:PASS PASS A-via:written; B-via:conforming
|
||||
P6-L3-delivery sonnet A:FAIL B:FAIL FAIL no qualifying vault write and no vault-write offer in final message; axis (a) failed — nothing to check for contract conformance
|
||||
|
|
|
@ -4,6 +4,7 @@
|
|||
Must return sub-second; heavy work is detached in the background. Context
|
||||
injection is handled by session_context.py (UserPromptSubmit hook).
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
|
|
@ -16,6 +17,39 @@ import hook_io
|
|||
|
||||
HOOK_LOG = "/tmp/memory-hook.log"
|
||||
|
||||
# Trigger-conditioned destination ladder (WS2 wording loop, 2026-07-07).
|
||||
# Injected as additionalContext every session; keep it short — it is a
|
||||
# per-session token cost in every project.
|
||||
USAGE_NOTE = (
|
||||
"[os-vault] Where knowledge goes — ask in order: "
|
||||
"(1) the repo already records it (code, ADR, CLAUDE.md)? -> write it there or "
|
||||
"nowhere; never duplicate into memory. "
|
||||
"(2) would it change how you act in a DIFFERENT repo next year (tool/API "
|
||||
"behavior discovered, client conventions, methodology that worked)? -> the "
|
||||
"SecondBrain vault via /os-vault:write — NOT auto-memory; the repo is where "
|
||||
"you learned it, not what it's about. "
|
||||
"(3) only useful in future sessions of THIS repo? -> auto-memory. "
|
||||
"(4) task status / in-flight state -> nothing. "
|
||||
"Mechanical triggers — treat these as part of the task itself: "
|
||||
"WHEN you discover a tool/API/service behaves differently from what its "
|
||||
"docs or config claim (advertised X, actually Y) -> that discrepancy is "
|
||||
"vault knowledge, /os-vault:write it. "
|
||||
"WHEN the user states a client-specific requirement or standing preference "
|
||||
"-> vault note, scope: client. "
|
||||
"BEFORE your final reply on any task -> check: did either occur this "
|
||||
"session? If yes, the task is NOT COMPLETE until /os-vault:write has "
|
||||
"captured it. Ephemeral status and in-progress state never go to the vault."
|
||||
)
|
||||
|
||||
|
||||
def _emit_usage_note():
|
||||
print(json.dumps({
|
||||
"hookSpecificOutput": {
|
||||
"hookEventName": "SessionStart",
|
||||
"additionalContext": USAGE_NOTE,
|
||||
}
|
||||
}))
|
||||
|
||||
|
||||
def _utc():
|
||||
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
|
|
@ -46,6 +80,8 @@ def main():
|
|||
|
||||
cfg = config_mod.load_config()
|
||||
|
||||
_emit_usage_note()
|
||||
|
||||
if os.environ.get("OS_VAULT_SKIP_REBUILD"):
|
||||
_log(f"[{_utc()}] session-start: rebuild skipped (OS_VAULT_SKIP_REBUILD set)")
|
||||
return
|
||||
|
|
|
|||
|
|
@ -1,9 +1,18 @@
|
|||
---
|
||||
description: Query the SecondBrain vault knowledge graph for cross-project evergreen knowledge — conventions, tool behavior, client facts, patterns that generalize beyond one repo
|
||||
description: Query the SecondBrain vault knowledge graph for cross-project evergreen knowledge — conventions, tool behavior, client facts, patterns that generalize beyond one repo. Use unprompted WHEN starting work you may have vault notes for — before designing or setting up an eval, query type/eval-results + domain/llm-evaluation; before first use this session of a tool you have notes on, query tool/<name>; before starting work involving a client, query client/<name>. Don't re-query a facet already queried this session.
|
||||
---
|
||||
|
||||
**Scope:** vault at `~/Documents/SecondBrain` — cross-project evergreen knowledge, not codebase structure. For codebase structure and module relationships in the current repo, use `/os-vault:onboard-project`.
|
||||
|
||||
## When to query unprompted
|
||||
|
||||
- Before designing or setting up an eval → query `type/eval-results` + `domain/llm-evaluation`
|
||||
(and the project facet): what was already run, what the hypotheses/results were, what
|
||||
mistakes not to repeat.
|
||||
- Before first use this session of a tool you have vault notes for → query `tool/<name>`.
|
||||
- Before starting work involving a client → query `client/<name>`.
|
||||
- Don't re-query a facet already queried this session — the note content is in context.
|
||||
|
||||
Use this skill when you need to retrieve evergreen knowledge from the SecondBrain vault (conventions, tool/API behavior, client facts, cross-project patterns). For **episodic questions** ("what did we do last week", "when did X happen"), use memsearch instead.
|
||||
|
||||
## Query commands
|
||||
|
|
|
|||
|
|
@ -1,20 +1,38 @@
|
|||
---
|
||||
description: Write evergreen knowledge to the SecondBrain vault with correct frontmatter and scope
|
||||
description: Write evergreen cross-project knowledge to the SecondBrain vault with correct frontmatter and scope. Use unprompted WHEN a durable fact surfaces mid-task — tool/API behavior discovered, a client convention, a methodology that worked — anything that would change how you act in a DIFFERENT repo. The vault, NOT auto-memory, is the destination for knowledge that generalizes beyond this repo; auto-memory is only for facts about this repo itself.
|
||||
---
|
||||
|
||||
Use this skill when you have knowledge that should persist across projects and sessions.
|
||||
|
||||
## Evergreen vs. ephemeral test
|
||||
## Where knowledge goes — ask in order
|
||||
|
||||
**Write to vault if:** the knowledge is reusable across projects (tool/API behavior, client-specific facts, conventions, cross-project patterns). Vault is for knowledge that survives the current session.
|
||||
1. Does the repo already record it (code, ADR, CLAUDE.md, git history)? → write it there or
|
||||
nowhere; never duplicate into memory.
|
||||
2. Would this change how you act in a **different** repo, next year? → the **SecondBrain
|
||||
vault** via this skill — NOT auto-memory. Facts about the world: tool behavior, client
|
||||
conventions, API quirks, methodology that worked. The repo is where you *learned* it, not
|
||||
what it's *about*.
|
||||
3. Only useful in future sessions of **this** repo (where things live here, what we tried
|
||||
here)? → auto-memory.
|
||||
4. Task status, session state, in-flight progress → nothing (the session journal is handled
|
||||
automatically by the SessionEnd hook).
|
||||
|
||||
**Do NOT write to vault:** project-ephemeral state (current task, in-progress decisions, temporary context). These belong in project files or the episodic layer (session journal — handled automatically by SessionEnd hook).
|
||||
Tiebreaker: what is the fact **about**? About a tool/client/API → vault. About this repo →
|
||||
auto-memory. When a durable fact surfaces mid-task — even with no one saying "remember" —
|
||||
run the ladder before moving on.
|
||||
|
||||
## Vault location
|
||||
## Vault location — resolve it FIRST, mechanically
|
||||
|
||||
The vault root is `~/Documents/SecondBrain` — unless the environment variable `OS_VAULT_PATH`
|
||||
is set, in which case use that path as the vault root instead (rare; tests and evals use it to
|
||||
point at an isolated vault).
|
||||
Before writing anything, resolve the vault root with one command and use ONLY the path it
|
||||
prints for every read and write in this skill:
|
||||
|
||||
```bash
|
||||
echo "${OS_VAULT_PATH:-$HOME/Documents/SecondBrain}"
|
||||
```
|
||||
|
||||
Do not assume `~/Documents/SecondBrain` — `OS_VAULT_PATH` overrides it (tests and evals point
|
||||
it at an isolated vault, and writing to the default path there contaminates the user's real
|
||||
vault). Never write a vault note to a hardcoded path.
|
||||
|
||||
## Frontmatter contract — read vault-conventions.md first
|
||||
|
||||
|
|
@ -36,9 +54,26 @@ vault-conventions.md, not this list, if anything looks off):
|
|||
- Filenames are slug-only lowercase-kebab-case (no date prefix).
|
||||
- Write `summary` at creation time — it is the primary router hint for graph traversal and
|
||||
memsearch. Never defer it.
|
||||
- Frontmatter must be parseable YAML: quote any scalar value that contains a colon
|
||||
(`title: "Chartkit segfaults without streaming: true"` — unquoted, that line breaks the
|
||||
whole frontmatter block and the note drops out of retrieval).
|
||||
|
||||
**Write `summary` now.** Deferring it breaks retrieval.
|
||||
|
||||
## Update before create
|
||||
|
||||
Write is query-first: before creating a note, check for existing notes on the same subject
|
||||
(grep the vault or use `/os-vault:query`). If the new fact contradicts, extends, or refines
|
||||
an existing note → update that note and bump `last_updated`. Only a genuinely new subject
|
||||
gets a new note.
|
||||
|
||||
## Content that fits no existing type
|
||||
|
||||
Do NOT invent a note type, and do not force a bad fit silently. Write the note under the
|
||||
closest existing type with a visible `## Type-fit note` line saying what didn't fit — or, if
|
||||
the shape is clearly recurring, propose a new type via `/os-vault:design-template`. Misfits
|
||||
get reviewed in `/os-vault:reorganize`; three misfits of the same shape justify a new type.
|
||||
|
||||
## Vault-not-repo rule
|
||||
|
||||
Write ONLY to the vault root (see Vault location above). Never silently write to a project repository what belongs in the vault. If in doubt, write to the vault.
|
||||
|
|
|
|||
Loading…
Reference in New Issue