diff --git a/CLAUDE.md b/CLAUDE.md index df341fd..950d996 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -81,7 +81,7 @@ to those two and fix the stale doc. - Hook wiring: `~/.claude/settings.json` (hook entries invoke `/usr/bin/python3` with absolute paths into cc-os) **Graphify** — v0.8.31 at `/home/jared/.local/bin/graphify` -- Vault graph: `~/Documents/SecondBrain/graphify-out/` — disposable, rebuilt by SessionStart hook +- Vault graph: `~/Documents/SecondBrain/graphify-out/` — disposable, structural index rebuilt by SessionStart hook; handles relational/graph queries. Distinct from vault-index.json (planned, Plan B Phase 1): a flat `{tag: [{path, title, summary}]}` lookup rebuilt at SessionEnd, queried by the `/memory-find` skill for fast tag-based SB note discovery. - Project graph: `/graphify-out/` — same pattern; gitignore it in each project repo - Vault conventions: `~/Documents/SecondBrain/vault-conventions.md` — frontmatter contract + tag taxonomy (canonical name decided 2026-06-09; formerly referred to as CONVENTIONS.md) diff --git a/docs/memory-system/09-sb-content-plan.md b/docs/memory-system/09-sb-content-plan.md index 1fd480e..39eb050 100644 --- a/docs/memory-system/09-sb-content-plan.md +++ b/docs/memory-system/09-sb-content-plan.md @@ -1,6 +1,6 @@ # SecondBrain Content Plan (Plan A) -_Status: Planning. Last updated: 2026-06-27._ +_Status: Planning. Last updated: 2026-06-28._ > Plan A of two. Plan A solves the **content problem** — the vault is dormant because it holds > too little valuable knowledge to be worth querying. Plan B (separate doc) solves the @@ -70,6 +70,8 @@ what types always go to SB and where. Project-specific exceptions are stored in SB note (`project/[name].md`) and injected alongside the global rules at SessionStart for that project. +Inline classification works best as opt-out — the AI should assume a type is SB-eligible and skip only when the value gate fails, not the reverse. + ### Layer 2 — SessionEnd catch-all A SessionEnd agent audits the git diff since the last session for knowledge that belongs in SB @@ -79,6 +81,8 @@ but wasn't written inline. The agent uses judgment: - If the type exists but is specific to this project → it becomes a project-specific exception stored in the project note. +**Implementation note:** The Layer 2 "agent" requires an invocation mechanism decision before Plan B Phase 3 is built. See Plan B (`10-sb-findability-plan.md`) Phase 3 prerequisite note. Until this decision is made and built, Layer 1 is the primary write path. + ### The feedback loop Global rules handle obvious cases inline → SessionEnd catches non-obvious ones → catches that diff --git a/docs/memory-system/10-sb-findability-plan.md b/docs/memory-system/10-sb-findability-plan.md index a10affb..d2708e1 100644 --- a/docs/memory-system/10-sb-findability-plan.md +++ b/docs/memory-system/10-sb-findability-plan.md @@ -1,6 +1,6 @@ # SecondBrain Findability Plan -_Last updated: 2026-06-27_ +_Last updated: 2026-06-28_ _Status: Planning_ ## Problem @@ -34,7 +34,7 @@ This plan covers four phases, strictly sequenced — each phase depends on the p ## Surfaces Touched -- **Plugin code:** `cc-os/plugins/memory/hooks/session_start.py` (extend), `cc-os/plugins/memory/hooks/vault_index_rebuild.py` (new), `cc-os/plugins/memory/hooks/session_end.py` (extend for Phase 3 Parts A/B/C) +- **Plugin code:** `cc-os/plugins/memory/hooks/session_context.py` (extend for Phase 2 injection + Phase 3 queue processing), `cc-os/plugins/memory/hooks/vault_index_rebuild.py` (new), `cc-os/plugins/memory/hooks/session_end.py` (extend for Phase 3 queue writing + git SHA capture) - **Skills:** `cc-os/plugins/memory/skills/memory-find.md` (new) - **Vault content:** `~/Documents/SecondBrain/vault-index.json` (new), `~/Documents/SecondBrain/project/` directory (new), hub notes as density warrants - **Settings:** `~/.claude/settings.json` — add `vault_index_rebuild.py` as fourth SessionEnd hook (separate hook, mirrors `memsearch_sync.py` / `vault_sync.py` pattern) @@ -44,11 +44,12 @@ This plan covers four phases, strictly sequenced — each phase depends on the p | Risk | Mitigation | |------|-----------| | **Index staleness** | Rebuild at SessionEnd only (batched). Stale index is acceptable within a session; regenerated within one session boundary. Stale results are surfaced as a known gap, not a silent failure. | -| **SessionEnd agent latency** | Agent runs at session close. Target: under 30 seconds, same constraint as `memsearch_sync.py` and `vault_sync.py`. Cap at 5 candidate notes per run; deeper audits are on-demand via `/memory-audit`. | +| **SessionEnd agent latency** | Phase 3 uses queue-and-defer: SessionEnd writes a candidate queue file (fast Python, <1s), next SessionStart processes it with the already-running LLM. No API call at session close; no terminal blocking. `/memory-audit` skill available for on-demand thorough audits. | | **Duplicate notes** | Phase 3 Part C (pre-action deduplication) adds a tag-intersection check before any note creation. Dedup runs before create — not after. | | **False positives in discrepancy detection** | Phase 3 Part B flags, does not auto-update. Human confirms before any note change. Auto-update deferred until flag-then-confirm workflow is stable and trusted. | | **Hub note timing** | Hub notes should not be created before cluster density exists (3+ related notes). Phase 1 defers hub creation to emergence — noted as a natural byproduct, not a scheduled deliverable. | | **Injection token budget** | SessionStart injection targets ~200 tokens total. If project note + global rules exceed budget, truncate global rules, not the project note. Project-specific context is higher signal at the point of use. | +| **Silent hook failure** | `session_start.py` / `session_context.py`: inject a one-line warning when vault sync or graph rebuild last-success timestamp is older than `stale_threshold_days × 2`. Turn invisible decay into a visible session prompt. | ## Build Order @@ -60,7 +61,7 @@ Two deliverables ship together: Hub notes are deferred from Plan A because their value is relational — they only earn their place once notes are dense enough to navigate between. A hub note is a map, not content: title + one-sentence purpose + bulleted links to member notes with one-line descriptions. Do not create hub notes on a schedule; create them as a natural byproduct when a cluster of 3+ related notes exists. Planned first hubs (after content density warrants): agent-orchestration hub, rails-conventions hub. **vault-index.json** -Location: `~/Documents/SecondBrain/vault-index.json`. Structure: `{tag: [{path, title, summary}]}`. Built by scanning vault notes: frontmatter tags + first 1–2 sentences as summary. Rebuilt at SessionEnd — batched, not real-time, not per-note-write. Tracked in git so it is pushed with every vault sync. +Location: `~/Documents/SecondBrain/vault-index.json`. Structure: `{generated_at: "ISO-8601", tags: {tag: [{path, title, summary}]}}`. Built by scanning vault notes: frontmatter tags + first 1–2 sentences as summary. Rebuilt at SessionEnd — batched, not real-time, not per-note-write. Tracked in git so it is pushed with every vault sync. The `generated_at` field lets `/memory-find` surface staleness (e.g., "index is 3 sessions old") rather than silently serving stale results. **Rebuild hook** Add `vault_index_rebuild.py` as a fourth SessionEnd hook. Hook order at SessionEnd: `session_end.py` → `memsearch_sync.py` → `vault_index_rebuild.py` → `vault_sync.py`. Rebuild runs after vault notes are written but before `vault_sync.py` pushes to Forgejo, so the pushed index is current. @@ -77,7 +78,7 @@ Location: `cc-os/plugins/memory/skills/memory-find.md`. Accepts a tag argument. Trigger: on-demand only. The AI is instructed to call it when it encounters something that might be in SB. Automatic triggering before every action is deferred — too noisy until the index has density. **SessionStart injection** -Extend `session_start.py` to inject two things at session start (~200 tokens total): +Extend `session_context.py` (the UserPromptSubmit hook — `session_start.py` has no injection output channel) to inject two things at session start (~200 tokens total): 1. Global type classification rules (~100 tokens): "SecondBrain is your knowledge layer. Types that always go to SB: convention (rule/pattern), reference (lookup/architecture), howto (procedure). Use `/memory-find [tag]` to query. Use `/memory-recall [path]` to read full notes. DRY: if knowledge already exists in SB, reference it — don't duplicate." 2. Project note (if exists for current project): project name, one-sentence description, tags this project produces. Tells the AI what types to write inline and what to look up. Project note format: `project/[name].md` containing name, one-sentence description, bulleted tag list, and optional `produces:` section for project-specific note types. @@ -85,6 +86,15 @@ Extend `session_start.py` to inject two things at session start (~200 tokens tot Prerequisite: Phase 1 and Phase 2 complete. Runs after `session_end.py` writes the episodic journal. +> **Invocation model: queue-and-defer.** Python subprocess hooks cannot invoke the Claude model inline — the session has ended, and a direct Anthropic API call from a hook would block the terminal for 30–60+ seconds. The solution: split the work across two hooks. +> +> - **SessionEnd** (`session_end.py`, fast Python only): writes a candidate queue file to `~/.claude/memory-queue/YYYY-MM-DD-{session_id}.json` containing the git diff summary, files changed, and session context. No LLM involved. Takes under 1 second. +> - **Next SessionStart** (`session_context.py`): if a queue file exists from a prior session, inject it as a task for the already-running LLM. The LLM processes the queue during normal session startup — no separate API call, no blocking. +> +> Trade-off: there is a one-session lag before notes get created. This is acceptable — the knowledge is captured (in the queue file) immediately; only the SB write is deferred. The `/memory-audit` skill remains available for on-demand thorough audits in the current session. + +> **Prerequisite (git anchor):** Phase 3 Part A's "audit git diff since last session" requires a session-boundary git SHA. This is not currently stored anywhere. Before implementing Phase 3, add a step to `session_end.py` (or the vault journal entry it writes) to record `git rev-parse HEAD` for the active project. Without this, "since last session" has no anchor and the diff scope is undefined. + **Part A — Create new notes** Agent audits git diff since last session for knowledge that belongs in SB but wasn't written inline. Uses value gate (longevity + reusability + mutability) to assess candidates. Agent judgment: global type gap → update global classification rules; project-specific type → add to project note in SB. Creates notes using the appropriate template. Does NOT auto-update existing notes — flag only (see Part B). Cap at 5 candidates per run to stay within latency budget. @@ -95,6 +105,8 @@ When a session executed a procedure already in SB, agent checks actual vs. docum 3. Flags ambiguity — process differed but unclear which is correct; human resolves before any change. Auto-update deferred until flag-then-confirm workflow is stable and trusted. +> **Deferred:** Part B is cut from the initial Phase 3 build. Discrepancy detection is only meaningful when the vault has sufficient procedure note density. It also competes for the same time/token budget as Part A, which has clearer value at this stage. Add Part B when the vault has 25+ howto notes that are actively used. + **Part C — Pre-action deduplication** Before creating a new note, agent checks vault-index.json for tag collision: do any existing notes carry the same primary tags? If yes, surface the existing note first — create only if the existing note doesn't cover the case. Implementation: tag intersection query on vault-index.json (fast, no embeddings needed). This closes the "we already have a note for that" gap. @@ -108,8 +120,10 @@ This phase is complete when the SessionEnd agent's "create" suggestions drop to ## Open Questions -1. **SessionEnd agent token budget:** Target under 30 seconds. If Phase 3 Parts A + B + C together exceed the budget, split: Part C (dedup check) runs inline at note-create time, Parts A + B run as the SessionEnd agent. This preserves deduplication without blocking on the full audit. +1. **Queue file format for Phase 3:** The candidate queue file written by SessionEnd needs a schema. Minimum fields: session_id, timestamp, project_root, git_sha (see git anchor prerequisite), files_changed (list), diff_summary (truncated). Design this schema before building Phase 3. 2. **When does /memory-find trigger automatically vs. on-demand?** V1: on-demand only. The AI is instructed to call it when it encounters something that might be in SB. Automatic triggering (before every action) is deferred — too noisy until the index has density. 3. **Project note bootstrapping:** Who creates the initial project note for a new project? Options: (a) user creates manually using template, (b) `/memory-audit` suggests it, (c) `memory-project` skill creates it during onboarding. Current design: option (c) — `memory-project` skill writes a `project/[name].md` to the vault as part of onboarding. Confirm this in the memory-project skill spec. + +4. **Queue file retention policy:** How long should unprocessed queue files persist? If the next session is in a different project directory, does `session_context.py` still pick up the queue? Simplest answer: always process any queue file regardless of current project — knowledge extraction is global. Retain queue files until processed; delete on successful processing.