130 lines
13 KiB
Markdown
130 lines
13 KiB
Markdown
# SecondBrain Findability Plan
|
||
|
||
_Last updated: 2026-06-28_
|
||
_Status: Planning_
|
||
|
||
## Problem
|
||
|
||
Plan A (`09-sb-content-plan.md`) populates the vault with typed, templated notes. But notes that exist and cannot be found are no better than notes that do not exist. The vault needs three complementary findability layers: a queryable index so the AI can discover notes by tag without reading every file; session-start injection so the AI knows the vault exists and how to query it from the first prompt; and an end-of-session agent that closes the knowledge loop — creating notes that should have been written, detecting drift between documented and actual procedures, and preventing duplicate notes before they are created. Without these layers, the vault is a write-only system. This plan builds the read and update path.
|
||
|
||
## Design Principles
|
||
|
||
- **Vault as library, not navigation:** Don't pre-compute "when to read X." Make querying cheap and let the AI decide at the point of need. Query results surface paths + summaries; AI reads summaries and decides which notes to fetch.
|
||
- **Minimal injection overhead:** SessionStart targets ~200 tokens total for all vault context. No embedded links or pre-computed indexes in project repos.
|
||
- **Projects contain product; SecondBrain contains knowledge:** Project repos contain no SB infrastructure — project notes live in SB, not in repos.
|
||
- **Experience-driven updates:** When a session executes a documented procedure and finds a discrepancy, update the note. Not date-based.
|
||
|
||
## Scope
|
||
|
||
This plan covers four phases, strictly sequenced — each phase depends on the previous:
|
||
|
||
1. Hub/index type + vault-index.json
|
||
2. `/memory-find` skill + SessionStart injection
|
||
3. SessionEnd agent (create + detect + dedup)
|
||
4. Feedback loop hardening
|
||
|
||
## Out of Scope
|
||
|
||
- **Atomic note splitting:** Large notes from Plan A that exceed focus — deferred to Plan C and beyond.
|
||
- **SQLite/FTS upgrade:** When vault-index.json query latency becomes measurable (>200 notes or >15 matches per query consistently).
|
||
- **Auto-update without human confirmation:** Deferred until flag-then-confirm is stable and trusted.
|
||
- **SessionStart vault pull:** Multi-machine sync; push-only is the current locked design.
|
||
- **Meeting, task, QoL opportunity note types:** Require findability layer to be useful; define after Plan B is operational.
|
||
- **memsearch ↔ SB integration:** memsearch (episodic) and SB (semantic) remain separate by design (ADR-006, ADR-010).
|
||
|
||
## Surfaces Touched
|
||
|
||
- **Plugin code:** `cc-os/plugins/memory/hooks/session_context.py` (extend for Phase 2 injection + Phase 3 queue processing), `cc-os/plugins/memory/hooks/vault_index_rebuild.py` (new), `cc-os/plugins/memory/hooks/session_end.py` (extend for Phase 3 queue writing + git SHA capture)
|
||
- **Skills:** `cc-os/plugins/memory/skills/memory-find.md` (new)
|
||
- **Vault content:** `~/Documents/SecondBrain/vault-index.json` (new), `~/Documents/SecondBrain/project/` directory (new), hub notes as density warrants
|
||
- **Settings:** `~/.claude/settings.json` — add `vault_index_rebuild.py` as fourth SessionEnd hook (separate hook, mirrors `memsearch_sync.py` / `vault_sync.py` pattern)
|
||
|
||
## Risks
|
||
|
||
| Risk | Mitigation |
|
||
|------|-----------|
|
||
| **Index staleness** | Rebuild at SessionEnd only (batched). Stale index is acceptable within a session; regenerated within one session boundary. Stale results are surfaced as a known gap, not a silent failure. |
|
||
| **SessionEnd agent latency** | Phase 3 uses queue-and-defer: SessionEnd writes a candidate queue file (fast Python, <1s), next SessionStart processes it with the already-running LLM. No API call at session close; no terminal blocking. `/memory-audit` skill available for on-demand thorough audits. |
|
||
| **Duplicate notes** | Phase 3 Part C (pre-action deduplication) adds a tag-intersection check before any note creation. Dedup runs before create — not after. |
|
||
| **False positives in discrepancy detection** | Phase 3 Part B flags, does not auto-update. Human confirms before any note change. Auto-update deferred until flag-then-confirm workflow is stable and trusted. |
|
||
| **Hub note timing** | Hub notes should not be created before cluster density exists (3+ related notes). Phase 1 defers hub creation to emergence — noted as a natural byproduct, not a scheduled deliverable. |
|
||
| **Injection token budget** | SessionStart injection targets ~200 tokens total. If project note + global rules exceed budget, truncate global rules, not the project note. Project-specific context is higher signal at the point of use. |
|
||
| **Silent hook failure** | `session_start.py` / `session_context.py`: inject a one-line warning when vault sync or graph rebuild last-success timestamp is older than `stale_threshold_days × 2`. Turn invisible decay into a visible session prompt. |
|
||
|
||
## Build Order
|
||
|
||
### Phase 1: Hub/index type + vault-index.json (Size: M)
|
||
|
||
Two deliverables ship together:
|
||
|
||
**Hub/index note type**
|
||
Hub notes are deferred from Plan A because their value is relational — they only earn their place once notes are dense enough to navigate between. A hub note is a map, not content: title + one-sentence purpose + bulleted links to member notes with one-line descriptions. Do not create hub notes on a schedule; create them as a natural byproduct when a cluster of 3+ related notes exists. Planned first hubs (after content density warrants): agent-orchestration hub, rails-conventions hub.
|
||
|
||
**vault-index.json**
|
||
Location: `~/Documents/SecondBrain/vault-index.json`. Structure: `{generated_at: "ISO-8601", tags: {tag: [{path, title, summary}]}}`. Built by scanning vault notes: frontmatter tags + first 1–2 sentences as summary. Rebuilt at SessionEnd — batched, not real-time, not per-note-write. Tracked in git so it is pushed with every vault sync. The `generated_at` field lets `/memory-find` surface staleness (e.g., "index is 3 sessions old") rather than silently serving stale results.
|
||
|
||
**Rebuild hook**
|
||
Add `vault_index_rebuild.py` as a fourth SessionEnd hook. Hook order at SessionEnd: `session_end.py` → `memsearch_sync.py` → `vault_index_rebuild.py` → `vault_sync.py`. Rebuild runs after vault notes are written but before `vault_sync.py` pushes to Forgejo, so the pushed index is current.
|
||
|
||
Decision: separate hook (not extending `session_end.py`) — mirrors `memsearch_sync.py` / `vault_sync.py` pattern, cleaner separation of concerns.
|
||
|
||
### Phase 2: /memory-find skill + SessionStart injection (Size: S)
|
||
|
||
Prerequisite: Phase 1 complete (vault-index.json exists).
|
||
|
||
**/memory-find skill**
|
||
Location: `cc-os/plugins/memory/skills/memory-find.md`. Accepts a tag argument. Loads vault-index.json. Filters for matching tags. Returns bulleted list: path, title, summary — one line per match. No embedded note fetching — AI reads summaries and decides which notes to open with Read tool calls or `/memory-recall`. No hard result limit in V1; revisit if queries consistently return 15+ matches.
|
||
|
||
Trigger: on-demand only. The AI is instructed to call it when it encounters something that might be in SB. Automatic triggering before every action is deferred — too noisy until the index has density.
|
||
|
||
**SessionStart injection**
|
||
Extend `session_context.py` (the UserPromptSubmit hook — `session_start.py` has no injection output channel) to inject two things at session start (~200 tokens total):
|
||
1. Global type classification rules (~100 tokens): "SecondBrain is your knowledge layer. Types that always go to SB: convention (rule/pattern), reference (lookup/architecture), howto (procedure). Use `/memory-find [tag]` to query. Use `/memory-recall [path]` to read full notes. DRY: if knowledge already exists in SB, reference it — don't duplicate."
|
||
2. Project note (if exists for current project): project name, one-sentence description, tags this project produces. Tells the AI what types to write inline and what to look up. Project note format: `project/[name].md` containing name, one-sentence description, bulleted tag list, and optional `produces:` section for project-specific note types.
|
||
|
||
### Phase 3: SessionEnd agent (Size: L)
|
||
|
||
Prerequisite: Phase 1 and Phase 2 complete. Runs after `session_end.py` writes the episodic journal.
|
||
|
||
> **Invocation model: queue-and-defer.** Python subprocess hooks cannot invoke the Claude model inline — the session has ended, and a direct Anthropic API call from a hook would block the terminal for 30–60+ seconds. The solution: split the work across two hooks.
|
||
>
|
||
> - **SessionEnd** (`session_end.py`, fast Python only): writes a candidate queue file to `~/.claude/memory-queue/YYYY-MM-DD-{session_id}.json` containing the git diff summary, files changed, and session context. No LLM involved. Takes under 1 second.
|
||
> - **Next SessionStart** (`session_context.py`): if a queue file exists from a prior session, inject it as a task for the already-running LLM. The LLM processes the queue during normal session startup — no separate API call, no blocking.
|
||
>
|
||
> Trade-off: there is a one-session lag before notes get created. This is acceptable — the knowledge is captured (in the queue file) immediately; only the SB write is deferred. The `/memory-audit` skill remains available for on-demand thorough audits in the current session.
|
||
|
||
> **Prerequisite (git anchor):** Phase 3 Part A's "audit git diff since last session" requires a session-boundary git SHA. This is not currently stored anywhere. Before implementing Phase 3, add a step to `session_end.py` (or the vault journal entry it writes) to record `git rev-parse HEAD` for the active project. Without this, "since last session" has no anchor and the diff scope is undefined.
|
||
|
||
**Part A — Create new notes**
|
||
Agent audits git diff since last session for knowledge that belongs in SB but wasn't written inline. Uses value gate (longevity + reusability + mutability) to assess candidates. Agent judgment: global type gap → update global classification rules; project-specific type → add to project note in SB. Creates notes using the appropriate template. Does NOT auto-update existing notes — flag only (see Part B). Cap at 5 candidates per run to stay within latency budget.
|
||
|
||
**Part B — Detect discrepancies with existing notes**
|
||
When a session executed a procedure already in SB, agent checks actual vs. documented. Three outcomes:
|
||
1. Confirms — process matched the note, nothing to do.
|
||
2. Flags for update — process differed; agent reports the discrepancy and proposed change; human confirms before note is updated.
|
||
3. Flags ambiguity — process differed but unclear which is correct; human resolves before any change.
|
||
Auto-update deferred until flag-then-confirm workflow is stable and trusted.
|
||
|
||
> **Deferred:** Part B is cut from the initial Phase 3 build. Discrepancy detection is only meaningful when the vault has sufficient procedure note density. It also competes for the same time/token budget as Part A, which has clearer value at this stage. Add Part B when the vault has 25+ howto notes that are actively used.
|
||
|
||
**Part C — Pre-action deduplication**
|
||
Before creating a new note, agent checks vault-index.json for tag collision: do any existing notes carry the same primary tags? If yes, surface the existing note first — create only if the existing note doesn't cover the case. Implementation: tag intersection query on vault-index.json (fast, no embeddings needed). This closes the "we already have a note for that" gap.
|
||
|
||
### Phase 4: Feedback loop hardening (Size: S)
|
||
|
||
Prerequisite: Phase 3 stable.
|
||
|
||
Project-specific type exceptions discovered by the SessionEnd agent are written to the project's SB note (`project/[name].md`) under a `produces:` section. Next SessionStart for that project injects these exceptions alongside global rules. Over time: global rules handle obvious cases → SessionEnd catches less → per-project notes accumulate correct type exceptions → system approaches self-managing.
|
||
|
||
This phase is complete when the SessionEnd agent's "create" suggestions drop to near zero for a project — meaning global and per-project rules have absorbed the common cases.
|
||
|
||
## Open Questions
|
||
|
||
1. **Queue file format for Phase 3:** The candidate queue file written by SessionEnd needs a schema. Minimum fields: session_id, timestamp, project_root, git_sha (see git anchor prerequisite), files_changed (list), diff_summary (truncated). Design this schema before building Phase 3.
|
||
|
||
2. **When does /memory-find trigger automatically vs. on-demand?** V1: on-demand only. The AI is instructed to call it when it encounters something that might be in SB. Automatic triggering (before every action) is deferred — too noisy until the index has density.
|
||
|
||
3. **Project note bootstrapping:** Who creates the initial project note for a new project? Options: (a) user creates manually using template, (b) `/memory-audit` suggests it, (c) `memory-project` skill creates it during onboarding. Current design: option (c) — `memory-project` skill writes a `project/[name].md` to the vault as part of onboarding. Confirm this in the memory-project skill spec.
|
||
|
||
4. **Queue file retention policy:** How long should unprocessed queue files persist? If the next session is in a different project directory, does `session_context.py` still pick up the queue? Simplest answer: always process any queue file regardless of current project — knowledge extraction is global. Retain queue files until processed; delete on successful processing.
|