Harden SB content + findability plans via round-table review
Key changes: - Plan B Phase 3: replace on-demand-only framing with queue-and-defer pattern (SessionEnd writes candidate queue fast in Python; next SessionStart processes it with already-running LLM — no API call, no terminal block, one-session lag is acceptable) - Plan B Phase 3: cut Part B (discrepancy detection) until vault has 25+ active howto notes - Plan B Phase 2: fix injection hook name (session_context.py, not session_start.py — the latter has no output channel) - Plan B Phase 1: add generated_at timestamp to vault-index.json structure so /memory-find can surface staleness - Plan B: add silent hook failure risk row to Risks table - Plan B: update Open Questions to reflect queue-and-defer decisions - Plan A Layer 1: clarify opt-out framing for inline classification - Plan A Layer 2: add cross-ref to Plan B Phase 3 invocation decision - CLAUDE.md: clarify Graphify vault graph vs vault-index.json roles Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
c582bb1928
commit
4a87acc97f
|
|
@ -81,7 +81,7 @@ to those two and fix the stale doc.
|
|||
- Hook wiring: `~/.claude/settings.json` (hook entries invoke `/usr/bin/python3` with absolute paths into cc-os)
|
||||
|
||||
**Graphify** — v0.8.31 at `/home/jared/.local/bin/graphify`
|
||||
- Vault graph: `~/Documents/SecondBrain/graphify-out/` — disposable, rebuilt by SessionStart hook
|
||||
- Vault graph: `~/Documents/SecondBrain/graphify-out/` — disposable, structural index rebuilt by SessionStart hook; handles relational/graph queries. Distinct from vault-index.json (planned, Plan B Phase 1): a flat `{tag: [{path, title, summary}]}` lookup rebuilt at SessionEnd, queried by the `/memory-find` skill for fast tag-based SB note discovery.
|
||||
- Project graph: `<project-root>/graphify-out/` — same pattern; gitignore it in each project repo
|
||||
- Vault conventions: `~/Documents/SecondBrain/vault-conventions.md` — frontmatter contract + tag taxonomy (canonical name decided 2026-06-09; formerly referred to as CONVENTIONS.md)
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# SecondBrain Content Plan (Plan A)
|
||||
|
||||
_Status: Planning. Last updated: 2026-06-27._
|
||||
_Status: Planning. Last updated: 2026-06-28._
|
||||
|
||||
> Plan A of two. Plan A solves the **content problem** — the vault is dormant because it holds
|
||||
> too little valuable knowledge to be worth querying. Plan B (separate doc) solves the
|
||||
|
|
@ -70,6 +70,8 @@ what types always go to SB and where. Project-specific exceptions are stored in
|
|||
SB note (`project/[name].md`) and injected alongside the global rules at SessionStart for that
|
||||
project.
|
||||
|
||||
Inline classification works best as opt-out — the AI should assume a type is SB-eligible and skip only when the value gate fails, not the reverse.
|
||||
|
||||
### Layer 2 — SessionEnd catch-all
|
||||
|
||||
A SessionEnd agent audits the git diff since the last session for knowledge that belongs in SB
|
||||
|
|
@ -79,6 +81,8 @@ but wasn't written inline. The agent uses judgment:
|
|||
- If the type exists but is specific to this project → it becomes a project-specific exception
|
||||
stored in the project note.
|
||||
|
||||
**Implementation note:** The Layer 2 "agent" requires an invocation mechanism decision before Plan B Phase 3 is built. See Plan B (`10-sb-findability-plan.md`) Phase 3 prerequisite note. Until this decision is made and built, Layer 1 is the primary write path.
|
||||
|
||||
### The feedback loop
|
||||
|
||||
Global rules handle obvious cases inline → SessionEnd catches non-obvious ones → catches that
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# SecondBrain Findability Plan
|
||||
|
||||
_Last updated: 2026-06-27_
|
||||
_Last updated: 2026-06-28_
|
||||
_Status: Planning_
|
||||
|
||||
## Problem
|
||||
|
|
@ -34,7 +34,7 @@ This plan covers four phases, strictly sequenced — each phase depends on the p
|
|||
|
||||
## Surfaces Touched
|
||||
|
||||
- **Plugin code:** `cc-os/plugins/memory/hooks/session_start.py` (extend), `cc-os/plugins/memory/hooks/vault_index_rebuild.py` (new), `cc-os/plugins/memory/hooks/session_end.py` (extend for Phase 3 Parts A/B/C)
|
||||
- **Plugin code:** `cc-os/plugins/memory/hooks/session_context.py` (extend for Phase 2 injection + Phase 3 queue processing), `cc-os/plugins/memory/hooks/vault_index_rebuild.py` (new), `cc-os/plugins/memory/hooks/session_end.py` (extend for Phase 3 queue writing + git SHA capture)
|
||||
- **Skills:** `cc-os/plugins/memory/skills/memory-find.md` (new)
|
||||
- **Vault content:** `~/Documents/SecondBrain/vault-index.json` (new), `~/Documents/SecondBrain/project/` directory (new), hub notes as density warrants
|
||||
- **Settings:** `~/.claude/settings.json` — add `vault_index_rebuild.py` as fourth SessionEnd hook (separate hook, mirrors `memsearch_sync.py` / `vault_sync.py` pattern)
|
||||
|
|
@ -44,11 +44,12 @@ This plan covers four phases, strictly sequenced — each phase depends on the p
|
|||
| Risk | Mitigation |
|
||||
|------|-----------|
|
||||
| **Index staleness** | Rebuild at SessionEnd only (batched). Stale index is acceptable within a session; regenerated within one session boundary. Stale results are surfaced as a known gap, not a silent failure. |
|
||||
| **SessionEnd agent latency** | Agent runs at session close. Target: under 30 seconds, same constraint as `memsearch_sync.py` and `vault_sync.py`. Cap at 5 candidate notes per run; deeper audits are on-demand via `/memory-audit`. |
|
||||
| **SessionEnd agent latency** | Phase 3 uses queue-and-defer: SessionEnd writes a candidate queue file (fast Python, <1s), next SessionStart processes it with the already-running LLM. No API call at session close; no terminal blocking. `/memory-audit` skill available for on-demand thorough audits. |
|
||||
| **Duplicate notes** | Phase 3 Part C (pre-action deduplication) adds a tag-intersection check before any note creation. Dedup runs before create — not after. |
|
||||
| **False positives in discrepancy detection** | Phase 3 Part B flags, does not auto-update. Human confirms before any note change. Auto-update deferred until flag-then-confirm workflow is stable and trusted. |
|
||||
| **Hub note timing** | Hub notes should not be created before cluster density exists (3+ related notes). Phase 1 defers hub creation to emergence — noted as a natural byproduct, not a scheduled deliverable. |
|
||||
| **Injection token budget** | SessionStart injection targets ~200 tokens total. If project note + global rules exceed budget, truncate global rules, not the project note. Project-specific context is higher signal at the point of use. |
|
||||
| **Silent hook failure** | `session_start.py` / `session_context.py`: inject a one-line warning when vault sync or graph rebuild last-success timestamp is older than `stale_threshold_days × 2`. Turn invisible decay into a visible session prompt. |
|
||||
|
||||
## Build Order
|
||||
|
||||
|
|
@ -60,7 +61,7 @@ Two deliverables ship together:
|
|||
Hub notes are deferred from Plan A because their value is relational — they only earn their place once notes are dense enough to navigate between. A hub note is a map, not content: title + one-sentence purpose + bulleted links to member notes with one-line descriptions. Do not create hub notes on a schedule; create them as a natural byproduct when a cluster of 3+ related notes exists. Planned first hubs (after content density warrants): agent-orchestration hub, rails-conventions hub.
|
||||
|
||||
**vault-index.json**
|
||||
Location: `~/Documents/SecondBrain/vault-index.json`. Structure: `{tag: [{path, title, summary}]}`. Built by scanning vault notes: frontmatter tags + first 1–2 sentences as summary. Rebuilt at SessionEnd — batched, not real-time, not per-note-write. Tracked in git so it is pushed with every vault sync.
|
||||
Location: `~/Documents/SecondBrain/vault-index.json`. Structure: `{generated_at: "ISO-8601", tags: {tag: [{path, title, summary}]}}`. Built by scanning vault notes: frontmatter tags + first 1–2 sentences as summary. Rebuilt at SessionEnd — batched, not real-time, not per-note-write. Tracked in git so it is pushed with every vault sync. The `generated_at` field lets `/memory-find` surface staleness (e.g., "index is 3 sessions old") rather than silently serving stale results.
|
||||
|
||||
**Rebuild hook**
|
||||
Add `vault_index_rebuild.py` as a fourth SessionEnd hook. Hook order at SessionEnd: `session_end.py` → `memsearch_sync.py` → `vault_index_rebuild.py` → `vault_sync.py`. Rebuild runs after vault notes are written but before `vault_sync.py` pushes to Forgejo, so the pushed index is current.
|
||||
|
|
@ -77,7 +78,7 @@ Location: `cc-os/plugins/memory/skills/memory-find.md`. Accepts a tag argument.
|
|||
Trigger: on-demand only. The AI is instructed to call it when it encounters something that might be in SB. Automatic triggering before every action is deferred — too noisy until the index has density.
|
||||
|
||||
**SessionStart injection**
|
||||
Extend `session_start.py` to inject two things at session start (~200 tokens total):
|
||||
Extend `session_context.py` (the UserPromptSubmit hook — `session_start.py` has no injection output channel) to inject two things at session start (~200 tokens total):
|
||||
1. Global type classification rules (~100 tokens): "SecondBrain is your knowledge layer. Types that always go to SB: convention (rule/pattern), reference (lookup/architecture), howto (procedure). Use `/memory-find [tag]` to query. Use `/memory-recall [path]` to read full notes. DRY: if knowledge already exists in SB, reference it — don't duplicate."
|
||||
2. Project note (if exists for current project): project name, one-sentence description, tags this project produces. Tells the AI what types to write inline and what to look up. Project note format: `project/[name].md` containing name, one-sentence description, bulleted tag list, and optional `produces:` section for project-specific note types.
|
||||
|
||||
|
|
@ -85,6 +86,15 @@ Extend `session_start.py` to inject two things at session start (~200 tokens tot
|
|||
|
||||
Prerequisite: Phase 1 and Phase 2 complete. Runs after `session_end.py` writes the episodic journal.
|
||||
|
||||
> **Invocation model: queue-and-defer.** Python subprocess hooks cannot invoke the Claude model inline — the session has ended, and a direct Anthropic API call from a hook would block the terminal for 30–60+ seconds. The solution: split the work across two hooks.
|
||||
>
|
||||
> - **SessionEnd** (`session_end.py`, fast Python only): writes a candidate queue file to `~/.claude/memory-queue/YYYY-MM-DD-{session_id}.json` containing the git diff summary, files changed, and session context. No LLM involved. Takes under 1 second.
|
||||
> - **Next SessionStart** (`session_context.py`): if a queue file exists from a prior session, inject it as a task for the already-running LLM. The LLM processes the queue during normal session startup — no separate API call, no blocking.
|
||||
>
|
||||
> Trade-off: there is a one-session lag before notes get created. This is acceptable — the knowledge is captured (in the queue file) immediately; only the SB write is deferred. The `/memory-audit` skill remains available for on-demand thorough audits in the current session.
|
||||
|
||||
> **Prerequisite (git anchor):** Phase 3 Part A's "audit git diff since last session" requires a session-boundary git SHA. This is not currently stored anywhere. Before implementing Phase 3, add a step to `session_end.py` (or the vault journal entry it writes) to record `git rev-parse HEAD` for the active project. Without this, "since last session" has no anchor and the diff scope is undefined.
|
||||
|
||||
**Part A — Create new notes**
|
||||
Agent audits git diff since last session for knowledge that belongs in SB but wasn't written inline. Uses value gate (longevity + reusability + mutability) to assess candidates. Agent judgment: global type gap → update global classification rules; project-specific type → add to project note in SB. Creates notes using the appropriate template. Does NOT auto-update existing notes — flag only (see Part B). Cap at 5 candidates per run to stay within latency budget.
|
||||
|
||||
|
|
@ -95,6 +105,8 @@ When a session executed a procedure already in SB, agent checks actual vs. docum
|
|||
3. Flags ambiguity — process differed but unclear which is correct; human resolves before any change.
|
||||
Auto-update deferred until flag-then-confirm workflow is stable and trusted.
|
||||
|
||||
> **Deferred:** Part B is cut from the initial Phase 3 build. Discrepancy detection is only meaningful when the vault has sufficient procedure note density. It also competes for the same time/token budget as Part A, which has clearer value at this stage. Add Part B when the vault has 25+ howto notes that are actively used.
|
||||
|
||||
**Part C — Pre-action deduplication**
|
||||
Before creating a new note, agent checks vault-index.json for tag collision: do any existing notes carry the same primary tags? If yes, surface the existing note first — create only if the existing note doesn't cover the case. Implementation: tag intersection query on vault-index.json (fast, no embeddings needed). This closes the "we already have a note for that" gap.
|
||||
|
||||
|
|
@ -108,8 +120,10 @@ This phase is complete when the SessionEnd agent's "create" suggestions drop to
|
|||
|
||||
## Open Questions
|
||||
|
||||
1. **SessionEnd agent token budget:** Target under 30 seconds. If Phase 3 Parts A + B + C together exceed the budget, split: Part C (dedup check) runs inline at note-create time, Parts A + B run as the SessionEnd agent. This preserves deduplication without blocking on the full audit.
|
||||
1. **Queue file format for Phase 3:** The candidate queue file written by SessionEnd needs a schema. Minimum fields: session_id, timestamp, project_root, git_sha (see git anchor prerequisite), files_changed (list), diff_summary (truncated). Design this schema before building Phase 3.
|
||||
|
||||
2. **When does /memory-find trigger automatically vs. on-demand?** V1: on-demand only. The AI is instructed to call it when it encounters something that might be in SB. Automatic triggering (before every action) is deferred — too noisy until the index has density.
|
||||
|
||||
3. **Project note bootstrapping:** Who creates the initial project note for a new project? Options: (a) user creates manually using template, (b) `/memory-audit` suggests it, (c) `memory-project` skill creates it during onboarding. Current design: option (c) — `memory-project` skill writes a `project/[name].md` to the vault as part of onboarding. Confirm this in the memory-project skill spec.
|
||||
|
||||
4. **Queue file retention policy:** How long should unprocessed queue files persist? If the next session is in a different project directory, does `session_context.py` still pick up the queue? Simplest answer: always process any queue file regardless of current project — knowledge extraction is global. Retain queue files until processed; delete on successful processing.
|
||||
|
|
|
|||
Loading…
Reference in New Issue