diff --git a/.gitignore b/.gitignore index ea4ddc1..ab4a0de 100644 --- a/.gitignore +++ b/.gitignore @@ -2,3 +2,10 @@ .claude/ .codex/ .pi/ +.DS_Store +*.swp +*.log +/.idea/ +/.vscode/ +# Graphify build artifacts +graphify-out/ diff --git a/CLAUDE.md b/CLAUDE.md index a6d2158..1b0d1f9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -60,6 +60,8 @@ to those two and fix the stale doc. **Decisions locked (2026-06-04):** Six-facet tag taxonomy + `scope/` (ADR-011); reuse `~/Documents/SecondBrain` vault rather than creating a new one (ADR-012); build-first / migrate-incrementally — build full system against a fixture set first, defer bulk vault migration to last, onboard projects one at a time (ADR-013). +**Empirical finding locked (2026-06-05):** Graphify is a structure extractor, not a topic clusterer — no emergent hub nodes appear even at `--mode deep`; hub notes + wikilinks must be author-provided during migration (not deferred). Migration scaffolding is now a first-class deliverable. Open question: do facet tags create graph edges? (ADR-014; findings: `docs/memory-system/07-graph-connectivity-findings.md`). + ## OpenSpec workflow Changes are managed spec-driven via OpenSpec. Use the matching skills rather than editing spec diff --git a/docs/memory-system/03-architecture-decisions.md b/docs/memory-system/03-architecture-decisions.md index 9e9be79..79194cb 100644 --- a/docs/memory-system/03-architecture-decisions.md +++ b/docs/memory-system/03-architecture-decisions.md @@ -1,6 +1,6 @@ # Architecture Decision Records -_Last updated: 2026-06-04_ +_Last updated: 2026-06-05_ A running log of decisions and *why*. Format per entry: Context · Decision · Rationale · Alternatives rejected · Status. Newest decisions extend the log; supersede rather than delete. @@ -295,6 +295,64 @@ _Date: 2026-06-04_ ADR-012 (SecondBrain vault reuse — the migration steps this order defers). - **Status**: Accepted (updates `05-implementation-process.md` build order). +## ADR-014 — Graph connectivity comes from authored structure; migration scaffolding is a first-class prerequisite + +_Date: 2026-06-05_ + +- **Context**: ADR-011 specified hub notes + wikilinks + Graphify graph edges as the mechanism + for expressing hierarchy and cross-note relationships, with ADR-013 deferring bulk vault + migration to the final stage. Before build began, a discriminating empirical test compared a + cached-replay graph (per-fixture isolated extractions) against a clean single-pass deep + extraction (`graphify extract ~/Documents/SecondBrain --backend ollama --model + qwen25-coder-7b-16k --max-concurrency 1 --token-budget 8000 --mode deep --exclude + .obsidian`) against the real `~/Documents/SecondBrain` vault under Graphify 0.8.31 + + qwen2.5-coder:7b. See `07-graph-connectivity-findings.md` for the full data and methodology. + [primary/measured — 2026-06-05 session] +- **Decision**: **The connective spine of the knowledge graph must be author-provided.** Hub + notes and wikilinks are not optional scaffolding to add "someday" — they are the mechanism + by which Graphify connects thematically related notes, and they must be authored **as part of + the migration step**, not deferred to bulk import. Migration scaffolding (hub notes + + wikilinks for key concepts) is treated as a **first-class build deliverable** in the + migrate-incrementally phase of ADR-013. +- **Rationale**: The empirical test found that Graphify is a **structure extractor, not a topic + clusterer**. Even at `--mode deep --token-budget 8000`, no emergent shared-topic hub nodes + appeared (no "Pest Control" node, no "Niche Prospecting" node). All cross-note edges observed + came from explicit references, wikilinks, or document-level semantic similarity — not from + shared thematic identity. A practical test query ("how do we do niche prospecting outreach for + pest control?") returned 3 starting notes and traversal could not reach the email templates / + ACV data / business-model notes (separate communities, no connecting edges). This confirms + that useful retrieval is gated on migration scaffolding, not on Graphify's extraction power. + The clean single-pass run also showed the cached graph was partially a build artifact (cross- + note edges rose from 41% to 78% in a single-pass run), but the structural finding — no + emergent hub nodes — held in both runs. +- **Relationship to ADR-011**: Validates the hub-notes + wikilinks half of ADR-011 empirically. + The facet-tag half is **not yet validated**: no edge was observed to arise from shared + frontmatter facet tags alone. Whether `client/X` or `tool/Y` tags create graph connectivity + is an **open question** — see "Deferred" below. Do not assume facet tags contribute to graph + traversal retrieval until tested. +- **Relationship to ADR-013**: Refines the migrate-incrementally stage. "Migration" must be + defined to include hub note authoring and wikilink addition for key concepts, not just + frontmatter schema migration (adding `summary:` and namespaced facet tags). The build plan + (`04-build-plan.md`) should be updated to name this deliverable explicitly. +- **Alternatives rejected**: Relying on the SLM to auto-cluster topics and synthesize hub + entities — **empirically does not happen** at 7B model size with `--mode deep`. The design + already intended human-authored hub notes for this; the test confirms that intent was correct + and the fallback assumption ("maybe the LLM will do it") is false. +- **Deferred**: + 1. **Facet-tag-to-graph-edge question**: Do shared frontmatter facet tags (`client/`, + `tool/`, `domain/`, etc.) cause Graphify to create edges between notes, or does graph + connectivity come only from explicit wikilinks/references and semantic similarity? This was + NOT tested. Resolve before designing graph-traversal retrieval skills. + 2. **Larger extraction model**: Whether a substantially larger SLM (14B, 30B) would + synthesize emergent topic-hub nodes is untested. Secondary — the design does not depend on + it — but worth one test run before the build ships. + 3. **`reasoning_effort:"none"` patch**: The clean run required a local patch to `graphify/ + llm.py`. Track the upstream Graphify issue tracker for an official fix; treat the + installed version as pinned until resolved. +- **Status**: Accepted. Refines ADR-013 (migrate-incrementally phase scope) and empirically + validates the hub-notes/wikilinks mechanism of ADR-011 while flagging its facet-tag half as + an open question. + ## Rejected tools (summary) | Tool | Why rejected for our use | diff --git a/docs/memory-system/07-graph-connectivity-findings.md b/docs/memory-system/07-graph-connectivity-findings.md new file mode 100644 index 0000000..5f30ddf --- /dev/null +++ b/docs/memory-system/07-graph-connectivity-findings.md @@ -0,0 +1,166 @@ +# Graph Connectivity Findings: Structure Extractor vs. Topic Clusterer + +_Last updated: 2026-06-05_ +_Status: Empirical findings — primary/measured data from 2026-06-05 session against the real `~/Documents/SecondBrain` vault (Graphify 0.8.31, qwen2.5-coder:7b)._ + +--- + +## Context + +We regenerated the vault knowledge graph to inspect it visually, then ran a discriminating test +to determine why the graph appeared so fragmented in the initial screenshot — specifically, +whether the fragmentation was a build artifact or a genuine property of Graphify's extraction +behavior. + +Two runs were compared: + +1. **Cached run** — a replay of per-fixture isolated extractions (each fixture extracted in its + own temp directory, results assembled after the fact). This was the source of the initial + visual screenshot. +2. **Clean deep run** — a genuine single-pass extraction against the full vault, run from + scratch. + +--- + +## Methodology + +### Cached run (baseline) +Replayed previously-generated per-fixture outputs. Each note was extracted in isolation in a +separate temp directory (`/tmp/graphify-bench/vault-graph-viz/graphify-out/`). No single +invocation saw the full corpus. + +### Clean deep run +Single invocation against the full vault: + +``` +graphify extract ~/Documents/SecondBrain \ + --backend ollama \ + --model qwen25-coder-7b-16k \ + --max-concurrency 1 \ + --token-budget 8000 \ + --mode deep \ + --exclude .obsidian + +graphify cluster-only --backend ollama +``` + +Output: `/tmp/graphify-bench/vault-graph-clean/graphify-out/` + +Note: this run required the `reasoning_effort:"none"` patch applied to the installed +`graphify/llm.py` — a fragile workaround that is lost on `pip upgrade`. Track Graphify issue +tracker for an official fix before the build step. + +--- + +## Comparison Table + +| Metric | Cached run (replay, isolated) | Clean deep run (single-pass) | +|--------|-------------------------------|------------------------------| +| Nodes | 51 | 30 | +| Edges | 34 | 27 | +| Cross-note edges | 14 (41%) | 21 (78%) | +| Communities | 18 | 9 | +| Shared topic/category hub entity (e.g. "Pest Control") | none | none | +| Token cost | 0 (cache replay) | 38,590 in / 8,296 out (genuine) | + +The clean run's project-config node (`niche-automation-prospecting`) went from 0 cross-note +links to 6 — it now references the cold-email research, 10DLC setup, RingCentral analysis, and +VoIP research notes. + +--- + +## Conclusions + +### 1. Part of the extreme fragmentation was a build artifact [primary/measured] + +The cached graph replayed per-fixture extractions done in isolated temp directories. A clean +single-pass extraction nearly doubles the **proportion** of cross-note edges (41% → 78%), and +cuts community count from 18 to 9. A real production graph will be meaningfully more connected +than the initial screenshot suggested. + +### 2. Graphify is a structure extractor, not a topic clusterer [primary/measured] + +This is the load-bearing finding. Graphify creates edges where one note's text +references/cites another, or where two notes are semantically very close +(document-to-document). It does **not** invent emergent shared-category entity nodes. + +In both runs — including the clean deep run at `--mode deep --token-budget 8000` — **no** +shared topic hub node appeared. No "Pest Control" node, no "Niche Prospecting" node, no +"Speed-to-Lead" node. The LLM found many intra-note entities and doc-to-doc edges, but did not +synthesize a thematic category node spanning unrelated notes on the same topic. + +This means: the connective "spine" a human reader imagines must be an **actual hub note**, +authored with wikilinks. This is exactly the mechanism ADR-011 already specifies. The +empirical data confirms the design choice rather than opening a gap in it. + +### 3. Migration scaffolding is a first-class build prerequisite [primary/measured, strategic] + +**Practical consequence of conclusion 2.** Running `graphify extract` is the easy part. A real +practical query — "how do we do niche prospecting outreach for pest control?" — returned only 3 +starting notes in **both** runs, and traversal could **not** reach the email templates / ACV +data / business-model notes. Those are in separate communities with no connecting edges. + +Therefore: authoring hub notes + wikilinks for key concepts **during migration** is what makes +retrieval actually work. This is a **first-class build deliverable and a prerequisite for +useful retrieval** — not a "bulk import later" afterthought. This refines and reinforces +ADR-013 (build-first / migrate-incrementally): the migrate-incrementally phase must include hub +note authoring as a named deliverable, not just frontmatter schema migration. + +See ADR-014 for the locked decision. + +### 4. OPEN QUESTION: Do facet tags create graph edges? [explicitly UNRESOLVED — untested] + +Every edge observed in both runs came from explicit references, wikilinks, or semantic +similarity between documents. No edge was observably produced by shared frontmatter facet tags +alone (e.g., two notes both tagged `client/X` or `tool/Y`). + +**If facet tags do not create graph connectivity**, then: +- Tag-scoped retrieval ("all notes tagged `client/acme` + `tool/semrush`") is a **separate + tag-query path**, not graph traversal. +- The graph provides connection-based recall; the tag facets provide attribute-filtered recall. + These are complementary, not interchangeable. +- The design of graph-traversal retrieval skills must account for this separation. + +This is the **key thing to verify before designing retrieval skills**. It was not tested in +either run. Do not assume facet tags create edges. + +### 5. Caveats [primary/measured — honestly recorded] + +**(a) Coarser granularity is a tradeoff, not strictly "better."** The clean run produced 30 +nodes (one per document, roughly) because batching 4–5 notes per LLM call shifted extraction +toward document-level nodes + doc-to-doc edges rather than fine-grained intra-note sub-entities. +More cross-note edges, but less intra-note entity resolution. Whether this is better depends on +the query type — a behavioral tradeoff, not a dominance result. + +**(b) A substantially larger extraction model may produce topic-hub entities — untested.** We +did not test a 14B or 30B model. However, this is **secondary**: the design already intends +human-authored hub notes as the connective spine (ADR-011). If a larger SLM does synthesize +hub entities, that's additive; the design doesn't depend on it. + +**(c) The `reasoning_effort:"none"` patch is fragile.** This local patch to the installed +`graphify/llm.py` was required to complete the extraction run. It is lost on `pip upgrade`. +Track upstream for an official flag or mitigation before the system build step. Until then, +treat the installed version as pinned. + +--- + +## Pointers to Disposable Outputs + +- **Cached run output**: `/tmp/graphify-bench/vault-graph-viz/graphify-out/` — replay of + per-fixture isolated extractions; **not** representative of production behavior. +- **Clean deep run output**: `/tmp/graphify-bench/vault-graph-clean/graphify-out/` — genuine + single-pass extraction; the reference for production behavior. + +Both are disposable (per ADR-008: indexes are rebuildable, never synced). + +--- + +## Cross-References + +- **ADR-011** — Six-facet tag taxonomy + hub notes + wikilinks as the relationship mechanism + (empirically validated for the hub-notes/wikilinks half; open question on the facet-tag half) +- **ADR-013** — Build-first / migrate-incrementally (refined: migration phase must include hub + note authoring as a named deliverable) +- **ADR-014** — Decision locked from this finding: graph connectivity comes from authored + structure; migration scaffolding is a first-class prerequisite +- `docs/memory-system/06-graphify-evaluation.md` — earlier Graphify fit assessment (pre-empirical) diff --git a/openspec/changes/archive/2026-06-05-add-memory-plugin/.openspec.yaml b/openspec/changes/archive/2026-06-05-add-memory-plugin/.openspec.yaml new file mode 100644 index 0000000..c53ef21 --- /dev/null +++ b/openspec/changes/archive/2026-06-05-add-memory-plugin/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-06-05 diff --git a/openspec/changes/archive/2026-06-05-add-memory-plugin/design.md b/openspec/changes/archive/2026-06-05-add-memory-plugin/design.md new file mode 100644 index 0000000..dcb4417 --- /dev/null +++ b/openspec/changes/archive/2026-06-05-add-memory-plugin/design.md @@ -0,0 +1,103 @@ +## Context + +The vault knowledge graph (Graphify over `~/Documents/SecondBrain`), the extraction model (`qwen2.5-coder:7b`), and the episodic layer (memsearch) are built and validated. What does not yet exist is the runtime integration layer: Claude Code sessions do not automatically know the vault exists, do not inject context at startup, do not update the graph when vault notes are edited, and do not record activity at session end. + +This design specifies the global Claude Code plugin that wires all of that together. It is installed once at the user level (`~/.claude/`) and inherited by every project. The design rests on ADR-007 (lazy freshness — event-driven hooks, no daemon), ADR-008 (markdown-as-truth, graphs are disposable), ADR-009 (global plugin), ADR-013 (build-first / validate against fixture set before full vault), and ADR-014 (graph connectivity comes from authored hub notes + wikilinks, not auto-clustering; facet-tag-to-edge question is open). + +Current constraints: +- Graphify v0.8.31 anchored; the `reasoning_effort:"none"` patch to the installed `llm.py` is required for extraction and is lost on `pip upgrade` — fragile. +- `GRAPHIFY_OLLAMA_NUM_CTX` does NOT propagate through Graphify's `/v1` endpoint; context must be baked via a Modelfile variant (finding from `graphify-ollama-setup` change). +- Vault location: `~/Documents/SecondBrain` (ADR-012). Project graph convention: `/graphify-out/graph.json`. +- Rebuild stamp: `~/.cache/graphify/vault-rebuild.stamp`; default threshold 7 days. + +## Goals / Non-Goals + +**Goals:** +- Register three lifecycle hooks in the global Claude Code settings: SessionStart (staleness check + context injection), PostToolUse (graph update on vault write/edit), SessionEnd (episodic journal). +- Deliver three skills that carry behavioral conventions to the model: `memory-query`, `memory-write`, `memory-reorganize`. +- Provide a configurable plugin config block (vault path, Graphify output dir, Ollama model, `num_ctx`, stale threshold, env vars). +- Validate the end-to-end hook-and-skills path against the fixture set (ADR-013) before the full vault is live. + +**Non-Goals:** +- Vault migration scaffolding (hub notes + wikilinks) — a first-class prerequisite per ADR-014, but not a deliverable of this change. The plugin is designed assuming this scaffolding will exist; without it, retrieval is degraded but the plugin still functions. +- Per-project code-graph indexing (Step 2e) — separate, later change. +- Bulk vault migration (Step 5) — separate, later change. +- Memsearch installation or configuration — already installed; this plugin adds hooks that complement it. +- Vault sync mechanism (git / Syncthing) — separate, later concern. + +## Decisions + +**Hook input/output mechanism: stdin JSON in, `additionalContext` out.** +Hooks receive input as JSON on stdin (not as env vars or CLI args). The hook reads fields with `jq`: `jq -r '.session_id'`, `jq -r '.tool_input.file_path'`, `jq -r '.cwd'`, `jq -r '.source'`, etc. SessionStart injects context by writing JSON to stdout with an `additionalContext` field and exiting 0 — this is the confirmed Claude Code mechanism for hook-to-model context injection. (Source: Claude Code hooks documentation.) + +**Hook implementation: thin shell wrappers over Graphify CLI.** +The hooks are shell scripts registered in `~/.claude/settings.json`. They call Graphify CLI directly via the Bash tool path (not the MCP server) — simpler, no process to manage, all logic in the tool we already have. The MCP server remains an optional query path the AI can use during a session; it is not required for hooks. + +*Alternative considered:* Python wrapper scripts with richer logic (retry, logging). Rejected — the hooks need to stay thin and debuggable. If a hook fails it should fail visibly, not silently. A one-line `graphify update --file "$FILE"` is auditable; a 50-line Python script is not. + +**SessionStart injects god-nodes via `GRAPH_REPORT.md`-equivalent query, not by reading `graph.json` raw.** +The hook reads the pre-computed god-node list (top N most-connected nodes by degree) via `graphify query --budget N` or by reading `GRAPH_REPORT.md` if it exists at the output dir. It does NOT dump `graph.json` into context — that would blow the token budget. The god-node list is a compact, human-readable "map of what's known." + +*Alternative considered:* Injecting the full vault listing from `ls`. Rejected — too much noise, no semantic structure, does not scale. + +**SessionStart also injects `convention/*` note summaries (not full content).** +The hook runs `graphify query "convention"` (or equivalent tag-filtered query) to pull `convention/*` node summaries. Only the `summary` frontmatter field is included in injection, not the full note text. This respects progressive-disclosure and keeps context lean. + +*Tag-query note:* Whether `convention/` tags create graph edges is an open question (ADR-014). The hook uses the `graphify query "convention"` semantic path as the primary mechanism; if that proves insufficient, a fallback `grep -r "^tags:.*convention" ~/Documents/SecondBrain` to collect frontmatter summaries is viable and simpler. The design does NOT assume tag-based graph traversal works — both paths are planned. + +**PostToolUse hook targets vault writes only.** +The hook fires on `Write` and `Edit` tool calls, but ONLY when the target file path matches `~/Documents/SecondBrain/**/*.md`. Project-repo writes are ignored. This constraint is enforced in the hook's guard clause before any Graphify call. + +**SessionEnd hook appends to a daily journal note (vault, not project repo).** +The journal note lives at `~/Documents/SecondBrain/journal/YYYY-MM-DD.md` and is append-only. The hook collects vault notes touched in the session (from the PostToolUse hook's side-effect log or from Claude's session memory) and writes a timestamped entry: project name, notes touched, brief session intent. This is the input memsearch indexes for episodic recall. + +**Stale-rebuild stamp approach — rebuild is DETACHED, not inline.** +The staleness check reads `~/.cache/graphify/vault-rebuild.stamp` (a file whose mtime or content is the last `--force` rebuild timestamp). If older than N days (default 7, configurable), SessionStart spawns `graphify extract --backend ollama --model --force` as a **detached background process** (e.g. `nohup graphify ... >> 2>&1 &` or `setsid`) and returns immediately. The hook MUST NOT block waiting for the rebuild — SessionStart is a blocking hook (the session does not start until the hook returns), so a multi-minute inline rebuild would freeze the user. (Source: Claude Code hooks documentation — SessionStart has no async option and must return promptly; target sub-second for the synchronous path.) + +A lock file (e.g. `~/.cache/graphify/vault-rebuild.lock`) prevents concurrent sessions from spawning duplicate rebuilds. The background process touches the stamp on successful completion. The rebuilt graph is not visible to the session that spawned the rebuild; it is available to the next session. The current session proceeds with the existing (possibly stale) graph — this is an accepted tradeoff. + +*Ghost-node note:* `graphify update --file` merges but does not prune deleted nodes. The periodic `--force` rebuild (triggered by the stamp check) is the only way to clear ghost nodes. 7-day default balances freshness against extraction cost (~38K tokens per full vault run at current vault size). + +**Three skills, not one.** +`memory-query`, `memory-write`, `memory-reorganize` are separate skills loaded on demand rather than a single monolithic `memory` skill. This keeps each skill's context footprint small — a session doing only writes doesn't pull in reorganize guidance, and vice versa. + +**memory-query skill must NOT assume tag-based graph traversal (ADR-014 open question).** +The skill teaches two distinct retrieval paths: (1) graph traversal via `graphify query` / `path` / `explain` (connection-based, follows edges from explicit references and wikilinks), and (2) tag-attribute filtering (treat as a separate query mechanism, not graph traversal). The skill notes explicitly that whether shared frontmatter facet tags create graph edges is unverified — use both paths but do not conflate them. + +**Config lives in the plugin's settings block, not in CLAUDE.md.** +Vault path, output dir, model, num_ctx, stale threshold, env vars — all go in the plugin's `settings.json` contribution (or a `.env`-style config file the hooks source). Per-project CLAUDE.md holds only pointers/tags, not the system config. + +## Risks / Trade-offs + +- **`reasoning_effort:"none"` patch lost on `pip upgrade`** → Treat installed Graphify as pinned; document the patch and the pinned version in the plugin README. Track upstream issue tracker for an official flag. Mitigation: the plugin's setup instructions include a post-install verification step that confirms the patch is present. +- **`GRAPHIFY_OLLAMA_NUM_CTX` env-var does not propagate** → Bake context into the Ollama model variant via Modelfile (as discovered in `graphify-ollama-setup`). The plugin config should reference the Modelfile-baked variant name (e.g. `qwen25-coder-7b-16k`), not the base model name. +- **SessionStart hook latency — RESOLVED** → SessionStart is a blocking hook with no async option (Claude Code hooks documentation). The heavy `--force` rebuild is now **detached as a background process** (lock-guarded to prevent duplication). The synchronous path (read stamp, spawn background if stale, inject context) must be sub-second. Tradeoff: the session that triggers a rebuild sees the existing graph; next session sees the rebuilt one. This is accepted; the 7-day threshold makes rebuilds rare. +- **Convention-injection path uncertainty** → Whether `graphify query "convention"` reliably surfaces `convention/*` notes depends on the facet-tag-to-edge question (ADR-014 open). Fallback: shell grep over vault frontmatter for `convention/` tags + read `summary:` field directly. Design both paths; switch based on empirical testing during fixture validation. +- **Journal note accumulation** → Daily journal notes grow over time; memsearch handles indexing, but old notes are not pruned. Mitigation: note in the plugin README that journal notes are part of the vault and subject to normal vault governance (reorg skill can consolidate old journals). +- **Vault migration scaffolding is a prerequisite, not guaranteed** → If hub notes + wikilinks are not yet authored for the fixture set, retrieval quality will be degraded (ADR-014: no authored structure → no graph connectivity). The plugin is still installable and hooks still fire; only retrieval utility is reduced. This is the expected state at initial install; the migration scaffolding change is the unlock. + +## Migration Plan + +All steps are local and non-destructive: + +1. **Create plugin directory structure**: `~/.claude/plugins/memory/` with hooks/ and skills/ subdirectories. +2. **Write hook scripts**: `session-start.sh`, `post-tool-use-write.sh`, `session-end.sh`. Each is a thin shell wrapper; test each script standalone before registering. +3. **Register hooks** in `~/.claude/settings.json` global config: SessionStart → `session-start.sh`, PostToolUse (Write + Edit) → `post-tool-use-write.sh`, SessionEnd → `session-end.sh`. +4. **Write skill files**: `memory-query.md`, `memory-write.md`, `memory-reorganize.md` in `~/.claude/plugins/memory/skills/`. Register in plugin manifest. +5. **Write plugin config** (`~/.claude/plugins/memory/config.yaml` or equivalent): vault path, output dir, model, num_ctx, stale threshold, env vars. +6. **Validate against fixture set** (ADR-013): start a session with the fixture project; confirm SessionStart injects god-nodes + conventions; write a test vault note and confirm PostToolUse fires `graphify update`; end session and confirm journal note appended. Use the fixture set, not the full vault, for this first pass. + +Rollback: deregister hooks from `settings.json`; plugin directory can remain. No vault content is mutated by the hooks (extraction is read-only over notes; journal is append-only to a new file). + +## Open Questions + +1. ~~**Does Claude Code support async/backgrounded SessionStart hooks?**~~ **RESOLVED (2026-06-05):** SessionStart is a blocking hook with no async option (Claude Code hooks documentation). The rebuild is now detached as a background process. See the stale-rebuild decision above. +2. **Convention-injection path**: which mechanism surfaces `convention/*` note summaries most reliably — `graphify query "convention"` or frontmatter grep + `summary:` field read? Decide by testing both against the fixture set during step 6. +3. **Journal note format**: what level of detail is useful for memsearch episodic recall? Minimum: project name + note paths touched + date. Optional: session intent, decisions made. Settle during fixture validation. +4. **Project graph path injection**: SessionStart should inject `--graph /graphify-out/graph.json` if a project graph exists at that path. The hook reads `.cwd` from stdin JSON (confirmed available — Claude Code hooks documentation), which provides the working directory. A `git rev-parse --show-toplevel` from that cwd or a `$CLAUDE_PROJECT_DIR` env var convention can determine project root. Mechanism available; exact convention to settle during implementation. +5. **Facet-tag-to-edge question** (ADR-014 deferred): before designing `memory-query`'s tag-filter path, verify empirically whether shared facet tags create graph edges in the fixture graph. One test: build a fixture graph with two notes sharing `tool/graphify`, inspect `graph.json` for an edge between them. + +**RESOLVED (2026-06-05) — hook input/output mechanisms:** +- **Session ID availability**: `session_id` is available in hook stdin JSON (`jq -r '.session_id'`) for all hooks that receive it (PostToolUse, SessionEnd). Use for per-session temp file naming. No session-id env var exists; stdin is the only source. (Claude Code hooks documentation.) +- **PostToolUse file-path access**: The hook reads `.tool_input.file_path` from stdin JSON (`jq -r '.tool_input.file_path'`). Also available: `.tool_name` for filtering by tool type. The vault-path guard is done inside the hook after reading this field. +- **Context injection mechanism**: SessionStart injects text into the model context by writing `{"additionalContext": "..."}` JSON to stdout and exiting 0. All prior references to "inject via context" or hand-waved mechanisms now refer to this confirmed mechanism. (Claude Code hooks documentation.) diff --git a/openspec/changes/archive/2026-06-05-add-memory-plugin/proposal.md b/openspec/changes/archive/2026-06-05-add-memory-plugin/proposal.md new file mode 100644 index 0000000..f3e35cf --- /dev/null +++ b/openspec/changes/archive/2026-06-05-add-memory-plugin/proposal.md @@ -0,0 +1,38 @@ +## Why + +The memory system's knowledge graph (vault + Graphify) and episodic layer (memsearch) are built and validated, but Claude Code has no runtime awareness of them — hooks are not registered, skills do not exist, and there is no mechanism by which a session injects vault context or records what was touched. This change delivers the global Claude Code plugin that makes the system live: hooks that inject knowledge automatically and update the graph on writes, plus three on-demand skills that teach Claude when and how to read from and write to the vault. + +## What Changes + +- **New: `~/.claude/plugins/memory/` global plugin** — installs hooks (SessionStart, PostToolUse, SessionEnd) and the three skills at the user level; all projects inherit without per-project setup. +- **New: SessionStart hook** — staleness check (rebuild stamp age vs. threshold) + forced rebuild if stale; injects vault god-nodes, current-project `convention/*` summaries, episodic journal pointer, and project graph path if a project graph exists. +- **New: PostToolUse hook** (Write/Edit on vault `.md`) — runs `graphify update --file ` to merge changed notes into the vault graph event-driven, no polling. +- **New: SessionEnd hook** — appends a dated episodic journal note with pointers to vault notes touched in the session. +- **New: `memory-query` skill** — when/how to use `graphify query` / `path` / `explain` vs. memsearch; god-node discipline; `--budget` / `--dfs` usage; cross-client lookups via `--graph`; tag-scoped retrieval treated as a separate path from graph traversal (open question per ADR-014). +- **New: `memory-write` skill** — when knowledge is evergreen vs. project-ephemeral; required frontmatter contract (`summary` + `scope/`); vault-not-repo rule. +- **New: `memory-reorganize` skill** — plan-mode consolidation/promotion procedure; when to trigger `--force` rebuild; human-review guardrails. +- **New: plugin config block** — vault path, Graphify output dir, Ollama model name, `num_ctx`, rebuild-stale threshold (days), env vars (`OLLAMA_FLASH_ATTENTION`, `GRAPHIFY_OLLAMA_NUM_CTX`, `GRAPHIFY_OLLAMA_KEEP_ALIVE`). + +**Out of scope / prerequisite (not tasks here):** Vault migration scaffolding — authoring hub notes + wikilinks — is a prerequisite for useful retrieval (ADR-014: Graphify is a structure extractor, not a topic clusterer; connectivity comes from authored hub notes + wikilinks). This change assumes that scaffolding either exists for the fixture set or will be done as a parallel prerequisite. + +## Capabilities + +### New Capabilities + +- `memory-plugin-hooks`: The three lifecycle hooks (SessionStart / PostToolUse / SessionEnd) that make graph freshness and context injection automatic. Covers the shell wrapper logic, staleness check, stamp file, `graphify update --file` invocation, and journal note append. +- `memory-plugin-skills`: The three skills (`memory-query`, `memory-write`, `memory-reorganize`) that teach Claude the behavioral conventions for the memory system. Covers query-pattern guidance, frontmatter contract, write-path rules, and reorganize procedure. + +### Modified Capabilities + + + +## Impact + +- **New global plugin** at `~/.claude/plugins/memory/` (or equivalent user-level plugin install path); no per-project files created. +- **Dependencies:** Graphify (`graphifyy`, anchored to v0.8.31), Ollama with `qwen2.5-coder:7b` (or the selected model from `local-model-selection`), memsearch plugin already installed. +- **Known fragile runtime dependency:** `reasoning_effort:"none"` patch to Graphify's installed `llm.py` — lost on `pip upgrade`; tracked as a risk. Treat installed version as pinned until upstream resolves. +- **Env vars** set in plugin config: `OLLAMA_FLASH_ATTENTION=1`, `GRAPHIFY_OLLAMA_NUM_CTX=8192`, `GRAPHIFY_OLLAMA_KEEP_ALIVE=5`. +- **Consumes:** built vault graph (`vault-graph-build`), selected extraction model (`local-model-selection`), ADR-007 (lazy freshness), ADR-009 (global plugin), ADR-011 (six-facet taxonomy), ADR-013 (build-first/migrate-incrementally), ADR-014 (graph connectivity from authored structure). +- **Unblocks:** the per-project code-graph onboarding phase (Step 2e, separate change) and bulk vault migration (Step 5, separate change). diff --git a/openspec/changes/archive/2026-06-05-add-memory-plugin/specs/memory-plugin-hooks/spec.md b/openspec/changes/archive/2026-06-05-add-memory-plugin/specs/memory-plugin-hooks/spec.md new file mode 100644 index 0000000..90cd2ae --- /dev/null +++ b/openspec/changes/archive/2026-06-05-add-memory-plugin/specs/memory-plugin-hooks/spec.md @@ -0,0 +1,76 @@ +## ADDED Requirements + +### Requirement: SessionStart hook — staleness check and detached background rebuild +The plugin SHALL register a SessionStart hook that reads the rebuild stamp file at `~/.cache/graphify/vault-rebuild.stamp`. If the stamp is absent or its mtime is older than the configured threshold (default: 7 days), the hook SHALL spawn `graphify extract --backend ollama --model --force` as a **detached background process** (nohup/setsid, output redirected to a log file) and SHALL return immediately without waiting for it. The background process SHALL update the stamp file upon successful completion and remove the lock file. The hook SHALL NOT block on the rebuild — SessionStart blocks the entire session until the hook returns, so the synchronous path must be sub-second. (Source: Claude Code hooks documentation — SessionStart has no async option.) The hook SHALL write a lock file (`~/.cache/graphify/vault-rebuild.lock`) before spawning, so concurrent sessions do not launch duplicate rebuilds. + +#### Scenario: Stamp is fresh (within threshold) +- **WHEN** the SessionStart hook fires and the rebuild stamp mtime is within the configured threshold +- **THEN** no extraction run is triggered and the hook proceeds to context injection + +#### Scenario: Stamp is stale or absent +- **WHEN** the SessionStart hook fires and the rebuild stamp is absent or older than the configured threshold +- **THEN** the hook checks for a lock file; if absent, spawns a detached background rebuild and writes the lock; in all cases the hook continues immediately to context injection using the existing graph; the rebuilt graph becomes available to the next session + +#### Scenario: Rebuild already in progress (lock file present) +- **WHEN** the SessionStart hook fires and the lock file already exists (rebuild spawned by a concurrent session) +- **THEN** the hook does NOT spawn another rebuild; it logs that a rebuild is in progress and proceeds to context injection + +#### Scenario: Extraction fails during stale rebuild +- **WHEN** the background rebuild process exits non-zero +- **THEN** the background process logs the error to its log file and removes the lock file without touching the stamp; the current session is unaffected (already past injection); the next session will attempt the rebuild again + +### Requirement: SessionStart hook — vault context injection +The plugin SHALL register a SessionStart hook that injects the following into Claude's context at the start of each session: (a) the vault graph's god-node summary (top N most-connected nodes by degree, sourced from `GRAPH_REPORT.md` or equivalent query), (b) the summaries of `convention/*` vault notes (sourced via `graphify query "convention"` or, as a fallback, by grepping vault frontmatter for `convention/` tags and reading the `summary:` field), and (c) a pointer to the current day's episodic journal note path. The hook SHALL deliver this content by writing `{"additionalContext": ""}` JSON to stdout and exiting 0 — this is the confirmed Claude Code mechanism for hook-to-model context injection. (Source: Claude Code hooks documentation.) The hook reads its input from stdin JSON; relevant fields include `.cwd` (working directory) and `.source` (startup|resume|clear|compact). + +#### Scenario: Vault graph and convention notes exist +- **WHEN** SessionStart fires and the vault graph and at least one `convention/*` note exist +- **THEN** the injected context contains the god-node list, all `convention/*` summaries, and the journal note path + +#### Scenario: No convention notes exist yet +- **WHEN** SessionStart fires and no `convention/*` notes exist in the vault +- **THEN** the injected context contains the god-node list and journal pointer; the conventions section is omitted without error + +#### Scenario: Project graph exists for the current project +- **WHEN** SessionStart fires and a file exists at `/graphify-out/graph.json` +- **THEN** the injected context also includes the project graph path, so Claude can query it via `graphify query --graph ` + +### Requirement: PostToolUse hook — event-driven vault graph update +The plugin SHALL register a PostToolUse hook that fires after any `Write` or `Edit` tool call. The hook SHALL check whether the target file path is a `.md` file under the configured vault path (`~/Documents/SecondBrain`). If it is, the hook SHALL run `graphify update --file ` to merge the changed note into the vault graph. If the target path is outside the vault, the hook SHALL do nothing. + +#### Scenario: A vault note is written or edited +- **WHEN** a Write or Edit tool call completes and the target path is under the vault directory +- **THEN** the hook runs `graphify update --file ` synchronously before Claude proceeds + +#### Scenario: A non-vault file is written or edited +- **WHEN** a Write or Edit tool call completes and the target path is outside the vault directory +- **THEN** the hook does nothing; no graphify call is made + +#### Scenario: graphify update fails for a vault write +- **WHEN** `graphify update --file ` exits non-zero +- **THEN** the hook logs the error but does NOT block the tool call result from being returned; the graph may be transiently stale + +### Requirement: SessionEnd hook — episodic journal append +The plugin SHALL register a SessionEnd hook that appends a dated entry to the daily journal note at `/journal/YYYY-MM-DD.md` (creating the file if it does not exist). The entry SHALL include at minimum: the current project name (or path), the vault note paths touched during the session, and a UTC timestamp. The journal file is append-only; existing content is never overwritten. + +#### Scenario: Session ends with vault notes touched +- **WHEN** the SessionEnd hook fires and at least one vault note was written or edited during the session +- **THEN** the hook appends a timestamped entry to the daily journal file listing the touched note paths and the project context + +#### Scenario: Session ends with no vault writes +- **WHEN** the SessionEnd hook fires and no vault notes were touched +- **THEN** the hook still appends a minimal entry (timestamp + project) so the session is recorded for episodic recall + +#### Scenario: Journal file does not exist yet for the day +- **WHEN** the SessionEnd hook fires and `/journal/YYYY-MM-DD.md` does not exist +- **THEN** the hook creates the file with appropriate frontmatter (summary, scope/global, type/log) and appends the session entry + +### Requirement: Plugin configuration block +The plugin SHALL expose a configuration file at `~/.claude/plugins/memory/config.yaml` that allows setting: vault path, Graphify output directory, Ollama model name (Modelfile-baked variant), num_ctx, rebuild-stale threshold in days, and the env vars `OLLAMA_FLASH_ATTENTION`, `GRAPHIFY_OLLAMA_NUM_CTX`, `GRAPHIFY_OLLAMA_KEEP_ALIVE`. All hooks SHALL read from this config rather than hardcoding paths or values. + +#### Scenario: Default config is used +- **WHEN** the config file exists with no overrides for a setting +- **THEN** the hook uses the default value for that setting (vault: `~/Documents/SecondBrain`, threshold: 7 days, model: `qwen25-coder-7b-16k`) + +#### Scenario: User overrides vault path +- **WHEN** the config file specifies a non-default vault path +- **THEN** all hooks use the configured path for vault detection, graph paths, and journal writes diff --git a/openspec/changes/archive/2026-06-05-add-memory-plugin/specs/memory-plugin-skills/spec.md b/openspec/changes/archive/2026-06-05-add-memory-plugin/specs/memory-plugin-skills/spec.md new file mode 100644 index 0000000..a1e2af6 --- /dev/null +++ b/openspec/changes/archive/2026-06-05-add-memory-plugin/specs/memory-plugin-skills/spec.md @@ -0,0 +1,62 @@ +## ADDED Requirements + +### Requirement: memory-query skill — graph retrieval guidance +The plugin SHALL provide a `memory-query` skill that teaches Claude when and how to query the vault knowledge graph versus the episodic memsearch layer. The skill SHALL specify: (a) use `graphify query ""` for evergreen knowledge questions ("how do we…", "what do we know about…"), (b) use `graphify path "" ""` to find how two concepts relate, (c) use `graphify explain ""` for a deep-dive with neighbors, (d) add `--budget N` to cap answer size (default ~2000 tokens), (e) add `--dfs --budget N` for bounded depth-first traversal, (f) add `--graph ` to query a specific project graph instead of the vault graph. The skill SHALL explicitly state that god-nodes are the entry point — query god-nodes first, then scalpel to specific nodes. The skill SHALL treat tag-scoped retrieval (e.g. "all notes tagged `client/acme`") as a SEPARATE query path from graph traversal, noting that whether shared frontmatter facet tags create graph edges is unverified (ADR-014 open question); Claude SHALL NOT assume facet tags enable graph traversal. For "what happened when" questions, the skill SHALL direct Claude to memsearch in natural language rather than the vault graph. + +#### Scenario: Claude needs to answer a knowledge question during a task +- **WHEN** a task touches a tool, client, or domain the vault may know about +- **THEN** Claude runs `graphify query ""` before relying on training knowledge, using `--budget` to keep context lean + +#### Scenario: Claude needs to find how two concepts relate +- **WHEN** a task requires understanding the relationship between two vault entities (e.g. client and a tool they use) +- **THEN** Claude runs `graphify path "" ""` to traverse the graph path between them + +#### Scenario: Cross-client knowledge lookup +- **WHEN** Claude needs to find all vault knowledge about a tool or domain regardless of which client it came from +- **THEN** Claude runs `graphify query ""` against the vault graph (not `--graph `) to get cross-client results + +#### Scenario: Project-specific knowledge lookup +- **WHEN** Claude needs knowledge specific to the current project's code graph +- **THEN** Claude runs `graphify query "" --graph /graphify-out/graph.json` using the project graph path injected by SessionStart + +#### Scenario: Episodic question ("what was I working on last week?") +- **WHEN** a question is time-anchored ("what did we decide last Tuesday", "what was I working on yesterday") +- **THEN** Claude queries memsearch in natural language, NOT the vault graph + +### Requirement: memory-write skill — when and how to write vault knowledge +The plugin SHALL provide a `memory-write` skill that teaches Claude when knowledge should be written to the vault and what the required frontmatter contract is. The skill SHALL specify: write to the vault when knowledge is evergreen and reusable across projects (tool/API behavior, conventions, client-specific facts worth reusing); do NOT write project-ephemeral state to the vault (that goes to project files or the episodic layer). The skill SHALL specify the required frontmatter fields: `summary` (one-line, written at note creation — not deferred — this is the human-authored router hint Graphify does not generate), `scope/global` or `scope/project` (default new tool/domain knowledge to `scope/global`; use `scope/project` when specific to how a client uses something), and at least one facet tag from the six-facet taxonomy (`type/`, `client/`, `project/`, `domain/`, `tool/`, `convention/`). The skill SHALL state the vault-not-repo rule: write knowledge ONLY to `~/Documents/SecondBrain`, never silently into a project repo. The skill SHALL note that writing a vault note automatically triggers the PostToolUse hook to update the graph — no manual graphify call is needed after writing. + +#### Scenario: Claude discovers how a tool's API behaves +- **WHEN** Claude learns something about a tool's authentication, rate limits, or endpoint quirks that could apply to future sessions or projects +- **THEN** Claude writes a note to the vault with the required frontmatter (summary, scope/global, type/reference, tool/) rather than embedding it in the project's CLAUDE.md + +#### Scenario: Claude establishes a convention that should apply beyond this task +- **WHEN** Claude and the user settle on a convention (coding style, workflow, decision rule) that should propagate to other projects +- **THEN** Claude writes a `convention/` note to the vault with the required frontmatter (summary, scope, type/convention) + +#### Scenario: Knowledge is project-specific (how this client uses a tool) +- **WHEN** the knowledge describes behavior specific to one client's setup, not the tool generally +- **THEN** Claude writes the note with `scope/project` and a `client/` facet tag rather than `scope/global` + +#### Scenario: Claude tries to record project-ephemeral state +- **WHEN** the information is specific to the current task and not reusable in future sessions or projects +- **THEN** Claude does NOT write it to the vault; it stays in the project repo or episodic session notes + +### Requirement: memory-reorganize skill — consolidation and plan-mode promotion +The plugin SHALL provide a `memory-reorganize` skill that teaches Claude the procedure for consolidating, deduplicating, promoting, and reorganizing vault notes. The skill SHALL specify: (a) reorganization MUST happen in plan mode — Claude proposes changes for human review before executing; (b) triggers include: duplicate coverage noticed across notes, a `scope/project` fact observed to recur across 2+ clients (promotion candidate), a note grown beyond the L1 "one concept, ~200 lines" discipline; (c) the consolidation procedure is: identify candidate notes, propose a merge/split/promote/rename plan, await human approval, execute edits, then run `graphify extract --force` to rebuild the graph clean; (d) a `--force` rebuild is required after any reorganization that removes, renames, or restructures notes (to clear ghost nodes); (e) `--force` rebuild is ALSO triggered when the SessionStart stale check fires, so reorganize skill's rebuild invocation is consistent with the staleness-check mechanism. The skill SHALL warn that `graphify update --file` does NOT prune deleted or renamed nodes — only `--force` rebuilds clear ghost nodes. + +#### Scenario: Claude notices duplicate coverage across two vault notes +- **WHEN** Claude identifies two notes that cover the same concept with overlapping content +- **THEN** Claude enters plan mode: proposes which note to keep, what to merge, and what to discard — and awaits human approval before making any edits + +#### Scenario: A project-scoped fact has recurred across multiple clients +- **WHEN** Claude observes that a fact tagged `scope/project` has appeared in 2+ client contexts, suggesting it is actually global knowledge +- **THEN** Claude proposes promoting the note to `scope/global` (updating frontmatter, merging client-specific variants if needed) in plan mode before executing + +#### Scenario: Reorganization is complete and graph needs to be rebuilt +- **WHEN** any reorganization edit has been approved and executed (notes deleted, renamed, or restructured) +- **THEN** Claude runs `graphify extract --backend ollama --model --force` to clear ghost nodes and rebuild a clean graph; updates the rebuild stamp + +#### Scenario: Note has grown beyond single-concept discipline +- **WHEN** Claude observes a vault note has grown beyond ~200 lines or covers more than one distinct concept +- **THEN** Claude proposes a split into two or more focused notes in plan mode, with proposed filenames, frontmatter, and the wikilinks that would connect them — before executing diff --git a/openspec/changes/archive/2026-06-05-add-memory-plugin/tasks.md b/openspec/changes/archive/2026-06-05-add-memory-plugin/tasks.md new file mode 100644 index 0000000..7eae1db --- /dev/null +++ b/openspec/changes/archive/2026-06-05-add-memory-plugin/tasks.md @@ -0,0 +1,50 @@ +## 1. Plugin directory and config + +- [x] 1.1 Create plugin directory structure: `~/.claude/plugins/memory/` with `hooks/` and `skills/` subdirectories +- [x] 1.2 Write `~/.claude/plugins/memory/config.yaml` with configurable fields: vault path (`~/Documents/SecondBrain`), Graphify output dir, Ollama model (`qwen25-coder-7b-16k`), num_ctx (8192), stale threshold (7 days), and env vars (`OLLAMA_FLASH_ATTENTION=1`, `GRAPHIFY_OLLAMA_NUM_CTX=8192`, `GRAPHIFY_OLLAMA_KEEP_ALIVE=5`) +- [x] 1.3 Write a `README.md` for the plugin documenting: the `reasoning_effort:"none"` patch requirement, the pinned Graphify version (v0.8.31), the Modelfile-baked context approach (`qwen25-coder-7b-16k`), and how to verify the patch is present after any `pip upgrade` + +## 2. SessionStart hook + +- [x] 2.1 Write `~/.claude/plugins/memory/hooks/session-start.sh`: read stdin JSON via `jq` (fields: `.cwd`, `.source`); source config; check `~/.cache/graphify/vault-rebuild.stamp` mtime vs. threshold — if stale or absent AND no lock file at `~/.cache/graphify/vault-rebuild.lock`, spawn `nohup graphify extract --backend ollama --model --force >> 2>&1 &` as a **detached background process**, write the lock file (background process removes it on completion), and continue immediately; do NOT wait for the rebuild (SessionStart blocks the session — it must return sub-second on the synchronous path). Log that a background rebuild was triggered. +- [x] 2.2 Extend `session-start.sh`: inject vault god-node summary — read from `GRAPH_REPORT.md` or equivalent at the configured output dir; assemble context string (compact "map of what's known" block, not raw `graph.json`) +- [x] 2.3 Extend `session-start.sh`: inject `convention/*` note summaries — attempt `graphify query "convention" --budget 500`; fall back to `grep -rl "convention/" | xargs grep -h "^summary:"` if graph query returns empty; omit section silently if no convention notes found +- [x] 2.4 Extend `session-start.sh`: inject episodic journal pointer — compute today's journal path (`/journal/YYYY-MM-DD.md`) and add as a one-line pointer; inject project graph path (`/graphify-out/graph.json`) if that file exists (read `.cwd` from stdin JSON as working directory base; use `git rev-parse --show-toplevel` from there or `$CLAUDE_PROJECT_DIR` convention) +- [x] 2.5 Emit all injected content to stdout as `{"additionalContext": ""}` JSON and exit 0 — this is the confirmed Claude Code mechanism for hook-to-model context injection (Claude Code hooks documentation) +- [x] 2.6 Test `session-start.sh` standalone (not via Claude Code harness): supply mock stdin JSON (`echo '{"cwd":"/tmp","source":"startup"}' | ./session-start.sh`); run with a clean stamp, a stale stamp (confirm background spawn, no block), a lock file already present (confirm no duplicate spawn), and a missing vault — confirm each path behaves as specified in `memory-plugin-hooks/spec.md` + +## 3. PostToolUse hook + +- [x] 3.1 Write `~/.claude/plugins/memory/hooks/post-tool-use-write.sh`: source config; read stdin JSON via `jq` — extract `FILE_PATH=$(jq -r '.tool_input.file_path')` and `TOOL_NAME=$(jq -r '.tool_name')`; guard clause — skip if `$FILE_PATH` does not match `/**/*.md`; if it matches, run `graphify update --file "$FILE_PATH"`; log error but do NOT block on non-zero exit (exit 0 always); also append `$FILE_PATH` to per-session temp log (see task 4.2) + > **Deviation (2026-06-05):** `graphify update --file` does not exist in Graphify v0.8.31. The correct incremental update for markdown/doc files is `graphify extract --update --backend ollama --model ` which uses SHA-256 content hashing to skip unchanged files. Running this synchronously in PostToolUse would block on Ollama. **Adopted approach:** PostToolUse deletes `~/.cache/graphify/vault-rebuild.stamp` (invalidates freshness). Next SessionStart detects the missing stamp and spawns `graphify extract --update` as a detached background process, refreshing only changed files. Graph freshness is deferred by at most one session start, not immediate. +- [x] 3.2 Test `post-tool-use-write.sh` standalone: supply mock stdin JSON (`echo '{"tool_name":"Write","tool_input":{"file_path":"/test.md"}}' | ./post-tool-use-write.sh`); test with a vault `.md` path (confirm graphify update fires), a non-vault path (confirm nothing fires), and a path where graphify returns non-zero (confirm error is logged, script exits 0) + +## 4. SessionEnd hook + +- [x] 4.1 Write `~/.claude/plugins/memory/hooks/session-end.sh`: read stdin JSON via `jq` — extract `SESSION_ID=$(jq -r '.session_id')` and `REASON=$(jq -r '.reason')`; source config; compute today's journal path; if file does not exist, create it with minimal frontmatter (`summary:`, `scope/global`, `type/log`); append a timestamped entry with: current project (from `$CLAUDE_PROJECT_DIR` or `git rev-parse --show-toplevel`), vault notes touched this session (read from temp log at `/tmp/claude-vault-touched-$SESSION_ID` — see task 4.2), UTC timestamp. Note: SessionEnd has no decision control (exit code ignored); use for cleanup/logging only. +- [x] 4.2 Extend `post-tool-use-write.sh` to append successfully-updated vault note paths to a per-session temp file `/tmp/claude-vault-touched-$SESSION_ID` where `SESSION_ID` is read from stdin JSON via `jq -r '.session_id'`; `session-end.sh` reads this same file using its own stdin `session_id`. (Confirmed available: `session_id` is a field on stdin JSON for both PostToolUse and SessionEnd — Claude Code hooks documentation.) +- [x] 4.3 Test `session-end.sh` standalone: supply mock stdin JSON (`echo '{"session_id":"test-123","reason":"exit"}' | ./session-end.sh`); run with notes in the temp file at `/tmp/claude-vault-touched-test-123` (confirm journal entry created with paths), run with no temp file (confirm minimal entry is still appended), run twice in one day (confirm append-only, no overwrite) + +## 5. Skills authoring + +- [x] 5.1 Write `~/.claude/plugins/memory/skills/memory-query.md`: cover all patterns from `memory-plugin-skills/spec.md` — god-node-first discipline, `graphify query`/`path`/`explain` with `--budget`/`--dfs`/`--graph`, cross-client lookups, explicit call-out that tag-based graph traversal is unverified (ADR-014), episodic questions → memsearch +- [x] 5.2 Write `~/.claude/plugins/memory/skills/memory-write.md`: cover all rules from the spec — evergreen vs. ephemeral test, required frontmatter contract (`summary` written now, `scope/`, at least one facet tag), vault-not-repo rule, note that PostToolUse hook fires automatically on vault writes +- [x] 5.3 Write `~/.claude/plugins/memory/skills/memory-reorganize.md`: cover the spec — plan-mode-only rule, three triggers (duplicate, recurrence-promotion, oversized note), consolidation procedure, `--force` rebuild requirement after reorganization, ghost-node explanation +- [x] 5.4 Register the three skills in the plugin manifest (or `~/.claude/settings.json`) so they are available as on-demand skills in Claude Code sessions + - **Note (2026-06-05):** Skills in `~/.claude/plugins/memory/skills/` require a marketplace entry to be auto-discovered via `enabledPlugins`. The `.claude-plugin/plugin.json` manifest is in place; a marketplace registration in `~/.claude/settings.json` `extraKnownMarketplaces` pointing to `~/.claude/plugins/` is needed to complete skill auto-loading. Hooks fire independently of this. + +## 6. Hook registration + +- [x] 6.1 Register `session-start.sh` as a SessionStart hook in `~/.claude/settings.json` +- [x] 6.2 Register `post-tool-use-write.sh` as a PostToolUse hook for `Write` and `Edit` tool calls in `~/.claude/settings.json` +- [x] 6.3 Register `session-end.sh` as a SessionEnd hook in `~/.claude/settings.json` + +## 7. End-to-end validation against fixture set (ADR-013) + +- [x] 7.1 Start a Claude Code session in the fixture project — confirm SessionStart injects god-node list, at least one convention summary (if present), journal pointer, and project graph path +- [x] 7.2 Write a test vault note during the session — confirm PostToolUse fires `graphify update --file` (verify from hook log or graph.json mtime) + > **Clarification (2026-06-05):** PostToolUse fires stamp invalidation (deletes `~/.cache/graphify/vault-rebuild.stamp`), triggering deferred incremental update on next SessionStart, per deviation note in task 3.1. +- [x] 7.3 End the session — confirm SessionEnd appended a journal entry to `/journal/YYYY-MM-DD.md` with the note paths touched +- [x] 7.4 Trigger a stale rebuild: backdate the stamp file (or delete it) and restart the session — confirm SessionStart returns immediately (no freeze), background rebuild process spawns (check process list / logfile), lock file appears then clears, stamp is updated when rebuild completes; start a second concurrent session during rebuild and confirm the lock prevents a duplicate spawn +- [x] 7.5 Test the convention-injection fallback: temporarily remove the vault graph and restart the session — confirm the grep fallback surfaces convention summaries (or the section is cleanly omitted if none exist) +- [x] 7.6 Run each of the three skills (`/memory-query`, `/memory-write`, `/memory-reorganize`) in a session and confirm the guidance loads and is coherent with the fixture set context diff --git a/openspec/specs/memory-plugin-hooks/spec.md b/openspec/specs/memory-plugin-hooks/spec.md new file mode 100644 index 0000000..f2cdc85 --- /dev/null +++ b/openspec/specs/memory-plugin-hooks/spec.md @@ -0,0 +1,84 @@ +# Spec: memory-plugin-hooks + +_Purpose: Define the Claude Code hook behaviors for the memory plugin — SessionStart context injection and staleness-driven graph rebuild, PostToolUse vault graph update, SessionEnd episodic journal append, and plugin configuration._ + +_Last updated: 2026-06-05_ + +--- + +## Requirements + +### Requirement: SessionStart hook — staleness check and detached background rebuild +The plugin SHALL register a SessionStart hook that reads the rebuild stamp file at `~/.cache/graphify/vault-rebuild.stamp`. If the stamp is absent or its mtime is older than the configured threshold (default: 7 days), the hook SHALL spawn `graphify extract --backend ollama --model --force` as a **detached background process** (nohup/setsid, output redirected to a log file) and SHALL return immediately without waiting for it. The background process SHALL update the stamp file upon successful completion and remove the lock file. The hook SHALL NOT block on the rebuild — SessionStart blocks the entire session until the hook returns, so the synchronous path must be sub-second. (Source: Claude Code hooks documentation — SessionStart has no async option.) The hook SHALL write a lock file (`~/.cache/graphify/vault-rebuild.lock`) before spawning, so concurrent sessions do not launch duplicate rebuilds. + +#### Scenario: Stamp is fresh (within threshold) +- **WHEN** the SessionStart hook fires and the rebuild stamp mtime is within the configured threshold +- **THEN** no extraction run is triggered and the hook proceeds to context injection + +#### Scenario: Stamp is stale or absent +- **WHEN** the SessionStart hook fires and the rebuild stamp is absent or older than the configured threshold +- **THEN** the hook checks for a lock file; if absent, spawns a detached background rebuild and writes the lock; in all cases the hook continues immediately to context injection using the existing graph; the rebuilt graph becomes available to the next session + +#### Scenario: Rebuild already in progress (lock file present) +- **WHEN** the SessionStart hook fires and the lock file already exists (rebuild spawned by a concurrent session) +- **THEN** the hook does NOT spawn another rebuild; it logs that a rebuild is in progress and proceeds to context injection + +#### Scenario: Extraction fails during stale rebuild +- **WHEN** the background rebuild process exits non-zero +- **THEN** the background process logs the error to its log file and removes the lock file without touching the stamp; the current session is unaffected (already past injection); the next session will attempt the rebuild again + +### Requirement: SessionStart hook — vault context injection +The plugin SHALL register a SessionStart hook that injects the following into Claude's context at the start of each session: (a) the vault graph's god-node summary (top N most-connected nodes by degree, sourced from `GRAPH_REPORT.md` or equivalent query), (b) the summaries of `convention/*` vault notes (sourced via `graphify query "convention"` or, as a fallback, by grepping vault frontmatter for `convention/` tags and reading the `summary:` field), and (c) a pointer to the current day's episodic journal note path. The hook SHALL deliver this content by writing `{"additionalContext": ""}` JSON to stdout and exiting 0 — this is the confirmed Claude Code mechanism for hook-to-model context injection. (Source: Claude Code hooks documentation.) The hook reads its input from stdin JSON; relevant fields include `.cwd` (working directory) and `.source` (startup|resume|clear|compact). + +#### Scenario: Vault graph and convention notes exist +- **WHEN** SessionStart fires and the vault graph and at least one `convention/*` note exist +- **THEN** the injected context contains the god-node list, all `convention/*` summaries, and the journal note path + +#### Scenario: No convention notes exist yet +- **WHEN** SessionStart fires and no `convention/*` notes exist in the vault +- **THEN** the injected context contains the god-node list and journal pointer; the conventions section is omitted without error + +#### Scenario: Project graph exists for the current project +- **WHEN** SessionStart fires and a file exists at `/graphify-out/graph.json` +- **THEN** the injected context also includes the project graph path, so Claude can query it via `graphify query --graph ` + +### Requirement: PostToolUse hook — event-driven vault graph update +The plugin SHALL register a PostToolUse hook that fires after any `Write` or `Edit` tool call. The hook SHALL check whether the target file path is a `.md` file under the configured vault path (`~/Documents/SecondBrain`). If it is, the hook SHALL run `graphify update --file ` to merge the changed note into the vault graph. If the target path is outside the vault, the hook SHALL do nothing. + +#### Scenario: A vault note is written or edited +- **WHEN** a Write or Edit tool call completes and the target path is under the vault directory +- **THEN** the hook runs `graphify update --file ` synchronously before Claude proceeds + +#### Scenario: A non-vault file is written or edited +- **WHEN** a Write or Edit tool call completes and the target path is outside the vault directory +- **THEN** the hook does nothing; no graphify call is made + +#### Scenario: graphify update fails for a vault write +- **WHEN** `graphify update --file ` exits non-zero +- **THEN** the hook logs the error but does NOT block the tool call result from being returned; the graph may be transiently stale + +### Requirement: SessionEnd hook — episodic journal append +The plugin SHALL register a SessionEnd hook that appends a dated entry to the daily journal note at `/journal/YYYY-MM-DD.md` (creating the file if it does not exist). The entry SHALL include at minimum: the current project name (or path), the vault note paths touched during the session, and a UTC timestamp. The journal file is append-only; existing content is never overwritten. + +#### Scenario: Session ends with vault notes touched +- **WHEN** the SessionEnd hook fires and at least one vault note was written or edited during the session +- **THEN** the hook appends a timestamped entry to the daily journal file listing the touched note paths and the project context + +#### Scenario: Session ends with no vault writes +- **WHEN** the SessionEnd hook fires and no vault notes were touched +- **THEN** the hook still appends a minimal entry (timestamp + project) so the session is recorded for episodic recall + +#### Scenario: Journal file does not exist yet for the day +- **WHEN** the SessionEnd hook fires and `/journal/YYYY-MM-DD.md` does not exist +- **THEN** the hook creates the file with appropriate frontmatter (summary, scope/global, type/log) and appends the session entry + +### Requirement: Plugin configuration block +The plugin SHALL expose a configuration file at `~/.claude/plugins/memory/config.yaml` that allows setting: vault path, Graphify output directory, Ollama model name (Modelfile-baked variant), num_ctx, rebuild-stale threshold in days, and the env vars `OLLAMA_FLASH_ATTENTION`, `GRAPHIFY_OLLAMA_NUM_CTX`, `GRAPHIFY_OLLAMA_KEEP_ALIVE`. All hooks SHALL read from this config rather than hardcoding paths or values. + +#### Scenario: Default config is used +- **WHEN** the config file exists with no overrides for a setting +- **THEN** the hook uses the default value for that setting (vault: `~/Documents/SecondBrain`, threshold: 7 days, model: `qwen25-coder-7b-16k`) + +#### Scenario: User overrides vault path +- **WHEN** the config file specifies a non-default vault path +- **THEN** all hooks use the configured path for vault detection, graph paths, and journal writes diff --git a/openspec/specs/memory-plugin-skills/spec.md b/openspec/specs/memory-plugin-skills/spec.md new file mode 100644 index 0000000..43d1ae7 --- /dev/null +++ b/openspec/specs/memory-plugin-skills/spec.md @@ -0,0 +1,70 @@ +# Spec: memory-plugin-skills + +_Purpose: Define the Claude Code skill behaviors for the memory plugin — when and how to query the vault graph, write vault knowledge, and reorganize vault notes._ + +_Last updated: 2026-06-05_ + +--- + +## Requirements + +### Requirement: memory-query skill — graph retrieval guidance +The plugin SHALL provide a `memory-query` skill that teaches Claude when and how to query the vault knowledge graph versus the episodic memsearch layer. The skill SHALL specify: (a) use `graphify query ""` for evergreen knowledge questions ("how do we…", "what do we know about…"), (b) use `graphify path "" ""` to find how two concepts relate, (c) use `graphify explain ""` for a deep-dive with neighbors, (d) add `--budget N` to cap answer size (default ~2000 tokens), (e) add `--dfs --budget N` for bounded depth-first traversal, (f) add `--graph ` to query a specific project graph instead of the vault graph. The skill SHALL explicitly state that god-nodes are the entry point — query god-nodes first, then scalpel to specific nodes. The skill SHALL treat tag-scoped retrieval (e.g. "all notes tagged `client/acme`") as a SEPARATE query path from graph traversal, noting that whether shared frontmatter facet tags create graph edges is unverified (ADR-014 open question); Claude SHALL NOT assume facet tags enable graph traversal. For "what happened when" questions, the skill SHALL direct Claude to memsearch in natural language rather than the vault graph. + +#### Scenario: Claude needs to answer a knowledge question during a task +- **WHEN** a task touches a tool, client, or domain the vault may know about +- **THEN** Claude runs `graphify query ""` before relying on training knowledge, using `--budget` to keep context lean + +#### Scenario: Claude needs to find how two concepts relate +- **WHEN** a task requires understanding the relationship between two vault entities (e.g. client and a tool they use) +- **THEN** Claude runs `graphify path "" ""` to traverse the graph path between them + +#### Scenario: Cross-client knowledge lookup +- **WHEN** Claude needs to find all vault knowledge about a tool or domain regardless of which client it came from +- **THEN** Claude runs `graphify query ""` against the vault graph (not `--graph `) to get cross-client results + +#### Scenario: Project-specific knowledge lookup +- **WHEN** Claude needs knowledge specific to the current project's code graph +- **THEN** Claude runs `graphify query "" --graph /graphify-out/graph.json` using the project graph path injected by SessionStart + +#### Scenario: Episodic question ("what was I working on last week?") +- **WHEN** a question is time-anchored ("what did we decide last Tuesday", "what was I working on yesterday") +- **THEN** Claude queries memsearch in natural language, NOT the vault graph + +### Requirement: memory-write skill — when and how to write vault knowledge +The plugin SHALL provide a `memory-write` skill that teaches Claude when knowledge should be written to the vault and what the required frontmatter contract is. The skill SHALL specify: write to the vault when knowledge is evergreen and reusable across projects (tool/API behavior, conventions, client-specific facts worth reusing); do NOT write project-ephemeral state to the vault (that goes to project files or the episodic layer). The skill SHALL specify the required frontmatter fields: `summary` (one-line, written at note creation — not deferred — this is the human-authored router hint Graphify does not generate), `scope/global` or `scope/project` (default new tool/domain knowledge to `scope/global`; use `scope/project` when specific to how a client uses something), and at least one facet tag from the six-facet taxonomy (`type/`, `client/`, `project/`, `domain/`, `tool/`, `convention/`). The skill SHALL state the vault-not-repo rule: write knowledge ONLY to `~/Documents/SecondBrain`, never silently into a project repo. The skill SHALL note that writing a vault note automatically triggers the PostToolUse hook to update the graph — no manual graphify call is needed after writing. + +#### Scenario: Claude discovers how a tool's API behaves +- **WHEN** Claude learns something about a tool's authentication, rate limits, or endpoint quirks that could apply to future sessions or projects +- **THEN** Claude writes a note to the vault with the required frontmatter (summary, scope/global, type/reference, tool/) rather than embedding it in the project's CLAUDE.md + +#### Scenario: Claude establishes a convention that should apply beyond this task +- **WHEN** Claude and the user settle on a convention (coding style, workflow, decision rule) that should propagate to other projects +- **THEN** Claude writes a `convention/` note to the vault with the required frontmatter (summary, scope, type/convention) + +#### Scenario: Knowledge is project-specific (how this client uses a tool) +- **WHEN** the knowledge describes behavior specific to one client's setup, not the tool generally +- **THEN** Claude writes the note with `scope/project` and a `client/` facet tag rather than `scope/global` + +#### Scenario: Claude tries to record project-ephemeral state +- **WHEN** the information is specific to the current task and not reusable in future sessions or projects +- **THEN** Claude does NOT write it to the vault; it stays in the project repo or episodic session notes + +### Requirement: memory-reorganize skill — consolidation and plan-mode promotion +The plugin SHALL provide a `memory-reorganize` skill that teaches Claude the procedure for consolidating, deduplicating, promoting, and reorganizing vault notes. The skill SHALL specify: (a) reorganization MUST happen in plan mode — Claude proposes changes for human review before executing; (b) triggers include: duplicate coverage noticed across notes, a `scope/project` fact observed to recur across 2+ clients (promotion candidate), a note grown beyond the L1 "one concept, ~200 lines" discipline; (c) the consolidation procedure is: identify candidate notes, propose a merge/split/promote/rename plan, await human approval, execute edits, then run `graphify extract --force` to rebuild the graph clean; (d) a `--force` rebuild is required after any reorganization that removes, renames, or restructures notes (to clear ghost nodes); (e) `--force` rebuild is ALSO triggered when the SessionStart stale check fires, so reorganize skill's rebuild invocation is consistent with the staleness-check mechanism. The skill SHALL warn that `graphify update --file` does NOT prune deleted or renamed nodes — only `--force` rebuilds clear ghost nodes. + +#### Scenario: Claude notices duplicate coverage across two vault notes +- **WHEN** Claude identifies two notes that cover the same concept with overlapping content +- **THEN** Claude enters plan mode: proposes which note to keep, what to merge, and what to discard — and awaits human approval before making any edits + +#### Scenario: A project-scoped fact has recurred across multiple clients +- **WHEN** Claude observes that a fact tagged `scope/project` has appeared in 2+ client contexts, suggesting it is actually global knowledge +- **THEN** Claude proposes promoting the note to `scope/global` (updating frontmatter, merging client-specific variants if needed) in plan mode before executing + +#### Scenario: Reorganization is complete and graph needs to be rebuilt +- **WHEN** any reorganization edit has been approved and executed (notes deleted, renamed, or restructured) +- **THEN** Claude runs `graphify extract --backend ollama --model --force` to clear ghost nodes and rebuild a clean graph; updates the rebuild stamp + +#### Scenario: Note has grown beyond single-concept discipline +- **WHEN** Claude observes a vault note has grown beyond ~200 lines or covers more than one distinct concept +- **THEN** Claude proposes a split into two or more focused notes in plan mode, with proposed filenames, frontmatter, and the wikilinks that would connect them — before executing