SecondBrain/2026-06-09-claude-code-memo...

181 lines
21 KiB
Markdown
Raw Normal View History

---
summary: Architecture Decision Records for the Claude Code personal memory system — 14 decisions covering memory-type split, tooling choices, vault structure, freshness model, and graph connectivity.
tags:
- type/adr
- domain/knowledge-graphs
- domain/llm
- tool/graphify
- scope/global
source: cc-os
date: 2026-06-09
---
# Claude Code Memory System — Architecture Decision Records
A running log of decisions and *why*. Format per entry: Context · Decision · Rationale · Alternatives rejected · Status. Newest decisions extend the log; supersede rather than delete.
---
## ADR-001 — Two memory types, kept as separate systems
- **Context**: Earlier attempts to make one tool serve both "what happened" and "how do we do X" felt forced (e.g. trying to make memsearch filter knowledge by tags).
- **Decision**: Model **episodic** memory and **semantic/knowledge** memory as two separate systems with different tools.
- **Rationale**: They have different lifecycles (episodic accretes and decays; knowledge is deliberately maintained), different write paths (auto-captured vs curated with guardrails), and different query patterns ("when did we…" vs "how do we…"). Separation dissolves the earlier integration tension entirely.
- **Alternatives rejected**: One unified store (memsearch-for-everything, or a single `thoughts` table) — conflates the two and forces awkward filtering.
- **Status**: Accepted.
## ADR-002 — memsearch for the episodic layer
- **Context**: Need timeline/"what happened" memory that's NL-queryable and lazy.
- **Decision**: Adopt **memsearch** (Zilliz) off-the-shelf for episodic memory.
- **Rationale**: It already implements the daily-notes + "dreaming" pattern and the markdown-as-truth / disposable-shadow-index philosophy. Embedded **Milvus Lite** (single file), hybrid BM25+vector+RRF search, local ONNX embeddings (no API key/cost), a FileWatcher that handles deletions — **no Docker, no server**. Two-line install.
- **Alternatives rejected**: claude-mem (MCP-based — Claude must actively call search; opaque blobs vs readable markdown; overkill features). Hand-building daily notes + dreaming ourselves (reinventing a solved tool).
- **Status**: Accepted.
## ADR-003 — Flat vault with namespaced tags, not folders
- **Context**: Want a flat Obsidian vault with tags as virtual indexes, and cross-cutting filters (client × tool × convention).
- **Decision**: One **flat markdown vault**; organize via **namespaced, nested tags** (`tool/`, `client/`, `domain/`, `convention/`, `scope/`). Slashes are valid Obsidian nested tags, so `#tool` matches all children.
- **Rationale**: A note can carry several namespaces at once (`tool/semrush` + `client/clientx` + `convention/react-ts`) — folders can't express that. Enables "filter by client+tool to narrow the index." Enumerable virtual indexes ("what clients/tools exist").
- **Alternatives rejected**: Folder hierarchy (single-axis; can't do cross-cutting filters). Pure-prefix path filtering via memsearch `source_prefix` (would force directories back in).
- **Trade-off accepted**: Tags give the human/Obsidian free filtering, but the AI gets nothing for free from tags — they must be materialized into a queryable index.
- **Status**: **Refined by ADR-011** (type/ and project/ namespaces added; hierarchy-vs-facets clarified). Core decision — flat vault, namespaced tags — stands.
## ADR-004 — SQLite + Sequel (Ruby) tag index as the knowledge-layer cache
- **Context**: The AI can't use Obsidian tags directly; tag filtering needs a machine-queryable index.
- **Decision**: A small **Ruby program using the Sequel ORM over SQLite**, exposed as a **CLI**. Schema: `files(path, mtime, summary, scope)`, `tags(name)`, `files_tags` join (`many_to_many`). The summary is a **column on `files`** (an attribute), not a join.
- **Rationale**: Normalized `tags` table makes enumerating the vocabulary a first-class cheap query. The `summary` column is what turns the index from a *finder* into a *router* — the AI sees enough to pick a file without opening it (progressive disclosure, low tokens). **Failure-mode guard**: markdown is always authoritative; the SQLite file is a disposable cache that is never synced and can be rebuilt from frontmatter anytime.
- **Alternatives rejected**: Plain-markdown generated `INDEX.md` (must regenerate; grep-at-scale is token-heavy). Frontmatter grep on demand (scales badly). Milvus/Postgres for knowledge (overkill).
- **Query output**: returns **path + summary + matched tags** — tags are cheap and show *why* a result matched.
- **Status**: **Superseded by ADR-010** (Graphify replaces the Ruby/SQLite tag index). The `summary` + namespaced-tag frontmatter is **retained as note metadata**; only the bespoke Ruby/SQLite index and CLI are dropped.
## ADR-005 — Structured-first; semantic search over the vault deferred
- **Context**: Tag filtering may miss notes whose wording doesn't match the query.
- **Decision**: Ship the knowledge layer **structured-only** (tags + summaries). **Defer** meaning-based search until it demonstrably bites.
- **Rationale**: Structured tagging is the lightweight/fast thing; follow the "only level up when it bites" principle.
- **Status**: **Superseded by ADR-010.** Graphify makes the knowledge layer a graph from day one, giving structured *and* connection-based recall together. The "only level up when it bites" instinct carries forward to whether a *vector* layer is ever needed on top of the graph.
## ADR-006 — QMD as the (deferred) semantic-over-knowledge layer
- **Context**: When ADR-005's structured-only proves insufficient, want a set-and-forget semantic layer over the vault, local and Docker-free.
- **Decision**: Earmark **QMD** (github.com/tobi/qmd) for that role; do **not** install yet.
- **Rationale**: Local markdown search using **SQLite + FTS5/BM25 + local vector embeddings (EmbeddingGemma-300M GGUF) + LLM rerank**; CLI + optional MCP server; no Docker, no API keys. Complements the tag index (QMD filters by path/collection context, not first-class frontmatter tags).
- **Alternatives rejected**: Pointing memsearch at the vault (mixes episodic and knowledge corpora; its filtering is path-prefix not tags). A bespoke embedding index (reinvents QMD).
- **Status**: **Superseded by ADR-010.** Graphify's knowledge graph fills the semantic-recall role (traversal/`explain` over connections) without a separate vector system, so QMD is no longer earmarked. Revisit a vector layer only if graph traversal demonstrably misses cases where embedding similarity would win.
## ADR-007 — Lazy freshness: write-hook + session-start reconcile, no daemon/cron
- **Context**: The cache must reflect new/edited/deleted/renamed notes without becoming a resource hog or going stale.
- **Decision**: **Option A (lazy).** A `PostToolUse` hook updates the index on **AI** writes (single-file, prunes on delete). **Manual** edits are caught by a **session-start reconcile** (`index update --since` + prune of vanished paths). **No daemon, no cron.**
- **Rationale**: The AI is the primary writer, so write-time hooks give event-driven freshness with no polling. The user rarely edits the vault by hand, so a session-start reconcile is enough; a continuous `inotify` daemon would add an always-on process to manage for negligible benefit.
- **Alternatives rejected**: `inotify`/`listen` daemon (live freshness, but always-on process to manage — unnecessary). Cron reconcile (session-start covers it).
- **Status**: Accepted.
## ADR-008 — Markdown-as-truth; sync the vault, not the indexes
- **Context**: Must be accessible on a VPS / multiple machines but run local-fast.
- **Decision**: Sync the **markdown vault** to a remote machine via **git or Syncthing** (choice deferred to build time). **Graphs/indexes (Milvus Lite, Graphify `graphify-out/`) are rebuilt per machine and never synced.**
- **Rationale**: Markdown is plain text — git/Syncthing sync it trivially; lazy (hourly or continuous-async) is enough. Indexes are disposable caches; syncing binary DBs invites conflicts for no gain. Local reads stay fast; ownership and portability stay with the user.
- **Alternatives rejected**: Hosted DBs (always-remote, adds per-query latency and monthly cost, conflicts with local-fast; ownership weaker). Only worth it for real-time cross-tool memory, which was deemed overkill.
- **Status**: Accepted.
## ADR-009 — Package as a global Claude Code plugin with skills
- **Context**: Every project, on every machine, should know how to use the vault — write conventions, query patterns, the hooks, and the CLI — without per-project setup.
- **Decision**: Ship hooks + scripts + CRUD know-how as a **global Claude Code plugin with skills**, installed at the user level.
- **Rationale**: Skills carry the "when to write / what conventions / how & when to query" guidance to the model; the plugin registers the session-start / session-end / PostToolUse hooks and wires up Graphify. Global install = consistent behavior everywhere; single source of truth for the conventions themselves.
- **Status**: Accepted.
## ADR-010 — Graphify knowledge graph as the knowledge layer (supersedes ADR-004/005/006)
- **Context**: ADR-004 specced a hand-built Ruby/Sequel/SQLite tag index (+ CLI) as the machine-queryable layer over the vault, with ADR-005/006 deferring meaning-based recall to a future QMD vector layer. Before building any of it, **Graphify** (`graphify`, PyPI `graphifyy`) was evaluated — a tool that turns a folder into a queryable knowledge graph (local tree-sitter AST for code, local-SLM entity/relationship extraction for docs).
- **Decision**: Use **Graphify as the knowledge-layer engine** over the vault, with a **local Ollama** backend for doc extraction and free AST for per-project code graphs. **Drop** the Ruby/SQLite tag-index CLI (ADR-004) and the earmarked QMD layer (ADR-006); **retain** the `summary` + namespaced-tag frontmatter from ADR-003/004 as note metadata.
- **Rationale**: One off-the-shelf tool delivers both what the tag index was for (structured retrieval) and what QMD was deferred for (connection/meaning-based recall via graph traversal + `explain`) — without writing or maintaining a bespoke index, and without a vector store. Code graphs come free. Keeps the markdown-as-truth, no-Docker, no-API-key, local-first properties.
- **What's retained / changed**: `summary` stays the human-written router hint Graphify does not generate; namespaced tags stay useful for Obsidian filtering and as node attributes. How tightly metadata feeds graph queries is a **build-time refinement**, not settled here.
- **Trade-off accepted**: Graphify's `--update` doesn't prune deleted nodes (stale-node drift) — mitigated by a periodic `--force` rebuild on the session-start staleness check (ADR-007's lazy model still applies). Graphify also moves fast (flags are version-dependent; anchor to the installed version) and its headline token-savings numbers are corpus-dependent — benchmark your own.
- **Alternatives rejected**: Building the Ruby/SQLite index as originally planned (more code to own; no semantic recall); adding QMD as a second system on top (two stores where one graph suffices).
- **Status**: Accepted.
## ADR-011 — Faceted tag taxonomy: six independent namespaces (refines ADR-003)
_Date: 2026-06-04_
- **Context**: ADR-003 introduced five namespaces (`tool/`, `client/`, `domain/`, `convention/`, `scope/`). During vault-reuse assessment it became clear that (1) the existing vault uses a de-facto first-tag convention for note kind that should be made explicit and machine-queryable, and (2) for a freelancer working many projects per client, project identity deserves a first-class namespace.
- **Decision**: Knowledge-vault notes are classified by **six independent, flat tag facets** that sit side-by-side, never nested into one another:
- `type/` — note kind: `research`, `howto`, `adr`, `hub`, `plan`, `log`, `clip`, etc.
- `client/` — which client
- `project/` — which project (first-class; a freelancer's projects are the primary unit of work)
- `domain/` — knowledge domain / topic area
- `tool/` — tool-specific knowledge
- `convention/` — conventions
- …plus `scope/global` or `scope/project`
Hierarchy and relationships are expressed via **hub notes** (`type/hub`), **wikilinks**, and **Graphify knowledge-graph edges** — NOT via nested tag paths.
By convention `type/` is listed **first** in frontmatter.
- **Rationale**: The vault is flat — hierarchy is not expressed through folder paths or tag nesting. A many-to-many reality (many projects per client, knowledge domains spanning clients) maps badly onto a single-parent tree. A project hub note links out to both its `client/` and relevant `domain/` tags rather than being buried under either. Per-type `_templates` for **core types only** (research, howto, adr, hub); the long tail stays freeform. Consistent per-type structure also improves Graphify's local-SLM extraction reliability.
- **Alternatives rejected**: Hierarchical nesting in the style of `domain/{product}/{project}.md` folder structure. Rejected because: (1) the vault is flat; (2) the many-to-many reality maps badly onto a single-parent folder tree; (3) nesting one facet through another creates traversal coupling. Faceted parallel tags are the flat-vault analogue of what the Graphify graph already does with edges, so they compose naturally with the knowledge layer.
- **Status**: Accepted (supersedes the namespace list in ADR-003; core flat-vault + namespaced-tags decision stands).
## ADR-012 — Reuse an existing vault rather than creating a new one
_Date: 2026-06-04_
- **Context**: The design called for a flat markdown vault as the semantic knowledge layer. The question was whether to stand up a new vault from scratch or adopt an existing one.
- **Decision**: **Adopt an existing vault** rather than creating a new vault.
- **Rationale**: An existing vault that is already flat (all notes at root, only a `_templates/` exception), already articulates the correct "durable knowledge, not working memory" role, and contains real notes is worth preserving. Two patterns the existing vault may already include that improve on a bare-spec design:
1. An "act without being asked" section specifying *when* the AI should proactively query the vault — a behavioral spec.
2. Project-config hub notes with a tag-inference table (auto-tag by path pattern) that operationalizes *how* to tag a note from a given project.
- **Adaptations typically required (migration cost)**:
- Add `summary:` frontmatter to existing notes.
- Migrate flat unnamespaced tags to the six-facet namespaced form (per ADR-011).
- Add `scope/global` or `scope/project` to each note.
- Initialize git in the vault (required by ADR-008's sync strategy).
- Replace any legacy vault-search script references with `graphify query` (ADR-010).
These are mechanical schema migrations, not structural rework.
- **Alternatives rejected**: Starting fresh. Rejected because the hardest design decision — flat structure, durable-knowledge-only role, governance philosophy — is already made and practiced in an existing vault. The improved behavioral patterns and the existing notes are worth preserving.
- **Status**: Accepted.
## ADR-013 — Build-first / migrate-incrementally (build-order inversion)
_Date: 2026-06-04_
- **Context**: The original build order front-loaded bulk vault migration as Step 1 — migrating all existing notes and all projects to the ADR-011 six-facet taxonomy before the system existed to validate them. This committed to a schema and workflow before any end-to-end path had been exercised. The risk: locking in an approach that fails at scale, with no feedback loop until the entire vault has been touched.
- **Decision**: **Invert the build order.** The full system is built and validated against a small **510 note fixture set** first. Bulk vault migration is deferred to the final stage. The first real-data validation uses **one small project that contains both code AND documents**, exercising both the local-SLM doc-extraction path and the tree-sitter code path in the same run. After that single project validates end-to-end, remaining projects are onboarded **one at a time** with an observe-and-adjust step between each.
- **Rationale**: Validates the ADR-011 taxonomy and vault conventions against the real Graphify extraction pipeline before the entire vault is committed. The first mixed code+docs project surfaces both extraction paths (SLM for docs, tree-sitter for code) early, when corrections are cheap. Per-project rollout keeps the blast radius of any schema or workflow correction small.
- **Alternatives rejected**:
- **Keep migration-first**: Front-loads all notes and all projects before any end-to-end validation exists.
- **Big-bang migrate everything after build**: Build against fixtures, then migrate all notes and all projects in one batch at the end. Per-project rollout with intermediate checkpoints is strictly safer.
- **Status**: Accepted.
## ADR-014 — Graph connectivity comes from authored structure; migration scaffolding is a first-class prerequisite
_Date: 2026-06-05_
- **Context**: ADR-011 specified hub notes + wikilinks + Graphify graph edges as the mechanism for expressing hierarchy and cross-note relationships, with ADR-013 deferring bulk vault migration. Before build began, an empirical test compared a cached-replay graph (per-fixture isolated extractions) against a clean single-pass deep extraction against a real vault under Graphify 0.8.31 + qwen2.5-coder:7b. [primary/measured — 2026-06-05]
- **Decision**: **The connective spine of the knowledge graph must be author-provided.** Hub notes and wikilinks are not optional scaffolding to add "someday" — they are the mechanism by which Graphify connects thematically related notes, and they must be authored **as part of the migration step**, not deferred to bulk import. Migration scaffolding (hub notes + wikilinks for key concepts) is a **first-class build deliverable**.
- **Rationale**: The empirical test found that Graphify is a **structure extractor, not a topic clusterer**. Even at `--mode deep --token-budget 8000`, no emergent shared-topic hub nodes appeared. All cross-note edges observed came from explicit references, wikilinks, or document-level semantic similarity — not from shared thematic identity. A practical test query ("how do we do niche prospecting outreach for pest control?") returned 3 starting notes and traversal could not reach related notes in separate communities (no connecting edges). This confirms that useful retrieval is gated on migration scaffolding, not on Graphify's extraction power.
- **Relationship to ADR-011**: Validates the hub-notes + wikilinks half of ADR-011 empirically. The facet-tag half is **not yet validated**: no edge was observed to arise from shared frontmatter facet tags alone. Whether `client/X` or `tool/Y` tags create graph connectivity is an **open question** — do not assume facet tags contribute to graph traversal retrieval until tested.
- **Relationship to ADR-013**: Refines the migrate-incrementally stage. "Migration" must include hub note authoring and wikilink addition for key concepts, not just frontmatter schema migration.
- **Alternatives rejected**: Relying on the SLM to auto-cluster topics and synthesize hub entities — **empirically does not happen** at 7B model size with `--mode deep`. The design already intended human-authored hub notes; the test confirms that intent was correct and the fallback assumption ("maybe the LLM will do it") is false.
- **Deferred**:
1. **Facet-tag-to-graph-edge question**: Do shared frontmatter facet tags (`client/`, `tool/`, `domain/`, etc.) cause Graphify to create edges between notes, or does graph connectivity come only from explicit wikilinks/references and semantic similarity? This was NOT tested. Resolve before designing graph-traversal retrieval skills.
2. **Larger extraction model**: Whether a substantially larger SLM (14B, 30B) would synthesize emergent topic-hub nodes is untested.
3. **`reasoning_effort:"none"` patch**: The clean run required a local patch to `graphify/llm.py`. Track the upstream Graphify issue tracker for an official fix; treat the installed version as pinned until resolved.
- **Status**: Accepted. Refines ADR-013 (migrate-incrementally phase scope) and empirically validates the hub-notes/wikilinks mechanism of ADR-011 while flagging its facet-tag half as an open question.
---
## Rejected tools (summary)
| Tool | Why rejected |
|------|--------------------------|
| MemPalace | Storage not readable markdown; isolated drawers (knowledge not interconnected); fights self-managing + cross-linking goals |
| Recall / LightRAG | Content knowledge bases / deep research, not operational memory; Recall = hosted, you don't own data; LightRAG = enterprise overkill |
| OpenBrain / Mem0 | Always-remote DB → latency + cost; conflicts with local-fast lazy-sync; only pays off for real-time cross-tool memory (overkill) |
| Postgres / Milvus server | Unnecessary — Graphify's local graph (knowledge) + Milvus Lite (memsearch episodic) cover everything locally with no Docker |
| claude-mem | MCP-based (Claude must call search); opaque blobs vs readable markdown; feature overkill |
| Ruby/SQLite tag index CLI; QMD vector layer | Superseded by Graphify before build — one knowledge graph replaces both the structured index and the deferred semantic layer (ADR-010) |