cc-os/docs/memory-system/03-architecture-decisions.md

366 lines
26 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Architecture Decision Records
_Last updated: 2026-06-05_
A running log of decisions and *why*. Format per entry: Context · Decision · Rationale ·
Alternatives rejected · Status. Newest decisions extend the log; supersede rather than delete.
---
## ADR-001 — Two memory types, kept as separate systems
- **Context**: Earlier attempts to make one tool serve both "what happened" and "how do we do
X" felt forced (e.g. trying to make memsearch filter knowledge by tags).
- **Decision**: Model **episodic** memory and **semantic/knowledge** memory as two separate
systems with different tools.
- **Rationale**: They have different lifecycles (episodic accretes and decays; knowledge is
deliberately maintained), different write paths (auto-captured vs curated with guardrails),
and different query patterns ("when did we…" vs "how do we…"). Separation dissolves the
earlier integration tension entirely.
- **Alternatives rejected**: One unified store (memsearch-for-everything, or OpenBrain's single
`thoughts` table) — conflates the two and forces awkward filtering.
- **Status**: Accepted.
## ADR-002 — memsearch for the episodic layer
- **Context**: Need timeline/"what happened" memory (Goal 3) that's NL-queryable and lazy.
- **Decision**: Adopt **memsearch** (Zilliz) off-the-shelf for episodic memory.
- **Rationale**: It already implements the OpenClaw daily-notes + "dreaming" pattern and the
markdown-as-truth / disposable-shadow-index philosophy we'd otherwise build. Embedded
**Milvus Lite** (single file), hybrid BM25+vector+RRF search, local ONNX embeddings (no API
key/cost), a FileWatcher that handles deletions — **no Docker, no server**. Two-line install.
- **Alternatives rejected**: claude-mem (MCP-based — Claude must actively call search; opaque
blobs vs readable markdown; overkill features). Hand-building daily notes + dreaming
ourselves (reinventing a solved tool).
- **Status**: Accepted.
## ADR-003 — Flat vault with namespaced tags, not folders
- **Context**: Connelly/Huryn organize by folders (`tools/`, `domain/`). User wants a flat
Obsidian vault with tags as virtual indexes, and cross-cutting filters (client × tool ×
convention).
- **Decision**: One **flat markdown vault**; organize via **namespaced, nested tags**
(`tool/`, `client/`, `domain/`, `convention/`, `scope/`). Slashes are valid Obsidian nested
tags, so `#tool` matches all children.
- **Rationale**: A note can carry several namespaces at once (`tool/semrush` +
`client/sesame3g` + `convention/react-ts`) — folders can't express that. Enables "filter by
client+tool to narrow the index." Enumerable virtual indexes ("what clients/tools exist").
- **Alternatives rejected**: Folder hierarchy (single-axis; can't do cross-cutting filters).
Pure-prefix path filtering via memsearch `source_prefix` (would force directories back in).
- **Trade-off accepted**: Tags give the *human/Obsidian* free filtering, but the *AI* gets
nothing for free from tags — we must materialize them into a queryable index (see ADR-004).
- **Status**: **Refined by ADR-011** (type/ and project/ namespaces added; hierarchy-vs-facets
clarified). Core decision — flat vault, namespaced tags — stands.
## ADR-004 — SQLite + Sequel (Ruby) tag index as the knowledge-layer cache
- **Context**: The AI can't use Obsidian tags directly; tag filtering needs a machine-queryable
index. A previous `~/Documents/SecondBrain/` tag database was lost track of.
- **Decision**: A small **Ruby program using the Sequel ORM over SQLite**, exposed as a **CLI**.
Schema: `files(path, mtime, summary, scope)`, `tags(name)`, `files_tags` join
(`many_to_many`). The summary is a **column on `files`** (an attribute), not a join.
- **Rationale**: Normalized `tags` table makes enumerating the vocabulary a first-class cheap
query (the "virtual index" goal). The `summary` column is what turns the index from a
*finder* into a *router* — the AI sees enough to pick a file without opening it (progressive
disclosure, low tokens). Ruby + Sequel + CLI keeps the contract clean and the DB swappable;
the AI never touches SQLite directly.
- **Failure-mode guard (the lost-SecondBrain lesson)**: **markdown is always authoritative; the
SQLite file is a disposable cache** that is never synced and can be rebuilt from frontmatter
anytime (`index update --rebuild`).
- **Alternatives rejected**: Plain-markdown generated `INDEX.md` (must regenerate; grep-at-scale
is token-heavy). Frontmatter grep on demand (scales badly). Milvus/Postgres for knowledge
(overkill; QMD/memsearch prove SQLite is enough — see ADR-006/008).
- **Query output**: returns **path + summary + matched tags** (option C) — tags are cheap and
show *why* a result matched, useful for cross-client queries.
- **Status**: **Superseded by ADR-010** (Graphify replaces the Ruby/SQLite tag index). The
`summary` + namespaced-tag frontmatter this ADR introduced is **retained as note metadata**;
only the bespoke Ruby/SQLite index and its CLI are dropped.
## ADR-005 — Structured-first; semantic search over the vault deferred
- **Context**: Tag filtering ("client/sesame3g + tool/semrush") may miss notes whose wording
doesn't match the query ("how do we use semrush" vs a note titled "search analytics
integration").
- **Decision**: Ship the knowledge layer **structured-only** (tags + summaries). **Defer**
meaning-based search over the vault until it demonstrably bites.
- **Rationale**: Structured tagging is the lightweight/fast thing the user wants, and the
summary+tag design is built to make it work. Follow the video's "only level up when it bites."
- **Status**: **Superseded by ADR-010.** The premise (ship structured-only, bolt on semantics
later) no longer holds: Graphify makes the knowledge layer a graph from day one, giving
structured *and* connection-based recall together. The "only level up when it bites" instinct
carries forward to whether a *vector* layer is ever needed on top of the graph.
## ADR-006 — QMD as the (deferred) semantic-over-knowledge layer
- **Context**: When ADR-005's structured-only proves insufficient, we want a set-and-forget
semantic layer over the vault, local and Docker-free.
- **Decision**: Earmark **QMD** (github.com/tobi/qmd) for that role; do **not** install yet.
- **Rationale**: Local markdown search using **SQLite + FTS5/BM25 + local vector embeddings
(EmbeddingGemma-300M GGUF) + LLM rerank**; CLI + optional **MCP server**; no Docker, no API
keys. Validates that SQLite + a local vector model suffices (no Milvus/Postgres for
knowledge). Complements the tag index (QMD filters by path/collection context, not
first-class frontmatter tags), so it adds semantic recall without replacing structured
filtering.
- **Alternatives rejected**: Pointing memsearch at the vault (mixes episodic and knowledge
corpora; its filtering is path-prefix not tags). A bespoke embedding index (reinvents QMD).
- **Status**: **Superseded by ADR-010.** Graphify's knowledge graph fills the
semantic-recall role (traversal/`explain` over connections) without a separate vector system,
so QMD is no longer earmarked. Revisit a vector layer only if graph traversal demonstrably
misses cases where embedding similarity would win.
## ADR-007 — Lazy freshness: write-hook + session-start reconcile, no daemon/cron
- **Context**: The cache must reflect new/edited/deleted/renamed notes without becoming a
resource hog or going stale on renames.
- **Decision**: **Option A (lazy).** A `PostToolUse` hook updates the index on **AI** writes
(single-file, prunes on delete). **Manual** edits are caught by a **session-start reconcile**
(`index update --since` + prune of vanished paths). **No daemon, no cron.**
- **Rationale**: The AI is the primary writer, so write-time hooks give event-driven freshness
with no polling. The user rarely edits the vault by hand, so a session-start reconcile is
enough; a continuous `inotify` daemon (the `listen` gem) would add an always-on process to
manage/sync for negligible benefit. Matches the user's "lazy sync is fine" stance.
- **Alternatives rejected**: `inotify`/`listen` daemon (live freshness, but always-on process
to manage — unnecessary). Cron reconcile ("seems silly" per user; session-start covers it).
- **Status**: Accepted.
## ADR-008 — Markdown-as-truth; sync the vault, not the indexes
- **Context**: Must be accessible on a VPS / multiple machines but run local-fast (Goal 4).
- **Decision**: Sync the **markdown vault** to the VPS via **git or Syncthing** (choice deferred
to build time). **Graphs/indexes (Milvus Lite, Graphify `graphify-out/`) are rebuilt per
machine and never synced.**
- **Rationale**: Markdown is plain text — git/Syncthing sync it trivially; lazy (hourly or
continuous-async) is enough. Indexes are disposable caches; syncing binary DBs invites
conflicts for no gain. Local reads stay fast; ownership and portability stay with the user.
- **Alternatives rejected**: **OpenBrain / Mem0** hosted DBs — always-remote, adds per-query
latency and monthly cost, conflicts with local-fast; ownership weaker (Mem0 especially).
Only worth it for real-time cross-tool memory, which the user called overkill.
- **Status**: Accepted.
## ADR-009 — Package as a global Claude Code plugin with skills
- **Context**: Every project, on every machine, should know how to use the vault — write
conventions, query patterns, the hooks, and the CLI — without per-project setup.
- **Decision**: Ship hooks + scripts + CRUD know-how as a **global Claude Code plugin with
skills**, installed at the user level.
- **Rationale**: Skills carry the "when to write / what conventions / how & when to query"
guidance to the model; the plugin registers the session-start / session-end / PostToolUse
hooks and wires up Graphify (extraction/update/query + MCP server). Global install =
consistent behavior everywhere; single source of truth for the conventions themselves.
- **Status**: Accepted (to be built — see 04-build-plan.md).
## ADR-010 — Graphify knowledge graph as the knowledge layer (supersedes ADR-004/005/006)
- **Context**: ADR-004 specced a hand-built Ruby/Sequel/SQLite tag index (+ CLI) as the
machine-queryable layer over the vault, with ADR-005/006 deferring meaning-based recall to a
future QMD vector layer. Before building any of it, we evaluated **Graphify** (`graphify`,
PyPI `graphifyy`) — a tool that turns a folder into a queryable knowledge graph (local
tree-sitter AST for code, local-SLM entity/relationship extraction for docs). See
`06-graphify-evaluation.md`.
- **Decision**: Use **Graphify as the knowledge-layer engine** over the vault, with a **local
Ollama** backend for doc extraction and free AST for per-project code graphs. **Drop** the
Ruby/SQLite tag-index CLI (ADR-004) and the earmarked QMD layer (ADR-006); **retain** the
`summary` + namespaced-tag frontmatter from ADR-003/004 as note metadata.
- **Rationale**: One off-the-shelf tool delivers both what the tag index was for (structured
retrieval) and what QMD was deferred for (connection/meaning-based recall via graph
traversal + `explain`) — without writing or maintaining a bespoke index, and without a vector
store. Code graphs come free. Keeps the markdown-as-truth, no-Docker, no-API-key, local-first
properties (extraction runs against local Ollama). Net scope reduction: the entire Ruby build
(old critical-path Step 2) and the QMD layer are removed.
- **What's retained / changed**: `summary` stays the human-written router hint Graphify does not
generate; namespaced tags stay useful for Obsidian filtering and as node attributes. How
tightly metadata should feed graph queries is a **build-time refinement**, not settled here.
- **Trade-off accepted**: Graphify's `--update` doesn't prune deleted nodes (stale-node drift) —
mitigated by a periodic `--force` rebuild on the session-start staleness check (ADR-007's lazy
model still applies). Graphify also moves fast (flags are version-dependent; anchored to
v0.8.30) and its headline token-savings numbers are corpus-dependent — benchmark our own.
- **Alternatives rejected**: Building the Ruby/SQLite index as originally planned (more code to
own; no semantic recall); adding QMD as a second system on top (two stores where one graph
suffices).
- **Status**: Accepted (to be built — see 04-build-plan.md and 06-graphify-evaluation.md).
## ADR-011 — Faceted tag taxonomy: six independent namespaces (refines ADR-003)
_Date: 2026-06-04_
- **Context**: ADR-003 introduced five namespaces (`tool/`, `client/`, `domain/`,
`convention/`, `scope/`). During vault-reuse assessment (ADR-012) it became clear that (1)
the existing SecondBrain vault uses a de-facto first-tag convention for note kind
(research/plan/log/adr/howto) that should be made explicit and machine-queryable, and (2)
for a freelancer working many projects per client, project identity deserves a first-class
namespace rather than being implied by `client/` or `domain/`.
- **Decision**: Knowledge-vault notes are classified by **six independent, flat tag facets**
that sit side-by-side, never nested into one another:
- `type/` — note kind: `research`, `howto`, `adr`, `hub`, `plan`, `log`, `clip`, etc.
- `client/` — which client
- `project/` — which project (first-class; a freelancer's projects are the primary unit of
work)
- `domain/` — knowledge domain / topic area
- `tool/` — tool-specific knowledge
- `convention/` — conventions
- …plus `scope/global` or `scope/project` (retained from ADR-003)
Hierarchy and relationships are expressed via **hub notes** (`type/hub`), **wikilinks**, and
**Graphify knowledge-graph edges** — NOT via nested tag paths.
By convention `type/` is listed **first** in frontmatter, preserving the SecondBrain vault's
existing type-first ordering habit and making the note kind immediately visible.
- **Rationale**: The vault is flat — hierarchy is not expressed through folder paths or tag
nesting. The user's reality is many-to-many (many projects per client, knowledge domains
spanning clients), which a single-parent tree models badly and forces false hierarchy. A
project hub note links out to both its `client/` and relevant `domain/` tags rather than
being buried under either. Per-type `_templates` will be provided for **core types only**
(research, howto, adr, hub); the long tail stays freeform until a pattern earns a template.
Consistent per-type structure also improves Graphify's local-SLM extraction reliability.
- **Alternatives rejected**: Hierarchical nesting in the style of John Conneely's
`domain/{product}/{project}.md` folder structure (from the youngleaders.tech article "How I
finally sorted my Claude Code memory" — **secondary/interview-grade source, not verified
against primary implementation**). Rejected because: (1) the vault is flat — hierarchy is
not expressed through folder paths; (2) the user's many-to-many reality maps badly onto a
single-parent folder tree and forces false hierarchy; (3) nesting one facet through another
(e.g. `domain/client/project`) creates Law-of-Demeter-style traversal coupling. Conneely's
structure was the inspiration but diverges here on hierarchy-vs-facets. Faceted parallel tags
are the flat-vault analogue of what the Graphify graph already does with edges, so they
compose naturally with the chosen knowledge layer.
- **Status**: Accepted (supersedes the namespace list in ADR-003; core flat-vault +
namespaced-tags decision stands).
## ADR-012 — Reuse the existing SecondBrain vault as the knowledge vault
_Date: 2026-06-04_
- **Context**: The design called for a flat markdown vault as the semantic knowledge layer
(ADR-003/008/010). The question was whether to stand up a new `~/brain` vault from scratch
or adopt the existing `~/Documents/SecondBrain` vault.
- **Decision**: **Adopt `~/Documents/SecondBrain`** as the knowledge vault rather than
creating a new vault.
- **Rationale**: Assessment found the SecondBrain vault is already flat (all notes at root,
only a `_templates/` exception — exactly what the design permits), already articulates the
correct "durable knowledge, not working memory" role in its `CLAUDE.md` and
`vault-conventions.md`, and contains ~20 real notes. It also includes two patterns that
**improve on the current design** and should be adopted:
1. `vault-conventions.md`'s "act without being asked" section specifying *when* the AI
should proactively query the vault — a behavioral spec the cc-os docs lacked.
2. Project-config hub notes with a tag-inference table (auto-tag by path pattern) that
operationalizes *how* to tag a note from a given project.
- **Adaptations required (migration cost)**:
- Add `summary:` frontmatter to existing notes.
- Migrate flat unnamespaced tags to the six-facet namespaced form (per ADR-011).
- Add `scope/global` or `scope/project` to each note.
- Initialize git in the vault (no `.git` exists yet — required by ADR-008's sync strategy).
- Replace the vault's `~/.claude/scripts/vault_search.rb` reference (script does not exist)
with `graphify query` (ADR-010).
These are mechanical schema migrations, not structural rework.
- **Alternatives rejected**: Starting fresh with a new `~/brain` vault. Rejected because the
hardest design decision — flat structure, durable-knowledge-only role, governance philosophy
— is already made and practiced in SecondBrain. The improved behavioral patterns
(proactive-query spec, tag-inference table) and the existing notes are worth preserving; the
remaining work is mechanical migration.
- **Status**: Accepted.
## ADR-013 — Build-first / migrate-incrementally (build-order inversion)
_Date: 2026-06-04_
- **Context**: The build runbook (`05-implementation-process.md`) originally front-loaded bulk
vault migration as Step 1 — migrating all ~20 existing SecondBrain notes and all projects to
the ADR-011 six-facet taxonomy before the system existed to validate them. This committed to
a schema and workflow (the tag taxonomy from ADR-011, the vault-reuse choice from ADR-012,
and Graphify extraction behavior) before any end-to-end path had been exercised. The risk:
locking in an approach that fails at scale, with no feedback loop until the entire vault has
been touched.
- **Decision**: **Invert the build order.** The full system is built and validated against a
small **510 note fixture set** first. Bulk vault migration is deferred to the final stage.
The first real-data validation uses **one small project that contains both code AND
documents**, exercising both the local-SLM doc-extraction path and the tree-sitter code path
in the same run. After that single project validates end-to-end, remaining projects are
onboarded **one at a time** with an observe-and-adjust step between each.
- **Rationale**: Validates the ADR-011 taxonomy and ADR-012 vault conventions against the real
Graphify extraction pipeline before the entire vault is committed. The first mixed code+docs
project surfaces both extraction paths (SLM for docs, tree-sitter for code) early, when
corrections are cheap. Per-project rollout keeps the blast radius of any schema or workflow
correction small; each project is an opportunity to observe and adjust rather than discover
problems across 20 notes at once. This is consistent with the "markdown-as-truth, indexes are
disposable" principle (ADR-008): the vault notes are durable, but the extraction schema should
be validated before it shapes all of them.
- **Alternatives rejected**:
- **Keep migration-first (status quo)**: Front-loads all ~20 notes and all projects before
any end-to-end validation exists. Commits to ADR-011's taxonomy and ADR-012's migration
steps against the full vault without a feedback loop — exactly the gap this decision closes.
- **Big-bang migrate everything after build**: Build against fixtures, then migrate all notes
and all projects in one batch at the end. Avoids the pre-build commitment problem but still
risks a single large irreversible migration with no observe-and-adjust loop between units.
Per-project rollout with intermediate checkpoints is strictly safer.
- **Cross-references**: ADR-011 (six-facet tag taxonomy — the schema being validated);
ADR-012 (SecondBrain vault reuse — the migration steps this order defers).
- **Status**: Accepted (updates `05-implementation-process.md` build order).
## ADR-014 — Graph connectivity comes from authored structure; migration scaffolding is a first-class prerequisite
_Date: 2026-06-05_
- **Context**: ADR-011 specified hub notes + wikilinks + Graphify graph edges as the mechanism
for expressing hierarchy and cross-note relationships, with ADR-013 deferring bulk vault
migration to the final stage. Before build began, a discriminating empirical test compared a
cached-replay graph (per-fixture isolated extractions) against a clean single-pass deep
extraction (`graphify extract ~/Documents/SecondBrain --backend ollama --model
qwen25-coder-7b-16k --max-concurrency 1 --token-budget 8000 --mode deep --exclude
.obsidian`) against the real `~/Documents/SecondBrain` vault under Graphify 0.8.31 +
qwen2.5-coder:7b. See `07-graph-connectivity-findings.md` for the full data and methodology.
[primary/measured — 2026-06-05 session]
- **Decision**: **The connective spine of the knowledge graph must be author-provided.** Hub
notes and wikilinks are not optional scaffolding to add "someday" — they are the mechanism
by which Graphify connects thematically related notes, and they must be authored **as part of
the migration step**, not deferred to bulk import. Migration scaffolding (hub notes +
wikilinks for key concepts) is treated as a **first-class build deliverable** in the
migrate-incrementally phase of ADR-013.
- **Rationale**: The empirical test found that Graphify is a **structure extractor, not a topic
clusterer**. Even at `--mode deep --token-budget 8000`, no emergent shared-topic hub nodes
appeared (no "Pest Control" node, no "Niche Prospecting" node). All cross-note edges observed
came from explicit references, wikilinks, or document-level semantic similarity — not from
shared thematic identity. A practical test query ("how do we do niche prospecting outreach for
pest control?") returned 3 starting notes and traversal could not reach the email templates /
ACV data / business-model notes (separate communities, no connecting edges). This confirms
that useful retrieval is gated on migration scaffolding, not on Graphify's extraction power.
The clean single-pass run also showed the cached graph was partially a build artifact (cross-
note edges rose from 41% to 78% in a single-pass run), but the structural finding — no
emergent hub nodes — held in both runs.
- **Relationship to ADR-011**: Validates the hub-notes + wikilinks half of ADR-011 empirically.
The facet-tag half is **not yet validated**: no edge was observed to arise from shared
frontmatter facet tags alone. Whether `client/X` or `tool/Y` tags create graph connectivity
is an **open question** — see "Deferred" below. Do not assume facet tags contribute to graph
traversal retrieval until tested.
- **Relationship to ADR-013**: Refines the migrate-incrementally stage. "Migration" must be
defined to include hub note authoring and wikilink addition for key concepts, not just
frontmatter schema migration (adding `summary:` and namespaced facet tags). The build plan
(`04-build-plan.md`) should be updated to name this deliverable explicitly.
- **Alternatives rejected**: Relying on the SLM to auto-cluster topics and synthesize hub
entities — **empirically does not happen** at 7B model size with `--mode deep`. The design
already intended human-authored hub notes for this; the test confirms that intent was correct
and the fallback assumption ("maybe the LLM will do it") is false.
- **Deferred**:
1. **Facet-tag-to-graph-edge question**: Do shared frontmatter facet tags (`client/`,
`tool/`, `domain/`, etc.) cause Graphify to create edges between notes, or does graph
connectivity come only from explicit wikilinks/references and semantic similarity? This was
NOT tested. Resolve before designing graph-traversal retrieval skills.
2. **Larger extraction model**: Whether a substantially larger SLM (14B, 30B) would
synthesize emergent topic-hub nodes is untested. Secondary — the design does not depend on
it — but worth one test run before the build ships.
3. **`reasoning_effort:"none"` patch**: The clean run required a local patch to `graphify/
llm.py`. Track the upstream Graphify issue tracker for an official fix; treat the
installed version as pinned until resolved.
- **Status**: Accepted. Refines ADR-013 (migrate-incrementally phase scope) and empirically
validates the hub-notes/wikilinks mechanism of ADR-011 while flagging its facet-tag half as
an open question.
## Rejected tools (summary)
| Tool | Why rejected for our use |
|------|--------------------------|
| MemPalace (L4) | Storage not readable markdown; isolated drawers (knowledge not interconnected); fights self-managing + cross-linking goals |
| Recall / LightRAG (L5) | Content knowledge bases / deep research, not operational memory; Recall = hosted, you don't own data; LightRAG = enterprise overkill |
| OpenBrain / Mem0 (L6) | Always-remote DB → latency + cost; conflicts with local-fast lazy-sync; only pays off for real-time cross-tool memory (user: overkill) |
| Postgres / Milvus server | Unnecessary — Graphify's local graph (knowledge) + Milvus Lite (memsearch episodic) cover everything locally with no Docker |
| claude-mem | MCP-based (Claude must call search); opaque blobs vs readable markdown; feature overkill |
| Ruby/SQLite tag index CLI; QMD vector layer | Superseded by Graphify before build — one knowledge graph replaces both the structured index and the deferred semantic layer (ADR-010) |