Add 3-layer memory system design documentation
This documents the complete architecture for a personal, cross-project memory system for Claude Code: three-layer (episodic/knowledge/deferred- semantic) design using Milvus Lite for fast local storage with lazy remote sync. Includes video synthesis, architecture decisions with rejected alternatives (MemPalace, Mem0, Postgres, LightRAG), concrete build plan with scripts and hooks, and initial handoff for implementation.
This commit is contained in:
jared
parent
b0695dd4b5
commit
54b00364cd
|
|
@ -0,0 +1,39 @@
|
||||||
|
# Memory System — Documentation Set
|
||||||
|
|
||||||
|
A personal, cross-project memory system for Claude Code: lightweight, fast, tag-organized,
|
||||||
|
self-evolving, local-fast with lazy remote sync. Built for a multi-client freelancer who
|
||||||
|
wants the AI to learn things once and reuse them everywhere, without bloating project repos.
|
||||||
|
|
||||||
|
## Read in this order
|
||||||
|
|
||||||
|
1. **[01-video-synthesis.md](01-video-synthesis.md)** — Synthesis of the "6 Levels of Claude
|
||||||
|
Code Memory" video. What each level/tool is, where memory lives, how it's retrieved, and
|
||||||
|
the author's recommendations. Background reading; the source of many ideas below.
|
||||||
|
2. **[02-system-design.md](02-system-design.md)** — The architecture we settled on. Three
|
||||||
|
layers (episodic / knowledge / deferred-semantic), the knowledge-layer internals, data
|
||||||
|
model, hooks, sync, and guardrails. **Start here if you only read one.**
|
||||||
|
3. **[03-architecture-decisions.md](03-architecture-decisions.md)** — ADR log. Every decision
|
||||||
|
and *why*, including what we rejected and why (MemPalace, OpenBrain/Mem0, Postgres,
|
||||||
|
Recall/LightRAG) and what we deferred (semantic search over the vault).
|
||||||
|
4. **[04-build-plan.md](04-build-plan.md)** — How a human builds this, step by step. The
|
||||||
|
scripts, the hooks, the CRUD lifecycle, the AI's write/query conventions, and the
|
||||||
|
Claude Code plugin + skills that package it for global install.
|
||||||
|
5. **[05-handoff.md](05-handoff.md)** — Where we are, what's decided, what's open, and the
|
||||||
|
first concrete actions for the next session.
|
||||||
|
|
||||||
|
## One-paragraph summary
|
||||||
|
|
||||||
|
Two complementary memory types kept as **separate systems**: **episodic** ("what happened,
|
||||||
|
when") handled by **memsearch** (Milvus Lite, embedded, auto-captured session/journal notes),
|
||||||
|
and **semantic/knowledge** ("how do we…") handled by a **flat Obsidian markdown vault** that
|
||||||
|
is the single source of truth, organized by **namespaced nested tags** (`tool/`, `client/`,
|
||||||
|
`domain/`, `convention/`, `scope/`) rather than folders, and indexed by a small **Ruby +
|
||||||
|
Sequel + SQLite** tag index exposed through a CLI. Retrieval is **hook-injected and
|
||||||
|
on-demand** so project repos stay thin. Freshness is **lazy** (a write-time hook plus a
|
||||||
|
session-start reconcile — no daemon, no cron). Everything is markdown-as-truth and syncs to a
|
||||||
|
VPS via **git/Syncthing**; the indexes are disposable and rebuilt per machine. A meaning-based
|
||||||
|
search layer over the vault (**QMD**) is designed-in but **deferred** until structured tagging
|
||||||
|
proves insufficient. The whole thing ships as a **global Claude Code plugin with skills** so
|
||||||
|
every project knows how to use it.
|
||||||
|
|
||||||
|
_Last updated: 2026-06-03_
|
||||||
|
|
@ -0,0 +1,122 @@
|
||||||
|
# Video Synthesis — The 6 Levels of Claude Code Memory
|
||||||
|
|
||||||
|
Synthesis of the YouTube video "The 6 Levels of Claude Code Memory" plus the referenced
|
||||||
|
articles (John Connelly / Pawel Huryn). This is background: the source of several ideas in our
|
||||||
|
design. Our system borrows from Levels 1–3 and deliberately ignores 4–6.
|
||||||
|
|
||||||
|
## The framing
|
||||||
|
|
||||||
|
Every memory system answers the same question: **when you give Claude Code a task, how does it
|
||||||
|
pull the right context at the right time?** Each level differs on just two axes:
|
||||||
|
|
||||||
|
- **Where memory lives** — the storage mechanism and file structure (markdown vs vectors,
|
||||||
|
local vs cloud).
|
||||||
|
- **How Claude gets it** — the retrieval stage (auto-injected into context, searched on
|
||||||
|
demand in a DB, etc.).
|
||||||
|
|
||||||
|
The recurring enemy is **context rot**: as you load more context, the model recalls less of it
|
||||||
|
reliably. The cure throughout is **progressive disclosure** — load a small *index*, pull the
|
||||||
|
detail only when needed.
|
||||||
|
|
||||||
|
## The levels
|
||||||
|
|
||||||
|
### Level 1 — What ships natively (CLAUDE.md + memory.md / automemory)
|
||||||
|
|
||||||
|
- **CLAUDE.md**: plain markdown in the project, always loaded every session like a system
|
||||||
|
prompt. Hierarchical (project-folder → root → individual project); lower levels inherit the
|
||||||
|
parent but local rules win on conflict.
|
||||||
|
- **Common mistake**: stuffing it full (brand guide, tone doc, client list) → burns context →
|
||||||
|
context rot. **Rule of thumb: keep CLAUDE.md under 200 lines**; push larger context into
|
||||||
|
separate files *referenced* from CLAUDE.md so they load only when needed.
|
||||||
|
- **memory.md / automemory**: `/memory` shows imported/project/user memory. Auto-memory keeps a
|
||||||
|
per-project `memory.md` that acts as an **index of pointers** to many small memory files —
|
||||||
|
Claude quietly takes notes in the background and builds the index.
|
||||||
|
- Anthropic is clearly working on this natively (leaked references to an unreleased always-on
|
||||||
|
consolidation daemon, "Chyros"). Native memory will only get better.
|
||||||
|
|
||||||
|
### Level 2 — Forcing reliable recall (Connelly hook / Huryn structure)
|
||||||
|
|
||||||
|
- Paste a memory-management prompt + rules into CLAUDE.md. Structured memory rooted at
|
||||||
|
`~/.claude/memory/`: `memory.md` (index), `general.md` (cross-project facts/prefs/env),
|
||||||
|
`domain/<topic>.md` (one file per topic), `tools/<tool>.md` (one file per tool, e.g.
|
||||||
|
`slack.md` with config/workarounds/edge cases).
|
||||||
|
- A **session-start hook** auto-injects the **index** (not full content) into every session and
|
||||||
|
sub-agent — not relying on Claude choosing to read it.
|
||||||
|
- A **"reorganize memory"** command periodically reads all memory files, removes
|
||||||
|
duplicates/outdated entries, merges related ones, splits overloaded files, sorts by date,
|
||||||
|
and rebuilds the index.
|
||||||
|
- Huryn's updated post (productcompass.pm) adds **active hypothesis tracking**, a **catalog of
|
||||||
|
"false beliefs,"** and an **AI-proposes-reorganization / human-keeps-editorial-control**
|
||||||
|
loop. The transferable mechanic (not the content-writing domain) is the hypothesis/
|
||||||
|
false-belief log + propose-and-approve loop — useful for per-client "what we tried / decided
|
||||||
|
/ what didn't work."
|
||||||
|
- **Author's take: most people should stop here.**
|
||||||
|
|
||||||
|
### Level 3 — Search by meaning, not keywords (memsearch; OpenClaw template)
|
||||||
|
|
||||||
|
- Add only if you've used Claude Code > 1 month, have many memory files, and have asked
|
||||||
|
something you *know* is in your notes that Claude couldn't find.
|
||||||
|
- **OpenClaw memory design (3 layers)**: (1) `memory.md` = long-term durable facts, loaded at
|
||||||
|
session start; (2) **daily note files** = one per date, a running log, today+yesterday
|
||||||
|
auto-loaded, older left on disk; (3) optional **"dreaming"** = background process that scores
|
||||||
|
daily notes and promotes recurring ones into long-term memory, forgetting stale stuff.
|
||||||
|
- **memsearch** (by Zilliz): ports this into Claude Code as a two-line plugin. Markdown-first,
|
||||||
|
same chunking/structure. Chunks into semantic vectors; a **user-prompt-submit hook
|
||||||
|
auto-injects the top-3 semantic matches** into every prompt. Plain-readable markdown.
|
||||||
|
- Alternative **claude-mem**: MCP-based, 3-tier storage (summaries/timeline/observations),
|
||||||
|
dashboard/team/cost features. Author's view: overkill, and MCP-based means Claude must
|
||||||
|
actively call the search tool; stores opaque blobs vs memsearch's readable markdown.
|
||||||
|
|
||||||
|
### Level 4 — Verbatim conversation recall (MemPalace)
|
||||||
|
|
||||||
|
- Local RAG, free, claims highest published benchmark. Stores words **verbatim** (nothing
|
||||||
|
summarized) indexed in a dense symbolic dialect ("memory palace": wings → rooms → closets →
|
||||||
|
drawers). Two DBs: SQL (entities/relationships) + Chroma (vector chunks). Background hooks on
|
||||||
|
session-end/pre-compaction. ~42ms retrieval.
|
||||||
|
- **Downsides**: storage is **not readable markdown**; drawers are **isolated** (knowledge not
|
||||||
|
interconnected); local-only.
|
||||||
|
|
||||||
|
### Level 5 — Self-organizing knowledge base (Karpathy LLM wiki; Recall; LightRAG)
|
||||||
|
|
||||||
|
- Karpathy's **LLM wiki**: `raw/` (you drop sources, Claude reads, never writes) + `wiki/`
|
||||||
|
(Claude owns entirely). Plain markdown, Obsidian graph. **Recall** is a hosted done-for-you
|
||||||
|
version (you don't own the data). **LightRAG** is enterprise knowledge-graph overkill.
|
||||||
|
- **Author's take**: these are for **content knowledge bases / deep research**, NOT operational
|
||||||
|
memory ("what did we decide about client X's landing page in March"). Skip for our use case.
|
||||||
|
|
||||||
|
### Level 6 — One brain for all AI tools (OpenBrain; Mem0)
|
||||||
|
|
||||||
|
- **OpenBrain**: memory in a **Postgres DB you own** (Supabase), one `thoughts` table (text +
|
||||||
|
embedding + tags + timestamp), MCP server fronting it so any AI tool shares the same memory.
|
||||||
|
Most portable/future-proof. **Downsides**: longer/harder setup; **always remote → latency on
|
||||||
|
every query**; small monthly cost. **Mem0**: cross-tool layer, fast setup, but data lives on
|
||||||
|
their servers permanently.
|
||||||
|
- **Author's take**: only if you live across many AI tools and want real-time shared memory.
|
||||||
|
|
||||||
|
## Author's recommendation
|
||||||
|
|
||||||
|
- Just starting → Level 1 done right.
|
||||||
|
- A bit in → Level 2 (Connelly hook). **Most should stop here.**
|
||||||
|
- Lots of context, losing old decisions → Level 3 (memsearch) or Level 4 (MemPalace).
|
||||||
|
- Levels 5–6 are a different realm for specific use cases.
|
||||||
|
- **Levels 1+2+3 stack** (similar folder structures). The author personally runs up to Level 3:
|
||||||
|
OpenClaw conventions + semantic search + injection hooks.
|
||||||
|
|
||||||
|
## What we took / left (see ADRs for why)
|
||||||
|
|
||||||
|
- **Took**: progressive-disclosure index (L1/L2), per-tool/per-domain granularity + reorganize
|
||||||
|
command + session hooks (L2), Huryn's propose-and-approve reorg loop, OpenClaw daily-notes +
|
||||||
|
dreaming and memsearch (L3).
|
||||||
|
- **Replaced**: L2's **folders** with **namespaced tags** in a flat vault.
|
||||||
|
- **Left**: MemPalace (opaque, isolated), Recall/LightRAG (content KB, not operational),
|
||||||
|
OpenBrain/Mem0 (always-remote, fights local-fast).
|
||||||
|
|
||||||
|
## Reference links
|
||||||
|
|
||||||
|
- John & Pawel's system — youngleaders.tech/p/how-i-finally-sorted-my-claude-code-memory
|
||||||
|
- Pawel Huryn (updated) — productcompass.pm/p/self-improving-claude-system
|
||||||
|
- memsearch — github.com/zilliztech/memsearch
|
||||||
|
- MemPalace — github.com/MemPalace/mempalace
|
||||||
|
- Karpathy LLM wiki — gist.github.com/karpathy
|
||||||
|
- Recall — recall.it · Mem0 — mem0.ai · OpenBrain — github.com/NateBJones-Project
|
||||||
|
- QMD (our deferred semantic layer) — github.com/tobi/qmd
|
||||||
|
|
@ -0,0 +1,173 @@
|
||||||
|
# System Design
|
||||||
|
|
||||||
|
_Status: approved 2026-06-03. Implementation not yet started._
|
||||||
|
|
||||||
|
## Goals (what this system must do)
|
||||||
|
|
||||||
|
1. **Thin projects** — keep as little AI context inside each project repo as possible. Projects
|
||||||
|
focus on project files; knowledge is pulled in on demand or injected by hooks.
|
||||||
|
2. **Cross-project / cross-client knowledge** — the AI learns something once (e.g. the SEMrush
|
||||||
|
API) and references it from anywhere. Two scopes: **global** (broadly useful) and
|
||||||
|
**project/client-specific** (how a given client uses a tool) — both globally reachable.
|
||||||
|
Ask anything client- or project-related from any project.
|
||||||
|
3. **Timeline awareness** — from any project, lightweight awareness of recent activity ("what
|
||||||
|
was I doing an hour ago / yesterday"), with the ability to drill deeper.
|
||||||
|
4. **Remote, local-fast** — accessible anywhere (VPS / personal OS) but runs local-fast; lazy
|
||||||
|
sync (minutes/hourly) is fine; real-time is overkill.
|
||||||
|
|
||||||
|
Desired properties: **lightweight** (low tokens), **fast** (out of the way), **flexible**
|
||||||
|
(cross project/client), **self-evolving** (AI maintains it under clear rules), **easy to
|
||||||
|
manage** (AI-managed), **semi-structured** (organization that can evolve).
|
||||||
|
|
||||||
|
## Core principle: two memory types, kept separate
|
||||||
|
|
||||||
|
| Type | Question | Lifecycle | Write path | Our tool |
|
||||||
|
|------|----------|-----------|-----------|----------|
|
||||||
|
| **Episodic** | "What happened, when?" | accretes & decays | auto-captured | **memsearch** |
|
||||||
|
| **Semantic / knowledge** | "How do we…?" | deliberately maintained | curated by you/AI | **Obsidian vault + tag index** |
|
||||||
|
|
||||||
|
This is the classic **episodic vs. semantic** memory split. Keeping them separate is the key
|
||||||
|
architectural decision — they have different lifecycles, write paths, and query patterns, and
|
||||||
|
forcing one tool to do both is what made every earlier design feel forced.
|
||||||
|
|
||||||
|
## The three layers
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────────────┐
|
||||||
|
│ EPISODIC ── memsearch (Milvus Lite, embedded, no Docker) │
|
||||||
|
│ auto-captured session/journal notes · NL semantic recall │
|
||||||
|
│ answers "when did we…", "what was I doing yesterday" │
|
||||||
|
├─────────────────────────────────────────────────────────────────┤
|
||||||
|
│ KNOWLEDGE ── flat Obsidian vault (single source of truth) │
|
||||||
|
│ + Ruby/Sequel/SQLite tag index (CLI) │
|
||||||
|
│ structured tag filtering · answers "how do we…", "what do we │
|
||||||
|
│ know about X for client Y" │
|
||||||
|
├─────────────────────────────────────────────────────────────────┤
|
||||||
|
│ SEMANTIC-OVER-KNOWLEDGE ── QMD (SQLite+vector, MCP) [DEFERRED] │
|
||||||
|
│ meaning-based recall over the vault when tags miss │
|
||||||
|
│ add ONLY when structured tagging proves insufficient │
|
||||||
|
└─────────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
All three are **local-first, markdown-as-truth, no Docker, no server, no API keys.**
|
||||||
|
|
||||||
|
## Layer 1 — Episodic (memsearch)
|
||||||
|
|
||||||
|
- **What it is**: a Claude Code plugin (by Zilliz) that auto-captures session notes as daily
|
||||||
|
markdown, chunks them, and stores a **shadow index** in **Milvus Lite** (a single embedded
|
||||||
|
file — no server, no Docker). Hybrid search = BM25 + dense vectors + RRF, local ONNX
|
||||||
|
embeddings (`bge-m3`, no API key/cost). A FileWatcher (1500ms debounce) handles updates and
|
||||||
|
deletions; markdown stays the source of truth.
|
||||||
|
- **Why off-the-shelf**: it already implements the OpenClaw daily-notes + "dreaming" pattern
|
||||||
|
and the markdown-as-truth / disposable-shadow-index philosophy we'd otherwise hand-build.
|
||||||
|
- **Role in our system**: satisfies Goal 3 (timeline). The AI queries it in natural language
|
||||||
|
("what was decided about X last week"). We do **not** make it filter by our tags — it owns
|
||||||
|
the episodic corpus only.
|
||||||
|
|
||||||
|
## Layer 2 — Knowledge (vault + tag index)
|
||||||
|
|
||||||
|
The heart of the system, and the part we build.
|
||||||
|
|
||||||
|
### Vault
|
||||||
|
- **Flat markdown directory**, single source of truth, **configurable location** (NOT forced
|
||||||
|
into `~/.claude/`; symlink if a tool insists). Browsable in Obsidian as a viewer.
|
||||||
|
- **Replaces project-local documentation**: instead of docs scattered per repo, knowledge
|
||||||
|
lives once in the vault and is pulled into any project on demand.
|
||||||
|
|
||||||
|
### Frontmatter contract (every note)
|
||||||
|
```yaml
|
||||||
|
---
|
||||||
|
summary: One line, written at creation. The router shows this so the AI can pick a
|
||||||
|
file without opening it.
|
||||||
|
tags:
|
||||||
|
- tool/semrush # namespaced, nested (slash = Obsidian nested tag)
|
||||||
|
- client/sesame3g
|
||||||
|
- domain/seo
|
||||||
|
- scope/project # or scope/global
|
||||||
|
---
|
||||||
|
```
|
||||||
|
- **Namespaces** are the "virtual indexes": `tool/`, `client/`, `domain/`, `convention/`,
|
||||||
|
`scope/`. `#tool` matches all children — native prefix filtering, no folders needed.
|
||||||
|
- **Two knowledge scopes** via `scope/global` vs `scope/project` (+ a `client/` tag): global =
|
||||||
|
broadly useful tool/domain knowledge; project = how a specific client uses it. Both are
|
||||||
|
globally queryable; the scope tag is the shortcut that avoids scanning every client's usage.
|
||||||
|
|
||||||
|
### Index (the tag cache)
|
||||||
|
A small **Ruby program, Sequel ORM, SQLite** — the disposable structured cache over the vault.
|
||||||
|
|
||||||
|
- **Schema**:
|
||||||
|
- `files(id, path, mtime, summary, scope)` — one row per note.
|
||||||
|
- `tags(id, name)` — one row per distinct tag (enables enumerating the vocabulary:
|
||||||
|
"what clients/tools do I have notes on?").
|
||||||
|
- `files_tags(file_id, tag_id)` — `many_to_many` join.
|
||||||
|
- **CLI** (the only interface; the AI never touches SQLite directly):
|
||||||
|
- `index update --since <mtime>` — incremental: pulls `.md` files with `mtime >` last cache
|
||||||
|
time, re-reads their frontmatter, upserts; **also reconciles** (prunes rows whose path no
|
||||||
|
longer exists).
|
||||||
|
- `index update --rebuild` — full rebuild from scratch (default `false`).
|
||||||
|
- `index query --client X --tool Y [--scope global] [--domain Z]` — returns **path +
|
||||||
|
summary + matched tags** for each hit (decided: option C). Tags shown so the AI sees *why*
|
||||||
|
a file matched.
|
||||||
|
- `index tags --namespace tool/` — enumerate a virtual index.
|
||||||
|
- **Source of truth rule**: markdown is authoritative; the SQLite file is a rebuildable cache
|
||||||
|
that is **never synced** and can be deleted/rebuilt anytime.
|
||||||
|
|
||||||
|
### Freshness (lazy — chosen Option A)
|
||||||
|
- **AI writes** → a `PostToolUse` hook on `Write`/`Edit` targeting vault `.md` files calls
|
||||||
|
`index update --file <path>` (updates exactly that file, prunes if deleted). Event-driven,
|
||||||
|
no polling, no staleness for AI edits.
|
||||||
|
- **Manual edits** (rare) → caught by a **session-start reconcile** (`index update --since` +
|
||||||
|
prune). **No daemon, no cron.**
|
||||||
|
|
||||||
|
### Retrieval (hook-injected + on-demand)
|
||||||
|
- **Session-start hook** injects: (a) a compact index/overview, (b) the current project's
|
||||||
|
declared `convention/*` tags **resolved to files** (so coding conventions auto-pull and a
|
||||||
|
convention edit propagates to every project using that tag), (c) a pointer to recent
|
||||||
|
episodic journal.
|
||||||
|
- **On demand**: the AI runs `index query` to pull specific knowledge into context only when
|
||||||
|
the task needs it. Projects stay thin — their CLAUDE.md holds **tags/pointers**, not content.
|
||||||
|
|
||||||
|
## Layer 3 — Semantic over knowledge (QMD) — DEFERRED
|
||||||
|
|
||||||
|
- **What**: QMD (github.com/tobi/qmd) — local markdown search, **SQLite + FTS5/BM25 + local
|
||||||
|
vector embeddings (EmbeddingGemma-300M GGUF) + LLM rerank**. CLI + optional **MCP server**.
|
||||||
|
No Docker, no API keys. Proves SQLite + a local vector model is enough — no Milvus/Postgres
|
||||||
|
for knowledge.
|
||||||
|
- **Why deferred**: start structured-only. Add QMD as a **set-and-forget** semantic layer over
|
||||||
|
the vault **only when** we catch ourselves failing to retrieve notes we know exist (the
|
||||||
|
video's "only level up when it bites"). It complements, not replaces, the tag index (QMD
|
||||||
|
filters by path/collection context, not first-class frontmatter tags).
|
||||||
|
|
||||||
|
## Timeline (Goal 3) details
|
||||||
|
- A **session-end hook** appends a daily journal note (one file per date) with pointers to the
|
||||||
|
project/knowledge files touched. memsearch indexes these; today+yesterday are cheap to load,
|
||||||
|
older entries are reachable by query for drill-down.
|
||||||
|
|
||||||
|
## Self-evolution guardrails
|
||||||
|
- The AI **writes only to the vault**, never silently into project repos.
|
||||||
|
- **Required frontmatter schema** (summary + namespaced tags) is enforced so the index stays
|
||||||
|
queryable.
|
||||||
|
- **Daily notes are append-only**; consolidation/reorg is a **separate, reviewable step run in
|
||||||
|
plan mode** (Connelly's reorganize + Huryn's propose-and-approve loop).
|
||||||
|
- **Promotion to `scope/global`** requires a rule (e.g. a fact recurring N times) — not every
|
||||||
|
stray note gets promoted.
|
||||||
|
|
||||||
|
## Sync (Goal 4)
|
||||||
|
- The **vault** syncs to the VPS via **git** (versioned history, hourly) or **Syncthing**
|
||||||
|
(continuous, zero-thought). Decision deferred to build time.
|
||||||
|
- **Indexes are never synced** — Milvus Lite and (later) QMD shadow indexes are rebuilt per
|
||||||
|
machine. Sync only the markdown.
|
||||||
|
|
||||||
|
## Packaging
|
||||||
|
- The whole thing ships as a **global Claude Code plugin with skills** (hooks + scripts +
|
||||||
|
CRUD know-how) so every project, on every machine, knows how to use the vault effectively.
|
||||||
|
See [04-build-plan.md](04-build-plan.md).
|
||||||
|
|
||||||
|
## How each goal is met
|
||||||
|
|
||||||
|
| Goal | Met by |
|
||||||
|
|------|--------|
|
||||||
|
| 1. Thin projects | Knowledge in the vault, not repos; CLAUDE.md holds tags/pointers; on-demand `index query` |
|
||||||
|
| 2. Cross-project/client knowledge, global vs project scopes | Flat vault + namespaced tags + `scope/` + `client/`; enumerable virtual indexes |
|
||||||
|
| 3. Timeline | memsearch episodic layer + session-end journal hook |
|
||||||
|
| 4. Remote, local-fast | Markdown vault synced via git/Syncthing; disposable per-machine indexes |
|
||||||
|
|
@ -0,0 +1,149 @@
|
||||||
|
# Architecture Decision Records
|
||||||
|
|
||||||
|
A running log of decisions and *why*. Format per entry: Context · Decision · Rationale ·
|
||||||
|
Alternatives rejected · Status. Newest decisions extend the log; supersede rather than delete.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ADR-001 — Two memory types, kept as separate systems
|
||||||
|
|
||||||
|
- **Context**: Earlier attempts to make one tool serve both "what happened" and "how do we do
|
||||||
|
X" felt forced (e.g. trying to make memsearch filter knowledge by tags).
|
||||||
|
- **Decision**: Model **episodic** memory and **semantic/knowledge** memory as two separate
|
||||||
|
systems with different tools.
|
||||||
|
- **Rationale**: They have different lifecycles (episodic accretes and decays; knowledge is
|
||||||
|
deliberately maintained), different write paths (auto-captured vs curated with guardrails),
|
||||||
|
and different query patterns ("when did we…" vs "how do we…"). Separation dissolves the
|
||||||
|
earlier integration tension entirely.
|
||||||
|
- **Alternatives rejected**: One unified store (memsearch-for-everything, or OpenBrain's single
|
||||||
|
`thoughts` table) — conflates the two and forces awkward filtering.
|
||||||
|
- **Status**: Accepted.
|
||||||
|
|
||||||
|
## ADR-002 — memsearch for the episodic layer
|
||||||
|
|
||||||
|
- **Context**: Need timeline/"what happened" memory (Goal 3) that's NL-queryable and lazy.
|
||||||
|
- **Decision**: Adopt **memsearch** (Zilliz) off-the-shelf for episodic memory.
|
||||||
|
- **Rationale**: It already implements the OpenClaw daily-notes + "dreaming" pattern and the
|
||||||
|
markdown-as-truth / disposable-shadow-index philosophy we'd otherwise build. Embedded
|
||||||
|
**Milvus Lite** (single file), hybrid BM25+vector+RRF search, local ONNX embeddings (no API
|
||||||
|
key/cost), a FileWatcher that handles deletions — **no Docker, no server**. Two-line install.
|
||||||
|
- **Alternatives rejected**: claude-mem (MCP-based — Claude must actively call search; opaque
|
||||||
|
blobs vs readable markdown; overkill features). Hand-building daily notes + dreaming
|
||||||
|
ourselves (reinventing a solved tool).
|
||||||
|
- **Status**: Accepted.
|
||||||
|
|
||||||
|
## ADR-003 — Flat vault with namespaced tags, not folders
|
||||||
|
|
||||||
|
- **Context**: Connelly/Huryn organize by folders (`tools/`, `domain/`). User wants a flat
|
||||||
|
Obsidian vault with tags as virtual indexes, and cross-cutting filters (client × tool ×
|
||||||
|
convention).
|
||||||
|
- **Decision**: One **flat markdown vault**; organize via **namespaced, nested tags**
|
||||||
|
(`tool/`, `client/`, `domain/`, `convention/`, `scope/`). Slashes are valid Obsidian nested
|
||||||
|
tags, so `#tool` matches all children.
|
||||||
|
- **Rationale**: A note can carry several namespaces at once (`tool/semrush` +
|
||||||
|
`client/sesame3g` + `convention/react-ts`) — folders can't express that. Enables "filter by
|
||||||
|
client+tool to narrow the index." Enumerable virtual indexes ("what clients/tools exist").
|
||||||
|
- **Alternatives rejected**: Folder hierarchy (single-axis; can't do cross-cutting filters).
|
||||||
|
Pure-prefix path filtering via memsearch `source_prefix` (would force directories back in).
|
||||||
|
- **Trade-off accepted**: Tags give the *human/Obsidian* free filtering, but the *AI* gets
|
||||||
|
nothing for free from tags — we must materialize them into a queryable index (see ADR-004).
|
||||||
|
- **Status**: Accepted.
|
||||||
|
|
||||||
|
## ADR-004 — SQLite + Sequel (Ruby) tag index as the knowledge-layer cache
|
||||||
|
|
||||||
|
- **Context**: The AI can't use Obsidian tags directly; tag filtering needs a machine-queryable
|
||||||
|
index. A previous `~/Documents/SecondBrain/` tag database was lost track of.
|
||||||
|
- **Decision**: A small **Ruby program using the Sequel ORM over SQLite**, exposed as a **CLI**.
|
||||||
|
Schema: `files(path, mtime, summary, scope)`, `tags(name)`, `files_tags` join
|
||||||
|
(`many_to_many`). The summary is a **column on `files`** (an attribute), not a join.
|
||||||
|
- **Rationale**: Normalized `tags` table makes enumerating the vocabulary a first-class cheap
|
||||||
|
query (the "virtual index" goal). The `summary` column is what turns the index from a
|
||||||
|
*finder* into a *router* — the AI sees enough to pick a file without opening it (progressive
|
||||||
|
disclosure, low tokens). Ruby + Sequel + CLI keeps the contract clean and the DB swappable;
|
||||||
|
the AI never touches SQLite directly.
|
||||||
|
- **Failure-mode guard (the lost-SecondBrain lesson)**: **markdown is always authoritative; the
|
||||||
|
SQLite file is a disposable cache** that is never synced and can be rebuilt from frontmatter
|
||||||
|
anytime (`index update --rebuild`).
|
||||||
|
- **Alternatives rejected**: Plain-markdown generated `INDEX.md` (must regenerate; grep-at-scale
|
||||||
|
is token-heavy). Frontmatter grep on demand (scales badly). Milvus/Postgres for knowledge
|
||||||
|
(overkill; QMD/memsearch prove SQLite is enough — see ADR-006/008).
|
||||||
|
- **Query output**: returns **path + summary + matched tags** (option C) — tags are cheap and
|
||||||
|
show *why* a result matched, useful for cross-client queries.
|
||||||
|
- **Status**: Accepted.
|
||||||
|
|
||||||
|
## ADR-005 — Structured-first; semantic search over the vault deferred
|
||||||
|
|
||||||
|
- **Context**: Tag filtering ("client/sesame3g + tool/semrush") may miss notes whose wording
|
||||||
|
doesn't match the query ("how do we use semrush" vs a note titled "search analytics
|
||||||
|
integration").
|
||||||
|
- **Decision**: Ship the knowledge layer **structured-only** (tags + summaries). **Defer**
|
||||||
|
meaning-based search over the vault until it demonstrably bites.
|
||||||
|
- **Rationale**: Structured tagging is the lightweight/fast thing the user wants, and the
|
||||||
|
summary+tag design is built to make it work. Follow the video's "only level up when it bites."
|
||||||
|
- **Status**: Accepted (semantic deferred).
|
||||||
|
|
||||||
|
## ADR-006 — QMD as the (deferred) semantic-over-knowledge layer
|
||||||
|
|
||||||
|
- **Context**: When ADR-005's structured-only proves insufficient, we want a set-and-forget
|
||||||
|
semantic layer over the vault, local and Docker-free.
|
||||||
|
- **Decision**: Earmark **QMD** (github.com/tobi/qmd) for that role; do **not** install yet.
|
||||||
|
- **Rationale**: Local markdown search using **SQLite + FTS5/BM25 + local vector embeddings
|
||||||
|
(EmbeddingGemma-300M GGUF) + LLM rerank**; CLI + optional **MCP server**; no Docker, no API
|
||||||
|
keys. Validates that SQLite + a local vector model suffices (no Milvus/Postgres for
|
||||||
|
knowledge). Complements the tag index (QMD filters by path/collection context, not
|
||||||
|
first-class frontmatter tags), so it adds semantic recall without replacing structured
|
||||||
|
filtering.
|
||||||
|
- **Alternatives rejected**: Pointing memsearch at the vault (mixes episodic and knowledge
|
||||||
|
corpora; its filtering is path-prefix not tags). A bespoke embedding index (reinvents QMD).
|
||||||
|
- **Status**: Deferred / earmarked.
|
||||||
|
|
||||||
|
## ADR-007 — Lazy freshness: write-hook + session-start reconcile, no daemon/cron
|
||||||
|
|
||||||
|
- **Context**: The cache must reflect new/edited/deleted/renamed notes without becoming a
|
||||||
|
resource hog or going stale on renames.
|
||||||
|
- **Decision**: **Option A (lazy).** A `PostToolUse` hook updates the index on **AI** writes
|
||||||
|
(single-file, prunes on delete). **Manual** edits are caught by a **session-start reconcile**
|
||||||
|
(`index update --since` + prune of vanished paths). **No daemon, no cron.**
|
||||||
|
- **Rationale**: The AI is the primary writer, so write-time hooks give event-driven freshness
|
||||||
|
with no polling. The user rarely edits the vault by hand, so a session-start reconcile is
|
||||||
|
enough; a continuous `inotify` daemon (the `listen` gem) would add an always-on process to
|
||||||
|
manage/sync for negligible benefit. Matches the user's "lazy sync is fine" stance.
|
||||||
|
- **Alternatives rejected**: `inotify`/`listen` daemon (live freshness, but always-on process
|
||||||
|
to manage — unnecessary). Cron reconcile ("seems silly" per user; session-start covers it).
|
||||||
|
- **Status**: Accepted.
|
||||||
|
|
||||||
|
## ADR-008 — Markdown-as-truth; sync the vault, not the indexes
|
||||||
|
|
||||||
|
- **Context**: Must be accessible on a VPS / multiple machines but run local-fast (Goal 4).
|
||||||
|
- **Decision**: Sync the **markdown vault** to the VPS via **git or Syncthing** (choice deferred
|
||||||
|
to build time). **Indexes (Milvus Lite, future QMD) are rebuilt per machine and never
|
||||||
|
synced.**
|
||||||
|
- **Rationale**: Markdown is plain text — git/Syncthing sync it trivially; lazy (hourly or
|
||||||
|
continuous-async) is enough. Indexes are disposable caches; syncing binary DBs invites
|
||||||
|
conflicts for no gain. Local reads stay fast; ownership and portability stay with the user.
|
||||||
|
- **Alternatives rejected**: **OpenBrain / Mem0** hosted DBs — always-remote, adds per-query
|
||||||
|
latency and monthly cost, conflicts with local-fast; ownership weaker (Mem0 especially).
|
||||||
|
Only worth it for real-time cross-tool memory, which the user called overkill.
|
||||||
|
- **Status**: Accepted.
|
||||||
|
|
||||||
|
## ADR-009 — Package as a global Claude Code plugin with skills
|
||||||
|
|
||||||
|
- **Context**: Every project, on every machine, should know how to use the vault — write
|
||||||
|
conventions, query patterns, the hooks, and the CLI — without per-project setup.
|
||||||
|
- **Decision**: Ship hooks + scripts + CRUD know-how as a **global Claude Code plugin with
|
||||||
|
skills**, installed at the user level.
|
||||||
|
- **Rationale**: Skills carry the "when to write / what conventions / how & when to query"
|
||||||
|
guidance to the model; the plugin registers the session-start / session-end / PostToolUse
|
||||||
|
hooks and bundles the Ruby CLI. Global install = consistent behavior everywhere; single
|
||||||
|
source of truth for the conventions themselves.
|
||||||
|
- **Status**: Accepted (to be built — see 04-build-plan.md).
|
||||||
|
|
||||||
|
## Rejected tools (summary)
|
||||||
|
|
||||||
|
| Tool | Why rejected for our use |
|
||||||
|
|------|--------------------------|
|
||||||
|
| MemPalace (L4) | Storage not readable markdown; isolated drawers (knowledge not interconnected); fights self-managing + cross-linking goals |
|
||||||
|
| Recall / LightRAG (L5) | Content knowledge bases / deep research, not operational memory; Recall = hosted, you don't own data; LightRAG = enterprise overkill |
|
||||||
|
| OpenBrain / Mem0 (L6) | Always-remote DB → latency + cost; conflicts with local-fast lazy-sync; only pays off for real-time cross-tool memory (user: overkill) |
|
||||||
|
| Postgres / Milvus server | Unnecessary — SQLite (tag index) + Milvus Lite (memsearch) + QMD's SQLite cover everything locally with no Docker |
|
||||||
|
| claude-mem | MCP-based (Claude must call search); opaque blobs vs readable markdown; feature overkill |
|
||||||
|
|
@ -0,0 +1,155 @@
|
||||||
|
# Build Plan
|
||||||
|
|
||||||
|
How a human builds this system, step by step, and answers to the operational questions:
|
||||||
|
which scripts and hooks, how the AI knows when to write and what conventions to follow, how and
|
||||||
|
when it queries, the CRUD hooks, and how it's packaged as a global Claude Code plugin with
|
||||||
|
skills.
|
||||||
|
|
||||||
|
> This is a build outline, not the implementation plan. The next session should turn this into
|
||||||
|
> a proper implementation plan (writing-plans skill) and execute it.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Part A — Build order (human builder's path)
|
||||||
|
|
||||||
|
Build bottom-up: the vault and CLI first (usable standalone), then the hooks, then the plugin
|
||||||
|
that packages it.
|
||||||
|
|
||||||
|
### Step 1 — Vault skeleton & conventions
|
||||||
|
- Decide the vault location (default: a synced home dir, e.g. `~/brain`; symlink into
|
||||||
|
`~/.claude/memory` only if a tool insists). **Vault is the single source of truth.**
|
||||||
|
- Write `CONVENTIONS.md` in the vault: the frontmatter contract and the tag namespaces
|
||||||
|
(`tool/`, `client/`, `domain/`, `convention/`, `scope/`). This doubles as the spec the
|
||||||
|
skills teach the AI.
|
||||||
|
- Seed a few real notes (e.g. `tool/semrush` global + a `client/<x>` project note) to test
|
||||||
|
against.
|
||||||
|
|
||||||
|
### Step 2 — The Ruby tag-index CLI (the core build)
|
||||||
|
- Ruby project, `sequel` + `sqlite3` gems. SQLite file in a cache dir (e.g.
|
||||||
|
`~/.cache/memory-index/index.sqlite`) — disposable, never synced.
|
||||||
|
- **Schema / migrations**: `files(id, path, mtime, summary, scope)`, `tags(id, name)`,
|
||||||
|
`files_tags(file_id, tag_id)`; `many_to_many` between files and tags.
|
||||||
|
- **Frontmatter parser**: read YAML frontmatter from a `.md` file → `{summary, scope, tags[]}`.
|
||||||
|
Fail loudly (or quarantine) notes missing the required `summary`/tags so the contract is
|
||||||
|
enforced.
|
||||||
|
- **Commands** (a thin CLI dispatch — `thor`/`optparse` or plain):
|
||||||
|
- `index update --since <iso8601|auto>` — find `.md` with `mtime >` last-cache-time, upsert
|
||||||
|
them; **reconcile**: delete rows for paths no longer on disk.
|
||||||
|
- `index update --file <path>` — upsert a single file (used by the write hook); prune if the
|
||||||
|
path is gone.
|
||||||
|
- `index update --rebuild` (default off) — drop & rebuild from a full vault scan.
|
||||||
|
- `index query --client X --tool Y [--domain Z] [--scope global]` — AND the filters; output
|
||||||
|
JSON lines of `{path, summary, tags[]}` (option C).
|
||||||
|
- `index tags [--namespace tool/]` — enumerate the vocabulary (virtual index).
|
||||||
|
- Store `last_cache_time` (a tiny meta table or a stamp file) so `--since auto` works.
|
||||||
|
- **Tests**: create/edit/delete/rename a note, assert the index reflects it; rebuild equals
|
||||||
|
incremental.
|
||||||
|
|
||||||
|
### Step 3 — Hooks (maintenance + retrieval)
|
||||||
|
See Part C for exact mapping. Implement as small shell wrappers that call the Ruby CLI:
|
||||||
|
- `pre-tool-memory` style **PostToolUse** updater (write path → `index update --file`).
|
||||||
|
- **SessionStart** reconcile + inject (index overview + resolved `convention/*` files + journal
|
||||||
|
pointer).
|
||||||
|
- **SessionEnd** journal appender (daily note with pointers).
|
||||||
|
|
||||||
|
### Step 4 — Episodic layer (memsearch)
|
||||||
|
- Install memsearch (`/plugin marketplace add zilliztech/memsearch`, then `plugin install
|
||||||
|
memsearch`), local to start. Verify daily memory files appear after a few conversations.
|
||||||
|
- Decide whether memsearch indexes our session-end journal notes or its own capture (likely its
|
||||||
|
own; our journal can point into the knowledge vault).
|
||||||
|
|
||||||
|
### Step 5 — Sync
|
||||||
|
- Pick **git** (versioned, hourly push/pull) or **Syncthing** (continuous, zero-thought) for the
|
||||||
|
vault → VPS. Configure on each machine. **Do not sync the SQLite/Milvus caches.**
|
||||||
|
|
||||||
|
### Step 6 — Package as a global plugin (Part D)
|
||||||
|
- Wrap Steps 2–3 into a Claude Code plugin with skills; install at user level.
|
||||||
|
|
||||||
|
### Step 7 (deferred) — QMD semantic layer
|
||||||
|
- Only when structured-only retrieval misses notes you know exist: `qmd` over the vault,
|
||||||
|
optionally as an MCP server the AI can query.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Part B — The AI's write & query conventions (skills teach these)
|
||||||
|
|
||||||
|
### When the AI WRITES to the vault
|
||||||
|
Trigger writes when the AI learns something **evergreen and reusable across projects** — not
|
||||||
|
project-ephemeral state (that's the episodic layer / project files). Concretely:
|
||||||
|
- It worked out how a tool/API behaves (e.g. SEMrush auth, rate limits, an endpoint quirk).
|
||||||
|
- It established a convention, decision, or preference that should apply beyond this task.
|
||||||
|
- It discovered a client-specific fact worth reusing (how *this* client uses a tool).
|
||||||
|
|
||||||
|
**What conventions to follow when writing** (enforced by the frontmatter contract):
|
||||||
|
- One concept per note; keep notes small (the L1 "under ~200 lines / one topic" discipline).
|
||||||
|
- **Required frontmatter**: a one-line `summary` (written now, not deferred), `scope/global`
|
||||||
|
or `scope/project`, and namespaced tags (`tool/…`, `client/…`, `domain/…`, optionally
|
||||||
|
`convention/…`).
|
||||||
|
- **Scope rule**: default new tool/domain knowledge to `scope/global`; mark `scope/project` +
|
||||||
|
a `client/<name>` tag when it's specific to how a client uses something.
|
||||||
|
- **Write only to the vault**, never silently into a project repo.
|
||||||
|
- **Promotion to global** (project → global) and **consolidation** are *not* done inline —
|
||||||
|
they happen in the reorganize step (Part C), in plan mode, for human review.
|
||||||
|
|
||||||
|
### When & how the AI QUERIES
|
||||||
|
- **At session start**: the injected index overview + resolved `convention/*` files tell it
|
||||||
|
what's available without reading everything.
|
||||||
|
- **On demand, during a task**: when the task touches a tool/client/domain, run
|
||||||
|
`index query --tool semrush --client sesame3g` → get `{path, summary, tags}` → open only the
|
||||||
|
files whose summary matches. This is the progressive-disclosure loop that keeps tokens low.
|
||||||
|
- **Cross-client lookups**: query by `--tool` alone (omit client) to surface what was learned
|
||||||
|
with *any* client; tags in the output show provenance.
|
||||||
|
- **"What happened" questions** go to the **episodic layer (memsearch)** in natural language,
|
||||||
|
not the tag index.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Part C — Hooks & CRUD mapping
|
||||||
|
|
||||||
|
CRUD over the knowledge vault, and which hook/command services each operation:
|
||||||
|
|
||||||
|
| Operation | Trigger | Mechanism |
|
||||||
|
|-----------|---------|-----------|
|
||||||
|
| **Create** | AI writes a new note | AI writes `.md` (Write tool) → **PostToolUse** hook → `index update --file` |
|
||||||
|
| **Read / query** | AI needs knowledge | AI calls `index query …` (no hook; on-demand CLI) |
|
||||||
|
| **Update** | AI/user edits a note | AI edit → **PostToolUse** hook → `index update --file`; user edit → **SessionStart** reconcile |
|
||||||
|
| **Delete / rename** | note removed/renamed | covered by `--file` (prune) on AI ops; by **SessionStart** reconcile (prune vanished paths) on manual ops |
|
||||||
|
| **Inject** | new session/sub-agent starts | **SessionStart** hook injects index overview + resolved `convention/*` + journal pointer |
|
||||||
|
| **Journal** | session ends | **SessionEnd** hook appends a dated journal note with pointers |
|
||||||
|
| **Reorganize** | periodic, user-invoked | `reorganize memory` run in **plan mode**: dedupe, merge, split, re-tag, promote `scope/global`, rebuild index — human approves |
|
||||||
|
|
||||||
|
Hooks are thin shell wrappers over the Ruby CLI so the logic lives in one place.
|
||||||
|
|
||||||
|
**Hooks summary:**
|
||||||
|
- **SessionStart** — reconcile (`index update --since auto`) + inject context.
|
||||||
|
- **PostToolUse** (on `Write`/`Edit` of vault `.md`) — `index update --file`.
|
||||||
|
- **SessionEnd** — append daily journal note.
|
||||||
|
- (memsearch brings its own `UserPromptSubmit` + capture hooks for the episodic layer.)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Part D — Claude Code plugin with skills (global install)
|
||||||
|
|
||||||
|
Goal: one global install so every project/machine knows the vault, conventions, hooks, and CLI.
|
||||||
|
|
||||||
|
**Plugin contents:**
|
||||||
|
- **Hooks** registered in settings: SessionStart, PostToolUse, SessionEnd (the shell wrappers
|
||||||
|
from Part C), pointed at a configurable vault path + cache dir.
|
||||||
|
- **The Ruby CLI** (the tag index) bundled or installed as a dependency, on `PATH`.
|
||||||
|
- **Skills** (these carry the *know-how* to the model):
|
||||||
|
- `memory-write` — when to record evergreen knowledge, the frontmatter contract, scope rules,
|
||||||
|
"vault not repo." (Part B write section.)
|
||||||
|
- `memory-query` — how/when to query the tag index vs the episodic layer; the progressive-
|
||||||
|
disclosure loop; cross-client lookups. (Part B query section.)
|
||||||
|
- `memory-reorganize` — the plan-mode consolidation/promotion procedure + guardrails.
|
||||||
|
- **Config**: vault path, cache path, sync method — set once at user level.
|
||||||
|
|
||||||
|
**Why a plugin + skills (not just CLAUDE.md):** hooks must be registered by the harness (the
|
||||||
|
plugin does that), and the conventions are taught as skills so they load on demand without
|
||||||
|
bloating every project's context. A single global install keeps the conventions themselves a
|
||||||
|
single source of truth — edit the skill once, every project follows.
|
||||||
|
|
||||||
|
**Open question for build time**: do the coding-`convention/*` notes live *in the vault* (data,
|
||||||
|
edited freely, resolved by the SessionStart hook) while the *memory-system skills* live *in the
|
||||||
|
plugin* (behavior, versioned)? Recommended: yes — conventions are data in the vault; the skills
|
||||||
|
are the mechanism.
|
||||||
|
|
@ -0,0 +1,82 @@
|
||||||
|
# Session Handoff
|
||||||
|
|
||||||
|
_Written 2026-06-03 at the end of the brainstorming session. For the next session to pick up._
|
||||||
|
|
||||||
|
## What this is
|
||||||
|
|
||||||
|
Designing a personal, cross-project **memory system for Claude Code** for a multi-client
|
||||||
|
freelancer. We finished **brainstorming/architecture** and wrote the design + ADRs. **No
|
||||||
|
implementation has started.** The source material is the transcript at
|
||||||
|
`/home/jared/Documents/cc-os/memory-systems-compared060326` (the "6 Levels of Claude Code
|
||||||
|
Memory" video).
|
||||||
|
|
||||||
|
## Where we are
|
||||||
|
|
||||||
|
- ✅ Synthesized the video (`01-video-synthesis.md`).
|
||||||
|
- ✅ Settled the architecture and got user sign-off (`02-system-design.md`).
|
||||||
|
- ✅ Recorded every decision + rejected/deferred options (`03-architecture-decisions.md`).
|
||||||
|
- ✅ Wrote the build outline + operational answers (`04-build-plan.md`).
|
||||||
|
- ⏳ **Next**: turn `04-build-plan.md` into a real implementation plan and start building.
|
||||||
|
|
||||||
|
## The design in 30 seconds
|
||||||
|
|
||||||
|
Three local, markdown-as-truth, no-Docker layers:
|
||||||
|
1. **Episodic** ("what happened, when") = **memsearch** (Milvus Lite, off-the-shelf).
|
||||||
|
2. **Knowledge** ("how do we…") = **flat Obsidian vault** (single source of truth) + a
|
||||||
|
**Ruby/Sequel/SQLite tag index** with a CLI. Organized by **namespaced nested tags**
|
||||||
|
(`tool/ client/ domain/ convention/ scope/`), not folders.
|
||||||
|
3. **Semantic over the vault** = **QMD**, **deferred** until structured tagging proves
|
||||||
|
insufficient.
|
||||||
|
|
||||||
|
Retrieval is hook-injected + on-demand (projects stay thin). Freshness is lazy (write-time
|
||||||
|
hook + session-start reconcile; no daemon/cron). Vault syncs to a VPS via git/Syncthing;
|
||||||
|
indexes are disposable and rebuilt per machine. Ships as a **global Claude Code plugin with
|
||||||
|
skills**.
|
||||||
|
|
||||||
|
## Decisions locked (don't relitigate without reason)
|
||||||
|
|
||||||
|
- Two separate systems for episodic vs knowledge (ADR-001).
|
||||||
|
- memsearch for episodic (ADR-002). Flat vault + tags, not folders (ADR-003).
|
||||||
|
- SQLite + Sequel (Ruby) CLI tag index; markdown authoritative, cache disposable (ADR-004).
|
||||||
|
- Structured-first; semantic (QMD) deferred (ADR-005/006).
|
||||||
|
- Lazy freshness: PostToolUse write hook + SessionStart reconcile, no daemon/cron (ADR-007).
|
||||||
|
- Sync the vault not the indexes; reject OpenBrain/Mem0/Postgres (ADR-008).
|
||||||
|
- Package as a global plugin with skills (ADR-009).
|
||||||
|
- Query output = path + summary + matched tags (option C).
|
||||||
|
|
||||||
|
## Open questions for build time
|
||||||
|
|
||||||
|
1. **Vault location** — default `~/brain` (or similar synced home dir)? Symlink into
|
||||||
|
`~/.claude/memory` only if a tool requires it.
|
||||||
|
2. **Sync mechanism** — git (versioned history, hourly) vs Syncthing (continuous, zero-thought).
|
||||||
|
3. **Convention notes placement** — confirmed direction: coding `convention/*` live as **data
|
||||||
|
in the vault** (resolved by the SessionStart hook); the memory-system **skills live in the
|
||||||
|
plugin** (behavior, versioned). Validate when building.
|
||||||
|
4. **memsearch journal** — does memsearch index our SessionEnd journal notes, or only its own
|
||||||
|
auto-capture? Decide how the journal points into the knowledge vault.
|
||||||
|
5. **Promotion rule** — the concrete threshold for project→`scope/global` promotion (e.g.
|
||||||
|
recurrence count) used by the reorganize step.
|
||||||
|
6. **CLI ergonomics** — command framework (thor vs optparse), output format details, where
|
||||||
|
`last_cache_time` is stored.
|
||||||
|
|
||||||
|
## Recommended first actions next session
|
||||||
|
|
||||||
|
1. Re-read `02-system-design.md` and `04-build-plan.md`.
|
||||||
|
2. Invoke the **writing-plans** skill to convert `04-build-plan.md` Part A into a staged
|
||||||
|
implementation plan (Step 2, the Ruby tag-index CLI, is the critical path — build and test
|
||||||
|
it standalone first).
|
||||||
|
3. Resolve open questions 1–2 (vault location + sync) before writing code, since they affect
|
||||||
|
paths in the hooks and CLI.
|
||||||
|
4. Build Step 2 (CLI) with tests, then Step 3 (hooks), then validate end-to-end on the seeded
|
||||||
|
notes before touching the plugin packaging (Step 6).
|
||||||
|
|
||||||
|
## Context notes
|
||||||
|
|
||||||
|
- Working dir `/home/jared/Documents/cc-os` is **not a git repo**; consider `git init` if we
|
||||||
|
want history for these docs and the build.
|
||||||
|
- User prefers **Docker when reasonable** — but this design deliberately needs **none** (all
|
||||||
|
layers are embedded/local). Only full Milvus or QMD-server modes would involve Docker, and
|
||||||
|
we're not using those.
|
||||||
|
- User is Ruby-comfortable (chose Ruby + Sequel deliberately).
|
||||||
|
- A past `~/Documents/SecondBrain/` attempt (tag DB) was lost track of — the "markdown
|
||||||
|
authoritative, cache disposable & rebuildable" rule (ADR-004) is the explicit fix for that.
|
||||||
Loading…
Reference in New Issue