cc-os/docs/memory-system/02-system-design.md

System Design

Status: approved 2026-06-03. Implementation not yet started.

Goals (what this system must do)

1. Thin projects — keep as little AI context inside each project repo as possible. Projects focus on project files; knowledge is pulled in on demand or injected by hooks.
2. Cross-project / cross-client knowledge — the AI learns something once (e.g. the SEMrush API) and references it from anywhere. Two scopes: global (broadly useful) and project/client-specific (how a given client uses a tool) — both globally reachable. Ask anything client- or project-related from any project.
3. Timeline awareness — from any project, lightweight awareness of recent activity ("what was I doing an hour ago / yesterday"), with the ability to drill deeper.
4. Remote, local-fast — accessible anywhere (VPS / personal OS) but runs local-fast; lazy sync (minutes/hourly) is fine; real-time is overkill.

Desired properties: lightweight (low tokens), fast (out of the way), flexible (cross project/client), self-evolving (AI maintains it under clear rules), easy to manage (AI-managed), semi-structured (organization that can evolve).

Core principle: two memory types, kept separate

Type	Question	Lifecycle	Write path	Our tool
Episodic	"What happened, when?"	accretes & decays	auto-captured	memsearch
Semantic / knowledge	"How do we…?"	deliberately maintained	curated by you/AI	Obsidian vault + tag index

This is the classic episodic vs. semantic memory split. Keeping them separate is the key architectural decision — they have different lifecycles, write paths, and query patterns, and forcing one tool to do both is what made every earlier design feel forced.

The three layers

┌─────────────────────────────────────────────────────────────────┐
│ EPISODIC  ── memsearch (Milvus Lite, embedded, no Docker)         │
│   auto-captured session/journal notes · NL semantic recall        │
│   answers "when did we…", "what was I doing yesterday"            │
├─────────────────────────────────────────────────────────────────┤
│ KNOWLEDGE ── flat Obsidian vault (single source of truth)         │
│   + Ruby/Sequel/SQLite tag index (CLI)                            │
│   structured tag filtering · answers "how do we…", "what do we    │
│   know about X for client Y"                                       │
├─────────────────────────────────────────────────────────────────┤
│ SEMANTIC-OVER-KNOWLEDGE ── QMD (SQLite+vector, MCP)   [DEFERRED]   │
│   meaning-based recall over the vault when tags miss              │
│   add ONLY when structured tagging proves insufficient            │
└─────────────────────────────────────────────────────────────────┘

All three are local-first, markdown-as-truth, no Docker, no server, no API keys.

Layer 1 — Episodic (memsearch)

• What it is: a Claude Code plugin (by Zilliz) that auto-captures session notes as daily markdown, chunks them, and stores a shadow index in Milvus Lite (a single embedded file — no server, no Docker). Hybrid search = BM25 + dense vectors + RRF, local ONNX embeddings (bge-m3, no API key/cost). A FileWatcher (1500ms debounce) handles updates and deletions; markdown stays the source of truth.
• Why off-the-shelf: it already implements the OpenClaw daily-notes + "dreaming" pattern and the markdown-as-truth / disposable-shadow-index philosophy we'd otherwise hand-build.
• Role in our system: satisfies Goal 3 (timeline). The AI queries it in natural language ("what was decided about X last week"). We do not make it filter by our tags — it owns the episodic corpus only.

Layer 2 — Knowledge (vault + tag index)

The heart of the system, and the part we build.

Vault

• Flat markdown directory, single source of truth, configurable location (NOT forced into ~/.claude/; symlink if a tool insists). Browsable in Obsidian as a viewer.
• Replaces project-local documentation: instead of docs scattered per repo, knowledge lives once in the vault and is pulled into any project on demand.

Frontmatter contract (every note)

---
summary: One line, written at creation. The router shows this so the AI can pick a
  file without opening it.
tags:
  - tool/semrush          # namespaced, nested (slash = Obsidian nested tag)
  - client/sesame3g
  - domain/seo
  - scope/project         # or scope/global
---

• Namespaces are the "virtual indexes": tool/, client/, domain/, convention/, scope/. #tool matches all children — native prefix filtering, no folders needed.
• Two knowledge scopes via scope/global vs scope/project (+ a client/ tag): global = broadly useful tool/domain knowledge; project = how a specific client uses it. Both are globally queryable; the scope tag is the shortcut that avoids scanning every client's usage.

Index (the tag cache)

A small Ruby program, Sequel ORM, SQLite — the disposable structured cache over the vault.

• Schema:
  • files(id, path, mtime, summary, scope) — one row per note.
  • tags(id, name) — one row per distinct tag (enables enumerating the vocabulary: "what clients/tools do I have notes on?").
  • files_tags(file_id, tag_id) — many_to_many join.
• CLI (the only interface; the AI never touches SQLite directly):
  • index update --since <mtime> — incremental: pulls .md files with mtime > last cache time, re-reads their frontmatter, upserts; also reconciles (prunes rows whose path no longer exists).
  • index update --rebuild — full rebuild from scratch (default false).
  • index query --client X --tool Y [--scope global] [--domain Z] — returns path + summary + matched tags for each hit (decided: option C). Tags shown so the AI sees why a file matched.
  • index tags --namespace tool/ — enumerate a virtual index.
• Source of truth rule: markdown is authoritative; the SQLite file is a rebuildable cache that is never synced and can be deleted/rebuilt anytime.

Freshness (lazy — chosen Option A)

• AI writes → a PostToolUse hook on Write/Edit targeting vault .md files calls index update --file <path> (updates exactly that file, prunes if deleted). Event-driven, no polling, no staleness for AI edits.
• Manual edits (rare) → caught by a session-start reconcile (index update --since + prune). No daemon, no cron.

Retrieval (hook-injected + on-demand)

• Session-start hook injects: (a) a compact index/overview, (b) the current project's declared convention/* tags resolved to files (so coding conventions auto-pull and a convention edit propagates to every project using that tag), (c) a pointer to recent episodic journal.
• On demand: the AI runs index query to pull specific knowledge into context only when the task needs it. Projects stay thin — their CLAUDE.md holds tags/pointers, not content.

Layer 3 — Semantic over knowledge (QMD) — DEFERRED

• What: QMD (github.com/tobi/qmd) — local markdown search, SQLite + FTS5/BM25 + local vector embeddings (EmbeddingGemma-300M GGUF) + LLM rerank. CLI + optional MCP server. No Docker, no API keys. Proves SQLite + a local vector model is enough — no Milvus/Postgres for knowledge.
• Why deferred: start structured-only. Add QMD as a set-and-forget semantic layer over the vault only when we catch ourselves failing to retrieve notes we know exist (the video's "only level up when it bites"). It complements, not replaces, the tag index (QMD filters by path/collection context, not first-class frontmatter tags).

Timeline (Goal 3) details

• A session-end hook appends a daily journal note (one file per date) with pointers to the project/knowledge files touched. memsearch indexes these; today+yesterday are cheap to load, older entries are reachable by query for drill-down.

Self-evolution guardrails

• The AI writes only to the vault, never silently into project repos.
• Required frontmatter schema (summary + namespaced tags) is enforced so the index stays queryable.
• Daily notes are append-only; consolidation/reorg is a separate, reviewable step run in plan mode (Connelly's reorganize + Huryn's propose-and-approve loop).
• Promotion to scope/global requires a rule (e.g. a fact recurring N times) — not every stray note gets promoted.

Sync (Goal 4)

• The vault syncs to the VPS via git (versioned history, hourly) or Syncthing (continuous, zero-thought). Decision deferred to build time.
• Indexes are never synced — Milvus Lite and (later) QMD shadow indexes are rebuilt per machine. Sync only the markdown.

Packaging

• The whole thing ships as a global Claude Code plugin with skills (hooks + scripts + CRUD know-how) so every project, on every machine, knows how to use the vault effectively. See 04-build-plan.md.

How each goal is met

Goal	Met by
1. Thin projects	Knowledge in the vault, not repos; CLAUDE.md holds tags/pointers; on-demand index query
2. Cross-project/client knowledge, global vs project scopes	Flat vault + namespaced tags + scope/ + client/; enumerable virtual indexes
3. Timeline	memsearch episodic layer + session-end journal hook
4. Remote, local-fast	Markdown vault synced via git/Syncthing; disposable per-machine indexes