From 235fd45ff7c7fba52dfab40a1900f14d4478c710 Mon Sep 17 00:00:00 2001 From: jared Date: Thu, 4 Jun 2026 09:35:43 -0400 Subject: [PATCH] Update design docs and add implementation runbook per ADR-011/012 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update CLAUDE.md: clarify the knowledge layer now specifies the ~/Documents/SecondBrain vault, six-facet taxonomy, and reference the locked decisions (ADR-011, ADR-012). Update 02-system-design.md: align with faceted taxonomy (type/ client/project/domain/tool/convention/ plus scope/), express hierarchy via hubs + wikilinks + Graphify edges rather than nested paths, name the vault explicitly, update build-plan timestamp. Add 05-implementation-process.md: concrete build runbook integrating the locked decisions — seven steps bottom-up from vault migration through plugin packaging, with the Graphify + Ollama model benchmark (Step 2c) marked as the critical gate. Open questions deferred or defaulted; most are non-blocking. Design milestone: tag taxonomy and vault location locked. --- CLAUDE.md | 9 +- docs/memory-system/02-system-design.md | 34 +- .../05-implementation-process.md | 306 ++++++++++++++++++ 3 files changed, 332 insertions(+), 17 deletions(-) create mode 100644 docs/memory-system/05-implementation-process.md diff --git a/CLAUDE.md b/CLAUDE.md index b683246..1704d41 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -45,10 +45,9 @@ the design, update this paragraph (and the relevant `docs/memory-system/` files match.** Keep it accurate, don't preserve it for its own sake. Two memory types kept as **separate systems**: **episodic** ("what happened, when") handled by -**memsearch** (Milvus Lite, embedded), and **semantic/knowledge** ("how do we…") handled by a -flat **Obsidian markdown vault** as the single source of truth. Notes keep `summary` + -namespaced tags (`tool/`/`client/`/`domain/`/`convention/`/`scope/`) as metadata, and the vault -is queried via a **Graphify knowledge graph** (local Ollama SLM for doc extraction, free +**memsearch** (Milvus Lite, embedded), and **semantic/knowledge** ("how do we…") handled by the +existing **`~/Documents/SecondBrain` Obsidian vault** as the single source of truth. Notes keep +`summary` + six flat, parallel namespaced facets (`type/`/`client/`/`project/`/`domain/`/`tool/`/`convention/`) plus `scope/` as metadata; hierarchy and relationships are expressed via hub notes (`type/hub`), wikilinks, and Graphify graph edges — not nested tag paths. The vault is queried via a **Graphify knowledge graph** (local Ollama SLM for doc extraction, free tree-sitter AST for code). Retrieval is hook-injected + on-demand so project repos stay thin; freshness is lazy (write-time hook + SessionStart reconcile, no daemon/cron); the vault syncs to a VPS while @@ -59,6 +58,8 @@ tag-index CLI and also covers the deferred QMD semantic layer. `04-build-plan.md `06-graphify-evaluation.md` reflect this; if an older doc still describes the Ruby CLI, defer to those two and fix the stale doc. +**Decisions locked (2026-06-04):** Six-facet tag taxonomy + `scope/` (ADR-011); reuse `~/Documents/SecondBrain` vault rather than creating a new one (ADR-012). + ## OpenSpec workflow Changes are managed spec-driven via OpenSpec. Use the matching skills rather than editing spec diff --git a/docs/memory-system/02-system-design.md b/docs/memory-system/02-system-design.md index 7280856..49fc2bf 100644 --- a/docs/memory-system/02-system-design.md +++ b/docs/memory-system/02-system-design.md @@ -1,7 +1,9 @@ # System Design _Status: approved 2026-06-03; knowledge layer revised 2026-06-04 (Graphify replaces the Ruby -tag-index CLI and the deferred QMD layer — see ADR-010). Implementation not yet started._ +tag-index CLI and the deferred QMD layer — see ADR-010); tag taxonomy and vault location locked +2026-06-04 (six-facet taxonomy — see ADR-011; reuse ~/Documents/SecondBrain — see ADR-012). +Implementation not yet started._ ## Goals (what this system must do) @@ -39,8 +41,8 @@ forcing one tool to do both is what made every earlier design feel forced. │ auto-captured session/journal notes · NL semantic recall │ │ answers "when did we…", "what was I doing yesterday" │ ├─────────────────────────────────────────────────────────────────┤ -│ KNOWLEDGE ── flat Obsidian vault (single source of truth) │ -│ notes carry summary + namespaced tags (metadata) │ +│ KNOWLEDGE ── ~/Documents/SecondBrain Obsidian vault (single SOT) │ +│ notes carry summary + six facet tags + scope/ (metadata) │ │ + Graphify knowledge graph (local SLM over docs; AST over code) │ │ graph queries (god nodes / query / path) · answers "how do we…",│ │ "what do we know about X for client Y", "what relates to Y" │ @@ -72,8 +74,7 @@ filters), it is just no longer backed by a bespoke index. The heart of the system, and the part we build. ### Vault -- **Flat markdown directory**, single source of truth, **configurable location** (NOT forced - into `~/.claude/`; symlink if a tool insists). Browsable in Obsidian as a viewer. +- **Flat markdown directory**, single source of truth — reuses the existing **`~/Documents/SecondBrain`** Obsidian vault (ADR-012); not forced into `~/.claude/`; symlink if a tool insists. Browsable in Obsidian as a viewer. - **Replaces project-local documentation**: instead of docs scattered per repo, knowledge lives once in the vault and is pulled into any project on demand. @@ -83,14 +84,20 @@ The heart of the system, and the part we build. summary: One line, written at creation. The router shows this so the AI can pick a file without opening it. tags: - - tool/semrush # namespaced, nested (slash = Obsidian nested tag) + - type/reference # listed first by convention; e.g. type/hub, type/how-to - client/sesame3g + - project/website-redesign - domain/seo + - tool/semrush + - convention/api-style - scope/project # or scope/global --- ``` -- **Namespaces** are the "virtual indexes": `tool/`, `client/`, `domain/`, `convention/`, - `scope/`. `#tool` matches all children — native prefix filtering, no folders needed. +- **Six flat facets** (ADR-011): `type/`, `client/`, `project/`, `domain/`, `tool/`, + `convention/` — plus `scope/`. Each facet is independent and parallel (never nested into each + other). `#tool` matches all `tool/*` values — native Obsidian prefix filtering, no folders + needed. Hierarchy and relationships are expressed via **hub notes** (`type/hub`), + **wikilinks**, and **Graphify graph edges**, not nested tag paths. - **Two knowledge scopes** via `scope/global` vs `scope/project` (+ a `client/` tag): global = broadly useful tool/domain knowledge; project = how a specific client uses it. Both are globally queryable; the scope tag is the shortcut that avoids scanning every client's usage. @@ -113,11 +120,12 @@ layer (ADR-010): one graph gives both structured and semantic retrieval, without - **Query** (via CLI and an MCP server exposing `query_graph` / `get_node` / `shortest_path`): ask for **god nodes first**, then scalpel down with `graphify query` / `path` / `explain`. Prompt the graph; don't dump the corpus into context. -- **Metadata still matters**: the `summary` + namespaced tags remain first-class note +- **Metadata still matters**: the `summary` + six facet tags remain first-class note attributes — `summary` is the human-written router hint Graphify does **not** generate, and - the `tag/` namespaces stay useful for Obsidian filtering and as node attributes. They are - retained even though they no longer back a bespoke index. *(How tightly metadata feeds graph - queries is a refinement for build time.)* + the facet namespaces (`type/`, `client/`, `project/`, `domain/`, `tool/`, `convention/`, + `scope/`) stay useful for Obsidian filtering and as node attributes. They are retained even + though they no longer back a bespoke index. *(How tightly metadata feeds graph queries is a + refinement for build time.)* - **Source of truth rule**: markdown is authoritative; the graph (`graphify-out/`) is a rebuildable artifact that is **never synced** and can be deleted/rebuilt anytime (`graphify ... --force`). @@ -179,6 +187,6 @@ similarity would clearly win, revisit then (the video's "only level up when it b | Goal | Met by | |------|--------| | 1. Thin projects | Knowledge in the vault, not repos; CLAUDE.md holds tags/pointers; on-demand `index query` | -| 2. Cross-project/client knowledge, global vs project scopes | Flat vault + namespaced tags + `scope/` + `client/`; Graphify knowledge graph (god nodes + traversal) over it | +| 2. Cross-project/client knowledge, global vs project scopes | `~/Documents/SecondBrain` vault + six-facet tags (`type/`/`client/`/`project/`/`domain/`/`tool/`/`convention/`) + `scope/`; Graphify knowledge graph (god nodes + traversal) over it | | 3. Timeline | memsearch episodic layer + session-end journal hook | | 4. Remote, local-fast | Markdown vault synced via git/Syncthing; disposable per-machine graphs/indexes | diff --git a/docs/memory-system/05-implementation-process.md b/docs/memory-system/05-implementation-process.md new file mode 100644 index 0000000..bd68b69 --- /dev/null +++ b/docs/memory-system/05-implementation-process.md @@ -0,0 +1,306 @@ +# Implementation Process + +_Last updated: 2026-06-04_ | _Status: Ready to plan — awaiting Step 2c benchmark results_ + +This document distills `04-build-plan.md` into a concrete, staged build process and folds in +two locked decisions: **ADR-011** (faceted six-namespace taxonomy) and **ADR-012** (reuse the +existing `~/Documents/SecondBrain` vault rather than creating a new one). Read `04-build-plan.md` +for the underlying rationale, query/CRUD conventions, and plugin internals; read +`03-architecture-decisions.md` for the decision log. This document is the runbook. + +> This is still a build process outline, not a detailed implementation plan. The recommended +> next move is to feed Step 2 (the critical path) into the **writing-plans skill** to produce an +> executable, task-level plan. Most open questions in Steps 3–6 can be defaulted; Step 2c's +> model benchmark is the genuine gate that the rest depends on — do not hardcode a model before +> it runs. + +--- + +## Build order + +Bottom-up: vault migration first (usable standalone), then Graphify, then hooks, then plugin. + +--- + +## Step 1 — Adopt and migrate the SecondBrain vault + +**Per ADR-012:** reuse `~/Documents/SecondBrain` as the knowledge vault rather than creating a +new `~/brain`. The vault is already flat, already scoped to durable knowledge, and already +articulates the correct governance philosophy. Migration cost is mechanical. + +### 1a — Initialize git + +The vault has no `.git` yet. Initialize it so ADR-008's sync strategy works: + +```bash +git init ~/Documents/SecondBrain +``` + +### 1b — Update vault governance notes + +Update three files to the new schema: + +- **`CLAUDE.md`** — update vault path references, Graphify commands, ADR-011 namespace list. +- **`vault-conventions.md`** — update tag-namespace list to the six-facet form; keep the + existing "act without being asked" proactive-retrieval section intact (it's better than + what cc-os had — adopt it, don't overwrite it). +- **Project-config hub note(s)** — update the tag-inference table values to namespaced form + (e.g. `semrush-work → tool/semrush`, `pest-control → domain/pest-control`). Keep the + table; it operationalizes auto-tagging by path pattern and should be preserved. + +**Frontmatter contract** (apply to all notes, new and migrated): + +```yaml +--- +summary: "One-line, human-written router hint (required; Graphify does not generate this)" +tags: + - type/ # listed first by convention + - client/ # as applicable + - project/ # as applicable — first-class for a freelancer + - domain/ # as applicable + - tool/ # as applicable + - convention/ # as applicable + - scope/global # or scope/project +--- +``` + +`type/` listed first preserves the vault's existing type-first ordering habit and makes note +kind immediately visible. + +### 1c — Migrate existing notes (~20) + +For each existing note: + +1. Add `summary:` frontmatter (one line; write it now — it's the human-authored router hint). +2. Convert flat unnamespaced tags to the six-facet namespaced form. Examples: + - `research` → `type/research` + - `pest-control` → `domain/pest-control` + - `niche-automation-prospecting` → `domain/niche-automation-prospecting` (or split across + `domain/` + `project/` if the note is project-specific) + - `semrush` → `tool/semrush` +3. Add `scope/global` or `scope/project` to each note. + +Keep 5–10 migrated notes aside as representative test cases for Step 2c's extraction benchmark. +Include variety: a tool note, a client/project note, a convention note, a domain note, and at +least one note dense with relationships between concepts. + +### 1d — Create `_templates` for core note types only + +Add templates for: `research`, `howto`, `adr`, `hub`. The long tail stays freeform until a +pattern earns a template (per ADR-011). Consistent per-type structure improves Graphify's +local-SLM extraction reliability. + +### 1e — Fix the broken vault search reference + +`vault-conventions.md` (or `CLAUDE.md`) currently references +`~/.claude/scripts/vault_search.rb`, which does not exist. Replace every reference to that +script with `graphify query` (per ADR-010). + +--- + +## Step 2 — Graphify + Ollama setup (CRITICAL PATH) + +This step is the genuine gate. All hook behavior, model selection, and extraction tuning depend +on the benchmark in 2c. Do not skip or defer 2c. + +### 2a — Install and verify Graphify + +The PyPI package name has a double-y — this is the correct install command: + +```bash +pip install graphifyy +``` + +Verify: `graphify --version` + +### 2b — Configure Ollama + +Set these in your shell profile (or the plugin env block in Step 6): + +```bash +OLLAMA_FLASH_ATTENTION=1 # 30–50% VRAM savings on KV cache — always set +GRAPHIFY_OLLAMA_NUM_CTX=8192 # 8K is sufficient for vault notes (200–2000 words) + # leaves comfortable headroom for prompt overhead +GRAPHIFY_OLLAMA_KEEP_ALIVE=5 # set when packaging in Step 6 +``` + +Verify context allocation after the first extraction call: `ollama ps` shows allocated context. + +### 2c — Model benchmark (THE GATE) + +Run a small extraction test before committing to any model. Use the 5–10 representative +migrated notes from Step 1c as the test set. + +**Models to benchmark:** + +| Model | Backend | Size | Notes | +|---|---|---|---| +| `gemma4:e2b` | Ollama local | 7.2 GB | Primary candidate — fast, large context | +| `qwen3.5:2b` | Ollama local | 2.7 GB | Smallest; fallback if VRAM is constrained | +| `gemma4:e4b` | Ollama local | 9.6 GB | Highest local quality; slower | +| `claude-haiku-4-5` | Claude API | — | API baseline | +| `claude-sonnet-4-6` | Claude API | — | Mid-tier reference | +| `claude-opus-4-8` | Claude API | — | **Gold standard — score local models against this** | + +**What to measure per model:** + +1. Extraction speed (wall-clock time per note) +2. Entity quality: right concepts extracted, no hallucinated entities +3. Relationship quality: edges plausible and correctly typed, no missing key edges +4. Confidence tag accuracy: `INFERRED` vs `AMBIGUOUS` appropriately flagged + +**Decision rule:** choose the fastest local model whose entity/relationship quality is "close +enough" to Opus output. Do not hardcode `gemma4:e2b` — the benchmark decides. + +Review god-node quality in `GRAPH_REPORT.md` after each run. Opus output is the scoring rubric. + +### 2d — Build the initial vault graph + +After 2c picks a model: + +```bash +graphify extract --path ~/Documents/SecondBrain \ + --backend ollama --model \ + --token-budget 512 --max-concurrency 2 +``` + +Review `GRAPH_REPORT.md`. God nodes should be the most-connected tools, clients, and domain +concepts — confirm they make sense given the vault's content. + +### 2e — Per-project code graphs (free; no model) + +For each active client project: + +```bash +graphify extract --path ~/projects// --no-docs +``` + +`--no-docs` runs only tree-sitter AST — zero token cost. Use `--update` on subsequent runs; +use `--force` when files have been deleted (to clear stale nodes). Keep each project's +`graphify-out/` alongside the project; do not merge client projects into one graph. + +--- + +## Step 3 — Hooks + +Thin shell wrappers around Graphify. The logic lives in Graphify; the hooks only invoke it. + +| Hook | Trigger | Action | +|---|---|---| +| **PostToolUse** | AI writes/edits a vault `.md` | `graphify update --file ` | +| **SessionStart** | Session opens | Stale check → `--force` rebuild if needed → inject context | +| **SessionEnd** | Session closes | Append dated journal note to vault | + +**Stale-check mechanism (SessionStart):** read the mtime of +`~/.cache/graphify/vault-rebuild.stamp`. If older than N days (7 is the starting guess — tune +after observing drift), run `graphify --force` on the vault and write a new stamp. Then inject: +god-node summary from `GRAPH_REPORT.md` + `convention/*` note summaries + journal pointer. + +**Note on `--update` vs `--force`:** `graphify update --file` does not prune deleted nodes. +Stale/ghost nodes accumulate from manual deletes and renames. The periodic `--force` rebuild +triggered by the stale check is the mitigation — it rebuilds the graph clean. + +--- + +## Step 4 — Episodic layer (memsearch) + +Install memsearch for time-anchored "what happened / what was I working on" queries. Graphify +does **not** replace this — they serve different query patterns (knowledge graph vs timeline +recall). + +``` +/plugin marketplace add zilliztech/memsearch +plugin install memsearch +``` + +Verify daily memory files appear after a few conversations. The open question of whether +memsearch indexes SessionEnd journal notes or only its own auto-capture is deferred to build +time (see Open questions §7). + +--- + +## Step 5 — Vault sync + +Sync the markdown vault to the VPS. Pick one: + +- **git** — versioned, hourly push/pull via cron; explicit audit trail. +- **Syncthing** — continuous, bidirectional, zero-thought after setup. + +**What to sync:** the vault only (`~/Documents/SecondBrain`). + +**What NOT to sync:** `graphify-out/` directories, Milvus Lite caches, Ollama models. These are +disposable — rebuilt per machine from the vault (markdown is the single source of truth, per +ADR-008). + +--- + +## Step 6 — Package as a global Claude Code plugin + +One global install so every project and machine shares the same vault conventions, hooks, and +Graphify config. + +**Plugin contents:** + +- **Hooks** registered in settings: SessionStart, PostToolUse, SessionEnd — shell wrappers from + Step 3, parameterized by vault path and Graphify output dir. +- **Graphify CLI on PATH** — the AI calls `graphify query`, `graphify path`, `graphify explain` + via the Bash tool. No server process per graph; project-specific graphs are queried with + `--graph /graphify-out/graph.json`. +- **Skills** (carry the know-how to the model): + - `memory-write` — when to record evergreen knowledge, frontmatter contract, scope rule, vault not repo. + - `memory-query` — `graphify query` vs memsearch; god-node discipline; `--budget`/`--dfs`; cross-client lookups; progressive disclosure via `summary`. + - `memory-reorganize` — plan-mode consolidation/promotion procedure; when to trigger `--force` rebuild; human-approval guardrails. +- **Env vars** baked in: `OLLAMA_FLASH_ATTENTION=1`, `GRAPHIFY_OLLAMA_NUM_CTX=8192`, + `GRAPHIFY_OLLAMA_KEEP_ALIVE=5`. +- **Config** (set once at user level): vault path, Graphify output dir, Ollama model name, + stale-rebuild threshold in days. + +--- + +## Open questions / decisions still to settle + +These are deferred to build time. Most can be defaulted without blocking; the only genuine gate +is **§6 (Step 2c benchmark)** — model selection is blocked until it runs. + +1. **Vault symlink** — `~/Documents/SecondBrain` is confirmed as the vault (RESOLVED per + ADR-012). The open sub-question: symlink it into `~/.claude/memory` or not? Only needed if + a tool requires that path. + +2. **Sync mechanism** — git (versioned, hourly) vs Syncthing (continuous). Both are valid; + choose at build time based on preference. + +3. **Stale rebuild threshold** — 7 days is the starting guess. Tune after observing how quickly + ghost-node drift becomes noticeable in practice. + +4. **Per-project graph path convention** — how does SessionStart know which `graph.json` to + inject? Proposed convention: `/graphify-out/graph.json`, injected only if the + file exists at that path. Not yet made explicit in the plugin spec. + +5. **`summary` field discipline** — Graphify extracts entities and edges but does not write + summaries. The human (or AI at note-creation time) must write `summary:` frontmatter. Confirm + this holds in practice and add a lint/reminder to the memory-write skill if it drifts. + +6. **Model benchmark results (Step 2c)** — all model choices are deferred until the benchmark + runs. Do not hardcode `gemma4:e2b` (it's the primary candidate, not the confirmed pick). + +7. **memsearch + journal integration** — does memsearch index SessionEnd journal notes or only + its own auto-capture? How does the journal pointer injected at SessionStart reference the + vault? Nail down at memsearch install time (Step 4). + +8. **MCP server management** — vault graph + N project graphs = N+1 potential instances. The + build plan leans toward a single Graphify CLI on PATH with `--graph ` rather than + running multiple `graphify serve` instances, but this is not yet made explicit in the plugin + spec. Confirm the CLI-only approach or define the server lifecycle. + +**Resolved:** The type-tag taxonomy is settled via ADR-011 (faceted, six independent namespaces; +`type/` listed first). No further debate needed on taxonomy shape. + +--- + +## Recommended next move + +Turn **Step 2** into a detailed implementation plan via the **writing-plans skill**. It is the +critical path: Graphify install, Ollama configuration, the model benchmark (2c), and the initial +graph build are the smallest set of tasks that unblock everything else. Open questions §1–5 and +§7–8 can be defaulted or deferred; only §6 (the benchmark) genuinely blocks model-dependent +decisions.