Update design docs and add implementation runbook per ADR-011/012

Update CLAUDE.md: clarify the knowledge layer now specifies the
~/Documents/SecondBrain vault, six-facet taxonomy, and reference the
locked decisions (ADR-011, ADR-012).

Update 02-system-design.md: align with faceted taxonomy (type/
client/project/domain/tool/convention/ plus scope/), express hierarchy
via hubs + wikilinks + Graphify edges rather than nested paths, name
the vault explicitly, update build-plan timestamp.

Add 05-implementation-process.md: concrete build runbook integrating the
locked decisions — seven steps bottom-up from vault migration through
plugin packaging, with the Graphify + Ollama model benchmark (Step 2c)
marked as the critical gate. Open questions deferred or defaulted; most
are non-blocking.

Design milestone: tag taxonomy and vault location locked.
This commit is contained in:
jared 2026-06-04 09:35:43 -04:00
parent f73cb46a53
commit 235fd45ff7
3 changed files with 332 additions and 17 deletions

View File

@ -45,10 +45,9 @@ the design, update this paragraph (and the relevant `docs/memory-system/` files
match.** Keep it accurate, don't preserve it for its own sake.
Two memory types kept as **separate systems**: **episodic** ("what happened, when") handled by
**memsearch** (Milvus Lite, embedded), and **semantic/knowledge** ("how do we…") handled by a
flat **Obsidian markdown vault** as the single source of truth. Notes keep `summary` +
namespaced tags (`tool/`/`client/`/`domain/`/`convention/`/`scope/`) as metadata, and the vault
is queried via a **Graphify knowledge graph** (local Ollama SLM for doc extraction, free
**memsearch** (Milvus Lite, embedded), and **semantic/knowledge** ("how do we…") handled by the
existing **`~/Documents/SecondBrain` Obsidian vault** as the single source of truth. Notes keep
`summary` + six flat, parallel namespaced facets (`type/`/`client/`/`project/`/`domain/`/`tool/`/`convention/`) plus `scope/` as metadata; hierarchy and relationships are expressed via hub notes (`type/hub`), wikilinks, and Graphify graph edges — not nested tag paths. The vault is queried via a **Graphify knowledge graph** (local Ollama SLM for doc extraction, free
tree-sitter AST for code). Retrieval is hook-injected + on-demand so project repos stay thin;
freshness is lazy
(write-time hook + SessionStart reconcile, no daemon/cron); the vault syncs to a VPS while
@ -59,6 +58,8 @@ tag-index CLI and also covers the deferred QMD semantic layer. `04-build-plan.md
`06-graphify-evaluation.md` reflect this; if an older doc still describes the Ruby CLI, defer
to those two and fix the stale doc.
**Decisions locked (2026-06-04):** Six-facet tag taxonomy + `scope/` (ADR-011); reuse `~/Documents/SecondBrain` vault rather than creating a new one (ADR-012).
## OpenSpec workflow
Changes are managed spec-driven via OpenSpec. Use the matching skills rather than editing spec

View File

@ -1,7 +1,9 @@
# System Design
_Status: approved 2026-06-03; knowledge layer revised 2026-06-04 (Graphify replaces the Ruby
tag-index CLI and the deferred QMD layer — see ADR-010). Implementation not yet started._
tag-index CLI and the deferred QMD layer — see ADR-010); tag taxonomy and vault location locked
2026-06-04 (six-facet taxonomy — see ADR-011; reuse ~/Documents/SecondBrain — see ADR-012).
Implementation not yet started._
## Goals (what this system must do)
@ -39,8 +41,8 @@ forcing one tool to do both is what made every earlier design feel forced.
│ auto-captured session/journal notes · NL semantic recall │
│ answers "when did we…", "what was I doing yesterday" │
├─────────────────────────────────────────────────────────────────┤
│ KNOWLEDGE ── flat Obsidian vault (single source of truth)
│ notes carry summary + namespaced tags (metadata)
│ KNOWLEDGE ── ~/Documents/SecondBrain Obsidian vault (single SOT)
│ notes carry summary + six facet tags + scope/ (metadata)
│ + Graphify knowledge graph (local SLM over docs; AST over code) │
│ graph queries (god nodes / query / path) · answers "how do we…",│
│ "what do we know about X for client Y", "what relates to Y" │
@ -72,8 +74,7 @@ filters), it is just no longer backed by a bespoke index.
The heart of the system, and the part we build.
### Vault
- **Flat markdown directory**, single source of truth, **configurable location** (NOT forced
into `~/.claude/`; symlink if a tool insists). Browsable in Obsidian as a viewer.
- **Flat markdown directory**, single source of truth — reuses the existing **`~/Documents/SecondBrain`** Obsidian vault (ADR-012); not forced into `~/.claude/`; symlink if a tool insists. Browsable in Obsidian as a viewer.
- **Replaces project-local documentation**: instead of docs scattered per repo, knowledge
lives once in the vault and is pulled into any project on demand.
@ -83,14 +84,20 @@ The heart of the system, and the part we build.
summary: One line, written at creation. The router shows this so the AI can pick a
file without opening it.
tags:
- tool/semrush # namespaced, nested (slash = Obsidian nested tag)
- type/reference # listed first by convention; e.g. type/hub, type/how-to
- client/sesame3g
- project/website-redesign
- domain/seo
- tool/semrush
- convention/api-style
- scope/project # or scope/global
---
```
- **Namespaces** are the "virtual indexes": `tool/`, `client/`, `domain/`, `convention/`,
`scope/`. `#tool` matches all children — native prefix filtering, no folders needed.
- **Six flat facets** (ADR-011): `type/`, `client/`, `project/`, `domain/`, `tool/`,
`convention/` — plus `scope/`. Each facet is independent and parallel (never nested into each
other). `#tool` matches all `tool/*` values — native Obsidian prefix filtering, no folders
needed. Hierarchy and relationships are expressed via **hub notes** (`type/hub`),
**wikilinks**, and **Graphify graph edges**, not nested tag paths.
- **Two knowledge scopes** via `scope/global` vs `scope/project` (+ a `client/` tag): global =
broadly useful tool/domain knowledge; project = how a specific client uses it. Both are
globally queryable; the scope tag is the shortcut that avoids scanning every client's usage.
@ -113,11 +120,12 @@ layer (ADR-010): one graph gives both structured and semantic retrieval, without
- **Query** (via CLI and an MCP server exposing `query_graph` / `get_node` / `shortest_path`):
ask for **god nodes first**, then scalpel down with `graphify query` / `path` / `explain`.
Prompt the graph; don't dump the corpus into context.
- **Metadata still matters**: the `summary` + namespaced tags remain first-class note
- **Metadata still matters**: the `summary` + six facet tags remain first-class note
attributes — `summary` is the human-written router hint Graphify does **not** generate, and
the `tag/` namespaces stay useful for Obsidian filtering and as node attributes. They are
retained even though they no longer back a bespoke index. *(How tightly metadata feeds graph
queries is a refinement for build time.)*
the facet namespaces (`type/`, `client/`, `project/`, `domain/`, `tool/`, `convention/`,
`scope/`) stay useful for Obsidian filtering and as node attributes. They are retained even
though they no longer back a bespoke index. *(How tightly metadata feeds graph queries is a
refinement for build time.)*
- **Source of truth rule**: markdown is authoritative; the graph (`graphify-out/`) is a
rebuildable artifact that is **never synced** and can be deleted/rebuilt anytime
(`graphify ... --force`).
@ -179,6 +187,6 @@ similarity would clearly win, revisit then (the video's "only level up when it b
| Goal | Met by |
|------|--------|
| 1. Thin projects | Knowledge in the vault, not repos; CLAUDE.md holds tags/pointers; on-demand `index query` |
| 2. Cross-project/client knowledge, global vs project scopes | Flat vault + namespaced tags + `scope/` + `client/`; Graphify knowledge graph (god nodes + traversal) over it |
| 2. Cross-project/client knowledge, global vs project scopes | `~/Documents/SecondBrain` vault + six-facet tags (`type/`/`client/`/`project/`/`domain/`/`tool/`/`convention/`) + `scope/`; Graphify knowledge graph (god nodes + traversal) over it |
| 3. Timeline | memsearch episodic layer + session-end journal hook |
| 4. Remote, local-fast | Markdown vault synced via git/Syncthing; disposable per-machine graphs/indexes |

View File

@ -0,0 +1,306 @@
# Implementation Process
_Last updated: 2026-06-04_ | _Status: Ready to plan — awaiting Step 2c benchmark results_
This document distills `04-build-plan.md` into a concrete, staged build process and folds in
two locked decisions: **ADR-011** (faceted six-namespace taxonomy) and **ADR-012** (reuse the
existing `~/Documents/SecondBrain` vault rather than creating a new one). Read `04-build-plan.md`
for the underlying rationale, query/CRUD conventions, and plugin internals; read
`03-architecture-decisions.md` for the decision log. This document is the runbook.
> This is still a build process outline, not a detailed implementation plan. The recommended
> next move is to feed Step 2 (the critical path) into the **writing-plans skill** to produce an
> executable, task-level plan. Most open questions in Steps 36 can be defaulted; Step 2c's
> model benchmark is the genuine gate that the rest depends on — do not hardcode a model before
> it runs.
---
## Build order
Bottom-up: vault migration first (usable standalone), then Graphify, then hooks, then plugin.
---
## Step 1 — Adopt and migrate the SecondBrain vault
**Per ADR-012:** reuse `~/Documents/SecondBrain` as the knowledge vault rather than creating a
new `~/brain`. The vault is already flat, already scoped to durable knowledge, and already
articulates the correct governance philosophy. Migration cost is mechanical.
### 1a — Initialize git
The vault has no `.git` yet. Initialize it so ADR-008's sync strategy works:
```bash
git init ~/Documents/SecondBrain
```
### 1b — Update vault governance notes
Update three files to the new schema:
- **`CLAUDE.md`** — update vault path references, Graphify commands, ADR-011 namespace list.
- **`vault-conventions.md`** — update tag-namespace list to the six-facet form; keep the
existing "act without being asked" proactive-retrieval section intact (it's better than
what cc-os had — adopt it, don't overwrite it).
- **Project-config hub note(s)** — update the tag-inference table values to namespaced form
(e.g. `semrush-work → tool/semrush`, `pest-control → domain/pest-control`). Keep the
table; it operationalizes auto-tagging by path pattern and should be preserved.
**Frontmatter contract** (apply to all notes, new and migrated):
```yaml
---
summary: "One-line, human-written router hint (required; Graphify does not generate this)"
tags:
- type/<kind> # listed first by convention
- client/<name> # as applicable
- project/<name> # as applicable — first-class for a freelancer
- domain/<topic> # as applicable
- tool/<name> # as applicable
- convention/<name> # as applicable
- scope/global # or scope/project
---
```
`type/` listed first preserves the vault's existing type-first ordering habit and makes note
kind immediately visible.
### 1c — Migrate existing notes (~20)
For each existing note:
1. Add `summary:` frontmatter (one line; write it now — it's the human-authored router hint).
2. Convert flat unnamespaced tags to the six-facet namespaced form. Examples:
- `research``type/research`
- `pest-control``domain/pest-control`
- `niche-automation-prospecting``domain/niche-automation-prospecting` (or split across
`domain/` + `project/` if the note is project-specific)
- `semrush``tool/semrush`
3. Add `scope/global` or `scope/project` to each note.
Keep 510 migrated notes aside as representative test cases for Step 2c's extraction benchmark.
Include variety: a tool note, a client/project note, a convention note, a domain note, and at
least one note dense with relationships between concepts.
### 1d — Create `_templates` for core note types only
Add templates for: `research`, `howto`, `adr`, `hub`. The long tail stays freeform until a
pattern earns a template (per ADR-011). Consistent per-type structure improves Graphify's
local-SLM extraction reliability.
### 1e — Fix the broken vault search reference
`vault-conventions.md` (or `CLAUDE.md`) currently references
`~/.claude/scripts/vault_search.rb`, which does not exist. Replace every reference to that
script with `graphify query` (per ADR-010).
---
## Step 2 — Graphify + Ollama setup (CRITICAL PATH)
This step is the genuine gate. All hook behavior, model selection, and extraction tuning depend
on the benchmark in 2c. Do not skip or defer 2c.
### 2a — Install and verify Graphify
The PyPI package name has a double-y — this is the correct install command:
```bash
pip install graphifyy
```
Verify: `graphify --version`
### 2b — Configure Ollama
Set these in your shell profile (or the plugin env block in Step 6):
```bash
OLLAMA_FLASH_ATTENTION=1 # 3050% VRAM savings on KV cache — always set
GRAPHIFY_OLLAMA_NUM_CTX=8192 # 8K is sufficient for vault notes (2002000 words)
# leaves comfortable headroom for prompt overhead
GRAPHIFY_OLLAMA_KEEP_ALIVE=5 # set when packaging in Step 6
```
Verify context allocation after the first extraction call: `ollama ps` shows allocated context.
### 2c — Model benchmark (THE GATE)
Run a small extraction test before committing to any model. Use the 510 representative
migrated notes from Step 1c as the test set.
**Models to benchmark:**
| Model | Backend | Size | Notes |
|---|---|---|---|
| `gemma4:e2b` | Ollama local | 7.2 GB | Primary candidate — fast, large context |
| `qwen3.5:2b` | Ollama local | 2.7 GB | Smallest; fallback if VRAM is constrained |
| `gemma4:e4b` | Ollama local | 9.6 GB | Highest local quality; slower |
| `claude-haiku-4-5` | Claude API | — | API baseline |
| `claude-sonnet-4-6` | Claude API | — | Mid-tier reference |
| `claude-opus-4-8` | Claude API | — | **Gold standard — score local models against this** |
**What to measure per model:**
1. Extraction speed (wall-clock time per note)
2. Entity quality: right concepts extracted, no hallucinated entities
3. Relationship quality: edges plausible and correctly typed, no missing key edges
4. Confidence tag accuracy: `INFERRED` vs `AMBIGUOUS` appropriately flagged
**Decision rule:** choose the fastest local model whose entity/relationship quality is "close
enough" to Opus output. Do not hardcode `gemma4:e2b` — the benchmark decides.
Review god-node quality in `GRAPH_REPORT.md` after each run. Opus output is the scoring rubric.
### 2d — Build the initial vault graph
After 2c picks a model:
```bash
graphify extract --path ~/Documents/SecondBrain \
--backend ollama --model <chosen-model> \
--token-budget 512 --max-concurrency 2
```
Review `GRAPH_REPORT.md`. God nodes should be the most-connected tools, clients, and domain
concepts — confirm they make sense given the vault's content.
### 2e — Per-project code graphs (free; no model)
For each active client project:
```bash
graphify extract --path ~/projects/<client>/<project> --no-docs
```
`--no-docs` runs only tree-sitter AST — zero token cost. Use `--update` on subsequent runs;
use `--force` when files have been deleted (to clear stale nodes). Keep each project's
`graphify-out/` alongside the project; do not merge client projects into one graph.
---
## Step 3 — Hooks
Thin shell wrappers around Graphify. The logic lives in Graphify; the hooks only invoke it.
| Hook | Trigger | Action |
|---|---|---|
| **PostToolUse** | AI writes/edits a vault `.md` | `graphify update --file <path>` |
| **SessionStart** | Session opens | Stale check → `--force` rebuild if needed → inject context |
| **SessionEnd** | Session closes | Append dated journal note to vault |
**Stale-check mechanism (SessionStart):** read the mtime of
`~/.cache/graphify/vault-rebuild.stamp`. If older than N days (7 is the starting guess — tune
after observing drift), run `graphify --force` on the vault and write a new stamp. Then inject:
god-node summary from `GRAPH_REPORT.md` + `convention/*` note summaries + journal pointer.
**Note on `--update` vs `--force`:** `graphify update --file` does not prune deleted nodes.
Stale/ghost nodes accumulate from manual deletes and renames. The periodic `--force` rebuild
triggered by the stale check is the mitigation — it rebuilds the graph clean.
---
## Step 4 — Episodic layer (memsearch)
Install memsearch for time-anchored "what happened / what was I working on" queries. Graphify
does **not** replace this — they serve different query patterns (knowledge graph vs timeline
recall).
```
/plugin marketplace add zilliztech/memsearch
plugin install memsearch
```
Verify daily memory files appear after a few conversations. The open question of whether
memsearch indexes SessionEnd journal notes or only its own auto-capture is deferred to build
time (see Open questions §7).
---
## Step 5 — Vault sync
Sync the markdown vault to the VPS. Pick one:
- **git** — versioned, hourly push/pull via cron; explicit audit trail.
- **Syncthing** — continuous, bidirectional, zero-thought after setup.
**What to sync:** the vault only (`~/Documents/SecondBrain`).
**What NOT to sync:** `graphify-out/` directories, Milvus Lite caches, Ollama models. These are
disposable — rebuilt per machine from the vault (markdown is the single source of truth, per
ADR-008).
---
## Step 6 — Package as a global Claude Code plugin
One global install so every project and machine shares the same vault conventions, hooks, and
Graphify config.
**Plugin contents:**
- **Hooks** registered in settings: SessionStart, PostToolUse, SessionEnd — shell wrappers from
Step 3, parameterized by vault path and Graphify output dir.
- **Graphify CLI on PATH** — the AI calls `graphify query`, `graphify path`, `graphify explain`
via the Bash tool. No server process per graph; project-specific graphs are queried with
`--graph <project-root>/graphify-out/graph.json`.
- **Skills** (carry the know-how to the model):
- `memory-write` — when to record evergreen knowledge, frontmatter contract, scope rule, vault not repo.
- `memory-query``graphify query` vs memsearch; god-node discipline; `--budget`/`--dfs`; cross-client lookups; progressive disclosure via `summary`.
- `memory-reorganize` — plan-mode consolidation/promotion procedure; when to trigger `--force` rebuild; human-approval guardrails.
- **Env vars** baked in: `OLLAMA_FLASH_ATTENTION=1`, `GRAPHIFY_OLLAMA_NUM_CTX=8192`,
`GRAPHIFY_OLLAMA_KEEP_ALIVE=5`.
- **Config** (set once at user level): vault path, Graphify output dir, Ollama model name,
stale-rebuild threshold in days.
---
## Open questions / decisions still to settle
These are deferred to build time. Most can be defaulted without blocking; the only genuine gate
is **§6 (Step 2c benchmark)** — model selection is blocked until it runs.
1. **Vault symlink**`~/Documents/SecondBrain` is confirmed as the vault (RESOLVED per
ADR-012). The open sub-question: symlink it into `~/.claude/memory` or not? Only needed if
a tool requires that path.
2. **Sync mechanism** — git (versioned, hourly) vs Syncthing (continuous). Both are valid;
choose at build time based on preference.
3. **Stale rebuild threshold** — 7 days is the starting guess. Tune after observing how quickly
ghost-node drift becomes noticeable in practice.
4. **Per-project graph path convention** — how does SessionStart know which `graph.json` to
inject? Proposed convention: `<project-root>/graphify-out/graph.json`, injected only if the
file exists at that path. Not yet made explicit in the plugin spec.
5. **`summary` field discipline** — Graphify extracts entities and edges but does not write
summaries. The human (or AI at note-creation time) must write `summary:` frontmatter. Confirm
this holds in practice and add a lint/reminder to the memory-write skill if it drifts.
6. **Model benchmark results (Step 2c)** — all model choices are deferred until the benchmark
runs. Do not hardcode `gemma4:e2b` (it's the primary candidate, not the confirmed pick).
7. **memsearch + journal integration** — does memsearch index SessionEnd journal notes or only
its own auto-capture? How does the journal pointer injected at SessionStart reference the
vault? Nail down at memsearch install time (Step 4).
8. **MCP server management** — vault graph + N project graphs = N+1 potential instances. The
build plan leans toward a single Graphify CLI on PATH with `--graph <path>` rather than
running multiple `graphify serve` instances, but this is not yet made explicit in the plugin
spec. Confirm the CLI-only approach or define the server lifecycle.
**Resolved:** The type-tag taxonomy is settled via ADR-011 (faceted, six independent namespaces;
`type/` listed first). No further debate needed on taxonomy shape.
---
## Recommended next move
Turn **Step 2** into a detailed implementation plan via the **writing-plans skill**. It is the
critical path: Graphify install, Ollama configuration, the model benchmark (2c), and the initial
graph build are the smallest set of tasks that unblock everything else. Open questions §15 and
§78 can be defaulted or deferred; only §6 (the benchmark) genuinely blocks model-dependent
decisions.