cc-os/docs/memory-system/04-build-plan.md

# Build Plan

_Last updated: 2026-06-04_

How a human builds this system, step by step, and answers to the operational questions:
which scripts and hooks, how the AI knows when to write and what conventions to follow, how and
when it queries, the CRUD hooks, and how it's packaged as a global Claude Code plugin with
skills.

> This is a build outline, not the implementation plan. The next session should turn this into
> a proper implementation plan (writing-plans skill) and execute it.

**Architecture decision (2026-06-03):** Graphify replaces the Ruby/SQLite tag index.
The Ruby CLI (old Step 2) is skipped. Graphify handles both knowledge-graph queries over the
vault and free AST-based code graphs per project. The deferred QMD semantic layer (old Step 7)
is also skipped — Graphify covers it without vectors. See `06-graphify-evaluation.md` for
the full rationale.

**Architecture decision (2026-06-04):** Build order inverted (ADR-013). Build and validate
the full system against a small 5–10 note fixture set first; defer bulk vault migration to last.
Validate end-to-end on one small pilot project containing both code and documents before
onboarding any others. Onboard remaining projects one at a time, with observe-and-adjust between
each. Steps 2d and 2e below are updated accordingly.

---

## Part A — Build order (human builder's path)

Build bottom-up: the vault and Graphify first (usable standalone), then the hooks, then the
plugin that packages it.

### Step 1 — Vault skeleton & conventions
- Vault location is settled by ADR-012: `~/Documents/SecondBrain` (the existing Obsidian
  vault). No new vault is created; symlink into `~/.claude/memory` only if a tool insists.
  **Vault is the single source of truth.**
- Write `CONVENTIONS.md` in the vault: the frontmatter contract. The required fields are
  `summary` (one-line, author-written — this is the human-readable router hint Graphify
  does not generate) and `scope` (`scope/global` or `scope/project`). Tags are now
  supplementary, not the primary query mechanism.
- Seed a few real notes (e.g. `tool/semrush`, `client/<x>`) to use as extraction test cases
  in Step 2.

### Step 2 — Graphify + Ollama setup (the knowledge graph layer)

This replaces the Ruby tag-index CLI. Graphify builds a knowledge graph over vault docs via a
local SLM, and a free code graph over project files via AST.

#### 2a — Install Graphify
```
pip install graphify   # or follow current install docs
```
Verify: `graphify --version`

#### 2b — Configure Ollama for extraction
The 2024 approach (Modelfile with `PARAMETER num_ctx`) still works but the preferred
programmatic method is now per-request via the API (Graphify supports this via
`GRAPHIFY_OLLAMA_NUM_CTX`):

```bash
# Set in your shell profile or service env:
OLLAMA_FLASH_ATTENTION=1          # 30-50% VRAM savings on KV cache — always set this
GRAPHIFY_OLLAMA_NUM_CTX=8192      # 8K is sufficient for vault notes (200-2000 words)
                                  # and leaves comfortable headroom for prompt overhead
```

To verify context is being used: `ollama ps` shows allocated context after first call.

Available models (as of 2026-06-03, in order of interest):
- `gemma4:e2b` — 7.2 GB, ~2B effective params, 128K native window, fast; **start here**
- `qwen3.5:2b` — 2.7 GB, smallest, good fallback if VRAM is constrained
- `gemma4:e4b` — 9.6 GB, more capable, slower

#### 2c — Claude reference-set benchmark (THE GATE, before committing to a local model)

This step produces the **gold-standard reference set** — one structured extraction output per
fixture note per Claude tier. It does **not** choose the final extraction model, and it does
**not** measure speed.

**Input:** the 5–10 fixture notes selected in Step 1 (from the runbook).

**What to run:** dispatch one Claude Code subagent per tier (Claude-tiers only — Ollama models
are not reachable in this environment):

| Tier | Model ID | Role |
|---|---|---|
| Haiku | `claude-haiku-4-5` | Lightweight reference |
| Sonnet | `claude-sonnet-4-6` | Mid-tier reference |
| Opus | `claude-opus-4-8` | **Gold standard** (scoring rubric) |

Each subagent receives only the fixture note text plus the shared extraction spec — no design
docs, no project context (fairness contract). See ADR-013.

**Metrics — quality only (wall-clock speed is explicitly out of scope here):**
1. Entity correctness — right concepts extracted, no hallucinated entities
2. Relationship plausibility and typing — edges plausible, correctly typed, no missing key edges
3. Confidence-tag accuracy — `INFERRED` vs `AMBIGUOUS` applied appropriately

**Deliverables** (produced by this step):
- Dispatch prompt (copy/paste-able): `docs/memory-system/benchmark/dispatch-prompt.md`
- Shared extraction schema: `docs/memory-system/benchmark/extraction-spec.md`
- Per-tier outputs: `docs/memory-system/benchmark/reference-outputs/<note-slug>.<tier>.md`

Opus output is the rubric. **Deferred later step:** local Ollama models (gemma4:e2b, qwen3.5:2b,
gemma4:e4b) are timed AND scored against these references — that scoring run is where speed
re-enters and the final model is chosen. Do not hardcode a model before that run completes.

**Status (2026-06-04): EXECUTED.** 6 cross-domain fixtures × 3 Claude tiers = 18 reference
fragments generated in `benchmark/reference-outputs/`. Run as-is (no vault frontmatter
modification); verified clean. Fixtures listed in `benchmark/dispatch-prompt.md`.

Authoritative detail lives in `docs/memory-system/05-implementation-process.md` §2c. Local-model gut-check done (2026-06-04): `gemma4:e4b` is the candidate; GPU fix pending reboot; Graphify owns the ollama call — see `docs/memory-system/benchmark/local-llm-findings-2026-06-04.md`.

#### 2d — Initial fixture graph build (ADR-013: small-first)
Run the initial build against the small fixture set (5–10 notes from Step 1/2c), not the
full vault. The bulk vault build is deferred to after the system is validated end-to-end.

```bash
graphify extract --path ~/Documents/SecondBrain --backend ollama --model gemma4:e2b \
  --token-budget 512 --max-concurrency 2
```
Tune `--token-budget` (semantic chunk size) and `--max-concurrency` based on VRAM headroom.
Review `GRAPH_REPORT.md` — check god nodes make sense (they should be your most-connected
tools, clients, and domain concepts).

**Full vault migration** (the `~/Documents/SecondBrain` build above run over all notes) is the final step —
deferred to after end-to-end validation on the pilot project. Do not bulk-migrate the vault
until the system is verified working on the fixture set and pilot project.

#### 2e — Per-project code graphs (free, no model needed)
**ADR-013 order:** start with ONE pilot project that contains both code and documents; validate
end-to-end before onboarding others. Onboard remaining projects one at a time, with
observe-and-adjust between each. Do not batch all projects at once.

For the pilot project (and each subsequent project, one at a time):
```bash
graphify extract --path ~/projects/<client>/<project> --no-docs
```
`--no-docs` skips LLM extraction; only AST is run. Zero token cost. Add `--update` on
subsequent runs; use `--force` when you've deleted files (to clear stale nodes).

Store each project's `graphify-out/` alongside the project, or in a configurable cache dir.
Keep per-project graphs separate — do not merge client projects into one graph.

### Step 3 — Hooks (maintenance + retrieval)

Hooks now call Graphify instead of the Ruby CLI.

- **PostToolUse** (on Write/Edit of vault `.md`): `graphify update --file <path>` to merge
  the changed note into the vault graph. Note: Graphify's `--update` does not prune deleted
  nodes — a periodic `--force` rebuild (e.g. weekly, or triggered by SessionStart stale
  check) is needed to clear ghost nodes.
- **SessionStart**: reconcile (check vault graph mtime vs last rebuild; trigger `--force` if
  stale threshold exceeded) + inject context (query MCP server for god nodes + inject
  `convention/*` note summaries + journal pointer).
- **SessionEnd**: append daily journal note with pointers to vault notes touched.

**Stale-node mitigation strategy:** track last `--force` rebuild time in a stamp file
(e.g. `~/.cache/graphify/vault-rebuild.stamp`). SessionStart hook triggers `--force` if
older than N days (7 is a reasonable starting point).

### Step 4 — Episodic layer (memsearch)
- Install memsearch (`/plugin marketplace add zilliztech/memsearch`, then `plugin install
  memsearch`), local to start. Verify daily memory files appear after a few conversations.
- Decide whether memsearch indexes our session-end journal notes or its own capture (likely its
  own; our journal can point into the knowledge vault).
- Graphify does **not** replace memsearch here. Time-anchored semantic queries ("what was I
  working on last Tuesday?") are better served by vector similarity over session logs.

### Step 5 — Sync
- Pick **git** (versioned, hourly push/pull) or **Syncthing** (continuous, zero-thought) for
  the vault → VPS. Configure on each machine.
- **Do not sync** the Graphify `graphify-out/` directories, Milvus caches, or the Ollama
  models. Graphs are rebuilt per machine from the vault (the single source of truth).

### Step 6 — Package as a global plugin (Part D)
- Wrap Steps 2–3 into a Claude Code plugin with skills; install at user level.

### Step 7 — (SKIPPED) QMD semantic layer
Covered by Graphify. The knowledge-graph approach provides structured semantic retrieval
without vectors. Only revisit if Graphify's graph queries prove insufficient for a use case
where embedding similarity would outperform graph traversal.

---

## Part B — The AI's write & query conventions (skills teach these)

### When the AI WRITES to the vault
Trigger writes when the AI learns something **evergreen and reusable across projects** — not
project-ephemeral state (that's the episodic layer / project files). Concretely:
- It worked out how a tool/API behaves (e.g. SEMrush auth, rate limits, an endpoint quirk).
- It established a convention, decision, or preference that should apply beyond this task.
- It discovered a client-specific fact worth reusing (how *this* client uses a tool).

**What conventions to follow when writing** (enforced by the frontmatter contract):
- One concept per note; keep notes small (the L1 "under ~200 lines / one topic" discipline).
- **Required frontmatter**: a one-line `summary` (written now, not deferred — this is the
  human-authored router hint; Graphify extracts entities but does not write summaries),
  and `scope/global` or `scope/project`.
- **Scope rule**: default new tool/domain knowledge to `scope/global`; mark `scope/project`
  when it's specific to how a client uses something.
- **Write only to the vault**, never silently into a project repo.
- **Promotion to global** and **consolidation** happen in the reorganize step (Part C), in
  plan mode, for human review.

### When & how the AI QUERIES

**Knowledge queries (vault graph)** via Graphify CLI (Bash tool):
- `graphify query "how do we authenticate with SEMrush?"` — semantic graph search
- `graphify path "client/sesame3g" "tool/semrush"` — how these two concepts connect
- `graphify explain "convention/seo-workflow"` — node deep-dive with neighbors
- Add `--budget N` to cap answer size (default ~2000 tokens); `--dfs --budget N` for bounded traversal
- Add `--graph <path>` to query a specific project's `graph.json` instead of the vault graph

**At session start**: god-node summary (injected by hook from `GRAPH_REPORT.md`) gives the
architecture spine — what are the highest-connectivity concepts in the vault. The AI then
queries specific nodes on demand rather than reading everything.

**On demand, during a task**: run `graphify query` when the task touches a tool/client/domain.
The graph returns relevant node clusters; the AI opens only the vault notes whose `summary`
matches the task. The `summary` frontmatter field remains the human-written progressive-disclosure hint.

**Cross-client lookups**: `graphify query "tool semrush"` returns all nodes connected to the
SEMrush concept, regardless of which client project they came from. Graph edges show provenance.

**"What happened" questions** go to **memsearch** in natural language — not the graph.

---

## Part C — Hooks & CRUD mapping

| Operation | Trigger | Mechanism |
|-----------|---------|-----------|
| **Create** | AI writes a new vault note | AI writes `.md` (Write tool) → **PostToolUse** hook → `graphify update --file` |
| **Read / query** | AI needs knowledge | AI runs `graphify query` / `graphify explain` via Bash (no hook; on-demand) |
| **Update** | AI/user edits a note | AI edit → **PostToolUse** → `graphify update --file`; user edit → **SessionStart** reconcile |
| **Delete / rename** | note removed/renamed | `graphify update --file` prunes AI-deleted paths; **SessionStart** triggers `--force` rebuild if stale (catches manual deletes) |
| **Inject** | new session starts | **SessionStart** hook: god-node summary + `convention/*` summaries + journal pointer |
| **Journal** | session ends | **SessionEnd** hook appends a dated journal note with pointers |
| **Reorganize** | periodic, user-invoked | `reorganize memory` in **plan mode**: dedupe, merge, split, re-scope, then `graphify --force` to rebuild clean graph — human approves |
| **Stale graph rebuild** | SessionStart stale check | If rebuild stamp > N days old: `graphify --force` on vault; resets stamp |

Hooks are thin shell wrappers; the logic lives in Graphify.

**Hooks summary:**
- **SessionStart** — stale check + `--force` if needed, then inject (god nodes + conventions + journal pointer).
- **PostToolUse** (on `Write`/`Edit` of vault `.md`) — `graphify update --file`.
- **SessionEnd** — append daily journal note.
- (memsearch brings its own hooks for the episodic layer.)

---

## Part D — Claude Code plugin with skills (global install)

Goal: one global install so every project/machine knows the vault, conventions, hooks, and
Graphify config.

**Plugin contents:**
- **Hooks** registered in settings: SessionStart, PostToolUse, SessionEnd (shell wrappers
  from Part C), pointed at a configurable vault path + Graphify output dir.
- **Graphify CLI** on `PATH` — the AI uses the Bash tool to run `graphify query`,
  `graphify path`, `graphify explain` directly. No server process. Project-specific graphs
  are queried with `--graph <path>` pointing at the project's `graphify-out/graph.json`.
- **Skills** (these carry the *know-how* to the model):
  - `memory-write` — when to record evergreen knowledge, the frontmatter contract (summary
    required, scope rule), "vault not repo."
  - `memory-query` — how/when to use `graphify query` vs memsearch; god-node discipline;
    `--budget`/`--dfs` usage; cross-client lookups via `--graph`; progressive-disclosure
    via summary field.
  - `memory-reorganize` — the plan-mode consolidation/promotion procedure + guardrails;
    when to trigger `graphify --force` rebuild.
- **Config**: vault path, Graphify output dir, Ollama model name, `num_ctx`, rebuild-stale
  threshold in days — set once at user level.
- **Env vars** set by plugin: `OLLAMA_FLASH_ATTENTION=1`, `GRAPHIFY_OLLAMA_NUM_CTX=8192`,
  `GRAPHIFY_OLLAMA_KEEP_ALIVE=5`.

**Why a plugin + skills (not just CLAUDE.md):** hooks must be registered by the harness;
conventions taught as skills load on demand without bloating every project's context. A single
global install keeps conventions a single source of truth.

**Convention notes placement:** coding `convention/*` notes live as **data in the vault**
(edited freely, resolved by SessionStart hook); memory-system **skills live in the plugin**
(behavior, versioned).

---

## Open questions

1. **Vault location** — Settled by ADR-012: `~/Documents/SecondBrain`. Symlink into
   `~/.claude/memory` only if a tool requires it.
2. **Sync mechanism** — git (versioned, hourly) vs Syncthing (continuous)?
3. **Stale rebuild threshold** — how many days before SessionStart triggers `--force`?
   7 days is the starting guess; tune after observing drift in practice.
4. **Per-project graph path** — how does the AI know which `graph.json` to pass to
   `--graph` for the current project? Convention: `<project-root>/graphify-out/graph.json`,
   injected by SessionStart hook if a project graph exists at that path.
5. **Summary field ownership** — Graphify extracts entities but does NOT write summaries.
   The human (or AI, on vault note creation) must write the `summary` frontmatter. Confirm
   this discipline holds in practice.
6. **Model benchmark results** — defer all model decisions until Step 2c benchmark is done.
   Don't hardcode `gemma4:e2b` until testing confirms it's the right pick.
7. **memsearch + journal** — does memsearch index our SessionEnd journal notes, or only
   its own auto-capture, and how does the journal point into the knowledge vault?