cc-os/docs/memory-system/04-build-plan.md

# Build Plan

_Last updated: 2026-07-02_

How a human builds this system, step by step, and answers to the operational questions:
which scripts and hooks, how the AI knows when to write and what conventions to follow, how and
when it queries, the CRUD hooks, and how it's packaged as a global Claude Code plugin with
skills.

> This is a build outline, not the implementation plan. The next session should turn this into
> a proper implementation plan (writing-plans skill) and execute it.

**Architecture decision (2026-06-03):** Graphify replaces the Ruby/SQLite tag index.
The Ruby CLI (old Step 2) is skipped. Graphify handles both knowledge-graph queries over the
vault and free AST-based code graphs per project. The deferred QMD semantic layer (old Step 7)
is also skipped — Graphify covers it without vectors. See `06-graphify-evaluation.md` for
the full rationale.

**Architecture decision (2026-06-04):** Build order inverted (ADR-013). Build and validate
the full system against a small 5–10 note fixture set first; defer bulk vault migration to last.
Validate end-to-end on one small pilot project containing both code and documents before
onboarding any others. Onboard remaining projects one at a time, with observe-and-adjust between
each. Steps 2d and 2e below are updated accordingly.

---

## Part A — Build order (human builder's path)

Build bottom-up: the vault and Graphify first (usable standalone), then the hooks, then the
plugin that packages it.

### Step 1 — Vault skeleton & conventions
- Vault location is settled by ADR-012: `~/Documents/SecondBrain` (the existing Obsidian
  vault). No new vault is created; symlink into `~/.claude/memory` only if a tool insists.
  **Vault is the single source of truth.**
- Write `CONVENTIONS.md` in the vault: the frontmatter contract. The required fields are
  `summary` (one-line, author-written — this is the human-readable router hint Graphify
  does not generate) and `scope` (`scope/global` or `scope/project`). Tags are now
  supplementary, not the primary query mechanism.
  **Naming resolved (2026-06-09):** `vault-conventions.md` is the canonical name (not
  `CONVENTIONS.md`). The file exists at `~/Documents/SecondBrain/vault-conventions.md`.
- Seed a few real notes (e.g. `tool/semrush`, `client/<x>`) to use as extraction test cases
  in Step 2.

**Status: COMPLETE (2026-06-09)**
- `vault-conventions.md` (renamed 2026-06-09 from `CONVENTIONS.md`) exists in the vault at
  `~/Documents/SecondBrain/vault-conventions.md` — frontmatter contract + tag taxonomy.
- All 6 fixture notes exist in the vault (seeded for extraction testing in Step 2c).
- 14 Graphify handbook + memory-system design notes migrated to the vault (2026-06-09) as
  part of migration scaffolding — first-class deliverable per the empirical finding locked
  2026-06-05.

### Step 2 — Graphify + Ollama setup (the knowledge graph layer)

This replaces the Ruby tag-index CLI. Graphify builds a knowledge graph over vault docs via a
local SLM, and a free code graph over project files via AST.

#### 2a — Install Graphify **DONE (2026-06-08)**
```
pip install graphify   # or follow current install docs
```
Verify: `graphify --version`

**Status:** `graphify` v0.8.31 installed at `/home/jared/.local/bin/graphify`.

#### 2b — Configure Ollama for extraction **DONE (2026-06-08)**
The 2024 approach (Modelfile with `PARAMETER num_ctx`) still works but the preferred
programmatic method is now per-request via the API (Graphify supports this via
`GRAPHIFY_OLLAMA_NUM_CTX`):

```bash
# Set in your shell profile or service env:
OLLAMA_FLASH_ATTENTION=1          # 30-50% VRAM savings on KV cache — always set this
GRAPHIFY_OLLAMA_NUM_CTX=8192      # 8K is sufficient for vault notes (200-2000 words)
                                  # and leaves comfortable headroom for prompt overhead
```

To verify context is being used: `ollama ps` shows allocated context after first call.

**Status:** All env vars set in `~/.claude/plugins/os-vault/config.yaml`:
`OLLAMA_FLASH_ATTENTION=1`, `GRAPHIFY_OLLAMA_NUM_CTX=8192`, `GRAPHIFY_OLLAMA_KEEP_ALIVE=5`.
Model locked to `qwen25-coder-7b-16k` (per ADR and memory note — this is Graphify's default
and the confirmed working extraction model).

Available models (as of 2026-06-03, in order of interest):
- `gemma4:e2b` — 7.2 GB, ~2B effective params, 128K native window, fast; **start here**
- `qwen3.5:2b` — 2.7 GB, smallest, good fallback if VRAM is constrained
- `gemma4:e4b` — 9.6 GB, more capable, slower

#### 2c — Claude reference-set benchmark (THE GATE, before committing to a local model)

This step produces the **gold-standard reference set** — one structured extraction output per
fixture note per Claude tier. It does **not** choose the final extraction model, and it does
**not** measure speed.

**Input:** the 5–10 fixture notes selected in Step 1 (from the runbook).

**What to run:** dispatch one Claude Code subagent per tier (Claude-tiers only — Ollama models
are not reachable in this environment):

| Tier | Model ID | Role |
|---|---|---|
| Haiku | `claude-haiku-4-5` | Lightweight reference |
| Sonnet | `claude-sonnet-4-6` | Mid-tier reference |
| Opus | `claude-opus-4-8` | **Gold standard** (scoring rubric) |

Each subagent receives only the fixture note text plus the shared extraction spec — no design
docs, no project context (fairness contract). See ADR-013.

**Metrics — quality only (wall-clock speed is explicitly out of scope here):**
1. Entity correctness — right concepts extracted, no hallucinated entities
2. Relationship plausibility and typing — edges plausible, correctly typed, no missing key edges
3. Confidence-tag accuracy — `INFERRED` vs `AMBIGUOUS` applied appropriately

**Deliverables** (produced by this step):
- Dispatch prompt (copy/paste-able): `docs/memory-system/benchmark/dispatch-prompt.md`
- Shared extraction schema: `docs/memory-system/benchmark/extraction-spec.md`
- Per-tier outputs: `docs/memory-system/benchmark/reference-outputs/<note-slug>.<tier>.md`

Opus output is the rubric. **Deferred later step:** local Ollama models (gemma4:e2b, qwen3.5:2b,
gemma4:e4b) are timed AND scored against these references — that scoring run is where speed
re-enters and the final model is chosen. Do not hardcode a model before that run completes.

**Status (2026-06-04): EXECUTED.** 6 cross-domain fixtures × 3 Claude tiers = 18 reference
fragments generated in `benchmark/reference-outputs/`. Run as-is (no vault frontmatter
modification); verified clean. Fixtures listed in `benchmark/dispatch-prompt.md`.

Authoritative detail lives in `docs/memory-system/05-implementation-process.md` §2c. Local-model gut-check done (2026-06-04): `gemma4:e4b` is the candidate; GPU fix pending reboot; Graphify owns the ollama call — see `docs/memory-system/benchmark/local-llm-findings-2026-06-04.md`.

#### 2d — Initial fixture graph build (ADR-013: small-first)
Run the initial build against the small fixture set (5–10 notes from Step 1/2c), not the
full vault. The bulk vault build is deferred to after the system is validated end-to-end.

```bash
graphify extract --path ~/Documents/SecondBrain --backend ollama --model gemma4:e2b \
  --token-budget 512 --max-concurrency 2
```
Tune `--token-budget` (semantic chunk size) and `--max-concurrency` based on VRAM headroom.
Review `GRAPH_REPORT.md` — check god nodes make sense (they should be your most-connected
tools, clients, and domain concepts).

**Note (2026-06-08):** The `~/Documents/SecondBrain` vault already has a live `graphify-out/`
graph (32 files, built 2026-06-05). The fixture-first approach from ADR-013 may therefore be
superseded — the vault graph exists before Step 1 (conventions/fixtures) is formally complete.

**Resolved (2026-06-15): DONE/superseded.** The live vault `graphify-out/` over the full vault
IS the baseline graph. A fixture-only build is moot now that the vault holds real scaffolding
notes (not just fixtures). No separate fixture-only build will be run.

**Full vault migration** (the `~/Documents/SecondBrain` build above run over all notes) is the final step —
deferred to after end-to-end validation on the pilot project. Do not bulk-migrate the vault
until the system is verified working on the fixture set and pilot project.

#### 2e — Per-project code graphs (free, no model needed)
**ADR-013 order:** start with ONE pilot project that contains both code and documents; validate
end-to-end before onboarding others. Onboard remaining projects one at a time, with
observe-and-adjust between each. Do not batch all projects at once.

For the pilot project (and each subsequent project, one at a time):
```bash
graphify extract --path ~/projects/<client>/<project> --no-docs
```
`--no-docs` skips LLM extraction; only AST is run. Zero token cost. Add `--update` on
subsequent runs; use `--force` when you've deleted files (to clear stale nodes).

Store each project's `graphify-out/` alongside the project, or in a configurable cache dir.
Keep per-project graphs separate — do not merge client projects into one graph.

**Note (2026-06-08):** A `graphify-out/` for the `cc-os` project exists (built 2026-06-05),
but `cc-os` is a docs-only repo, not a pilot project in the ADR-013 sense. A pilot project
with real code has not yet been onboarded.

**Status: PILOT DONE (2026-06-15).** The llf-schema project (`/home/jared/dev/llf-schema`,
PHP 8.2 WordPress plugin) was onboarded via the `/os-vault:onboard-project` skill: `graphify extract`
(tree-sitter AST on 90 code files + Ollama doc pass on 9 doc files, ~3.5 min) + `cluster-only`
→ 605 nodes / 930 edges / 52 communities (~524K `graph.json`). `graphify-out/` added to that
repo's `.gitignore`. **Finding:** Ollama doc pass was lossy on structured WordPress docs (101
warnings, recursive bisection) but the AST pass carried the code structure — acceptable for
code-heavy repos. **Remaining under 2e:** onboard additional projects incrementally.

### Step 3 — Hooks (maintenance + retrieval) **DONE (2026-06-08)**

Hooks now call Graphify instead of the Ruby CLI.

- **PostToolUse** (on Write/Edit of vault `.md`): `graphify update --file <path>` to merge
  the changed note into the vault graph. Note: Graphify's `--update` does not prune deleted
  nodes — a periodic `--force` rebuild (e.g. weekly, or triggered by SessionStart stale
  check) is needed to clear ghost nodes.
- **SessionStart**: reconcile (check vault graph mtime vs last rebuild; trigger `--force` if
  stale threshold exceeded) + inject context (emits project graph path if a
  `graphify-out/graph.json` exists at the project root; vault context injection was removed).
- **SessionEnd**: append daily journal note with pointers to vault notes touched.

**Stale-node mitigation strategy:** track last `--force` rebuild time in a stamp file
(e.g. `~/.cache/graphify/vault-rebuild.stamp`). SessionStart hook triggers `--force` if
older than N days (7 is a reasonable starting point).

### Step 4 — Episodic layer (memsearch) — DONE
- Install memsearch (`/plugin marketplace add zilliztech/memsearch`, then `plugin install
  memsearch`), local to start. Verify daily memory files appear after a few conversations.
- Decide whether memsearch indexes our session-end journal notes or its own capture (likely its
  own; our journal can point into the knowledge vault).
- Graphify does **not** replace memsearch here. Time-anchored semantic queries ("what was I
  working on last Tuesday?") are better served by vector similarity over session logs.

### Step 5 — Sync — PARTIAL

Step 5 covers two distinct sync lines; they are at different statuses:

#### 5a — memsearch episodic memory git sync — **DONE (2026-06-09)**
`~/.memsearch` is now a git repo on branch `main` with remote `origin` pointing to a private
self-hosted Forgejo repo (`ssh://git@forgejo.swansoncloud.com:2222/jared/memsearch.git`;
web: `https://forgejo.swansoncloud.com/jared/memsearch`). Initial commit `106cebc`.

A whitelist `.gitignore` tracks **only** `memory/*.md` (the daily session files) and
`.gitignore`. Excluded as derived/disposable: `milvus.db` (Milvus Lite index — rebuildable via
`memsearch index`), `config.toml`, and the bge-m3 ONNX model (lives in `~/.cache/huggingface`).

Auto-commit and push are wired into the cc-os memory plugin's **own `memsearch_sync.py`** SessionEnd hook (split from `session_end.py` by ADR-016, 2026-06-12; behavior preserved — see ADR-015)
(not the marketplace plugin, which can be clobbered on update). The block guards on
`~/.memsearch/.git` existing, commits only when staged content exists (message:
`memsearch: session memory <date>`), and pushes with `timeout 30 ... || true`. Wrapped in a
subshell with trailing `|| true` — cannot fail session shutdown. See **ADR-015** for rationale
and the cross-client commingling, which is an accepted design choice (single global store
across all clients by intent — see ADR-015, resolved 2026-06-09).

#### 5b — Obsidian vault → VPS sync — **DONE (2026-06-15)**
The vault at `~/Documents/SecondBrain` was initialized as a git repo and pushed to a new
private Forgejo repo (`ssh://git@forgejo.swansoncloud.com:2222/jared/SecondBrain.git`;
web: `https://forgejo.swansoncloud.com/jared/SecondBrain`), mirroring the memsearch sync
pattern. 52 files committed on 2026-06-15 (notes, journal, templates, vault-conventions.md,
CLAUDE.md, .obsidian config); `graphify-out/` correctly excluded via the existing `.gitignore`.
**Decision:** git sync chosen over Syncthing.

**Auto-sync hook — DONE (2026-06-15):** `vault_sync.py` added as the third SessionEnd hook
(after `session_end.py` and `memsearch_sync.py`, timeout 30). Auto-commits `git add -A` and
pushes the vault to `forgejo.swansoncloud.com/jared/SecondBrain` on every session end, so the
daily journal note from `session_end.py` is included in the commit. Mirrors the
`memsearch_sync.py` pattern; uses `cfg.vault_path`. Smoke-tested (exit 0) 2026-06-15.

**Multi-machine freshness — DONE (2026-07-10):** `session_start.py` now runs a fast-forward-only
`git pull` (`timeout 10`, `--ff-only --quiet`) against both `cfg.vault_path` and
`cfg.memsearch_dir` before the staleness check. Silent on success; a quiet one-line stderr note
on failure (offline machine, diverged history) that never blocks or delays session start.

**Do not sync** the Graphify `graphify-out/` directories, Milvus caches, or the Ollama
models. Graphs are rebuilt per machine from the vault (the single source of truth).

### Step 6 — Package as a global plugin (Part D) **DONE (2026-06-08)**
- Wrap Steps 2–3 into a Claude Code plugin with skills; install at user level.

**Status:** Global plugin installed at `~/.claude/plugins/os-vault/` with all hook scripts and
5 skills (`query`, `write`, `reorganize`, `onboard-project`, `design-template`). Plugin enabled in
`~/.claude/settings.json`.

**Source-in-git + bash→Python port — DONE (2026-06-12):** Plugin source moved into git at
`cc-os/plugins/os-vault/` (OpenSpec change `memory-plugin-source-and-port`). Bash hooks ported to
Python using a deep-module architecture (shared `config.py`, `hook_io.py`, `session_state.py`;
thin entry-point scripts under `hooks/`). memsearch sync split into `memsearch_sync.py` as a
dedicated second SessionEnd hook (ADR-015 behavior relocated, not reversed). Cutover done via
symlink `~/.claude/plugins/os-vault → cc-os/plugins/os-vault/` and settings.json hook rewrite to
Python absolute paths. Fresh-session test passed 2026-06-12. See ADR-016.

### Step 7 — (SKIPPED) QMD semantic layer
Covered by Graphify. The knowledge-graph approach provides structured semantic retrieval
without vectors. Only revisit if Graphify's graph queries prove insufficient for a use case
where embedding similarity would outperform graph traversal.

---

## Part B — The AI's write & query conventions (skills teach these)

### When the AI WRITES to the vault
Trigger writes when the AI learns something **evergreen and reusable across projects** — not
project-ephemeral state (that's the episodic layer / project files). Concretely:
- It worked out how a tool/API behaves (e.g. SEMrush auth, rate limits, an endpoint quirk).
- It established a convention, decision, or preference that should apply beyond this task.
- It discovered a client-specific fact worth reusing (how *this* client uses a tool).

**What conventions to follow when writing** (enforced by the frontmatter contract):
- One concept per note; keep notes small (the L1 "under ~200 lines / one topic" discipline).
- **Required frontmatter**: a one-line `summary` (written now, not deferred — this is the
  human-authored router hint; Graphify extracts entities but does not write summaries),
  and `scope/global` or `scope/project`.
- **Scope rule**: default new tool/domain knowledge to `scope/global`; mark `scope/project`
  when it's specific to how a client uses something.
- **Write only to the vault**, never silently into a project repo.
- **Promotion to global** and **consolidation** happen in the reorganize step (Part C), in
  plan mode, for human review.

### When & how the AI QUERIES

**Knowledge queries (vault graph)** via Graphify CLI (Bash tool):
- `graphify query "how do we authenticate with SEMrush?"` — semantic graph search
- `graphify path "client/sesame3g" "tool/semrush"` — how these two concepts connect
- `graphify explain "convention/seo-workflow"` — node deep-dive with neighbors
- Add `--budget N` to cap answer size (default ~2000 tokens); `--dfs --budget N` for bounded traversal
- Add `--graph <path>` to query a specific project's `graph.json` instead of the vault graph

**At session start**: the session_context.py hook emits the project graph path (pointing at
`<project-root>/graphify-out/graph.json`) if one exists — vault context injection (god-node
summary, convention summaries, journal pointer) was removed. The AI queries the vault or
project graph on demand rather than having it injected.

**On demand, during a task**: run `graphify query` when the task touches a tool/client/domain.
The graph returns relevant node clusters; the AI opens only the vault notes whose `summary`
matches the task. The `summary` frontmatter field remains the human-written progressive-disclosure hint.

**Cross-client lookups**: `graphify query "tool semrush"` returns all nodes connected to the
SEMrush concept, regardless of which client project they came from. Graph edges show provenance.

**"What happened" questions** go to **memsearch** in natural language — not the graph.

---

## Part C — Hooks & CRUD mapping

| Operation | Trigger | Mechanism |
|-----------|---------|-----------|
| **Create** | AI writes a new vault note | AI writes `.md` (Write tool) → **PostToolUse** hook → `graphify update --file` |
| **Read / query** | AI needs knowledge | AI runs `graphify query` / `graphify explain` via Bash (no hook; on-demand) |
| **Update** | AI/user edits a note | AI edit → **PostToolUse** → `graphify update --file`; user edit → **SessionStart** reconcile |
| **Delete / rename** | note removed/renamed | `graphify update --file` prunes AI-deleted paths; **SessionStart** triggers `--force` rebuild if stale (catches manual deletes) |
| **Inject** | new session starts | **SessionStart** hook: emits project graph path (vault context injection removed) |
| **Journal** | session ends | **SessionEnd** hook appends a dated journal note with pointers |
| **Reorganize** | periodic, user-invoked | `reorganize memory` in **plan mode**: dedupe, merge, split, re-scope, then `graphify --force` to rebuild clean graph — human approves |
| **Stale graph rebuild** | SessionStart stale check | If rebuild stamp > N days old: `graphify --force` on vault; resets stamp |

Hooks are thin shell wrappers; the logic lives in Graphify.

**Hooks summary:**
- **SessionStart** — stale check + `--force` if needed, then inject project graph path (vault context injection was removed; session_context.py now only emits the project graph path).
- **PostToolUse** (on `Write`/`Edit` of vault `.md`) — `graphify update --file`.
- **SessionEnd** — append daily journal note.
- (memsearch brings its own hooks for the episodic layer.)

**Status:** All 4 hooks wired in `~/.claude/settings.json`: SessionStart, UserPromptSubmit
(context injection), PostToolUse (vault writes), SessionEnd (journal).

---

## Part D — Claude Code plugin with skills (global install)

Goal: one global install so every project/machine knows the vault, conventions, hooks, and
Graphify config.

**Plugin contents:**
- **Hooks** registered in settings: SessionStart, PostToolUse, SessionEnd (shell wrappers
  from Part C), pointed at a configurable vault path + Graphify output dir.
- **Graphify CLI** on `PATH` — the AI uses the Bash tool to run `graphify query`,
  `graphify path`, `graphify explain` directly. No server process. Project-specific graphs
  are queried with `--graph <path>` pointing at the project's `graphify-out/graph.json`.
- **Skills** (these carry the *know-how* to the model):
  - `/os-vault:write` — when to record evergreen knowledge, the frontmatter contract (summary
    required, scope rule), "vault not repo."
  - `/os-vault:query` — how/when to use `graphify query` vs memsearch; god-node discipline;
    `--budget`/`--dfs` usage; cross-client lookups via `--graph`; progressive-disclosure
    via summary field.
  - `/os-vault:reorganize` — the plan-mode consolidation/promotion procedure + guardrails;
    when to trigger `graphify --force` rebuild.
  - `/os-vault:onboard-project` **DONE (2026-06-08)** — full lifecycle skill for per-project Graphify graphs:
    - **Onboard**: `graphify extract . --backend ollama --model qwen2.5-coder:7b`, then `graphify cluster-only .`, add `graphify-out/` to `.gitignore`
    - **Onboard flow (updated 2026-06-17, ADR-017)**: Assessment-first — the skill surveys the repo and generates a per-project `.graphifyignore` for user confirmation before running extract, because Graphify does not honor `.gitignore`.
    - **Query**: `graphify query ... --graph <path>` for project-context queries (wraps `/os-vault:query`)
    - **Update**: `graphify update .` for incremental updates (no API cost) or `--force` rebuild after structural changes
    - **Remove**: delete `graphify-out/` when no longer needed
- **Config**: vault path, Graphify output dir, Ollama model name, `num_ctx`, rebuild-stale
  threshold in days — set once at user level.
- **Env vars** set by plugin: `OLLAMA_FLASH_ATTENTION=1`, `GRAPHIFY_OLLAMA_NUM_CTX=8192`,
  `GRAPHIFY_OLLAMA_KEEP_ALIVE=5`.

**Why a plugin + skills (not just CLAUDE.md):** hooks must be registered by the harness;
conventions taught as skills load on demand without bloating every project's context. A single
global install keeps conventions a single source of truth.

**Convention notes placement:** coding `convention/*` notes live as **data in the vault**
(edited freely, resolved by SessionStart hook); memory-system **skills live in the plugin**
(behavior, versioned).

---

## Open questions

1. **Vault location** — Settled by ADR-012: `~/Documents/SecondBrain`. Symlink into
   `~/.claude/memory` only if a tool requires it.
2. **Sync mechanism** — ~~git (versioned, hourly) vs Syncthing (continuous)?~~ **RESOLVED (2026-06-15): git.**
3. **Stale rebuild threshold** — how many days before SessionStart triggers `--force`?
   7 days is the starting guess; tune after observing drift in practice.
4. **Per-project graph path** — how does the AI know which `graph.json` to pass to
   `--graph` for the current project? Convention: `<project-root>/graphify-out/graph.json`,
   injected by SessionStart hook if a project graph exists at that path.
5. **Summary field ownership** — Graphify extracts entities but does NOT write summaries.
   The human (or AI, on vault note creation) must write the `summary` frontmatter. Confirm
   this discipline holds in practice.
6. **Model benchmark results** — defer all model decisions until Step 2c benchmark is done.
   Don't hardcode `gemma4:e2b` until testing confirms it's the right pick.
7. **memsearch + journal** — does memsearch index our SessionEnd journal notes, or only
   its own auto-capture, and how does the journal point into the knowledge vault?