cc-os/docs/memory-system/05-implementation-process.md

# Implementation Process

_Last updated: 2026-07-02_ | _Status: Steps 2a/2b/2d executed — toolchain installed, context fix applied, vault graph built; model selected (qwen2.5-coder:7b); Step 2c (reference set) previously complete. Step 2 fully executed._

This document distills `04-build-plan.md` into a concrete, staged build process and folds in
two locked decisions: **ADR-011** (faceted six-namespace taxonomy) and **ADR-012** (reuse the
existing `~/Documents/SecondBrain` vault rather than creating a new one). Read `04-build-plan.md`
for the underlying rationale, query/CRUD conventions, and plugin internals; read
`03-architecture-decisions.md` for the decision log. This document is the runbook.

> This is still a build process outline, not a detailed implementation plan. **Step 2 is fully
> executed as of 2026-06-05**: toolchain installed (2a), context-baking approach confirmed (2b),
> reference set generated (2c), vault graph built (2d). Selected model: **qwen2.5-coder:7b**
> (Modelfile variant `qwen25-coder-7b-16k`, num_ctx 16384). The next phase is Step 3 (Hooks).
> Most open questions in Steps 3–6 can be defaulted at build time.

---

## Build order

Bottom-up: fixture selection first (feeds the Step 2c gate), then Graphify setup, then hooks, then plugin. Bulk vault migration is deferred to after the system is validated end-to-end.

---

## Step 1 — Select benchmark fixtures (and defer bulk migration)

**Per ADR-012:** `~/Documents/SecondBrain` is the vault. The vault is already flat, already
scoped to durable knowledge, and already articulates the correct governance philosophy.

**Bulk vault migration is DEFERRED.** Do not migrate all notes now. The only pre-build activity
in this step is selecting a small fixture set that feeds the Step 2c benchmark gate. Everything
else — tagging ~20 existing notes, updating governance docs, initializing git, fixing broken
references — happens after the system is validated end-to-end.

> **Four sequencing concepts to keep distinct — they are not interchangeable:**
>
> 1. **Benchmark fixtures** — 5–10 notes selected NOW, in this step. They feed the Step 2c
>    extraction benchmark gate. They are living fixtures, kept and reused throughout development.
> 2. **Bulk vault migration** — converting all ~20 existing notes to the six-facet tag schema,
>    updating governance docs, initializing git. DEFERRED to last, after the system is fully
>    validated.
> 3. **Initial validation project** — the first real end-to-end test after the system is built:
>    one small project that contains BOTH code AND documents. POST-BUILD, before any rollout.
> 4. **Project-by-project rollout** — onboarding remaining projects one at a time, observing
>    and adjusting between each. Follows the initial validation project.

### 1a — Select fixture notes

**Status (2026-06-04): DONE.** 6 cross-domain fixtures selected and listed in
`benchmark/dispatch-prompt.md`. Run as-is (no vault frontmatter modification per the
methodology decision — fixtures used exactly as they exist in the vault).

Choose 5–10 notes from `~/Documents/SecondBrain` with deliberate variety. The selection must
include at minimum:

- A **tool note** (e.g., a note about a CLI tool or SaaS product)
- A **client/project note** (scoped, not global)
- A **convention note** (a `type/convention` or workflow note)
- A **domain note** (a topic or subject-area note)
- At least one **relationship-dense note** — a note where several concepts interrelate and
  Graphify should emit multiple typed edges and confidence tags

These notes must be usable as-is by a subagent that receives only the note text plus the shared
extraction spec (see Step 2c). Apply `summary:` frontmatter and the six-facet tag schema to
each fixture note now; do not wait for bulk migration.

**Frontmatter contract** (required for fixture notes immediately; apply to all notes at bulk
migration time):

```yaml
---
summary: "One-line, human-written router hint (required; Graphify does not generate this)"
tags:
  - type/<kind>        # listed first by convention
  - client/<name>      # as applicable
  - project/<name>     # as applicable — first-class for a freelancer
  - domain/<topic>     # as applicable
  - tool/<name>        # as applicable
  - convention/<name>  # as applicable
  - scope/global       # or scope/project
---
```

`type/` listed first preserves the vault's existing type-first ordering habit and makes note
kind immediately visible.

### 1b — Deferred bulk migration work (do not start until post-validation)

The following items are deferred to after the system is validated end-to-end on fixtures and
on the initial validation project (see §Post-build sequence below):

- **Initialize git:** `git init ~/Documents/SecondBrain` (enables ADR-008 sync) — before the first commit, add `graphify-out/` to the vault's `.gitignore`: `~/Documents/SecondBrain/graphify-out/cache/` (AST/tree-sitter parse cache) currently sits inside the vault and must stay untracked (ADR-008 — indexes are disposable/excluded from version control)
- **Update vault governance notes:** `CLAUDE.md` (vault path refs, Graphify commands,
  ADR-011 namespace list), `vault-conventions.md` (six-facet tag list; preserve the
  "act without being asked" proactive-retrieval section), and project-config hub note(s)
  (update tag-inference table to namespaced form, e.g. `semrush-work → tool/semrush`)
- **Migrate remaining notes (~20):** add `summary:` frontmatter and convert unnamespaced tags
  to six-facet form (e.g. `research → type/research`, `pest-control → domain/pest-control`,
  `semrush → tool/semrush`). Add `scope/global` or `scope/project` to each note.
- **Create `_templates` for core note types:** `research`, `howto`, `adr`, `hub`. The long
  tail stays freeform until a pattern earns a template (per ADR-011).
- **Fix the broken vault search reference:** `vault-conventions.md` (or `CLAUDE.md`) currently
  references `~/.claude/scripts/vault_search.rb`, which does not exist. Replace every reference
  with `graphify query` (per ADR-010).

### Post-build sequence (after Steps 2–6 are complete)

After the system is built and validated on fixtures:

**First:** Validate end-to-end on **one small project that contains BOTH code AND documents.**
This project is the initial validation target — not a production migration. It exercises both
extraction paths simultaneously: vault notes use the local-SLM extraction path; project code
uses the tree-sitter AST path. These are different extraction paths, which is precisely why the
first validation project must contain both. Observe how the two paths compose before proceeding.

**Then:** Onboard remaining projects **one at a time.** Observe extraction quality, god-node
shape, and hook behavior between each project before onboarding the next. Adjust configuration
as needed.

**Finally:** Execute bulk vault migration (items in 1b above) and run full vault graph build.

**Migration-unit granularity open question:** the right unit — a whole project repo, a
vault-note cluster, a client boundary — is not yet established. The first validation project
("one small project with both code and documents") is the anchor; refine the granularity
definition after observing that first real migration.

---

## Step 2 — Graphify + Ollama setup (CRITICAL PATH)

This step is the genuine gate. All hook behavior, model selection, and extraction tuning depend
on the benchmark in 2c. Do not skip or defer 2c.

### 2a — Install and verify Graphify

**Status (2026-06-05): EXECUTED.** graphify 0.8.31 installed at `~/.local/bin/graphify`; `graphify --version` → `graphify 0.8.31`. ollama 0.30.3, systemd service on 127.0.0.1:11434; GPU RTX 3060 12GB VRAM confirmed; all extraction ran 100% on GPU (no CPU spill per `ollama ps`). [verified: local shell, 2026-06-05]

The PyPI package name has a double-y — this is the correct install command:

```bash
pip install graphifyy
```

Verify: `graphify --version`

### 2b — Configure Ollama

**Status (2026-06-05): EXECUTED — but the env-var approach was superseded.** `GRAPHIFY_OLLAMA_NUM_CTX` does NOT propagate through graphify's ollama `/v1/chat/completions` endpoint: ollama's OpenAI-compatible `/v1` endpoint silently ignores per-request `options.num_ctx`, pinning context at the model's baked default (4096 by default). At 4096, graphify's extraction output is truncated mid-response and the chunk is discarded — producing an empty graph. [verified: local A/B test, 2026-06-05]

**SUPERSEDED APPROACH — do not use `GRAPHIFY_OLLAMA_NUM_CTX`:**

```bash
# This env var has NO EFFECT through graphify's /v1 path — do not set
# GRAPHIFY_OLLAMA_NUM_CTX=8192
```

**CORRECT APPROACH — bake context into a Modelfile variant:**

```bash
# Create a Modelfile variant with the desired context baked in:
# FROM <model>
# PARAMETER num_ctx 16384
# ollama create <name>-16k -f Modelfile
```

The `/v1` endpoint DOES honor the model's baked default. Verified `ollama ps` CONTEXT=16384, 100% GPU. This is the same lever the vault build (Step 2d) and plugin (Step 6) must use. Non-invasive: no sudo, no systemd restart, no shell-profile edit; reversible via `ollama rm`.

Still set in shell profile (these remain valid):

```bash
OLLAMA_FLASH_ATTENTION=1        # 30–50% VRAM savings on KV cache — always set
GRAPHIFY_OLLAMA_KEEP_ALIVE=5    # set when packaging in Step 6
```

Two additional operational requirements confirmed: `OLLAMA_BASE_URL` must end in `/v1` (e.g., `http://127.0.0.1:11434/v1`) or every graphify call 404s; `OLLAMA_API_KEY` must be set to any non-empty value unless the host is loopback.

Verify context allocation after the first extraction call: `ollama ps` shows allocated context.

### 2c — Claude reference-set benchmark (THE GATE)

**Status (2026-06-04): EXECUTED (prior round).** 6 cross-domain fixtures × 3 Claude tiers = 18 reference fragments generated in `benchmark/reference-outputs/`. Run as-is (no vault frontmatter modification); verified clean. Gate is passed — Ollama model scoring is unblocked. Full local-model scoring (task 3.4) and model selection (task 3.6) are also complete — see `benchmark/scoring-results-2026-06-04.md`.

Produce a reference set of Graphify-shaped extraction outputs before committing to any local
Ollama model. Use the 5–10 fixture notes selected in Step 1a as the input set.

**What this step produces — and what it does NOT decide:**

This step produces the **gold-standard reference set**: one structured extraction output per
fixture note per Claude tier. The reference set is the scoring rubric against which local Ollama
models are scored in a later step. This step does NOT choose the final extraction model —
local-model selection happens when Ollama models are timed and scored against these references.

**Why Claude-only here:** local Ollama models cannot be run in this benchmarking environment.
Only Claude tiers are reachable via dispatched subagents. The Claude outputs serve as quality
anchors; Ollama speed and quality are measured separately, against these anchors.

**Dispatch one Claude Code subagent per tier:**

| Tier | Model ID | Role |
|---|---|---|
| Haiku | `claude-haiku-4-5` | Lightweight reference |
| Sonnet | `claude-sonnet-4-6` | Mid-tier reference |
| Opus | `claude-opus-4-8` | **Gold standard** |

Each subagent receives only the fixture note text plus the shared extraction spec at
`docs/memory-system/benchmark/extraction-spec.md`. It must not read `CLAUDE.md`, design docs,
or pull any project context — each subagent operates with the note and spec alone (fairness
contract).

Each subagent emits a Graphify-shaped structured fragment containing:
- Extracted entities (named concepts, tools, people, projects)
- Typed relationships between entities
- Confidence tags: `INFERRED` (inferred but not stated) and `AMBIGUOUS` (could be interpreted
  multiple ways) applied where appropriate

**Metrics — quality only (wall-clock speed is explicitly out of scope here):**

Wall-clock timing is untrackable across dispatched subagents and is not measured in this step.
Speed re-enters the picture in the later Ollama-scoring step, where local models are timed
against these reference outputs. For this step, measure quality only:

1. **Entity correctness** — right concepts extracted, no hallucinated entities
2. **Relationship plausibility and typing** — edges plausible, correctly typed, no missing key edges
3. **Confidence-tag accuracy** — `INFERRED` vs `AMBIGUOUS` applied appropriately

**Deliverable files:**

- Dispatch prompt (copy/paste-able): `docs/memory-system/benchmark/dispatch-prompt.md`
- Shared extraction schema: `docs/memory-system/benchmark/extraction-spec.md`
- Per-model outputs: `docs/memory-system/benchmark/reference-outputs/<note-slug>.<tier>.md`
  where `tier` is one of `haiku`, `sonnet`, or `opus`

`claude-opus-4-8` output is the gold standard. When Ollama models are benchmarked later, their
outputs are scored by how closely they match the Opus reference for each fixture note.

### 2d — Build the initial vault graph

**Status (2026-06-05): EXECUTED.** Full vault graph built with qwen2.5-coder:7b (Modelfile variant `qwen25-coder-7b-16k`, num_ctx 16384). Result: **57 nodes / 43 edges / 15 communities**. God-nodes are plausible: Speed-to-Lead is the dominant hub, consistent with the vault's current content emphasis. [verified: local shell, 2026-06-05]

Command used (deviates from the template below — actual values recorded for reference):

```bash
graphify extract ~/Documents/SecondBrain \
  --backend ollama --model qwen25-coder-7b-16k \
  --max-concurrency 1 --token-budget 4000 \
  --exclude .obsidian --out /tmp/graphify-bench/vault-graph
```

Note: `--max-concurrency 1` (not 2 — untested at 2 on 12GB GPU); `--token-budget 4000` (not 512 — fits the 16k context comfortably). Template for future runs:

```bash
graphify extract --path ~/Documents/SecondBrain \
  --backend ollama --model qwen25-coder-7b-16k \
  --token-budget 4000 --max-concurrency 1 --exclude .obsidian
```

Review `GRAPH_REPORT.md`. God nodes should be the most-connected tools, clients, and domain
concepts — confirm they make sense given the vault's content.

### 2e — Per-project code graphs (free; no model)

For each active client project:

```bash
graphify extract --path ~/projects/<client>/<project> --no-docs
```

`--no-docs` runs only tree-sitter AST — zero token cost. Use `--update` on subsequent runs;
use `--force` when files have been deleted (to clear stale nodes). Keep each project's
`graphify-out/` alongside the project; do not merge client projects into one graph.

---

## Step 3 — Hooks

Thin shell wrappers around Graphify. The logic lives in Graphify; the hooks only invoke it.

| Hook | Trigger | Action |
|---|---|---|
| **PostToolUse** | AI writes/edits a vault `.md` | `graphify update --file <path>` |
| **SessionStart** | Session opens | Stale check → `--force` rebuild if needed → inject context |
| **SessionEnd** | Session closes | Append dated journal note to vault |

**Stale-check mechanism (SessionStart):** read the mtime of
`~/.cache/graphify/vault-rebuild.stamp`. If older than N days (7 is the starting guess — tune
after observing drift), run `graphify --force` on the vault and write a new stamp. Then inject:
god-node summary from `GRAPH_REPORT.md` + `convention/*` note summaries + journal pointer.

**Note on `--update` vs `--force`:** `graphify update --file` does not prune deleted nodes.
Stale/ghost nodes accumulate from manual deletes and renames. The periodic `--force` rebuild
triggered by the stale check is the mitigation — it rebuilds the graph clean.

---

## Step 4 — Episodic layer (memsearch)

Install memsearch for time-anchored "what happened / what was I working on" queries. Graphify
does **not** replace this — they serve different query patterns (knowledge graph vs timeline
recall).

```
/plugin marketplace add zilliztech/memsearch
plugin install memsearch
```

Verify daily memory files appear after a few conversations. The open question of whether
memsearch indexes SessionEnd journal notes or only its own auto-capture is deferred to build
time (see Open questions §7).

---

## Step 5 — Vault sync

Sync the markdown vault to the VPS. Pick one:

- **git** — versioned, hourly push/pull via cron; explicit audit trail.
- **Syncthing** — continuous, bidirectional, zero-thought after setup.

**What to sync:** the vault only (`~/Documents/SecondBrain`).

**What NOT to sync:** `graphify-out/` directories, Milvus Lite caches, Ollama models. These are
disposable — rebuilt per machine from the vault (markdown is the single source of truth, per
ADR-008).

---

## Step 6 — Package as a global Claude Code plugin

One global install so every project and machine shares the same vault conventions, hooks, and
Graphify config.

**Plugin contents:**

- **Hooks** registered in settings: SessionStart, PostToolUse, SessionEnd — shell wrappers from
  Step 3, parameterized by vault path and Graphify output dir.
- **Graphify CLI on PATH** — the AI calls `graphify query`, `graphify path`, `graphify explain`
  via the Bash tool. No server process per graph; project-specific graphs are queried with
  `--graph <project-root>/graphify-out/graph.json`.
- **Skills** (carry the know-how to the model):
  - `/os-vault:write` — when to record evergreen knowledge, frontmatter contract, scope rule, vault not repo.
  - `/os-vault:query` — `graphify query` vs memsearch; god-node discipline; `--budget`/`--dfs`; cross-client lookups; progressive disclosure via `summary`.
  - `/os-vault:reorganize` — plan-mode consolidation/promotion procedure; when to trigger `--force` rebuild; human-approval guardrails.
- **Env vars** baked in: `OLLAMA_FLASH_ATTENTION=1`, `GRAPHIFY_OLLAMA_KEEP_ALIVE=5`,
  `OLLAMA_BASE_URL=http://127.0.0.1:11434/v1`, `OLLAMA_API_KEY=<any-non-empty>`. Note:
  `GRAPHIFY_OLLAMA_NUM_CTX` is NOT effective through graphify's `/v1` path — context must
  be baked into the Modelfile variant instead (see Step 2b).
- **Config** (set once at user level): vault path, Graphify output dir, Ollama model name,
  stale-rebuild threshold in days.

---

## Open questions / decisions still to settle

These are deferred to build time. Most can be defaulted without blocking. **§6 (model selection) is RESOLVED** — qwen2.5-coder:7b selected, vault graph built. Remaining questions (§1–5, §7–8) do not block Step 3 or later.

1. **Vault symlink** — `~/Documents/SecondBrain` is confirmed as the vault (RESOLVED per
   ADR-012). The open sub-question: symlink it into `~/.claude/memory` or not? Only needed if
   a tool requires that path.

2. **Sync mechanism** — git (versioned, hourly) vs Syncthing (continuous). Both are valid;
   choose at build time based on preference.

3. **Stale rebuild threshold** — 7 days is the starting guess. Tune after observing how quickly
   ghost-node drift becomes noticeable in practice.

4. **Per-project graph path convention** — how does SessionStart know which `graph.json` to
   inject? Proposed convention: `<project-root>/graphify-out/graph.json`, injected only if the
   file exists at that path. Not yet made explicit in the plugin spec.

5. **`summary` field discipline** — Graphify extracts entities and edges but does not write
   summaries. The human (or AI at note-creation time) must write `summary:` frontmatter. Confirm
   this holds in practice and add a lint/reminder to the `/os-vault:write` skill if it drifts.

6. **Step 2c reference set + model selection** — **RESOLVED (2026-06-04/05):** The reference
   set was generated: 18 fragments (6 fixtures × 3 tiers) in `benchmark/reference-outputs/`.
   Local-model scoring complete (see `benchmark/scoring-results-2026-06-04.md`). **Selected
   model: qwen2.5-coder:7b** (run as Modelfile variant `qwen25-coder-7b-16k`, num_ctx 16384)
   — graphify's shipped default and the only tested candidate that cleared the
   relationship/edge gate. All smaller candidates tested (gemma4:e4b 8.0B, gemma4:e2b 5.1B,
   qwen3.5:2b 2.3B) failed to produce relationship-bearing graphs through graphify's extraction
   prompt on this hardware. Model selection is complete.

7. **memsearch + journal integration** — does memsearch index SessionEnd journal notes or only
   its own auto-capture? How does the journal pointer injected at SessionStart reference the
   vault? Nail down at memsearch install time (Step 4).

8. **MCP server management** — vault graph + N project graphs = N+1 potential instances. The
   build plan leans toward a single Graphify CLI on PATH with `--graph <path>` rather than
   running multiple `graphify serve` instances, but this is not yet made explicit in the plugin
   spec. Confirm the CLI-only approach or define the server lifecycle.

**Resolved:** The type-tag taxonomy is settled via ADR-011 (faceted, six independent namespaces;
`type/` listed first). No further debate needed on taxonomy shape.

---

## Recommended next move

**Step 2 is complete.** The next phase is **Step 3 (Hooks)** — thin shell wrappers around
Graphify for PostToolUse, SessionStart, and SessionEnd — followed by Step 4 (memsearch
episodic layer) and Step 6 (package as global plugin). Open questions §1–5 and §7–8 can be
defaulted or deferred at build time. §6 (model selection) is resolved.