jared
/
cc-os
1
0
Fork 0
Code Issues 18 Pull Requests Packages Projects Releases Wiki Activity
cc-os/docs/memory-system/04-build-plan.md

16 KiB
Raw Blame History

Build Plan

Last updated: 2026-06-04

How a human builds this system, step by step, and answers to the operational questions: which scripts and hooks, how the AI knows when to write and what conventions to follow, how and when it queries, the CRUD hooks, and how it's packaged as a global Claude Code plugin with skills.

This is a build outline, not the implementation plan. The next session should turn this into a proper implementation plan (writing-plans skill) and execute it.

Architecture decision (2026-06-03): Graphify replaces the Ruby/SQLite tag index. The Ruby CLI (old Step 2) is skipped. Graphify handles both knowledge-graph queries over the vault and free AST-based code graphs per project. The deferred QMD semantic layer (old Step 7) is also skipped — Graphify covers it without vectors. See 06-graphify-evaluation.md for the full rationale.

Architecture decision (2026-06-04): Build order inverted (ADR-013). Build and validate the full system against a small 5–10 note fixture set first; defer bulk vault migration to last. Validate end-to-end on one small pilot project containing both code and documents before onboarding any others. Onboard remaining projects one at a time, with observe-and-adjust between each. Steps 2d and 2e below are updated accordingly.

Part A — Build order (human builder's path)

Build bottom-up: the vault and Graphify first (usable standalone), then the hooks, then the plugin that packages it.

Step 1 — Vault skeleton & conventions

  • Vault location is settled by ADR-012: ~/Documents/SecondBrain (the existing Obsidian vault). No new vault is created; symlink into ~/.claude/memory only if a tool insists. Vault is the single source of truth.
  • Write CONVENTIONS.md in the vault: the frontmatter contract. The required fields are summary (one-line, author-written — this is the human-readable router hint Graphify does not generate) and scope (scope/global or scope/project). Tags are now supplementary, not the primary query mechanism.
  • Seed a few real notes (e.g. tool/semrush, client/<x>) to use as extraction test cases in Step 2.

Step 2 — Graphify + Ollama setup (the knowledge graph layer)

This replaces the Ruby tag-index CLI. Graphify builds a knowledge graph over vault docs via a local SLM, and a free code graph over project files via AST.

2a — Install Graphify

pip install graphify   # or follow current install docs

Verify: graphify --version

2b — Configure Ollama for extraction

The 2024 approach (Modelfile with PARAMETER num_ctx) still works but the preferred programmatic method is now per-request via the API (Graphify supports this via GRAPHIFY_OLLAMA_NUM_CTX):

# Set in your shell profile or service env:
OLLAMA_FLASH_ATTENTION=1          # 30-50% VRAM savings on KV cache — always set this
GRAPHIFY_OLLAMA_NUM_CTX=8192      # 8K is sufficient for vault notes (200-2000 words)
                                  # and leaves comfortable headroom for prompt overhead

To verify context is being used: ollama ps shows allocated context after first call.

Available models (as of 2026-06-03, in order of interest):

  • gemma4:e2b — 7.2 GB, ~2B effective params, 128K native window, fast; start here
  • qwen3.5:2b — 2.7 GB, smallest, good fallback if VRAM is constrained
  • gemma4:e4b — 9.6 GB, more capable, slower

2c — Claude reference-set benchmark (THE GATE, before committing to a local model)

This step produces the gold-standard reference set — one structured extraction output per fixture note per Claude tier. It does not choose the final extraction model, and it does not measure speed.

Input: the 5–10 fixture notes selected in Step 1 (from the runbook).

What to run: dispatch one Claude Code subagent per tier (Claude-tiers only — Ollama models are not reachable in this environment):

Tier Model ID Role
Haiku claude-haiku-4-5 Lightweight reference
Sonnet claude-sonnet-4-6 Mid-tier reference
Opus claude-opus-4-8 Gold standard (scoring rubric)

Each subagent receives only the fixture note text plus the shared extraction spec — no design docs, no project context (fairness contract). See ADR-013.

Metrics — quality only (wall-clock speed is explicitly out of scope here):

  1. Entity correctness — right concepts extracted, no hallucinated entities
  2. Relationship plausibility and typing — edges plausible, correctly typed, no missing key edges
  3. Confidence-tag accuracy — INFERRED vs AMBIGUOUS applied appropriately

Deliverables (produced by this step):

  • Dispatch prompt (copy/paste-able): docs/memory-system/benchmark/dispatch-prompt.md
  • Shared extraction schema: docs/memory-system/benchmark/extraction-spec.md
  • Per-tier outputs: docs/memory-system/benchmark/reference-outputs/<note-slug>.<tier>.md

Opus output is the rubric. Deferred later step: local Ollama models (gemma4:e2b, qwen3.5:2b, gemma4:e4b) are timed AND scored against these references — that scoring run is where speed re-enters and the final model is chosen. Do not hardcode a model before that run completes.

Authoritative detail lives in docs/memory-system/05-implementation-process.md §2c.

2d — Initial fixture graph build (ADR-013: small-first)

Run the initial build against the small fixture set (5–10 notes from Step 1/2c), not the full vault. The bulk vault build is deferred to after the system is validated end-to-end.

graphify extract --path ~/Documents/SecondBrain --backend ollama --model gemma4:e2b \
  --token-budget 512 --max-concurrency 2

Tune --token-budget (semantic chunk size) and --max-concurrency based on VRAM headroom. Review GRAPH_REPORT.md — check god nodes make sense (they should be your most-connected tools, clients, and domain concepts).

Full vault migration (the ~/Documents/SecondBrain build above run over all notes) is the final step — deferred to after end-to-end validation on the pilot project. Do not bulk-migrate the vault until the system is verified working on the fixture set and pilot project.

2e — Per-project code graphs (free, no model needed)

ADR-013 order: start with ONE pilot project that contains both code and documents; validate end-to-end before onboarding others. Onboard remaining projects one at a time, with observe-and-adjust between each. Do not batch all projects at once.

For the pilot project (and each subsequent project, one at a time):

graphify extract --path ~/projects/<client>/<project> --no-docs

--no-docs skips LLM extraction; only AST is run. Zero token cost. Add --update on subsequent runs; use --force when you've deleted files (to clear stale nodes).

Store each project's graphify-out/ alongside the project, or in a configurable cache dir. Keep per-project graphs separate — do not merge client projects into one graph.

Step 3 — Hooks (maintenance + retrieval)

Hooks now call Graphify instead of the Ruby CLI.

  • PostToolUse (on Write/Edit of vault .md): graphify update --file <path> to merge the changed note into the vault graph. Note: Graphify's --update does not prune deleted nodes — a periodic --force rebuild (e.g. weekly, or triggered by SessionStart stale check) is needed to clear ghost nodes.
  • SessionStart: reconcile (check vault graph mtime vs last rebuild; trigger --force if stale threshold exceeded) + inject context (query MCP server for god nodes + inject convention/* note summaries + journal pointer).
  • SessionEnd: append daily journal note with pointers to vault notes touched.

Stale-node mitigation strategy: track last --force rebuild time in a stamp file (e.g. ~/.cache/graphify/vault-rebuild.stamp). SessionStart hook triggers --force if older than N days (7 is a reasonable starting point).

Step 4 — Episodic layer (memsearch)

  • Install memsearch (/plugin marketplace add zilliztech/memsearch, then plugin install memsearch), local to start. Verify daily memory files appear after a few conversations.
  • Decide whether memsearch indexes our session-end journal notes or its own capture (likely its own; our journal can point into the knowledge vault).
  • Graphify does not replace memsearch here. Time-anchored semantic queries ("what was I working on last Tuesday?") are better served by vector similarity over session logs.

Step 5 — Sync

  • Pick git (versioned, hourly push/pull) or Syncthing (continuous, zero-thought) for the vault → VPS. Configure on each machine.
  • Do not sync the Graphify graphify-out/ directories, Milvus caches, or the Ollama models. Graphs are rebuilt per machine from the vault (the single source of truth).

Step 6 — Package as a global plugin (Part D)

  • Wrap Steps 2–3 into a Claude Code plugin with skills; install at user level.

Step 7 — (SKIPPED) QMD semantic layer

Covered by Graphify. The knowledge-graph approach provides structured semantic retrieval without vectors. Only revisit if Graphify's graph queries prove insufficient for a use case where embedding similarity would outperform graph traversal.

Part B — The AI's write & query conventions (skills teach these)

When the AI WRITES to the vault

Trigger writes when the AI learns something evergreen and reusable across projects — not project-ephemeral state (that's the episodic layer / project files). Concretely:

  • It worked out how a tool/API behaves (e.g. SEMrush auth, rate limits, an endpoint quirk).
  • It established a convention, decision, or preference that should apply beyond this task.
  • It discovered a client-specific fact worth reusing (how this client uses a tool).

What conventions to follow when writing (enforced by the frontmatter contract):

  • One concept per note; keep notes small (the L1 "under ~200 lines / one topic" discipline).
  • Required frontmatter: a one-line summary (written now, not deferred — this is the human-authored router hint; Graphify extracts entities but does not write summaries), and scope/global or scope/project.
  • Scope rule: default new tool/domain knowledge to scope/global; mark scope/project when it's specific to how a client uses something.
  • Write only to the vault, never silently into a project repo.
  • Promotion to global and consolidation happen in the reorganize step (Part C), in plan mode, for human review.

When & how the AI QUERIES

Knowledge queries (vault graph) via Graphify CLI (Bash tool):

  • graphify query "how do we authenticate with SEMrush?" — semantic graph search
  • graphify path "client/sesame3g" "tool/semrush" — how these two concepts connect
  • graphify explain "convention/seo-workflow" — node deep-dive with neighbors
  • Add --budget N to cap answer size (default ~2000 tokens); --dfs --budget N for bounded traversal
  • Add --graph <path> to query a specific project's graph.json instead of the vault graph

At session start: god-node summary (injected by hook from GRAPH_REPORT.md) gives the architecture spine — what are the highest-connectivity concepts in the vault. The AI then queries specific nodes on demand rather than reading everything.

On demand, during a task: run graphify query when the task touches a tool/client/domain. The graph returns relevant node clusters; the AI opens only the vault notes whose summary matches the task. The summary frontmatter field remains the human-written progressive-disclosure hint.

Cross-client lookups: graphify query "tool semrush" returns all nodes connected to the SEMrush concept, regardless of which client project they came from. Graph edges show provenance.

"What happened" questions go to memsearch in natural language — not the graph.

Part C — Hooks & CRUD mapping

Operation Trigger Mechanism
Create AI writes a new vault note AI writes .md (Write tool) → PostToolUse hook → graphify update --file
Read / query AI needs knowledge AI runs graphify query / graphify explain via Bash (no hook; on-demand)
Update AI/user edits a note AI edit → PostToolUse → graphify update --file; user edit → SessionStart reconcile
Delete / rename note removed/renamed graphify update --file prunes AI-deleted paths; SessionStart triggers --force rebuild if stale (catches manual deletes)
Inject new session starts SessionStart hook: god-node summary + convention/* summaries + journal pointer
Journal session ends SessionEnd hook appends a dated journal note with pointers
Reorganize periodic, user-invoked reorganize memory in plan mode: dedupe, merge, split, re-scope, then graphify --force to rebuild clean graph — human approves
Stale graph rebuild SessionStart stale check If rebuild stamp > N days old: graphify --force on vault; resets stamp

Hooks are thin shell wrappers; the logic lives in Graphify.

Hooks summary:

  • SessionStart — stale check + --force if needed, then inject (god nodes + conventions + journal pointer).
  • PostToolUse (on Write/Edit of vault .md) — graphify update --file.
  • SessionEnd — append daily journal note.
  • (memsearch brings its own hooks for the episodic layer.)

Part D — Claude Code plugin with skills (global install)

Goal: one global install so every project/machine knows the vault, conventions, hooks, and Graphify config.

Plugin contents:

  • Hooks registered in settings: SessionStart, PostToolUse, SessionEnd (shell wrappers from Part C), pointed at a configurable vault path + Graphify output dir.
  • Graphify CLI on PATH — the AI uses the Bash tool to run graphify query, graphify path, graphify explain directly. No server process. Project-specific graphs are queried with --graph <path> pointing at the project's graphify-out/graph.json.
  • Skills (these carry the know-how to the model):
    • memory-write — when to record evergreen knowledge, the frontmatter contract (summary required, scope rule), "vault not repo."
    • memory-query — how/when to use graphify query vs memsearch; god-node discipline; --budget/--dfs usage; cross-client lookups via --graph; progressive-disclosure via summary field.
    • memory-reorganize — the plan-mode consolidation/promotion procedure + guardrails; when to trigger graphify --force rebuild.
  • Config: vault path, Graphify output dir, Ollama model name, num_ctx, rebuild-stale threshold in days — set once at user level.
  • Env vars set by plugin: OLLAMA_FLASH_ATTENTION=1, GRAPHIFY_OLLAMA_NUM_CTX=8192, GRAPHIFY_OLLAMA_KEEP_ALIVE=5.

Why a plugin + skills (not just CLAUDE.md): hooks must be registered by the harness; conventions taught as skills load on demand without bloating every project's context. A single global install keeps conventions a single source of truth.

Convention notes placement: coding convention/* notes live as data in the vault (edited freely, resolved by SessionStart hook); memory-system skills live in the plugin (behavior, versioned).

Open questions

  1. Vault location — Settled by ADR-012: ~/Documents/SecondBrain. Symlink into ~/.claude/memory only if a tool requires it.
  2. Sync mechanism — git (versioned, hourly) vs Syncthing (continuous)?
  3. Stale rebuild threshold — how many days before SessionStart triggers --force? 7 days is the starting guess; tune after observing drift in practice.
  4. Per-project graph path — how does the AI know which graph.json to pass to --graph for the current project? Convention: <project-root>/graphify-out/graph.json, injected by SessionStart hook if a project graph exists at that path.
  5. Summary field ownership — Graphify extracts entities but does NOT write summaries. The human (or AI, on vault note creation) must write the summary frontmatter. Confirm this discipline holds in practice.
  6. Model benchmark results — defer all model decisions until Step 2c benchmark is done. Don't hardcode gemma4:e2b until testing confirms it's the right pick.
  7. memsearch + journal — does memsearch index our SessionEnd journal notes, or only its own auto-capture, and how does the journal point into the knowledge vault?