24 KiB

Raw Permalink Blame History

Build Plan

Last updated: 2026-07-02

How a human builds this system, step by step, and answers to the operational questions: which scripts and hooks, how the AI knows when to write and what conventions to follow, how and when it queries, the CRUD hooks, and how it's packaged as a global Claude Code plugin with skills.

This is a build outline, not the implementation plan. The next session should turn this into a proper implementation plan (writing-plans skill) and execute it.

Architecture decision (2026-06-03): Graphify replaces the Ruby/SQLite tag index. The Ruby CLI (old Step 2) is skipped. Graphify handles both knowledge-graph queries over the vault and free AST-based code graphs per project. The deferred QMD semantic layer (old Step 7) is also skipped — Graphify covers it without vectors. See 06-graphify-evaluation.md for the full rationale.

Architecture decision (2026-06-04): Build order inverted (ADR-013). Build and validate the full system against a small 5–10 note fixture set first; defer bulk vault migration to last. Validate end-to-end on one small pilot project containing both code and documents before onboarding any others. Onboard remaining projects one at a time, with observe-and-adjust between each. Steps 2d and 2e below are updated accordingly.

Part A — Build order (human builder's path)

Build bottom-up: the vault and Graphify first (usable standalone), then the hooks, then the plugin that packages it.

Step 1 — Vault skeleton & conventions

Vault location is settled by ADR-012: ~/Documents/SecondBrain (the existing Obsidian vault). No new vault is created; symlink into ~/.claude/memory only if a tool insists. Vault is the single source of truth.
Write CONVENTIONS.md in the vault: the frontmatter contract. The required fields are summary (one-line, author-written — this is the human-readable router hint Graphify does not generate) and scope (scope/global or scope/project). Tags are now supplementary, not the primary query mechanism. Naming resolved (2026-06-09): vault-conventions.md is the canonical name (not CONVENTIONS.md). The file exists at ~/Documents/SecondBrain/vault-conventions.md.
Seed a few real notes (e.g. tool/semrush, client/<x>) to use as extraction test cases in Step 2.

Status: COMPLETE (2026-06-09)

vault-conventions.md (renamed 2026-06-09 from CONVENTIONS.md) exists in the vault at ~/Documents/SecondBrain/vault-conventions.md — frontmatter contract + tag taxonomy.
All 6 fixture notes exist in the vault (seeded for extraction testing in Step 2c).
14 Graphify handbook + memory-system design notes migrated to the vault (2026-06-09) as part of migration scaffolding — first-class deliverable per the empirical finding locked 2026-06-05.

Step 2 — Graphify + Ollama setup (the knowledge graph layer)

This replaces the Ruby tag-index CLI. Graphify builds a knowledge graph over vault docs via a local SLM, and a free code graph over project files via AST.

2a — Install Graphify DONE (2026-06-08)

pip install graphify   # or follow current install docs

Verify: graphify --version

Status: graphify v0.8.31 installed at /home/jared/.local/bin/graphify.

2b — Configure Ollama for extraction DONE (2026-06-08)

The 2024 approach (Modelfile with PARAMETER num_ctx) still works but the preferred programmatic method is now per-request via the API (Graphify supports this via GRAPHIFY_OLLAMA_NUM_CTX):

# Set in your shell profile or service env:
OLLAMA_FLASH_ATTENTION=1          # 30-50% VRAM savings on KV cache — always set this
GRAPHIFY_OLLAMA_NUM_CTX=8192      # 8K is sufficient for vault notes (200-2000 words)
                                  # and leaves comfortable headroom for prompt overhead

To verify context is being used: ollama ps shows allocated context after first call.

Status: All env vars set in ~/.claude/plugins/os-vault/config.yaml: OLLAMA_FLASH_ATTENTION=1, GRAPHIFY_OLLAMA_NUM_CTX=8192, GRAPHIFY_OLLAMA_KEEP_ALIVE=5. Model locked to qwen25-coder-7b-16k (per ADR and memory note — this is Graphify's default and the confirmed working extraction model).

Available models (as of 2026-06-03, in order of interest):

gemma4:e2b — 7.2 GB, ~2B effective params, 128K native window, fast; start here
qwen3.5:2b — 2.7 GB, smallest, good fallback if VRAM is constrained
gemma4:e4b — 9.6 GB, more capable, slower

2c — Claude reference-set benchmark (THE GATE, before committing to a local model)

This step produces the gold-standard reference set — one structured extraction output per fixture note per Claude tier. It does not choose the final extraction model, and it does not measure speed.

Input: the 5–10 fixture notes selected in Step 1 (from the runbook).

What to run: dispatch one Claude Code subagent per tier (Claude-tiers only — Ollama models are not reachable in this environment):

Tier	Model ID	Role
Haiku	`claude-haiku-4-5`	Lightweight reference
Sonnet	`claude-sonnet-4-6`	Mid-tier reference
Opus	`claude-opus-4-8`	Gold standard (scoring rubric)

Each subagent receives only the fixture note text plus the shared extraction spec — no design docs, no project context (fairness contract). See ADR-013.

Metrics — quality only (wall-clock speed is explicitly out of scope here):

Entity correctness — right concepts extracted, no hallucinated entities
Relationship plausibility and typing — edges plausible, correctly typed, no missing key edges
Confidence-tag accuracy — INFERRED vs AMBIGUOUS applied appropriately

Deliverables (produced by this step):

Dispatch prompt (copy/paste-able): docs/memory-system/benchmark/dispatch-prompt.md
Shared extraction schema: docs/memory-system/benchmark/extraction-spec.md
Per-tier outputs: docs/memory-system/benchmark/reference-outputs/<note-slug>.<tier>.md

Opus output is the rubric. Deferred later step: local Ollama models (gemma4:e2b, qwen3.5:2b, gemma4:e4b) are timed AND scored against these references — that scoring run is where speed re-enters and the final model is chosen. Do not hardcode a model before that run completes.

Status (2026-06-04): EXECUTED. 6 cross-domain fixtures × 3 Claude tiers = 18 reference fragments generated in benchmark/reference-outputs/. Run as-is (no vault frontmatter modification); verified clean. Fixtures listed in benchmark/dispatch-prompt.md.

Authoritative detail lives in docs/memory-system/05-implementation-process.md §2c. Local-model gut-check done (2026-06-04): gemma4:e4b is the candidate; GPU fix pending reboot; Graphify owns the ollama call — see docs/memory-system/benchmark/local-llm-findings-2026-06-04.md.

2d — Initial fixture graph build (ADR-013: small-first)

Run the initial build against the small fixture set (5–10 notes from Step 1/2c), not the full vault. The bulk vault build is deferred to after the system is validated end-to-end.

graphify extract --path ~/Documents/SecondBrain --backend ollama --model gemma4:e2b \
  --token-budget 512 --max-concurrency 2

Tune --token-budget (semantic chunk size) and --max-concurrency based on VRAM headroom. Review GRAPH_REPORT.md — check god nodes make sense (they should be your most-connected tools, clients, and domain concepts).

Note (2026-06-08): The ~/Documents/SecondBrain vault already has a live graphify-out/ graph (32 files, built 2026-06-05). The fixture-first approach from ADR-013 may therefore be superseded — the vault graph exists before Step 1 (conventions/fixtures) is formally complete.

Resolved (2026-06-15): DONE/superseded. The live vault graphify-out/ over the full vault IS the baseline graph. A fixture-only build is moot now that the vault holds real scaffolding notes (not just fixtures). No separate fixture-only build will be run.

Full vault migration (the ~/Documents/SecondBrain build above run over all notes) is the final step — deferred to after end-to-end validation on the pilot project. Do not bulk-migrate the vault until the system is verified working on the fixture set and pilot project.

2e — Per-project code graphs (free, no model needed)

ADR-013 order: start with ONE pilot project that contains both code and documents; validate end-to-end before onboarding others. Onboard remaining projects one at a time, with observe-and-adjust between each. Do not batch all projects at once.

For the pilot project (and each subsequent project, one at a time):

graphify extract --path ~/projects/<client>/<project> --no-docs

--no-docs skips LLM extraction; only AST is run. Zero token cost. Add --update on subsequent runs; use --force when you've deleted files (to clear stale nodes).

Store each project's graphify-out/ alongside the project, or in a configurable cache dir. Keep per-project graphs separate — do not merge client projects into one graph.

Note (2026-06-08): A graphify-out/ for the cc-os project exists (built 2026-06-05), but cc-os is a docs-only repo, not a pilot project in the ADR-013 sense. A pilot project with real code has not yet been onboarded.

Status: PILOT DONE (2026-06-15). The llf-schema project (/home/jared/dev/llf-schema, PHP 8.2 WordPress plugin) was onboarded via the /os-vault:onboard-project skill: graphify extract (tree-sitter AST on 90 code files + Ollama doc pass on 9 doc files, ~3.5 min) + cluster-only → 605 nodes / 930 edges / 52 communities (~524K graph.json). graphify-out/ added to that repo's .gitignore. Finding: Ollama doc pass was lossy on structured WordPress docs (101 warnings, recursive bisection) but the AST pass carried the code structure — acceptable for code-heavy repos. Remaining under 2e: onboard additional projects incrementally.

Step 3 — Hooks (maintenance + retrieval) DONE (2026-06-08)

Hooks now call Graphify instead of the Ruby CLI.

PostToolUse (on Write/Edit of vault .md): graphify update --file <path> to merge the changed note into the vault graph. Note: Graphify's --update does not prune deleted nodes — a periodic --force rebuild (e.g. weekly, or triggered by SessionStart stale check) is needed to clear ghost nodes.
SessionStart: reconcile (check vault graph mtime vs last rebuild; trigger --force if stale threshold exceeded) + inject context (emits project graph path if a graphify-out/graph.json exists at the project root; vault context injection was removed).
SessionEnd: append daily journal note with pointers to vault notes touched.

Stale-node mitigation strategy: track last --force rebuild time in a stamp file (e.g. ~/.cache/graphify/vault-rebuild.stamp). SessionStart hook triggers --force if older than N days (7 is a reasonable starting point).

Step 4 — Episodic layer (memsearch) — DONE

Install memsearch (/plugin marketplace add zilliztech/memsearch, then plugin install memsearch), local to start. Verify daily memory files appear after a few conversations.
Decide whether memsearch indexes our session-end journal notes or its own capture (likely its own; our journal can point into the knowledge vault).
Graphify does not replace memsearch here. Time-anchored semantic queries ("what was I working on last Tuesday?") are better served by vector similarity over session logs.

Step 5 — Sync — PARTIAL

Step 5 covers two distinct sync lines; they are at different statuses:

5a — memsearch episodic memory git sync — DONE (2026-06-09)

~/.memsearch is now a git repo on branch main with remote origin pointing to a private self-hosted Forgejo repo (ssh://git@forgejo.swansoncloud.com:2222/jared/memsearch.git; web: https://forgejo.swansoncloud.com/jared/memsearch). Initial commit 106cebc.

A whitelist .gitignore tracks only memory/*.md (the daily session files) and .gitignore. Excluded as derived/disposable: milvus.db (Milvus Lite index — rebuildable via memsearch index), config.toml, and the bge-m3 ONNX model (lives in ~/.cache/huggingface).

Auto-commit and push are wired into the cc-os memory plugin's own memsearch_sync.py SessionEnd hook (split from session_end.py by ADR-016, 2026-06-12; behavior preserved — see ADR-015) (not the marketplace plugin, which can be clobbered on update). The block guards on ~/.memsearch/.git existing, commits only when staged content exists (message: memsearch: session memory <date>), and pushes with timeout 30 ... || true. Wrapped in a subshell with trailing || true — cannot fail session shutdown. See ADR-015 for rationale and the cross-client commingling, which is an accepted design choice (single global store across all clients by intent — see ADR-015, resolved 2026-06-09).

5b — Obsidian vault → VPS sync — DONE (2026-06-15)

The vault at ~/Documents/SecondBrain was initialized as a git repo and pushed to a new private Forgejo repo (ssh://git@forgejo.swansoncloud.com:2222/jared/SecondBrain.git; web: https://forgejo.swansoncloud.com/jared/SecondBrain), mirroring the memsearch sync pattern. 52 files committed on 2026-06-15 (notes, journal, templates, vault-conventions.md, CLAUDE.md, .obsidian config); graphify-out/ correctly excluded via the existing .gitignore. Decision: git sync chosen over Syncthing.

Auto-sync hook — DONE (2026-06-15): vault_sync.py added as the third SessionEnd hook (after session_end.py and memsearch_sync.py, timeout 30). Auto-commits git add -A and pushes the vault to forgejo.swansoncloud.com/jared/SecondBrain on every session end, so the daily journal note from session_end.py is included in the commit. Mirrors the memsearch_sync.py pattern; uses cfg.vault_path. Smoke-tested (exit 0) 2026-06-15.

Multi-machine freshness — DONE (2026-07-10): session_start.py now runs a fast-forward-only git pull (timeout 10, --ff-only --quiet) against both cfg.vault_path and cfg.memsearch_dir before the staleness check. Silent on success; a quiet one-line stderr note on failure (offline machine, diverged history) that never blocks or delays session start.

Do not sync the Graphify graphify-out/ directories, Milvus caches, or the Ollama models. Graphs are rebuilt per machine from the vault (the single source of truth).

Step 6 — Package as a global plugin (Part D) DONE (2026-06-08)

Wrap Steps 2–3 into a Claude Code plugin with skills; install at user level.

Status: Global plugin installed at ~/.claude/plugins/os-vault/ with all hook scripts and 5 skills (query, write, reorganize, onboard-project, design-template). Plugin enabled in ~/.claude/settings.json.

Source-in-git + bash→Python port — DONE (2026-06-12): Plugin source moved into git at cc-os/plugins/os-vault/ (OpenSpec change memory-plugin-source-and-port). Bash hooks ported to Python using a deep-module architecture (shared config.py, hook_io.py, session_state.py; thin entry-point scripts under hooks/). memsearch sync split into memsearch_sync.py as a dedicated second SessionEnd hook (ADR-015 behavior relocated, not reversed). Cutover done via symlink ~/.claude/plugins/os-vault → cc-os/plugins/os-vault/ and settings.json hook rewrite to Python absolute paths. Fresh-session test passed 2026-06-12. See ADR-016.

Step 7 — (SKIPPED) QMD semantic layer

Covered by Graphify. The knowledge-graph approach provides structured semantic retrieval without vectors. Only revisit if Graphify's graph queries prove insufficient for a use case where embedding similarity would outperform graph traversal.

Part B — The AI's write & query conventions (skills teach these)

When the AI WRITES to the vault

Trigger writes when the AI learns something evergreen and reusable across projects — not project-ephemeral state (that's the episodic layer / project files). Concretely:

It worked out how a tool/API behaves (e.g. SEMrush auth, rate limits, an endpoint quirk).
It established a convention, decision, or preference that should apply beyond this task.
It discovered a client-specific fact worth reusing (how this client uses a tool).

What conventions to follow when writing (enforced by the frontmatter contract):

One concept per note; keep notes small (the L1 "under ~200 lines / one topic" discipline).
Required frontmatter: a one-line summary (written now, not deferred — this is the human-authored router hint; Graphify extracts entities but does not write summaries), and scope/global or scope/project.
Scope rule: default new tool/domain knowledge to scope/global; mark scope/project when it's specific to how a client uses something.
Write only to the vault, never silently into a project repo.
Promotion to global and consolidation happen in the reorganize step (Part C), in plan mode, for human review.

When & how the AI QUERIES

Knowledge queries (vault graph) via Graphify CLI (Bash tool):

graphify query "how do we authenticate with SEMrush?" — semantic graph search
graphify path "client/sesame3g" "tool/semrush" — how these two concepts connect
graphify explain "convention/seo-workflow" — node deep-dive with neighbors
Add --budget N to cap answer size (default ~2000 tokens); --dfs --budget N for bounded traversal
Add --graph <path> to query a specific project's graph.json instead of the vault graph

At session start: the session_context.py hook emits the project graph path (pointing at <project-root>/graphify-out/graph.json) if one exists — vault context injection (god-node summary, convention summaries, journal pointer) was removed. The AI queries the vault or project graph on demand rather than having it injected.

On demand, during a task: run graphify query when the task touches a tool/client/domain. The graph returns relevant node clusters; the AI opens only the vault notes whose summary matches the task. The summary frontmatter field remains the human-written progressive-disclosure hint.

Cross-client lookups: graphify query "tool semrush" returns all nodes connected to the SEMrush concept, regardless of which client project they came from. Graph edges show provenance.

"What happened" questions go to memsearch in natural language — not the graph.

Part C — Hooks & CRUD mapping

Operation	Trigger	Mechanism
Create	AI writes a new vault note	AI writes `.md` (Write tool) → PostToolUse hook → `graphify update --file`
Read / query	AI needs knowledge	AI runs `graphify query` / `graphify explain` via Bash (no hook; on-demand)
Update	AI/user edits a note	AI edit → PostToolUse → `graphify update --file`; user edit → SessionStart reconcile
Delete / rename	note removed/renamed	`graphify update --file` prunes AI-deleted paths; SessionStart triggers `--force` rebuild if stale (catches manual deletes)
Inject	new session starts	SessionStart hook: emits project graph path (vault context injection removed)
Journal	session ends	SessionEnd hook appends a dated journal note with pointers
Reorganize	periodic, user-invoked	`reorganize memory` in plan mode: dedupe, merge, split, re-scope, then `graphify --force` to rebuild clean graph — human approves
Stale graph rebuild	SessionStart stale check	If rebuild stamp > N days old: `graphify --force` on vault; resets stamp

Hooks are thin shell wrappers; the logic lives in Graphify.

Hooks summary:

SessionStart — stale check + --force if needed, then inject project graph path (vault context injection was removed; session_context.py now only emits the project graph path).
PostToolUse (on Write/Edit of vault .md) — graphify update --file.
SessionEnd — append daily journal note.
(memsearch brings its own hooks for the episodic layer.)

Status: All 4 hooks wired in ~/.claude/settings.json: SessionStart, UserPromptSubmit (context injection), PostToolUse (vault writes), SessionEnd (journal).

Part D — Claude Code plugin with skills (global install)

Goal: one global install so every project/machine knows the vault, conventions, hooks, and Graphify config.

Plugin contents:

Hooks registered in settings: SessionStart, PostToolUse, SessionEnd (shell wrappers from Part C), pointed at a configurable vault path + Graphify output dir.
Graphify CLI on PATH — the AI uses the Bash tool to run graphify query, graphify path, graphify explain directly. No server process. Project-specific graphs are queried with --graph <path> pointing at the project's graphify-out/graph.json.
Skills (these carry the know-how to the model):
- /os-vault:write — when to record evergreen knowledge, the frontmatter contract (summary required, scope rule), "vault not repo."
- /os-vault:query — how/when to use graphify query vs memsearch; god-node discipline; --budget/--dfs usage; cross-client lookups via --graph; progressive-disclosure via summary field.
- /os-vault:reorganize — the plan-mode consolidation/promotion procedure + guardrails; when to trigger graphify --force rebuild.
- /os-vault:onboard-project DONE (2026-06-08) — full lifecycle skill for per-project Graphify graphs:
  - Onboard: graphify extract . --backend ollama --model qwen2.5-coder:7b, then graphify cluster-only ., add graphify-out/ to .gitignore
  - Onboard flow (updated 2026-06-17, ADR-017): Assessment-first — the skill surveys the repo and generates a per-project .graphifyignore for user confirmation before running extract, because Graphify does not honor .gitignore.
  - Query: graphify query ... --graph <path> for project-context queries (wraps /os-vault:query)
  - Update: graphify update . for incremental updates (no API cost) or --force rebuild after structural changes
  - Remove: delete graphify-out/ when no longer needed
Config: vault path, Graphify output dir, Ollama model name, num_ctx, rebuild-stale threshold in days — set once at user level.
Env vars set by plugin: OLLAMA_FLASH_ATTENTION=1, GRAPHIFY_OLLAMA_NUM_CTX=8192, GRAPHIFY_OLLAMA_KEEP_ALIVE=5.

Why a plugin + skills (not just CLAUDE.md): hooks must be registered by the harness; conventions taught as skills load on demand without bloating every project's context. A single global install keeps conventions a single source of truth.

Convention notes placement: coding convention/* notes live as data in the vault (edited freely, resolved by SessionStart hook); memory-system skills live in the plugin (behavior, versioned).

Open questions

Vault location — Settled by ADR-012: ~/Documents/SecondBrain. Symlink into ~/.claude/memory only if a tool requires it.
Sync mechanism — ~~git (versioned, hourly) vs Syncthing (continuous)?~~ RESOLVED (2026-06-15): git.
Stale rebuild threshold — how many days before SessionStart triggers --force? 7 days is the starting guess; tune after observing drift in practice.
Per-project graph path — how does the AI know which graph.json to pass to --graph for the current project? Convention: <project-root>/graphify-out/graph.json, injected by SessionStart hook if a project graph exists at that path.
Summary field ownership — Graphify extracts entities but does NOT write summaries. The human (or AI, on vault note creation) must write the summary frontmatter. Confirm this discipline holds in practice.
Model benchmark results — defer all model decisions until Step 2c benchmark is done. Don't hardcode gemma4:e2b until testing confirms it's the right pick.
memsearch + journal — does memsearch index our SessionEnd journal notes, or only its own auto-capture, and how does the journal point into the knowledge vault?

24 KiB Raw Permalink Blame History Unescape Escape