jared
/
cc-os
1
0
Fork 0
Code Issues 18 Pull Requests Packages Projects Releases Wiki Activity
cc-os/docs/memory-system/06-graphify-evaluation.md

7.5 KiB
Raw Blame History

Graphify Evaluation: Fit for the Three-Source Memory System

Evaluated: 2026-06-03. Source docs: /docs/graphify/ (00–09 + external-tips).

Summary Verdict

Graphify is a strong fit for Layer 2 (knowledge) and a natural fit for project-file code analysis. It is NOT the right tool for Layer 1 (episodic/session logs), and it does not "connect" the three sources on its own — that bridge still needs custom glue.

What Graphify Actually Is

Graphify builds a knowledge graph from code and documents:

  • Nodes: entities (functions, classes, concepts, topics)
  • Edges: typed relationships with confidence tags (EXTRACTED = deterministic AST, INFERRED = LLM, AMBIGUOUS = flagged)
  • Storage: graph.json + optional exports (Obsidian sidecar, GraphML, Cypher)
  • Query: semantic graph traversal (query, path, explain) — not vector/embedding search
  • MCP server: graphify.serve exposes query_graph, get_node, shortest_path as Claude tools

The core cost split:

  • Code → tree-sitter AST, 33 languages, deterministic, zero LLM cost
  • Docs → LLM entity extraction, can use local SLMs via Ollama (--backend ollama)

Fit by Layer

Layer 1: Episodic ("what happened") — memsearch / Milvus Lite

Graphify is not the right tool here.

Episodic queries are time-anchored semantic lookups: "what was I working on last Tuesday?", "what client did we discuss the Stripe integration for?" Vector similarity over session logs is well-matched to this. A knowledge graph of session entities would add build overhead without improving recall for timeline queries. Keep memsearch.

Layer 2: Knowledge ("how do we…") — Obsidian vault + tag index

Graphify is a strong candidate to replace or complement the tag index.

What it replaces: The tagging-discipline requirement. Instead of manually adding #tool/semrush #client/sesame3g to every note, a local SLM (Ollama + Qwen2.5 7B or Phi-4 14B) extracts entities automatically. You get entity nodes and relationship edges from the vault without frontmatter authoring.

What it adds that tags cannot: Inferred cross-note relationships. Tags only filter ("give me notes tagged tool/semrush"). A graph can say "tool/semrush is connected to client/sesame3g via 3 project notes, and both reference convention/seo-workflow." That's a graph query, not a tag query.

What it does NOT replace: The summary column in the tag index schema, which is a human-written first-class router hint. Graphify extracts entities, not prose summaries. If summaries are the primary token-efficiency mechanism (see 02-system-design.md:98-102), they need to remain author-controlled.

Critical limitation: Stale node drift. Graphify's --update does not prune deleted symbols/notes — you must --force rebuild to clear ghost nodes. The current design's SQLite is explicitly disposable-and-rebuildable from frontmatter; a Graphify graph is a snapshot with known drift. This is a real tradeoff.

Recommendation: Use Graphify's Obsidian sidecar export (--obsidian) to augment vault notes with auto-extracted entity metadata, and the MCP server for graph queries. Keep the summary frontmatter field as author-controlled. Consider Graphify as the semantic enrichment step that the deferred QMD layer was meant to fill — but without vectors.

Project Files (code) — currently unindexed

Graphify is a no-brainer add here.

Project-file code analysis via AST is free (no LLM, no API key needed at runtime), deterministic, and produces exactly the call-graph / symbol map that "what does this codebase do?" queries need. Every client project gets a graph.json at zero ongoing cost. Query via the MCP server.

Limitation from issue #1122: graphify extract demands a backend credential even for pure-code extraction. Workaround: set a dummy key or use --backend ollama with a running (or not) Ollama instance — confirm at build time.

The "Three Sources Connected" Question

Graphify does not natively connect three separate graphs. It builds one graph per ingestion run. To connect project files, vault, and session logs in one queryable surface, you'd need:

  1. A combined ingestion run that points at all three source trees, OR
  2. Three separate graphs queried via three MCP tool instances and synthesized at the Claude layer

Option 1 risks cross-contamination of unrelated client projects. Option 2 is the safer model: each source has its own graph, and Claude stitches results from three MCP calls.

The more honest framing: Graphify replaces tags as the vault entity index, adds a free code graph per project, and leaves episodic (memsearch) alone. That's a meaningful simplification — you no longer need the Ruby/SQLite tag index CLI — but it's a layer replacement, not a unified connector.

What Changes in the Current Design

Current design With Graphify
Ruby/Sequel/SQLite tag index CLI Replaced by graphify query + MCP server
Manual #tool/ #client/ #domain/ tagging discipline Auto-extracted by local SLM on vault ingestion
summary frontmatter (author-written, router hint) Unchanged — Graphify doesn't generate these
QMD semantic layer (deferred) Graphify fills this role without vectors
memsearch (episodic) Unchanged
Project files (unindexed) Free code graph via AST

What's saved: The entire Ruby CLI build (Step 2 on the critical path in 04-build-plan.md) could be skipped. That's significant scope reduction.

What's added: A local Ollama SLM running on this machine for vault doc extraction. This is a new infrastructure dependency. On low-RAM machines, a 7B model at 4-bit takes ~5 GB VRAM/RAM.

Open Questions This Creates

  1. Rebuild strategy: How often do you rebuild the vault graph? On every SessionStart (expensive), on vault git commit (right cadence), or manually? The --update flag helps but stale-node drift means periodic --force rebuilds are needed.

  2. Model choice for vault extraction: No official recommendation. Test Qwen2.5 7B and Phi-4 14B against a sample of your vault notes and review god-node quality before committing.

  3. Summary field fate: If Graphify extracts entities but doesn't write summaries, does the human still maintain summary frontmatter? Or does a local SLM generate those too?

  4. Cross-client isolation: Project code graphs should be per-client-project, not merged. How do you namespace them? Separate graphify-out/ per project? Or a single graph with source labels as node properties?

  5. MCP server management: Running graphify.serve for vault + N project graphs means N+1 MCP server instances. Is that manageable, or do you build a single meta-server?

References

Topic Source
Code AST extraction (free, deterministic) docs/graphify/03-ingesting-code-ast.md:9–100
Local SLM (Ollama) setup docs/graphify/05-local-models-and-backends.md:63–156
Query verbs + bounded traversal docs/graphify/06-querying-and-god-nodes.md:51–93
Stale node drift limitation docs/graphify/07-token-economics-and-updates.md:136–148
Token savings by repo size docs/graphify/external-tips.md:31–45
Tag index schema and role docs/memory-system/02-system-design.md:95–113
Graph rejection decisions (LightRAG) docs/memory-system/03-architecture-decisions.md:83, 145
Critical path (Ruby CLI = Step 2) docs/memory-system/04-build-plan.md
Open handoff questions docs/memory-system/05-handoff.md:47–60