117 lines
7.4 KiB
Markdown
117 lines
7.4 KiB
Markdown
# Graphify Evaluation: Fit for the Three-Source Memory System
|
||
|
||
> Evaluated: 2026-06-03. Source docs: `/docs/graphify/` (00–09 + external-tips).
|
||
|
||
---
|
||
|
||
## Summary Verdict
|
||
|
||
**Graphify is a strong fit for Layer 2 (knowledge) and a natural fit for project-file code analysis. It is NOT the right tool for Layer 1 (episodic/session logs), and it does not "connect" the three sources on its own — that bridge still needs custom glue.**
|
||
|
||
---
|
||
|
||
## What Graphify Actually Is
|
||
|
||
Graphify builds a knowledge graph from code and documents:
|
||
|
||
- **Nodes**: entities (functions, classes, concepts, topics)
|
||
- **Edges**: typed relationships with confidence tags (`EXTRACTED` = deterministic AST, `INFERRED` = LLM, `AMBIGUOUS` = flagged)
|
||
- **Storage**: `graph.json` + optional exports (Obsidian sidecar, GraphML, Cypher)
|
||
- **Query**: semantic graph traversal (`query`, `path`, `explain`) — not vector/embedding search
|
||
- **MCP server**: `graphify.serve` exposes `query_graph`, `get_node`, `shortest_path` as Claude tools
|
||
|
||
The core cost split:
|
||
- **Code** → tree-sitter AST, 33 languages, deterministic, **zero LLM cost**
|
||
- **Docs** → LLM entity extraction, can use local SLMs via Ollama (`--backend ollama`)
|
||
|
||
---
|
||
|
||
## Fit by Layer
|
||
|
||
### Layer 1: Episodic ("what happened") — memsearch / Milvus Lite
|
||
|
||
**Graphify is not the right tool here.**
|
||
|
||
Episodic queries are time-anchored semantic lookups: "what was I working on last Tuesday?", "what client did we discuss the Stripe integration for?" Vector similarity over session logs is well-matched to this. A knowledge graph of session entities would add build overhead without improving recall for timeline queries. **Keep memsearch.**
|
||
|
||
### Layer 2: Knowledge ("how do we…") — Obsidian vault + tag index
|
||
|
||
**Graphify is a strong candidate to replace or complement the tag index.**
|
||
|
||
**What it replaces:** The tagging-discipline requirement. Instead of manually adding `#tool/semrush #client/sesame3g` to every note, a local SLM (Ollama + Qwen2.5 7B or Phi-4 14B) extracts entities automatically. You get entity nodes and relationship edges from the vault without frontmatter authoring.
|
||
|
||
**What it adds that tags cannot:** Inferred cross-note relationships. Tags only filter ("give me notes tagged tool/semrush"). A graph can say "tool/semrush is connected to client/sesame3g via 3 project notes, and both reference convention/seo-workflow." That's a graph query, not a tag query.
|
||
|
||
**What it does NOT replace:** The `summary` column in the tag index schema, which is a human-written first-class router hint. Graphify extracts entities, not prose summaries. If summaries are the primary token-efficiency mechanism (see `02-system-design.md:98-102`), they need to remain author-controlled.
|
||
|
||
**Critical limitation:** Stale node drift. Graphify's `--update` does not prune deleted symbols/notes — you must `--force` rebuild to clear ghost nodes. The current design's SQLite is explicitly disposable-and-rebuildable from frontmatter; a Graphify graph is a snapshot with known drift. This is a real tradeoff.
|
||
|
||
**Recommendation:** Use Graphify's Obsidian sidecar export (`--obsidian`) to augment vault notes with auto-extracted entity metadata, and the MCP server for graph queries. Keep the summary frontmatter field as author-controlled. Consider Graphify as the **semantic enrichment step** that the deferred QMD layer was meant to fill — but without vectors.
|
||
|
||
### Project Files (code) — currently unindexed
|
||
|
||
**Graphify is a no-brainer add here.**
|
||
|
||
Project-file code analysis via AST is free (no LLM, no API key needed at runtime), deterministic, and produces exactly the call-graph / symbol map that "what does this codebase do?" queries need. Every client project gets a `graph.json` at zero ongoing cost. Query via the MCP server.
|
||
|
||
**Limitation from issue #1122:** `graphify extract` demands a backend credential even for pure-code extraction. Workaround: set a dummy key or use `--backend ollama` with a running (or not) Ollama instance — confirm at build time.
|
||
|
||
---
|
||
|
||
## The "Three Sources Connected" Question
|
||
|
||
Graphify does **not** natively connect three separate graphs. It builds one graph per ingestion run. To connect project files, vault, and session logs in one queryable surface, you'd need:
|
||
|
||
1. **A combined ingestion run** that points at all three source trees, OR
|
||
2. **Three separate graphs** queried via three MCP tool instances and synthesized at the Claude layer
|
||
|
||
Option 1 risks cross-contamination of unrelated client projects. Option 2 is the safer model: each source has its own graph, and Claude stitches results from three MCP calls.
|
||
|
||
The more honest framing: Graphify replaces tags as the **vault entity index**, adds a **free code graph per project**, and leaves episodic (memsearch) alone. That's a meaningful simplification — you no longer need the Ruby/SQLite tag index CLI — but it's a layer replacement, not a unified connector.
|
||
|
||
---
|
||
|
||
## What Changes in the Current Design
|
||
|
||
| Current design | With Graphify |
|
||
|---|---|
|
||
| Ruby/Sequel/SQLite tag index CLI | Replaced by `graphify query` + MCP server |
|
||
| Manual `#tool/ #client/ #domain/` tagging discipline | Auto-extracted by local SLM on vault ingestion |
|
||
| `summary` frontmatter (author-written, router hint) | Unchanged — Graphify doesn't generate these |
|
||
| QMD semantic layer (deferred) | Graphify fills this role without vectors |
|
||
| memsearch (episodic) | Unchanged |
|
||
| Project files (unindexed) | Free code graph via AST |
|
||
|
||
**What's saved:** The entire Ruby CLI build (Step 2 on the critical path in `04-build-plan.md`) could be skipped. That's significant scope reduction.
|
||
|
||
**What's added:** A local Ollama SLM running on this machine for vault doc extraction. This is a new infrastructure dependency. On low-RAM machines, a 7B model at 4-bit takes ~5 GB VRAM/RAM.
|
||
|
||
---
|
||
|
||
## Open Questions This Creates
|
||
|
||
1. **Rebuild strategy**: How often do you rebuild the vault graph? On every SessionStart (expensive), on vault git commit (right cadence), or manually? The `--update` flag helps but stale-node drift means periodic `--force` rebuilds are needed.
|
||
|
||
2. **Model choice for vault extraction**: No official recommendation. Test Qwen2.5 7B and Phi-4 14B against a sample of your vault notes and review god-node quality before committing.
|
||
|
||
3. **Summary field fate**: If Graphify extracts entities but doesn't write summaries, does the human still maintain summary frontmatter? Or does a local SLM generate those too?
|
||
|
||
4. **Cross-client isolation**: Project code graphs should be per-client-project, not merged. How do you namespace them? Separate `graphify-out/` per project? Or a single graph with source labels as node properties?
|
||
|
||
5. **MCP server management**: Running `graphify.serve` for vault + N project graphs means N+1 MCP server instances. Is that manageable, or do you build a single meta-server?
|
||
|
||
---
|
||
|
||
## References
|
||
|
||
| Topic | Source |
|
||
|---|---|
|
||
| Code AST extraction (free, deterministic) | `docs/graphify/03-ingesting-code-ast.md:9–100` |
|
||
| Local SLM (Ollama) setup | `docs/graphify/05-local-models-and-backends.md:63–156` |
|
||
| Query verbs + bounded traversal | `docs/graphify/06-querying-and-god-nodes.md:51–93` |
|
||
| Stale node drift limitation | `docs/graphify/07-token-economics-and-updates.md:136–148` |
|
||
| Token savings by repo size | `docs/graphify/external-tips.md:31–45` |
|
||
| Knowledge layer + retained metadata (summary/tags) | `docs/memory-system/02-system-design.md` § Layer 2 |
|
||
| Graph rejection decisions (LightRAG) | `docs/memory-system/03-architecture-decisions.md:83, 145` |
|
||
| Build order + open questions | `docs/memory-system/04-build-plan.md` |
|