167 lines
7.7 KiB
Markdown
167 lines
7.7 KiB
Markdown
# Graph Connectivity Findings: Structure Extractor vs. Topic Clusterer
|
||
|
||
_Last updated: 2026-06-05_
|
||
_Status: Empirical findings — primary/measured data from 2026-06-05 session against the real `~/Documents/SecondBrain` vault (Graphify 0.8.31, qwen2.5-coder:7b)._
|
||
|
||
---
|
||
|
||
## Context
|
||
|
||
We regenerated the vault knowledge graph to inspect it visually, then ran a discriminating test
|
||
to determine why the graph appeared so fragmented in the initial screenshot — specifically,
|
||
whether the fragmentation was a build artifact or a genuine property of Graphify's extraction
|
||
behavior.
|
||
|
||
Two runs were compared:
|
||
|
||
1. **Cached run** — a replay of per-fixture isolated extractions (each fixture extracted in its
|
||
own temp directory, results assembled after the fact). This was the source of the initial
|
||
visual screenshot.
|
||
2. **Clean deep run** — a genuine single-pass extraction against the full vault, run from
|
||
scratch.
|
||
|
||
---
|
||
|
||
## Methodology
|
||
|
||
### Cached run (baseline)
|
||
Replayed previously-generated per-fixture outputs. Each note was extracted in isolation in a
|
||
separate temp directory (`/tmp/graphify-bench/vault-graph-viz/graphify-out/`). No single
|
||
invocation saw the full corpus.
|
||
|
||
### Clean deep run
|
||
Single invocation against the full vault:
|
||
|
||
```
|
||
graphify extract ~/Documents/SecondBrain \
|
||
--backend ollama \
|
||
--model qwen25-coder-7b-16k \
|
||
--max-concurrency 1 \
|
||
--token-budget 8000 \
|
||
--mode deep \
|
||
--exclude .obsidian
|
||
|
||
graphify cluster-only <out> --backend ollama
|
||
```
|
||
|
||
Output: `/tmp/graphify-bench/vault-graph-clean/graphify-out/`
|
||
|
||
Note: this run required the `reasoning_effort:"none"` patch applied to the installed
|
||
`graphify/llm.py` — a fragile workaround that is lost on `pip upgrade`. Track Graphify issue
|
||
tracker for an official fix before the build step.
|
||
|
||
---
|
||
|
||
## Comparison Table
|
||
|
||
| Metric | Cached run (replay, isolated) | Clean deep run (single-pass) |
|
||
|--------|-------------------------------|------------------------------|
|
||
| Nodes | 51 | 30 |
|
||
| Edges | 34 | 27 |
|
||
| Cross-note edges | 14 (41%) | 21 (78%) |
|
||
| Communities | 18 | 9 |
|
||
| Shared topic/category hub entity (e.g. "Pest Control") | none | none |
|
||
| Token cost | 0 (cache replay) | 38,590 in / 8,296 out (genuine) |
|
||
|
||
The clean run's project-config node (`niche-automation-prospecting`) went from 0 cross-note
|
||
links to 6 — it now references the cold-email research, 10DLC setup, RingCentral analysis, and
|
||
VoIP research notes.
|
||
|
||
---
|
||
|
||
## Conclusions
|
||
|
||
### 1. Part of the extreme fragmentation was a build artifact [primary/measured]
|
||
|
||
The cached graph replayed per-fixture extractions done in isolated temp directories. A clean
|
||
single-pass extraction nearly doubles the **proportion** of cross-note edges (41% → 78%), and
|
||
cuts community count from 18 to 9. A real production graph will be meaningfully more connected
|
||
than the initial screenshot suggested.
|
||
|
||
### 2. Graphify is a structure extractor, not a topic clusterer [primary/measured]
|
||
|
||
This is the load-bearing finding. Graphify creates edges where one note's text
|
||
references/cites another, or where two notes are semantically very close
|
||
(document-to-document). It does **not** invent emergent shared-category entity nodes.
|
||
|
||
In both runs — including the clean deep run at `--mode deep --token-budget 8000` — **no**
|
||
shared topic hub node appeared. No "Pest Control" node, no "Niche Prospecting" node, no
|
||
"Speed-to-Lead" node. The LLM found many intra-note entities and doc-to-doc edges, but did not
|
||
synthesize a thematic category node spanning unrelated notes on the same topic.
|
||
|
||
This means: the connective "spine" a human reader imagines must be an **actual hub note**,
|
||
authored with wikilinks. This is exactly the mechanism ADR-011 already specifies. The
|
||
empirical data confirms the design choice rather than opening a gap in it.
|
||
|
||
### 3. Migration scaffolding is a first-class build prerequisite [primary/measured, strategic]
|
||
|
||
**Practical consequence of conclusion 2.** Running `graphify extract` is the easy part. A real
|
||
practical query — "how do we do niche prospecting outreach for pest control?" — returned only 3
|
||
starting notes in **both** runs, and traversal could **not** reach the email templates / ACV
|
||
data / business-model notes. Those are in separate communities with no connecting edges.
|
||
|
||
Therefore: authoring hub notes + wikilinks for key concepts **during migration** is what makes
|
||
retrieval actually work. This is a **first-class build deliverable and a prerequisite for
|
||
useful retrieval** — not a "bulk import later" afterthought. This refines and reinforces
|
||
ADR-013 (build-first / migrate-incrementally): the migrate-incrementally phase must include hub
|
||
note authoring as a named deliverable, not just frontmatter schema migration.
|
||
|
||
See ADR-014 for the locked decision.
|
||
|
||
### 4. OPEN QUESTION: Do facet tags create graph edges? [explicitly UNRESOLVED — untested]
|
||
|
||
Every edge observed in both runs came from explicit references, wikilinks, or semantic
|
||
similarity between documents. No edge was observably produced by shared frontmatter facet tags
|
||
alone (e.g., two notes both tagged `client/X` or `tool/Y`).
|
||
|
||
**If facet tags do not create graph connectivity**, then:
|
||
- Tag-scoped retrieval ("all notes tagged `client/acme` + `tool/semrush`") is a **separate
|
||
tag-query path**, not graph traversal.
|
||
- The graph provides connection-based recall; the tag facets provide attribute-filtered recall.
|
||
These are complementary, not interchangeable.
|
||
- The design of graph-traversal retrieval skills must account for this separation.
|
||
|
||
This is the **key thing to verify before designing retrieval skills**. It was not tested in
|
||
either run. Do not assume facet tags create edges.
|
||
|
||
### 5. Caveats [primary/measured — honestly recorded]
|
||
|
||
**(a) Coarser granularity is a tradeoff, not strictly "better."** The clean run produced 30
|
||
nodes (one per document, roughly) because batching 4–5 notes per LLM call shifted extraction
|
||
toward document-level nodes + doc-to-doc edges rather than fine-grained intra-note sub-entities.
|
||
More cross-note edges, but less intra-note entity resolution. Whether this is better depends on
|
||
the query type — a behavioral tradeoff, not a dominance result.
|
||
|
||
**(b) A substantially larger extraction model may produce topic-hub entities — untested.** We
|
||
did not test a 14B or 30B model. However, this is **secondary**: the design already intends
|
||
human-authored hub notes as the connective spine (ADR-011). If a larger SLM does synthesize
|
||
hub entities, that's additive; the design doesn't depend on it.
|
||
|
||
**(c) The `reasoning_effort:"none"` patch is fragile.** This local patch to the installed
|
||
`graphify/llm.py` was required to complete the extraction run. It is lost on `pip upgrade`.
|
||
Track upstream for an official flag or mitigation before the system build step. Until then,
|
||
treat the installed version as pinned.
|
||
|
||
---
|
||
|
||
## Pointers to Disposable Outputs
|
||
|
||
- **Cached run output**: `/tmp/graphify-bench/vault-graph-viz/graphify-out/` — replay of
|
||
per-fixture isolated extractions; **not** representative of production behavior.
|
||
- **Clean deep run output**: `/tmp/graphify-bench/vault-graph-clean/graphify-out/` — genuine
|
||
single-pass extraction; the reference for production behavior.
|
||
|
||
Both are disposable (per ADR-008: indexes are rebuildable, never synced).
|
||
|
||
---
|
||
|
||
## Cross-References
|
||
|
||
- **ADR-011** — Six-facet tag taxonomy + hub notes + wikilinks as the relationship mechanism
|
||
(empirically validated for the hub-notes/wikilinks half; open question on the facet-tag half)
|
||
- **ADR-013** — Build-first / migrate-incrementally (refined: migration phase must include hub
|
||
note authoring as a named deliverable)
|
||
- **ADR-014** — Decision locked from this finding: graph connectivity comes from authored
|
||
structure; migration scaffolding is a first-class prerequisite
|
||
- `docs/memory-system/06-graphify-evaluation.md` — earlier Graphify fit assessment (pre-empirical)
|