Compare commits

...

10 Commits

Author SHA1 Message Date
jared 6a5b09f253 Archive graphify-ollama-setup and sync new main specs
Move the completed graphify-ollama-setup change to changes/archive/2026-06-05-graphify-ollama-setup/ and promote its two delta specs into the stable spec set: local-model-selection and vault-graph-build. Verification passed with no critical issues; 16/16 tasks done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-05 11:48:02 -04:00
jared 3e38398500 Complete graphify-ollama-setup OpenSpec change
Executes all 16 tasks: local extraction-model benchmark with qwen2.5-coder:7b
selected as the default (7B parameter count, multi-modal, outperforms Mistral on
structured output), disposable vault graph build workflow, and supporting docs.
Adds extraction model options handbook, updates implementation process with
reconciliation strategy, and archives the scoring results.
2026-06-05 11:44:40 -04:00
jared 3823bb7c91 Record post-reboot GPU timing and mark deferred #3 resolved
GPU is fully resident and functional post-reboot (~74 tok/s steady-state).
Driver mismatch is resolved; gemma4:e4b confirmed at/above section C's
estimate. Deferred follow-up #3 (Confirm GPU timing post-reboot) marked
RESOLVED.
2026-06-04 14:52:21 -04:00
jared 65595c3995 Consolidate local-LLM gut-check findings and update pointers
Add new doc docs/memory-system/benchmark/local-llm-findings-2026-06-04.md
consolidating the local-LLM doc-extraction gut-check run. Key findings:
gemma4:e4b is the best installed model and adequate for the extraction
role (below frontier haiku but acceptable, catching ~45–60% of Opus's
high-value entities); GPU went unused due to NVIDIA driver version
mismatch (fix = reboot, pending); and Graphify owns the ollama call
(HTTP API, prompt/chunking/context/parsing internal) so raw-ollama
tuning is not production config. Add progressive-disclosure pointers
from 04-build-plan.md and 05-implementation-process.md.
2026-06-04 14:37:39 -04:00
jared 77355b4b44 Execute Step 2c: Claude reference-set benchmark — 18 gold-standard outputs
Completed the benchmark run that gates Ollama model selection. Generated 18
reference fragments (6 cross-domain fixtures × 3 Claude tiers) and tightened
extraction spec rules based on first-run learnings. Updated implementation
status to reflect that reference set is complete and Ollama scoring is
unblocked. Populated fixture list in dispatch-prompt with final selection.
2026-06-04 12:49:22 -04:00
jared b408a62136 Align 04-build-plan vault path with ADR-012
Replace the stale ~/brain placeholder with the locked vault location
~/Documents/SecondBrain so the build plan agrees with ADR-012 and the
runbook. Pre-existing inconsistency, surfaced during change verification.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-04 11:18:31 -04:00
jared 77febc8aca Add incremental-migration + reference-benchmark change (ADR-013)
Invert the build order to build-first / migrate-incrementally and
redesign Step 2c as a Claude reference-set benchmark.

- ADR-013 records the build-order inversion; CLAUDE.md locked-decisions
  pointer updated
- New benchmark deliverable under docs/memory-system/benchmark/:
  shared extraction spec + copy/paste dispatch prompt + reference-outputs
- Runbook (05): Step 1 = fixture selection with bulk migration deferred;
  Step 2c = Claude-tiers-only reference set, quality-only metrics, Opus
  as gold-standard rubric (produces references, not a model choice)
- 04-build-plan reconciled to the new build order and benchmark design
- Sync incremental-migration + reference-extraction-benchmark specs into
  openspec/specs/; archive the completed change

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-04 11:17:32 -04:00
jared 235fd45ff7 Update design docs and add implementation runbook per ADR-011/012
Update CLAUDE.md: clarify the knowledge layer now specifies the
~/Documents/SecondBrain vault, six-facet taxonomy, and reference the
locked decisions (ADR-011, ADR-012).

Update 02-system-design.md: align with faceted taxonomy (type/
client/project/domain/tool/convention/ plus scope/), express hierarchy
via hubs + wikilinks + Graphify edges rather than nested paths, name
the vault explicitly, update build-plan timestamp.

Add 05-implementation-process.md: concrete build runbook integrating the
locked decisions — seven steps bottom-up from vault migration through
plugin packaging, with the Graphify + Ollama model benchmark (Step 2c)
marked as the critical gate. Open questions deferred or defaulted; most
are non-blocking.

Design milestone: tag taxonomy and vault location locked.
2026-06-04 09:35:43 -04:00
jared f73cb46a53 ADR-011: Faceted tag taxonomy; ADR-012: reuse SecondBrain vault
Lock two design decisions: (1) six independent namespaced tag facets
(type/client/project/domain/tool/convention/) plus scope/, with
hierarchy expressed via hub notes and Graphify edges, not nested paths;
(2) adopt the existing ~/Documents/SecondBrain vault as the knowledge
layer rather than creating a new one — it is already flat, governs to
the correct semantics, and contains useful patterns (proactive-query
spec, tag-inference table).

See ADR-011 and ADR-012 for rationale, rejected alternatives, and
migration cost (mechanical schema updates on existing ~20 notes).
2026-06-04 09:35:38 -04:00
jared aa1ffe30fe Remove git commands from permitted direct tool uses
Under the new orchestrator-subagent pattern, version control is
delegated — the orchestrator (Claude) dispatches git operations
to subagents rather than calling them directly. Update the allowed
operations list to reflect this constraint.
2026-06-04 08:27:00 -04:00
49 changed files with 4879 additions and 51 deletions

View File

@ -45,10 +45,9 @@ the design, update this paragraph (and the relevant `docs/memory-system/` files
match.** Keep it accurate, don't preserve it for its own sake.
Two memory types kept as **separate systems**: **episodic** ("what happened, when") handled by
**memsearch** (Milvus Lite, embedded), and **semantic/knowledge** ("how do we…") handled by a
flat **Obsidian markdown vault** as the single source of truth. Notes keep `summary` +
namespaced tags (`tool/`/`client/`/`domain/`/`convention/`/`scope/`) as metadata, and the vault
is queried via a **Graphify knowledge graph** (local Ollama SLM for doc extraction, free
**memsearch** (Milvus Lite, embedded), and **semantic/knowledge** ("how do we…") handled by the
existing **`~/Documents/SecondBrain` Obsidian vault** as the single source of truth. Notes keep
`summary` + six flat, parallel namespaced facets (`type/`/`client/`/`project/`/`domain/`/`tool/`/`convention/`) plus `scope/` as metadata; hierarchy and relationships are expressed via hub notes (`type/hub`), wikilinks, and Graphify graph edges — not nested tag paths. The vault is queried via a **Graphify knowledge graph** (local Ollama SLM for doc extraction, free
tree-sitter AST for code). Retrieval is hook-injected + on-demand so project repos stay thin;
freshness is lazy
(write-time hook + SessionStart reconcile, no daemon/cron); the vault syncs to a VPS while
@ -59,6 +58,8 @@ tag-index CLI and also covers the deferred QMD semantic layer. `04-build-plan.md
`06-graphify-evaluation.md` reflect this; if an older doc still describes the Ruby CLI, defer
to those two and fix the stale doc.
**Decisions locked (2026-06-04):** Six-facet tag taxonomy + `scope/` (ADR-011); reuse `~/Documents/SecondBrain` vault rather than creating a new one (ADR-012); build-first / migrate-incrementally — build full system against a fixture set first, defer bulk vault migration to last, onboard projects one at a time (ADR-013).
## OpenSpec workflow
Changes are managed spec-driven via OpenSpec. Use the matching skills rather than editing spec
@ -89,7 +90,6 @@ Delegate all file I/O and shell commands to subagents via the Agent tool. No exc
**Permitted direct tool uses — only these, no others:**
- **Git commands** (`git status`, `git log`, `git diff`, `git commit`, `git push`) — version control is orchestrator-level.
- **Skill invocations via the Skill tool** — the skill handles its own operations.
- **Conversational responses requiring zero tool calls.**

View File

@ -31,6 +31,10 @@ then verified and corrected against the official GitHub repository and supplemen
command quick-reference + setting-up-across-many-projects mini-guide. **Keep this one open while working.**
10. **[external-tips.md](external-tips.md)** — Independent/community tips, gotchas with issue links, and an
even-handed look at the token-savings debate.
11. **[10-extraction-model-options.md](10-extraction-model-options.md)** — Why Graphify uses a general
structured-output LLM (not a purpose-built KG extractor), the architecture constraints that make
drop-in specialist models (Triplex, GLiNER, REBEL) non-starters, and an honest assessment of whether
the Triplex adapter route is worth experimenting with.
## One-paragraph summary

View File

@ -138,7 +138,9 @@ graphify extract ./docs --backend ollama --api-timeout 900 # 15-minute timeout
### Which Ollama model?
**There is no official Graphify-recommended/tested model list.** Neither the v8 README nor the site publishes one (verified — see Open questions). So follow the rule the task itself sets and the interview implies: **pick an instruction-following model that fits your RAM/VRAM.** `[interview]` Document extraction is "read this chunk, emit structured entities/relationships," which is an instruction-following + structured-output job, not deep reasoning.
**There is no official Graphify-recommended/tested model list in the README or on the site.** Neither the v8 README nor the site publishes one (verified — see Open questions). So follow the rule the task itself sets and the interview implies: **pick an instruction-following model that fits your RAM/VRAM.** `[interview]` Document extraction is "read this chunk, emit structured entities/relationships," which is an instruction-following + structured-output job, not deep reasoning.
> **Correction — there IS a shipped code default (2026-06-05).** Although the README/site publish no model recommendation, installed graphify 0.8.31 **hardcodes** a fallback in `llm.py:67`: `"default_model": os.environ.get("OLLAMA_MODEL", "qwen2.5-coder:7b")`. `[github]` So if you run `graphify extract --backend ollama` without setting `OLLAMA_MODEL`, the binary silently uses `qwen2.5-coder:7b`. The README's silence on a recommended model is accurate as a documentation statement; it is not accurate as a claim that "no default exists." The choice of a *coder* model for document extraction is a revealed preference — the binding requirement is structured-JSON instruction-following discipline, not domain NER. See [10-extraction-model-options.md](10-extraction-model-options.md) for the full analysis.
Rough sizing rule of thumb: a model needs roughly its parameter count in **GB of (V)RAM** at common 4-bit quantization (a 7B model ≈ ~5 GB; a 30B+ model wants 24 GB+ VRAM). Leave headroom for the context window.
@ -195,7 +197,7 @@ graphify extract ./src
## Open questions / unverified
- **No official Ollama model recommendation exists.** Verified absent from the v8 README env/backend sections and not surfaced on the (403-gated) official site. The model names here are community/general-Ollama signals, tagged as such — not Graphify-tested defaults.
- **No official Ollama model recommendation exists in the README**, but the installed binary hardcodes `qwen2.5-coder:7b` as the Ollama fallback (`llm.py:67`) — see the Correction callout above and `10-extraction-model-options.md`. `[github]`
- **`GRAPHIFY_OLLAMA_KEEP_ALIVE` default value** is documented by *behavior* ("minutes to keep loaded; `0` to unload after each chunk") but the README does not state the numeric default. `[github]`
- **`gemma3:27b` exact tag / VRAM numbers** come from a single user issue (#792), not Graphify docs. The original ASR transcript said "Gemma 4 31b," which does not match a shipped tag; treat the community `gemma3:27b`-class figure as illustrative, not authoritative. `[community](https://github.com/safishamsi/graphify/issues/792)`
- **The "SLMs as good as frontier models soon" vision** is the creator's aspiration, not a benchmark. `[unverified claim]` / `[interview]`

View File

@ -0,0 +1,131 @@
# 10 — Extraction Model Options: Why a General Structured-Output LLM, and Is a Purpose-Built KG Model Worth It?
_Last updated: 2026-06-05_ | _Status: reference / decision record_
This document explains why Graphify's document extraction uses a general instruction-following / structured-output LLM rather than a purpose-built knowledge-graph extraction model (e.g. Triplex, GLiNER, REBEL), and honestly assesses whether the purpose-built route is worth experimenting with for this project.
See also: [05 — Local models & backends](05-local-models-and-backends.md) · [Benchmark results (2026-06-04)](../memory-system/benchmark/scoring-results-2026-06-04.md)
---
## The architecture constraint (this is the crux)
Graphify owns **both** the prompt and the output contract. It is not a thin wrapper you can swap a model into — the extraction pipeline is opinionated end-to-end:
**Fixed system prompt.** For document/markdown extraction, Graphify sends its own system prompt `_EXTRACTION_SYSTEM` (`llm.py:218-232`). `[github]` There is no user hook to supply a custom extraction prompt; the prompt is hardcoded.
**Fixed output schema.** The model must return exactly this JSON structure — no approximation accepted:
```json
{
"nodes": [{"id": "stem_entity", "label": "...", "file_type": "code|document|paper|image|rationale|concept",
"source_file": "...", "source_location": null, "source_url": null,
"captured_at": null, "author": null, "contributor": null}],
"edges": [{"source": "node_id", "target": "node_id",
"relation": "calls|implements|references|cites|conceptually_related_to|shares_data_with|semantically_similar_to",
"confidence": "EXTRACTED|INFERRED|AMBIGUOUS",
"confidence_score": 1.0, "source_file": "...", "source_location": null, "weight": 1.0}],
"hyperedges": [], "input_tokens": 0, "output_tokens": 0
}
```
The `relation` field is constrained to a fixed vocabulary of seven values. The `confidence` field is a three-value enum (`EXTRACTED`/`INFERRED`/`AMBIGUOUS`). `[github]` (`llm.py:218-232`)
**No JSON-mode / grammar enforcement.** For the Ollama backend, Graphify passes only `{"options": {"num_ctx": ...}, "keep_alive": ...}` as extra parameters (`llm.py:506`). `[github]` There is no `response_format`, grammar constraint, or structured-output enforcement. Schema compliance rests entirely on the model's instruction-following ability.
**Lenient but not magic parser.** The response goes through `_parse_llm_json` (`llm.py:269-303`), which strips optional markdown fences and then tries `json.loads`. If that fails, it scans for the first balanced JSON object in the text. A model that emits its own schema — triples, spans, BIO tags — will produce an empty or nonsense graph, not a graceful fallback. `[github]`
**Temperature forced to 0 for Ollama.** The Ollama backend config sets `"temperature": 0` (`llm.py:65-73`). `[github]`
**Deep mode.** `--mode deep` appends `_DEEP_EXTRACTION_SUFFIX` to the same prompt, asking for extra `INFERRED` edges with a more conservative framing (`llm.py:234-247`). `[github]` The schema stays identical.
---
## The shipped default: why a coder model?
The Ollama fallback default is `qwen2.5-coder:7b` (`llm.py:67`): `[github]`
```python
"default_model": os.environ.get("OLLAMA_MODEL", "qwen2.5-coder:7b"),
```
Even though Graphify sends *documents* (not code) for extraction, the maintainer chose a coder/structured-output-tuned model. This is a revealed preference: the binding requirement is **structured-JSON instruction-following discipline**, not domain entity recognition. A coder/instruct model is the right axis; a NER specialist is the wrong one; frontier reasoning is overkill.
**Config knobs for the Ollama backend** (all `[github]` from README env-var table unless noted):
| Knob | How to set | What it does |
|---|---|---|
| `OLLAMA_MODEL` | env var | Override the `qwen2.5-coder:7b` default |
| `OLLAMA_BASE_URL` | env var | Must end in `/v1` (e.g. `http://127.0.0.1:11434/v1`) — any other suffix produces 404s on every call |
| `GRAPHIFY_OLLAMA_NUM_CTX` | env var | **See GOTCHA below** |
| `GRAPHIFY_OLLAMA_KEEP_ALIVE` | env var | Minutes to keep model resident; default `30m`; set `0` to unload after each chunk |
| `--token-budget` | CLI flag | Per-chunk input cap (tokens); pack multiple files per chunk |
| `--max-concurrency` | CLI flag | Set 12 for a single local GPU |
| `--mode deep` | CLI flag | Appends deep-inference suffix; elicits more INFERRED edges |
**GOTCHA — num_ctx does not propagate through graphify (verified 2026-06-04 on this project).** `[github]` Graphify posts to Ollama's OpenAI-compatible `/v1/chat/completions` endpoint. That endpoint **silently ignores** the per-request `options.num_ctx` that Graphify sends (`llm.py:506`). Proven by A/B: a POST to `/v1/chat/completions` with `num_ctx=8192` left `ollama ps` showing CONTEXT=4096; the same value through the native `/api/chat` endpoint was honoured. Therefore `GRAPHIFY_OLLAMA_NUM_CTX` has **no effect** through Graphify — context pins at Ollama's `/v1` default of 4096. At 4096 tokens, Graphify's extraction output JSON is truncated mid-response (`finish_reason=length`) and the chunk is **discarded**, producing an empty graph.
**Workaround (validated):** bake context into a Modelfile variant:
```bash
# Create a 16 k-context variant of qwen2.5-coder:7b:
cat <<'EOF' | ollama create qwen25-coder-7b-16k -f -
FROM qwen2.5-coder:7b
PARAMETER num_ctx 16384
EOF
OLLAMA_MODEL=qwen25-coder-7b-16k graphify extract ./docs --backend ollama
```
The `/v1` endpoint honours the model's baked-in default. Non-invasive: no sudo, no systemd restart; reversible with `ollama rm`. Full investigation: `docs/memory-system/benchmark/scoring-results-2026-06-04.md`.
**Local patch — thinking mode disabled (applied 2026-06-04):** `reasoning_effort: "none"` was added at `llm.py:71` in the installed binary to suppress thinking output for any thinking-capable model. The original is backed up at `/tmp/graphify-bench/llm.py.orig`. This patch is a **no-op for `qwen2.5-coder:7b`** (no thinking mode) but will be **lost on `pip install --upgrade`** and would need re-applying for any future thinking-capable candidate. Needs a production path: upstream PR or a maintained local wrapper. `[github]`
---
## Purpose-built KG / entity / relation models — and why they don't drop in
Every purpose-built extraction model assumes **it** owns the prompt and output contract. Graphify owns both. That's the structural incompatibility.
| Model | Architecture | Ollama-pullable? | Follows Graphify's fixed prompt + schema? | Verdict |
|---|---|---|---|---|
| **SciPhi/Triplex** (Phi-3-3.8B, fine-tuned for KG triples) | Autoregressive causal LM | Yes (`ollama pull sciphi/triplex`) | No — locked to its own `{entity_types}/{predicates}` input template; emits subjectpredicateobject triples, not Graphify's nodes/edges JSON | Can't drop in as-is |
| **GLiNER** (lightweight NER) | BERT-like bidirectional encoder | No | No — NER-only (span outputs); cannot emit arbitrary JSON or follow a system prompt | Wrong architecture |
| **REBEL / Relik** (relation extraction) | Seq2seq BART with special-token triples | No | No — emits `<triplet>/<subj>/<obj>` tokens; custom parsing required | Wrong architecture |
Sources: HuggingFace SciPhi/Triplex model card, `ollama.com/library/sciphi/triplex`, GitHub `urchade/GLiNER`, GitHub `Babelscape/rebel`. `[unverified claim]` — line numbers in those repos were not pinned.
---
## Is the purpose-built route worth experimenting with?
**User hypothesis:** if the JSON-shape mismatch is the only blocker, write a shim converting Triplex's triple output into Graphify's schema with a custom prompt. Triplex is marketed as very low-cost/fast (small 3.8B Phi-3 model).
What it would actually take `[speculative]`:
1. **A model-specific prompt path.** Triplex requires its `{entity_types}/{predicates}` template, not `_EXTRACTION_SYSTEM`. Graphify currently hard-codes one system prompt; a Triplex adapter needs to inject a different one for that model (or backend).
2. **A model-specific output parser.** Convert subjectpredicateobject triples into Graphify's `nodes` + `edges` JSON: assign `file_type` to synthesized nodes, map predicates onto Graphify's fixed relation vocabulary (or accept free-text), synthesize `confidence` since Triplex emits no `EXTRACTED`/`INFERRED`/`AMBIGUOUS` enum, and reconstruct per-file provenance fields (`source_file`, `source_location`).
3. **A new backend/model branch in `llm.py`.** This is not a one-line patch — it is a backend adapter (prompt + parser) of moderate size.
**Architecturally consistent.** Graphify already has model-specific branches: the Kimi/moonshot backend sets `extra_body={"thinking": {"type": "disabled"}}` (`llm.py:461-462`) `[github]` to suppress reasoning output. A Triplex adapter follows the same pattern and could in principle be an upstream PR.
**Quality unknowns `[speculative]`.** Triplex targets generic NER + triples. Graphify's design wants typed nodes, a fixed relation vocabulary, EXTRACTED/INFERRED/AMBIGUOUS confidence, and per-file provenance. None of those would be native outputs from Triplex — they'd be synthesized, lowering fidelity compared to a model that follows Graphify's schema directly.
**Cost/speed `[speculative]`.** Triplex is plausibly faster per token (3.8B vs 7B), but `qwen2.5-coder:7b` at Q4 on a 12GB GPU is already rapid. The adapter effort isn't justified unless the default model proves too slow or too heavy for the freshness budget.
**Verdict `[speculative]`:** Realistic but moderate effort. Worth it **only if**: (a) `qwen2.5-coder:7b` proves too slow or too heavy for the project's freshness budget on the full vault, **and** (b) a quick Triplex spike shows acceptable triple quality on the vault fixtures. Neither condition is known yet. **PARKED — revisit after the qwen2.5-coder:7b benchmark result is in hand.**
**Cheap first probe:** `ollama run sciphi/triplex` on one fixture note using Triplex's native `{entity_types}/{predicates}` template. If the triples are coherent and well-typed, the adapter is worth scoping. If they are sparse or incorrectly typed, don't bother.
---
## Why qwen2.5-coder:7b is the right axis, not a domain specialist
To be concrete: the extractors that do better than `qwen2.5-coder:7b` at Graphify's actual task are models that are **better at structured-JSON instruction following**, not models that are better at entity recognition or triple extraction in isolation. Relevant axis: instruction-following + JSON discipline. Irrelevant axis: NER F1, triple-completeness on academic benchmarks. `[speculative]` (Based on observed failure modes: models that failed in the 2026-06-04 benchmark failed by producing invalid JSON or consuming the output budget with thinking tokens — not by extracting the wrong entities.)
---
## Pointers
- Full local-model investigation + the num_ctx/thinking findings: `docs/memory-system/benchmark/scoring-results-2026-06-04.md`
- Backend config details and Ollama setup: `docs/graphify/05-local-models-and-backends.md`
- Graphify source (installed): `~/.local/lib/python3.14/site-packages/graphify/llm.py` — anchored to 0.8.31

View File

@ -1,7 +1,9 @@
# System Design
_Status: approved 2026-06-03; knowledge layer revised 2026-06-04 (Graphify replaces the Ruby
tag-index CLI and the deferred QMD layer — see ADR-010). Implementation not yet started._
tag-index CLI and the deferred QMD layer — see ADR-010); tag taxonomy and vault location locked
2026-06-04 (six-facet taxonomy — see ADR-011; reuse ~/Documents/SecondBrain — see ADR-012).
Implementation not yet started._
## Goals (what this system must do)
@ -39,8 +41,8 @@ forcing one tool to do both is what made every earlier design feel forced.
│ auto-captured session/journal notes · NL semantic recall │
│ answers "when did we…", "what was I doing yesterday" │
├─────────────────────────────────────────────────────────────────┤
│ KNOWLEDGE ── flat Obsidian vault (single source of truth)
│ notes carry summary + namespaced tags (metadata)
│ KNOWLEDGE ── ~/Documents/SecondBrain Obsidian vault (single SOT)
│ notes carry summary + six facet tags + scope/ (metadata)
│ + Graphify knowledge graph (local SLM over docs; AST over code) │
│ graph queries (god nodes / query / path) · answers "how do we…",│
│ "what do we know about X for client Y", "what relates to Y" │
@ -72,8 +74,7 @@ filters), it is just no longer backed by a bespoke index.
The heart of the system, and the part we build.
### Vault
- **Flat markdown directory**, single source of truth, **configurable location** (NOT forced
into `~/.claude/`; symlink if a tool insists). Browsable in Obsidian as a viewer.
- **Flat markdown directory**, single source of truth — reuses the existing **`~/Documents/SecondBrain`** Obsidian vault (ADR-012); not forced into `~/.claude/`; symlink if a tool insists. Browsable in Obsidian as a viewer.
- **Replaces project-local documentation**: instead of docs scattered per repo, knowledge
lives once in the vault and is pulled into any project on demand.
@ -83,14 +84,20 @@ The heart of the system, and the part we build.
summary: One line, written at creation. The router shows this so the AI can pick a
file without opening it.
tags:
- tool/semrush # namespaced, nested (slash = Obsidian nested tag)
- type/reference # listed first by convention; e.g. type/hub, type/how-to
- client/sesame3g
- project/website-redesign
- domain/seo
- tool/semrush
- convention/api-style
- scope/project # or scope/global
---
```
- **Namespaces** are the "virtual indexes": `tool/`, `client/`, `domain/`, `convention/`,
`scope/`. `#tool` matches all children — native prefix filtering, no folders needed.
- **Six flat facets** (ADR-011): `type/`, `client/`, `project/`, `domain/`, `tool/`,
`convention/` — plus `scope/`. Each facet is independent and parallel (never nested into each
other). `#tool` matches all `tool/*` values — native Obsidian prefix filtering, no folders
needed. Hierarchy and relationships are expressed via **hub notes** (`type/hub`),
**wikilinks**, and **Graphify graph edges**, not nested tag paths.
- **Two knowledge scopes** via `scope/global` vs `scope/project` (+ a `client/` tag): global =
broadly useful tool/domain knowledge; project = how a specific client uses it. Both are
globally queryable; the scope tag is the shortcut that avoids scanning every client's usage.
@ -113,11 +120,12 @@ layer (ADR-010): one graph gives both structured and semantic retrieval, without
- **Query** (via CLI and an MCP server exposing `query_graph` / `get_node` / `shortest_path`):
ask for **god nodes first**, then scalpel down with `graphify query` / `path` / `explain`.
Prompt the graph; don't dump the corpus into context.
- **Metadata still matters**: the `summary` + namespaced tags remain first-class note
- **Metadata still matters**: the `summary` + six facet tags remain first-class note
attributes — `summary` is the human-written router hint Graphify does **not** generate, and
the `tag/` namespaces stay useful for Obsidian filtering and as node attributes. They are
retained even though they no longer back a bespoke index. *(How tightly metadata feeds graph
queries is a refinement for build time.)*
the facet namespaces (`type/`, `client/`, `project/`, `domain/`, `tool/`, `convention/`,
`scope/`) stay useful for Obsidian filtering and as node attributes. They are retained even
though they no longer back a bespoke index. *(How tightly metadata feeds graph queries is a
refinement for build time.)*
- **Source of truth rule**: markdown is authoritative; the graph (`graphify-out/`) is a
rebuildable artifact that is **never synced** and can be deleted/rebuilt anytime
(`graphify ... --force`).
@ -179,6 +187,6 @@ similarity would clearly win, revisit then (the video's "only level up when it b
| Goal | Met by |
|------|--------|
| 1. Thin projects | Knowledge in the vault, not repos; CLAUDE.md holds tags/pointers; on-demand `index query` |
| 2. Cross-project/client knowledge, global vs project scopes | Flat vault + namespaced tags + `scope/` + `client/`; Graphify knowledge graph (god nodes + traversal) over it |
| 2. Cross-project/client knowledge, global vs project scopes | `~/Documents/SecondBrain` vault + six-facet tags (`type/`/`client/`/`project/`/`domain/`/`tool/`/`convention/`) + `scope/`; Graphify knowledge graph (god nodes + traversal) over it |
| 3. Timeline | memsearch episodic layer + session-end journal hook |
| 4. Remote, local-fast | Markdown vault synced via git/Syncthing; disposable per-machine graphs/indexes |

View File

@ -1,5 +1,7 @@
# Architecture Decision Records
_Last updated: 2026-06-04_
A running log of decisions and *why*. Format per entry: Context · Decision · Rationale ·
Alternatives rejected · Status. Newest decisions extend the log; supersede rather than delete.
@ -47,7 +49,8 @@ Alternatives rejected · Status. Newest decisions extend the log; supersede rath
Pure-prefix path filtering via memsearch `source_prefix` (would force directories back in).
- **Trade-off accepted**: Tags give the *human/Obsidian* free filtering, but the *AI* gets
nothing for free from tags — we must materialize them into a queryable index (see ADR-004).
- **Status**: Accepted.
- **Status**: **Refined by ADR-011** (type/ and project/ namespaces added; hierarchy-vs-facets
clarified). Core decision — flat vault, namespaced tags — stands.
## ADR-004 — SQLite + Sequel (Ruby) tag index as the knowledge-layer cache
@ -176,6 +179,122 @@ Alternatives rejected · Status. Newest decisions extend the log; supersede rath
suffices).
- **Status**: Accepted (to be built — see 04-build-plan.md and 06-graphify-evaluation.md).
## ADR-011 — Faceted tag taxonomy: six independent namespaces (refines ADR-003)
_Date: 2026-06-04_
- **Context**: ADR-003 introduced five namespaces (`tool/`, `client/`, `domain/`,
`convention/`, `scope/`). During vault-reuse assessment (ADR-012) it became clear that (1)
the existing SecondBrain vault uses a de-facto first-tag convention for note kind
(research/plan/log/adr/howto) that should be made explicit and machine-queryable, and (2)
for a freelancer working many projects per client, project identity deserves a first-class
namespace rather than being implied by `client/` or `domain/`.
- **Decision**: Knowledge-vault notes are classified by **six independent, flat tag facets**
that sit side-by-side, never nested into one another:
- `type/` — note kind: `research`, `howto`, `adr`, `hub`, `plan`, `log`, `clip`, etc.
- `client/` — which client
- `project/` — which project (first-class; a freelancer's projects are the primary unit of
work)
- `domain/` — knowledge domain / topic area
- `tool/` — tool-specific knowledge
- `convention/` — conventions
- …plus `scope/global` or `scope/project` (retained from ADR-003)
Hierarchy and relationships are expressed via **hub notes** (`type/hub`), **wikilinks**, and
**Graphify knowledge-graph edges** — NOT via nested tag paths.
By convention `type/` is listed **first** in frontmatter, preserving the SecondBrain vault's
existing type-first ordering habit and making the note kind immediately visible.
- **Rationale**: The vault is flat — hierarchy is not expressed through folder paths or tag
nesting. The user's reality is many-to-many (many projects per client, knowledge domains
spanning clients), which a single-parent tree models badly and forces false hierarchy. A
project hub note links out to both its `client/` and relevant `domain/` tags rather than
being buried under either. Per-type `_templates` will be provided for **core types only**
(research, howto, adr, hub); the long tail stays freeform until a pattern earns a template.
Consistent per-type structure also improves Graphify's local-SLM extraction reliability.
- **Alternatives rejected**: Hierarchical nesting in the style of John Conneely's
`domain/{product}/{project}.md` folder structure (from the youngleaders.tech article "How I
finally sorted my Claude Code memory" — **secondary/interview-grade source, not verified
against primary implementation**). Rejected because: (1) the vault is flat — hierarchy is
not expressed through folder paths; (2) the user's many-to-many reality maps badly onto a
single-parent folder tree and forces false hierarchy; (3) nesting one facet through another
(e.g. `domain/client/project`) creates Law-of-Demeter-style traversal coupling. Conneely's
structure was the inspiration but diverges here on hierarchy-vs-facets. Faceted parallel tags
are the flat-vault analogue of what the Graphify graph already does with edges, so they
compose naturally with the chosen knowledge layer.
- **Status**: Accepted (supersedes the namespace list in ADR-003; core flat-vault +
namespaced-tags decision stands).
## ADR-012 — Reuse the existing SecondBrain vault as the knowledge vault
_Date: 2026-06-04_
- **Context**: The design called for a flat markdown vault as the semantic knowledge layer
(ADR-003/008/010). The question was whether to stand up a new `~/brain` vault from scratch
or adopt the existing `~/Documents/SecondBrain` vault.
- **Decision**: **Adopt `~/Documents/SecondBrain`** as the knowledge vault rather than
creating a new vault.
- **Rationale**: Assessment found the SecondBrain vault is already flat (all notes at root,
only a `_templates/` exception — exactly what the design permits), already articulates the
correct "durable knowledge, not working memory" role in its `CLAUDE.md` and
`vault-conventions.md`, and contains ~20 real notes. It also includes two patterns that
**improve on the current design** and should be adopted:
1. `vault-conventions.md`'s "act without being asked" section specifying *when* the AI
should proactively query the vault — a behavioral spec the cc-os docs lacked.
2. Project-config hub notes with a tag-inference table (auto-tag by path pattern) that
operationalizes *how* to tag a note from a given project.
- **Adaptations required (migration cost)**:
- Add `summary:` frontmatter to existing notes.
- Migrate flat unnamespaced tags to the six-facet namespaced form (per ADR-011).
- Add `scope/global` or `scope/project` to each note.
- Initialize git in the vault (no `.git` exists yet — required by ADR-008's sync strategy).
- Replace the vault's `~/.claude/scripts/vault_search.rb` reference (script does not exist)
with `graphify query` (ADR-010).
These are mechanical schema migrations, not structural rework.
- **Alternatives rejected**: Starting fresh with a new `~/brain` vault. Rejected because the
hardest design decision — flat structure, durable-knowledge-only role, governance philosophy
— is already made and practiced in SecondBrain. The improved behavioral patterns
(proactive-query spec, tag-inference table) and the existing notes are worth preserving; the
remaining work is mechanical migration.
- **Status**: Accepted.
## ADR-013 — Build-first / migrate-incrementally (build-order inversion)
_Date: 2026-06-04_
- **Context**: The build runbook (`05-implementation-process.md`) originally front-loaded bulk
vault migration as Step 1 — migrating all ~20 existing SecondBrain notes and all projects to
the ADR-011 six-facet taxonomy before the system existed to validate them. This committed to
a schema and workflow (the tag taxonomy from ADR-011, the vault-reuse choice from ADR-012,
and Graphify extraction behavior) before any end-to-end path had been exercised. The risk:
locking in an approach that fails at scale, with no feedback loop until the entire vault has
been touched.
- **Decision**: **Invert the build order.** The full system is built and validated against a
small **510 note fixture set** first. Bulk vault migration is deferred to the final stage.
The first real-data validation uses **one small project that contains both code AND
documents**, exercising both the local-SLM doc-extraction path and the tree-sitter code path
in the same run. After that single project validates end-to-end, remaining projects are
onboarded **one at a time** with an observe-and-adjust step between each.
- **Rationale**: Validates the ADR-011 taxonomy and ADR-012 vault conventions against the real
Graphify extraction pipeline before the entire vault is committed. The first mixed code+docs
project surfaces both extraction paths (SLM for docs, tree-sitter for code) early, when
corrections are cheap. Per-project rollout keeps the blast radius of any schema or workflow
correction small; each project is an opportunity to observe and adjust rather than discover
problems across 20 notes at once. This is consistent with the "markdown-as-truth, indexes are
disposable" principle (ADR-008): the vault notes are durable, but the extraction schema should
be validated before it shapes all of them.
- **Alternatives rejected**:
- **Keep migration-first (status quo)**: Front-loads all ~20 notes and all projects before
any end-to-end validation exists. Commits to ADR-011's taxonomy and ADR-012's migration
steps against the full vault without a feedback loop — exactly the gap this decision closes.
- **Big-bang migrate everything after build**: Build against fixtures, then migrate all notes
and all projects in one batch at the end. Avoids the pre-build commitment problem but still
risks a single large irreversible migration with no observe-and-adjust loop between units.
Per-project rollout with intermediate checkpoints is strictly safer.
- **Cross-references**: ADR-011 (six-facet tag taxonomy — the schema being validated);
ADR-012 (SecondBrain vault reuse — the migration steps this order defers).
- **Status**: Accepted (updates `05-implementation-process.md` build order).
## Rejected tools (summary)
| Tool | Why rejected for our use |

View File

@ -1,5 +1,7 @@
# Build Plan
_Last updated: 2026-06-04_
How a human builds this system, step by step, and answers to the operational questions:
which scripts and hooks, how the AI knows when to write and what conventions to follow, how and
when it queries, the CRUD hooks, and how it's packaged as a global Claude Code plugin with
@ -14,6 +16,12 @@ vault and free AST-based code graphs per project. The deferred QMD semantic laye
is also skipped — Graphify covers it without vectors. See `06-graphify-evaluation.md` for
the full rationale.
**Architecture decision (2026-06-04):** Build order inverted (ADR-013). Build and validate
the full system against a small 510 note fixture set first; defer bulk vault migration to last.
Validate end-to-end on one small pilot project containing both code and documents before
onboarding any others. Onboard remaining projects one at a time, with observe-and-adjust between
each. Steps 2d and 2e below are updated accordingly.
---
## Part A — Build order (human builder's path)
@ -22,8 +30,9 @@ Build bottom-up: the vault and Graphify first (usable standalone), then the hook
plugin that packages it.
### Step 1 — Vault skeleton & conventions
- Decide the vault location (default: a synced home dir, e.g. `~/brain`; symlink into
`~/.claude/memory` only if a tool insists). **Vault is the single source of truth.**
- Vault location is settled by ADR-012: `~/Documents/SecondBrain` (the existing Obsidian
vault). No new vault is created; symlink into `~/.claude/memory` only if a tool insists.
**Vault is the single source of truth.**
- Write `CONVENTIONS.md` in the vault: the frontmatter contract. The required fields are
`summary` (one-line, author-written — this is the human-readable router hint Graphify
does not generate) and `scope` (`scope/global` or `scope/project`). Tags are now
@ -61,49 +70,68 @@ Available models (as of 2026-06-03, in order of interest):
- `qwen3.5:2b` — 2.7 GB, smallest, good fallback if VRAM is constrained
- `gemma4:e4b` — 9.6 GB, more capable, slower
#### 2c — Model benchmarking (before committing to a model)
#### 2c — Claude reference-set benchmark (THE GATE, before committing to a local model)
Run a small extraction test across all local models plus the three Claude API models. The goal
is to find the best speed/accuracy tradeoff for entity+relationship extraction from vault notes.
This step produces the **gold-standard reference set** — one structured extraction output per
fixture note per Claude tier. It does **not** choose the final extraction model, and it does
**not** measure speed.
**Test set:** 510 representative vault notes spanning different note types (tool note, client
note, convention note, domain note). Include one note that is dense with relationships.
**Input:** the 510 fixture notes selected in Step 1 (from the runbook).
**What to measure per model:**
1. Extraction speed (tokens/sec or wall-clock time per note)
2. Entity quality: are the right concepts extracted? Any hallucinated entities?
3. Relationship quality: are edges plausible and correctly typed? Missing relationships?
4. Confidence tag accuracy: are `INFERRED` vs `AMBIGUOUS` edges appropriately flagged?
**What to run:** dispatch one Claude Code subagent per tier (Claude-tiers only — Ollama models
are not reachable in this environment):
**Models to test:**
| Model | Backend | Notes |
| Tier | Model ID | Role |
|---|---|---|
| `gemma4:e2b` | Ollama local | Primary candidate — fast, 7.2 GB |
| `qwen3.5:2b` | Ollama local | Smallest, fastest |
| `gemma4:e4b` | Ollama local | Highest local quality |
| `claude-haiku-4-5` | Claude API | Baseline API option |
| `claude-sonnet-4-6` | Claude API | Mid-tier reference |
| `claude-opus-4-8` | Claude API | **Gold standard** — judge quality against this |
| Haiku | `claude-haiku-4-5` | Lightweight reference |
| Sonnet | `claude-sonnet-4-6` | Mid-tier reference |
| Opus | `claude-opus-4-8` | **Gold standard** (scoring rubric) |
Run using subagents in Claude Code: dispatch one agent per model, each extracting the same
test set, return structured JSON of entities+relationships. Review god-node quality in
`GRAPH_REPORT.md` after each run. Use Opus output as the scoring rubric for the local models.
Each subagent receives only the fixture note text plus the shared extraction spec — no design
docs, no project context (fairness contract). See ADR-013.
**Decision rule:** Choose the fastest local model whose entity/relationship quality is
"close enough" to Opus (subjective; likely Gemma4:e2b or Gemma4:e4b based on the description).
API models are a fallback option for high-stakes notes, not the default.
**Metrics — quality only (wall-clock speed is explicitly out of scope here):**
1. Entity correctness — right concepts extracted, no hallucinated entities
2. Relationship plausibility and typing — edges plausible, correctly typed, no missing key edges
3. Confidence-tag accuracy — `INFERRED` vs `AMBIGUOUS` applied appropriately
**Deliverables** (produced by this step):
- Dispatch prompt (copy/paste-able): `docs/memory-system/benchmark/dispatch-prompt.md`
- Shared extraction schema: `docs/memory-system/benchmark/extraction-spec.md`
- Per-tier outputs: `docs/memory-system/benchmark/reference-outputs/<note-slug>.<tier>.md`
Opus output is the rubric. **Deferred later step:** local Ollama models (gemma4:e2b, qwen3.5:2b,
gemma4:e4b) are timed AND scored against these references — that scoring run is where speed
re-enters and the final model is chosen. Do not hardcode a model before that run completes.
**Status (2026-06-04): EXECUTED.** 6 cross-domain fixtures × 3 Claude tiers = 18 reference
fragments generated in `benchmark/reference-outputs/`. Run as-is (no vault frontmatter
modification); verified clean. Fixtures listed in `benchmark/dispatch-prompt.md`.
Authoritative detail lives in `docs/memory-system/05-implementation-process.md` §2c. Local-model gut-check done (2026-06-04): `gemma4:e4b` is the candidate; GPU fix pending reboot; Graphify owns the ollama call — see `docs/memory-system/benchmark/local-llm-findings-2026-06-04.md`.
#### 2d — Initial fixture graph build (ADR-013: small-first)
Run the initial build against the small fixture set (510 notes from Step 1/2c), not the
full vault. The bulk vault build is deferred to after the system is validated end-to-end.
#### 2d — Initial vault graph build
```bash
graphify extract --path ~/brain --backend ollama --model gemma4:e2b \
graphify extract --path ~/Documents/SecondBrain --backend ollama --model gemma4:e2b \
--token-budget 512 --max-concurrency 2
```
Tune `--token-budget` (semantic chunk size) and `--max-concurrency` based on VRAM headroom.
Review `GRAPH_REPORT.md` — check god nodes make sense (they should be your most-connected
tools, clients, and domain concepts).
**Full vault migration** (the `~/Documents/SecondBrain` build above run over all notes) is the final step —
deferred to after end-to-end validation on the pilot project. Do not bulk-migrate the vault
until the system is verified working on the fixture set and pilot project.
#### 2e — Per-project code graphs (free, no model needed)
For each client project:
**ADR-013 order:** start with ONE pilot project that contains both code and documents; validate
end-to-end before onboarding others. Onboard remaining projects one at a time, with
observe-and-adjust between each. Do not batch all projects at once.
For the pilot project (and each subsequent project, one at a time):
```bash
graphify extract --path ~/projects/<client>/<project> --no-docs
```
@ -257,7 +285,8 @@ global install keeps conventions a single source of truth.
## Open questions
1. **Vault location**`~/brain` (synced home dir)? Symlink into `~/.claude/memory`?
1. **Vault location** — Settled by ADR-012: `~/Documents/SecondBrain`. Symlink into
`~/.claude/memory` only if a tool requires it.
2. **Sync mechanism** — git (versioned, hourly) vs Syncthing (continuous)?
3. **Stale rebuild threshold** — how many days before SessionStart triggers `--force`?
7 days is the starting guess; tune after observing drift in practice.

View File

@ -0,0 +1,402 @@
# Implementation Process
_Last updated: 2026-06-05_ | _Status: Steps 2a/2b/2d executed — toolchain installed, context fix applied, vault graph built; model selected (qwen2.5-coder:7b); Step 2c (reference set) previously complete. Step 2 fully executed._
This document distills `04-build-plan.md` into a concrete, staged build process and folds in
two locked decisions: **ADR-011** (faceted six-namespace taxonomy) and **ADR-012** (reuse the
existing `~/Documents/SecondBrain` vault rather than creating a new one). Read `04-build-plan.md`
for the underlying rationale, query/CRUD conventions, and plugin internals; read
`03-architecture-decisions.md` for the decision log. This document is the runbook.
> This is still a build process outline, not a detailed implementation plan. **Step 2 is fully
> executed as of 2026-06-05**: toolchain installed (2a), context-baking approach confirmed (2b),
> reference set generated (2c), vault graph built (2d). Selected model: **qwen2.5-coder:7b**
> (Modelfile variant `qwen25-coder-7b-16k`, num_ctx 16384). The next phase is Step 3 (Hooks).
> Most open questions in Steps 36 can be defaulted at build time.
---
## Build order
Bottom-up: fixture selection first (feeds the Step 2c gate), then Graphify setup, then hooks, then plugin. Bulk vault migration is deferred to after the system is validated end-to-end.
---
## Step 1 — Select benchmark fixtures (and defer bulk migration)
**Per ADR-012:** `~/Documents/SecondBrain` is the vault. The vault is already flat, already
scoped to durable knowledge, and already articulates the correct governance philosophy.
**Bulk vault migration is DEFERRED.** Do not migrate all notes now. The only pre-build activity
in this step is selecting a small fixture set that feeds the Step 2c benchmark gate. Everything
else — tagging ~20 existing notes, updating governance docs, initializing git, fixing broken
references — happens after the system is validated end-to-end.
> **Four sequencing concepts to keep distinct — they are not interchangeable:**
>
> 1. **Benchmark fixtures** — 510 notes selected NOW, in this step. They feed the Step 2c
> extraction benchmark gate. They are living fixtures, kept and reused throughout development.
> 2. **Bulk vault migration** — converting all ~20 existing notes to the six-facet tag schema,
> updating governance docs, initializing git. DEFERRED to last, after the system is fully
> validated.
> 3. **Initial validation project** — the first real end-to-end test after the system is built:
> one small project that contains BOTH code AND documents. POST-BUILD, before any rollout.
> 4. **Project-by-project rollout** — onboarding remaining projects one at a time, observing
> and adjusting between each. Follows the initial validation project.
### 1a — Select fixture notes
**Status (2026-06-04): DONE.** 6 cross-domain fixtures selected and listed in
`benchmark/dispatch-prompt.md`. Run as-is (no vault frontmatter modification per the
methodology decision — fixtures used exactly as they exist in the vault).
Choose 510 notes from `~/Documents/SecondBrain` with deliberate variety. The selection must
include at minimum:
- A **tool note** (e.g., a note about a CLI tool or SaaS product)
- A **client/project note** (scoped, not global)
- A **convention note** (a `type/convention` or workflow note)
- A **domain note** (a topic or subject-area note)
- At least one **relationship-dense note** — a note where several concepts interrelate and
Graphify should emit multiple typed edges and confidence tags
These notes must be usable as-is by a subagent that receives only the note text plus the shared
extraction spec (see Step 2c). Apply `summary:` frontmatter and the six-facet tag schema to
each fixture note now; do not wait for bulk migration.
**Frontmatter contract** (required for fixture notes immediately; apply to all notes at bulk
migration time):
```yaml
---
summary: "One-line, human-written router hint (required; Graphify does not generate this)"
tags:
- type/<kind> # listed first by convention
- client/<name> # as applicable
- project/<name> # as applicable — first-class for a freelancer
- domain/<topic> # as applicable
- tool/<name> # as applicable
- convention/<name> # as applicable
- scope/global # or scope/project
---
```
`type/` listed first preserves the vault's existing type-first ordering habit and makes note
kind immediately visible.
### 1b — Deferred bulk migration work (do not start until post-validation)
The following items are deferred to after the system is validated end-to-end on fixtures and
on the initial validation project (see §Post-build sequence below):
- **Initialize git:** `git init ~/Documents/SecondBrain` (enables ADR-008 sync) — before the first commit, add `graphify-out/` to the vault's `.gitignore`: `~/Documents/SecondBrain/graphify-out/cache/` (AST/tree-sitter parse cache) currently sits inside the vault and must stay untracked (ADR-008 — indexes are disposable/excluded from version control)
- **Update vault governance notes:** `CLAUDE.md` (vault path refs, Graphify commands,
ADR-011 namespace list), `vault-conventions.md` (six-facet tag list; preserve the
"act without being asked" proactive-retrieval section), and project-config hub note(s)
(update tag-inference table to namespaced form, e.g. `semrush-work → tool/semrush`)
- **Migrate remaining notes (~20):** add `summary:` frontmatter and convert unnamespaced tags
to six-facet form (e.g. `research → type/research`, `pest-control → domain/pest-control`,
`semrush → tool/semrush`). Add `scope/global` or `scope/project` to each note.
- **Create `_templates` for core note types:** `research`, `howto`, `adr`, `hub`. The long
tail stays freeform until a pattern earns a template (per ADR-011).
- **Fix the broken vault search reference:** `vault-conventions.md` (or `CLAUDE.md`) currently
references `~/.claude/scripts/vault_search.rb`, which does not exist. Replace every reference
with `graphify query` (per ADR-010).
### Post-build sequence (after Steps 26 are complete)
After the system is built and validated on fixtures:
**First:** Validate end-to-end on **one small project that contains BOTH code AND documents.**
This project is the initial validation target — not a production migration. It exercises both
extraction paths simultaneously: vault notes use the local-SLM extraction path; project code
uses the tree-sitter AST path. These are different extraction paths, which is precisely why the
first validation project must contain both. Observe how the two paths compose before proceeding.
**Then:** Onboard remaining projects **one at a time.** Observe extraction quality, god-node
shape, and hook behavior between each project before onboarding the next. Adjust configuration
as needed.
**Finally:** Execute bulk vault migration (items in 1b above) and run full vault graph build.
**Migration-unit granularity open question:** the right unit — a whole project repo, a
vault-note cluster, a client boundary — is not yet established. The first validation project
("one small project with both code and documents") is the anchor; refine the granularity
definition after observing that first real migration.
---
## Step 2 — Graphify + Ollama setup (CRITICAL PATH)
This step is the genuine gate. All hook behavior, model selection, and extraction tuning depend
on the benchmark in 2c. Do not skip or defer 2c.
### 2a — Install and verify Graphify
**Status (2026-06-05): EXECUTED.** graphify 0.8.31 installed at `~/.local/bin/graphify`; `graphify --version``graphify 0.8.31`. ollama 0.30.3, systemd service on 127.0.0.1:11434; GPU RTX 3060 12GB VRAM confirmed; all extraction ran 100% on GPU (no CPU spill per `ollama ps`). [verified: local shell, 2026-06-05]
The PyPI package name has a double-y — this is the correct install command:
```bash
pip install graphifyy
```
Verify: `graphify --version`
### 2b — Configure Ollama
**Status (2026-06-05): EXECUTED — but the env-var approach was superseded.** `GRAPHIFY_OLLAMA_NUM_CTX` does NOT propagate through graphify's ollama `/v1/chat/completions` endpoint: ollama's OpenAI-compatible `/v1` endpoint silently ignores per-request `options.num_ctx`, pinning context at the model's baked default (4096 by default). At 4096, graphify's extraction output is truncated mid-response and the chunk is discarded — producing an empty graph. [verified: local A/B test, 2026-06-05]
**SUPERSEDED APPROACH — do not use `GRAPHIFY_OLLAMA_NUM_CTX`:**
```bash
# This env var has NO EFFECT through graphify's /v1 path — do not set
# GRAPHIFY_OLLAMA_NUM_CTX=8192
```
**CORRECT APPROACH — bake context into a Modelfile variant:**
```bash
# Create a Modelfile variant with the desired context baked in:
# FROM <model>
# PARAMETER num_ctx 16384
# ollama create <name>-16k -f Modelfile
```
The `/v1` endpoint DOES honor the model's baked default. Verified `ollama ps` CONTEXT=16384, 100% GPU. This is the same lever the vault build (Step 2d) and plugin (Step 6) must use. Non-invasive: no sudo, no systemd restart, no shell-profile edit; reversible via `ollama rm`.
Still set in shell profile (these remain valid):
```bash
OLLAMA_FLASH_ATTENTION=1 # 3050% VRAM savings on KV cache — always set
GRAPHIFY_OLLAMA_KEEP_ALIVE=5 # set when packaging in Step 6
```
Two additional operational requirements confirmed: `OLLAMA_BASE_URL` must end in `/v1` (e.g., `http://127.0.0.1:11434/v1`) or every graphify call 404s; `OLLAMA_API_KEY` must be set to any non-empty value unless the host is loopback.
Verify context allocation after the first extraction call: `ollama ps` shows allocated context.
### 2c — Claude reference-set benchmark (THE GATE)
**Status (2026-06-04): EXECUTED (prior round).** 6 cross-domain fixtures × 3 Claude tiers = 18 reference fragments generated in `benchmark/reference-outputs/`. Run as-is (no vault frontmatter modification); verified clean. Gate is passed — Ollama model scoring is unblocked. Full local-model scoring (task 3.4) and model selection (task 3.6) are also complete — see `benchmark/scoring-results-2026-06-04.md`.
Produce a reference set of Graphify-shaped extraction outputs before committing to any local
Ollama model. Use the 510 fixture notes selected in Step 1a as the input set.
**What this step produces — and what it does NOT decide:**
This step produces the **gold-standard reference set**: one structured extraction output per
fixture note per Claude tier. The reference set is the scoring rubric against which local Ollama
models are scored in a later step. This step does NOT choose the final extraction model —
local-model selection happens when Ollama models are timed and scored against these references.
**Why Claude-only here:** local Ollama models cannot be run in this benchmarking environment.
Only Claude tiers are reachable via dispatched subagents. The Claude outputs serve as quality
anchors; Ollama speed and quality are measured separately, against these anchors.
**Dispatch one Claude Code subagent per tier:**
| Tier | Model ID | Role |
|---|---|---|
| Haiku | `claude-haiku-4-5` | Lightweight reference |
| Sonnet | `claude-sonnet-4-6` | Mid-tier reference |
| Opus | `claude-opus-4-8` | **Gold standard** |
Each subagent receives only the fixture note text plus the shared extraction spec at
`docs/memory-system/benchmark/extraction-spec.md`. It must not read `CLAUDE.md`, design docs,
or pull any project context — each subagent operates with the note and spec alone (fairness
contract).
Each subagent emits a Graphify-shaped structured fragment containing:
- Extracted entities (named concepts, tools, people, projects)
- Typed relationships between entities
- Confidence tags: `INFERRED` (inferred but not stated) and `AMBIGUOUS` (could be interpreted
multiple ways) applied where appropriate
**Metrics — quality only (wall-clock speed is explicitly out of scope here):**
Wall-clock timing is untrackable across dispatched subagents and is not measured in this step.
Speed re-enters the picture in the later Ollama-scoring step, where local models are timed
against these reference outputs. For this step, measure quality only:
1. **Entity correctness** — right concepts extracted, no hallucinated entities
2. **Relationship plausibility and typing** — edges plausible, correctly typed, no missing key edges
3. **Confidence-tag accuracy**`INFERRED` vs `AMBIGUOUS` applied appropriately
**Deliverable files:**
- Dispatch prompt (copy/paste-able): `docs/memory-system/benchmark/dispatch-prompt.md`
- Shared extraction schema: `docs/memory-system/benchmark/extraction-spec.md`
- Per-model outputs: `docs/memory-system/benchmark/reference-outputs/<note-slug>.<tier>.md`
where `tier` is one of `haiku`, `sonnet`, or `opus`
`claude-opus-4-8` output is the gold standard. When Ollama models are benchmarked later, their
outputs are scored by how closely they match the Opus reference for each fixture note.
### 2d — Build the initial vault graph
**Status (2026-06-05): EXECUTED.** Full vault graph built with qwen2.5-coder:7b (Modelfile variant `qwen25-coder-7b-16k`, num_ctx 16384). Result: **57 nodes / 43 edges / 15 communities**. God-nodes are plausible: Speed-to-Lead is the dominant hub, consistent with the vault's current content emphasis. [verified: local shell, 2026-06-05]
Command used (deviates from the template below — actual values recorded for reference):
```bash
graphify extract ~/Documents/SecondBrain \
--backend ollama --model qwen25-coder-7b-16k \
--max-concurrency 1 --token-budget 4000 \
--exclude .obsidian --out /tmp/graphify-bench/vault-graph
```
Note: `--max-concurrency 1` (not 2 — untested at 2 on 12GB GPU); `--token-budget 4000` (not 512 — fits the 16k context comfortably). Template for future runs:
```bash
graphify extract --path ~/Documents/SecondBrain \
--backend ollama --model qwen25-coder-7b-16k \
--token-budget 4000 --max-concurrency 1 --exclude .obsidian
```
Review `GRAPH_REPORT.md`. God nodes should be the most-connected tools, clients, and domain
concepts — confirm they make sense given the vault's content.
### 2e — Per-project code graphs (free; no model)
For each active client project:
```bash
graphify extract --path ~/projects/<client>/<project> --no-docs
```
`--no-docs` runs only tree-sitter AST — zero token cost. Use `--update` on subsequent runs;
use `--force` when files have been deleted (to clear stale nodes). Keep each project's
`graphify-out/` alongside the project; do not merge client projects into one graph.
---
## Step 3 — Hooks
Thin shell wrappers around Graphify. The logic lives in Graphify; the hooks only invoke it.
| Hook | Trigger | Action |
|---|---|---|
| **PostToolUse** | AI writes/edits a vault `.md` | `graphify update --file <path>` |
| **SessionStart** | Session opens | Stale check → `--force` rebuild if needed → inject context |
| **SessionEnd** | Session closes | Append dated journal note to vault |
**Stale-check mechanism (SessionStart):** read the mtime of
`~/.cache/graphify/vault-rebuild.stamp`. If older than N days (7 is the starting guess — tune
after observing drift), run `graphify --force` on the vault and write a new stamp. Then inject:
god-node summary from `GRAPH_REPORT.md` + `convention/*` note summaries + journal pointer.
**Note on `--update` vs `--force`:** `graphify update --file` does not prune deleted nodes.
Stale/ghost nodes accumulate from manual deletes and renames. The periodic `--force` rebuild
triggered by the stale check is the mitigation — it rebuilds the graph clean.
---
## Step 4 — Episodic layer (memsearch)
Install memsearch for time-anchored "what happened / what was I working on" queries. Graphify
does **not** replace this — they serve different query patterns (knowledge graph vs timeline
recall).
```
/plugin marketplace add zilliztech/memsearch
plugin install memsearch
```
Verify daily memory files appear after a few conversations. The open question of whether
memsearch indexes SessionEnd journal notes or only its own auto-capture is deferred to build
time (see Open questions §7).
---
## Step 5 — Vault sync
Sync the markdown vault to the VPS. Pick one:
- **git** — versioned, hourly push/pull via cron; explicit audit trail.
- **Syncthing** — continuous, bidirectional, zero-thought after setup.
**What to sync:** the vault only (`~/Documents/SecondBrain`).
**What NOT to sync:** `graphify-out/` directories, Milvus Lite caches, Ollama models. These are
disposable — rebuilt per machine from the vault (markdown is the single source of truth, per
ADR-008).
---
## Step 6 — Package as a global Claude Code plugin
One global install so every project and machine shares the same vault conventions, hooks, and
Graphify config.
**Plugin contents:**
- **Hooks** registered in settings: SessionStart, PostToolUse, SessionEnd — shell wrappers from
Step 3, parameterized by vault path and Graphify output dir.
- **Graphify CLI on PATH** — the AI calls `graphify query`, `graphify path`, `graphify explain`
via the Bash tool. No server process per graph; project-specific graphs are queried with
`--graph <project-root>/graphify-out/graph.json`.
- **Skills** (carry the know-how to the model):
- `memory-write` — when to record evergreen knowledge, frontmatter contract, scope rule, vault not repo.
- `memory-query``graphify query` vs memsearch; god-node discipline; `--budget`/`--dfs`; cross-client lookups; progressive disclosure via `summary`.
- `memory-reorganize` — plan-mode consolidation/promotion procedure; when to trigger `--force` rebuild; human-approval guardrails.
- **Env vars** baked in: `OLLAMA_FLASH_ATTENTION=1`, `GRAPHIFY_OLLAMA_KEEP_ALIVE=5`,
`OLLAMA_BASE_URL=http://127.0.0.1:11434/v1`, `OLLAMA_API_KEY=<any-non-empty>`. Note:
`GRAPHIFY_OLLAMA_NUM_CTX` is NOT effective through graphify's `/v1` path — context must
be baked into the Modelfile variant instead (see Step 2b).
- **Config** (set once at user level): vault path, Graphify output dir, Ollama model name,
stale-rebuild threshold in days.
---
## Open questions / decisions still to settle
These are deferred to build time. Most can be defaulted without blocking. **§6 (model selection) is RESOLVED** — qwen2.5-coder:7b selected, vault graph built. Remaining questions (§15, §78) do not block Step 3 or later.
1. **Vault symlink**`~/Documents/SecondBrain` is confirmed as the vault (RESOLVED per
ADR-012). The open sub-question: symlink it into `~/.claude/memory` or not? Only needed if
a tool requires that path.
2. **Sync mechanism** — git (versioned, hourly) vs Syncthing (continuous). Both are valid;
choose at build time based on preference.
3. **Stale rebuild threshold** — 7 days is the starting guess. Tune after observing how quickly
ghost-node drift becomes noticeable in practice.
4. **Per-project graph path convention** — how does SessionStart know which `graph.json` to
inject? Proposed convention: `<project-root>/graphify-out/graph.json`, injected only if the
file exists at that path. Not yet made explicit in the plugin spec.
5. **`summary` field discipline** — Graphify extracts entities and edges but does not write
summaries. The human (or AI at note-creation time) must write `summary:` frontmatter. Confirm
this holds in practice and add a lint/reminder to the memory-write skill if it drifts.
6. **Step 2c reference set + model selection****RESOLVED (2026-06-04/05):** The reference
set was generated: 18 fragments (6 fixtures × 3 tiers) in `benchmark/reference-outputs/`.
Local-model scoring complete (see `benchmark/scoring-results-2026-06-04.md`). **Selected
model: qwen2.5-coder:7b** (run as Modelfile variant `qwen25-coder-7b-16k`, num_ctx 16384)
— graphify's shipped default and the only tested candidate that cleared the
relationship/edge gate. All smaller candidates tested (gemma4:e4b 8.0B, gemma4:e2b 5.1B,
qwen3.5:2b 2.3B) failed to produce relationship-bearing graphs through graphify's extraction
prompt on this hardware. Model selection is complete.
7. **memsearch + journal integration** — does memsearch index SessionEnd journal notes or only
its own auto-capture? How does the journal pointer injected at SessionStart reference the
vault? Nail down at memsearch install time (Step 4).
8. **MCP server management** — vault graph + N project graphs = N+1 potential instances. The
build plan leans toward a single Graphify CLI on PATH with `--graph <path>` rather than
running multiple `graphify serve` instances, but this is not yet made explicit in the plugin
spec. Confirm the CLI-only approach or define the server lifecycle.
**Resolved:** The type-tag taxonomy is settled via ADR-011 (faceted, six independent namespaces;
`type/` listed first). No further debate needed on taxonomy shape.
---
## Recommended next move
**Step 2 is complete.** The next phase is **Step 3 (Hooks)** — thin shell wrappers around
Graphify for PostToolUse, SessionStart, and SessionEnd — followed by Step 4 (memsearch
episodic layer) and Step 6 (package as global plugin). Open questions §15 and §78 can be
defaulted or deferred at build time. §6 (model selection) is resolved.

View File

@ -0,0 +1,210 @@
# Claude-Tier Reference-Set Benchmark — Dispatch Prompt
_Last updated: 2026-06-04_
_Status: active — copy-paste this into a Claude Code session to generate the reference set_
## Overview
This prompt dispatches **one subagent per Claude tier** (`claude-haiku-4-5`, `claude-sonnet-4-6`,
`claude-opus-4-8`) over each fixture note selected in Step 1c of the build runbook. Each subagent
emits a Graphify-shaped structured fragment. The per-model output files become the **gold-standard
reference set** against which local Ollama doc-extraction models are scored in a later step.
**`claude-opus-4-8`'s output is the gold-standard rubric.**
Evaluation is on **quality only**: entity correctness, relationship plausibility and typing, and
confidence-tag accuracy. Wall-clock speed is NOT a metric for this Claude reference run — speed
re-enters only when local Ollama models are timed against this reference set.
---
## Step 1 — Select Fixture Notes (operator action, before dispatching)
Fixture notes are selected at build time from `~/Documents/SecondBrain` per Step 1c of the
runbook. Choose **510 notes** with deliberate variety:
- One **tool note** (documents a specific tool or library)
- One **client/project note** (describes a client engagement or project)
- One **convention note** (captures a working convention or practice)
- One **domain note** (covers a knowledge or technical domain)
- One **relationship-dense note** (many named entities or cross-references)
- Additional notes as needed for coverage
Assign each note a **kebab-case slug** (used in output filenames). Populate the placeholder list
below before dispatching:
```
# FIXTURES — selected 2026-06-04; benchmark run same day (no vault frontmatter modification)
# Format: <note-slug>:<vault-relative-path> (vault root: ~/Documents/SecondBrain)
FIXTURES=(
10dlc-isv-setup-guide:2026-03-31-10dlc-isv-setup-guide-oncadence.md
pest-control-enterprise-revenue:2026-03-13-pest-control-enterprise-revenue-architecture-and-seasonality.md
ai-coding-conventions-synthesis:2026-03-13-ai-coding-conventions-organization-external-research-synthesis.md
pest-control-sms-market-research-stats:2026-03-13-pest-control-after-hours-sms-lead-capture-market-research-stats.md
oo-principles-plugin-concept:2026-03-13-oo-principles-plugin-concept-design-recommendations.md
pest-control-email-abc-hub:2026-03-14-pest-control-email-a-b-c-test-experiment-hub.md
)
```
---
## Step 2 — Output File Convention
For each fixture note, the run produces **one output file per Claude tier** — three files per note,
`3 × N` files total (where N is the number of fixture notes).
Output path pattern:
```
docs/memory-system/benchmark/reference-outputs/<note-slug>.<tier>.md
```
Where `<tier>` is one of: `haiku`, `sonnet`, `opus`
Examples (using a hypothetical slug `graphify-tool-overview`):
- `reference-outputs/graphify-tool-overview.haiku.md`
- `reference-outputs/graphify-tool-overview.sonnet.md`
- `reference-outputs/graphify-tool-overview.opus.md`
Each file contains that tier's extraction fragment for **one fixture note** — the `note_slug:`
block for that note only. Do not combine multiple notes into one file.
---
## Step 3 — Dispatch (copy-paste this block into a Claude Code session)
> **Operator:** replace `<NOTE-SLUG>`, `<NOTE-TEXT>`, and model names as needed. Dispatch all
> three subagents for each fixture note. They can run in parallel across tiers for the same note.
---
### Dispatch template (repeat for each fixture note × each tier)
```
Dispatch a subagent using model <MODEL> to perform the following task.
=== FAIRNESS CONTRACT ===
You will receive exactly two inputs:
1. The raw text of one vault note (below).
2. The shared extraction spec and output schema (below).
You MUST NOT read any repository files (CLAUDE.md, design docs, specs, tasks), access the
vault directory structure, or use any project or system context. If your environment has
injected any such context automatically, you must ignore it entirely — treat it as if it
does not exist. The only allowed inputs are the note text and the extraction spec below.
=== END FAIRNESS CONTRACT ===
=== NOTE TEXT ===
<NOTE-TEXT>
=== END NOTE TEXT ===
=== EXTRACTION SPEC AND OUTPUT SCHEMA ===
## Facet Vocabulary (closed)
Notes carry six flat, namespaced facets plus one scope tag. These are the ONLY valid values
for the optional `facet` field on entities. Do not invent new namespaces.
| Prefix | Meaning |
|--------------|-----------------------------------------|
| `type/` | What kind of thing the note is about |
| `client/` | A client identity |
| `project/` | A project name |
| `domain/` | A knowledge or technical domain |
| `tool/` | A specific tool, library, or CLI |
| `convention/`| A working convention or practice |
| `scope/` | Applicability scope (cross-cutting tag) |
An entity should carry a `facet` value only when the note text directly supports mapping it
to one of the above. Absence of a `facet` field is correct when no mapping is warranted.
## Output Schema
Emit ONLY the following YAML structure. No other keys, no prose, no summary.
```yaml
# --- Graphify extraction fragment ---
# One block per fixture note. Repeat this structure for each note.
note_slug: <kebab-case-identifier-for-the-note> # operator fills this in at run time
entities:
- name: <string> # exact or near-exact name as it appears in the note
type: <string> # e.g. Person, Tool, Project, Concept, Convention, Client, Domain
facet: <string> # OPTIONAL — must be one of the seven prefixes above, e.g. "tool/graphify"
confidence: <string> # OPTIONAL — omit if the entity is directly stated
# Values: INFERRED | AMBIGUOUS
relationships:
- source: <entity name> # must match a name in the entities list above
type: <string> # free-text verb phrase, e.g. "uses", "depends_on", "implements", "replaces"
target: <entity name> # must match a name in the entities list above
confidence: <string> # OPTIONAL — omit if the relationship is directly stated
# Values: INFERRED | AMBIGUOUS
```
## Confidence tag semantics
| Tag | Meaning |
|------------|--------------------------------------------------------------------------------------|
| _(absent)_ | The entity or relationship is **directly stated** in the note text. |
| `INFERRED` | Not stated literally but **reasonably deducible** from the note text alone. |
| `AMBIGUOUS`| Supportable by the text but **uncertain or admits multiple readings**. |
## Rules
- **Entities only:** extract entities that are named (not vague category references).
- **No invented names:** entity names must be grounded in the note text.
- **Relationship types:** free-text verb phrases; do not normalize to a fixed vocabulary.
- **Facet mapping:** use the closed vocabulary above; only assign when the note text supports it.
- **No extra keys:** do not add summaries, scores, embeddings, or metadata fields.
- **One block per note:** if processing multiple fixture notes, emit one `note_slug:` block per
note, separated by `---`.
=== END EXTRACTION SPEC ===
Write your output to:
docs/memory-system/benchmark/reference-outputs/<NOTE-SLUG>.<TIER>.md
where <NOTE-SLUG> is the kebab-case slug for this note and <TIER> is one of: haiku, sonnet, opus.
The file must contain ONLY the YAML extraction fragment — no preamble, no explanation.
```
**MODEL and TIER substitutions:**
| Dispatch | MODEL | TIER |
|----------|------------------------|---------|
| 1st | `claude-haiku-4-5` | `haiku` |
| 2nd | `claude-sonnet-4-6` | `sonnet`|
| 3rd | `claude-opus-4-8` | `opus` |
---
## Step 4 — After All Subagents Complete
1. Verify `3 × N` files exist in `docs/memory-system/benchmark/reference-outputs/`.
2. For each fixture note, diff the three tier files side by side to understand tier-level
disagreements in entity recognition, relationship typing, and confidence tagging.
3. `claude-opus-4-8`'s `.opus.md` file for each note is the **gold-standard rubric** against
which local Ollama extraction will later be scored.
4. Do NOT run Ollama models in this step — that is a separate later step that uses these files
as the scoring reference.
---
## Notes on the Fairness Contract
The fairness contract requires that each subagent reasons from **note text + extraction spec
only**. Two mechanisms can break this:
1. **Deliberate reads:** the subagent reads repository files or the vault. The dispatch template
forbids this explicitly.
2. **Injected context:** Claude Code automatically injects `CLAUDE.md` and project context into
subagent sessions. The dispatch template explicitly instructs the subagent to **ignore any
injected context** and treat it as non-existent. This is the critical instruction — "don't
read files" is not sufficient on its own.
If you observe tier outputs that appear to reflect knowledge of the vault structure or system
design (beyond what the note text contains), treat that output as contaminated and re-run that
subagent with stronger isolation instructions.

View File

@ -0,0 +1,95 @@
# Graphify Document-Extraction Benchmark — Shared Extraction Spec
_Last updated: 2026-06-04 — rules tightened after first benchmark run (bare-slash facets and undeclared relationship entities)_
_Status: active — used as the shared schema for the Claude-tier reference-set benchmark_
## Purpose
This spec defines the **minimal structured fragment** that every benchmarking subagent (one per
Claude tier) must emit when given a raw vault note. The fragment mimics what Graphify's local SLM
doc-extraction step produces: a list of **entities**, a list of **typed relationships** (edges
between entities), and per-item **confidence tags** where applicable.
The schema is intentionally minimal. It exists so that:
1. Per-model output files are **diffable entity-by-entity and edge-by-edge** across tiers.
2. The same schema is used unchanged when local Ollama models are later scored against this
reference set — making the comparison apples-to-apples.
Subagents receive only the raw note text plus this spec. No repository files, no vault structure,
no project context. If any such context is injected by the environment, it must be ignored.
---
## Facet Vocabulary (closed)
Notes in the SecondBrain vault carry six flat, namespaced facets plus one scope tag. These are the
**only** valid values for the optional `facet` field on entities. Do not invent new namespaces.
| Prefix | Meaning |
|--------------|-----------------------------------------|
| `type/` | What kind of thing the note is about |
| `client/` | A client identity |
| `project/` | A project name |
| `domain/` | A knowledge or technical domain |
| `tool/` | A specific tool, library, or CLI |
| `convention/`| A working convention or practice |
| `scope/` | Applicability scope (cross-cutting tag) |
An entity should carry a `facet` value only when the note text directly supports mapping it to one
of the above. Absence of a `facet` field is correct when no mapping is warranted.
**Sub-value required (STRICT):** A `facet` value MUST include a non-empty sub-value after the
namespace slash — e.g. `tool/cursor`, `convention/tdd`, `domain/distributed-systems`. A bare
namespace with nothing after the slash (e.g. `tool/`, `domain/`) is **invalid** and must never
appear in output. If a suitable sub-value cannot be identified from the note text, omit the
`facet` field entirely.
---
## Output Schema
The required output format. Every subagent MUST emit this structure and no other top-level keys.
```yaml
# --- Graphify extraction fragment ---
# One block per fixture note. Repeat this structure for each note.
note_slug: <kebab-case-identifier-for-the-note> # operator fills this in at run time
entities:
- name: <string> # exact or near-exact name as it appears in the note
type: <string> # e.g. Person, Tool, Project, Concept, Convention, Client, Domain
facet: <string> # OPTIONAL — must be one of the seven prefixes above, e.g. "tool/graphify"
confidence: <string> # OPTIONAL — omit if the entity is directly stated
# Values: INFERRED | AMBIGUOUS
relationships:
- source: <entity name> # must match a name in the entities list above
type: <string> # free-text verb phrase, e.g. "uses", "depends_on", "implements", "replaces"
target: <entity name> # must match a name in the entities list above
confidence: <string> # OPTIONAL — omit if the relationship is directly stated
# Values: INFERRED | AMBIGUOUS
```
### Confidence tag semantics
| Tag | Meaning |
|------------|--------------------------------------------------------------------------------------|
| _(absent)_ | The entity or relationship is **directly stated** in the note text. |
| `INFERRED` | Not stated literally but **reasonably deducible** from the note text alone. |
| `AMBIGUOUS`| Supportable by the text but **uncertain or admits multiple readings**. |
### Rules
- **Entities only:** extract entities that are named (not vague category references).
- **No invented names:** entity names must be grounded in the note text.
- **Relationship types:** free-text verb phrases; do not normalize to a fixed vocabulary.
- **Facet mapping:** use the closed vocabulary above; only assign when the note text supports it.
- **No extra keys:** do not add summaries, scores, embeddings, or metadata fields.
- **One block per note:** if processing multiple fixture notes, emit one `note_slug:` block per note,
separated by `---`.
- **Referential integrity (STRICT):** every relationship `source` and `target` MUST exactly match
the `name` of an entity declared in the `entities` list of the same block. Do not reference
undeclared concept strings, free-form text, or the note's own `note_slug` as a `source` or
`target`. If a relationship needs a concept that is not yet in the entity list, declare it as an
entity first (with an appropriate `confidence` tag if it is inferred), then reference it.

View File

@ -0,0 +1,274 @@
# Local-LLM Doc-Extraction Gut-Check — 2026-06-04
_Last updated: 2026-06-04_ | _Status: gut-check complete; gemma4:e4b is the candidate; GPU confirmed post-reboot (~74 tok/s, see appendix); Graphify validation is the next step._
This document consolidates the local-LLM doc-extraction gut-check run on 2026-06-04. It is
the authoritative findings record for that session and feeds forward into the Graphify-ollama
validation step.
**Four things this document keeps strictly separate:**
- **(A) Model-capability signal** — what carries forward; the durable conclusion.
- **(B) Raw-ollama harness gotchas** — benchmark-process notes only; explicitly NOT production config.
- **(C) GPU driver fix** — real diagnosis; timing estimate, not a measured result.
- **(D) Graphify config surface** — the actual production guidance.
Cross-references:
- Claude-tier reference run: `docs/memory-system/benchmark/run-findings-2026-06-04.md`
- 18 Claude reference fragments: `docs/memory-system/benchmark/reference-outputs/` (`.haiku.md`, `.sonnet.md`, `.opus.md` files only)
- Authoritative Graphify-ollama section: `docs/graphify/05-local-models-and-backends.md`
---
## TL;DR — What worked
- The local model class is **good-enough** for the doc-extraction role. Best installed model:
**`gemma4:e4b`** (~9.6 GB, fits the RTX 3060's 12 GB). It produced valid schema on 6/6
fixtures with clean, well-namespaced facets (0 bad facets out of 6) and note-grounded
entities (no contamination).
- It catches the high-value entities (~4560% of Opus's entity recall), missing Opus's long
tail — acceptable for the role.
- **Critical reframe:** in production we will NOT hand-invoke ollama. Graphify owns the ollama
call (prompt, chunking, context window, parsing) over its HTTP API. The hand-rolled benchmark
validated *model capability*, not the production pipeline. The raw-ollama tuning discovered
during this run is Graphify-internal, not our config surface.
---
## (A) Model-capability signal — measured via a hand-rolled prompt/parser (NOT Graphify)
This signal was gathered through a custom prompt and a custom YAML parser, NOT Graphify's
extraction path. It says "the model class is good-enough." It does NOT say "this is what
Graphify will output" — Graphify uses a different prompt, schema, and parser.
### Results across the 3 installed models (all reasoning models; only these 3 are on the machine)
| Model | Size | Parse rate | Entities/note | Facet quality | Verdict |
|---|---|---|---|---|---|
| `gemma4:e4b` | ~9.6 GB | 6/6 (2 needed trivial colon-quote repair) | 1334 (~4560% of Opus) | 0 bad/6 | **Usable — best pick** |
| `gemma4:e2b` | ~7.2 GB | 4/6 (2 hard YAML breaks — dropped `source:` key) | 2331 when valid | 0 bad/6 | Usable only behind a tolerant parser + retry |
| `qwen3.5:2b` | ~2.7 GB | parses but collapses to ~1 entity/note | 1 | n/a | Not usable as tested |
All three land below haiku (the weakest Claude frontier tier: 1937 entities, 6/6 clean parse,
clean referential integrity). "Below frontier-floor" does not mean "unusable" — `gemma4:e4b`'s
misses are the long tail; the high-value entities are present.
### gemma4:e4b quirks observed
- Occasional top-level key rename (e.g. `entity_relationships` instead of the spec's key).
- A `target: Concept` placeholder leak on the longest note — it echoed the schema's example
value rather than a concrete entity. Seen once across 6 fixtures.
These quirks were caught by the hand-rolled parser and are noted as model-class behavior.
Whether Graphify's parser handles them gracefully is a separate question (see section D).
---
## (B) Raw-ollama harness gotchas — benchmark-process notes ONLY
> **EXPLICIT CAVEAT:** This section records why the first benchmark pass produced junk and
> what was done to fix the harness. These knobs are NOT our production settings. They are
> Graphify-internal concerns (see section D). Do not treat this section as production config.
### Problem: default context truncation produced garbage output
Bare `ollama run` with the default (~4K) context truncated output mid-stream. Reasoning models
burn their token budget on chain-of-thought first; the model got guillotined mid-output, leaving
malformed YAML.
### Harness fix (benchmark only)
- Called the HTTP API at `/api/generate` instead of `ollama run`.
- Set `num_ctx=8192` (per-request via the API body).
- Set `think=false` to suppress chain-of-thought tokens and keep the output budget for the
extraction result.
- Added a tolerant YAML parser that retries on soft parse errors and auto-quotes bare scalars
containing `:`.
These changes produced the results in section A. They explain the benchmark outcome; they do
not define the production path, which Graphify handles internally.
---
## (C) GPU not used during this run — root cause and fix
### Root cause: NVIDIA driver version mismatch
| Item | Value |
|---|---|
| In-kernel module | 580.142 |
| On-disk / userspace driver | 580.159.03 |
The driver was updated on disk on 2026-06-03 but the stale module stayed resident. Result:
`nvidia-smi` fails — "Failed to initialize NVML: Driver/library version mismatch." Because
NVML/CUDA cannot initialize, ollama finds no CUDA device (`total_vram="0 B"`) and runs 100%
on CPU.
Same-machine proof: the ollama log at 07:57 on 2026-06-03 DID find the RTX 3060
(`compute=8.6 ... 12.0 GiB`); the 16:12 service instance the benchmark ran under found no CUDA.
**CPU eval rate observed:** ~10.6 tok/s → ~187313 s/note. The 9.6 GB model fits the 12 GB
card; VRAM is not the bottleneck — the driver mismatch is.
### Fix (user must run — non-interactive sudo was blocked during this session)
The running kernel already matches the on-disk 580.159.03 module, so a reboot loads the correct
module:
```bash
sudo reboot
# then verify:
nvidia-smi # should print the RTX 3060 table, no NVML error
# only if CUDA still fails after reboot:
sudo dnf reinstall 'akmod-nvidia*' 'xorg-x11-drv-nvidia*' # then reboot again
```
### ESTIMATE — post-reboot GPU timing (NOT measured; blocked by sudo this session)
With the RTX 3060 and the 4-bit `gemma4:e4b` model fully resident, ~4060 tok/s plus faster
prompt processing should drop extraction to roughly single-digit to ~20 s/note — i.e.,
plausibly within a "well under a minute" target.
**This is an estimate to be confirmed post-reboot using the curl command in the appendix below.**
It is not a measured result.
---
## (D) Graphify config surface — production guidance
> This is the only section that contains production configuration guidance. Sections B and C
> are process notes; section A is a model-capability signal only.
### How Graphify talks to Ollama
Graphify talks to Ollama over its **HTTP API** at `http://localhost:11434`, not by shelling out
to `ollama run`. `[github-inferred]` from the env-var/timeout/num_ctx design; the exact endpoint
(`/api/generate` vs `/api/chat`) is `[unverified]` — package internals are not browseable on
GitHub.
Graphify **owns** the prompt template, chunking, context sizing (auto-sized `num_ctx` by
default), and output parsing. `[github]` for the env-var knobs; prompt and parser specifics are
`[github-inferred]` / `[unverified]`. Whether Graphify sets `think=false` and whether it does
tolerant-YAML / retry are `[unverified]`.
Implication: the hand-rolled extraction-spec prompt and raw-ollama settings from section B
likely do **not** apply to Graphify's ollama path. Our levers are the documented external knobs
only.
### Text-vs-code split
Confirmed `[github]` (verbatim README): code is extracted locally via tree-sitter AST with zero
LLM calls; docs/PDFs/images go through the LLM backend (Ollama here). Ollama is invoked **for
text documents only.**
### Model selection
No official Graphify-recommended Ollama model. `[github]` — absence verified at v0.8.30.
Community starting points: `qwen2.5:7b`, `phi4:14b`. `[community]` Default behavior: auto-detect
the running model. Override via `OLLAMA_MODEL`.
Per this benchmark, `gemma4:e4b` is the best installed candidate — but this must be validated
through Graphify's own extraction path, not the hand-rolled benchmark (see Deferred
follow-ups §1 below).
### Environment variables and CLI flags (all `[github]` from v0.8.30 README)
| Knob | Default | Notes |
|---|---|---|
| `OLLAMA_BASE_URL` | `http://localhost:11434` | Ollama endpoint |
| `OLLAMA_MODEL` | auto-detect | Override to pin a model |
| `GRAPHIFY_OLLAMA_NUM_CTX` | auto-sized | Override context window |
| `GRAPHIFY_OLLAMA_KEEP_ALIVE` | unspecified | Minutes to keep model loaded; `0` = unload after each chunk |
| `--token-budget` | — | Chunk size (tokens) |
| `--max-concurrency` | — | Parallel extraction workers |
| `--api-timeout` | 600 s | Increase for slower hardware |
### Known sharp edge
Context saturation across consecutive chunks on the Ollama backend can exhaust VRAM after a
few chunks. `[community, issue #798]` Mitigate by lowering `GRAPHIFY_OLLAMA_NUM_CTX` and/or
setting `GRAPHIFY_OLLAMA_KEEP_ALIVE=0`.
---
## Appendix — Exact commands
### Graphify Ollama path `[github]`
```bash
# Install (package is graphifyy — double-y; command is graphify):
uv tool install "graphifyy[ollama]"
# Pull the candidate model (gemma4:e4b is the local best pick per this benchmark):
ollama pull gemma4:e4b
# Start Ollama if not already running as a service:
ollama serve
# Basic extraction:
graphify extract ./docs --backend ollama
# Constrained single-GPU tuning:
GRAPHIFY_OLLAMA_NUM_CTX=8192 graphify extract ./docs --backend ollama \
--token-budget 4000 --max-concurrency 2 --api-timeout 900
# Free VRAM between chunks (mitigates issue #798):
GRAPHIFY_OLLAMA_KEEP_ALIVE=0 graphify extract ./docs --backend ollama
```
### Post-reboot GPU re-timing (benchmark verification only — NOT production)
Run this after rebooting to confirm the GPU is now used and to measure actual tok/s:
```bash
F="$HOME/Documents/SecondBrain/2026-03-31-10dlc-isv-setup-guide-oncadence.md"
PROMPT=$(python3 -c "import json;print(json.dumps('Extract entities from this note:\n\n'+open('$F').read()))")
time curl -s http://127.0.0.1:11434/api/generate \
-d "{\"model\":\"gemma4:e4b\",\"prompt\":$PROMPT,\"stream\":false,\"think\":false,\"options\":{\"num_ctx\":8192}}" \
| python3 -c "import json,sys;d=json.load(sys.stdin);ec=d['eval_count'];ed=d['eval_duration'];print('eval tok/s:',round(ec/(ed/1e9),2),'| eval_count:',ec,'| total wall (s):',round(d['total_duration']/1e9,2))"
```
This measures raw model tok/s via the hand-rolled harness — it does NOT measure Graphify's
production extraction performance.
#### Measured results (run 2026-06-04, post-reboot)
GPU is now fully active and resident — the driver mismatch diagnosed in section C is
**RESOLVED**. `nvidia-smi` is healthy (Driver 580.159.03, RTX 3060, no NVML error). The
`gemma4:e4b` model loaded fully GPU-resident: `/api/ps` reports `size_vram` 3.29 GB ==
`size` 3.29 GB (no CPU fallback), with 4462 MiB used on the card.
| Pass | eval tok/s | eval_count | total wall (s) | load (s) | prompt_eval (s) | prompt_eval_count |
|---|---|---|---|---|---|---|
| Cold | 74.27 | 701 | 76.04 | 38.15 | 28.44 | 905 |
| Warm | 74.25 | 717 | 10.13 | 0.43 | 0.04 | — |
Conclusion: steady-state generation is **~74 tok/s** — at or above the section C estimate of
4060 tok/s. Warm extraction lands at **~10 s/note**; the cold pass is ~76 s, dominated by the
one-time 38 s model load plus a 28 s cold prompt-eval. This confirms section C's "well under a
minute once the model is resident" prediction.
> **Caveat retained:** these are raw model tok/s from the hand-rolled curl harness, NOT
> Graphify production extraction performance (see Deferred follow-up §1).
---
## Deferred follow-ups
1. **Definitive validation: run Graphify's own ollama path over the fixtures** — NOT re-scoring
the hand-rolled benchmark. Do NOT conclude the next step is "score Graphify output against
the 18 Opus references": that is apples-to-oranges (different prompt/schema/output). The 18
Opus references validated model capability; they are not a scoring rubric for Graphify's
graph output.
2. **Resolve `[unverified]` Graphify internals** (HTTP endpoint, think-mode, retry/parsing):
after `uv tool install "graphifyy[ollama]"`, read the installed client source under the
tool's site-packages. This does not require the GPU — a cheap deferred follow-up.
3. **Confirm GPU timing post-reboot****RESOLVED (2026-06-04).** GPU re-timing ran post-reboot;
measured ~74 tok/s steady-state (at/above the section C estimate). See the "Measured results"
table under the Post-reboot GPU re-timing appendix above.
4. **Raw per-model benchmark outputs from this run** were throwaway scratch (wrong harness,
hand-rolled prompt) and are not committed.

View File

@ -0,0 +1,88 @@
note_slug: 10dlc-isv-setup-guide
entities:
- name: OnCadence
type: Project
facet: project/oncadence
- name: Twilio
type: Tool
facet: tool/twilio
- name: TrustHub API
type: Tool
facet: tool/twilio
- name: 10DLC
type: Domain
facet: domain/sms
- name: ISV architecture
type: Convention
facet: convention/isv-integration
- name: Secondary Customer Profile
type: Concept
- name: Brand registration
type: Concept
- name: Campaign registration
type: Concept
- name: Primary Business Profile
type: Concept
- name: pest control
type: Domain
facet: domain/pest-control
- name: opt-out language
type: Convention
- name: EIN
type: Concept
- name: subaccount
type: Concept
- name: T-Mobile
type: Tool
- name: Facebook Business page
type: Convention
facet: convention/web-presence
- name: Google Business Profile
type: Convention
facet: convention/web-presence
- name: Sole Proprietors
type: Concept
- name: long-code number
type: Concept
- name: SMS carrier surcharge
type: Concept
relationships:
- source: OnCadence
type: uses
target: Twilio
- source: OnCadence
type: implements
target: ISV architecture
- source: Twilio
type: provides
target: TrustHub API
- source: OnCadence
type: targets
target: pest control
- source: TrustHub API
type: manages
target: Brand registration
- source: TrustHub API
type: manages
target: Campaign registration
- source: Secondary Customer Profile
type: requires
target: EIN
- source: Brand registration
type: precedes
target: Campaign registration
- source: subaccount
type: carries
target: Secondary Customer Profile
- source: opt-out language
type: required_in
target: Campaign registration
- source: Sole Proprietors
type: limited_to
target: T-Mobile
- source: long-code number
type: isolated_per
target: subaccount
confidence: INFERRED

View File

@ -0,0 +1,106 @@
# --- Graphify extraction fragment ---
note_slug: 10dlc-isv-setup-guide
entities:
- name: OnCadence
type: Project
facet: project/oncadence
- name: 10DLC
type: Concept
facet: domain/10dlc
- name: Twilio
type: Tool
facet: tool/twilio
- name: TrustHub API
type: Tool
facet: tool/twilio-trusthub
- name: ISV
type: Concept
- name: CSP
type: Concept
- name: pest control clients
type: Client
facet: client/pest-control
- name: Secondary Customer Profile
type: Concept
- name: Primary Business Profile
type: Concept
- name: Brand registration
type: Concept
- name: Campaign registration
type: Concept
- name: EIN
type: Concept
- name: IRS
type: Concept
- name: Facebook Business page
type: Concept
- name: Google Business Profile
type: Concept
- name: T-Mobile
type: Concept
- name: Sole Proprietors
type: Concept
- name: long-code number
type: Concept
- name: STOP opt-out language
type: Convention
facet: convention/sms-opt-out
- name: subaccount-per-client architecture
type: Concept
facet: convention/subaccount-per-client
relationships:
- source: OnCadence
type: is_primary_account_of
target: Twilio
- source: OnCadence
type: acts_as
target: ISV
- source: OnCadence
type: acts_as
target: CSP
- source: OnCadence
type: manages_10dlc_for
target: pest control clients
- source: OnCadence
type: submits_via
target: TrustHub API
- source: OnCadence
type: creates_once
target: Primary Business Profile
- source: pest control clients
type: each_assigned
target: Secondary Customer Profile
- source: pest control clients
type: each_assigned
target: Brand registration
- source: pest control clients
type: each_assigned
target: Campaign registration
- source: Brand registration
type: must_match
target: EIN
- source: EIN
type: issued_by
target: IRS
confidence: INFERRED
- source: Sole Proprietors
type: lack
target: EIN
- source: Sole Proprietors
type: limited_to_throughput_on
target: T-Mobile
- source: Facebook Business page
type: accepted_for
target: Brand registration
confidence: INFERRED
- source: Campaign registration
type: must_include
target: STOP opt-out language
- source: OnCadence
type: implements
target: subaccount-per-client architecture
- source: pest control clients
type: each_assigned
target: long-code number

View File

@ -0,0 +1,66 @@
# --- Graphify extraction fragment ---
note_slug: 10dlc-isv-setup-guide
entities:
- name: OnCadence
type: Project
facet: project/OnCadence
- name: Twilio
type: Tool
facet: tool/Twilio
- name: 10DLC
type: Concept
facet: domain/SMS
- name: TrustHub API
type: Tool
facet: tool/TrustHub API
- name: ISV
type: Concept
facet: domain/SMS
- name: Secondary Customer Profile
type: Concept
- name: Brand registration
type: Concept
- name: Campaign registration
type: Concept
- name: Primary Business Profile
type: Concept
- name: T-Mobile
type: Client
- name: Sole Proprietor
type: Concept
- name: pest control client
type: Client
facet: client/pest-control
relationships:
- source: OnCadence
type: acts as ISV/CSP on
target: Twilio
- source: OnCadence
type: submits registrations on behalf of
target: pest control client
- source: OnCadence
type: uses
target: TrustHub API
- source: pest control client
type: registered as
target: Secondary Customer Profile
- source: pest control client
type: requires
target: Brand registration
- source: pest control client
type: requires
target: Campaign registration
- source: OnCadence
type: creates once
target: Primary Business Profile
- source: Sole Proprietor
type: has lower throughput on
target: T-Mobile
- source: 10DLC
type: governs
target: Campaign registration
- source: TrustHub API
type: is part of
target: Twilio

View File

@ -0,0 +1,26 @@
# Reference Outputs
_Last updated: 2026-06-04_
This directory holds the per-model Graphify-shaped extraction fragments generated by the
Claude-tier reference-set benchmark (see `../dispatch-prompt.md`).
## File naming convention
```
<note-slug>.<tier>.md
```
Where:
- `<note-slug>` is the kebab-case identifier for the fixture note (assigned in Step 1c of the
build runbook)
- `<tier>` is one of: `haiku`, `sonnet`, `opus`
## Contents
Each file contains the YAML extraction fragment for one fixture note as produced by one Claude
tier. Files are generated by running the dispatch prompt — they do not exist until that step
is executed.
`claude-opus-4-8`'s `.opus.md` files are the **gold-standard rubric** against which local Ollama
doc-extraction models are scored in a later step.

View File

@ -0,0 +1,149 @@
note_slug: ai-coding-conventions-synthesis
entities:
- name: Claude
type: tool
facet: tool/claude
- name: Cursor
type: tool
facet: tool/cursor
- name: Copilot
type: tool
facet: tool/copilot
- name: CLAUDE.md
type: document_type
facet: convention/root-file
- name: .cursorrules
type: document_type
facet: convention/root-file
- name: AGENTS.md
type: document_type
facet: convention/hierarchical-scoping
- name: Three-Tier Architecture
type: framework
facet: convention/context-architecture
- name: Constitution
type: framework_component
facet: convention/constitution-pattern
- name: Domain Specialist Agents
type: framework_component
facet: convention/specialist-agents
- name: Knowledge Base
type: framework_component
- name: Codified Context
type: research_paper
facet: domain/context-engineering
- name: arXiv 2602.20478
type: publication
- name: Nuxt Content
type: tool
facet: tool/nuxt
- name: MCP
type: tool
facet: tool/mcp
- name: Token Efficiency Strategies
type: practice_set
facet: convention/token-optimization
- name: Just-in-time retrieval
type: practice
facet: convention/context-loading
- name: Linters
type: tool_category
facet: tool/linting
- name: Formatters
type: tool_category
facet: tool/formatting
- name: Feedback loop documentation
type: practice
facet: convention/feedback-loop
- name: Process encoding
type: practice
facet: convention/process-encoding
- name: Trigger tables
type: mechanism
facet: convention/routing-mechanism
- name: Reference documentation
type: practice
facet: convention/documentation-strategy
- name: Hierarchical override
type: mechanism
facet: convention/context-scoping
- name: github.com/arisvas4/codified-context-infrastructure
type: reference
- name: Anthropic Effective Context Engineering
type: reference
- name: PatrickJS/awesome-cursorrules
type: reference
- name: alexop.dev
type: reference
- name: mbleigh.dev
type: reference
- name: agents.md specification
type: standard
- name: 99 Bottles OOP — Full Software Design Process Map
type: reference
- name: OO Principles Plugin Concept — Design Recommendations
type: reference
relationships:
- source: Three-Tier Architecture
type: comprises
target: Constitution
- source: Three-Tier Architecture
type: comprises
target: Domain Specialist Agents
- source: Three-Tier Architecture
type: comprises
target: Knowledge Base
- source: Constitution
type: routes to
target: Domain Specialist Agents
- source: Domain Specialist Agents
type: contains
target: Knowledge Base
confidence: INFERRED
- source: Knowledge Base
type: served via
target: MCP
- source: CLAUDE.md
type: implements
target: Constitution
- source: .cursorrules
type: implements
target: Constitution
- source: AGENTS.md
type: implements
target: Hierarchical override
- source: Token Efficiency Strategies
type: includes
target: Just-in-time retrieval
- source: Token Efficiency Strategies
type: includes
target: Linters
- source: Token Efficiency Strategies
type: includes
target: Formatters
- source: Token Efficiency Strategies
type: includes
target: Feedback loop documentation
- source: Linters
type: replaces
target: Reference documentation
confidence: INFERRED
- source: Formatters
type: replaces
target: Reference documentation
confidence: INFERRED
- source: Process encoding
type: encodes
target: Trigger tables
- source: Trigger tables
type: enforces
target: Constitution
confidence: INFERRED
- source: Codified Context
type: describes
target: Three-Tier Architecture
- source: Codified Context
type: published at
target: arXiv 2602.20478

View File

@ -0,0 +1,156 @@
# --- Graphify extraction fragment ---
note_slug: ai-coding-conventions-synthesis
entities:
- name: Claude
type: Tool
facet: tool/claude
- name: Cursor
type: Tool
facet: tool/cursor
- name: Copilot
type: Tool
facet: tool/copilot
- name: Codified Context
type: Concept
- name: Three-Tier Architecture
type: Concept
- name: Constitution
type: Concept
- name: Domain Specialist Agents
type: Concept
- name: Knowledge Base
type: Concept
- name: MCP
type: Tool
facet: tool/mcp
- name: Just-in-time retrieval
type: Convention
facet: convention/just-in-time-retrieval
- name: Trigger tables
type: Concept
- name: Progressive disclosure
type: Convention
facet: convention/progressive-disclosure
- name: Positive framing only
type: Convention
facet: convention/positive-framing
- name: /learn pattern
type: Convention
facet: convention/feedback-loop-documentation
- name: CLAUDE.md
type: Convention
facet: convention/root-file
- name: .cursorrules
type: Convention
facet: convention/root-file
- name: AGENTS.md
type: Convention
facet: convention/root-file
- name: Cursor .mdc
type: Tool
- name: alwaysApply
type: Convention
- name: Auto-attached rules
type: Convention
- name: Agent-requested rules
type: Convention
- name: Manual rules
type: Convention
- name: Hierarchical scoping
type: Convention
facet: convention/hierarchical-scoping
- name: Nuxt Content
type: Tool
facet: tool/nuxt-content
- name: Codified Context paper
type: Concept
- name: github.com/arisvas4/codified-context-infrastructure
type: Project
- name: Anthropic Effective Context Engineering
type: Concept
- name: PatrickJS/awesome-cursorrules
type: Project
- name: alexop.dev — Stop Bloating Your CLAUDE.md
type: Concept
- name: mbleigh.dev — Rules for Rules
type: Concept
- name: agents.md
type: Project
- name: OpenAI
type: Client
- name: Anthropic
type: Client
- name: Context engineering
type: Domain
facet: domain/context-engineering
- name: AI coding conventions
type: Domain
facet: domain/ai-coding-conventions
- name: 99 Bottles OOP — Full Software Design Process Map
type: Concept
- name: OO Principles Plugin Concept — Design Recommendations
type: Concept
- name: Linters
type: Tool
- name: Formatters
type: Tool
- name: Type checkers
type: Tool
relationships:
- source: Three-Tier Architecture
type: comprises
target: Constitution
- source: Three-Tier Architecture
type: comprises
target: Domain Specialist Agents
- source: Three-Tier Architecture
type: comprises
target: Knowledge Base
- source: Codified Context
type: documents
target: Three-Tier Architecture
- source: Constitution
type: contains
target: Trigger tables
- source: Knowledge Base
type: served_via
target: MCP
- source: Codified Context paper
type: described_in
target: github.com/arisvas4/codified-context-infrastructure
confidence: INFERRED
- source: Constitution
type: routes_tasks_to
target: Domain Specialist Agents
- source: OpenAI
type: uses
target: AGENTS.md
- source: Anthropic
type: published
target: Anthropic Effective Context Engineering
confidence: INFERRED
- source: Cursor
type: uses
target: .cursorrules
confidence: INFERRED
- source: Cursor .mdc
type: supports
target: alwaysApply
- source: Cursor .mdc
type: supports
target: Auto-attached rules
- source: Cursor .mdc
type: supports
target: Agent-requested rules
- source: Cursor .mdc
type: supports
target: Manual rules
- source: Linters
type: replace
target: alwaysApply
confidence: INFERRED
- source: Trigger tables
type: route_to
target: Domain Specialist Agents

View File

@ -0,0 +1,135 @@
# --- Graphify extraction fragment ---
note_slug: ai-coding-conventions-synthesis
entities:
- name: Codified Context
type: Concept
facet: convention/codified-context
confidence: INFERRED
- name: Three-Tier Architecture
type: Concept
facet: convention/three-tier-architecture
- name: Constitution
type: Concept
facet: convention/constitution
- name: Domain Specialist Agents
type: Concept
facet: convention/domain-specialist-agents
- name: Knowledge Base
type: Concept
facet: convention/knowledge-base
- name: CLAUDE.md
type: Convention
facet: tool/claude-md
- name: AGENTS.md
type: Convention
facet: tool/agents-md
- name: Cursor
type: Tool
facet: tool/cursor
- name: Claude
type: Tool
facet: tool/claude
- name: Copilot
type: Tool
facet: tool/copilot
- name: MCP
type: Tool
facet: tool/mcp
- name: arXiv 2602.20478
type: Reference
- name: github.com/arisvas4/codified-context-infrastructure
type: Reference
- name: Anthropic Effective Context Engineering
type: Reference
- name: PatrickJS/awesome-cursorrules
type: Reference
- name: alexop.dev — Stop Bloating Your CLAUDE.md
type: Reference
- name: mbleigh.dev — Rules for Rules
type: Reference
- name: agents.md
type: Reference
- name: progressive-disclosure
type: Concept
facet: convention/progressive-disclosure
- name: trigger tables
type: Concept
facet: convention/trigger-tables
- name: feedback loop documentation
type: Concept
facet: convention/feedback-loop-documentation
- name: hierarchical scoping
type: Concept
facet: convention/hierarchical-scoping
- name: token efficiency
type: Concept
facet: convention/token-efficiency
- name: just-in-time retrieval
type: Concept
facet: convention/just-in-time-retrieval
- name: cursor rules
type: Convention
facet: tool/cursor-rules
- name: hyperthrive_dev
type: Person
confidence: AMBIGUOUS
relationships:
- source: Three-Tier Architecture
type: documented_in
target: arXiv 2602.20478
- source: Three-Tier Architecture
type: consists_of
target: Constitution
- source: Three-Tier Architecture
type: consists_of
target: Domain Specialist Agents
- source: Three-Tier Architecture
type: consists_of
target: Knowledge Base
- source: Constitution
type: uses
target: trigger tables
- source: Domain Specialist Agents
type: invoked_by
target: trigger tables
- source: Knowledge Base
type: served_via
target: MCP
- source: CLAUDE.md
type: implements
target: Constitution
confidence: INFERRED
- source: AGENTS.md
type: implements
target: Constitution
confidence: INFERRED
- source: cursor rules
type: implements
target: hierarchical scoping
- source: Three-Tier Architecture
type: enables
target: token efficiency
- source: progressive-disclosure
type: supports
target: token efficiency
- source: just-in-time retrieval
type: supports
target: token efficiency
- source: feedback loop documentation
type: converts_errors_into
target: Constitution
confidence: INFERRED
- source: Codified Context
type: described_by
target: arXiv 2602.20478
- source: Codified Context
type: exemplified_by
target: github.com/arisvas4/codified-context-infrastructure
- source: Claude
type: guided_by
target: CLAUDE.md
- source: Cursor
type: guided_by
target: cursor rules

View File

@ -0,0 +1,150 @@
note_slug: oo-principles-plugin-concept
entities:
- name: OO Principles Plugin
type: plugin
facet: type/plugin
- name: 99 Bottles of OOP
type: reference-book
facet: domain/oo-design
- name: Shameless Green
type: principle
facet: convention/shameless-green
- name: Flocking Rules
type: principle
facet: convention/flocking-rules
- name: Replace Conditional with Polymorphism
type: refactoring-technique
facet: convention/polymorphism-refactoring
- name: Single Responsibility Principle
type: principle
facet: convention/srp
- name: Law of Demeter
type: principle
facet: convention/law-of-demeter
- name: Dependency Injection
type: principle
facet: convention/dependency-injection
- name: Open/Closed Principle
type: principle
facet: convention/open-closed
- name: Process/Routing Layer
type: architectural-component
facet: type/architecture
- name: Mechanic Layer
type: architectural-component
facet: type/architecture
- name: Theory Layer
type: architectural-component
facet: type/architecture
- name: PROCESS.md
type: documentation-file
facet: convention/process-routing
- name: concept cards
type: documentation-format
- name: NotebookLM notebook
type: knowledge-base
facet: tool/notebooklm
- name: Phase 1
type: development-phase
- name: Phase 2
type: development-phase
- name: Phase 3
type: development-phase
- name: 4-phase development lifecycle
type: process-model
- name: Approach 1: Process-First Entry Point
type: architectural-approach
- name: Approach 2: Situational Trigger Files
type: architectural-approach
- name: Approach 3: Phase-Bundled Context Packs
type: architectural-approach
- name: Approach 4: Annotated Process Graph
type: architectural-approach
- name: conventions/oo-principles/
type: workspace-structure
facet: project/hyperthrive
- name: hyperthrive_dev
type: project
facet: project/hyperthrive
- name: Codified Context paper
type: research-reference
- name: TDD
type: principle
facet: convention/tdd
- name: QUICK_REFERENCE.md
type: documentation-file
- name: role bundles
type: documentation-format
- name: AI Coding Conventions Organization — External Research Synthesis
type: reference-document
- name: 99 Bottles OOP — Full Software Design Process Map
type: reference-document
- name: CLAUDE.md
type: documentation-file
facet: tool/claude-code
relationships:
- source: OO Principles Plugin
type: is designed to encode
target: 4-phase development lifecycle
- source: OO Principles Plugin
type: models architecture on
target: Process/Routing Layer
- source: OO Principles Plugin
type: models architecture on
target: Mechanic Layer
- source: OO Principles Plugin
type: models architecture on
target: Theory Layer
- source: Mechanic Layer
type: should contain recipes for
target: Flocking Rules
- source: Mechanic Layer
type: should contain recipes for
target: Replace Conditional with Polymorphism
- source: Theory Layer
type: contains explanatory material for
target: concept cards
- source: Process/Routing Layer
type: references
target: Mechanic Layer
- source: Flocking Rules
type: competes with
target: Replace Conditional with Polymorphism
- source: Dependency Injection
type: should be checked before
target: Law of Demeter
- source: Law of Demeter
type: should be checked before
target: Single Responsibility Principle
- source: Phase 2
type: includes opportunistic refactoring guided by
target: Law of Demeter
- source: Phase 2
type: includes opportunistic refactoring guided by
target: Dependency Injection
- source: Approach 1: Process-First Entry Point
type: is recommended for
target: OO Principles Plugin
- source: NotebookLM notebook
type: serves as knowledge base for
target: OO Principles Plugin
- source: conventions/oo-principles/
type: is baseline for
target: OO Principles Plugin
- source: hyperthrive_dev
type: is source context for
target: OO Principles Plugin
- source: OO Principles Plugin
type: based on
target: 99 Bottles of OOP
- source: Process/Routing Layer
type: encodes
target: 4-phase development lifecycle
- source: Phase 1
type: applies
target: Shameless Green
- source: Phase 3
type: requires checking
target: Open/Closed Principle

View File

@ -0,0 +1,159 @@
# --- Graphify extraction fragment ---
note_slug: oo-principles-plugin-concept
entities:
- name: OO Principles Plugin
type: plugin
facet: type/plugin
- name: oo-principles conventions structure
type: conventions structure
facet: convention/oo-principles
- name: 99 Bottles of OOP
type: book
facet: domain/object-oriented-design
- name: NotebookLM notebook
type: knowledge base resource
facet: tool/notebooklm
- name: Codified Context paper
type: research paper
- name: TDD
type: principle
facet: convention/tdd
- name: Shameless Green
type: principle
facet: convention/shameless-green
- name: Flocking Rules
type: principle
facet: convention/flocking-rules
- name: SRP
type: principle
facet: convention/srp
- name: LoD
type: principle
facet: convention/law-of-demeter
- name: DI
type: principle
facet: convention/dependency-injection
- name: Replace Conditional with Polymorphism
type: principle
facet: convention/replace-conditional-with-polymorphism
- name: Open/Closed
type: principle
facet: convention/open-closed
- name: 4-phase development lifecycle
type: process model
- name: QUICK_REFERENCE.md
type: file
- name: cards/
type: directory
- name: bundles/
type: directory
- name: Two-Layer Hybrid
type: architecture
- name: Process/Routing Layer
type: layer
- name: Mechanic Layer
type: layer
- name: Theory Layer
type: layer
- name: PROCESS.md
type: file
- name: Approach 1 - Process-First Entry Point
type: architectural approach
- name: Approach 2 - Situational Trigger Files
type: architectural approach
- name: Approach 3 - Phase-Bundled Context Packs
type: architectural approach
- name: Approach 4 - Annotated Process Graph
type: architectural approach
- name: Phase 2
type: development phase
- name: three-tier architecture
type: architecture
- name: CLAUDE.md
type: file
- name: 99 Bottles OOP — Full Software Design Process Map
type: note
- name: AI Coding Conventions Organization — External Research Synthesis
type: note
relationships:
- source: OO Principles Plugin
type: evolves from
target: oo-principles conventions structure
- source: OO Principles Plugin
type: based on study of
target: 99 Bottles of OOP
- source: OO Principles Plugin
type: informed by research on
target: Codified Context paper
- source: OO Principles Plugin
type: can use as knowledge base layer
target: NotebookLM notebook
- source: oo-principles conventions structure
type: contains
target: QUICK_REFERENCE.md
- source: oo-principles conventions structure
type: contains
target: cards/
- source: oo-principles conventions structure
type: contains
target: bundles/
- source: Two-Layer Hybrid
type: includes
target: Process/Routing Layer
- source: Two-Layer Hybrid
type: includes
target: Mechanic Layer
- source: Two-Layer Hybrid
type: includes
target: Theory Layer
- source: Process/Routing Layer
type: encodes
target: 4-phase development lifecycle
- source: Process/Routing Layer
type: contains
target: PROCESS.md
- source: Process/Routing Layer
type: references
target: Mechanic Layer
- source: Theory Layer
type: retains
target: cards/
confidence: INFERRED
- source: Approach 1 - Process-First Entry Point
type: adds as entry point
target: PROCESS.md
- source: 99 Bottles of OOP
type: defines
target: Shameless Green
confidence: INFERRED
- source: 99 Bottles of OOP
type: defines
target: Flocking Rules
confidence: INFERRED
- source: Phase 2
type: addresses
target: LoD
- source: Phase 2
type: addresses
target: DI
- source: Replace Conditional with Polymorphism
type: alternative to
target: Flocking Rules
- source: 4-phase development lifecycle
type: included as loadable context pack in
target: OO Principles Plugin
- source: OO Principles Plugin
type: modeled on
target: three-tier architecture
- source: three-tier architecture
type: routing entry point is
target: CLAUDE.md
confidence: INFERRED
- source: oo-principles conventions structure
type: see also
target: 99 Bottles OOP — Full Software Design Process Map
- source: oo-principles conventions structure
type: see also
target: AI Coding Conventions Organization — External Research Synthesis

View File

@ -0,0 +1,126 @@
# --- Graphify extraction fragment ---
note_slug: oo-principles-plugin-concept
entities:
- name: OO Principles Plugin
type: project
facet: project/oo-principles-plugin
- name: conventions/oo-principles
type: directory-structure
facet: convention/oo-principles
- name: QUICK_REFERENCE.md
type: file
- name: concept cards
type: document-type
- name: role bundles
type: document-type
- name: PROCESS.md
type: file
- name: 99 Bottles of OOP
type: book
facet: domain/software-design
- name: Codified Context paper
type: external-reference
- name: NotebookLM
type: tool
facet: tool/notebooklm
- name: Shameless Green
type: development-principle
facet: convention/shameless-green
- name: Flocking Rules
type: development-principle
facet: convention/flocking-rules
- name: Replace Conditional with Polymorphism
type: development-principle
facet: convention/replace-conditional-with-polymorphism
- name: TDD
type: development-principle
facet: convention/tdd
- name: SRP
type: development-principle
facet: convention/srp
- name: Law of Demeter
type: development-principle
facet: convention/law-of-demeter
- name: Dependency Injection
type: development-principle
facet: convention/dependency-injection
- name: 4-phase development lifecycle
type: concept
facet: convention/4-phase-lifecycle
- name: Process-First Entry Point
type: architectural-approach
- name: Situational Trigger Files
type: architectural-approach
- name: Phase-Bundled Context Packs
type: architectural-approach
- name: Annotated Process Graph
type: architectural-approach
- name: Two-Layer Hybrid architecture
type: architectural-pattern
- name: hyperthrive_dev
type: client
facet: client/hyperthrive-dev
- name: Rails
type: tool
facet: tool/rails
- name: CLAUDE.md
type: file
- name: AI Coding Conventions Organization — External Research Synthesis
type: note
confidence: INFERRED
- name: 99 Bottles OOP — Full Software Design Process Map
type: note
confidence: INFERRED
relationships:
- source: OO Principles Plugin
type: evolved from
target: conventions/oo-principles
- source: OO Principles Plugin
type: uses architecture
target: Two-Layer Hybrid architecture
- source: Two-Layer Hybrid architecture
type: layer 1 is
target: PROCESS.md
- source: Two-Layer Hybrid architecture
type: layer 3 is
target: concept cards
- source: OO Principles Plugin
type: informed by
target: 99 Bottles of OOP
- source: OO Principles Plugin
type: informed by
target: Codified Context paper
- source: PROCESS.md
type: encodes
target: 4-phase development lifecycle
- source: Flocking Rules
type: contrasted with
target: Replace Conditional with Polymorphism
- source: Shameless Green
type: part of
target: 4-phase development lifecycle
- source: Process-First Entry Point
type: recommended over
target: Situational Trigger Files
- source: Process-First Entry Point
type: recommended over
target: Phase-Bundled Context Packs
- source: Process-First Entry Point
type: recommended over
target: Annotated Process Graph
- source: NotebookLM
type: serves as knowledge base for
target: OO Principles Plugin
- source: hyperthrive_dev
type: source workspace for
target: conventions/oo-principles
- source: CLAUDE.md
type: entry point for
target: conventions/oo-principles
confidence: INFERRED
- source: OO Principles Plugin
type: links to
target: 99 Bottles OOP — Full Software Design Process Map
- source: OO Principles Plugin
type: links to
target: AI Coding Conventions Organization — External Research Synthesis

View File

@ -0,0 +1,177 @@
note_slug: pest-control-email-abc-hub
entities:
- name: Pest Control Email A/B/C Test
type: experiment
- name: Golden 4-Sentence Framework
type: email draft variant
- name: Revised
type: email draft variant
- name: Copywriter
type: email draft variant
- name: pest-control-spring-2026
type: campaign
facet: project/pest-control-spring-2026
- name: Mike Deluca
type: persona
facet: client/pest-control-owner
confidence: INFERRED
- name: Sandra Kowalski
type: persona
facet: client/pest-control-owner
confidence: INFERRED
- name: Ray Tanner
type: persona
facet: client/pest-control-owner
confidence: INFERRED
- name: gatekeeper personas
type: judge group
- name: pest control owner-operators
type: judge group
- name: Group 3 Revised
type: email group variant
- name: Group 7 Copywriter
type: email group variant
- name: Group 9 Revised
type: email group variant
- name: Group 2 Copywriter
type: email group variant
- name: Group 5 Revised
type: email group variant
- name: Two-scenario contrast format
type: copy pattern
facet: convention/email-copy-pattern
- name: Scenario first, stat second
type: structural pattern
facet: convention/email-copy-pattern
- name: Reframe before solution
type: structural pattern
facet: convention/email-copy-pattern
- name: Single closing question with honest out
type: structural pattern
facet: convention/email-copy-pattern
- name: Review-shaming openers
type: anti-pattern
facet: convention/email-copy-pattern
- name: Fabricated revenue math
type: anti-pattern
facet: convention/email-copy-pattern
- name: Escape-hatch closes
type: anti-pattern
facet: convention/email-copy-pattern
- name: Self-aware meta-openers
type: anti-pattern
facet: convention/email-copy-pattern
- name: Saturday wasp nest scenario
type: email hook
- name: Scenario A vs. B two-format
type: email hook
- name: Lead disappearance problem reframe
type: email hook
- name: Bed bugs at 11pm scenario
type: email hook
- name: Termite LTV urgency
type: email hook
- name: GorillaDesk/P&L email
type: referenced email version
confidence: INFERRED
- name: LSA-anchor version
type: email variant
- name: Driven Results
type: citation source
confidence: INFERRED
relationships:
- source: Pest Control Email A/B/C Test
type: compares
target: Golden 4-Sentence Framework
- source: Pest Control Email A/B/C Test
type: compares
target: Revised
- source: Pest Control Email A/B/C Test
type: compares
target: Copywriter
- source: Pest Control Email A/B/C Test
type: part of campaign
target: pest-control-spring-2026
- source: Pest Control Email A/B/C Test
type: judged by
target: gatekeeper personas
- source: Pest Control Email A/B/C Test
type: judged by
target: pest control owner-operators
- source: Mike Deluca
type: participates in
target: Pest Control Email A/B/C Test
- source: Sandra Kowalski
type: participates in
target: Pest Control Email A/B/C Test
- source: Ray Tanner
type: participates in
target: Pest Control Email A/B/C Test
- source: Revised
type: wins
target: Pest Control Email A/B/C Test
- source: Copywriter
type: runner-up in
target: Pest Control Email A/B/C Test
- source: Group 3 Revised
type: unanimous sweep in
target: Pest Control Email A/B/C Test
- source: Group 7 Copywriter
type: unanimous sweep in
target: Pest Control Email A/B/C Test
- source: Two-scenario contrast format
type: recommended for adoption
target: pest-control-spring-2026
- source: Scenario first, stat second
type: recommended for adoption
target: pest-control-spring-2026
- source: Reframe before solution
type: recommended for adoption
target: pest-control-spring-2026
- source: Single closing question with honest out
type: recommended for adoption
target: pest-control-spring-2026
- source: Review-shaming openers
type: recommended to retire
target: pest-control-spring-2026
- source: Fabricated revenue math
type: recommended to retire
target: pest-control-spring-2026
- source: Escape-hatch closes
type: recommended to retire
target: pest-control-spring-2026
- source: Self-aware meta-openers
type: recommended to retire
target: pest-control-spring-2026
- source: Group 3 Revised
type: uses hook
target: Saturday wasp nest scenario
- source: Group 7 Copywriter
type: uses hook
target: Scenario A vs. B two-format
- source: Group 9 Revised
type: uses hook
target: Lead disappearance problem reframe
- source: Group 2 Copywriter
type: uses hook
target: Bed bugs at 11pm scenario
- source: Group 5 Revised
type: uses hook
target: Termite LTV urgency
- source: gatekeeper personas
type: predicted preference for
target: Revised
confidence: AMBIGUOUS
- source: pest control owner-operators
type: prefer
target: Revised
- source: LSA-anchor version
type: works for
target: paid-acquisition operators
confidence: INFERRED
- source: LSA-anchor version
type: rejected by
target: referral-only operator
confidence: INFERRED

View File

@ -0,0 +1,192 @@
# --- Graphify extraction fragment ---
note_slug: pest-control-email-abc-hub
entities:
- name: Pest Control Email A/B/C Test
type: experiment
facet: type/experiment
- name: pest-control-spring-2026
type: campaign
facet: project/pest-control-spring-2026
- name: Golden 4-Sentence Framework
type: copy framework
facet: convention/golden-4-sentence-framework
- name: draft
type: email version
- name: revised
type: email version
- name: copywriter
type: email version
- name: Mike Deluca
type: owner-operator persona
- name: Sandra Kowalski
type: owner-operator persona
- name: Ray Tanner
type: owner-operator persona
- name: gatekeeper panel
type: judging panel
- name: owner panel
type: judging panel
- name: Group 3
type: email group
- name: Group 7
type: email group
- name: Group 2
type: email group
- name: Group 4
type: email group
- name: Group 5
type: email group
- name: Group 8
type: email group
- name: Group 9
type: email group
- name: Saturday wasp nest scenario
type: email hook
- name: Scenario A vs. B two-format
type: email hook
- name: Lead disappearance problem reframe
type: email hook
- name: Bed bugs at 11pm scenario
type: email hook
- name: Termite LTV urgency
type: email hook
- name: Two-scenario contrast format
type: structural pattern
facet: convention/two-scenario-contrast-format
- name: Scenario first, stat second
type: structural pattern
facet: convention/scenario-first-stat-second
- name: Reframe before solution
type: structural pattern
facet: convention/reframe-before-solution
- name: Single closing question with an honest out
type: structural pattern
facet: convention/single-closing-question-honest-out
- name: Fabricated revenue math
type: retired pattern
- name: Review-shaming openers
type: retired pattern
- name: Escape-hatch closes
type: retired pattern
- name: Guessed specificity
type: retired pattern
- name: Self-aware meta-openers
type: retired pattern
- name: GorillaDesk/P&L email
type: email
- name: Driven Results citation
type: citation
- name: LSA-anchor version
type: email version
- name: $136,500 ROI projection
type: copy claim
relationships:
- source: Pest Control Email A/B/C Test
type: tested for campaign
target: pest-control-spring-2026
- source: Pest Control Email A/B/C Test
type: compared
target: draft
- source: Pest Control Email A/B/C Test
type: compared
target: revised
- source: Pest Control Email A/B/C Test
type: compared
target: copywriter
- source: draft
type: built on framework
target: Golden 4-Sentence Framework
- source: gatekeeper panel
type: judged
target: Pest Control Email A/B/C Test
- source: owner panel
type: judged
target: Pest Control Email A/B/C Test
- source: owner panel
type: includes persona
target: Mike Deluca
- source: owner panel
type: includes persona
target: Sandra Kowalski
- source: owner panel
type: includes persona
target: Ray Tanner
- source: revised
type: won overall
target: Pest Control Email A/B/C Test
- source: copywriter
type: led in raw first-place owner votes
target: revised
- source: Group 3
type: swept by version
target: revised
- source: Group 7
type: swept by version
target: copywriter
- source: Group 3
type: uses hook
target: Saturday wasp nest scenario
- source: Group 3
type: cites
target: Driven Results citation
- source: Group 7
type: uses hook
target: Scenario A vs. B two-format
- source: Group 9
type: uses hook
target: Lead disappearance problem reframe
- source: Group 2
type: uses hook
target: Bed bugs at 11pm scenario
- source: Group 5
type: uses hook
target: Termite LTV urgency
- source: Group 8
type: gatekeepers ranked first
target: GorillaDesk/P&L email
- source: Group 5
type: contains copy claim
target: $136,500 ROI projection
confidence: INFERRED
- source: LSA-anchor version
type: appears in
target: Group 4
- source: Group 7
type: exemplifies pattern
target: Two-scenario contrast format
- source: Group 9
type: exemplifies pattern
target: Reframe before solution
confidence: INFERRED
- source: Pest Control Email A/B/C Test
type: recommends for live sequence
target: Group 3
- source: Pest Control Email A/B/C Test
type: recommends for live sequence
target: Group 7
- source: Pest Control Email A/B/C Test
type: recommends for live sequence
target: Group 9
- source: Pest Control Email A/B/C Test
type: recommends for live sequence
target: Group 2
- source: Pest Control Email A/B/C Test
type: recommends for live sequence
target: Group 5
- source: Pest Control Email A/B/C Test
type: recommends retiring
target: Fabricated revenue math
- source: Pest Control Email A/B/C Test
type: recommends retiring
target: Review-shaming openers
- source: Pest Control Email A/B/C Test
type: recommends retiring
target: Escape-hatch closes
- source: Pest Control Email A/B/C Test
type: recommends retiring
target: Guessed specificity
- source: Pest Control Email A/B/C Test
type: recommends retiring
target: Self-aware meta-openers

View File

@ -0,0 +1,104 @@
# --- Graphify extraction fragment ---
note_slug: pest-control-email-abc-hub
entities:
- name: pest-control-spring-2026
type: campaign
facet: project/pest-control-spring-2026
- name: Golden 4-Sentence Framework
type: email framework
facet: convention/golden-4-sentence-framework
- name: Revised
type: email version
- name: Copywriter
type: email version
- name: Draft
type: email version
- name: Mike Deluca
type: owner persona
- name: Sandra Kowalski
type: owner persona
- name: Ray Tanner
type: owner persona
- name: Driven Results
type: organization
- name: GorillaDesk
type: tool
facet: tool/gorillaDesk
- name: Two-scenario contrast format
type: copy pattern
facet: convention/two-scenario-contrast-format
- name: Saturday wasp nest scenario
type: email hook
- name: Scenario A vs. B two-format
type: email structure
facet: convention/scenario-a-vs-b-two-format
- name: lead disappearance problem reframe
type: copy pattern
facet: convention/lead-disappearance-reframe
- name: LSA-anchor version
type: email version
- name: $136,500 ROI projection
type: copy element
- name: review-shaming opener
type: copy pattern
- name: escape-hatch close
type: copy pattern
- name: cold email
type: domain
facet: domain/cold-email
relationships:
- source: Revised
type: won overall in
target: pest-control-spring-2026
- source: Copywriter
type: runner-up in
target: pest-control-spring-2026
- source: Draft
type: placed third in
target: pest-control-spring-2026
- source: Mike Deluca
type: judged as owner persona in
target: pest-control-spring-2026
- source: Sandra Kowalski
type: judged as owner persona in
target: pest-control-spring-2026
- source: Ray Tanner
type: judged as owner persona in
target: pest-control-spring-2026
- source: Saturday wasp nest scenario
type: achieved unanimous sweep in
target: pest-control-spring-2026
- source: Scenario A vs. B two-format
type: achieved unanimous sweep in
target: pest-control-spring-2026
- source: Two-scenario contrast format
type: earned identical reactions from all owner personas in
target: pest-control-spring-2026
- source: LSA-anchor version
type: requires audience segmentation for
target: pest-control-spring-2026
- source: $136,500 ROI projection
type: recommended to retire from
target: pest-control-spring-2026
- source: review-shaming opener
type: recommended to retire from
target: pest-control-spring-2026
- source: escape-hatch close
type: recommended to retire from
target: pest-control-spring-2026
- source: Revised
type: uses citation from
target: Driven Results
- source: lead disappearance problem reframe
type: appears in
target: Revised
confidence: INFERRED
- source: GorillaDesk
type: referenced in email version
target: Copywriter
confidence: INFERRED
- source: Golden 4-Sentence Framework
type: basis for
target: Draft

View File

@ -0,0 +1,124 @@
note_slug: pest-control-enterprise-revenue
entities:
- name: Gemini
type: Tool
facet: tool/gemini
- name: Pest Control Enterprise
type: Concept
facet: domain/pest-control
- name: Recurring Revenue
type: Concept
- name: Termite Swarming
type: Concept
facet: domain/pest-control
- name: Growing Degree Days
type: Convention
facet: convention/gdd-forecasting
- name: Heatwave-Driven Migrations
type: Concept
facet: domain/pest-control
- name: First Freeze Rodent Push
type: Concept
facet: domain/pest-control
- name: PPC
type: Tool
facet: tool/ppc
- name: SEO
type: Tool
facet: tool/seo
- name: Customer Acquisition Cost
type: Concept
- name: Route Density
type: Concept
facet: domain/pest-control
- name: Rollins
type: Client
- name: Rentokil
type: Client
- name: Ecolab
type: Client
- name: Terminix
type: Client
- name: Churn Rate
type: Concept
- name: Communication Failure Patterns
type: Concept
facet: domain/pest-control
- name: Northeast Region
type: Concept
facet: scope/geographic
- name: Mid-Atlantic Region
type: Concept
facet: scope/geographic
- name: Southeast Region
type: Concept
facet: scope/geographic
- name: Southwest Region
type: Concept
facet: scope/geographic
- name: Northwest Region
type: Concept
facet: scope/geographic
- name: Commercial Pest Control
type: Concept
facet: domain/pest-control
- name: Offshore Customer Service
type: Concept
confidence: INFERRED
- name: M&A Valuation
type: Concept
facet: domain/pest-control
relationships:
- source: Pest Control Enterprise
type: achieves recurring revenue through
target: Recurring Revenue
- source: Pest Control Enterprise
type: experiences lead generation spike during
target: Termite Swarming
- source: Growing Degree Days
type: predicts onset of
target: Termite Swarming
- source: Pest Control Enterprise
type: experiences demand spike during
target: Heatwave-Driven Migrations
- source: Pest Control Enterprise
type: experiences demand spike during
target: First Freeze Rodent Push
- source: PPC
type: captures demand during
target: Termite Swarming
- source: SEO
type: reduces long-term CAC for
target: Pest Control Enterprise
- source: Route Density
type: drives revenue per technician in
target: Pest Control Enterprise
- source: Rollins
type: consolidates market share in
target: Pest Control Enterprise
- source: Rentokil
type: consolidates market share in
target: Pest Control Enterprise
- source: Ecolab
type: consolidates market share in
target: Pest Control Enterprise
- source: Terminix
type: consolidates market share in
target: Pest Control Enterprise
- source: Communication Failure Patterns
type: drives churn in
target: Pest Control Enterprise
- source: Churn Rate
type: correlates with customer satisfaction in
target: Communication Failure Patterns
- source: Offshore Customer Service
type: contributes to
target: Communication Failure Patterns
- source: Commercial Pest Control
type: represents market segment of
target: Pest Control Enterprise
- source: Gemini
type: source for
target: Pest Control Enterprise

View File

@ -0,0 +1,224 @@
# --- Graphify extraction fragment ---
note_slug: pest-control-enterprise-revenue
entities:
- name: Pest Control Enterprise
type: Domain
facet: domain/pest-control
- name: Recurring Revenue
type: Concept
- name: ACV
type: Concept
- name: Monthly maintenance
type: Concept
- name: Bi-monthly
type: Concept
- name: Tri-annual
type: Concept
- name: Quarterly
type: Concept
- name: Premium/Gold bundled
type: Concept
- name: Termite warranty renewal
type: Concept
- name: Initial service fee
type: Concept
- name: CAC
type: Concept
- name: LTV
type: Concept
- name: Termite monitoring/baiting
type: Concept
- name: Mosquito seasonal
type: Concept
- name: Rodent exclusion
type: Concept
- name: Wasp/hornet removal
type: Concept
- name: Bed bug remediation
type: Concept
- name: Communication Failure Patterns
type: Concept
- name: '"No one answered"'
type: Concept
- name: '"Voicemail full"'
type: Concept
- name: '"Dropped calls and ghosting"'
type: Concept
- name: Offshore agent disconnect
type: Concept
- name: CPL
type: Concept
- name: Churn
type: Concept
- name: Climatologically-Driven Lead Seasonality
type: Concept
- name: Termite Swarming
type: Concept
- name: Subterranean termites
type: Concept
- name: Growing Degree Days (GDD)
type: Concept
- name: PPC
type: Concept
- name: Heatwave-Driven Migrations
type: Concept
- name: First Freeze Rodent Push
type: Concept
- name: Northeast
type: Concept
- name: Mid-Atlantic
type: Concept
- name: Southeast
type: Concept
- name: Southwest
type: Concept
- name: Northwest
type: Concept
- name: Operational Unit Economics
type: Concept
- name: Revenue per technician
type: Concept
- name: COGS
type: Concept
- name: Direct labor
type: Concept
- name: Materials/chemicals
type: Concept
- name: Marketing/lead gen
type: Concept
- name: Vehicle ops
type: Concept
- name: Admin/management
type: Concept
- name: EBITDA
type: Concept
- name: SEO
type: Concept
- name: Marketing allocation model
type: Concept
- name: Commercial
type: Concept
- name: M&A
type: Concept
- name: Big Four
type: Concept
- name: Rollins
type: Client
- name: Rentokil
type: Client
- name: Ecolab
type: Client
- name: Terminix
type: Client
- name: Revenue multiples
type: Concept
- name: SDE multiples
type: Concept
- name: Route density
type: Concept
- name: Gemini
type: Tool
- name: SOPs
type: Concept
relationships:
- source: Recurring Revenue
type: comprises_share_of
target: Pest Control Enterprise
- source: Recurring Revenue
type: determines
target: M&A
- source: Initial service fee
type: offsets
target: CAC
- source: Initial service fee
type: inflates
target: ACV
- source: Termite warranty renewal
type: is_a
target: ACV
- source: Monthly maintenance
type: is_a
target: ACV
- source: Communication Failure Patterns
type: drives
target: Churn
- source: '"No one answered"'
type: wastes
target: CPL
- source: Offshore agent disconnect
type: misidentifies
target: Subterranean termites
- source: Termite Swarming
type: is_event_of
target: Climatologically-Driven Lead Seasonality
- source: Subterranean termites
type: cause
target: Termite Swarming
- source: Growing Degree Days (GDD)
type: predicts
target: Termite Swarming
- source: PPC
type: increases_with
target: Growing Degree Days (GDD)
- source: Heatwave-Driven Migrations
type: is_event_of
target: Climatologically-Driven Lead Seasonality
- source: First Freeze Rodent Push
type: is_event_of
target: Climatologically-Driven Lead Seasonality
- source: First Freeze Rodent Push
type: triggers
target: Rodent exclusion
- source: Northeast
type: experiences
target: Subterranean termites
- source: Mid-Atlantic
type: experiences
target: Subterranean termites
- source: Direct labor
type: component_of
target: COGS
- source: Materials/chemicals
type: component_of
target: COGS
- source: Marketing/lead gen
type: component_of
target: COGS
- source: Vehicle ops
type: component_of
target: COGS
- source: Admin/management
type: component_of
target: COGS
- source: Marketing allocation model
type: allocates_to
target: PPC
- source: Marketing allocation model
type: allocates_to
target: SEO
- source: Big Four
type: includes
target: Rollins
- source: Big Four
type: includes
target: Rentokil
- source: Big Four
type: includes
target: Ecolab
- source: Big Four
type: includes
target: Terminix
- source: Big Four
type: consolidates
target: Pest Control Enterprise
- source: SOPs
type: increases
target: SDE multiples
- source: Route density
type: drives
target: M&A
- source: Gemini
type: sourced
target: Pest Control Enterprise

View File

@ -0,0 +1,206 @@
# --- Graphify extraction fragment ---
note_slug: pest-control-enterprise-revenue
entities:
- name: Pest Control Enterprise
type: business-segment
facet: domain/pest-control
- name: Recurring Revenue
type: financial-concept
facet: domain/pest-control
- name: Monthly Maintenance Program
type: service-program
facet: domain/pest-control
- name: Bi-monthly Program
type: service-program
facet: domain/pest-control
- name: Tri-annual Program
type: service-program
facet: domain/pest-control
- name: Quarterly Program
type: service-program
facet: domain/pest-control
- name: Premium/Gold Bundled Program
type: service-program
facet: domain/pest-control
- name: Termite Warranty Renewal
type: service-program
facet: domain/pest-control
- name: Initial Service Fee
type: pricing-mechanism
facet: domain/pest-control
- name: Termite Monitoring/Baiting
type: specialty-service
facet: domain/pest-control
- name: Mosquito Seasonal Service
type: specialty-service
facet: domain/pest-control
- name: Rodent Exclusion
type: specialty-service
facet: domain/pest-control
- name: Wasp/Hornet Removal
type: specialty-service
facet: domain/pest-control
- name: Bed Bug Remediation
type: specialty-service
facet: domain/pest-control
- name: Missed Call Failure
type: communication-failure-pattern
facet: domain/pest-control
- name: Voicemail Full Failure
type: communication-failure-pattern
facet: domain/pest-control
- name: Dropped Calls and Ghosting
type: communication-failure-pattern
facet: domain/pest-control
- name: Offshore Agent Disconnect
type: communication-failure-pattern
facet: domain/pest-control
- name: Annual Churn
type: operational-metric
facet: domain/pest-control
- name: Termite Swarming Season
type: seasonal-lead-trigger
facet: domain/pest-control
- name: Heatwave-Driven Migrations
type: seasonal-lead-trigger
facet: domain/pest-control
- name: First Freeze Rodent Push
type: seasonal-lead-trigger
facet: domain/pest-control
- name: Growing Degree Days
type: forecasting-concept
facet: domain/pest-control
- name: Revenue per Technician
type: operational-metric
facet: domain/pest-control
- name: COGS Structure
type: financial-concept
facet: domain/pest-control
- name: Marketing Allocation Model
type: operational-framework
facet: domain/pest-control
- name: Commercial Pest Control
type: business-segment
facet: domain/pest-control
- name: Rollins
type: company
facet: domain/pest-control
- name: Rentokil
type: company
facet: domain/pest-control
- name: Ecolab
type: company
facet: domain/pest-control
- name: Terminix
type: company
facet: domain/pest-control
- name: Big Four
type: company-group
facet: domain/pest-control
- name: M&A Valuation
type: financial-concept
facet: domain/pest-control
- name: SDE Multiple
type: financial-metric
facet: domain/pest-control
- name: Revenue Multiple
type: financial-metric
facet: domain/pest-control
- name: Route Density
type: operational-metric
facet: domain/pest-control
- name: Customer Retention
type: operational-metric
facet: domain/pest-control
relationships:
- source: Pest Control Enterprise
type: generates
target: Recurring Revenue
- source: Recurring Revenue
type: determines
target: M&A Valuation
- source: Monthly Maintenance Program
type: is-tier-of
target: Recurring Revenue
- source: Bi-monthly Program
type: is-tier-of
target: Recurring Revenue
- source: Tri-annual Program
type: is-tier-of
target: Recurring Revenue
- source: Quarterly Program
type: is-tier-of
target: Recurring Revenue
- source: Premium/Gold Bundled Program
type: is-tier-of
target: Recurring Revenue
- source: Termite Warranty Renewal
type: is-tier-of
target: Recurring Revenue
- source: Initial Service Fee
type: offsets-CAC-for
target: Pest Control Enterprise
- source: Missed Call Failure
type: increases
target: Annual Churn
confidence: INFERRED
- source: Voicemail Full Failure
type: increases
target: Annual Churn
confidence: INFERRED
- source: Dropped Calls and Ghosting
type: increases
target: Annual Churn
confidence: INFERRED
- source: Offshore Agent Disconnect
type: causes
target: Dropped Calls and Ghosting
confidence: INFERRED
- source: Termite Swarming Season
type: drives-demand-for
target: Termite Monitoring/Baiting
- source: Heatwave-Driven Migrations
type: drives-demand-for
target: Mosquito Seasonal Service
- source: First Freeze Rodent Push
type: drives-demand-for
target: Rodent Exclusion
- source: Growing Degree Days
type: used-to-predict
target: Termite Swarming Season
- source: Marketing Allocation Model
type: allocates-spend-toward
target: Termite Swarming Season
- source: Rollins
type: member-of
target: Big Four
- source: Rentokil
type: member-of
target: Big Four
- source: Ecolab
type: member-of
target: Big Four
- source: Terminix
type: member-of
target: Big Four
- source: Big Four
type: consolidates
target: Pest Control Enterprise
- source: M&A Valuation
type: measured-by
target: SDE Multiple
- source: M&A Valuation
type: measured-by
target: Revenue Multiple
- source: Route Density
type: premium-driver-for
target: M&A Valuation
- source: Customer Retention
type: premium-driver-for
target: M&A Valuation
- source: Commercial Pest Control
type: segment-of
target: Pest Control Enterprise
- source: Annual Churn
type: inversely-affects
target: Customer Retention

View File

@ -0,0 +1,136 @@
note_slug: pest-control-sms-market-research-stats
entities:
- name: MIT/InsideSales.com Lead Response Management Study
type: research study
- name: Harvard Business Review
type: publication
facet: domain/sales-operations
- name: Velocify Ultimate Contact Strategy Study
type: research study
- name: InsideSales.com 2021
type: research study
- name: HomeAdvisor
type: company
facet: tool/homeadvisor
- name: Angi
type: company
facet: tool/angi
- name: CallRail
type: company
facet: tool/callrail
- name: PATLive
type: company
- name: Forbes
type: publication
- name: SellCell
type: publication
- name: Driven Results
type: research organization
- name: HubSpot
type: company
facet: tool/hubspot
- name: Google Ads
type: tool/google-ads
facet: tool/google-ads
- name: Google Local Services Ads
type: tool/google-local-services-ads
facet: tool/google-local-services-ads
- name: Thumbtack
type: company
facet: tool/thumbtack
- name: Coalmarch
type: company
- name: Valve+Meter
type: research organization
- name: Invoca
type: company
- name: PPMA
type: organization
- name: Scorpion
type: company
- name: NPMA
type: organization
- name: PCO Bookkeepers
type: organization
- name: Gartner
type: company
- name: D7 Networks
type: company
- name: CTIA
type: organization
- name: TransUnion
type: company
- name: Avochato
type: company
- name: Leads360
type: company
- name: Briostack
type: company
- name: Cube Creative Design
type: company
- name: ServiceTitan
type: company
facet: tool/servicetitan
- name: Point Loma Electric & Plumbing
type: company
- name: Hatch
type: company
- name: Shafer Services
type: company
- name: Aruza Pest Control
type: company
- name: After-Hours SMS Lead Capture
type: product concept
facet: project/after-hours-sms-lead-capture
- name: pest control industry
type: market segment
facet: domain/pest-control
relationships:
- source: MIT/InsideSales.com Lead Response Management Study
type: documents lead response time decay
target: After-Hours SMS Lead Capture
confidence: INFERRED
- source: Harvard Business Review
type: documents lead qualification speed
target: After-Hours SMS Lead Capture
confidence: INFERRED
- source: Velocify Ultimate Contact Strategy Study
type: documents conversion decay curve
target: After-Hours SMS Lead Capture
confidence: INFERRED
- source: HomeAdvisor
type: shows after-hours call volume in
target: pest control industry
- source: Angi
type: shows after-hours call volume in
target: pest control industry
- source: CallRail
type: documents call volume peaks in
target: pest control industry
- source: PATLive
type: measures caller abandonment in
target: pest control industry
- source: Driven Results
type: measured real-world conversion outcomes for
target: After-Hours SMS Lead Capture
confidence: INFERRED
- source: ServiceTitan
type: tracks adoption gap in
target: pest control industry
- source: NPMA
type: published gross margin data for
target: pest control industry
- source: Point Loma Electric & Plumbing
type: achieved booking rate boost with
target: After-Hours SMS Lead Capture
confidence: INFERRED
- source: Shafer Services
type: achieved lead booking increase with
target: After-Hours SMS Lead Capture
confidence: INFERRED
- source: Aruza Pest Control
type: achieved cost reduction with
target: After-Hours SMS Lead Capture
confidence: INFERRED

View File

@ -0,0 +1,134 @@
# --- Graphify extraction fragment ---
note_slug: pest-control-sms-market-research-stats
entities:
- name: After-Hours SMS Lead Capture & Qualification
type: product
facet: project/niche-automation-prospecting
- name: pest control companies
type: industry
facet: domain/pest-control
- name: niche-automation-prospecting
type: initiative
facet: project/niche-automation-prospecting
- name: pest-control-spring-2026
type: campaign
facet: project/pest-control-spring-2026
- name: MIT/InsideSales.com Lead Response Management Study (2007)
type: study
- name: Harvard Business Review (March 2011)
type: study
- name: Velocify "Ultimate Contact Strategy" Study (2012)
type: study
- name: InsideSales.com 2021
type: study
- name: HomeAdvisor
type: company
- name: Angi
type: company
- name: CallRail
type: company
- name: PATLive
type: company
- name: Forbes
type: publication
- name: SellCell 2024
type: study
- name: Driven Results
type: company
- name: HubSpot 2023
type: study
- name: Google Ads (PPC)
type: channel
facet: tool/google-ads
- name: Google Local Services Ads
type: channel
facet: tool/google-local-services-ads
- name: Thumbtack
type: company
- name: Coalmarch
type: company
- name: Valve+Meter
type: company
- name: Invoca
type: company
- name: NPMA & PCO Bookkeepers 2025 Cost Study
type: study
- name: PPMA survey
type: study
- name: Scorpion
type: company
- name: Gartner 2016
type: study
- name: D7 Networks
type: company
- name: CTIA
type: organization
- name: TransUnion
type: company
- name: Avochato
type: company
- name: Velocify/Leads360
type: study
- name: Briostack
type: company
- name: Cube Creative Design
type: company
- name: ServiceTitan 2025
type: study
- name: Point Loma Electric & Plumbing
type: company
- name: Hatch
type: company
- name: Shafer Services
type: company
- name: Aruza Pest Control
type: company
- name: SMS
type: channel
facet: domain/sms
- name: NPMA
type: organization
- name: ServiceTitan
type: company
relationships:
- source: After-Hours SMS Lead Capture & Qualification
type: is sold to
target: pest control companies
- source: After-Hours SMS Lead Capture & Qualification
type: uses channel
target: SMS
- source: MIT/InsideSales.com Lead Response Management Study (2007)
type: measured conversion decay for
target: pest control companies
confidence: INFERRED
- source: Velocify/Leads360
type: found texted prospects convert 40% higher for
target: SMS
confidence: INFERRED
- source: Aruza Pest Control
type: is a customer outcome case studied by
target: Cube Creative Design
- source: Point Loma Electric & Plumbing
type: is a case study by
target: Hatch
- source: Shafer Services
type: is a case study by
target: Hatch
- source: NPMA & PCO Bookkeepers 2025 Cost Study
type: reports gross margin for
target: pest control companies
- source: Coalmarch
type: provides cost-per-lead average for
target: pest control companies
- source: ServiceTitan 2025
type: surveyed
target: ServiceTitan
confidence: AMBIGUOUS
- source: Briostack
type: reports automation adoption gap for
target: pest control companies
- source: Gartner 2016
type: reports open and response rates for
target: SMS

View File

@ -0,0 +1,142 @@
# --- Graphify extraction fragment ---
note_slug: pest-control-sms-market-research-stats
entities:
- name: MIT/InsideSales.com Lead Response Management Study
type: research-study
facet: domain/lead-response
- name: Harvard Business Review Lead Response Study
type: research-study
facet: domain/lead-response
- name: Velocify Ultimate Contact Strategy Study
type: research-study
facet: domain/lead-response
- name: InsideSales.com 2021 Study
type: research-study
facet: domain/lead-response
- name: HomeAdvisor
type: organization
- name: Angi
type: organization
- name: CallRail
type: organization
- name: PATLive
type: organization
- name: Driven Results
type: organization
- name: HubSpot
type: organization
- name: Valve+Meter
type: organization
- name: Invoca
type: organization
- name: Coalmarch
type: organization
- name: NPMA
type: organization
- name: PCO Bookkeepers
type: organization
- name: PPMA
type: organization
- name: Scorpion
type: organization
- name: Briostack
type: organization
- name: Cube Creative Design
type: organization
- name: ServiceTitan
type: organization
- name: Gartner
type: organization
- name: TransUnion
type: organization
- name: Avochato
type: organization
- name: Hatch
type: organization
- name: Google Ads
type: tool
facet: tool/paid-search
- name: Google Local Services Ads
type: tool
facet: tool/paid-search
- name: Thumbtack
type: tool
facet: tool/lead-marketplace
- name: Point Loma Electric & Plumbing
type: organization
- name: Shafer Services
type: organization
- name: Aruza Pest Control
type: organization
- name: After-Hours SMS Lead Capture & Qualification
type: product-concept
facet: project/niche-automation-prospecting
- name: pest control
type: industry
facet: domain/pest-control
- name: home services
type: industry
facet: domain/field-service
- name: lead response time
type: concept
facet: domain/lead-response
- name: SMS marketing
type: concept
facet: domain/lead-capture
- name: lead conversion
type: concept
facet: domain/lead-response
- name: after-hours call handling
type: concept
facet: domain/field-service
- name: niche-automation-prospecting
type: project
facet: project/niche-automation-prospecting
relationships:
- source: MIT/InsideSales.com Lead Response Management Study
type: measures decay of
target: lead conversion
- source: Harvard Business Review Lead Response Study
type: measures decay of
target: lead conversion
- source: Velocify Ultimate Contact Strategy Study
type: measures decay of
target: lead conversion
- source: InsideSales.com 2021 Study
type: measures decay of
target: lead conversion
- source: After-Hours SMS Lead Capture & Qualification
type: targets industry
target: pest control
- source: After-Hours SMS Lead Capture & Qualification
type: addresses problem of
target: after-hours call handling
- source: After-Hours SMS Lead Capture & Qualification
type: leverages
target: SMS marketing
- source: Aruza Pest Control
type: is customer in case study by
target: Cube Creative Design
- source: Point Loma Electric & Plumbing
type: is customer in case study by
target: Hatch
- source: Shafer Services
type: is customer in case study by
target: Hatch
- source: Driven Results
type: sourced after-hours data for
target: home services
- source: Valve+Meter
type: conducted secret-shopper study of
target: home services
- source: NPMA
type: co-published cost study with
target: PCO Bookkeepers
- source: ServiceTitan
type: published industry survey on
target: home services
- source: niche-automation-prospecting
type: contains research for
target: After-Hours SMS Lead Capture & Qualification

View File

@ -0,0 +1,31 @@
# Benchmark Run Findings — 2026-06-04 (Step 2c reference set)
_Run: 2026-06-04 — reference set generated, verified, gate satisfied._
This document records the outcomes, surprises, and lessons from the Step 2c reference-extraction benchmark run. It is the authoritative findings record for this run date and serves as input to future benchmark comparisons when the Graphify/Ollama production pipeline is evaluated.
## What was run
- 6 cross-domain fixture notes hand-selected from ~/Documents/SecondBrain (4 pest-control / niche-automation-prospecting, 2 software-design), covering all five fixture categories (tool, client/project, convention, domain, relationship-dense). Fixtures are listed in dispatch-prompt.md.
- Each note extracted by 3 Claude tiers (claude-haiku-4-5, claude-sonnet-4-6, claude-opus-4-8) via subagent dispatch with model overrides = 18 reference fragments in reference-outputs/.
- Notes were run AS-IS (no vault frontmatter modification); the SecondBrain vault was backed up first to ~/Documents/SecondBrain-backup-20260604-114749.
- The .opus.md files are the gold-standard scoring rubric; haiku/sonnet are kept to characterize the tier quality gradient.
## Key finding: fairness-contract contamination (the gate working as designed)
- The first-pass haiku extraction of the `ai-coding-conventions-synthesis` note leaked three entities — `Graphify`, `Milvus Lite`, `SecondBrain vault` — that do NOT appear in the note text. They were injected from the cc-os CLAUDE.md / project context that Claude Code auto-loads into subagents. Sonnet and opus did not leak on the same note.
- Lesson 1: the soft "ignore any injected context" instruction in the fairness contract is NOT reliable on the haiku tier — re-running clean required an explicit forbidden-class hint (do not introduce knowledge-graph/vector-db/vault tooling unless it appears in the note text).
- Lesson 2 (important, prevents a wrong conclusion): this is largely a HARNESS ARTIFACT, not a property of the production pipeline. In the real Graphify→Ollama pipeline, the SLM receives only the prompt Graphify constructs — there is no CLAUDE.md injection path. So the benchmark's contamination exposure is HIGHER than production; the cleaned reference set is conservative. Do NOT conclude "haiku extracts poorly," and do NOT carry the anti-contamination hint into the production Graphify extraction prompt — it is unneeded there.
## Spec gaps found and fixed
- The original extraction-spec.md did not require a sub-value after a facet namespace, producing degenerate bare-slash facets (`domain/`, `tool/`) in several sonnet/haiku outputs. It also did not state strict referential integrity, producing relationships pointing at undeclared entities.
- Both rules were added to extraction-spec.md on 2026-06-04, and the affected files regenerated so the whole 18-file set conforms to the final spec.
## Tier richness gradient (observed)
- Opus is reliably the richest extractor (e.g. the enterprise-revenue note: opus ~59-61 entities vs haiku ~25). Sonnet is sometimes terser than haiku. On stat-dense notes all three tiers saturate to similar counts. Treat entity/relationship counts as indicative, not exact (extractor self-counts diverged from verified counts).
## Status
- All 18 files verified: valid YAML, conformant schema, referential integrity, no contamination. Step 2c gate satisfied 2026-06-04.

View File

@ -0,0 +1,246 @@
# Local-Model Extraction Scoring — Results (2026-06-04)
_Last updated: 2026-06-05_ | _Status: UNBLOCKED — qwen2.5-coder:7b selected; scoring (3.4), speed (3.3), and model selection (3.5/3.6) complete. Remaining: build vault graph (4), update docs (5)._
---
## Toolchain (Step 2a)
- graphify 0.8.31 (PyPI `graphifyy`) installed at `~/.local/bin/graphify`; `graphify --version``graphify 0.8.31`. [verified: local shell]
- ollama 0.30.3, systemd service `/usr/local/bin/ollama serve`, host 127.0.0.1:11434. [verified: local shell]
- GPU: RTX 3060, 12GB VRAM. All extraction ran 100% on GPU (no CPU spill) per `ollama ps`. [verified: local shell]
---
## Candidate models — true bases (resolves the e4b/e2b naming question)
Pinned via `ollama show <tag>` info cards. These are genuine next-gen families, NOT gemma3n/qwen2.5 renames: [verified: ollama show output]
| Local tag | Architecture | Params | Quant | Native ctx |
|---------------|--------------|---------------------------------|--------|------------|
| gemma4:e4b | gemma4 | 8.0B (MatFormer "effective 4B") | Q4_K_M | 131072 |
| gemma4:e2b | gemma4 | 5.1B (effective 2B) | Q4_K_M | 131072 |
| qwen3.5:2b | qwen35 | 2.3B | Q8_0 | 262144 |
---
## Critical config finding — num_ctx does not propagate through graphify (Step 2b)
- graphify posts to ollama's OpenAI-compatible `/v1/chat/completions` endpoint. That endpoint **silently ignores** the per-request `options.num_ctx` graphify sends. Proven by A/B: `POST /v1/chat/completions` with num_ctx=8192 → `ollama ps` CONTEXT=4096; `POST /api/chat` with num_ctx=8192 → CONTEXT=8192. [verified: local A/B test]
- Therefore `GRAPHIFY_OLLAMA_NUM_CTX` (env var) has **NO EFFECT** through graphify; context pins at ollama's /v1 default 4096.
- At 4096, graphify's extraction output JSON is truncated mid-response (finish_reason=length) and the chunk is **DISCARDED** → empty graph, regardless of (small) input note size.
- **WORKAROUND (validated):** bake context into a Modelfile variant — `FROM <model>` + `PARAMETER num_ctx 8192`, then `ollama create <name>-8k`. The /v1 endpoint **does** honor the model's baked default. Verified `ollama ps` CONTEXT=8192 (and 16384), 100% GPU. This is the same lever production (vault build) will need. Non-invasive: no sudo, no systemd restart, no shell-profile/dotfiles edit; reversible via `ollama rm`. [verified: local shell]
- Two other operational notes: graphify's `OLLAMA_BASE_URL` must **end in `/v1`** (e.g. `http://127.0.0.1:11434/v1`) or every call 404s; and the CLI requires `OLLAMA_API_KEY` set to any non-empty value unless the host is loopback (127.0.0.1/localhost/::1).
---
## Benchmark methodology
- Each of the 6 Step-2c fixtures was isolated in its own temp directory (graphify `extract` does NOT accept a single file path — it walks a directory; isolation also avoids conflating notes in one token-budget chunk and excludes `.obsidian/` config noise). [verified: local testing]
- Command shape: `OLLAMA_BASE_URL=http://127.0.0.1:11434/v1 graphify extract <fixture-dir> --backend ollama --model <8k-variant> --max-concurrency 1 --out <dir>`. Models run sequentially (single GPU), one resident at a time.
- Scoring was scoped to dimensions graphify's output and the `.opus` gold references share: entity correctness, relationship plausibility, and edge-level INFERRED/AMBIGUOUS confidence. **Excluded as non-comparable:** the six-facet taxonomy and per-entity type/confidence (graphify's built-in extraction prompt emits neither — per ADR-011 facets are note frontmatter metadata, never Graphify's extraction job), and free-text relationship TYPE strings (graphify uses a fixed relation vocabulary: `references`/`conceptually_related_to`/`semantically_similar_to`/etc., vs the references' free-text verbs).
---
## Results through graphify's real extraction path
All three tested as 8192-context Modelfile variants. **Scoring blocked** because outputs were degenerate:
| Model | Result | Failure mode |
|--------------------|-------------------------------------------|--------------|
| gemma4:e4b (8k) | EMPTY graphs (multiple fixtures, deterministic 5/5) | Hollow, invalid JSON (~5039 output tokens of unparseable content); graphify discards chunk. graphify's own warning: "model too small for JSON instruction following — try a larger model." |
| qwen3.5:2b (8k) | EMPTY graph | Output truncated at max_completion_tokens; qwen3.5 "thinking" mode appears to consume the output budget before emitting the JSON graph. |
| gemma4:e2b (8k) | 15 nodes, 0 edges (valid JSON) on the oo-principles fixture | Extracted plausible entities (OO Principles, Single Responsibility Principle, Law of Demeter, Dependency Injection, the four design approaches, etc.) but emitted ZERO relationships → unusable as a knowledge graph. |
**Caveat on variance:** model Modelfile defaults are temperature 1 (poor for structured extraction); one PRE-fix run of gemma4:e4b on oo-principles yielded 4 nodes/3 edges, indicating high run-to-run variance.
---
## Conclusion
No specified candidate (gemma4:e4b, gemma4:e2b, qwen3.5:2b) produces a usable entity+relationship graph through Graphify's **default extraction prompt** on this hardware. Per the change proposal's risk clause ("no candidate scores acceptably → surface as finding, don't force selection"), model selection (task 3.6) and the initial vault graph build (task 4) are **DEFERRED** pending a decision.
**NOTE:** The earlier gut-check (`local-llm-findings-2026-06-04.md`) reported gemma4:e4b doing "6/6 clean parse, 1334 entities" — but that used a **direct ollama call** with a simpler custom prompt, **NOT** graphify's actual extraction prompt. Through graphify's real path the result is far worse. The graphify-path numbers are the authoritative ones for production.
---
## Untested levers (candidate next steps, not yet attempted)
1. **Disable model "thinking" mode** — gemma4 and qwen3.5 are thinking-capable; thinking tokens likely burn the output budget, causing the hollow/truncated JSON. Most promising cheap lever, but exposing a think-toggle through graphify's /v1 path is unverified.
2. **`graphify extract --mode deep`** to elicit INFERRED edges (addresses e2b's 0-edge result).
3. **Override sampling temperature to 0** for structured extraction (Modelfile defaults are temp 1).
4. **Pull a stronger model that fits 12GB VRAM with 8k context** — e.g. a 78B at Q4 (~5GB, leaves headroom for context); graphify itself suggests a ~14B (~9GB, tighter). This expands the candidate set beyond the three the change specified (a download + scope decision).
---
## Raw artifacts (ephemeral, /tmp — will not survive reboot)
- e4b first benchmark (pre-fix, 4096): `/tmp/graphify-bench/out/e4b/`
- 8k-variant runs: `/tmp/graphify-bench/out2/`, `/tmp/graphify-bench/out3/`
- Created reversible ollama variants: `gemma4-e4b-8k`, `gemma4-e4b-16k`, `gemma4-e2b-8k`, `qwen35-2b-8k`
---
## Round 2 (2026-06-05): thinking-disable patch + edge gate
### The thinking-mode investigation (user-directed)
- GitHub issue safishamsi/graphify#792 turned out to be about CPU scaling, local API timeouts, and the OLLAMA_API_KEY auth gate — NOT thinking/reasoning. (Documents the same auth friction we hit.)
- Ollama's `/v1` (OpenAI-compat) endpoint, which Graphify uses, IGNORES `think:false` and `chat_template_kwargs:{enable_thinking:false}`, but DOES honor top-level `reasoning_effort:"none"`. The native `/api/chat` honors `think:false`. (Same /v1-drops-nonstandard-options pattern as num_ctx.)
- Graphify sends NO thinking control for the ollama backend by default; there is no env var / CLI flag / config file to inject it (confirmed by source trace of llm.py). `PARAMETER think false` in a Modelfile is rejected by `ollama create`; a Modelfile `SYSTEM /no_think` does not survive because Graphify sends its own system message and ollama's /v1 REPLACES (not merges) the system message.
- WORKING FIX (applied to the installed package): one-line patch adding `"reasoning_effort": "none"` to the ollama backend config dict in `~/.local/lib/python3.14/site-packages/graphify/llm.py` (~line 71). Graphify already applies `reasoning_effort` as a top-level kwarg if present, and it survives the extra_body overwrite. The author already uses the equivalent pattern (`extra_body={"thinking":{"type":"disabled"}}`) for the Kimi/moonshot backend — just not for ollama. PRODUCTION IMPLICATION: this is a local patch lost on `pip install --upgrade`; production needs an upstream PR (likely welcome) or a maintained patch/wrapper.
- Verified effect: with the patch, every model's raw output begins directly with `{"nodes":[...` — no reasoning preamble. Thinking is genuinely suppressed.
### The edge gate — the criterion that actually matters
A knowledge graph needs RELATIONSHIPS (edges), not just entities (nodes). Across ~10+ runs this whole investigation, LLM-extracted relationships appeared essentially once (e4b pre-patch: 3 edges, since non-reproducible). All other non-empty results were nodes-only. So the gate for selection is: does a config emit a plausible EDGE-bearing graph on one fixture?
Bounded test on the `oo-principles-plugin-concept-design-recommendations` fixture (a note that clearly contains relationships), thinking-disable confound checked by running both ON and OFF:
| Config | Exit | Nodes | Edges | Outcome |
|---|---|---|---|---|
| qwen3.5:2b @16k, thinking-OFF | 1 | — | empty | output truncated mid-JSON → invalid → discarded |
| qwen3.5:2b @16k, thinking-ON | 1 | — | empty | same truncation; 16k did not rescue it |
| gemma4:e2b @16k, thinking-ON | 0 | 17 | 0 | clean valid JSON, coherent on-topic nodes, ZERO edges |
| gemma4:e2b @16k, thinking-ON, --mode deep | 0 | 10 | 0 | deep mode (explicitly requests INFERRED edges) made it WORSE: fewer nodes, still 0 edges |
`hyperedges` was also empty `[]` in the clean runs — no relationships hid in the alternate key.
### Confound resolved: thinking-off is NOT strictly better
gemma4:e2b @8k: thinking-ON gave 15 nodes / 3307 output tokens (valid); thinking-OFF gave 1 node / 395 tokens. For e2b the thinking tokens were doing extraction work; the binding constraint is output room, not thinking. So the user's "thinking wastes the budget" premise holds for qwen (truncation) but inverts for e2b. Thinking-disable is a useful lever, not a universal win.
### Two distinct failure modes
- qwen3.5:2b (2.3B, Q8_0): capacity/truncation — runs out of output room emitting node JSON before reaching edges. 16k context insufficient; thinking on/off irrelevant.
- gemma4:e2b (5.1B, Q4_K_M): clean completion, extracts plausible NODES but emits zero RELATIONSHIPS even when deep mode explicitly asks for them. The model understands the doc (node labels: OO Principles, Single Responsibility Principle, Law of Demeter, Dependency Injection, Replace Conditional with Polymorphism, the 4 approaches, the 3 layers) — it just won't populate `links`.
- gemma4:e4b (8.0B, Q4_K_M): hollow/malformed JSON (dropped — not a context problem).
### Round-2 conclusion
The cheap-lever space is exhausted (thinking ON/OFF, 8k/16k context, --mode deep). No local 25B candidate produces a relationship-bearing graph through Graphify's extraction prompt on a 12GB GPU. Relationship extraction (not entity extraction) is the wall. Recommended path: a stronger model that fits 12GB with ~16k context — e.g. qwen2.5-coder:7b (~5GB Q4, non-thinking, strong at structured JSON+edges). This expands the candidate set beyond the three the change specified (a multi-GB download + scope decision) — pending user go-ahead.
### Round-2 ollama variants created (reversible via `ollama rm`)
gemma4-e4b-8k, gemma4-e4b-16k, gemma4-e2b-8k, gemma4-e2b-16k, qwen35-2b-8k, qwen35-2b-16k. Installed graphify llm.py currently carries the thinking-OFF patch (backup of original at /tmp/graphify-bench/llm.py.orig).
---
## Round 3 (2026-06-05): qwen2.5-coder:7b clears the edge gate — full benchmark
### Config that works
- Model: `qwen2.5-coder:7b` (graphify's hardcoded ollama default, llm.py:67), run as a Modelfile variant `qwen25-coder-7b-16k` (FROM qwen2.5-coder:7b + PARAMETER num_ctx 16384).
- Command: `OLLAMA_BASE_URL=http://127.0.0.1:11434/v1 graphify extract <fixture-dir> --backend ollama --model qwen25-coder-7b-16k --max-concurrency 1 --out <dir>`. The thinking-OFF llm.py patch is in place (no-op for qwen2.5-coder, which has no thinking mode).
- VRAM: 5.6GB resident, 100% GPU, 16384 context, no CPU spill on the 12GB RTX 3060.
- Speed (task 3.3): aggregate throughput ~59.4 tok/s (substantive runs cluster 5761 tok/s). Total wall-clock all 6 = 253.7s (4.2 min); avg 42.3s/fixture (49.9s excluding the degenerate 10dlc run). Use tok/s not wall-clock (wall-clock tracks output length).
### Per-fixture results (edge gate PASSED: 5/6 produce relationships)
| Fixture | Wall | Nodes | Edges | Out tok | Notes |
|---|---|---|---|---|---|
| oo-principles-plugin-concept | 44.9s | 11 | 10 | 2553 | gate fixture; cold-load incl. |
| 10dlc-isv-setup-guide-oncadence | 4.4s | 1 | 0 | 159 | under-extraction on smallest note (502 words), exit 0 valid JSON — not a crash |
| pest-control-enterprise-revenue | 64.2s | 23 | 17 | 3932 | only fixture with INFERRED edges (7) |
| ai-coding-conventions-synthesis | 37.9s | 10 | 9 | 2284 | |
| pest-control-after-hours-sms | 76.8s | 21 | 20 | 4641 | |
| pest-control-email-a-b-c-hub | 25.5s | 7 | 6 | 1489 | |
All exit 0, no truncation/hollow/invalid-JSON warnings. Confidence tags are provenance method tags: EXTRACTED (from text) vs INFERRED (model-inferred); INFERRED appeared only in enterprise-revenue.
### Full edge lists (for scoring vs .opus references next session)
**oo-principles (10, all EXTRACTED):** Process Layer→shares_data_with→Mechanic Layer; Process Layer→shares_data_with→Theory Layer; Mechanic Layer→shares_data_with→Cards Layer; Annotated Process Graph Layer→shares_data_with→Cards Layer; Bundles Layer→shares_data_with→Cards Layer; CLAUDE.md→shares_data_with→Cards Layer; NotebookLM Notebook→shares_data_with→Cards Layer; Phase Bundled Context Packs Layer→shares_data_with→Cards Layer; Refactoring Layer→shares_data_with→Cards Layer; Situational Trigger Files Layer→shares_data_with→Cards Layer.
**10dlc:** none.
**enterprise-revenue (17):** Marketing Allocation Model→references→Unit Economics; ACV Tiers→references→Unit Economics; M&A Context→references→ACV Tiers; Communication Failure Patterns→references→Unit Economics; Dropped calls and ghosting→conceptually_related_to→Communication Failure Patterns [INFERRED]; No one answered→conceptually_related_to→Communication Failure Patterns [INFERRED]; Offshore agent disconnect→conceptually_related_to→Communication Failure Patterns [INFERRED]; Voicemail full→conceptually_related_to→Communication Failure Patterns [INFERRED]; M&A Context→references→Communication Failure Patterns; Climatologically-Driven Lead Seasonality→references→Unit Economics; Fall First Freeze Rodent Push→conceptually_related_to→Climatologically-Driven Lead Seasonality [INFERRED]; Spring Termite Swarming→conceptually_related_to→Climatologically-Driven Lead Seasonality [INFERRED]; Summer Heatwave-Driven Migrations→conceptually_related_to→Climatologically-Driven Lead Seasonality [INFERRED]; M&A Context→references→Climatologically-Driven Lead Seasonality; M&A Context→references→Operational Unit Economics; Operational Unit Economics→references→COGS Structure; M&A Context→references→Unit Economics.
**ai-coding-conventions (9, all EXTRACTED):** Core Finding→references→{Context Linking Patterns, Enforcement Mechanisms, Key Tradeoffs, Notable Public References, Organizational Patterns, Process vs. Reference Design, See Also, The Three-Tier Architecture (arXiv 2602.20478), Token Efficiency Strategies}.
**after-hours-sms (20, all EXTRACTED):** document node→cites→ each of 20 study/source nodes (MIT/InsideSales 2007, HBR 2011, Velocify 2012, InsideSales 2021, HomeAdvisor/Angi 2024, Thumbtack 2024, Coalmarch 2025, Invoca 2024, Driven Results 2025, HubSpot 2023, Gartner 2016, D7 Networks/CTIA 2024, TransUnion 2024, Avochato 2019, Briostack 2024, ServiceTitan 2025, Point Loma Hatch, Shafer HVAC Hatch, Aruza Cube, ServiceTitan Data 2025).
**email-a-b-c-hub (6, all EXTRACTED):** Experiment Hub→references→{Blind A/B/C Methodology Guide, Per Group Copy and Scores, What Owners Respond to/Reject, V4 Copywriter Draft, V4 Golden Framework Draft, V4 Golden Framework Revised}.
### Round-3 takeaway
qwen2.5-coder:7b is the de-facto selected model pending the formal scoring pass (3.4). It's graphify's own default, fits 12GB with 16k context at ~59 tok/s, and produces typed, mostly-EXTRACTED edges. The earlier gut-check's gemma4:e4b pick was an artifact of using a simpler direct prompt, not graphify's real path. Source graph.json (ephemeral): /tmp/graphify-bench/out6/coder7b/<slug>/graphify-out/graph.json.
---
## 3.4 Scoring — qwen2.5-coder:7b vs Opus Gold Standard
**Sources used:**
- Candidate output: `scoring-results-2026-06-04.md`, Round 3 section (per-fixture node/edge lists)
- Gold-standard references: `reference-outputs/*.opus.md` (all 6 `.opus.md` files)
- Fixture mapping (from Round 3 table): `oo-principles-plugin-concept`, `10dlc-isv-setup-guide-oncadence`, `pest-control-enterprise-revenue`, `ai-coding-conventions-synthesis`, `pest-control-after-hours-sms`, `pest-control-email-a-b-c-hub`
### Dimensions Excluded (N/A)
**Facets:** The `.opus.md` references include facet annotations (e.g., `facet: tool/twilio`, `facet: convention/oo-principles`). These are note-level metadata per ADR-011 — they are not part of graphify's extraction output at all, so there is nothing to compare. N/A.
**Entity-type labels:** The opus references annotate entities with types (`type: layer`, `type: principle`, `type: Tool`, etc.). Graphify's extraction output produces a node list with names only — no type labels are emitted. N/A.
**Free-text relationship typing:** The opus references use descriptive, domain-specific verbs (`comprises_share_of`, `is_event_of`, `measured conversion decay for`, `tested for campaign`, etc.). Graphify uses a fixed relation vocabulary (`references`, `conceptually_related_to`, `semantically_similar_to`, `shares_data_with`, `cites`). These are structurally incommensurable — comparing relation-type strings would score graphify's vocabulary design, not extraction quality. N/A.
### Rubric
**Entity Correctness (a):** 15. How many of the entities graphify extracted correspond to real entities in the reference? Penalizes under-extraction (missing key nodes) and over-compression (collapsing distinct entities). Does not penalize naming style differences.
**Relationship Plausibility (b):** 15. Are the edges graphify drew present or derivable from the reference's relationship set? Penalizes phantom edges (no basis in reference), missing major structural relationships, and systematically wrong pairing patterns. Does not penalize relation-verb style.
**Confidence Accuracy (c):** 15. Where graphify tagged EXTRACTED vs INFERRED vs AMBIGUOUS, does that align with what the reference treated as explicit (no `confidence:` tag) vs inferred (`confidence: INFERRED`) vs ambiguous (`confidence: AMBIGUOUS`)? N/A on fixtures where graphify emitted no edges at all, or no INFERRED/AMBIGUOUS tags appeared in either output.
### Per-Fixture Scoring Table
| Fixture | (a) Entity Correctness | (b) Relationship Plausibility | (c) Confidence Accuracy | Notes |
|---|---|---|---|---|
| oo-principles-plugin-concept | 2 | 1 | N/A† | |
| 10dlc-isv-setup-guide | 1 | N/A (0 edges) | N/A | |
| pest-control-enterprise-revenue | 3 | 3 | 3 | |
| ai-coding-conventions-synthesis | 2 | 2 | N/A† | |
| pest-control-after-hours-sms | 3 | 2 | N/A† | |
| pest-control-email-a-b-c-hub | 2 | 2 | N/A† | |
†Graphify emitted no INFERRED/AMBIGUOUS tags on this fixture (all EXTRACTED), so there is no confidence signal to compare against the reference's INFERRED annotations. N/A does not mean graphify was wrong; it means the dimension is unobservable.
### Per-Fixture Justifications
**oo-principles-plugin-concept**
- (a) **2/5.** Captured the layer decomposition (Process, Mechanic, Theory + 4 named layers, CLAUDE.md, NotebookLM Notebook = 11 nodes) but missed the plugin as a named entity, all OO principle entities (TDD, SRP, LoD, DI, Shameless Green, Flocking Rules), the book/paper sources, and the 4-phase development lifecycle. ~30% of the ~30-entity reference set.
- (b) **1/5.** All 10 edges are `shares_data_with` → "Cards Layer". The reference relationships are structural/directional (includes, encodes, references, retains, alternative to, evolves from, modeled on). Homogenization collapses the architecture into a flat data-sharing star with no analog in the reference.
- (c) **N/A.** All graphify edges EXTRACTED; reference has 3 INFERRED. No INFERRED signal to compare.
**10dlc-isv-setup-guide**
- (a) **1/5.** Extracted 1 node (whole document as a single entity) vs 20 distinct reference entities (OnCadence, 10DLC, Twilio, TrustHub API, ISV, CSP, EIN, IRS, T-Mobile, subaccount-per-client architecture, etc.). Near-total extraction failure on a 502-word note.
- (b) **N/A.** 0 edges produced vs 17 reference relationships.
- (c) **N/A.** No edges.
**pest-control-enterprise-revenue**
- (a) **3/5.** 23 nodes capturing the major conceptual clusters (Marketing Allocation Model, ACV Tiers, M&A Context, Unit Economics, Communication Failure Patterns + 4 sub-patterns, Seasonality + 3 seasonal events, COGS Structure) but at shallower granularity than the ~55-entity reference (misses CAC/LTV/CPL/Churn, COGS components, regions, Big Four integrators, GDD, multiples).
- (b) **3/5.** 17 edges incl. 7 INFERRED. The INFERRED membership cluster (comm sub-patterns → Communication Failure Patterns; seasonal events → Seasonality) maps closely onto the reference's `is_event_of`/`component_of` structure. ~50% of reference edges missed (COGS breakdown, Big Four, causal chains).
- (c) **3/5.** The 7 INFERRED tags are defensible group-membership edges; not systematically miscalibrated, though it under-fires on EXTRACTED edges the reference treats as explicit, and never modeled the reference's `EIN → issued_by → IRS` background-knowledge INFERRED edge.
**ai-coding-conventions-synthesis**
- (a) **2/5.** Extracted the document's section-heading outline (10 nodes: Context Linking Patterns, Enforcement Mechanisms, Three-Tier Architecture, etc.) rather than the ~40 domain entities (Claude, Cursor, Copilot, Constitution, Domain Specialist Agents, .cursorrules, AGENTS.md, MCP, named projects). Only Three-Tier Architecture overlaps in substance.
- (b) **2/5.** All 9 edges are `Core Finding → references → {section heading}` — a hub-and-spoke from a synthetic aggregate node. Zero structural resolution vs the reference's compositional/directional relationships. 2 not 1 because no phantom entities introduced.
- (c) **N/A.** All EXTRACTED; reference's 2 INFERRED edges not modeled.
**pest-control-after-hours-sms**
- (a) **3/5.** 21 nodes (document + 20 study/source nodes) covering ~half the reference's company/study entities, but missing the product/project/campaign entities (After-Hours SMS product, niche-automation-prospecting initiative, pest-control-spring-2026 campaign) and several companies (Google Ads/LSA, NPMA, CallRail, PATLive, etc.).
- (b) **2/5.** All 20 edges are `document → cites → {study}` — a citation list. Defensible but reductive; the reference's productcustomer, productchannel, and studyfinding semantic edges are entirely missing.
- (c) **N/A.** All EXTRACTED; reference has 2 INFERRED + 1 AMBIGUOUS, none modeled.
**pest-control-email-a-b-c-hub**
- (a) **2/5.** Extracted the hub's table-of-contents (7 nodes: Experiment Hub + 6 linked sub-documents) but missed the experiment's analytic entities (3 personas, 9 email groups, 5 hooks, structural/retired patterns, 3 versions) — the primary conceptual content. ~45-entity reference.
- (b) **2/5.** All 6 edges are `Experiment Hub → references → {sub-document}` — the index, not the findings. Version outcomes, recommended groups, retired patterns all absent.
- (c) **N/A.** All EXTRACTED; reference's 2 INFERRED edges not modeled.
### Overall Summary
**Entity extraction: weak to moderate (avg ~2.2/5).** Consistently fewer/coarser entities than the opus reference. On hub-structured notes it extracts the structural outline (section headings, linked sub-docs) rather than conceptual entities; on dense domain notes it captures ~4055% of the reference entity set; on the smallest note (10dlc) it nearly fails (1 node vs 20).
**Relationship extraction: poor to moderate (avg ~1.9/5, excluding the N/A fixture).** Dominant failure mode is **homogenization**: nearly all relationship semantics collapse into one or two relation types (`references`, `cites`, `shares_data_with`) in a hub-and-spoke topology. enterprise-revenue is the only fixture with meaningfully differentiated edges — and the only one where INFERRED edges appeared — suggesting extraction quality correlates with note density.
**Confidence calibration: partially defensible where observable (3/5 on the one applicable fixture).** INFERRED tags only appeared on enterprise-revenue (7/17 edges) and were defensible group-membership edges. On the other five fixtures all edges were EXTRACTED, precluding comparison.
**Net assessment:** qwen2.5-coder:7b passes the edge gate (5/6 fixtures produce relationships) and extracts credible entities on content-dense notes, confirming it as a viable baseline over the tested alternatives. Its relationship extraction is systemically shallow: the relation flattening means the graph encodes "these things are related" but not "how," limiting downstream query quality. For a production vault graph the primary gap is relationship semantic resolution, not entity recall.
---
## 3.6 Model Selection
**Selected model: qwen2.5-coder:7b** (run as the Modelfile variant `qwen25-coder-7b-16k`, num_ctx 16384).
The selection is not a choice among viable candidates — it is the only candidate that cleared the relationship/edge gate. Every smaller candidate tested (gemma4:e4b 8.0B, gemma4:e2b 5.1B, qwen3.5:2b 2.3B) failed to produce relationship-bearing graphs through graphify's prompt on this hardware. The scoring in section 3.4 therefore characterizes qwen2.5-coder:7b's extraction quality against the opus gold standard rather than differentiating among alternatives; there was no viable alternative to differentiate.
This model is also graphify's own hardcoded ollama default (llm.py:67), which is a strong signal that it is the intended extraction model for this backend. It fits the 12GB RTX 3060 comfortably at 5.6GB VRAM resident with 16384 context and runs at ~59 tok/s, completing a full 6-fixture pass in ~4 minutes.
**Known limitations (carried forward honestly):** shallow relationship semantics — graphify's fixed relation vocabulary combined with qwen2.5-coder:7b's extraction behavior produces hub-and-spoke topologies dominated by `references`, `cites`, and `shares_data_with`, collapsing domain-specific structural relationships. Entity recall is weak on hub-structured notes (extracts the structural outline instead of conceptual entities) and near-failing on very short notes (10dlc: 1 node vs 20 in reference). These limitations are accepted for the initial build. Relationship semantic resolution is the known primary gap for future improvement.
**Speed:** ~59 tok/s on the 12GB GPU at max-concurrency 1.

View File

@ -0,0 +1,2 @@
schema: spec-driven
created: 2026-06-04

View File

@ -0,0 +1,42 @@
## Context
`cc-os` is a design-only repository; the build runbook is `docs/memory-system/05-implementation-process.md` and Step 2c is its documented gate. The hard constraint driving this change: only the Claude tiers are reachable here (via Claude Code subagents) — local Ollama models cannot be run in this benchmarking environment. The vault is `~/Documents/SecondBrain` (ADR-012) under the six-facet taxonomy (ADR-011). Graphify extracts entities plus typed edges plus confidence tags (`INFERRED`/`AMBIGUOUS`) from documents via a local SLM, and extracts code via tree-sitter AST (free, no model). The benchmark exists to choose the local doc-extraction model; this change does not make that choice, it produces the reference set that choice will be measured against.
## Goals / Non-Goals
**Goals:**
- A runnable benchmark that produces a reusable, diffable reference set across Claude tiers.
- A fairness contract that keeps the comparison as close to apples-to-apples as the environment allows.
- An incremental, observable build-and-migration path that validates the system before committing the whole vault to it.
**Non-Goals:**
- Choosing the final extraction model now (the reference set feeds that later decision).
- Pivoting away from local Ollama doc extraction (architecture stays intact per the existing ADRs).
- Running Ollama models within this change.
- Bulk-migrating the vault now.
## Decisions
- **Reference-set, not model selection.** Claude subagents produce gold-standard outputs, not a final pick. Rejected alternative: treating this as a "Claude-as-extraction-backend" pivot — that is an ADR-level architectural shift (cost, privacy) and is out of scope.
- **Mimic-extraction task with an explicit embedded output schema.** Subagents reason directly to the Graphify-shaped fragment rather than invoking `graphify extract`, because Ollama backends cannot run here and the goal is tier-vs-tier signal. The output schema (entities, typed relationships, `INFERRED`/`AMBIGUOUS` confidence) is embedded verbatim in the prompt so per-model files are diffable against each other now and against Ollama output later. Each subagent writes to its own per-model file.
- **Fairness contract: minimal context only.** Each subagent receives only the raw note text and the shared extraction spec. It is explicitly instructed not to read repository files (`CLAUDE.md`, design docs) or pull project context. This is the core property that makes the comparison meaningful.
- **Speed dropped for the Claude run.** Wall-clock per note is untrackable across dispatched subagents here, so quality is the only metric for the reference run: entity correctness, relationship plausibility/typing, and confidence-tag accuracy. Speed re-enters when local Ollama models are timed against the references.
- **Build-first / migrate-incrementally.** A 510 note fixture set (the variety already called for in Step 1c) feeds the gate immediately; bulk vault migration is deferred to last; first end-to-end validation runs against one small project containing both code and documents. The build-order inversion is recorded in ADR-013, and `CLAUDE.md`'s "Decisions locked" pointer is updated.
- **Migration-unit granularity, surfaced not hidden.** The first migration unit is named "one small project with both code and documents." Vault notes (local-SLM extraction path) and project code (tree-sitter path) are different extraction paths; design.md surfaces this so the user can react at proposal review rather than discovering it mid-migration.
## Risks / Trade-offs
- Fixtures unrepresentative of the real vault → choose deliberate variety per Step 1c (tool note, client/project note, convention note, domain note, one relationship-dense note) and keep them as living fixtures.
- Reference set encodes Claude-tier idiosyncrasies and biases later Ollama scoring → treat the references as a quality ceiling / scoring rubric, not literal ground truth; keep a human in the loop reviewing god-nodes.
- Deferring bulk migration delays real-world validation of the schema at scale → mitigated by the early end-to-end test on one small mixed project before broad rollout.
- Prompt context leakage breaks the fairness contract → the prompt enumerates the only allowed inputs and explicitly forbids reading repo/project files.
## Migration Plan
No code migration. The change edits docs and adds ADR-013; the only new artifact is the benchmark prompt file plus a directory for reference outputs (proposed: `docs/memory-system/benchmark/`). Rollback is a straight revert of the doc edits.
## Open Questions
- Migration-unit granularity (whole project repo vs. a vault-note cluster) — first unit is named now; refine after observing the first real migration.
- Where the prompt file and per-model reference outputs live (proposed default: `docs/memory-system/benchmark/`).
- Which specific small project is the initial mixed code+docs validation target — the user picks this at apply time (candidates under `~/projects/` and `~/dev/`).

View File

@ -0,0 +1,29 @@
## Why
Two gaps block executing the build runbook (`docs/memory-system/05-implementation-process.md`). First, Step 2c's model benchmark assumes head-to-head runs across local Ollama models and Claude API models, but the only models reachable from this environment are the Claude tiers via Claude Code subagents — so the benchmark cannot be run as written, and nothing captures a gold-standard to score local models against later. Second, the runbook front-loads bulk vault migration (all ~20 notes and all projects) as Step 1, committing to a schema and workflow before the system exists to validate them.
## What Changes
- Redesign Step 2c as a **reference-set benchmark**: a copy/paste-able Claude Code prompt — written as a committed file — that dispatches one subagent per Claude tier (`claude-haiku-4-5`, `claude-sonnet-4-6`, `claude-opus-4-8`) to mimic Graphify doc extraction on the fixture notes, each emitting the same Graphify-shaped structured fragment (entities, typed relationships, `INFERRED`/`AMBIGUOUS` confidence tags) to a per-model file. These per-model files become the gold-standard reference set that local Ollama models are scored against in a later step.
- Establish a **fairness contract** for the prompt: each subagent receives only the note text plus a shared minimal extraction spec — no `CLAUDE.md`, no design docs, no project context.
- Drop wall-clock speed as a metric for the Claude reference run (untrackable in this environment; quality only). Speed re-enters later when Ollama models are benchmarked against the reference set.
- **Resequence the build order**: build the full system against a small fixture set first; defer bulk vault migration and multi-project onboarding until the system is built and validated end-to-end on one small project that contains both code and documents; then onboard remaining projects one at a time, observing and adjusting per project.
- Keep four things explicitly distinct so the runbook does not blur them: benchmark fixtures (needed now, for the gate), bulk vault migration (deferred), the initial mixed code+docs validation project (post-build), and project-by-project rollout.
- Add **ADR-013** recording the build-order inversion (vault-migration-first → build-first / migrate-incrementally) and amend the "Decisions locked" pointer in `CLAUDE.md`.
## Capabilities
### New Capabilities
- `reference-extraction-benchmark`: the procedure, fairness contract, output schema, and deliverable prompt file for producing a reusable gold-standard Claude reference extraction set that local models are later scored against.
- `incremental-migration`: the resequenced build order — build and validate on fixtures plus one small mixed code+docs project before any bulk migration, then onboard projects one at a time.
### Modified Capabilities
<!-- None. No prior specs exist in openspec/specs/. -->
## Impact
- Docs: `docs/memory-system/05-implementation-process.md` (Step 1 sequencing + Step 2c rewrites), `docs/memory-system/03-architecture-decisions.md` (new ADR-013), `docs/memory-system/04-build-plan.md` (reconcile if it states the old order), `CLAUDE.md` ("Decisions locked" line).
- New deliverable: a committed, copy/paste-able benchmark dispatch prompt file, plus a home for the per-model reference outputs.
- No application code. This change touches design, runbook, and process only.

View File

@ -0,0 +1,47 @@
## ADDED Requirements
### Requirement: System is validated before bulk migration
The build SHALL be completed and validated end-to-end against a small fixture set and one small project containing both code and documents before any bulk vault migration is performed.
#### Scenario: Fixtures feed the gate without bulk migration
- **WHEN** the Step 2c benchmark gate is run
- **THEN** it uses a 510 note fixture set
- **AND** the remaining vault notes are not migrated at that point
#### Scenario: First end-to-end validation is a mixed project
- **WHEN** the system is first validated end-to-end
- **THEN** the target is one small project containing both code and documents
- **AND** validation covers both the document extraction path and the code (tree-sitter) path
### Requirement: Projects are onboarded one at a time
After the system is validated, projects SHALL be onboarded individually, with observation and adjustment between each, rather than migrated in bulk.
#### Scenario: One project at a time
- **WHEN** a project is onboarded after initial validation
- **THEN** it is migrated on its own
- **AND** its migration is observed and the process adjusted before the next project is onboarded
### Requirement: Bulk vault migration is deferred to last
Bulk migration of the full vault SHALL be deferred until after the system is built and validated, rather than performed as the first build step.
#### Scenario: Bulk migration ordering
- **WHEN** the build order is followed
- **THEN** bulk vault migration occurs after system validation
- **AND** only the fixture notes are migrated beforehand
### Requirement: Build-order inversion is recorded in an ADR
The inversion of the documented build order (from vault-migration-first to build-first / migrate-incrementally) SHALL be recorded in a new ADR and reflected in the project's locked-decisions pointer.
#### Scenario: ADR captures the inversion
- **WHEN** the build order is resequenced
- **THEN** a new ADR (ADR-013) records the inversion and its rationale
- **AND** the "Decisions locked" pointer in `CLAUDE.md` is updated to reference it

View File

@ -0,0 +1,61 @@
## ADDED Requirements
### Requirement: Reference set is produced from Claude tiers
The benchmark SHALL produce a reusable reference set by dispatching one Claude Code subagent per Claude tier (`claude-haiku-4-5`, `claude-sonnet-4-6`, `claude-opus-4-8`) over the fixture notes, with each subagent's output written to a distinct per-model file.
#### Scenario: One reference file per model
- **WHEN** the benchmark prompt is run on the fixture set
- **THEN** a separate output file is produced for each of the three Claude tiers
- **AND** each file contains that tier's extraction result for every fixture note
#### Scenario: Opus output is the scoring rubric
- **WHEN** the reference set is reviewed
- **THEN** the `claude-opus-4-8` output is treated as the gold-standard rubric against which the other tiers, and later the local Ollama models, are scored
### Requirement: Dispatch prompt enforces a fairness contract
The benchmark prompt SHALL give each subagent only the raw note text and a shared minimal extraction spec, and SHALL forbid reading repository files or pulling project context.
#### Scenario: Minimal context per subagent
- **WHEN** a subagent is dispatched for a fixture note
- **THEN** its input is limited to the note text and the shared extraction spec
- **AND** it is explicitly instructed not to read `CLAUDE.md`, design docs, or other project context
### Requirement: Output conforms to the Graphify-shaped schema
Each subagent SHALL emit a structured fragment matching the shape Graphify produces — entities, typed relationships, and `INFERRED`/`AMBIGUOUS` confidence tags — so per-model outputs are diffable against each other and against later Ollama output.
#### Scenario: Schema embedded in the prompt
- **WHEN** the benchmark prompt is authored
- **THEN** the required output schema is embedded verbatim in the prompt
- **AND** every per-model output file follows that schema
#### Scenario: Outputs are diffable
- **WHEN** two per-model output files for the same fixture note are compared
- **THEN** they share a common structure that allows entity-by-entity and edge-by-edge comparison
### Requirement: Claude reference run is scored on quality only
The Claude reference run SHALL be evaluated on extraction quality only — entity correctness, relationship plausibility and typing, and confidence-tag accuracy — and SHALL NOT use wall-clock speed as a metric.
#### Scenario: Speed excluded from the reference run
- **WHEN** the Claude reference run is evaluated
- **THEN** wall-clock time per note is not used as a metric
- **AND** speed is reintroduced only when local Ollama models are later benchmarked against the reference set
### Requirement: Prompt is a committed, reusable file
The benchmark dispatch prompt SHALL be written as a committed file that can be copy/pasted into a Claude Code session to reproduce the run.
#### Scenario: Prompt persisted as a file
- **WHEN** the benchmark is set up
- **THEN** the dispatch prompt exists as a committed file in the repository
- **AND** running it does not require reconstructing the prompt from memory

View File

@ -0,0 +1,30 @@
## 1. Record the decision
- [x] 1.1 Add ADR-013 to `docs/memory-system/03-architecture-decisions.md` recording the build-order inversion (vault-migration-first → build-first / migrate-incrementally), with rationale and rejected alternatives
- [x] 1.2 Update the "Decisions locked" pointer in `CLAUDE.md` to reference ADR-013
## 2. Author the benchmark prompt deliverable
- [x] 2.1 Choose and create the home for the prompt and per-model reference outputs (default `docs/memory-system/benchmark/`)
- [x] 2.2 Write the shared minimal extraction spec (entities, typed relationships, `INFERRED`/`AMBIGUOUS` confidence) as the schema the subagents must emit
- [x] 2.3 Write the copy/paste-able dispatch prompt file: one subagent per Claude tier (`claude-haiku-4-5`, `claude-sonnet-4-6`, `claude-opus-4-8`), fairness contract (note text + spec only; no repo/project context), per-model output files
- [x] 2.4 Confirm the prompt embeds the output schema verbatim and names the fixture inputs and per-model output paths
## 3. Rewrite Step 2c (benchmark) in the runbook
- [x] 3.1 Replace Step 2c's model table/metrics with the reference-set design: Claude tiers only, quality-only metrics, speed deferred to the later Ollama run
- [x] 3.2 State that the per-model reference set is the rubric local Ollama models are scored against later, and point to the prompt file
- [x] 3.3 Update Open question §6 to reflect that the Claude reference run produces references (not the final model choice)
## 4. Resequence Step 1 (migration) in the runbook
- [x] 4.1 Rewrite Step 1 so bulk vault migration is deferred; keep only fixture-note selection (510 notes, deliberate variety) as the pre-build step
- [x] 4.2 Add the post-build sequence: validate end-to-end on one small mixed code+docs project, then onboard projects one at a time with observe-and-adjust between each
- [x] 4.3 Make the four distinct items explicit in the runbook: benchmark fixtures / deferred bulk migration / initial mixed validation project / project-by-project rollout
- [x] 4.4 Note the migration-unit granularity question and that vault notes (SLM path) and project code (tree-sitter path) are different extraction paths
## 5. Reconcile and verify
- [x] 5.1 Check `docs/memory-system/04-build-plan.md` for any statement of the old "migration first" order and reconcile it with ADR-013
- [x] 5.2 Update the `_Last updated:_` / status lines on every edited design doc
- [x] 5.3 Re-read the edited runbook end-to-end to confirm Step 1 and Step 2c are internally consistent and the four distinct items are not blurred

View File

@ -0,0 +1,2 @@
schema: spec-driven
created: 2026-06-04

View File

@ -0,0 +1,33 @@
# Handoff — graphify-ollama-setup (2026-06-05)
## Where we are
Mid-application of the `graphify-ollama-setup` OpenSpec change. Toolchain installed; the hard part (finding a local model that produces relationship-bearing graphs through graphify) is SOLVED. Remaining: formal scoring, selection record, vault build, doc updates, verify.
## The working config (use this)
- Model: **qwen2.5-coder:7b** (graphify's shipped default). Run via Modelfile variant **`qwen25-coder-7b-16k`** (already created: `FROM qwen2.5-coder:7b` + `PARAMETER num_ctx 16384`).
- Always: `OLLAMA_BASE_URL=http://127.0.0.1:11434/v1` (MUST end in `/v1`), `--backend ollama --max-concurrency 1` (single 12GB GPU, run models sequentially).
- A one-line thinking-OFF patch (`reasoning_effort:"none"`) is applied to installed `~/.local/lib/python3.14/site-packages/graphify/llm.py` (backup: /tmp/graphify-bench/llm.py.orig; patched copy: /tmp/graphify-bench/llm.py.patched). No-op for qwen2.5-coder but harmless; note it for production (lost on `pip install --upgrade`).
## Gotchas (all the lessons, so you don't relearn them)
1. `GRAPHIFY_OLLAMA_NUM_CTX` env var does NOT propagate through graphify's ollama `/v1` endpoint — bake `num_ctx` into a Modelfile variant instead. Verify with `ollama ps` (CONTEXT column).
2. `graphify extract <path>` needs a DIRECTORY, not a single file. Each benchmark fixture was isolated in its own dir.
3. For the vault build, add `--exclude .obsidian` (graphify treats `.obsidian/*.json` as code and indexes it as noise).
4. `extract` produces `graphify-out/graph.json` + `.graphify_analysis.json` (god-nodes here). It does NOT produce `GRAPH_REPORT.md` — that's a separate step; read the analysis JSON for the god-node sanity check (task 4.2).
5. OLLAMA_API_KEY only needed if base_url is non-loopback; 127.0.0.1 avoids it.
6. Small models (gemma4:e4b 8B, gemma4:e2b 5.1B, qwen3.5:2b 2.3B) all FAIL the relationship/edge gate through graphify's prompt — don't revisit them.
## Remaining tasks (in order)
- **3.4 — Score** qwen2.5-coder:7b outputs vs the `.opus` references (`docs/memory-system/benchmark/reference-outputs/*.opus.md`) on the 3 COMPARABLE dimensions only: entity correctness, relationship plausibility, edge-level EXTRACTED/INFERRED/AMBIGUOUS confidence. EXCLUDE facets + entity-type + free-text relation-typing (graphify doesn't emit them; facets are note metadata per ADR-011). The qwen2.5-coder per-fixture node/edge lists are recorded verbatim in `docs/memory-system/benchmark/scoring-results-2026-06-04.md` (Round 3) so you can score WITHOUT re-running. Raw graph.json (if /tmp survived): `/tmp/graphify-bench/out6/coder7b/<slug>/graphify-out/graph.json`.
- **3.5 / 3.6 — Record + select:** add scoring scores + selection rationale to the findings doc; qwen2.5-coder:7b is the de-facto selection. Per the change, don't lock before 3.4 is done.
- **4 — Build vault graph:** `OLLAMA_BASE_URL=http://127.0.0.1:11434/v1 graphify extract ~/Documents/SecondBrain --backend ollama --model qwen25-coder-7b-16k --max-concurrency 2 --token-budget 4000 --exclude .obsidian --out <dir>`. (token-budget BELOW context so multi-file chunks fit 16k.) Then 4.2 god-node sanity check via `.graphify_analysis.json`; 4.3 confirm graph artifacts are gitignored/not vault-synced (ADR-008).
- **5.1 — Update** `docs/memory-system/05-implementation-process.md` status + Step 2 markers (2a/2b/2d executed; selected model = qwen2.5-coder:7b; the e4b/e2b inconsistency is resolved — see findings). NOTE: that doc currently claims 2a/2b "DONE" but they were only truly executed this round; reconcile honestly.
- **5.2 — Verify** with `/opsx:verify` then archive.
## Re-run extraction if /tmp was wiped (reboot)
Fixtures live in the vault; recreate isolated dirs by copying the 6 source notes (slugs in scoring-results-2026-06-04.md Round-3 table) into per-fixture temp dirs, then run the working command above per fixture.
## Pointers
- Findings + raw data: `docs/memory-system/benchmark/scoring-results-2026-06-04.md`
- Model-choice reference (incl. Triplex/purpose-built assessment): `docs/graphify/10-extraction-model-options.md`
- Task list + status block: `openspec/changes/graphify-ollama-setup/tasks.md`
- Optional cleanup: extra ollama variants from the investigation (gemma4-e4b-8k/16k, gemma4-e2b-8k/16k, qwen35-2b-8k/16k) can be removed with `ollama rm`; KEEP `qwen25-coder-7b-16k`.

View File

@ -0,0 +1,55 @@
## Context
This change implements Steps 2a2d of `docs/memory-system/05-implementation-process.md` — the critical path of the memory system. Step 2c (the Claude reference-set gate) is already executed: 18 gold-standard fragments (6 fixtures × 3 tiers) exist in `docs/memory-system/benchmark/reference-outputs/`, with the `claude-opus-4-8` outputs as the scoring rubric (specced under `reference-extraction-benchmark`). What remains is to stand up the local extraction toolchain, score local Ollama models against that rubric, pick the model, and build the first vault graph.
Constraints:
- The vault is the existing `~/Documents/SecondBrain` (ADR-012); the build runs against it as-is, no bulk migration (ADR-013, `incremental-migration` spec).
- Markdown is the single source of truth; graph artifacts are disposable (ADR-008).
- A feasibility gut-check found `gemma4:e4b` runs at ~74 tok/s on the local GPU, but quality against the references has not been measured. The implementation-process doc is internally inconsistent on the model (`e4b` vs `e2b`) — a tell that nothing is locked.
## Goals / Non-Goals
**Goals:**
- Install and verify Graphify; configure Ollama for extraction.
- Score candidate Ollama models against the existing Opus rubric on the 6 fixtures (quality + speed) and select the extraction model by evidence.
- Build the initial vault graph with the selected model and sanity-check god-nodes.
**Non-Goals:**
- Regenerating or modifying the Step 2c reference set (consumed, not produced here).
- Per-project code graphs / Step 2e (separate, free tree-sitter path; out of range).
- Hooks, memsearch, sync, plugin packaging (Steps 36).
- Bulk vault migration (deferred to last per ADR-013).
- Choosing the sync mechanism or stale-rebuild threshold (Open questions §23).
## Decisions
**Selection by scoring, not by gut-check.** The model is chosen by comparing each candidate's Graphify-shaped output to the Opus reference per fixture (entity correctness, relationship typing, confidence-tag accuracy) plus measured wall-clock speed. Alternative considered: adopt `gemma4:e4b` directly since feasibility passed — rejected because the gut-check validated speed, not extraction quality, and Open-question §6 explicitly says do not hardcode. `gemma4:e4b` enters as the front-runner candidate, nothing more.
**Score against the Opus tier as the rubric.** Haiku/Sonnet references exist but Opus is the gold standard (per `reference-extraction-benchmark`). Candidates are scored primarily against Opus; the other tiers provide a quality gradient for context.
**Ollama config travels with this step.** `OLLAMA_FLASH_ATTENTION=1` (KV-cache VRAM savings) and `GRAPHIFY_OLLAMA_NUM_CTX=8192` (sufficient for 2002000-word notes with prompt headroom) are set in the shell profile now and re-baked into the plugin env block at Step 6. `GRAPHIFY_OLLAMA_KEEP_ALIVE` is deferred to packaging. Verify with `ollama ps` after the first call.
**Build the full vault, then review god-nodes.** Rather than a synthetic subset, build over the real `~/Documents/SecondBrain` and use `GRAPH_REPORT.md`'s most-connected nodes as the sanity signal — the highest-traffic tools/clients/domains should surface as god-nodes. Cheap, and it exercises the real extraction path.
## Risks / Trade-offs
- **No candidate scores acceptably against the rubric** → If even the best candidate is well below the Opus reference, surface it as a finding rather than forcing a selection; the front-runner's speed does not rescue poor extraction quality. Selection may need a larger candidate or a revisit of token budget / context.
- **Scoring is partly qualitative** → Entity/relationship/confidence-tag comparison against references is judgement-based, not a single numeric pass/fail. Mitigation: record the per-fixture comparison and rationale in the result artifact so the choice is auditable, not asserted.
- **Vault content drift during build** → Building over the live vault means notes could change mid-run. Mitigation: the graph is disposable and rebuildable (`--force`); a one-shot initial build is acceptable.
- **GPU/VRAM regressions** → The ~74 tok/s figure was post-reboot; throughput can vary. Mitigation: `--max-concurrency 2` and a bounded `--token-budget` keep memory pressure predictable; record actual speed per candidate.
## Migration Plan
Sequential, low blast radius (local-only, no production system touched):
1. `pip install graphifyy`; verify `graphify --version`.
2. Export Ollama env vars; pull candidate model(s); verify context via `ollama ps`.
3. Run each candidate over the 6 fixtures; score vs. the Opus references; record results.
4. Select the model; build the initial vault graph; review `GRAPH_REPORT.md`.
Rollback: artifacts are disposable — delete `graphify-out/` and re-run; uninstall `graphifyy` if needed. No data is mutated in the vault (extraction is read-only over notes).
## Open Questions
- Exact candidate set beyond `gemma4:e4b` (e.g. whether to also score a smaller/larger sibling) — decided at run time based on what is pulled and how the front-runner scores.
- Where the scoring-result artifact lives (under `docs/memory-system/benchmark/` alongside the references is the natural home) — settle when writing it.
- `--token-budget` / `--max-concurrency` tuning — start from the doc's `512` / `2` and adjust if quality or VRAM demands it.

View File

@ -0,0 +1,32 @@
## Why
Step 2c produced the gold-standard reference set (18 Claude fragments across 6 fixtures), but no local extraction model has been scored against it yet — so the critical-path question "which Ollama model drives vault extraction?" is still open and blocks the initial vault graph build (Step 2d) and everything downstream (hooks, plugin packaging). This change installs and configures the extraction toolchain, scores candidate Ollama models against the existing references, and builds the first real vault graph.
## What Changes
- Install and verify Graphify (`graphifyy` package, `graphify` command) — Step 2a.
- Configure Ollama for extraction: flash attention, 8K context, keep-alive — Step 2b.
- Score candidate local Ollama models against the Step 2c gold-standard references on the same 6 fixtures, measuring both extraction quality (vs. the Opus rubric) and wall-clock speed, then select the extraction model. `gemma4:e4b` is the front-runner candidate (feasibility-validated at ~74 tok/s) but is **not** pre-selected — selection is decided by the scoring run.
- Build the initial vault graph with the selected model and review `GRAPH_REPORT.md` god-nodes for sanity — Step 2d.
- Step 2c (reference set) is a **completed prerequisite**, not work in this change; per-project code graphs (Step 2e) are **out of scope**.
## Capabilities
### New Capabilities
- `local-model-selection`: Score candidate Ollama extraction models against the Step 2c gold-standard reference set and select the model by evidence (quality vs. the Opus rubric + measured speed), rather than hardcoding one. Owns the Ollama runtime configuration (flash attention, context size) that the scoring run depends on.
- `vault-graph-build`: Build the initial `~/Documents/SecondBrain` vault knowledge graph with the selected extraction model and verify graph sanity by reviewing god-nodes against the vault's actual content.
### Modified Capabilities
<!-- None. This change consumes `reference-extraction-benchmark` (Step 2c, complete) as a
prerequisite and is a build-step under `incremental-migration`; it changes neither spec's
requirements. -->
## Impact
- **New dependencies:** Graphify (`graphifyy` on PyPI, anchored to v0.8.30), a running Ollama with at least one pulled candidate model.
- **Environment:** Ollama env vars (`OLLAMA_FLASH_ATTENTION`, `GRAPHIFY_OLLAMA_NUM_CTX`, `GRAPHIFY_OLLAMA_KEEP_ALIVE`) set in the shell profile now; rebaked into the plugin env block at Step 6.
- **Artifacts produced:** a model-scoring result recording the chosen model and its rationale; the initial vault `graphify-out/graph.json` + `GRAPH_REPORT.md` (disposable/rebuildable per ADR-008, not synced).
- **Consumes:** the 18 reference fragments in `docs/memory-system/benchmark/reference-outputs/` and the 6 fixtures.
- **Unblocks:** Step 2e (per-project code graphs), Step 3 (hooks), and downstream plugin packaging — all of which assume a selected model and a built vault graph.

View File

@ -0,0 +1,53 @@
## ADDED Requirements
### Requirement: Extraction toolchain is installed and verified
The Graphify CLI SHALL be installed from the `graphifyy` PyPI package and verified to run before any extraction is attempted, and a running Ollama with at least one pulled candidate model SHALL be available.
#### Scenario: Graphify is callable
- **WHEN** the toolchain setup completes
- **THEN** `graphify --version` returns a version without error
- **AND** at least one candidate Ollama model is pulled and listed by `ollama list`
### Requirement: Ollama runtime is configured for extraction
Ollama SHALL be configured with the settings the extraction run depends on: flash attention enabled (`OLLAMA_FLASH_ATTENTION=1`) and a context window sufficient for vault notes (`GRAPHIFY_OLLAMA_NUM_CTX=8192`), and the allocated context SHALL be verified after the first extraction call.
#### Scenario: Configuration is in effect
- **WHEN** the first extraction call is made
- **THEN** flash attention is enabled and the context size is 8192
- **AND** `ollama ps` shows the expected allocated context for the loaded model
### Requirement: Candidate models are scored against the gold-standard reference set
The extraction model SHALL be selected by scoring candidate Ollama models against the existing Step 2c reference set (the 18 fragments in `docs/memory-system/benchmark/reference-outputs/`) over the same 6 fixture notes. Each candidate's Graphify-shaped output SHALL be compared to the `claude-opus-4-8` gold-standard output for entity correctness, relationship plausibility and typing, and `INFERRED`/`AMBIGUOUS` confidence-tag accuracy, and wall-clock extraction speed SHALL be measured per candidate.
#### Scenario: Each candidate is scored on quality and speed
- **WHEN** a candidate model is run over the 6 fixtures
- **THEN** its output is scored against the Opus reference on entity correctness, relationship typing, and confidence-tag accuracy
- **AND** its wall-clock extraction speed is recorded
#### Scenario: Reference benchmark is consumed, not re-created
- **WHEN** scoring is performed
- **THEN** it reads the existing reference fragments produced by the `reference-extraction-benchmark` capability
- **AND** it does not regenerate or modify the Claude reference set
### Requirement: Model is selected by evidence, not hardcoded
The chosen extraction model SHALL be the one justified by the scoring run's recorded results, and no model SHALL be hardcoded as the selection before scoring completes. `gemma4:e4b` MAY be the front-runner candidate, but its selection SHALL depend on its scored quality, not its feasibility gut-check alone.
#### Scenario: Selection records its rationale
- **WHEN** a model is selected
- **THEN** a result artifact records the chosen model, its quality scores against the Opus rubric, and its measured speed
- **AND** the rationale references the scoring evidence rather than asserting a pre-chosen model
#### Scenario: No model is locked before scoring
- **WHEN** scoring has not yet run
- **THEN** no model is committed as the selection
- **AND** the front-runner candidate is treated as unconfirmed until scored

View File

@ -0,0 +1,37 @@
## ADDED Requirements
### Requirement: Initial vault graph is built with the selected model
The initial knowledge graph SHALL be built over the `~/Documents/SecondBrain` vault using the model selected by the `local-model-selection` capability, via Graphify's Ollama document-extraction backend, and SHALL NOT begin until a model has been selected.
#### Scenario: Build uses the selected model
- **WHEN** the initial vault graph is built
- **THEN** Graphify extracts over `~/Documents/SecondBrain` using the selected Ollama model
- **AND** the build does not run before model selection is complete
#### Scenario: Build scope is the vault only
- **WHEN** the initial graph is built
- **THEN** the extraction target is the vault, not any project code repository
- **AND** per-project code graphs are not built as part of this change
### Requirement: Graph sanity is verified via god-nodes
After the build, `GRAPH_REPORT.md` SHALL be reviewed to confirm the most-connected nodes (god-nodes) are the tools, clients, and domain concepts that the vault's content would predict, so an obviously wrong extraction is caught before downstream steps depend on it.
#### Scenario: God-nodes match vault content
- **WHEN** `GRAPH_REPORT.md` is reviewed after the build
- **THEN** the top god-nodes are recognizable high-traffic tools, clients, or domains from the vault
- **AND** an implausible god-node distribution is flagged rather than accepted
### Requirement: Graph artifacts are treated as disposable
The produced graph artifacts (`graphify-out/`, `GRAPH_REPORT.md`, any index caches) SHALL be treated as rebuildable from the markdown vault and SHALL NOT be synced as source of truth, consistent with the markdown-as-truth decision (ADR-008).
#### Scenario: Graph output is not the source of truth
- **WHEN** the graph artifacts are produced
- **THEN** they are rebuildable from the vault markdown
- **AND** they are excluded from vault sync rather than treated as authoritative

View File

@ -0,0 +1,32 @@
> **STATUS (2026-06-05, complete — 16/16 tasks done):** Toolchain installed (task 1 ✓). Key findings in `docs/memory-system/benchmark/scoring-results-2026-06-04.md`. Critical mechanics discovered: (a) `GRAPHIFY_OLLAMA_NUM_CTX` does NOT propagate through graphify's ollama `/v1` endpoint — bake context via a Modelfile variant instead; (b) graphify base_url must end in `/v1`; (c) a one-line thinking-OFF patch (`reasoning_effort:"none"`) was applied to the installed `llm.py` (backup at /tmp/graphify-bench/llm.py.orig). Small 25B models (gemma4:e4b/e2b, qwen3.5:2b) FAIL the relationship/edge gate. **qwen2.5-coder:7b (graphify's shipped default) PASSES** — full 6-fixture extraction done (~59 tok/s). All formal scoring (3.4), selection record (3.5/3.6), vault build (4), and doc updates (5) completed. Ready for archival.
## 1. Install and verify the toolchain (Step 2a)
- [x] 1.1 Install Graphify: `pip install graphifyy` (double-y package; `graphify` command)
- [x] 1.2 Verify it runs: `graphify --version` returns a version without error
- [x] 1.3 Confirm Ollama is running and pull the front-runner candidate `gemma4:e4b` (and any sibling to be scored); verify with `ollama list`
## 2. Configure Ollama for extraction (Step 2b)
- [x] 2.1 Export `OLLAMA_FLASH_ATTENTION=1` and `GRAPHIFY_OLLAMA_NUM_CTX=8192` in the shell profile (defer `GRAPHIFY_OLLAMA_KEEP_ALIVE` to Step 6 packaging) — superseded — env-var approach does not propagate via /v1; use Modelfile-baked num_ctx (see findings). No shell-profile/systemd change made per user decision.
- [x] 2.2 Run one extraction call and verify allocated context with `ollama ps` (expect 8192) — context verified at 16384 via Modelfile variant in `ollama ps` (env-var path is a no-op through graphify).
## 3. Score candidates against the Step 2c references
- [x] 3.1 Confirm the reference set is intact: 18 fragments in `docs/memory-system/benchmark/reference-outputs/` and the 6 fixtures in `benchmark/dispatch-prompt.md` (read-only; do not regenerate) — 18 reference fragments + 6 fixtures confirmed intact, used read-only.
- [x] 3.2 Run each candidate Ollama model over the same 6 fixtures via Graphify's Ollama backend, capturing Graphify-shaped output per fixture — all candidates run via graphify ollama backend; qwen2.5-coder:7b results recorded in findings.
- [x] 3.3 Record wall-clock extraction speed per candidate — qwen2.5-coder:7b ~59 tok/s; small-model speeds in findings.
- [x] 3.4 Score each candidate's output against the `claude-opus-4-8` reference per fixture: entity correctness, relationship plausibility/typing, and `INFERRED`/`AMBIGUOUS` confidence-tag accuracy — Scored qwen2.5-coder:7b on the 3 comparable dims (entity correctness ~2.2/5, relationship plausibility ~1.9/5, confidence calibration 3/5 where observable); excluded dims (facets/entity-type/relation-typing) documented N/A. Full scoring table in scoring-results-2026-06-04.md §3.4.
- [x] 3.5 Write a scoring-result artifact (under `docs/memory-system/benchmark/`) recording per-candidate quality scores, measured speed, and the selected model with its rationale — Scoring table + per-fixture justifications + selection rationale recorded in docs/memory-system/benchmark/scoring-results-2026-06-04.md (§3.4, §3.6).
- [x] 3.6 Select the extraction model from the recorded evidence — do not lock a model before 3.4 completes — Selected qwen2.5-coder:7b (variant qwen25-coder-7b-16k) — graphify's default and the only candidate clearing the edge gate; smaller models all failed. Known limitation: shallow relationship semantics. ~59 tok/s.
## 4. Build and verify the initial vault graph (Step 2d)
- [x] 4.1 Build the graph with the selected model: `graphify extract --path ~/Documents/SecondBrain --backend ollama --model <selected> --token-budget 512 --max-concurrency 2` — Built with `graphify extract ~/Documents/SecondBrain --backend ollama --model qwen25-coder-7b-16k --max-concurrency 1 --token-budget 4000 --exclude .obsidian --out /tmp/graphify-bench/vault-graph`. Ran at concurrency 1 (not 2 — untested on 12GB GPU) and token-budget 4000 (not 512 — fits the 16k context). Result: 57 nodes, 43 edges, 15 communities, ~$0 local.
- [x] 4.2 Review `GRAPH_REPORT.md`: confirm top god-nodes are recognizable high-traffic tools, clients, and domains from the vault; flag an implausible distribution rather than accepting it — God-node distribution plausible: Speed-to-Lead (deg 12) dominant, then Email A/B/C Experiment Hub, ACV Estimates, Claude, Vault Conventions, project-config hubs — all recognizable high-traffic vault concepts; no implausible top node. Read from .graphify_analysis.json (extract does not emit GRAPH_REPORT.md).
- [x] 4.3 Confirm graph artifacts (`graphify-out/`, `GRAPH_REPORT.md`) are treated as disposable/rebuildable and excluded from vault sync (ADR-008) — Satisfied by construction — `--out /tmp/graphify-bench/vault-graph` writes outside ~/Documents/SecondBrain; nothing written into the vault. Artifacts disposable/rebuildable per ADR-008.
## 5. Wrap up
- [x] 5.1 Update `docs/memory-system/05-implementation-process.md` status line and Step 2 markers to reflect 2a/2b/2d executed and the selected model (resolve the `e4b`/`e2b` inconsistency) — also: stale 'no recommended model' claim in docs/graphify/05 already corrected 2026-06-05; the e4b/e2b naming resolved — these are real gemma4 8.0B/5.1B + qwen35 2.3B, not gemma3n renames (see findings).
- [x] 5.2 Verify the change with `/opsx:verify` before archiving

View File

@ -0,0 +1,53 @@
# Spec: Incremental Migration
## Purpose
Defines the ordering constraints for onboarding the memory system into production: the system must be built and validated before any migration is performed, projects are onboarded one at a time, and bulk vault migration is deferred to last.
## Requirements
### Requirement: System is validated before bulk migration
The build SHALL be completed and validated end-to-end against a small fixture set and one small project containing both code and documents before any bulk vault migration is performed.
#### Scenario: Fixtures feed the gate without bulk migration
- **WHEN** the Step 2c benchmark gate is run
- **THEN** it uses a 510 note fixture set
- **AND** the remaining vault notes are not migrated at that point
#### Scenario: First end-to-end validation is a mixed project
- **WHEN** the system is first validated end-to-end
- **THEN** the target is one small project containing both code and documents
- **AND** validation covers both the document extraction path and the code (tree-sitter) path
### Requirement: Projects are onboarded one at a time
After the system is validated, projects SHALL be onboarded individually, with observation and adjustment between each, rather than migrated in bulk.
#### Scenario: One project at a time
- **WHEN** a project is onboarded after initial validation
- **THEN** it is migrated on its own
- **AND** its migration is observed and the process adjusted before the next project is onboarded
### Requirement: Bulk vault migration is deferred to last
Bulk migration of the full vault SHALL be deferred until after the system is built and validated, rather than performed as the first build step.
#### Scenario: Bulk migration ordering
- **WHEN** the build order is followed
- **THEN** bulk vault migration occurs after system validation
- **AND** only the fixture notes are migrated beforehand
### Requirement: Build-order inversion is recorded in an ADR
The inversion of the documented build order (from vault-migration-first to build-first / migrate-incrementally) SHALL be recorded in a new ADR and reflected in the project's locked-decisions pointer.
#### Scenario: ADR captures the inversion
- **WHEN** the build order is resequenced
- **THEN** a new ADR (ADR-013) records the inversion and its rationale
- **AND** the "Decisions locked" pointer in `CLAUDE.md` is updated to reference it

View File

@ -0,0 +1,59 @@
# Spec: local-model-selection
## Purpose
Install Graphify, configure Ollama for extraction (flash attention, context window), score candidate models against Claude-Opus references (entity correctness, relationship typing, confidence-tag accuracy, speed), and select the extraction model by evidence. As of 2026-06-04, `qwen2.5-coder:7b` is the selected model.
## Requirements
### Requirement: Extraction toolchain is installed and verified
The Graphify CLI SHALL be installed from the `graphifyy` PyPI package and verified to run before any extraction is attempted, and a running Ollama with at least one pulled candidate model SHALL be available.
#### Scenario: Graphify is callable
- **WHEN** the toolchain setup completes
- **THEN** `graphify --version` returns a version without error
- **AND** at least one candidate Ollama model is pulled and listed by `ollama list`
### Requirement: Ollama runtime is configured for extraction
Ollama SHALL be configured with the settings the extraction run depends on: flash attention enabled (`OLLAMA_FLASH_ATTENTION=1`) and a context window sufficient for vault notes (`GRAPHIFY_OLLAMA_NUM_CTX=8192`), and the allocated context SHALL be verified after the first extraction call.
#### Scenario: Configuration is in effect
- **WHEN** the first extraction call is made
- **THEN** flash attention is enabled and the context size is 8192
- **AND** `ollama ps` shows the expected allocated context for the loaded model
### Requirement: Candidate models are scored against the gold-standard reference set
The extraction model SHALL be selected by scoring candidate Ollama models against the existing Step 2c reference set (the 18 fragments in `docs/memory-system/benchmark/reference-outputs/`) over the same 6 fixture notes. Each candidate's Graphify-shaped output SHALL be compared to the `claude-opus-4-8` gold-standard output for entity correctness, relationship plausibility and typing, and `INFERRED`/`AMBIGUOUS` confidence-tag accuracy, and wall-clock extraction speed SHALL be measured per candidate.
#### Scenario: Each candidate is scored on quality and speed
- **WHEN** a candidate model is run over the 6 fixtures
- **THEN** its output is scored against the Opus reference on entity correctness, relationship typing, and confidence-tag accuracy
- **AND** its wall-clock extraction speed is recorded
#### Scenario: Reference benchmark is consumed, not re-created
- **WHEN** scoring is performed
- **THEN** it reads the existing reference fragments produced by the `reference-extraction-benchmark` capability
- **AND** it does not regenerate or modify the Claude reference set
### Requirement: Model is selected by evidence, not hardcoded
The chosen extraction model SHALL be the one justified by the scoring run's recorded results, and no model SHALL be hardcoded as the selection before scoring completes. `gemma4:e4b` MAY be the front-runner candidate, but its selection SHALL depend on its scored quality, not its feasibility gut-check alone.
#### Scenario: Selection records its rationale
- **WHEN** a model is selected
- **THEN** a result artifact records the chosen model, its quality scores against the Opus rubric, and its measured speed
- **AND** the rationale references the scoring evidence rather than asserting a pre-chosen model
#### Scenario: No model is locked before scoring
- **WHEN** scoring has not yet run
- **THEN** no model is committed as the selection
- **AND** the front-runner candidate is treated as unconfirmed until scored

View File

@ -0,0 +1,67 @@
# Spec: Reference Extraction Benchmark
## Purpose
Defines the process for producing a reusable Claude-tier reference set that serves as the quality rubric against which local Ollama extraction models are later scored.
## Requirements
### Requirement: Reference set is produced from Claude tiers
The benchmark SHALL produce a reusable reference set by dispatching one Claude Code subagent per Claude tier (`claude-haiku-4-5`, `claude-sonnet-4-6`, `claude-opus-4-8`) over the fixture notes, with each subagent's output written to a distinct per-model file.
#### Scenario: One reference file per model
- **WHEN** the benchmark prompt is run on the fixture set
- **THEN** a separate output file is produced for each of the three Claude tiers
- **AND** each file contains that tier's extraction result for every fixture note
#### Scenario: Opus output is the scoring rubric
- **WHEN** the reference set is reviewed
- **THEN** the `claude-opus-4-8` output is treated as the gold-standard rubric against which the other tiers, and later the local Ollama models, are scored
### Requirement: Dispatch prompt enforces a fairness contract
The benchmark prompt SHALL give each subagent only the raw note text and a shared minimal extraction spec, and SHALL forbid reading repository files or pulling project context.
#### Scenario: Minimal context per subagent
- **WHEN** a subagent is dispatched for a fixture note
- **THEN** its input is limited to the note text and the shared extraction spec
- **AND** it is explicitly instructed not to read `CLAUDE.md`, design docs, or other project context
### Requirement: Output conforms to the Graphify-shaped schema
Each subagent SHALL emit a structured fragment matching the shape Graphify produces — entities, typed relationships, and `INFERRED`/`AMBIGUOUS` confidence tags — so per-model outputs are diffable against each other and against later Ollama output.
#### Scenario: Schema embedded in the prompt
- **WHEN** the benchmark prompt is authored
- **THEN** the required output schema is embedded verbatim in the prompt
- **AND** every per-model output file follows that schema
#### Scenario: Outputs are diffable
- **WHEN** two per-model output files for the same fixture note are compared
- **THEN** they share a common structure that allows entity-by-entity and edge-by-edge comparison
### Requirement: Claude reference run is scored on quality only
The Claude reference run SHALL be evaluated on extraction quality only — entity correctness, relationship plausibility and typing, and confidence-tag accuracy — and SHALL NOT use wall-clock speed as a metric.
#### Scenario: Speed excluded from the reference run
- **WHEN** the Claude reference run is evaluated
- **THEN** wall-clock time per note is not used as a metric
- **AND** speed is reintroduced only when local Ollama models are later benchmarked against the reference set
### Requirement: Prompt is a committed, reusable file
The benchmark dispatch prompt SHALL be written as a committed file that can be copy/pasted into a Claude Code session to reproduce the run.
#### Scenario: Prompt persisted as a file
- **WHEN** the benchmark is set up
- **THEN** the dispatch prompt exists as a committed file in the repository
- **AND** running it does not require reconstructing the prompt from memory

View File

@ -0,0 +1,43 @@
# Spec: vault-graph-build
## Purpose
Build the initial knowledge graph over the `~/Documents/SecondBrain` vault using the selected Ollama model via Graphify; verify god-node plausibility; treat graph artifacts as disposable/rebuildable, not source-of-truth (ADR-008).
## Requirements
### Requirement: Initial vault graph is built with the selected model
The initial knowledge graph SHALL be built over the `~/Documents/SecondBrain` vault using the model selected by the `local-model-selection` capability, via Graphify's Ollama document-extraction backend, and SHALL NOT begin until a model has been selected.
#### Scenario: Build uses the selected model
- **WHEN** the initial vault graph is built
- **THEN** Graphify extracts over `~/Documents/SecondBrain` using the selected Ollama model
- **AND** the build does not run before model selection is complete
#### Scenario: Build scope is the vault only
- **WHEN** the initial graph is built
- **THEN** the extraction target is the vault, not any project code repository
- **AND** per-project code graphs are not built as part of this change
### Requirement: Graph sanity is verified via god-nodes
After the build, `GRAPH_REPORT.md` SHALL be reviewed to confirm the most-connected nodes (god-nodes) are the tools, clients, and domain concepts that the vault's content would predict, so an obviously wrong extraction is caught before downstream steps depend on it.
#### Scenario: God-nodes match vault content
- **WHEN** `GRAPH_REPORT.md` is reviewed after the build
- **THEN** the top god-nodes are recognizable high-traffic tools, clients, or domains from the vault
- **AND** an implausible god-node distribution is flagged rather than accepted
### Requirement: Graph artifacts are treated as disposable
The produced graph artifacts (`graphify-out/`, `GRAPH_REPORT.md`, any index caches) SHALL be treated as rebuildable from the markdown vault and SHALL NOT be synced as source of truth, consistent with the markdown-as-truth decision (ADR-008).
#### Scenario: Graph output is not the source of truth
- **WHEN** the graph artifacts are produced
- **THEN** they are rebuildable from the vault markdown
- **AND** they are excluded from vault sync rather than treated as authoritative