Compare commits
10 Commits
75a8a94192
...
6a5b09f253
| Author | SHA1 | Date |
|---|---|---|
|
|
6a5b09f253 | |
|
|
3e38398500 | |
|
|
3823bb7c91 | |
|
|
65595c3995 | |
|
|
77355b4b44 | |
|
|
b408a62136 | |
|
|
77febc8aca | |
|
|
235fd45ff7 | |
|
|
f73cb46a53 | |
|
|
aa1ffe30fe |
10
CLAUDE.md
10
CLAUDE.md
|
|
@ -45,10 +45,9 @@ the design, update this paragraph (and the relevant `docs/memory-system/` files
|
|||
match.** Keep it accurate, don't preserve it for its own sake.
|
||||
|
||||
Two memory types kept as **separate systems**: **episodic** ("what happened, when") handled by
|
||||
**memsearch** (Milvus Lite, embedded), and **semantic/knowledge** ("how do we…") handled by a
|
||||
flat **Obsidian markdown vault** as the single source of truth. Notes keep `summary` +
|
||||
namespaced tags (`tool/`/`client/`/`domain/`/`convention/`/`scope/`) as metadata, and the vault
|
||||
is queried via a **Graphify knowledge graph** (local Ollama SLM for doc extraction, free
|
||||
**memsearch** (Milvus Lite, embedded), and **semantic/knowledge** ("how do we…") handled by the
|
||||
existing **`~/Documents/SecondBrain` Obsidian vault** as the single source of truth. Notes keep
|
||||
`summary` + six flat, parallel namespaced facets (`type/`/`client/`/`project/`/`domain/`/`tool/`/`convention/`) plus `scope/` as metadata; hierarchy and relationships are expressed via hub notes (`type/hub`), wikilinks, and Graphify graph edges — not nested tag paths. The vault is queried via a **Graphify knowledge graph** (local Ollama SLM for doc extraction, free
|
||||
tree-sitter AST for code). Retrieval is hook-injected + on-demand so project repos stay thin;
|
||||
freshness is lazy
|
||||
(write-time hook + SessionStart reconcile, no daemon/cron); the vault syncs to a VPS while
|
||||
|
|
@ -59,6 +58,8 @@ tag-index CLI and also covers the deferred QMD semantic layer. `04-build-plan.md
|
|||
`06-graphify-evaluation.md` reflect this; if an older doc still describes the Ruby CLI, defer
|
||||
to those two and fix the stale doc.
|
||||
|
||||
**Decisions locked (2026-06-04):** Six-facet tag taxonomy + `scope/` (ADR-011); reuse `~/Documents/SecondBrain` vault rather than creating a new one (ADR-012); build-first / migrate-incrementally — build full system against a fixture set first, defer bulk vault migration to last, onboard projects one at a time (ADR-013).
|
||||
|
||||
## OpenSpec workflow
|
||||
|
||||
Changes are managed spec-driven via OpenSpec. Use the matching skills rather than editing spec
|
||||
|
|
@ -89,7 +90,6 @@ Delegate all file I/O and shell commands to subagents via the Agent tool. No exc
|
|||
|
||||
**Permitted direct tool uses — only these, no others:**
|
||||
|
||||
- **Git commands** (`git status`, `git log`, `git diff`, `git commit`, `git push`) — version control is orchestrator-level.
|
||||
- **Skill invocations via the Skill tool** — the skill handles its own operations.
|
||||
- **Conversational responses requiring zero tool calls.**
|
||||
|
||||
|
|
|
|||
|
|
@ -31,6 +31,10 @@ then verified and corrected against the official GitHub repository and supplemen
|
|||
command quick-reference + setting-up-across-many-projects mini-guide. **Keep this one open while working.**
|
||||
10. **[external-tips.md](external-tips.md)** — Independent/community tips, gotchas with issue links, and an
|
||||
even-handed look at the token-savings debate.
|
||||
11. **[10-extraction-model-options.md](10-extraction-model-options.md)** — Why Graphify uses a general
|
||||
structured-output LLM (not a purpose-built KG extractor), the architecture constraints that make
|
||||
drop-in specialist models (Triplex, GLiNER, REBEL) non-starters, and an honest assessment of whether
|
||||
the Triplex adapter route is worth experimenting with.
|
||||
|
||||
## One-paragraph summary
|
||||
|
||||
|
|
|
|||
|
|
@ -138,7 +138,9 @@ graphify extract ./docs --backend ollama --api-timeout 900 # 15-minute timeout
|
|||
|
||||
### Which Ollama model?
|
||||
|
||||
**There is no official Graphify-recommended/tested model list.** Neither the v8 README nor the site publishes one (verified — see Open questions). So follow the rule the task itself sets and the interview implies: **pick an instruction-following model that fits your RAM/VRAM.** `[interview]` Document extraction is "read this chunk, emit structured entities/relationships," which is an instruction-following + structured-output job, not deep reasoning.
|
||||
**There is no official Graphify-recommended/tested model list in the README or on the site.** Neither the v8 README nor the site publishes one (verified — see Open questions). So follow the rule the task itself sets and the interview implies: **pick an instruction-following model that fits your RAM/VRAM.** `[interview]` Document extraction is "read this chunk, emit structured entities/relationships," which is an instruction-following + structured-output job, not deep reasoning.
|
||||
|
||||
> **Correction — there IS a shipped code default (2026-06-05).** Although the README/site publish no model recommendation, installed graphify 0.8.31 **hardcodes** a fallback in `llm.py:67`: `"default_model": os.environ.get("OLLAMA_MODEL", "qwen2.5-coder:7b")`. `[github]` So if you run `graphify extract --backend ollama` without setting `OLLAMA_MODEL`, the binary silently uses `qwen2.5-coder:7b`. The README's silence on a recommended model is accurate as a documentation statement; it is not accurate as a claim that "no default exists." The choice of a *coder* model for document extraction is a revealed preference — the binding requirement is structured-JSON instruction-following discipline, not domain NER. See [10-extraction-model-options.md](10-extraction-model-options.md) for the full analysis.
|
||||
|
||||
Rough sizing rule of thumb: a model needs roughly its parameter count in **GB of (V)RAM** at common 4-bit quantization (a 7B model ≈ ~5 GB; a 30B+ model wants 24 GB+ VRAM). Leave headroom for the context window.
|
||||
|
||||
|
|
@ -195,7 +197,7 @@ graphify extract ./src
|
|||
|
||||
## Open questions / unverified
|
||||
|
||||
- **No official Ollama model recommendation exists.** Verified absent from the v8 README env/backend sections and not surfaced on the (403-gated) official site. The model names here are community/general-Ollama signals, tagged as such — not Graphify-tested defaults.
|
||||
- **No official Ollama model recommendation exists in the README**, but the installed binary hardcodes `qwen2.5-coder:7b` as the Ollama fallback (`llm.py:67`) — see the Correction callout above and `10-extraction-model-options.md`. `[github]`
|
||||
- **`GRAPHIFY_OLLAMA_KEEP_ALIVE` default value** is documented by *behavior* ("minutes to keep loaded; `0` to unload after each chunk") but the README does not state the numeric default. `[github]`
|
||||
- **`gemma3:27b` exact tag / VRAM numbers** come from a single user issue (#792), not Graphify docs. The original ASR transcript said "Gemma 4 31b," which does not match a shipped tag; treat the community `gemma3:27b`-class figure as illustrative, not authoritative. `[community](https://github.com/safishamsi/graphify/issues/792)`
|
||||
- **The "SLMs as good as frontier models soon" vision** is the creator's aspiration, not a benchmark. `[unverified claim]` / `[interview]`
|
||||
|
|
|
|||
|
|
@ -0,0 +1,131 @@
|
|||
# 10 — Extraction Model Options: Why a General Structured-Output LLM, and Is a Purpose-Built KG Model Worth It?
|
||||
|
||||
_Last updated: 2026-06-05_ | _Status: reference / decision record_
|
||||
|
||||
This document explains why Graphify's document extraction uses a general instruction-following / structured-output LLM rather than a purpose-built knowledge-graph extraction model (e.g. Triplex, GLiNER, REBEL), and honestly assesses whether the purpose-built route is worth experimenting with for this project.
|
||||
|
||||
See also: [05 — Local models & backends](05-local-models-and-backends.md) · [Benchmark results (2026-06-04)](../memory-system/benchmark/scoring-results-2026-06-04.md)
|
||||
|
||||
---
|
||||
|
||||
## The architecture constraint (this is the crux)
|
||||
|
||||
Graphify owns **both** the prompt and the output contract. It is not a thin wrapper you can swap a model into — the extraction pipeline is opinionated end-to-end:
|
||||
|
||||
**Fixed system prompt.** For document/markdown extraction, Graphify sends its own system prompt `_EXTRACTION_SYSTEM` (`llm.py:218-232`). `[github]` There is no user hook to supply a custom extraction prompt; the prompt is hardcoded.
|
||||
|
||||
**Fixed output schema.** The model must return exactly this JSON structure — no approximation accepted:
|
||||
|
||||
```json
|
||||
{
|
||||
"nodes": [{"id": "stem_entity", "label": "...", "file_type": "code|document|paper|image|rationale|concept",
|
||||
"source_file": "...", "source_location": null, "source_url": null,
|
||||
"captured_at": null, "author": null, "contributor": null}],
|
||||
"edges": [{"source": "node_id", "target": "node_id",
|
||||
"relation": "calls|implements|references|cites|conceptually_related_to|shares_data_with|semantically_similar_to",
|
||||
"confidence": "EXTRACTED|INFERRED|AMBIGUOUS",
|
||||
"confidence_score": 1.0, "source_file": "...", "source_location": null, "weight": 1.0}],
|
||||
"hyperedges": [], "input_tokens": 0, "output_tokens": 0
|
||||
}
|
||||
```
|
||||
|
||||
The `relation` field is constrained to a fixed vocabulary of seven values. The `confidence` field is a three-value enum (`EXTRACTED`/`INFERRED`/`AMBIGUOUS`). `[github]` (`llm.py:218-232`)
|
||||
|
||||
**No JSON-mode / grammar enforcement.** For the Ollama backend, Graphify passes only `{"options": {"num_ctx": ...}, "keep_alive": ...}` as extra parameters (`llm.py:506`). `[github]` There is no `response_format`, grammar constraint, or structured-output enforcement. Schema compliance rests entirely on the model's instruction-following ability.
|
||||
|
||||
**Lenient but not magic parser.** The response goes through `_parse_llm_json` (`llm.py:269-303`), which strips optional markdown fences and then tries `json.loads`. If that fails, it scans for the first balanced JSON object in the text. A model that emits its own schema — triples, spans, BIO tags — will produce an empty or nonsense graph, not a graceful fallback. `[github]`
|
||||
|
||||
**Temperature forced to 0 for Ollama.** The Ollama backend config sets `"temperature": 0` (`llm.py:65-73`). `[github]`
|
||||
|
||||
**Deep mode.** `--mode deep` appends `_DEEP_EXTRACTION_SUFFIX` to the same prompt, asking for extra `INFERRED` edges with a more conservative framing (`llm.py:234-247`). `[github]` The schema stays identical.
|
||||
|
||||
---
|
||||
|
||||
## The shipped default: why a coder model?
|
||||
|
||||
The Ollama fallback default is `qwen2.5-coder:7b` (`llm.py:67`): `[github]`
|
||||
|
||||
```python
|
||||
"default_model": os.environ.get("OLLAMA_MODEL", "qwen2.5-coder:7b"),
|
||||
```
|
||||
|
||||
Even though Graphify sends *documents* (not code) for extraction, the maintainer chose a coder/structured-output-tuned model. This is a revealed preference: the binding requirement is **structured-JSON instruction-following discipline**, not domain entity recognition. A coder/instruct model is the right axis; a NER specialist is the wrong one; frontier reasoning is overkill.
|
||||
|
||||
**Config knobs for the Ollama backend** (all `[github]` from README env-var table unless noted):
|
||||
|
||||
| Knob | How to set | What it does |
|
||||
|---|---|---|
|
||||
| `OLLAMA_MODEL` | env var | Override the `qwen2.5-coder:7b` default |
|
||||
| `OLLAMA_BASE_URL` | env var | Must end in `/v1` (e.g. `http://127.0.0.1:11434/v1`) — any other suffix produces 404s on every call |
|
||||
| `GRAPHIFY_OLLAMA_NUM_CTX` | env var | **See GOTCHA below** |
|
||||
| `GRAPHIFY_OLLAMA_KEEP_ALIVE` | env var | Minutes to keep model resident; default `30m`; set `0` to unload after each chunk |
|
||||
| `--token-budget` | CLI flag | Per-chunk input cap (tokens); pack multiple files per chunk |
|
||||
| `--max-concurrency` | CLI flag | Set 1–2 for a single local GPU |
|
||||
| `--mode deep` | CLI flag | Appends deep-inference suffix; elicits more INFERRED edges |
|
||||
|
||||
**GOTCHA — num_ctx does not propagate through graphify (verified 2026-06-04 on this project).** `[github]` Graphify posts to Ollama's OpenAI-compatible `/v1/chat/completions` endpoint. That endpoint **silently ignores** the per-request `options.num_ctx` that Graphify sends (`llm.py:506`). Proven by A/B: a POST to `/v1/chat/completions` with `num_ctx=8192` left `ollama ps` showing CONTEXT=4096; the same value through the native `/api/chat` endpoint was honoured. Therefore `GRAPHIFY_OLLAMA_NUM_CTX` has **no effect** through Graphify — context pins at Ollama's `/v1` default of 4096. At 4096 tokens, Graphify's extraction output JSON is truncated mid-response (`finish_reason=length`) and the chunk is **discarded**, producing an empty graph.
|
||||
|
||||
**Workaround (validated):** bake context into a Modelfile variant:
|
||||
|
||||
```bash
|
||||
# Create a 16 k-context variant of qwen2.5-coder:7b:
|
||||
cat <<'EOF' | ollama create qwen25-coder-7b-16k -f -
|
||||
FROM qwen2.5-coder:7b
|
||||
PARAMETER num_ctx 16384
|
||||
EOF
|
||||
OLLAMA_MODEL=qwen25-coder-7b-16k graphify extract ./docs --backend ollama
|
||||
```
|
||||
|
||||
The `/v1` endpoint honours the model's baked-in default. Non-invasive: no sudo, no systemd restart; reversible with `ollama rm`. Full investigation: `docs/memory-system/benchmark/scoring-results-2026-06-04.md`.
|
||||
|
||||
**Local patch — thinking mode disabled (applied 2026-06-04):** `reasoning_effort: "none"` was added at `llm.py:71` in the installed binary to suppress thinking output for any thinking-capable model. The original is backed up at `/tmp/graphify-bench/llm.py.orig`. This patch is a **no-op for `qwen2.5-coder:7b`** (no thinking mode) but will be **lost on `pip install --upgrade`** and would need re-applying for any future thinking-capable candidate. Needs a production path: upstream PR or a maintained local wrapper. `[github]`
|
||||
|
||||
---
|
||||
|
||||
## Purpose-built KG / entity / relation models — and why they don't drop in
|
||||
|
||||
Every purpose-built extraction model assumes **it** owns the prompt and output contract. Graphify owns both. That's the structural incompatibility.
|
||||
|
||||
| Model | Architecture | Ollama-pullable? | Follows Graphify's fixed prompt + schema? | Verdict |
|
||||
|---|---|---|---|---|
|
||||
| **SciPhi/Triplex** (Phi-3-3.8B, fine-tuned for KG triples) | Autoregressive causal LM | Yes (`ollama pull sciphi/triplex`) | No — locked to its own `{entity_types}/{predicates}` input template; emits subject–predicate–object triples, not Graphify's nodes/edges JSON | Can't drop in as-is |
|
||||
| **GLiNER** (lightweight NER) | BERT-like bidirectional encoder | No | No — NER-only (span outputs); cannot emit arbitrary JSON or follow a system prompt | Wrong architecture |
|
||||
| **REBEL / Relik** (relation extraction) | Seq2seq BART with special-token triples | No | No — emits `<triplet>/<subj>/<obj>` tokens; custom parsing required | Wrong architecture |
|
||||
|
||||
Sources: HuggingFace SciPhi/Triplex model card, `ollama.com/library/sciphi/triplex`, GitHub `urchade/GLiNER`, GitHub `Babelscape/rebel`. `[unverified claim]` — line numbers in those repos were not pinned.
|
||||
|
||||
---
|
||||
|
||||
## Is the purpose-built route worth experimenting with?
|
||||
|
||||
**User hypothesis:** if the JSON-shape mismatch is the only blocker, write a shim converting Triplex's triple output into Graphify's schema with a custom prompt. Triplex is marketed as very low-cost/fast (small 3.8B Phi-3 model).
|
||||
|
||||
What it would actually take `[speculative]`:
|
||||
|
||||
1. **A model-specific prompt path.** Triplex requires its `{entity_types}/{predicates}` template, not `_EXTRACTION_SYSTEM`. Graphify currently hard-codes one system prompt; a Triplex adapter needs to inject a different one for that model (or backend).
|
||||
2. **A model-specific output parser.** Convert subject–predicate–object triples into Graphify's `nodes` + `edges` JSON: assign `file_type` to synthesized nodes, map predicates onto Graphify's fixed relation vocabulary (or accept free-text), synthesize `confidence` since Triplex emits no `EXTRACTED`/`INFERRED`/`AMBIGUOUS` enum, and reconstruct per-file provenance fields (`source_file`, `source_location`).
|
||||
3. **A new backend/model branch in `llm.py`.** This is not a one-line patch — it is a backend adapter (prompt + parser) of moderate size.
|
||||
|
||||
**Architecturally consistent.** Graphify already has model-specific branches: the Kimi/moonshot backend sets `extra_body={"thinking": {"type": "disabled"}}` (`llm.py:461-462`) `[github]` to suppress reasoning output. A Triplex adapter follows the same pattern and could in principle be an upstream PR.
|
||||
|
||||
**Quality unknowns `[speculative]`.** Triplex targets generic NER + triples. Graphify's design wants typed nodes, a fixed relation vocabulary, EXTRACTED/INFERRED/AMBIGUOUS confidence, and per-file provenance. None of those would be native outputs from Triplex — they'd be synthesized, lowering fidelity compared to a model that follows Graphify's schema directly.
|
||||
|
||||
**Cost/speed `[speculative]`.** Triplex is plausibly faster per token (3.8B vs 7B), but `qwen2.5-coder:7b` at Q4 on a 12GB GPU is already rapid. The adapter effort isn't justified unless the default model proves too slow or too heavy for the freshness budget.
|
||||
|
||||
**Verdict `[speculative]`:** Realistic but moderate effort. Worth it **only if**: (a) `qwen2.5-coder:7b` proves too slow or too heavy for the project's freshness budget on the full vault, **and** (b) a quick Triplex spike shows acceptable triple quality on the vault fixtures. Neither condition is known yet. **PARKED — revisit after the qwen2.5-coder:7b benchmark result is in hand.**
|
||||
|
||||
**Cheap first probe:** `ollama run sciphi/triplex` on one fixture note using Triplex's native `{entity_types}/{predicates}` template. If the triples are coherent and well-typed, the adapter is worth scoping. If they are sparse or incorrectly typed, don't bother.
|
||||
|
||||
---
|
||||
|
||||
## Why qwen2.5-coder:7b is the right axis, not a domain specialist
|
||||
|
||||
To be concrete: the extractors that do better than `qwen2.5-coder:7b` at Graphify's actual task are models that are **better at structured-JSON instruction following**, not models that are better at entity recognition or triple extraction in isolation. Relevant axis: instruction-following + JSON discipline. Irrelevant axis: NER F1, triple-completeness on academic benchmarks. `[speculative]` (Based on observed failure modes: models that failed in the 2026-06-04 benchmark failed by producing invalid JSON or consuming the output budget with thinking tokens — not by extracting the wrong entities.)
|
||||
|
||||
---
|
||||
|
||||
## Pointers
|
||||
|
||||
- Full local-model investigation + the num_ctx/thinking findings: `docs/memory-system/benchmark/scoring-results-2026-06-04.md`
|
||||
- Backend config details and Ollama setup: `docs/graphify/05-local-models-and-backends.md`
|
||||
- Graphify source (installed): `~/.local/lib/python3.14/site-packages/graphify/llm.py` — anchored to 0.8.31
|
||||
|
|
@ -1,7 +1,9 @@
|
|||
# System Design
|
||||
|
||||
_Status: approved 2026-06-03; knowledge layer revised 2026-06-04 (Graphify replaces the Ruby
|
||||
tag-index CLI and the deferred QMD layer — see ADR-010). Implementation not yet started._
|
||||
tag-index CLI and the deferred QMD layer — see ADR-010); tag taxonomy and vault location locked
|
||||
2026-06-04 (six-facet taxonomy — see ADR-011; reuse ~/Documents/SecondBrain — see ADR-012).
|
||||
Implementation not yet started._
|
||||
|
||||
## Goals (what this system must do)
|
||||
|
||||
|
|
@ -39,8 +41,8 @@ forcing one tool to do both is what made every earlier design feel forced.
|
|||
│ auto-captured session/journal notes · NL semantic recall │
|
||||
│ answers "when did we…", "what was I doing yesterday" │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ KNOWLEDGE ── flat Obsidian vault (single source of truth) │
|
||||
│ notes carry summary + namespaced tags (metadata) │
|
||||
│ KNOWLEDGE ── ~/Documents/SecondBrain Obsidian vault (single SOT) │
|
||||
│ notes carry summary + six facet tags + scope/ (metadata) │
|
||||
│ + Graphify knowledge graph (local SLM over docs; AST over code) │
|
||||
│ graph queries (god nodes / query / path) · answers "how do we…",│
|
||||
│ "what do we know about X for client Y", "what relates to Y" │
|
||||
|
|
@ -72,8 +74,7 @@ filters), it is just no longer backed by a bespoke index.
|
|||
The heart of the system, and the part we build.
|
||||
|
||||
### Vault
|
||||
- **Flat markdown directory**, single source of truth, **configurable location** (NOT forced
|
||||
into `~/.claude/`; symlink if a tool insists). Browsable in Obsidian as a viewer.
|
||||
- **Flat markdown directory**, single source of truth — reuses the existing **`~/Documents/SecondBrain`** Obsidian vault (ADR-012); not forced into `~/.claude/`; symlink if a tool insists. Browsable in Obsidian as a viewer.
|
||||
- **Replaces project-local documentation**: instead of docs scattered per repo, knowledge
|
||||
lives once in the vault and is pulled into any project on demand.
|
||||
|
||||
|
|
@ -83,14 +84,20 @@ The heart of the system, and the part we build.
|
|||
summary: One line, written at creation. The router shows this so the AI can pick a
|
||||
file without opening it.
|
||||
tags:
|
||||
- tool/semrush # namespaced, nested (slash = Obsidian nested tag)
|
||||
- type/reference # listed first by convention; e.g. type/hub, type/how-to
|
||||
- client/sesame3g
|
||||
- project/website-redesign
|
||||
- domain/seo
|
||||
- tool/semrush
|
||||
- convention/api-style
|
||||
- scope/project # or scope/global
|
||||
---
|
||||
```
|
||||
- **Namespaces** are the "virtual indexes": `tool/`, `client/`, `domain/`, `convention/`,
|
||||
`scope/`. `#tool` matches all children — native prefix filtering, no folders needed.
|
||||
- **Six flat facets** (ADR-011): `type/`, `client/`, `project/`, `domain/`, `tool/`,
|
||||
`convention/` — plus `scope/`. Each facet is independent and parallel (never nested into each
|
||||
other). `#tool` matches all `tool/*` values — native Obsidian prefix filtering, no folders
|
||||
needed. Hierarchy and relationships are expressed via **hub notes** (`type/hub`),
|
||||
**wikilinks**, and **Graphify graph edges**, not nested tag paths.
|
||||
- **Two knowledge scopes** via `scope/global` vs `scope/project` (+ a `client/` tag): global =
|
||||
broadly useful tool/domain knowledge; project = how a specific client uses it. Both are
|
||||
globally queryable; the scope tag is the shortcut that avoids scanning every client's usage.
|
||||
|
|
@ -113,11 +120,12 @@ layer (ADR-010): one graph gives both structured and semantic retrieval, without
|
|||
- **Query** (via CLI and an MCP server exposing `query_graph` / `get_node` / `shortest_path`):
|
||||
ask for **god nodes first**, then scalpel down with `graphify query` / `path` / `explain`.
|
||||
Prompt the graph; don't dump the corpus into context.
|
||||
- **Metadata still matters**: the `summary` + namespaced tags remain first-class note
|
||||
- **Metadata still matters**: the `summary` + six facet tags remain first-class note
|
||||
attributes — `summary` is the human-written router hint Graphify does **not** generate, and
|
||||
the `tag/` namespaces stay useful for Obsidian filtering and as node attributes. They are
|
||||
retained even though they no longer back a bespoke index. *(How tightly metadata feeds graph
|
||||
queries is a refinement for build time.)*
|
||||
the facet namespaces (`type/`, `client/`, `project/`, `domain/`, `tool/`, `convention/`,
|
||||
`scope/`) stay useful for Obsidian filtering and as node attributes. They are retained even
|
||||
though they no longer back a bespoke index. *(How tightly metadata feeds graph queries is a
|
||||
refinement for build time.)*
|
||||
- **Source of truth rule**: markdown is authoritative; the graph (`graphify-out/`) is a
|
||||
rebuildable artifact that is **never synced** and can be deleted/rebuilt anytime
|
||||
(`graphify ... --force`).
|
||||
|
|
@ -179,6 +187,6 @@ similarity would clearly win, revisit then (the video's "only level up when it b
|
|||
| Goal | Met by |
|
||||
|------|--------|
|
||||
| 1. Thin projects | Knowledge in the vault, not repos; CLAUDE.md holds tags/pointers; on-demand `index query` |
|
||||
| 2. Cross-project/client knowledge, global vs project scopes | Flat vault + namespaced tags + `scope/` + `client/`; Graphify knowledge graph (god nodes + traversal) over it |
|
||||
| 2. Cross-project/client knowledge, global vs project scopes | `~/Documents/SecondBrain` vault + six-facet tags (`type/`/`client/`/`project/`/`domain/`/`tool/`/`convention/`) + `scope/`; Graphify knowledge graph (god nodes + traversal) over it |
|
||||
| 3. Timeline | memsearch episodic layer + session-end journal hook |
|
||||
| 4. Remote, local-fast | Markdown vault synced via git/Syncthing; disposable per-machine graphs/indexes |
|
||||
|
|
|
|||
|
|
@ -1,5 +1,7 @@
|
|||
# Architecture Decision Records
|
||||
|
||||
_Last updated: 2026-06-04_
|
||||
|
||||
A running log of decisions and *why*. Format per entry: Context · Decision · Rationale ·
|
||||
Alternatives rejected · Status. Newest decisions extend the log; supersede rather than delete.
|
||||
|
||||
|
|
@ -47,7 +49,8 @@ Alternatives rejected · Status. Newest decisions extend the log; supersede rath
|
|||
Pure-prefix path filtering via memsearch `source_prefix` (would force directories back in).
|
||||
- **Trade-off accepted**: Tags give the *human/Obsidian* free filtering, but the *AI* gets
|
||||
nothing for free from tags — we must materialize them into a queryable index (see ADR-004).
|
||||
- **Status**: Accepted.
|
||||
- **Status**: **Refined by ADR-011** (type/ and project/ namespaces added; hierarchy-vs-facets
|
||||
clarified). Core decision — flat vault, namespaced tags — stands.
|
||||
|
||||
## ADR-004 — SQLite + Sequel (Ruby) tag index as the knowledge-layer cache
|
||||
|
||||
|
|
@ -176,6 +179,122 @@ Alternatives rejected · Status. Newest decisions extend the log; supersede rath
|
|||
suffices).
|
||||
- **Status**: Accepted (to be built — see 04-build-plan.md and 06-graphify-evaluation.md).
|
||||
|
||||
## ADR-011 — Faceted tag taxonomy: six independent namespaces (refines ADR-003)
|
||||
|
||||
_Date: 2026-06-04_
|
||||
|
||||
- **Context**: ADR-003 introduced five namespaces (`tool/`, `client/`, `domain/`,
|
||||
`convention/`, `scope/`). During vault-reuse assessment (ADR-012) it became clear that (1)
|
||||
the existing SecondBrain vault uses a de-facto first-tag convention for note kind
|
||||
(research/plan/log/adr/howto) that should be made explicit and machine-queryable, and (2)
|
||||
for a freelancer working many projects per client, project identity deserves a first-class
|
||||
namespace rather than being implied by `client/` or `domain/`.
|
||||
- **Decision**: Knowledge-vault notes are classified by **six independent, flat tag facets**
|
||||
that sit side-by-side, never nested into one another:
|
||||
- `type/` — note kind: `research`, `howto`, `adr`, `hub`, `plan`, `log`, `clip`, etc.
|
||||
- `client/` — which client
|
||||
- `project/` — which project (first-class; a freelancer's projects are the primary unit of
|
||||
work)
|
||||
- `domain/` — knowledge domain / topic area
|
||||
- `tool/` — tool-specific knowledge
|
||||
- `convention/` — conventions
|
||||
- …plus `scope/global` or `scope/project` (retained from ADR-003)
|
||||
|
||||
Hierarchy and relationships are expressed via **hub notes** (`type/hub`), **wikilinks**, and
|
||||
**Graphify knowledge-graph edges** — NOT via nested tag paths.
|
||||
|
||||
By convention `type/` is listed **first** in frontmatter, preserving the SecondBrain vault's
|
||||
existing type-first ordering habit and making the note kind immediately visible.
|
||||
- **Rationale**: The vault is flat — hierarchy is not expressed through folder paths or tag
|
||||
nesting. The user's reality is many-to-many (many projects per client, knowledge domains
|
||||
spanning clients), which a single-parent tree models badly and forces false hierarchy. A
|
||||
project hub note links out to both its `client/` and relevant `domain/` tags rather than
|
||||
being buried under either. Per-type `_templates` will be provided for **core types only**
|
||||
(research, howto, adr, hub); the long tail stays freeform until a pattern earns a template.
|
||||
Consistent per-type structure also improves Graphify's local-SLM extraction reliability.
|
||||
- **Alternatives rejected**: Hierarchical nesting in the style of John Conneely's
|
||||
`domain/{product}/{project}.md` folder structure (from the youngleaders.tech article "How I
|
||||
finally sorted my Claude Code memory" — **secondary/interview-grade source, not verified
|
||||
against primary implementation**). Rejected because: (1) the vault is flat — hierarchy is
|
||||
not expressed through folder paths; (2) the user's many-to-many reality maps badly onto a
|
||||
single-parent folder tree and forces false hierarchy; (3) nesting one facet through another
|
||||
(e.g. `domain/client/project`) creates Law-of-Demeter-style traversal coupling. Conneely's
|
||||
structure was the inspiration but diverges here on hierarchy-vs-facets. Faceted parallel tags
|
||||
are the flat-vault analogue of what the Graphify graph already does with edges, so they
|
||||
compose naturally with the chosen knowledge layer.
|
||||
- **Status**: Accepted (supersedes the namespace list in ADR-003; core flat-vault +
|
||||
namespaced-tags decision stands).
|
||||
|
||||
## ADR-012 — Reuse the existing SecondBrain vault as the knowledge vault
|
||||
|
||||
_Date: 2026-06-04_
|
||||
|
||||
- **Context**: The design called for a flat markdown vault as the semantic knowledge layer
|
||||
(ADR-003/008/010). The question was whether to stand up a new `~/brain` vault from scratch
|
||||
or adopt the existing `~/Documents/SecondBrain` vault.
|
||||
- **Decision**: **Adopt `~/Documents/SecondBrain`** as the knowledge vault rather than
|
||||
creating a new vault.
|
||||
- **Rationale**: Assessment found the SecondBrain vault is already flat (all notes at root,
|
||||
only a `_templates/` exception — exactly what the design permits), already articulates the
|
||||
correct "durable knowledge, not working memory" role in its `CLAUDE.md` and
|
||||
`vault-conventions.md`, and contains ~20 real notes. It also includes two patterns that
|
||||
**improve on the current design** and should be adopted:
|
||||
1. `vault-conventions.md`'s "act without being asked" section specifying *when* the AI
|
||||
should proactively query the vault — a behavioral spec the cc-os docs lacked.
|
||||
2. Project-config hub notes with a tag-inference table (auto-tag by path pattern) that
|
||||
operationalizes *how* to tag a note from a given project.
|
||||
- **Adaptations required (migration cost)**:
|
||||
- Add `summary:` frontmatter to existing notes.
|
||||
- Migrate flat unnamespaced tags to the six-facet namespaced form (per ADR-011).
|
||||
- Add `scope/global` or `scope/project` to each note.
|
||||
- Initialize git in the vault (no `.git` exists yet — required by ADR-008's sync strategy).
|
||||
- Replace the vault's `~/.claude/scripts/vault_search.rb` reference (script does not exist)
|
||||
with `graphify query` (ADR-010).
|
||||
These are mechanical schema migrations, not structural rework.
|
||||
- **Alternatives rejected**: Starting fresh with a new `~/brain` vault. Rejected because the
|
||||
hardest design decision — flat structure, durable-knowledge-only role, governance philosophy
|
||||
— is already made and practiced in SecondBrain. The improved behavioral patterns
|
||||
(proactive-query spec, tag-inference table) and the existing notes are worth preserving; the
|
||||
remaining work is mechanical migration.
|
||||
- **Status**: Accepted.
|
||||
|
||||
## ADR-013 — Build-first / migrate-incrementally (build-order inversion)
|
||||
|
||||
_Date: 2026-06-04_
|
||||
|
||||
- **Context**: The build runbook (`05-implementation-process.md`) originally front-loaded bulk
|
||||
vault migration as Step 1 — migrating all ~20 existing SecondBrain notes and all projects to
|
||||
the ADR-011 six-facet taxonomy before the system existed to validate them. This committed to
|
||||
a schema and workflow (the tag taxonomy from ADR-011, the vault-reuse choice from ADR-012,
|
||||
and Graphify extraction behavior) before any end-to-end path had been exercised. The risk:
|
||||
locking in an approach that fails at scale, with no feedback loop until the entire vault has
|
||||
been touched.
|
||||
- **Decision**: **Invert the build order.** The full system is built and validated against a
|
||||
small **5–10 note fixture set** first. Bulk vault migration is deferred to the final stage.
|
||||
The first real-data validation uses **one small project that contains both code AND
|
||||
documents**, exercising both the local-SLM doc-extraction path and the tree-sitter code path
|
||||
in the same run. After that single project validates end-to-end, remaining projects are
|
||||
onboarded **one at a time** with an observe-and-adjust step between each.
|
||||
- **Rationale**: Validates the ADR-011 taxonomy and ADR-012 vault conventions against the real
|
||||
Graphify extraction pipeline before the entire vault is committed. The first mixed code+docs
|
||||
project surfaces both extraction paths (SLM for docs, tree-sitter for code) early, when
|
||||
corrections are cheap. Per-project rollout keeps the blast radius of any schema or workflow
|
||||
correction small; each project is an opportunity to observe and adjust rather than discover
|
||||
problems across 20 notes at once. This is consistent with the "markdown-as-truth, indexes are
|
||||
disposable" principle (ADR-008): the vault notes are durable, but the extraction schema should
|
||||
be validated before it shapes all of them.
|
||||
- **Alternatives rejected**:
|
||||
- **Keep migration-first (status quo)**: Front-loads all ~20 notes and all projects before
|
||||
any end-to-end validation exists. Commits to ADR-011's taxonomy and ADR-012's migration
|
||||
steps against the full vault without a feedback loop — exactly the gap this decision closes.
|
||||
- **Big-bang migrate everything after build**: Build against fixtures, then migrate all notes
|
||||
and all projects in one batch at the end. Avoids the pre-build commitment problem but still
|
||||
risks a single large irreversible migration with no observe-and-adjust loop between units.
|
||||
Per-project rollout with intermediate checkpoints is strictly safer.
|
||||
- **Cross-references**: ADR-011 (six-facet tag taxonomy — the schema being validated);
|
||||
ADR-012 (SecondBrain vault reuse — the migration steps this order defers).
|
||||
- **Status**: Accepted (updates `05-implementation-process.md` build order).
|
||||
|
||||
## Rejected tools (summary)
|
||||
|
||||
| Tool | Why rejected for our use |
|
||||
|
|
|
|||
|
|
@ -1,5 +1,7 @@
|
|||
# Build Plan
|
||||
|
||||
_Last updated: 2026-06-04_
|
||||
|
||||
How a human builds this system, step by step, and answers to the operational questions:
|
||||
which scripts and hooks, how the AI knows when to write and what conventions to follow, how and
|
||||
when it queries, the CRUD hooks, and how it's packaged as a global Claude Code plugin with
|
||||
|
|
@ -14,6 +16,12 @@ vault and free AST-based code graphs per project. The deferred QMD semantic laye
|
|||
is also skipped — Graphify covers it without vectors. See `06-graphify-evaluation.md` for
|
||||
the full rationale.
|
||||
|
||||
**Architecture decision (2026-06-04):** Build order inverted (ADR-013). Build and validate
|
||||
the full system against a small 5–10 note fixture set first; defer bulk vault migration to last.
|
||||
Validate end-to-end on one small pilot project containing both code and documents before
|
||||
onboarding any others. Onboard remaining projects one at a time, with observe-and-adjust between
|
||||
each. Steps 2d and 2e below are updated accordingly.
|
||||
|
||||
---
|
||||
|
||||
## Part A — Build order (human builder's path)
|
||||
|
|
@ -22,8 +30,9 @@ Build bottom-up: the vault and Graphify first (usable standalone), then the hook
|
|||
plugin that packages it.
|
||||
|
||||
### Step 1 — Vault skeleton & conventions
|
||||
- Decide the vault location (default: a synced home dir, e.g. `~/brain`; symlink into
|
||||
`~/.claude/memory` only if a tool insists). **Vault is the single source of truth.**
|
||||
- Vault location is settled by ADR-012: `~/Documents/SecondBrain` (the existing Obsidian
|
||||
vault). No new vault is created; symlink into `~/.claude/memory` only if a tool insists.
|
||||
**Vault is the single source of truth.**
|
||||
- Write `CONVENTIONS.md` in the vault: the frontmatter contract. The required fields are
|
||||
`summary` (one-line, author-written — this is the human-readable router hint Graphify
|
||||
does not generate) and `scope` (`scope/global` or `scope/project`). Tags are now
|
||||
|
|
@ -61,49 +70,68 @@ Available models (as of 2026-06-03, in order of interest):
|
|||
- `qwen3.5:2b` — 2.7 GB, smallest, good fallback if VRAM is constrained
|
||||
- `gemma4:e4b` — 9.6 GB, more capable, slower
|
||||
|
||||
#### 2c — Model benchmarking (before committing to a model)
|
||||
#### 2c — Claude reference-set benchmark (THE GATE, before committing to a local model)
|
||||
|
||||
Run a small extraction test across all local models plus the three Claude API models. The goal
|
||||
is to find the best speed/accuracy tradeoff for entity+relationship extraction from vault notes.
|
||||
This step produces the **gold-standard reference set** — one structured extraction output per
|
||||
fixture note per Claude tier. It does **not** choose the final extraction model, and it does
|
||||
**not** measure speed.
|
||||
|
||||
**Test set:** 5–10 representative vault notes spanning different note types (tool note, client
|
||||
note, convention note, domain note). Include one note that is dense with relationships.
|
||||
**Input:** the 5–10 fixture notes selected in Step 1 (from the runbook).
|
||||
|
||||
**What to measure per model:**
|
||||
1. Extraction speed (tokens/sec or wall-clock time per note)
|
||||
2. Entity quality: are the right concepts extracted? Any hallucinated entities?
|
||||
3. Relationship quality: are edges plausible and correctly typed? Missing relationships?
|
||||
4. Confidence tag accuracy: are `INFERRED` vs `AMBIGUOUS` edges appropriately flagged?
|
||||
**What to run:** dispatch one Claude Code subagent per tier (Claude-tiers only — Ollama models
|
||||
are not reachable in this environment):
|
||||
|
||||
**Models to test:**
|
||||
| Model | Backend | Notes |
|
||||
| Tier | Model ID | Role |
|
||||
|---|---|---|
|
||||
| `gemma4:e2b` | Ollama local | Primary candidate — fast, 7.2 GB |
|
||||
| `qwen3.5:2b` | Ollama local | Smallest, fastest |
|
||||
| `gemma4:e4b` | Ollama local | Highest local quality |
|
||||
| `claude-haiku-4-5` | Claude API | Baseline API option |
|
||||
| `claude-sonnet-4-6` | Claude API | Mid-tier reference |
|
||||
| `claude-opus-4-8` | Claude API | **Gold standard** — judge quality against this |
|
||||
| Haiku | `claude-haiku-4-5` | Lightweight reference |
|
||||
| Sonnet | `claude-sonnet-4-6` | Mid-tier reference |
|
||||
| Opus | `claude-opus-4-8` | **Gold standard** (scoring rubric) |
|
||||
|
||||
Run using subagents in Claude Code: dispatch one agent per model, each extracting the same
|
||||
test set, return structured JSON of entities+relationships. Review god-node quality in
|
||||
`GRAPH_REPORT.md` after each run. Use Opus output as the scoring rubric for the local models.
|
||||
Each subagent receives only the fixture note text plus the shared extraction spec — no design
|
||||
docs, no project context (fairness contract). See ADR-013.
|
||||
|
||||
**Decision rule:** Choose the fastest local model whose entity/relationship quality is
|
||||
"close enough" to Opus (subjective; likely Gemma4:e2b or Gemma4:e4b based on the description).
|
||||
API models are a fallback option for high-stakes notes, not the default.
|
||||
**Metrics — quality only (wall-clock speed is explicitly out of scope here):**
|
||||
1. Entity correctness — right concepts extracted, no hallucinated entities
|
||||
2. Relationship plausibility and typing — edges plausible, correctly typed, no missing key edges
|
||||
3. Confidence-tag accuracy — `INFERRED` vs `AMBIGUOUS` applied appropriately
|
||||
|
||||
**Deliverables** (produced by this step):
|
||||
- Dispatch prompt (copy/paste-able): `docs/memory-system/benchmark/dispatch-prompt.md`
|
||||
- Shared extraction schema: `docs/memory-system/benchmark/extraction-spec.md`
|
||||
- Per-tier outputs: `docs/memory-system/benchmark/reference-outputs/<note-slug>.<tier>.md`
|
||||
|
||||
Opus output is the rubric. **Deferred later step:** local Ollama models (gemma4:e2b, qwen3.5:2b,
|
||||
gemma4:e4b) are timed AND scored against these references — that scoring run is where speed
|
||||
re-enters and the final model is chosen. Do not hardcode a model before that run completes.
|
||||
|
||||
**Status (2026-06-04): EXECUTED.** 6 cross-domain fixtures × 3 Claude tiers = 18 reference
|
||||
fragments generated in `benchmark/reference-outputs/`. Run as-is (no vault frontmatter
|
||||
modification); verified clean. Fixtures listed in `benchmark/dispatch-prompt.md`.
|
||||
|
||||
Authoritative detail lives in `docs/memory-system/05-implementation-process.md` §2c. Local-model gut-check done (2026-06-04): `gemma4:e4b` is the candidate; GPU fix pending reboot; Graphify owns the ollama call — see `docs/memory-system/benchmark/local-llm-findings-2026-06-04.md`.
|
||||
|
||||
#### 2d — Initial fixture graph build (ADR-013: small-first)
|
||||
Run the initial build against the small fixture set (5–10 notes from Step 1/2c), not the
|
||||
full vault. The bulk vault build is deferred to after the system is validated end-to-end.
|
||||
|
||||
#### 2d — Initial vault graph build
|
||||
```bash
|
||||
graphify extract --path ~/brain --backend ollama --model gemma4:e2b \
|
||||
graphify extract --path ~/Documents/SecondBrain --backend ollama --model gemma4:e2b \
|
||||
--token-budget 512 --max-concurrency 2
|
||||
```
|
||||
Tune `--token-budget` (semantic chunk size) and `--max-concurrency` based on VRAM headroom.
|
||||
Review `GRAPH_REPORT.md` — check god nodes make sense (they should be your most-connected
|
||||
tools, clients, and domain concepts).
|
||||
|
||||
**Full vault migration** (the `~/Documents/SecondBrain` build above run over all notes) is the final step —
|
||||
deferred to after end-to-end validation on the pilot project. Do not bulk-migrate the vault
|
||||
until the system is verified working on the fixture set and pilot project.
|
||||
|
||||
#### 2e — Per-project code graphs (free, no model needed)
|
||||
For each client project:
|
||||
**ADR-013 order:** start with ONE pilot project that contains both code and documents; validate
|
||||
end-to-end before onboarding others. Onboard remaining projects one at a time, with
|
||||
observe-and-adjust between each. Do not batch all projects at once.
|
||||
|
||||
For the pilot project (and each subsequent project, one at a time):
|
||||
```bash
|
||||
graphify extract --path ~/projects/<client>/<project> --no-docs
|
||||
```
|
||||
|
|
@ -257,7 +285,8 @@ global install keeps conventions a single source of truth.
|
|||
|
||||
## Open questions
|
||||
|
||||
1. **Vault location** — `~/brain` (synced home dir)? Symlink into `~/.claude/memory`?
|
||||
1. **Vault location** — Settled by ADR-012: `~/Documents/SecondBrain`. Symlink into
|
||||
`~/.claude/memory` only if a tool requires it.
|
||||
2. **Sync mechanism** — git (versioned, hourly) vs Syncthing (continuous)?
|
||||
3. **Stale rebuild threshold** — how many days before SessionStart triggers `--force`?
|
||||
7 days is the starting guess; tune after observing drift in practice.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,402 @@
|
|||
# Implementation Process
|
||||
|
||||
_Last updated: 2026-06-05_ | _Status: Steps 2a/2b/2d executed — toolchain installed, context fix applied, vault graph built; model selected (qwen2.5-coder:7b); Step 2c (reference set) previously complete. Step 2 fully executed._
|
||||
|
||||
This document distills `04-build-plan.md` into a concrete, staged build process and folds in
|
||||
two locked decisions: **ADR-011** (faceted six-namespace taxonomy) and **ADR-012** (reuse the
|
||||
existing `~/Documents/SecondBrain` vault rather than creating a new one). Read `04-build-plan.md`
|
||||
for the underlying rationale, query/CRUD conventions, and plugin internals; read
|
||||
`03-architecture-decisions.md` for the decision log. This document is the runbook.
|
||||
|
||||
> This is still a build process outline, not a detailed implementation plan. **Step 2 is fully
|
||||
> executed as of 2026-06-05**: toolchain installed (2a), context-baking approach confirmed (2b),
|
||||
> reference set generated (2c), vault graph built (2d). Selected model: **qwen2.5-coder:7b**
|
||||
> (Modelfile variant `qwen25-coder-7b-16k`, num_ctx 16384). The next phase is Step 3 (Hooks).
|
||||
> Most open questions in Steps 3–6 can be defaulted at build time.
|
||||
|
||||
---
|
||||
|
||||
## Build order
|
||||
|
||||
Bottom-up: fixture selection first (feeds the Step 2c gate), then Graphify setup, then hooks, then plugin. Bulk vault migration is deferred to after the system is validated end-to-end.
|
||||
|
||||
---
|
||||
|
||||
## Step 1 — Select benchmark fixtures (and defer bulk migration)
|
||||
|
||||
**Per ADR-012:** `~/Documents/SecondBrain` is the vault. The vault is already flat, already
|
||||
scoped to durable knowledge, and already articulates the correct governance philosophy.
|
||||
|
||||
**Bulk vault migration is DEFERRED.** Do not migrate all notes now. The only pre-build activity
|
||||
in this step is selecting a small fixture set that feeds the Step 2c benchmark gate. Everything
|
||||
else — tagging ~20 existing notes, updating governance docs, initializing git, fixing broken
|
||||
references — happens after the system is validated end-to-end.
|
||||
|
||||
> **Four sequencing concepts to keep distinct — they are not interchangeable:**
|
||||
>
|
||||
> 1. **Benchmark fixtures** — 5–10 notes selected NOW, in this step. They feed the Step 2c
|
||||
> extraction benchmark gate. They are living fixtures, kept and reused throughout development.
|
||||
> 2. **Bulk vault migration** — converting all ~20 existing notes to the six-facet tag schema,
|
||||
> updating governance docs, initializing git. DEFERRED to last, after the system is fully
|
||||
> validated.
|
||||
> 3. **Initial validation project** — the first real end-to-end test after the system is built:
|
||||
> one small project that contains BOTH code AND documents. POST-BUILD, before any rollout.
|
||||
> 4. **Project-by-project rollout** — onboarding remaining projects one at a time, observing
|
||||
> and adjusting between each. Follows the initial validation project.
|
||||
|
||||
### 1a — Select fixture notes
|
||||
|
||||
**Status (2026-06-04): DONE.** 6 cross-domain fixtures selected and listed in
|
||||
`benchmark/dispatch-prompt.md`. Run as-is (no vault frontmatter modification per the
|
||||
methodology decision — fixtures used exactly as they exist in the vault).
|
||||
|
||||
Choose 5–10 notes from `~/Documents/SecondBrain` with deliberate variety. The selection must
|
||||
include at minimum:
|
||||
|
||||
- A **tool note** (e.g., a note about a CLI tool or SaaS product)
|
||||
- A **client/project note** (scoped, not global)
|
||||
- A **convention note** (a `type/convention` or workflow note)
|
||||
- A **domain note** (a topic or subject-area note)
|
||||
- At least one **relationship-dense note** — a note where several concepts interrelate and
|
||||
Graphify should emit multiple typed edges and confidence tags
|
||||
|
||||
These notes must be usable as-is by a subagent that receives only the note text plus the shared
|
||||
extraction spec (see Step 2c). Apply `summary:` frontmatter and the six-facet tag schema to
|
||||
each fixture note now; do not wait for bulk migration.
|
||||
|
||||
**Frontmatter contract** (required for fixture notes immediately; apply to all notes at bulk
|
||||
migration time):
|
||||
|
||||
```yaml
|
||||
---
|
||||
summary: "One-line, human-written router hint (required; Graphify does not generate this)"
|
||||
tags:
|
||||
- type/<kind> # listed first by convention
|
||||
- client/<name> # as applicable
|
||||
- project/<name> # as applicable — first-class for a freelancer
|
||||
- domain/<topic> # as applicable
|
||||
- tool/<name> # as applicable
|
||||
- convention/<name> # as applicable
|
||||
- scope/global # or scope/project
|
||||
---
|
||||
```
|
||||
|
||||
`type/` listed first preserves the vault's existing type-first ordering habit and makes note
|
||||
kind immediately visible.
|
||||
|
||||
### 1b — Deferred bulk migration work (do not start until post-validation)
|
||||
|
||||
The following items are deferred to after the system is validated end-to-end on fixtures and
|
||||
on the initial validation project (see §Post-build sequence below):
|
||||
|
||||
- **Initialize git:** `git init ~/Documents/SecondBrain` (enables ADR-008 sync) — before the first commit, add `graphify-out/` to the vault's `.gitignore`: `~/Documents/SecondBrain/graphify-out/cache/` (AST/tree-sitter parse cache) currently sits inside the vault and must stay untracked (ADR-008 — indexes are disposable/excluded from version control)
|
||||
- **Update vault governance notes:** `CLAUDE.md` (vault path refs, Graphify commands,
|
||||
ADR-011 namespace list), `vault-conventions.md` (six-facet tag list; preserve the
|
||||
"act without being asked" proactive-retrieval section), and project-config hub note(s)
|
||||
(update tag-inference table to namespaced form, e.g. `semrush-work → tool/semrush`)
|
||||
- **Migrate remaining notes (~20):** add `summary:` frontmatter and convert unnamespaced tags
|
||||
to six-facet form (e.g. `research → type/research`, `pest-control → domain/pest-control`,
|
||||
`semrush → tool/semrush`). Add `scope/global` or `scope/project` to each note.
|
||||
- **Create `_templates` for core note types:** `research`, `howto`, `adr`, `hub`. The long
|
||||
tail stays freeform until a pattern earns a template (per ADR-011).
|
||||
- **Fix the broken vault search reference:** `vault-conventions.md` (or `CLAUDE.md`) currently
|
||||
references `~/.claude/scripts/vault_search.rb`, which does not exist. Replace every reference
|
||||
with `graphify query` (per ADR-010).
|
||||
|
||||
### Post-build sequence (after Steps 2–6 are complete)
|
||||
|
||||
After the system is built and validated on fixtures:
|
||||
|
||||
**First:** Validate end-to-end on **one small project that contains BOTH code AND documents.**
|
||||
This project is the initial validation target — not a production migration. It exercises both
|
||||
extraction paths simultaneously: vault notes use the local-SLM extraction path; project code
|
||||
uses the tree-sitter AST path. These are different extraction paths, which is precisely why the
|
||||
first validation project must contain both. Observe how the two paths compose before proceeding.
|
||||
|
||||
**Then:** Onboard remaining projects **one at a time.** Observe extraction quality, god-node
|
||||
shape, and hook behavior between each project before onboarding the next. Adjust configuration
|
||||
as needed.
|
||||
|
||||
**Finally:** Execute bulk vault migration (items in 1b above) and run full vault graph build.
|
||||
|
||||
**Migration-unit granularity open question:** the right unit — a whole project repo, a
|
||||
vault-note cluster, a client boundary — is not yet established. The first validation project
|
||||
("one small project with both code and documents") is the anchor; refine the granularity
|
||||
definition after observing that first real migration.
|
||||
|
||||
---
|
||||
|
||||
## Step 2 — Graphify + Ollama setup (CRITICAL PATH)
|
||||
|
||||
This step is the genuine gate. All hook behavior, model selection, and extraction tuning depend
|
||||
on the benchmark in 2c. Do not skip or defer 2c.
|
||||
|
||||
### 2a — Install and verify Graphify
|
||||
|
||||
**Status (2026-06-05): EXECUTED.** graphify 0.8.31 installed at `~/.local/bin/graphify`; `graphify --version` → `graphify 0.8.31`. ollama 0.30.3, systemd service on 127.0.0.1:11434; GPU RTX 3060 12GB VRAM confirmed; all extraction ran 100% on GPU (no CPU spill per `ollama ps`). [verified: local shell, 2026-06-05]
|
||||
|
||||
The PyPI package name has a double-y — this is the correct install command:
|
||||
|
||||
```bash
|
||||
pip install graphifyy
|
||||
```
|
||||
|
||||
Verify: `graphify --version`
|
||||
|
||||
### 2b — Configure Ollama
|
||||
|
||||
**Status (2026-06-05): EXECUTED — but the env-var approach was superseded.** `GRAPHIFY_OLLAMA_NUM_CTX` does NOT propagate through graphify's ollama `/v1/chat/completions` endpoint: ollama's OpenAI-compatible `/v1` endpoint silently ignores per-request `options.num_ctx`, pinning context at the model's baked default (4096 by default). At 4096, graphify's extraction output is truncated mid-response and the chunk is discarded — producing an empty graph. [verified: local A/B test, 2026-06-05]
|
||||
|
||||
**SUPERSEDED APPROACH — do not use `GRAPHIFY_OLLAMA_NUM_CTX`:**
|
||||
|
||||
```bash
|
||||
# This env var has NO EFFECT through graphify's /v1 path — do not set
|
||||
# GRAPHIFY_OLLAMA_NUM_CTX=8192
|
||||
```
|
||||
|
||||
**CORRECT APPROACH — bake context into a Modelfile variant:**
|
||||
|
||||
```bash
|
||||
# Create a Modelfile variant with the desired context baked in:
|
||||
# FROM <model>
|
||||
# PARAMETER num_ctx 16384
|
||||
# ollama create <name>-16k -f Modelfile
|
||||
```
|
||||
|
||||
The `/v1` endpoint DOES honor the model's baked default. Verified `ollama ps` CONTEXT=16384, 100% GPU. This is the same lever the vault build (Step 2d) and plugin (Step 6) must use. Non-invasive: no sudo, no systemd restart, no shell-profile edit; reversible via `ollama rm`.
|
||||
|
||||
Still set in shell profile (these remain valid):
|
||||
|
||||
```bash
|
||||
OLLAMA_FLASH_ATTENTION=1 # 30–50% VRAM savings on KV cache — always set
|
||||
GRAPHIFY_OLLAMA_KEEP_ALIVE=5 # set when packaging in Step 6
|
||||
```
|
||||
|
||||
Two additional operational requirements confirmed: `OLLAMA_BASE_URL` must end in `/v1` (e.g., `http://127.0.0.1:11434/v1`) or every graphify call 404s; `OLLAMA_API_KEY` must be set to any non-empty value unless the host is loopback.
|
||||
|
||||
Verify context allocation after the first extraction call: `ollama ps` shows allocated context.
|
||||
|
||||
### 2c — Claude reference-set benchmark (THE GATE)
|
||||
|
||||
**Status (2026-06-04): EXECUTED (prior round).** 6 cross-domain fixtures × 3 Claude tiers = 18 reference fragments generated in `benchmark/reference-outputs/`. Run as-is (no vault frontmatter modification); verified clean. Gate is passed — Ollama model scoring is unblocked. Full local-model scoring (task 3.4) and model selection (task 3.6) are also complete — see `benchmark/scoring-results-2026-06-04.md`.
|
||||
|
||||
Produce a reference set of Graphify-shaped extraction outputs before committing to any local
|
||||
Ollama model. Use the 5–10 fixture notes selected in Step 1a as the input set.
|
||||
|
||||
**What this step produces — and what it does NOT decide:**
|
||||
|
||||
This step produces the **gold-standard reference set**: one structured extraction output per
|
||||
fixture note per Claude tier. The reference set is the scoring rubric against which local Ollama
|
||||
models are scored in a later step. This step does NOT choose the final extraction model —
|
||||
local-model selection happens when Ollama models are timed and scored against these references.
|
||||
|
||||
**Why Claude-only here:** local Ollama models cannot be run in this benchmarking environment.
|
||||
Only Claude tiers are reachable via dispatched subagents. The Claude outputs serve as quality
|
||||
anchors; Ollama speed and quality are measured separately, against these anchors.
|
||||
|
||||
**Dispatch one Claude Code subagent per tier:**
|
||||
|
||||
| Tier | Model ID | Role |
|
||||
|---|---|---|
|
||||
| Haiku | `claude-haiku-4-5` | Lightweight reference |
|
||||
| Sonnet | `claude-sonnet-4-6` | Mid-tier reference |
|
||||
| Opus | `claude-opus-4-8` | **Gold standard** |
|
||||
|
||||
Each subagent receives only the fixture note text plus the shared extraction spec at
|
||||
`docs/memory-system/benchmark/extraction-spec.md`. It must not read `CLAUDE.md`, design docs,
|
||||
or pull any project context — each subagent operates with the note and spec alone (fairness
|
||||
contract).
|
||||
|
||||
Each subagent emits a Graphify-shaped structured fragment containing:
|
||||
- Extracted entities (named concepts, tools, people, projects)
|
||||
- Typed relationships between entities
|
||||
- Confidence tags: `INFERRED` (inferred but not stated) and `AMBIGUOUS` (could be interpreted
|
||||
multiple ways) applied where appropriate
|
||||
|
||||
**Metrics — quality only (wall-clock speed is explicitly out of scope here):**
|
||||
|
||||
Wall-clock timing is untrackable across dispatched subagents and is not measured in this step.
|
||||
Speed re-enters the picture in the later Ollama-scoring step, where local models are timed
|
||||
against these reference outputs. For this step, measure quality only:
|
||||
|
||||
1. **Entity correctness** — right concepts extracted, no hallucinated entities
|
||||
2. **Relationship plausibility and typing** — edges plausible, correctly typed, no missing key edges
|
||||
3. **Confidence-tag accuracy** — `INFERRED` vs `AMBIGUOUS` applied appropriately
|
||||
|
||||
**Deliverable files:**
|
||||
|
||||
- Dispatch prompt (copy/paste-able): `docs/memory-system/benchmark/dispatch-prompt.md`
|
||||
- Shared extraction schema: `docs/memory-system/benchmark/extraction-spec.md`
|
||||
- Per-model outputs: `docs/memory-system/benchmark/reference-outputs/<note-slug>.<tier>.md`
|
||||
where `tier` is one of `haiku`, `sonnet`, or `opus`
|
||||
|
||||
`claude-opus-4-8` output is the gold standard. When Ollama models are benchmarked later, their
|
||||
outputs are scored by how closely they match the Opus reference for each fixture note.
|
||||
|
||||
### 2d — Build the initial vault graph
|
||||
|
||||
**Status (2026-06-05): EXECUTED.** Full vault graph built with qwen2.5-coder:7b (Modelfile variant `qwen25-coder-7b-16k`, num_ctx 16384). Result: **57 nodes / 43 edges / 15 communities**. God-nodes are plausible: Speed-to-Lead is the dominant hub, consistent with the vault's current content emphasis. [verified: local shell, 2026-06-05]
|
||||
|
||||
Command used (deviates from the template below — actual values recorded for reference):
|
||||
|
||||
```bash
|
||||
graphify extract ~/Documents/SecondBrain \
|
||||
--backend ollama --model qwen25-coder-7b-16k \
|
||||
--max-concurrency 1 --token-budget 4000 \
|
||||
--exclude .obsidian --out /tmp/graphify-bench/vault-graph
|
||||
```
|
||||
|
||||
Note: `--max-concurrency 1` (not 2 — untested at 2 on 12GB GPU); `--token-budget 4000` (not 512 — fits the 16k context comfortably). Template for future runs:
|
||||
|
||||
```bash
|
||||
graphify extract --path ~/Documents/SecondBrain \
|
||||
--backend ollama --model qwen25-coder-7b-16k \
|
||||
--token-budget 4000 --max-concurrency 1 --exclude .obsidian
|
||||
```
|
||||
|
||||
Review `GRAPH_REPORT.md`. God nodes should be the most-connected tools, clients, and domain
|
||||
concepts — confirm they make sense given the vault's content.
|
||||
|
||||
### 2e — Per-project code graphs (free; no model)
|
||||
|
||||
For each active client project:
|
||||
|
||||
```bash
|
||||
graphify extract --path ~/projects/<client>/<project> --no-docs
|
||||
```
|
||||
|
||||
`--no-docs` runs only tree-sitter AST — zero token cost. Use `--update` on subsequent runs;
|
||||
use `--force` when files have been deleted (to clear stale nodes). Keep each project's
|
||||
`graphify-out/` alongside the project; do not merge client projects into one graph.
|
||||
|
||||
---
|
||||
|
||||
## Step 3 — Hooks
|
||||
|
||||
Thin shell wrappers around Graphify. The logic lives in Graphify; the hooks only invoke it.
|
||||
|
||||
| Hook | Trigger | Action |
|
||||
|---|---|---|
|
||||
| **PostToolUse** | AI writes/edits a vault `.md` | `graphify update --file <path>` |
|
||||
| **SessionStart** | Session opens | Stale check → `--force` rebuild if needed → inject context |
|
||||
| **SessionEnd** | Session closes | Append dated journal note to vault |
|
||||
|
||||
**Stale-check mechanism (SessionStart):** read the mtime of
|
||||
`~/.cache/graphify/vault-rebuild.stamp`. If older than N days (7 is the starting guess — tune
|
||||
after observing drift), run `graphify --force` on the vault and write a new stamp. Then inject:
|
||||
god-node summary from `GRAPH_REPORT.md` + `convention/*` note summaries + journal pointer.
|
||||
|
||||
**Note on `--update` vs `--force`:** `graphify update --file` does not prune deleted nodes.
|
||||
Stale/ghost nodes accumulate from manual deletes and renames. The periodic `--force` rebuild
|
||||
triggered by the stale check is the mitigation — it rebuilds the graph clean.
|
||||
|
||||
---
|
||||
|
||||
## Step 4 — Episodic layer (memsearch)
|
||||
|
||||
Install memsearch for time-anchored "what happened / what was I working on" queries. Graphify
|
||||
does **not** replace this — they serve different query patterns (knowledge graph vs timeline
|
||||
recall).
|
||||
|
||||
```
|
||||
/plugin marketplace add zilliztech/memsearch
|
||||
plugin install memsearch
|
||||
```
|
||||
|
||||
Verify daily memory files appear after a few conversations. The open question of whether
|
||||
memsearch indexes SessionEnd journal notes or only its own auto-capture is deferred to build
|
||||
time (see Open questions §7).
|
||||
|
||||
---
|
||||
|
||||
## Step 5 — Vault sync
|
||||
|
||||
Sync the markdown vault to the VPS. Pick one:
|
||||
|
||||
- **git** — versioned, hourly push/pull via cron; explicit audit trail.
|
||||
- **Syncthing** — continuous, bidirectional, zero-thought after setup.
|
||||
|
||||
**What to sync:** the vault only (`~/Documents/SecondBrain`).
|
||||
|
||||
**What NOT to sync:** `graphify-out/` directories, Milvus Lite caches, Ollama models. These are
|
||||
disposable — rebuilt per machine from the vault (markdown is the single source of truth, per
|
||||
ADR-008).
|
||||
|
||||
---
|
||||
|
||||
## Step 6 — Package as a global Claude Code plugin
|
||||
|
||||
One global install so every project and machine shares the same vault conventions, hooks, and
|
||||
Graphify config.
|
||||
|
||||
**Plugin contents:**
|
||||
|
||||
- **Hooks** registered in settings: SessionStart, PostToolUse, SessionEnd — shell wrappers from
|
||||
Step 3, parameterized by vault path and Graphify output dir.
|
||||
- **Graphify CLI on PATH** — the AI calls `graphify query`, `graphify path`, `graphify explain`
|
||||
via the Bash tool. No server process per graph; project-specific graphs are queried with
|
||||
`--graph <project-root>/graphify-out/graph.json`.
|
||||
- **Skills** (carry the know-how to the model):
|
||||
- `memory-write` — when to record evergreen knowledge, frontmatter contract, scope rule, vault not repo.
|
||||
- `memory-query` — `graphify query` vs memsearch; god-node discipline; `--budget`/`--dfs`; cross-client lookups; progressive disclosure via `summary`.
|
||||
- `memory-reorganize` — plan-mode consolidation/promotion procedure; when to trigger `--force` rebuild; human-approval guardrails.
|
||||
- **Env vars** baked in: `OLLAMA_FLASH_ATTENTION=1`, `GRAPHIFY_OLLAMA_KEEP_ALIVE=5`,
|
||||
`OLLAMA_BASE_URL=http://127.0.0.1:11434/v1`, `OLLAMA_API_KEY=<any-non-empty>`. Note:
|
||||
`GRAPHIFY_OLLAMA_NUM_CTX` is NOT effective through graphify's `/v1` path — context must
|
||||
be baked into the Modelfile variant instead (see Step 2b).
|
||||
- **Config** (set once at user level): vault path, Graphify output dir, Ollama model name,
|
||||
stale-rebuild threshold in days.
|
||||
|
||||
---
|
||||
|
||||
## Open questions / decisions still to settle
|
||||
|
||||
These are deferred to build time. Most can be defaulted without blocking. **§6 (model selection) is RESOLVED** — qwen2.5-coder:7b selected, vault graph built. Remaining questions (§1–5, §7–8) do not block Step 3 or later.
|
||||
|
||||
1. **Vault symlink** — `~/Documents/SecondBrain` is confirmed as the vault (RESOLVED per
|
||||
ADR-012). The open sub-question: symlink it into `~/.claude/memory` or not? Only needed if
|
||||
a tool requires that path.
|
||||
|
||||
2. **Sync mechanism** — git (versioned, hourly) vs Syncthing (continuous). Both are valid;
|
||||
choose at build time based on preference.
|
||||
|
||||
3. **Stale rebuild threshold** — 7 days is the starting guess. Tune after observing how quickly
|
||||
ghost-node drift becomes noticeable in practice.
|
||||
|
||||
4. **Per-project graph path convention** — how does SessionStart know which `graph.json` to
|
||||
inject? Proposed convention: `<project-root>/graphify-out/graph.json`, injected only if the
|
||||
file exists at that path. Not yet made explicit in the plugin spec.
|
||||
|
||||
5. **`summary` field discipline** — Graphify extracts entities and edges but does not write
|
||||
summaries. The human (or AI at note-creation time) must write `summary:` frontmatter. Confirm
|
||||
this holds in practice and add a lint/reminder to the memory-write skill if it drifts.
|
||||
|
||||
6. **Step 2c reference set + model selection** — **RESOLVED (2026-06-04/05):** The reference
|
||||
set was generated: 18 fragments (6 fixtures × 3 tiers) in `benchmark/reference-outputs/`.
|
||||
Local-model scoring complete (see `benchmark/scoring-results-2026-06-04.md`). **Selected
|
||||
model: qwen2.5-coder:7b** (run as Modelfile variant `qwen25-coder-7b-16k`, num_ctx 16384)
|
||||
— graphify's shipped default and the only tested candidate that cleared the
|
||||
relationship/edge gate. All smaller candidates tested (gemma4:e4b 8.0B, gemma4:e2b 5.1B,
|
||||
qwen3.5:2b 2.3B) failed to produce relationship-bearing graphs through graphify's extraction
|
||||
prompt on this hardware. Model selection is complete.
|
||||
|
||||
7. **memsearch + journal integration** — does memsearch index SessionEnd journal notes or only
|
||||
its own auto-capture? How does the journal pointer injected at SessionStart reference the
|
||||
vault? Nail down at memsearch install time (Step 4).
|
||||
|
||||
8. **MCP server management** — vault graph + N project graphs = N+1 potential instances. The
|
||||
build plan leans toward a single Graphify CLI on PATH with `--graph <path>` rather than
|
||||
running multiple `graphify serve` instances, but this is not yet made explicit in the plugin
|
||||
spec. Confirm the CLI-only approach or define the server lifecycle.
|
||||
|
||||
**Resolved:** The type-tag taxonomy is settled via ADR-011 (faceted, six independent namespaces;
|
||||
`type/` listed first). No further debate needed on taxonomy shape.
|
||||
|
||||
---
|
||||
|
||||
## Recommended next move
|
||||
|
||||
**Step 2 is complete.** The next phase is **Step 3 (Hooks)** — thin shell wrappers around
|
||||
Graphify for PostToolUse, SessionStart, and SessionEnd — followed by Step 4 (memsearch
|
||||
episodic layer) and Step 6 (package as global plugin). Open questions §1–5 and §7–8 can be
|
||||
defaulted or deferred at build time. §6 (model selection) is resolved.
|
||||
|
|
@ -0,0 +1,210 @@
|
|||
# Claude-Tier Reference-Set Benchmark — Dispatch Prompt
|
||||
|
||||
_Last updated: 2026-06-04_
|
||||
_Status: active — copy-paste this into a Claude Code session to generate the reference set_
|
||||
|
||||
## Overview
|
||||
|
||||
This prompt dispatches **one subagent per Claude tier** (`claude-haiku-4-5`, `claude-sonnet-4-6`,
|
||||
`claude-opus-4-8`) over each fixture note selected in Step 1c of the build runbook. Each subagent
|
||||
emits a Graphify-shaped structured fragment. The per-model output files become the **gold-standard
|
||||
reference set** against which local Ollama doc-extraction models are scored in a later step.
|
||||
|
||||
**`claude-opus-4-8`'s output is the gold-standard rubric.**
|
||||
|
||||
Evaluation is on **quality only**: entity correctness, relationship plausibility and typing, and
|
||||
confidence-tag accuracy. Wall-clock speed is NOT a metric for this Claude reference run — speed
|
||||
re-enters only when local Ollama models are timed against this reference set.
|
||||
|
||||
---
|
||||
|
||||
## Step 1 — Select Fixture Notes (operator action, before dispatching)
|
||||
|
||||
Fixture notes are selected at build time from `~/Documents/SecondBrain` per Step 1c of the
|
||||
runbook. Choose **5–10 notes** with deliberate variety:
|
||||
|
||||
- One **tool note** (documents a specific tool or library)
|
||||
- One **client/project note** (describes a client engagement or project)
|
||||
- One **convention note** (captures a working convention or practice)
|
||||
- One **domain note** (covers a knowledge or technical domain)
|
||||
- One **relationship-dense note** (many named entities or cross-references)
|
||||
- Additional notes as needed for coverage
|
||||
|
||||
Assign each note a **kebab-case slug** (used in output filenames). Populate the placeholder list
|
||||
below before dispatching:
|
||||
|
||||
```
|
||||
# FIXTURES — selected 2026-06-04; benchmark run same day (no vault frontmatter modification)
|
||||
# Format: <note-slug>:<vault-relative-path> (vault root: ~/Documents/SecondBrain)
|
||||
FIXTURES=(
|
||||
10dlc-isv-setup-guide:2026-03-31-10dlc-isv-setup-guide-oncadence.md
|
||||
pest-control-enterprise-revenue:2026-03-13-pest-control-enterprise-revenue-architecture-and-seasonality.md
|
||||
ai-coding-conventions-synthesis:2026-03-13-ai-coding-conventions-organization-external-research-synthesis.md
|
||||
pest-control-sms-market-research-stats:2026-03-13-pest-control-after-hours-sms-lead-capture-market-research-stats.md
|
||||
oo-principles-plugin-concept:2026-03-13-oo-principles-plugin-concept-design-recommendations.md
|
||||
pest-control-email-abc-hub:2026-03-14-pest-control-email-a-b-c-test-experiment-hub.md
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 2 — Output File Convention
|
||||
|
||||
For each fixture note, the run produces **one output file per Claude tier** — three files per note,
|
||||
`3 × N` files total (where N is the number of fixture notes).
|
||||
|
||||
Output path pattern:
|
||||
|
||||
```
|
||||
docs/memory-system/benchmark/reference-outputs/<note-slug>.<tier>.md
|
||||
```
|
||||
|
||||
Where `<tier>` is one of: `haiku`, `sonnet`, `opus`
|
||||
|
||||
Examples (using a hypothetical slug `graphify-tool-overview`):
|
||||
- `reference-outputs/graphify-tool-overview.haiku.md`
|
||||
- `reference-outputs/graphify-tool-overview.sonnet.md`
|
||||
- `reference-outputs/graphify-tool-overview.opus.md`
|
||||
|
||||
Each file contains that tier's extraction fragment for **one fixture note** — the `note_slug:`
|
||||
block for that note only. Do not combine multiple notes into one file.
|
||||
|
||||
---
|
||||
|
||||
## Step 3 — Dispatch (copy-paste this block into a Claude Code session)
|
||||
|
||||
> **Operator:** replace `<NOTE-SLUG>`, `<NOTE-TEXT>`, and model names as needed. Dispatch all
|
||||
> three subagents for each fixture note. They can run in parallel across tiers for the same note.
|
||||
|
||||
---
|
||||
|
||||
### Dispatch template (repeat for each fixture note × each tier)
|
||||
|
||||
```
|
||||
Dispatch a subagent using model <MODEL> to perform the following task.
|
||||
|
||||
=== FAIRNESS CONTRACT ===
|
||||
You will receive exactly two inputs:
|
||||
1. The raw text of one vault note (below).
|
||||
2. The shared extraction spec and output schema (below).
|
||||
|
||||
You MUST NOT read any repository files (CLAUDE.md, design docs, specs, tasks), access the
|
||||
vault directory structure, or use any project or system context. If your environment has
|
||||
injected any such context automatically, you must ignore it entirely — treat it as if it
|
||||
does not exist. The only allowed inputs are the note text and the extraction spec below.
|
||||
=== END FAIRNESS CONTRACT ===
|
||||
|
||||
=== NOTE TEXT ===
|
||||
<NOTE-TEXT>
|
||||
=== END NOTE TEXT ===
|
||||
|
||||
=== EXTRACTION SPEC AND OUTPUT SCHEMA ===
|
||||
|
||||
## Facet Vocabulary (closed)
|
||||
|
||||
Notes carry six flat, namespaced facets plus one scope tag. These are the ONLY valid values
|
||||
for the optional `facet` field on entities. Do not invent new namespaces.
|
||||
|
||||
| Prefix | Meaning |
|
||||
|--------------|-----------------------------------------|
|
||||
| `type/` | What kind of thing the note is about |
|
||||
| `client/` | A client identity |
|
||||
| `project/` | A project name |
|
||||
| `domain/` | A knowledge or technical domain |
|
||||
| `tool/` | A specific tool, library, or CLI |
|
||||
| `convention/`| A working convention or practice |
|
||||
| `scope/` | Applicability scope (cross-cutting tag) |
|
||||
|
||||
An entity should carry a `facet` value only when the note text directly supports mapping it
|
||||
to one of the above. Absence of a `facet` field is correct when no mapping is warranted.
|
||||
|
||||
## Output Schema
|
||||
|
||||
Emit ONLY the following YAML structure. No other keys, no prose, no summary.
|
||||
|
||||
```yaml
|
||||
# --- Graphify extraction fragment ---
|
||||
# One block per fixture note. Repeat this structure for each note.
|
||||
|
||||
note_slug: <kebab-case-identifier-for-the-note> # operator fills this in at run time
|
||||
|
||||
entities:
|
||||
- name: <string> # exact or near-exact name as it appears in the note
|
||||
type: <string> # e.g. Person, Tool, Project, Concept, Convention, Client, Domain
|
||||
facet: <string> # OPTIONAL — must be one of the seven prefixes above, e.g. "tool/graphify"
|
||||
confidence: <string> # OPTIONAL — omit if the entity is directly stated
|
||||
# Values: INFERRED | AMBIGUOUS
|
||||
|
||||
relationships:
|
||||
- source: <entity name> # must match a name in the entities list above
|
||||
type: <string> # free-text verb phrase, e.g. "uses", "depends_on", "implements", "replaces"
|
||||
target: <entity name> # must match a name in the entities list above
|
||||
confidence: <string> # OPTIONAL — omit if the relationship is directly stated
|
||||
# Values: INFERRED | AMBIGUOUS
|
||||
```
|
||||
|
||||
## Confidence tag semantics
|
||||
|
||||
| Tag | Meaning |
|
||||
|------------|--------------------------------------------------------------------------------------|
|
||||
| _(absent)_ | The entity or relationship is **directly stated** in the note text. |
|
||||
| `INFERRED` | Not stated literally but **reasonably deducible** from the note text alone. |
|
||||
| `AMBIGUOUS`| Supportable by the text but **uncertain or admits multiple readings**. |
|
||||
|
||||
## Rules
|
||||
|
||||
- **Entities only:** extract entities that are named (not vague category references).
|
||||
- **No invented names:** entity names must be grounded in the note text.
|
||||
- **Relationship types:** free-text verb phrases; do not normalize to a fixed vocabulary.
|
||||
- **Facet mapping:** use the closed vocabulary above; only assign when the note text supports it.
|
||||
- **No extra keys:** do not add summaries, scores, embeddings, or metadata fields.
|
||||
- **One block per note:** if processing multiple fixture notes, emit one `note_slug:` block per
|
||||
note, separated by `---`.
|
||||
|
||||
=== END EXTRACTION SPEC ===
|
||||
|
||||
Write your output to:
|
||||
docs/memory-system/benchmark/reference-outputs/<NOTE-SLUG>.<TIER>.md
|
||||
|
||||
where <NOTE-SLUG> is the kebab-case slug for this note and <TIER> is one of: haiku, sonnet, opus.
|
||||
|
||||
The file must contain ONLY the YAML extraction fragment — no preamble, no explanation.
|
||||
```
|
||||
|
||||
**MODEL and TIER substitutions:**
|
||||
|
||||
| Dispatch | MODEL | TIER |
|
||||
|----------|------------------------|---------|
|
||||
| 1st | `claude-haiku-4-5` | `haiku` |
|
||||
| 2nd | `claude-sonnet-4-6` | `sonnet`|
|
||||
| 3rd | `claude-opus-4-8` | `opus` |
|
||||
|
||||
---
|
||||
|
||||
## Step 4 — After All Subagents Complete
|
||||
|
||||
1. Verify `3 × N` files exist in `docs/memory-system/benchmark/reference-outputs/`.
|
||||
2. For each fixture note, diff the three tier files side by side to understand tier-level
|
||||
disagreements in entity recognition, relationship typing, and confidence tagging.
|
||||
3. `claude-opus-4-8`'s `.opus.md` file for each note is the **gold-standard rubric** against
|
||||
which local Ollama extraction will later be scored.
|
||||
4. Do NOT run Ollama models in this step — that is a separate later step that uses these files
|
||||
as the scoring reference.
|
||||
|
||||
---
|
||||
|
||||
## Notes on the Fairness Contract
|
||||
|
||||
The fairness contract requires that each subagent reasons from **note text + extraction spec
|
||||
only**. Two mechanisms can break this:
|
||||
|
||||
1. **Deliberate reads:** the subagent reads repository files or the vault. The dispatch template
|
||||
forbids this explicitly.
|
||||
2. **Injected context:** Claude Code automatically injects `CLAUDE.md` and project context into
|
||||
subagent sessions. The dispatch template explicitly instructs the subagent to **ignore any
|
||||
injected context** and treat it as non-existent. This is the critical instruction — "don't
|
||||
read files" is not sufficient on its own.
|
||||
|
||||
If you observe tier outputs that appear to reflect knowledge of the vault structure or system
|
||||
design (beyond what the note text contains), treat that output as contaminated and re-run that
|
||||
subagent with stronger isolation instructions.
|
||||
|
|
@ -0,0 +1,95 @@
|
|||
# Graphify Document-Extraction Benchmark — Shared Extraction Spec
|
||||
|
||||
_Last updated: 2026-06-04 — rules tightened after first benchmark run (bare-slash facets and undeclared relationship entities)_
|
||||
_Status: active — used as the shared schema for the Claude-tier reference-set benchmark_
|
||||
|
||||
## Purpose
|
||||
|
||||
This spec defines the **minimal structured fragment** that every benchmarking subagent (one per
|
||||
Claude tier) must emit when given a raw vault note. The fragment mimics what Graphify's local SLM
|
||||
doc-extraction step produces: a list of **entities**, a list of **typed relationships** (edges
|
||||
between entities), and per-item **confidence tags** where applicable.
|
||||
|
||||
The schema is intentionally minimal. It exists so that:
|
||||
1. Per-model output files are **diffable entity-by-entity and edge-by-edge** across tiers.
|
||||
2. The same schema is used unchanged when local Ollama models are later scored against this
|
||||
reference set — making the comparison apples-to-apples.
|
||||
|
||||
Subagents receive only the raw note text plus this spec. No repository files, no vault structure,
|
||||
no project context. If any such context is injected by the environment, it must be ignored.
|
||||
|
||||
---
|
||||
|
||||
## Facet Vocabulary (closed)
|
||||
|
||||
Notes in the SecondBrain vault carry six flat, namespaced facets plus one scope tag. These are the
|
||||
**only** valid values for the optional `facet` field on entities. Do not invent new namespaces.
|
||||
|
||||
| Prefix | Meaning |
|
||||
|--------------|-----------------------------------------|
|
||||
| `type/` | What kind of thing the note is about |
|
||||
| `client/` | A client identity |
|
||||
| `project/` | A project name |
|
||||
| `domain/` | A knowledge or technical domain |
|
||||
| `tool/` | A specific tool, library, or CLI |
|
||||
| `convention/`| A working convention or practice |
|
||||
| `scope/` | Applicability scope (cross-cutting tag) |
|
||||
|
||||
An entity should carry a `facet` value only when the note text directly supports mapping it to one
|
||||
of the above. Absence of a `facet` field is correct when no mapping is warranted.
|
||||
|
||||
**Sub-value required (STRICT):** A `facet` value MUST include a non-empty sub-value after the
|
||||
namespace slash — e.g. `tool/cursor`, `convention/tdd`, `domain/distributed-systems`. A bare
|
||||
namespace with nothing after the slash (e.g. `tool/`, `domain/`) is **invalid** and must never
|
||||
appear in output. If a suitable sub-value cannot be identified from the note text, omit the
|
||||
`facet` field entirely.
|
||||
|
||||
---
|
||||
|
||||
## Output Schema
|
||||
|
||||
The required output format. Every subagent MUST emit this structure and no other top-level keys.
|
||||
|
||||
```yaml
|
||||
# --- Graphify extraction fragment ---
|
||||
# One block per fixture note. Repeat this structure for each note.
|
||||
|
||||
note_slug: <kebab-case-identifier-for-the-note> # operator fills this in at run time
|
||||
|
||||
entities:
|
||||
- name: <string> # exact or near-exact name as it appears in the note
|
||||
type: <string> # e.g. Person, Tool, Project, Concept, Convention, Client, Domain
|
||||
facet: <string> # OPTIONAL — must be one of the seven prefixes above, e.g. "tool/graphify"
|
||||
confidence: <string> # OPTIONAL — omit if the entity is directly stated
|
||||
# Values: INFERRED | AMBIGUOUS
|
||||
|
||||
relationships:
|
||||
- source: <entity name> # must match a name in the entities list above
|
||||
type: <string> # free-text verb phrase, e.g. "uses", "depends_on", "implements", "replaces"
|
||||
target: <entity name> # must match a name in the entities list above
|
||||
confidence: <string> # OPTIONAL — omit if the relationship is directly stated
|
||||
# Values: INFERRED | AMBIGUOUS
|
||||
```
|
||||
|
||||
### Confidence tag semantics
|
||||
|
||||
| Tag | Meaning |
|
||||
|------------|--------------------------------------------------------------------------------------|
|
||||
| _(absent)_ | The entity or relationship is **directly stated** in the note text. |
|
||||
| `INFERRED` | Not stated literally but **reasonably deducible** from the note text alone. |
|
||||
| `AMBIGUOUS`| Supportable by the text but **uncertain or admits multiple readings**. |
|
||||
|
||||
### Rules
|
||||
|
||||
- **Entities only:** extract entities that are named (not vague category references).
|
||||
- **No invented names:** entity names must be grounded in the note text.
|
||||
- **Relationship types:** free-text verb phrases; do not normalize to a fixed vocabulary.
|
||||
- **Facet mapping:** use the closed vocabulary above; only assign when the note text supports it.
|
||||
- **No extra keys:** do not add summaries, scores, embeddings, or metadata fields.
|
||||
- **One block per note:** if processing multiple fixture notes, emit one `note_slug:` block per note,
|
||||
separated by `---`.
|
||||
- **Referential integrity (STRICT):** every relationship `source` and `target` MUST exactly match
|
||||
the `name` of an entity declared in the `entities` list of the same block. Do not reference
|
||||
undeclared concept strings, free-form text, or the note's own `note_slug` as a `source` or
|
||||
`target`. If a relationship needs a concept that is not yet in the entity list, declare it as an
|
||||
entity first (with an appropriate `confidence` tag if it is inferred), then reference it.
|
||||
|
|
@ -0,0 +1,274 @@
|
|||
# Local-LLM Doc-Extraction Gut-Check — 2026-06-04
|
||||
|
||||
_Last updated: 2026-06-04_ | _Status: gut-check complete; gemma4:e4b is the candidate; GPU confirmed post-reboot (~74 tok/s, see appendix); Graphify validation is the next step._
|
||||
|
||||
This document consolidates the local-LLM doc-extraction gut-check run on 2026-06-04. It is
|
||||
the authoritative findings record for that session and feeds forward into the Graphify-ollama
|
||||
validation step.
|
||||
|
||||
**Four things this document keeps strictly separate:**
|
||||
|
||||
- **(A) Model-capability signal** — what carries forward; the durable conclusion.
|
||||
- **(B) Raw-ollama harness gotchas** — benchmark-process notes only; explicitly NOT production config.
|
||||
- **(C) GPU driver fix** — real diagnosis; timing estimate, not a measured result.
|
||||
- **(D) Graphify config surface** — the actual production guidance.
|
||||
|
||||
Cross-references:
|
||||
- Claude-tier reference run: `docs/memory-system/benchmark/run-findings-2026-06-04.md`
|
||||
- 18 Claude reference fragments: `docs/memory-system/benchmark/reference-outputs/` (`.haiku.md`, `.sonnet.md`, `.opus.md` files only)
|
||||
- Authoritative Graphify-ollama section: `docs/graphify/05-local-models-and-backends.md`
|
||||
|
||||
---
|
||||
|
||||
## TL;DR — What worked
|
||||
|
||||
- The local model class is **good-enough** for the doc-extraction role. Best installed model:
|
||||
**`gemma4:e4b`** (~9.6 GB, fits the RTX 3060's 12 GB). It produced valid schema on 6/6
|
||||
fixtures with clean, well-namespaced facets (0 bad facets out of 6) and note-grounded
|
||||
entities (no contamination).
|
||||
- It catches the high-value entities (~45–60% of Opus's entity recall), missing Opus's long
|
||||
tail — acceptable for the role.
|
||||
- **Critical reframe:** in production we will NOT hand-invoke ollama. Graphify owns the ollama
|
||||
call (prompt, chunking, context window, parsing) over its HTTP API. The hand-rolled benchmark
|
||||
validated *model capability*, not the production pipeline. The raw-ollama tuning discovered
|
||||
during this run is Graphify-internal, not our config surface.
|
||||
|
||||
---
|
||||
|
||||
## (A) Model-capability signal — measured via a hand-rolled prompt/parser (NOT Graphify)
|
||||
|
||||
This signal was gathered through a custom prompt and a custom YAML parser, NOT Graphify's
|
||||
extraction path. It says "the model class is good-enough." It does NOT say "this is what
|
||||
Graphify will output" — Graphify uses a different prompt, schema, and parser.
|
||||
|
||||
### Results across the 3 installed models (all reasoning models; only these 3 are on the machine)
|
||||
|
||||
| Model | Size | Parse rate | Entities/note | Facet quality | Verdict |
|
||||
|---|---|---|---|---|---|
|
||||
| `gemma4:e4b` | ~9.6 GB | 6/6 (2 needed trivial colon-quote repair) | 13–34 (~45–60% of Opus) | 0 bad/6 | **Usable — best pick** |
|
||||
| `gemma4:e2b` | ~7.2 GB | 4/6 (2 hard YAML breaks — dropped `source:` key) | 23–31 when valid | 0 bad/6 | Usable only behind a tolerant parser + retry |
|
||||
| `qwen3.5:2b` | ~2.7 GB | parses but collapses to ~1 entity/note | 1 | n/a | Not usable as tested |
|
||||
|
||||
All three land below haiku (the weakest Claude frontier tier: 19–37 entities, 6/6 clean parse,
|
||||
clean referential integrity). "Below frontier-floor" does not mean "unusable" — `gemma4:e4b`'s
|
||||
misses are the long tail; the high-value entities are present.
|
||||
|
||||
### gemma4:e4b quirks observed
|
||||
|
||||
- Occasional top-level key rename (e.g. `entity_relationships` instead of the spec's key).
|
||||
- A `target: Concept` placeholder leak on the longest note — it echoed the schema's example
|
||||
value rather than a concrete entity. Seen once across 6 fixtures.
|
||||
|
||||
These quirks were caught by the hand-rolled parser and are noted as model-class behavior.
|
||||
Whether Graphify's parser handles them gracefully is a separate question (see section D).
|
||||
|
||||
---
|
||||
|
||||
## (B) Raw-ollama harness gotchas — benchmark-process notes ONLY
|
||||
|
||||
> **EXPLICIT CAVEAT:** This section records why the first benchmark pass produced junk and
|
||||
> what was done to fix the harness. These knobs are NOT our production settings. They are
|
||||
> Graphify-internal concerns (see section D). Do not treat this section as production config.
|
||||
|
||||
### Problem: default context truncation produced garbage output
|
||||
|
||||
Bare `ollama run` with the default (~4K) context truncated output mid-stream. Reasoning models
|
||||
burn their token budget on chain-of-thought first; the model got guillotined mid-output, leaving
|
||||
malformed YAML.
|
||||
|
||||
### Harness fix (benchmark only)
|
||||
|
||||
- Called the HTTP API at `/api/generate` instead of `ollama run`.
|
||||
- Set `num_ctx=8192` (per-request via the API body).
|
||||
- Set `think=false` to suppress chain-of-thought tokens and keep the output budget for the
|
||||
extraction result.
|
||||
- Added a tolerant YAML parser that retries on soft parse errors and auto-quotes bare scalars
|
||||
containing `:`.
|
||||
|
||||
These changes produced the results in section A. They explain the benchmark outcome; they do
|
||||
not define the production path, which Graphify handles internally.
|
||||
|
||||
---
|
||||
|
||||
## (C) GPU not used during this run — root cause and fix
|
||||
|
||||
### Root cause: NVIDIA driver version mismatch
|
||||
|
||||
| Item | Value |
|
||||
|---|---|
|
||||
| In-kernel module | 580.142 |
|
||||
| On-disk / userspace driver | 580.159.03 |
|
||||
|
||||
The driver was updated on disk on 2026-06-03 but the stale module stayed resident. Result:
|
||||
`nvidia-smi` fails — "Failed to initialize NVML: Driver/library version mismatch." Because
|
||||
NVML/CUDA cannot initialize, ollama finds no CUDA device (`total_vram="0 B"`) and runs 100%
|
||||
on CPU.
|
||||
|
||||
Same-machine proof: the ollama log at 07:57 on 2026-06-03 DID find the RTX 3060
|
||||
(`compute=8.6 ... 12.0 GiB`); the 16:12 service instance the benchmark ran under found no CUDA.
|
||||
|
||||
**CPU eval rate observed:** ~10.6 tok/s → ~187–313 s/note. The 9.6 GB model fits the 12 GB
|
||||
card; VRAM is not the bottleneck — the driver mismatch is.
|
||||
|
||||
### Fix (user must run — non-interactive sudo was blocked during this session)
|
||||
|
||||
The running kernel already matches the on-disk 580.159.03 module, so a reboot loads the correct
|
||||
module:
|
||||
|
||||
```bash
|
||||
sudo reboot
|
||||
# then verify:
|
||||
nvidia-smi # should print the RTX 3060 table, no NVML error
|
||||
|
||||
# only if CUDA still fails after reboot:
|
||||
sudo dnf reinstall 'akmod-nvidia*' 'xorg-x11-drv-nvidia*' # then reboot again
|
||||
```
|
||||
|
||||
### ESTIMATE — post-reboot GPU timing (NOT measured; blocked by sudo this session)
|
||||
|
||||
With the RTX 3060 and the 4-bit `gemma4:e4b` model fully resident, ~40–60 tok/s plus faster
|
||||
prompt processing should drop extraction to roughly single-digit to ~20 s/note — i.e.,
|
||||
plausibly within a "well under a minute" target.
|
||||
|
||||
**This is an estimate to be confirmed post-reboot using the curl command in the appendix below.**
|
||||
It is not a measured result.
|
||||
|
||||
---
|
||||
|
||||
## (D) Graphify config surface — production guidance
|
||||
|
||||
> This is the only section that contains production configuration guidance. Sections B and C
|
||||
> are process notes; section A is a model-capability signal only.
|
||||
|
||||
### How Graphify talks to Ollama
|
||||
|
||||
Graphify talks to Ollama over its **HTTP API** at `http://localhost:11434`, not by shelling out
|
||||
to `ollama run`. `[github-inferred]` from the env-var/timeout/num_ctx design; the exact endpoint
|
||||
(`/api/generate` vs `/api/chat`) is `[unverified]` — package internals are not browseable on
|
||||
GitHub.
|
||||
|
||||
Graphify **owns** the prompt template, chunking, context sizing (auto-sized `num_ctx` by
|
||||
default), and output parsing. `[github]` for the env-var knobs; prompt and parser specifics are
|
||||
`[github-inferred]` / `[unverified]`. Whether Graphify sets `think=false` and whether it does
|
||||
tolerant-YAML / retry are `[unverified]`.
|
||||
|
||||
Implication: the hand-rolled extraction-spec prompt and raw-ollama settings from section B
|
||||
likely do **not** apply to Graphify's ollama path. Our levers are the documented external knobs
|
||||
only.
|
||||
|
||||
### Text-vs-code split
|
||||
|
||||
Confirmed `[github]` (verbatim README): code is extracted locally via tree-sitter AST with zero
|
||||
LLM calls; docs/PDFs/images go through the LLM backend (Ollama here). Ollama is invoked **for
|
||||
text documents only.**
|
||||
|
||||
### Model selection
|
||||
|
||||
No official Graphify-recommended Ollama model. `[github]` — absence verified at v0.8.30.
|
||||
Community starting points: `qwen2.5:7b`, `phi4:14b`. `[community]` Default behavior: auto-detect
|
||||
the running model. Override via `OLLAMA_MODEL`.
|
||||
|
||||
Per this benchmark, `gemma4:e4b` is the best installed candidate — but this must be validated
|
||||
through Graphify's own extraction path, not the hand-rolled benchmark (see Deferred
|
||||
follow-ups §1 below).
|
||||
|
||||
### Environment variables and CLI flags (all `[github]` from v0.8.30 README)
|
||||
|
||||
| Knob | Default | Notes |
|
||||
|---|---|---|
|
||||
| `OLLAMA_BASE_URL` | `http://localhost:11434` | Ollama endpoint |
|
||||
| `OLLAMA_MODEL` | auto-detect | Override to pin a model |
|
||||
| `GRAPHIFY_OLLAMA_NUM_CTX` | auto-sized | Override context window |
|
||||
| `GRAPHIFY_OLLAMA_KEEP_ALIVE` | unspecified | Minutes to keep model loaded; `0` = unload after each chunk |
|
||||
| `--token-budget` | — | Chunk size (tokens) |
|
||||
| `--max-concurrency` | — | Parallel extraction workers |
|
||||
| `--api-timeout` | 600 s | Increase for slower hardware |
|
||||
|
||||
### Known sharp edge
|
||||
|
||||
Context saturation across consecutive chunks on the Ollama backend can exhaust VRAM after a
|
||||
few chunks. `[community, issue #798]` Mitigate by lowering `GRAPHIFY_OLLAMA_NUM_CTX` and/or
|
||||
setting `GRAPHIFY_OLLAMA_KEEP_ALIVE=0`.
|
||||
|
||||
---
|
||||
|
||||
## Appendix — Exact commands
|
||||
|
||||
### Graphify Ollama path `[github]`
|
||||
|
||||
```bash
|
||||
# Install (package is graphifyy — double-y; command is graphify):
|
||||
uv tool install "graphifyy[ollama]"
|
||||
|
||||
# Pull the candidate model (gemma4:e4b is the local best pick per this benchmark):
|
||||
ollama pull gemma4:e4b
|
||||
|
||||
# Start Ollama if not already running as a service:
|
||||
ollama serve
|
||||
|
||||
# Basic extraction:
|
||||
graphify extract ./docs --backend ollama
|
||||
|
||||
# Constrained single-GPU tuning:
|
||||
GRAPHIFY_OLLAMA_NUM_CTX=8192 graphify extract ./docs --backend ollama \
|
||||
--token-budget 4000 --max-concurrency 2 --api-timeout 900
|
||||
|
||||
# Free VRAM between chunks (mitigates issue #798):
|
||||
GRAPHIFY_OLLAMA_KEEP_ALIVE=0 graphify extract ./docs --backend ollama
|
||||
```
|
||||
|
||||
### Post-reboot GPU re-timing (benchmark verification only — NOT production)
|
||||
|
||||
Run this after rebooting to confirm the GPU is now used and to measure actual tok/s:
|
||||
|
||||
```bash
|
||||
F="$HOME/Documents/SecondBrain/2026-03-31-10dlc-isv-setup-guide-oncadence.md"
|
||||
PROMPT=$(python3 -c "import json;print(json.dumps('Extract entities from this note:\n\n'+open('$F').read()))")
|
||||
time curl -s http://127.0.0.1:11434/api/generate \
|
||||
-d "{\"model\":\"gemma4:e4b\",\"prompt\":$PROMPT,\"stream\":false,\"think\":false,\"options\":{\"num_ctx\":8192}}" \
|
||||
| python3 -c "import json,sys;d=json.load(sys.stdin);ec=d['eval_count'];ed=d['eval_duration'];print('eval tok/s:',round(ec/(ed/1e9),2),'| eval_count:',ec,'| total wall (s):',round(d['total_duration']/1e9,2))"
|
||||
```
|
||||
|
||||
This measures raw model tok/s via the hand-rolled harness — it does NOT measure Graphify's
|
||||
production extraction performance.
|
||||
|
||||
#### Measured results (run 2026-06-04, post-reboot)
|
||||
|
||||
GPU is now fully active and resident — the driver mismatch diagnosed in section C is
|
||||
**RESOLVED**. `nvidia-smi` is healthy (Driver 580.159.03, RTX 3060, no NVML error). The
|
||||
`gemma4:e4b` model loaded fully GPU-resident: `/api/ps` reports `size_vram` 3.29 GB ==
|
||||
`size` 3.29 GB (no CPU fallback), with 4462 MiB used on the card.
|
||||
|
||||
| Pass | eval tok/s | eval_count | total wall (s) | load (s) | prompt_eval (s) | prompt_eval_count |
|
||||
|---|---|---|---|---|---|---|
|
||||
| Cold | 74.27 | 701 | 76.04 | 38.15 | 28.44 | 905 |
|
||||
| Warm | 74.25 | 717 | 10.13 | 0.43 | 0.04 | — |
|
||||
|
||||
Conclusion: steady-state generation is **~74 tok/s** — at or above the section C estimate of
|
||||
40–60 tok/s. Warm extraction lands at **~10 s/note**; the cold pass is ~76 s, dominated by the
|
||||
one-time 38 s model load plus a 28 s cold prompt-eval. This confirms section C's "well under a
|
||||
minute once the model is resident" prediction.
|
||||
|
||||
> **Caveat retained:** these are raw model tok/s from the hand-rolled curl harness, NOT
|
||||
> Graphify production extraction performance (see Deferred follow-up §1).
|
||||
|
||||
---
|
||||
|
||||
## Deferred follow-ups
|
||||
|
||||
1. **Definitive validation: run Graphify's own ollama path over the fixtures** — NOT re-scoring
|
||||
the hand-rolled benchmark. Do NOT conclude the next step is "score Graphify output against
|
||||
the 18 Opus references": that is apples-to-oranges (different prompt/schema/output). The 18
|
||||
Opus references validated model capability; they are not a scoring rubric for Graphify's
|
||||
graph output.
|
||||
|
||||
2. **Resolve `[unverified]` Graphify internals** (HTTP endpoint, think-mode, retry/parsing):
|
||||
after `uv tool install "graphifyy[ollama]"`, read the installed client source under the
|
||||
tool's site-packages. This does not require the GPU — a cheap deferred follow-up.
|
||||
|
||||
3. **Confirm GPU timing post-reboot** — **RESOLVED (2026-06-04).** GPU re-timing ran post-reboot;
|
||||
measured ~74 tok/s steady-state (at/above the section C estimate). See the "Measured results"
|
||||
table under the Post-reboot GPU re-timing appendix above.
|
||||
|
||||
4. **Raw per-model benchmark outputs from this run** were throwaway scratch (wrong harness,
|
||||
hand-rolled prompt) and are not committed.
|
||||
|
|
@ -0,0 +1,88 @@
|
|||
note_slug: 10dlc-isv-setup-guide
|
||||
|
||||
entities:
|
||||
- name: OnCadence
|
||||
type: Project
|
||||
facet: project/oncadence
|
||||
- name: Twilio
|
||||
type: Tool
|
||||
facet: tool/twilio
|
||||
- name: TrustHub API
|
||||
type: Tool
|
||||
facet: tool/twilio
|
||||
- name: 10DLC
|
||||
type: Domain
|
||||
facet: domain/sms
|
||||
- name: ISV architecture
|
||||
type: Convention
|
||||
facet: convention/isv-integration
|
||||
- name: Secondary Customer Profile
|
||||
type: Concept
|
||||
- name: Brand registration
|
||||
type: Concept
|
||||
- name: Campaign registration
|
||||
type: Concept
|
||||
- name: Primary Business Profile
|
||||
type: Concept
|
||||
- name: pest control
|
||||
type: Domain
|
||||
facet: domain/pest-control
|
||||
- name: opt-out language
|
||||
type: Convention
|
||||
- name: EIN
|
||||
type: Concept
|
||||
- name: subaccount
|
||||
type: Concept
|
||||
- name: T-Mobile
|
||||
type: Tool
|
||||
- name: Facebook Business page
|
||||
type: Convention
|
||||
facet: convention/web-presence
|
||||
- name: Google Business Profile
|
||||
type: Convention
|
||||
facet: convention/web-presence
|
||||
- name: Sole Proprietors
|
||||
type: Concept
|
||||
- name: long-code number
|
||||
type: Concept
|
||||
- name: SMS carrier surcharge
|
||||
type: Concept
|
||||
|
||||
relationships:
|
||||
- source: OnCadence
|
||||
type: uses
|
||||
target: Twilio
|
||||
- source: OnCadence
|
||||
type: implements
|
||||
target: ISV architecture
|
||||
- source: Twilio
|
||||
type: provides
|
||||
target: TrustHub API
|
||||
- source: OnCadence
|
||||
type: targets
|
||||
target: pest control
|
||||
- source: TrustHub API
|
||||
type: manages
|
||||
target: Brand registration
|
||||
- source: TrustHub API
|
||||
type: manages
|
||||
target: Campaign registration
|
||||
- source: Secondary Customer Profile
|
||||
type: requires
|
||||
target: EIN
|
||||
- source: Brand registration
|
||||
type: precedes
|
||||
target: Campaign registration
|
||||
- source: subaccount
|
||||
type: carries
|
||||
target: Secondary Customer Profile
|
||||
- source: opt-out language
|
||||
type: required_in
|
||||
target: Campaign registration
|
||||
- source: Sole Proprietors
|
||||
type: limited_to
|
||||
target: T-Mobile
|
||||
- source: long-code number
|
||||
type: isolated_per
|
||||
target: subaccount
|
||||
confidence: INFERRED
|
||||
|
|
@ -0,0 +1,106 @@
|
|||
# --- Graphify extraction fragment ---
|
||||
note_slug: 10dlc-isv-setup-guide
|
||||
|
||||
entities:
|
||||
- name: OnCadence
|
||||
type: Project
|
||||
facet: project/oncadence
|
||||
- name: 10DLC
|
||||
type: Concept
|
||||
facet: domain/10dlc
|
||||
- name: Twilio
|
||||
type: Tool
|
||||
facet: tool/twilio
|
||||
- name: TrustHub API
|
||||
type: Tool
|
||||
facet: tool/twilio-trusthub
|
||||
- name: ISV
|
||||
type: Concept
|
||||
- name: CSP
|
||||
type: Concept
|
||||
- name: pest control clients
|
||||
type: Client
|
||||
facet: client/pest-control
|
||||
- name: Secondary Customer Profile
|
||||
type: Concept
|
||||
- name: Primary Business Profile
|
||||
type: Concept
|
||||
- name: Brand registration
|
||||
type: Concept
|
||||
- name: Campaign registration
|
||||
type: Concept
|
||||
- name: EIN
|
||||
type: Concept
|
||||
- name: IRS
|
||||
type: Concept
|
||||
- name: Facebook Business page
|
||||
type: Concept
|
||||
- name: Google Business Profile
|
||||
type: Concept
|
||||
- name: T-Mobile
|
||||
type: Concept
|
||||
- name: Sole Proprietors
|
||||
type: Concept
|
||||
- name: long-code number
|
||||
type: Concept
|
||||
- name: STOP opt-out language
|
||||
type: Convention
|
||||
facet: convention/sms-opt-out
|
||||
- name: subaccount-per-client architecture
|
||||
type: Concept
|
||||
facet: convention/subaccount-per-client
|
||||
|
||||
relationships:
|
||||
- source: OnCadence
|
||||
type: is_primary_account_of
|
||||
target: Twilio
|
||||
- source: OnCadence
|
||||
type: acts_as
|
||||
target: ISV
|
||||
- source: OnCadence
|
||||
type: acts_as
|
||||
target: CSP
|
||||
- source: OnCadence
|
||||
type: manages_10dlc_for
|
||||
target: pest control clients
|
||||
- source: OnCadence
|
||||
type: submits_via
|
||||
target: TrustHub API
|
||||
- source: OnCadence
|
||||
type: creates_once
|
||||
target: Primary Business Profile
|
||||
- source: pest control clients
|
||||
type: each_assigned
|
||||
target: Secondary Customer Profile
|
||||
- source: pest control clients
|
||||
type: each_assigned
|
||||
target: Brand registration
|
||||
- source: pest control clients
|
||||
type: each_assigned
|
||||
target: Campaign registration
|
||||
- source: Brand registration
|
||||
type: must_match
|
||||
target: EIN
|
||||
- source: EIN
|
||||
type: issued_by
|
||||
target: IRS
|
||||
confidence: INFERRED
|
||||
- source: Sole Proprietors
|
||||
type: lack
|
||||
target: EIN
|
||||
- source: Sole Proprietors
|
||||
type: limited_to_throughput_on
|
||||
target: T-Mobile
|
||||
- source: Facebook Business page
|
||||
type: accepted_for
|
||||
target: Brand registration
|
||||
confidence: INFERRED
|
||||
- source: Campaign registration
|
||||
type: must_include
|
||||
target: STOP opt-out language
|
||||
- source: OnCadence
|
||||
type: implements
|
||||
target: subaccount-per-client architecture
|
||||
- source: pest control clients
|
||||
type: each_assigned
|
||||
target: long-code number
|
||||
|
|
@ -0,0 +1,66 @@
|
|||
# --- Graphify extraction fragment ---
|
||||
note_slug: 10dlc-isv-setup-guide
|
||||
|
||||
entities:
|
||||
- name: OnCadence
|
||||
type: Project
|
||||
facet: project/OnCadence
|
||||
- name: Twilio
|
||||
type: Tool
|
||||
facet: tool/Twilio
|
||||
- name: 10DLC
|
||||
type: Concept
|
||||
facet: domain/SMS
|
||||
- name: TrustHub API
|
||||
type: Tool
|
||||
facet: tool/TrustHub API
|
||||
- name: ISV
|
||||
type: Concept
|
||||
facet: domain/SMS
|
||||
- name: Secondary Customer Profile
|
||||
type: Concept
|
||||
- name: Brand registration
|
||||
type: Concept
|
||||
- name: Campaign registration
|
||||
type: Concept
|
||||
- name: Primary Business Profile
|
||||
type: Concept
|
||||
- name: T-Mobile
|
||||
type: Client
|
||||
- name: Sole Proprietor
|
||||
type: Concept
|
||||
- name: pest control client
|
||||
type: Client
|
||||
facet: client/pest-control
|
||||
|
||||
relationships:
|
||||
- source: OnCadence
|
||||
type: acts as ISV/CSP on
|
||||
target: Twilio
|
||||
- source: OnCadence
|
||||
type: submits registrations on behalf of
|
||||
target: pest control client
|
||||
- source: OnCadence
|
||||
type: uses
|
||||
target: TrustHub API
|
||||
- source: pest control client
|
||||
type: registered as
|
||||
target: Secondary Customer Profile
|
||||
- source: pest control client
|
||||
type: requires
|
||||
target: Brand registration
|
||||
- source: pest control client
|
||||
type: requires
|
||||
target: Campaign registration
|
||||
- source: OnCadence
|
||||
type: creates once
|
||||
target: Primary Business Profile
|
||||
- source: Sole Proprietor
|
||||
type: has lower throughput on
|
||||
target: T-Mobile
|
||||
- source: 10DLC
|
||||
type: governs
|
||||
target: Campaign registration
|
||||
- source: TrustHub API
|
||||
type: is part of
|
||||
target: Twilio
|
||||
|
|
@ -0,0 +1,26 @@
|
|||
# Reference Outputs
|
||||
|
||||
_Last updated: 2026-06-04_
|
||||
|
||||
This directory holds the per-model Graphify-shaped extraction fragments generated by the
|
||||
Claude-tier reference-set benchmark (see `../dispatch-prompt.md`).
|
||||
|
||||
## File naming convention
|
||||
|
||||
```
|
||||
<note-slug>.<tier>.md
|
||||
```
|
||||
|
||||
Where:
|
||||
- `<note-slug>` is the kebab-case identifier for the fixture note (assigned in Step 1c of the
|
||||
build runbook)
|
||||
- `<tier>` is one of: `haiku`, `sonnet`, `opus`
|
||||
|
||||
## Contents
|
||||
|
||||
Each file contains the YAML extraction fragment for one fixture note as produced by one Claude
|
||||
tier. Files are generated by running the dispatch prompt — they do not exist until that step
|
||||
is executed.
|
||||
|
||||
`claude-opus-4-8`'s `.opus.md` files are the **gold-standard rubric** against which local Ollama
|
||||
doc-extraction models are scored in a later step.
|
||||
|
|
@ -0,0 +1,149 @@
|
|||
note_slug: ai-coding-conventions-synthesis
|
||||
|
||||
entities:
|
||||
- name: Claude
|
||||
type: tool
|
||||
facet: tool/claude
|
||||
- name: Cursor
|
||||
type: tool
|
||||
facet: tool/cursor
|
||||
- name: Copilot
|
||||
type: tool
|
||||
facet: tool/copilot
|
||||
- name: CLAUDE.md
|
||||
type: document_type
|
||||
facet: convention/root-file
|
||||
- name: .cursorrules
|
||||
type: document_type
|
||||
facet: convention/root-file
|
||||
- name: AGENTS.md
|
||||
type: document_type
|
||||
facet: convention/hierarchical-scoping
|
||||
- name: Three-Tier Architecture
|
||||
type: framework
|
||||
facet: convention/context-architecture
|
||||
- name: Constitution
|
||||
type: framework_component
|
||||
facet: convention/constitution-pattern
|
||||
- name: Domain Specialist Agents
|
||||
type: framework_component
|
||||
facet: convention/specialist-agents
|
||||
- name: Knowledge Base
|
||||
type: framework_component
|
||||
- name: Codified Context
|
||||
type: research_paper
|
||||
facet: domain/context-engineering
|
||||
- name: arXiv 2602.20478
|
||||
type: publication
|
||||
- name: Nuxt Content
|
||||
type: tool
|
||||
facet: tool/nuxt
|
||||
- name: MCP
|
||||
type: tool
|
||||
facet: tool/mcp
|
||||
- name: Token Efficiency Strategies
|
||||
type: practice_set
|
||||
facet: convention/token-optimization
|
||||
- name: Just-in-time retrieval
|
||||
type: practice
|
||||
facet: convention/context-loading
|
||||
- name: Linters
|
||||
type: tool_category
|
||||
facet: tool/linting
|
||||
- name: Formatters
|
||||
type: tool_category
|
||||
facet: tool/formatting
|
||||
- name: Feedback loop documentation
|
||||
type: practice
|
||||
facet: convention/feedback-loop
|
||||
- name: Process encoding
|
||||
type: practice
|
||||
facet: convention/process-encoding
|
||||
- name: Trigger tables
|
||||
type: mechanism
|
||||
facet: convention/routing-mechanism
|
||||
- name: Reference documentation
|
||||
type: practice
|
||||
facet: convention/documentation-strategy
|
||||
- name: Hierarchical override
|
||||
type: mechanism
|
||||
facet: convention/context-scoping
|
||||
- name: github.com/arisvas4/codified-context-infrastructure
|
||||
type: reference
|
||||
- name: Anthropic Effective Context Engineering
|
||||
type: reference
|
||||
- name: PatrickJS/awesome-cursorrules
|
||||
type: reference
|
||||
- name: alexop.dev
|
||||
type: reference
|
||||
- name: mbleigh.dev
|
||||
type: reference
|
||||
- name: agents.md specification
|
||||
type: standard
|
||||
- name: 99 Bottles OOP — Full Software Design Process Map
|
||||
type: reference
|
||||
- name: OO Principles Plugin Concept — Design Recommendations
|
||||
type: reference
|
||||
|
||||
relationships:
|
||||
- source: Three-Tier Architecture
|
||||
type: comprises
|
||||
target: Constitution
|
||||
- source: Three-Tier Architecture
|
||||
type: comprises
|
||||
target: Domain Specialist Agents
|
||||
- source: Three-Tier Architecture
|
||||
type: comprises
|
||||
target: Knowledge Base
|
||||
- source: Constitution
|
||||
type: routes to
|
||||
target: Domain Specialist Agents
|
||||
- source: Domain Specialist Agents
|
||||
type: contains
|
||||
target: Knowledge Base
|
||||
confidence: INFERRED
|
||||
- source: Knowledge Base
|
||||
type: served via
|
||||
target: MCP
|
||||
- source: CLAUDE.md
|
||||
type: implements
|
||||
target: Constitution
|
||||
- source: .cursorrules
|
||||
type: implements
|
||||
target: Constitution
|
||||
- source: AGENTS.md
|
||||
type: implements
|
||||
target: Hierarchical override
|
||||
- source: Token Efficiency Strategies
|
||||
type: includes
|
||||
target: Just-in-time retrieval
|
||||
- source: Token Efficiency Strategies
|
||||
type: includes
|
||||
target: Linters
|
||||
- source: Token Efficiency Strategies
|
||||
type: includes
|
||||
target: Formatters
|
||||
- source: Token Efficiency Strategies
|
||||
type: includes
|
||||
target: Feedback loop documentation
|
||||
- source: Linters
|
||||
type: replaces
|
||||
target: Reference documentation
|
||||
confidence: INFERRED
|
||||
- source: Formatters
|
||||
type: replaces
|
||||
target: Reference documentation
|
||||
confidence: INFERRED
|
||||
- source: Process encoding
|
||||
type: encodes
|
||||
target: Trigger tables
|
||||
- source: Trigger tables
|
||||
type: enforces
|
||||
target: Constitution
|
||||
confidence: INFERRED
|
||||
- source: Codified Context
|
||||
type: describes
|
||||
target: Three-Tier Architecture
|
||||
- source: Codified Context
|
||||
type: published at
|
||||
target: arXiv 2602.20478
|
||||
|
|
@ -0,0 +1,156 @@
|
|||
# --- Graphify extraction fragment ---
|
||||
note_slug: ai-coding-conventions-synthesis
|
||||
|
||||
entities:
|
||||
- name: Claude
|
||||
type: Tool
|
||||
facet: tool/claude
|
||||
- name: Cursor
|
||||
type: Tool
|
||||
facet: tool/cursor
|
||||
- name: Copilot
|
||||
type: Tool
|
||||
facet: tool/copilot
|
||||
- name: Codified Context
|
||||
type: Concept
|
||||
- name: Three-Tier Architecture
|
||||
type: Concept
|
||||
- name: Constitution
|
||||
type: Concept
|
||||
- name: Domain Specialist Agents
|
||||
type: Concept
|
||||
- name: Knowledge Base
|
||||
type: Concept
|
||||
- name: MCP
|
||||
type: Tool
|
||||
facet: tool/mcp
|
||||
- name: Just-in-time retrieval
|
||||
type: Convention
|
||||
facet: convention/just-in-time-retrieval
|
||||
- name: Trigger tables
|
||||
type: Concept
|
||||
- name: Progressive disclosure
|
||||
type: Convention
|
||||
facet: convention/progressive-disclosure
|
||||
- name: Positive framing only
|
||||
type: Convention
|
||||
facet: convention/positive-framing
|
||||
- name: /learn pattern
|
||||
type: Convention
|
||||
facet: convention/feedback-loop-documentation
|
||||
- name: CLAUDE.md
|
||||
type: Convention
|
||||
facet: convention/root-file
|
||||
- name: .cursorrules
|
||||
type: Convention
|
||||
facet: convention/root-file
|
||||
- name: AGENTS.md
|
||||
type: Convention
|
||||
facet: convention/root-file
|
||||
- name: Cursor .mdc
|
||||
type: Tool
|
||||
- name: alwaysApply
|
||||
type: Convention
|
||||
- name: Auto-attached rules
|
||||
type: Convention
|
||||
- name: Agent-requested rules
|
||||
type: Convention
|
||||
- name: Manual rules
|
||||
type: Convention
|
||||
- name: Hierarchical scoping
|
||||
type: Convention
|
||||
facet: convention/hierarchical-scoping
|
||||
- name: Nuxt Content
|
||||
type: Tool
|
||||
facet: tool/nuxt-content
|
||||
- name: Codified Context paper
|
||||
type: Concept
|
||||
- name: github.com/arisvas4/codified-context-infrastructure
|
||||
type: Project
|
||||
- name: Anthropic Effective Context Engineering
|
||||
type: Concept
|
||||
- name: PatrickJS/awesome-cursorrules
|
||||
type: Project
|
||||
- name: alexop.dev — Stop Bloating Your CLAUDE.md
|
||||
type: Concept
|
||||
- name: mbleigh.dev — Rules for Rules
|
||||
type: Concept
|
||||
- name: agents.md
|
||||
type: Project
|
||||
- name: OpenAI
|
||||
type: Client
|
||||
- name: Anthropic
|
||||
type: Client
|
||||
- name: Context engineering
|
||||
type: Domain
|
||||
facet: domain/context-engineering
|
||||
- name: AI coding conventions
|
||||
type: Domain
|
||||
facet: domain/ai-coding-conventions
|
||||
- name: 99 Bottles OOP — Full Software Design Process Map
|
||||
type: Concept
|
||||
- name: OO Principles Plugin Concept — Design Recommendations
|
||||
type: Concept
|
||||
- name: Linters
|
||||
type: Tool
|
||||
- name: Formatters
|
||||
type: Tool
|
||||
- name: Type checkers
|
||||
type: Tool
|
||||
|
||||
relationships:
|
||||
- source: Three-Tier Architecture
|
||||
type: comprises
|
||||
target: Constitution
|
||||
- source: Three-Tier Architecture
|
||||
type: comprises
|
||||
target: Domain Specialist Agents
|
||||
- source: Three-Tier Architecture
|
||||
type: comprises
|
||||
target: Knowledge Base
|
||||
- source: Codified Context
|
||||
type: documents
|
||||
target: Three-Tier Architecture
|
||||
- source: Constitution
|
||||
type: contains
|
||||
target: Trigger tables
|
||||
- source: Knowledge Base
|
||||
type: served_via
|
||||
target: MCP
|
||||
- source: Codified Context paper
|
||||
type: described_in
|
||||
target: github.com/arisvas4/codified-context-infrastructure
|
||||
confidence: INFERRED
|
||||
- source: Constitution
|
||||
type: routes_tasks_to
|
||||
target: Domain Specialist Agents
|
||||
- source: OpenAI
|
||||
type: uses
|
||||
target: AGENTS.md
|
||||
- source: Anthropic
|
||||
type: published
|
||||
target: Anthropic Effective Context Engineering
|
||||
confidence: INFERRED
|
||||
- source: Cursor
|
||||
type: uses
|
||||
target: .cursorrules
|
||||
confidence: INFERRED
|
||||
- source: Cursor .mdc
|
||||
type: supports
|
||||
target: alwaysApply
|
||||
- source: Cursor .mdc
|
||||
type: supports
|
||||
target: Auto-attached rules
|
||||
- source: Cursor .mdc
|
||||
type: supports
|
||||
target: Agent-requested rules
|
||||
- source: Cursor .mdc
|
||||
type: supports
|
||||
target: Manual rules
|
||||
- source: Linters
|
||||
type: replace
|
||||
target: alwaysApply
|
||||
confidence: INFERRED
|
||||
- source: Trigger tables
|
||||
type: route_to
|
||||
target: Domain Specialist Agents
|
||||
|
|
@ -0,0 +1,135 @@
|
|||
# --- Graphify extraction fragment ---
|
||||
note_slug: ai-coding-conventions-synthesis
|
||||
|
||||
entities:
|
||||
- name: Codified Context
|
||||
type: Concept
|
||||
facet: convention/codified-context
|
||||
confidence: INFERRED
|
||||
- name: Three-Tier Architecture
|
||||
type: Concept
|
||||
facet: convention/three-tier-architecture
|
||||
- name: Constitution
|
||||
type: Concept
|
||||
facet: convention/constitution
|
||||
- name: Domain Specialist Agents
|
||||
type: Concept
|
||||
facet: convention/domain-specialist-agents
|
||||
- name: Knowledge Base
|
||||
type: Concept
|
||||
facet: convention/knowledge-base
|
||||
- name: CLAUDE.md
|
||||
type: Convention
|
||||
facet: tool/claude-md
|
||||
- name: AGENTS.md
|
||||
type: Convention
|
||||
facet: tool/agents-md
|
||||
- name: Cursor
|
||||
type: Tool
|
||||
facet: tool/cursor
|
||||
- name: Claude
|
||||
type: Tool
|
||||
facet: tool/claude
|
||||
- name: Copilot
|
||||
type: Tool
|
||||
facet: tool/copilot
|
||||
- name: MCP
|
||||
type: Tool
|
||||
facet: tool/mcp
|
||||
- name: arXiv 2602.20478
|
||||
type: Reference
|
||||
- name: github.com/arisvas4/codified-context-infrastructure
|
||||
type: Reference
|
||||
- name: Anthropic Effective Context Engineering
|
||||
type: Reference
|
||||
- name: PatrickJS/awesome-cursorrules
|
||||
type: Reference
|
||||
- name: alexop.dev — Stop Bloating Your CLAUDE.md
|
||||
type: Reference
|
||||
- name: mbleigh.dev — Rules for Rules
|
||||
type: Reference
|
||||
- name: agents.md
|
||||
type: Reference
|
||||
- name: progressive-disclosure
|
||||
type: Concept
|
||||
facet: convention/progressive-disclosure
|
||||
- name: trigger tables
|
||||
type: Concept
|
||||
facet: convention/trigger-tables
|
||||
- name: feedback loop documentation
|
||||
type: Concept
|
||||
facet: convention/feedback-loop-documentation
|
||||
- name: hierarchical scoping
|
||||
type: Concept
|
||||
facet: convention/hierarchical-scoping
|
||||
- name: token efficiency
|
||||
type: Concept
|
||||
facet: convention/token-efficiency
|
||||
- name: just-in-time retrieval
|
||||
type: Concept
|
||||
facet: convention/just-in-time-retrieval
|
||||
- name: cursor rules
|
||||
type: Convention
|
||||
facet: tool/cursor-rules
|
||||
- name: hyperthrive_dev
|
||||
type: Person
|
||||
confidence: AMBIGUOUS
|
||||
|
||||
relationships:
|
||||
- source: Three-Tier Architecture
|
||||
type: documented_in
|
||||
target: arXiv 2602.20478
|
||||
- source: Three-Tier Architecture
|
||||
type: consists_of
|
||||
target: Constitution
|
||||
- source: Three-Tier Architecture
|
||||
type: consists_of
|
||||
target: Domain Specialist Agents
|
||||
- source: Three-Tier Architecture
|
||||
type: consists_of
|
||||
target: Knowledge Base
|
||||
- source: Constitution
|
||||
type: uses
|
||||
target: trigger tables
|
||||
- source: Domain Specialist Agents
|
||||
type: invoked_by
|
||||
target: trigger tables
|
||||
- source: Knowledge Base
|
||||
type: served_via
|
||||
target: MCP
|
||||
- source: CLAUDE.md
|
||||
type: implements
|
||||
target: Constitution
|
||||
confidence: INFERRED
|
||||
- source: AGENTS.md
|
||||
type: implements
|
||||
target: Constitution
|
||||
confidence: INFERRED
|
||||
- source: cursor rules
|
||||
type: implements
|
||||
target: hierarchical scoping
|
||||
- source: Three-Tier Architecture
|
||||
type: enables
|
||||
target: token efficiency
|
||||
- source: progressive-disclosure
|
||||
type: supports
|
||||
target: token efficiency
|
||||
- source: just-in-time retrieval
|
||||
type: supports
|
||||
target: token efficiency
|
||||
- source: feedback loop documentation
|
||||
type: converts_errors_into
|
||||
target: Constitution
|
||||
confidence: INFERRED
|
||||
- source: Codified Context
|
||||
type: described_by
|
||||
target: arXiv 2602.20478
|
||||
- source: Codified Context
|
||||
type: exemplified_by
|
||||
target: github.com/arisvas4/codified-context-infrastructure
|
||||
- source: Claude
|
||||
type: guided_by
|
||||
target: CLAUDE.md
|
||||
- source: Cursor
|
||||
type: guided_by
|
||||
target: cursor rules
|
||||
|
|
@ -0,0 +1,150 @@
|
|||
note_slug: oo-principles-plugin-concept
|
||||
|
||||
entities:
|
||||
- name: OO Principles Plugin
|
||||
type: plugin
|
||||
facet: type/plugin
|
||||
- name: 99 Bottles of OOP
|
||||
type: reference-book
|
||||
facet: domain/oo-design
|
||||
- name: Shameless Green
|
||||
type: principle
|
||||
facet: convention/shameless-green
|
||||
- name: Flocking Rules
|
||||
type: principle
|
||||
facet: convention/flocking-rules
|
||||
- name: Replace Conditional with Polymorphism
|
||||
type: refactoring-technique
|
||||
facet: convention/polymorphism-refactoring
|
||||
- name: Single Responsibility Principle
|
||||
type: principle
|
||||
facet: convention/srp
|
||||
- name: Law of Demeter
|
||||
type: principle
|
||||
facet: convention/law-of-demeter
|
||||
- name: Dependency Injection
|
||||
type: principle
|
||||
facet: convention/dependency-injection
|
||||
- name: Open/Closed Principle
|
||||
type: principle
|
||||
facet: convention/open-closed
|
||||
- name: Process/Routing Layer
|
||||
type: architectural-component
|
||||
facet: type/architecture
|
||||
- name: Mechanic Layer
|
||||
type: architectural-component
|
||||
facet: type/architecture
|
||||
- name: Theory Layer
|
||||
type: architectural-component
|
||||
facet: type/architecture
|
||||
- name: PROCESS.md
|
||||
type: documentation-file
|
||||
facet: convention/process-routing
|
||||
- name: concept cards
|
||||
type: documentation-format
|
||||
- name: NotebookLM notebook
|
||||
type: knowledge-base
|
||||
facet: tool/notebooklm
|
||||
- name: Phase 1
|
||||
type: development-phase
|
||||
- name: Phase 2
|
||||
type: development-phase
|
||||
- name: Phase 3
|
||||
type: development-phase
|
||||
- name: 4-phase development lifecycle
|
||||
type: process-model
|
||||
- name: Approach 1: Process-First Entry Point
|
||||
type: architectural-approach
|
||||
- name: Approach 2: Situational Trigger Files
|
||||
type: architectural-approach
|
||||
- name: Approach 3: Phase-Bundled Context Packs
|
||||
type: architectural-approach
|
||||
- name: Approach 4: Annotated Process Graph
|
||||
type: architectural-approach
|
||||
- name: conventions/oo-principles/
|
||||
type: workspace-structure
|
||||
facet: project/hyperthrive
|
||||
- name: hyperthrive_dev
|
||||
type: project
|
||||
facet: project/hyperthrive
|
||||
- name: Codified Context paper
|
||||
type: research-reference
|
||||
- name: TDD
|
||||
type: principle
|
||||
facet: convention/tdd
|
||||
- name: QUICK_REFERENCE.md
|
||||
type: documentation-file
|
||||
- name: role bundles
|
||||
type: documentation-format
|
||||
- name: AI Coding Conventions Organization — External Research Synthesis
|
||||
type: reference-document
|
||||
- name: 99 Bottles OOP — Full Software Design Process Map
|
||||
type: reference-document
|
||||
- name: CLAUDE.md
|
||||
type: documentation-file
|
||||
facet: tool/claude-code
|
||||
|
||||
relationships:
|
||||
- source: OO Principles Plugin
|
||||
type: is designed to encode
|
||||
target: 4-phase development lifecycle
|
||||
- source: OO Principles Plugin
|
||||
type: models architecture on
|
||||
target: Process/Routing Layer
|
||||
- source: OO Principles Plugin
|
||||
type: models architecture on
|
||||
target: Mechanic Layer
|
||||
- source: OO Principles Plugin
|
||||
type: models architecture on
|
||||
target: Theory Layer
|
||||
- source: Mechanic Layer
|
||||
type: should contain recipes for
|
||||
target: Flocking Rules
|
||||
- source: Mechanic Layer
|
||||
type: should contain recipes for
|
||||
target: Replace Conditional with Polymorphism
|
||||
- source: Theory Layer
|
||||
type: contains explanatory material for
|
||||
target: concept cards
|
||||
- source: Process/Routing Layer
|
||||
type: references
|
||||
target: Mechanic Layer
|
||||
- source: Flocking Rules
|
||||
type: competes with
|
||||
target: Replace Conditional with Polymorphism
|
||||
- source: Dependency Injection
|
||||
type: should be checked before
|
||||
target: Law of Demeter
|
||||
- source: Law of Demeter
|
||||
type: should be checked before
|
||||
target: Single Responsibility Principle
|
||||
- source: Phase 2
|
||||
type: includes opportunistic refactoring guided by
|
||||
target: Law of Demeter
|
||||
- source: Phase 2
|
||||
type: includes opportunistic refactoring guided by
|
||||
target: Dependency Injection
|
||||
- source: Approach 1: Process-First Entry Point
|
||||
type: is recommended for
|
||||
target: OO Principles Plugin
|
||||
- source: NotebookLM notebook
|
||||
type: serves as knowledge base for
|
||||
target: OO Principles Plugin
|
||||
- source: conventions/oo-principles/
|
||||
type: is baseline for
|
||||
target: OO Principles Plugin
|
||||
- source: hyperthrive_dev
|
||||
type: is source context for
|
||||
target: OO Principles Plugin
|
||||
- source: OO Principles Plugin
|
||||
type: based on
|
||||
target: 99 Bottles of OOP
|
||||
- source: Process/Routing Layer
|
||||
type: encodes
|
||||
target: 4-phase development lifecycle
|
||||
- source: Phase 1
|
||||
type: applies
|
||||
target: Shameless Green
|
||||
- source: Phase 3
|
||||
type: requires checking
|
||||
target: Open/Closed Principle
|
||||
|
|
@ -0,0 +1,159 @@
|
|||
# --- Graphify extraction fragment ---
|
||||
note_slug: oo-principles-plugin-concept
|
||||
|
||||
entities:
|
||||
- name: OO Principles Plugin
|
||||
type: plugin
|
||||
facet: type/plugin
|
||||
- name: oo-principles conventions structure
|
||||
type: conventions structure
|
||||
facet: convention/oo-principles
|
||||
- name: 99 Bottles of OOP
|
||||
type: book
|
||||
facet: domain/object-oriented-design
|
||||
- name: NotebookLM notebook
|
||||
type: knowledge base resource
|
||||
facet: tool/notebooklm
|
||||
- name: Codified Context paper
|
||||
type: research paper
|
||||
- name: TDD
|
||||
type: principle
|
||||
facet: convention/tdd
|
||||
- name: Shameless Green
|
||||
type: principle
|
||||
facet: convention/shameless-green
|
||||
- name: Flocking Rules
|
||||
type: principle
|
||||
facet: convention/flocking-rules
|
||||
- name: SRP
|
||||
type: principle
|
||||
facet: convention/srp
|
||||
- name: LoD
|
||||
type: principle
|
||||
facet: convention/law-of-demeter
|
||||
- name: DI
|
||||
type: principle
|
||||
facet: convention/dependency-injection
|
||||
- name: Replace Conditional with Polymorphism
|
||||
type: principle
|
||||
facet: convention/replace-conditional-with-polymorphism
|
||||
- name: Open/Closed
|
||||
type: principle
|
||||
facet: convention/open-closed
|
||||
- name: 4-phase development lifecycle
|
||||
type: process model
|
||||
- name: QUICK_REFERENCE.md
|
||||
type: file
|
||||
- name: cards/
|
||||
type: directory
|
||||
- name: bundles/
|
||||
type: directory
|
||||
- name: Two-Layer Hybrid
|
||||
type: architecture
|
||||
- name: Process/Routing Layer
|
||||
type: layer
|
||||
- name: Mechanic Layer
|
||||
type: layer
|
||||
- name: Theory Layer
|
||||
type: layer
|
||||
- name: PROCESS.md
|
||||
type: file
|
||||
- name: Approach 1 - Process-First Entry Point
|
||||
type: architectural approach
|
||||
- name: Approach 2 - Situational Trigger Files
|
||||
type: architectural approach
|
||||
- name: Approach 3 - Phase-Bundled Context Packs
|
||||
type: architectural approach
|
||||
- name: Approach 4 - Annotated Process Graph
|
||||
type: architectural approach
|
||||
- name: Phase 2
|
||||
type: development phase
|
||||
- name: three-tier architecture
|
||||
type: architecture
|
||||
- name: CLAUDE.md
|
||||
type: file
|
||||
- name: 99 Bottles OOP — Full Software Design Process Map
|
||||
type: note
|
||||
- name: AI Coding Conventions Organization — External Research Synthesis
|
||||
type: note
|
||||
|
||||
relationships:
|
||||
- source: OO Principles Plugin
|
||||
type: evolves from
|
||||
target: oo-principles conventions structure
|
||||
- source: OO Principles Plugin
|
||||
type: based on study of
|
||||
target: 99 Bottles of OOP
|
||||
- source: OO Principles Plugin
|
||||
type: informed by research on
|
||||
target: Codified Context paper
|
||||
- source: OO Principles Plugin
|
||||
type: can use as knowledge base layer
|
||||
target: NotebookLM notebook
|
||||
- source: oo-principles conventions structure
|
||||
type: contains
|
||||
target: QUICK_REFERENCE.md
|
||||
- source: oo-principles conventions structure
|
||||
type: contains
|
||||
target: cards/
|
||||
- source: oo-principles conventions structure
|
||||
type: contains
|
||||
target: bundles/
|
||||
- source: Two-Layer Hybrid
|
||||
type: includes
|
||||
target: Process/Routing Layer
|
||||
- source: Two-Layer Hybrid
|
||||
type: includes
|
||||
target: Mechanic Layer
|
||||
- source: Two-Layer Hybrid
|
||||
type: includes
|
||||
target: Theory Layer
|
||||
- source: Process/Routing Layer
|
||||
type: encodes
|
||||
target: 4-phase development lifecycle
|
||||
- source: Process/Routing Layer
|
||||
type: contains
|
||||
target: PROCESS.md
|
||||
- source: Process/Routing Layer
|
||||
type: references
|
||||
target: Mechanic Layer
|
||||
- source: Theory Layer
|
||||
type: retains
|
||||
target: cards/
|
||||
confidence: INFERRED
|
||||
- source: Approach 1 - Process-First Entry Point
|
||||
type: adds as entry point
|
||||
target: PROCESS.md
|
||||
- source: 99 Bottles of OOP
|
||||
type: defines
|
||||
target: Shameless Green
|
||||
confidence: INFERRED
|
||||
- source: 99 Bottles of OOP
|
||||
type: defines
|
||||
target: Flocking Rules
|
||||
confidence: INFERRED
|
||||
- source: Phase 2
|
||||
type: addresses
|
||||
target: LoD
|
||||
- source: Phase 2
|
||||
type: addresses
|
||||
target: DI
|
||||
- source: Replace Conditional with Polymorphism
|
||||
type: alternative to
|
||||
target: Flocking Rules
|
||||
- source: 4-phase development lifecycle
|
||||
type: included as loadable context pack in
|
||||
target: OO Principles Plugin
|
||||
- source: OO Principles Plugin
|
||||
type: modeled on
|
||||
target: three-tier architecture
|
||||
- source: three-tier architecture
|
||||
type: routing entry point is
|
||||
target: CLAUDE.md
|
||||
confidence: INFERRED
|
||||
- source: oo-principles conventions structure
|
||||
type: see also
|
||||
target: 99 Bottles OOP — Full Software Design Process Map
|
||||
- source: oo-principles conventions structure
|
||||
type: see also
|
||||
target: AI Coding Conventions Organization — External Research Synthesis
|
||||
|
|
@ -0,0 +1,126 @@
|
|||
# --- Graphify extraction fragment ---
|
||||
note_slug: oo-principles-plugin-concept
|
||||
entities:
|
||||
- name: OO Principles Plugin
|
||||
type: project
|
||||
facet: project/oo-principles-plugin
|
||||
- name: conventions/oo-principles
|
||||
type: directory-structure
|
||||
facet: convention/oo-principles
|
||||
- name: QUICK_REFERENCE.md
|
||||
type: file
|
||||
- name: concept cards
|
||||
type: document-type
|
||||
- name: role bundles
|
||||
type: document-type
|
||||
- name: PROCESS.md
|
||||
type: file
|
||||
- name: 99 Bottles of OOP
|
||||
type: book
|
||||
facet: domain/software-design
|
||||
- name: Codified Context paper
|
||||
type: external-reference
|
||||
- name: NotebookLM
|
||||
type: tool
|
||||
facet: tool/notebooklm
|
||||
- name: Shameless Green
|
||||
type: development-principle
|
||||
facet: convention/shameless-green
|
||||
- name: Flocking Rules
|
||||
type: development-principle
|
||||
facet: convention/flocking-rules
|
||||
- name: Replace Conditional with Polymorphism
|
||||
type: development-principle
|
||||
facet: convention/replace-conditional-with-polymorphism
|
||||
- name: TDD
|
||||
type: development-principle
|
||||
facet: convention/tdd
|
||||
- name: SRP
|
||||
type: development-principle
|
||||
facet: convention/srp
|
||||
- name: Law of Demeter
|
||||
type: development-principle
|
||||
facet: convention/law-of-demeter
|
||||
- name: Dependency Injection
|
||||
type: development-principle
|
||||
facet: convention/dependency-injection
|
||||
- name: 4-phase development lifecycle
|
||||
type: concept
|
||||
facet: convention/4-phase-lifecycle
|
||||
- name: Process-First Entry Point
|
||||
type: architectural-approach
|
||||
- name: Situational Trigger Files
|
||||
type: architectural-approach
|
||||
- name: Phase-Bundled Context Packs
|
||||
type: architectural-approach
|
||||
- name: Annotated Process Graph
|
||||
type: architectural-approach
|
||||
- name: Two-Layer Hybrid architecture
|
||||
type: architectural-pattern
|
||||
- name: hyperthrive_dev
|
||||
type: client
|
||||
facet: client/hyperthrive-dev
|
||||
- name: Rails
|
||||
type: tool
|
||||
facet: tool/rails
|
||||
- name: CLAUDE.md
|
||||
type: file
|
||||
- name: AI Coding Conventions Organization — External Research Synthesis
|
||||
type: note
|
||||
confidence: INFERRED
|
||||
- name: 99 Bottles OOP — Full Software Design Process Map
|
||||
type: note
|
||||
confidence: INFERRED
|
||||
relationships:
|
||||
- source: OO Principles Plugin
|
||||
type: evolved from
|
||||
target: conventions/oo-principles
|
||||
- source: OO Principles Plugin
|
||||
type: uses architecture
|
||||
target: Two-Layer Hybrid architecture
|
||||
- source: Two-Layer Hybrid architecture
|
||||
type: layer 1 is
|
||||
target: PROCESS.md
|
||||
- source: Two-Layer Hybrid architecture
|
||||
type: layer 3 is
|
||||
target: concept cards
|
||||
- source: OO Principles Plugin
|
||||
type: informed by
|
||||
target: 99 Bottles of OOP
|
||||
- source: OO Principles Plugin
|
||||
type: informed by
|
||||
target: Codified Context paper
|
||||
- source: PROCESS.md
|
||||
type: encodes
|
||||
target: 4-phase development lifecycle
|
||||
- source: Flocking Rules
|
||||
type: contrasted with
|
||||
target: Replace Conditional with Polymorphism
|
||||
- source: Shameless Green
|
||||
type: part of
|
||||
target: 4-phase development lifecycle
|
||||
- source: Process-First Entry Point
|
||||
type: recommended over
|
||||
target: Situational Trigger Files
|
||||
- source: Process-First Entry Point
|
||||
type: recommended over
|
||||
target: Phase-Bundled Context Packs
|
||||
- source: Process-First Entry Point
|
||||
type: recommended over
|
||||
target: Annotated Process Graph
|
||||
- source: NotebookLM
|
||||
type: serves as knowledge base for
|
||||
target: OO Principles Plugin
|
||||
- source: hyperthrive_dev
|
||||
type: source workspace for
|
||||
target: conventions/oo-principles
|
||||
- source: CLAUDE.md
|
||||
type: entry point for
|
||||
target: conventions/oo-principles
|
||||
confidence: INFERRED
|
||||
- source: OO Principles Plugin
|
||||
type: links to
|
||||
target: 99 Bottles OOP — Full Software Design Process Map
|
||||
- source: OO Principles Plugin
|
||||
type: links to
|
||||
target: AI Coding Conventions Organization — External Research Synthesis
|
||||
|
|
@ -0,0 +1,177 @@
|
|||
note_slug: pest-control-email-abc-hub
|
||||
|
||||
entities:
|
||||
- name: Pest Control Email A/B/C Test
|
||||
type: experiment
|
||||
- name: Golden 4-Sentence Framework
|
||||
type: email draft variant
|
||||
- name: Revised
|
||||
type: email draft variant
|
||||
- name: Copywriter
|
||||
type: email draft variant
|
||||
- name: pest-control-spring-2026
|
||||
type: campaign
|
||||
facet: project/pest-control-spring-2026
|
||||
- name: Mike Deluca
|
||||
type: persona
|
||||
facet: client/pest-control-owner
|
||||
confidence: INFERRED
|
||||
- name: Sandra Kowalski
|
||||
type: persona
|
||||
facet: client/pest-control-owner
|
||||
confidence: INFERRED
|
||||
- name: Ray Tanner
|
||||
type: persona
|
||||
facet: client/pest-control-owner
|
||||
confidence: INFERRED
|
||||
- name: gatekeeper personas
|
||||
type: judge group
|
||||
- name: pest control owner-operators
|
||||
type: judge group
|
||||
- name: Group 3 Revised
|
||||
type: email group variant
|
||||
- name: Group 7 Copywriter
|
||||
type: email group variant
|
||||
- name: Group 9 Revised
|
||||
type: email group variant
|
||||
- name: Group 2 Copywriter
|
||||
type: email group variant
|
||||
- name: Group 5 Revised
|
||||
type: email group variant
|
||||
- name: Two-scenario contrast format
|
||||
type: copy pattern
|
||||
facet: convention/email-copy-pattern
|
||||
- name: Scenario first, stat second
|
||||
type: structural pattern
|
||||
facet: convention/email-copy-pattern
|
||||
- name: Reframe before solution
|
||||
type: structural pattern
|
||||
facet: convention/email-copy-pattern
|
||||
- name: Single closing question with honest out
|
||||
type: structural pattern
|
||||
facet: convention/email-copy-pattern
|
||||
- name: Review-shaming openers
|
||||
type: anti-pattern
|
||||
facet: convention/email-copy-pattern
|
||||
- name: Fabricated revenue math
|
||||
type: anti-pattern
|
||||
facet: convention/email-copy-pattern
|
||||
- name: Escape-hatch closes
|
||||
type: anti-pattern
|
||||
facet: convention/email-copy-pattern
|
||||
- name: Self-aware meta-openers
|
||||
type: anti-pattern
|
||||
facet: convention/email-copy-pattern
|
||||
- name: Saturday wasp nest scenario
|
||||
type: email hook
|
||||
- name: Scenario A vs. B two-format
|
||||
type: email hook
|
||||
- name: Lead disappearance problem reframe
|
||||
type: email hook
|
||||
- name: Bed bugs at 11pm scenario
|
||||
type: email hook
|
||||
- name: Termite LTV urgency
|
||||
type: email hook
|
||||
- name: GorillaDesk/P&L email
|
||||
type: referenced email version
|
||||
confidence: INFERRED
|
||||
- name: LSA-anchor version
|
||||
type: email variant
|
||||
- name: Driven Results
|
||||
type: citation source
|
||||
confidence: INFERRED
|
||||
|
||||
relationships:
|
||||
- source: Pest Control Email A/B/C Test
|
||||
type: compares
|
||||
target: Golden 4-Sentence Framework
|
||||
- source: Pest Control Email A/B/C Test
|
||||
type: compares
|
||||
target: Revised
|
||||
- source: Pest Control Email A/B/C Test
|
||||
type: compares
|
||||
target: Copywriter
|
||||
- source: Pest Control Email A/B/C Test
|
||||
type: part of campaign
|
||||
target: pest-control-spring-2026
|
||||
- source: Pest Control Email A/B/C Test
|
||||
type: judged by
|
||||
target: gatekeeper personas
|
||||
- source: Pest Control Email A/B/C Test
|
||||
type: judged by
|
||||
target: pest control owner-operators
|
||||
- source: Mike Deluca
|
||||
type: participates in
|
||||
target: Pest Control Email A/B/C Test
|
||||
- source: Sandra Kowalski
|
||||
type: participates in
|
||||
target: Pest Control Email A/B/C Test
|
||||
- source: Ray Tanner
|
||||
type: participates in
|
||||
target: Pest Control Email A/B/C Test
|
||||
- source: Revised
|
||||
type: wins
|
||||
target: Pest Control Email A/B/C Test
|
||||
- source: Copywriter
|
||||
type: runner-up in
|
||||
target: Pest Control Email A/B/C Test
|
||||
- source: Group 3 Revised
|
||||
type: unanimous sweep in
|
||||
target: Pest Control Email A/B/C Test
|
||||
- source: Group 7 Copywriter
|
||||
type: unanimous sweep in
|
||||
target: Pest Control Email A/B/C Test
|
||||
- source: Two-scenario contrast format
|
||||
type: recommended for adoption
|
||||
target: pest-control-spring-2026
|
||||
- source: Scenario first, stat second
|
||||
type: recommended for adoption
|
||||
target: pest-control-spring-2026
|
||||
- source: Reframe before solution
|
||||
type: recommended for adoption
|
||||
target: pest-control-spring-2026
|
||||
- source: Single closing question with honest out
|
||||
type: recommended for adoption
|
||||
target: pest-control-spring-2026
|
||||
- source: Review-shaming openers
|
||||
type: recommended to retire
|
||||
target: pest-control-spring-2026
|
||||
- source: Fabricated revenue math
|
||||
type: recommended to retire
|
||||
target: pest-control-spring-2026
|
||||
- source: Escape-hatch closes
|
||||
type: recommended to retire
|
||||
target: pest-control-spring-2026
|
||||
- source: Self-aware meta-openers
|
||||
type: recommended to retire
|
||||
target: pest-control-spring-2026
|
||||
- source: Group 3 Revised
|
||||
type: uses hook
|
||||
target: Saturday wasp nest scenario
|
||||
- source: Group 7 Copywriter
|
||||
type: uses hook
|
||||
target: Scenario A vs. B two-format
|
||||
- source: Group 9 Revised
|
||||
type: uses hook
|
||||
target: Lead disappearance problem reframe
|
||||
- source: Group 2 Copywriter
|
||||
type: uses hook
|
||||
target: Bed bugs at 11pm scenario
|
||||
- source: Group 5 Revised
|
||||
type: uses hook
|
||||
target: Termite LTV urgency
|
||||
- source: gatekeeper personas
|
||||
type: predicted preference for
|
||||
target: Revised
|
||||
confidence: AMBIGUOUS
|
||||
- source: pest control owner-operators
|
||||
type: prefer
|
||||
target: Revised
|
||||
- source: LSA-anchor version
|
||||
type: works for
|
||||
target: paid-acquisition operators
|
||||
confidence: INFERRED
|
||||
- source: LSA-anchor version
|
||||
type: rejected by
|
||||
target: referral-only operator
|
||||
confidence: INFERRED
|
||||
|
|
@ -0,0 +1,192 @@
|
|||
# --- Graphify extraction fragment ---
|
||||
note_slug: pest-control-email-abc-hub
|
||||
|
||||
entities:
|
||||
- name: Pest Control Email A/B/C Test
|
||||
type: experiment
|
||||
facet: type/experiment
|
||||
- name: pest-control-spring-2026
|
||||
type: campaign
|
||||
facet: project/pest-control-spring-2026
|
||||
- name: Golden 4-Sentence Framework
|
||||
type: copy framework
|
||||
facet: convention/golden-4-sentence-framework
|
||||
- name: draft
|
||||
type: email version
|
||||
- name: revised
|
||||
type: email version
|
||||
- name: copywriter
|
||||
type: email version
|
||||
- name: Mike Deluca
|
||||
type: owner-operator persona
|
||||
- name: Sandra Kowalski
|
||||
type: owner-operator persona
|
||||
- name: Ray Tanner
|
||||
type: owner-operator persona
|
||||
- name: gatekeeper panel
|
||||
type: judging panel
|
||||
- name: owner panel
|
||||
type: judging panel
|
||||
- name: Group 3
|
||||
type: email group
|
||||
- name: Group 7
|
||||
type: email group
|
||||
- name: Group 2
|
||||
type: email group
|
||||
- name: Group 4
|
||||
type: email group
|
||||
- name: Group 5
|
||||
type: email group
|
||||
- name: Group 8
|
||||
type: email group
|
||||
- name: Group 9
|
||||
type: email group
|
||||
- name: Saturday wasp nest scenario
|
||||
type: email hook
|
||||
- name: Scenario A vs. B two-format
|
||||
type: email hook
|
||||
- name: Lead disappearance problem reframe
|
||||
type: email hook
|
||||
- name: Bed bugs at 11pm scenario
|
||||
type: email hook
|
||||
- name: Termite LTV urgency
|
||||
type: email hook
|
||||
- name: Two-scenario contrast format
|
||||
type: structural pattern
|
||||
facet: convention/two-scenario-contrast-format
|
||||
- name: Scenario first, stat second
|
||||
type: structural pattern
|
||||
facet: convention/scenario-first-stat-second
|
||||
- name: Reframe before solution
|
||||
type: structural pattern
|
||||
facet: convention/reframe-before-solution
|
||||
- name: Single closing question with an honest out
|
||||
type: structural pattern
|
||||
facet: convention/single-closing-question-honest-out
|
||||
- name: Fabricated revenue math
|
||||
type: retired pattern
|
||||
- name: Review-shaming openers
|
||||
type: retired pattern
|
||||
- name: Escape-hatch closes
|
||||
type: retired pattern
|
||||
- name: Guessed specificity
|
||||
type: retired pattern
|
||||
- name: Self-aware meta-openers
|
||||
type: retired pattern
|
||||
- name: GorillaDesk/P&L email
|
||||
type: email
|
||||
- name: Driven Results citation
|
||||
type: citation
|
||||
- name: LSA-anchor version
|
||||
type: email version
|
||||
- name: $136,500 ROI projection
|
||||
type: copy claim
|
||||
|
||||
relationships:
|
||||
- source: Pest Control Email A/B/C Test
|
||||
type: tested for campaign
|
||||
target: pest-control-spring-2026
|
||||
- source: Pest Control Email A/B/C Test
|
||||
type: compared
|
||||
target: draft
|
||||
- source: Pest Control Email A/B/C Test
|
||||
type: compared
|
||||
target: revised
|
||||
- source: Pest Control Email A/B/C Test
|
||||
type: compared
|
||||
target: copywriter
|
||||
- source: draft
|
||||
type: built on framework
|
||||
target: Golden 4-Sentence Framework
|
||||
- source: gatekeeper panel
|
||||
type: judged
|
||||
target: Pest Control Email A/B/C Test
|
||||
- source: owner panel
|
||||
type: judged
|
||||
target: Pest Control Email A/B/C Test
|
||||
- source: owner panel
|
||||
type: includes persona
|
||||
target: Mike Deluca
|
||||
- source: owner panel
|
||||
type: includes persona
|
||||
target: Sandra Kowalski
|
||||
- source: owner panel
|
||||
type: includes persona
|
||||
target: Ray Tanner
|
||||
- source: revised
|
||||
type: won overall
|
||||
target: Pest Control Email A/B/C Test
|
||||
- source: copywriter
|
||||
type: led in raw first-place owner votes
|
||||
target: revised
|
||||
- source: Group 3
|
||||
type: swept by version
|
||||
target: revised
|
||||
- source: Group 7
|
||||
type: swept by version
|
||||
target: copywriter
|
||||
- source: Group 3
|
||||
type: uses hook
|
||||
target: Saturday wasp nest scenario
|
||||
- source: Group 3
|
||||
type: cites
|
||||
target: Driven Results citation
|
||||
- source: Group 7
|
||||
type: uses hook
|
||||
target: Scenario A vs. B two-format
|
||||
- source: Group 9
|
||||
type: uses hook
|
||||
target: Lead disappearance problem reframe
|
||||
- source: Group 2
|
||||
type: uses hook
|
||||
target: Bed bugs at 11pm scenario
|
||||
- source: Group 5
|
||||
type: uses hook
|
||||
target: Termite LTV urgency
|
||||
- source: Group 8
|
||||
type: gatekeepers ranked first
|
||||
target: GorillaDesk/P&L email
|
||||
- source: Group 5
|
||||
type: contains copy claim
|
||||
target: $136,500 ROI projection
|
||||
confidence: INFERRED
|
||||
- source: LSA-anchor version
|
||||
type: appears in
|
||||
target: Group 4
|
||||
- source: Group 7
|
||||
type: exemplifies pattern
|
||||
target: Two-scenario contrast format
|
||||
- source: Group 9
|
||||
type: exemplifies pattern
|
||||
target: Reframe before solution
|
||||
confidence: INFERRED
|
||||
- source: Pest Control Email A/B/C Test
|
||||
type: recommends for live sequence
|
||||
target: Group 3
|
||||
- source: Pest Control Email A/B/C Test
|
||||
type: recommends for live sequence
|
||||
target: Group 7
|
||||
- source: Pest Control Email A/B/C Test
|
||||
type: recommends for live sequence
|
||||
target: Group 9
|
||||
- source: Pest Control Email A/B/C Test
|
||||
type: recommends for live sequence
|
||||
target: Group 2
|
||||
- source: Pest Control Email A/B/C Test
|
||||
type: recommends for live sequence
|
||||
target: Group 5
|
||||
- source: Pest Control Email A/B/C Test
|
||||
type: recommends retiring
|
||||
target: Fabricated revenue math
|
||||
- source: Pest Control Email A/B/C Test
|
||||
type: recommends retiring
|
||||
target: Review-shaming openers
|
||||
- source: Pest Control Email A/B/C Test
|
||||
type: recommends retiring
|
||||
target: Escape-hatch closes
|
||||
- source: Pest Control Email A/B/C Test
|
||||
type: recommends retiring
|
||||
target: Guessed specificity
|
||||
- source: Pest Control Email A/B/C Test
|
||||
type: recommends retiring
|
||||
target: Self-aware meta-openers
|
||||
|
|
@ -0,0 +1,104 @@
|
|||
# --- Graphify extraction fragment ---
|
||||
note_slug: pest-control-email-abc-hub
|
||||
|
||||
entities:
|
||||
- name: pest-control-spring-2026
|
||||
type: campaign
|
||||
facet: project/pest-control-spring-2026
|
||||
- name: Golden 4-Sentence Framework
|
||||
type: email framework
|
||||
facet: convention/golden-4-sentence-framework
|
||||
- name: Revised
|
||||
type: email version
|
||||
- name: Copywriter
|
||||
type: email version
|
||||
- name: Draft
|
||||
type: email version
|
||||
- name: Mike Deluca
|
||||
type: owner persona
|
||||
- name: Sandra Kowalski
|
||||
type: owner persona
|
||||
- name: Ray Tanner
|
||||
type: owner persona
|
||||
- name: Driven Results
|
||||
type: organization
|
||||
- name: GorillaDesk
|
||||
type: tool
|
||||
facet: tool/gorillaDesk
|
||||
- name: Two-scenario contrast format
|
||||
type: copy pattern
|
||||
facet: convention/two-scenario-contrast-format
|
||||
- name: Saturday wasp nest scenario
|
||||
type: email hook
|
||||
- name: Scenario A vs. B two-format
|
||||
type: email structure
|
||||
facet: convention/scenario-a-vs-b-two-format
|
||||
- name: lead disappearance problem reframe
|
||||
type: copy pattern
|
||||
facet: convention/lead-disappearance-reframe
|
||||
- name: LSA-anchor version
|
||||
type: email version
|
||||
- name: $136,500 ROI projection
|
||||
type: copy element
|
||||
- name: review-shaming opener
|
||||
type: copy pattern
|
||||
- name: escape-hatch close
|
||||
type: copy pattern
|
||||
- name: cold email
|
||||
type: domain
|
||||
facet: domain/cold-email
|
||||
|
||||
relationships:
|
||||
- source: Revised
|
||||
type: won overall in
|
||||
target: pest-control-spring-2026
|
||||
- source: Copywriter
|
||||
type: runner-up in
|
||||
target: pest-control-spring-2026
|
||||
- source: Draft
|
||||
type: placed third in
|
||||
target: pest-control-spring-2026
|
||||
- source: Mike Deluca
|
||||
type: judged as owner persona in
|
||||
target: pest-control-spring-2026
|
||||
- source: Sandra Kowalski
|
||||
type: judged as owner persona in
|
||||
target: pest-control-spring-2026
|
||||
- source: Ray Tanner
|
||||
type: judged as owner persona in
|
||||
target: pest-control-spring-2026
|
||||
- source: Saturday wasp nest scenario
|
||||
type: achieved unanimous sweep in
|
||||
target: pest-control-spring-2026
|
||||
- source: Scenario A vs. B two-format
|
||||
type: achieved unanimous sweep in
|
||||
target: pest-control-spring-2026
|
||||
- source: Two-scenario contrast format
|
||||
type: earned identical reactions from all owner personas in
|
||||
target: pest-control-spring-2026
|
||||
- source: LSA-anchor version
|
||||
type: requires audience segmentation for
|
||||
target: pest-control-spring-2026
|
||||
- source: $136,500 ROI projection
|
||||
type: recommended to retire from
|
||||
target: pest-control-spring-2026
|
||||
- source: review-shaming opener
|
||||
type: recommended to retire from
|
||||
target: pest-control-spring-2026
|
||||
- source: escape-hatch close
|
||||
type: recommended to retire from
|
||||
target: pest-control-spring-2026
|
||||
- source: Revised
|
||||
type: uses citation from
|
||||
target: Driven Results
|
||||
- source: lead disappearance problem reframe
|
||||
type: appears in
|
||||
target: Revised
|
||||
confidence: INFERRED
|
||||
- source: GorillaDesk
|
||||
type: referenced in email version
|
||||
target: Copywriter
|
||||
confidence: INFERRED
|
||||
- source: Golden 4-Sentence Framework
|
||||
type: basis for
|
||||
target: Draft
|
||||
|
|
@ -0,0 +1,124 @@
|
|||
note_slug: pest-control-enterprise-revenue
|
||||
|
||||
entities:
|
||||
- name: Gemini
|
||||
type: Tool
|
||||
facet: tool/gemini
|
||||
- name: Pest Control Enterprise
|
||||
type: Concept
|
||||
facet: domain/pest-control
|
||||
- name: Recurring Revenue
|
||||
type: Concept
|
||||
- name: Termite Swarming
|
||||
type: Concept
|
||||
facet: domain/pest-control
|
||||
- name: Growing Degree Days
|
||||
type: Convention
|
||||
facet: convention/gdd-forecasting
|
||||
- name: Heatwave-Driven Migrations
|
||||
type: Concept
|
||||
facet: domain/pest-control
|
||||
- name: First Freeze Rodent Push
|
||||
type: Concept
|
||||
facet: domain/pest-control
|
||||
- name: PPC
|
||||
type: Tool
|
||||
facet: tool/ppc
|
||||
- name: SEO
|
||||
type: Tool
|
||||
facet: tool/seo
|
||||
- name: Customer Acquisition Cost
|
||||
type: Concept
|
||||
- name: Route Density
|
||||
type: Concept
|
||||
facet: domain/pest-control
|
||||
- name: Rollins
|
||||
type: Client
|
||||
- name: Rentokil
|
||||
type: Client
|
||||
- name: Ecolab
|
||||
type: Client
|
||||
- name: Terminix
|
||||
type: Client
|
||||
- name: Churn Rate
|
||||
type: Concept
|
||||
- name: Communication Failure Patterns
|
||||
type: Concept
|
||||
facet: domain/pest-control
|
||||
- name: Northeast Region
|
||||
type: Concept
|
||||
facet: scope/geographic
|
||||
- name: Mid-Atlantic Region
|
||||
type: Concept
|
||||
facet: scope/geographic
|
||||
- name: Southeast Region
|
||||
type: Concept
|
||||
facet: scope/geographic
|
||||
- name: Southwest Region
|
||||
type: Concept
|
||||
facet: scope/geographic
|
||||
- name: Northwest Region
|
||||
type: Concept
|
||||
facet: scope/geographic
|
||||
- name: Commercial Pest Control
|
||||
type: Concept
|
||||
facet: domain/pest-control
|
||||
- name: Offshore Customer Service
|
||||
type: Concept
|
||||
confidence: INFERRED
|
||||
- name: M&A Valuation
|
||||
type: Concept
|
||||
facet: domain/pest-control
|
||||
|
||||
relationships:
|
||||
- source: Pest Control Enterprise
|
||||
type: achieves recurring revenue through
|
||||
target: Recurring Revenue
|
||||
- source: Pest Control Enterprise
|
||||
type: experiences lead generation spike during
|
||||
target: Termite Swarming
|
||||
- source: Growing Degree Days
|
||||
type: predicts onset of
|
||||
target: Termite Swarming
|
||||
- source: Pest Control Enterprise
|
||||
type: experiences demand spike during
|
||||
target: Heatwave-Driven Migrations
|
||||
- source: Pest Control Enterprise
|
||||
type: experiences demand spike during
|
||||
target: First Freeze Rodent Push
|
||||
- source: PPC
|
||||
type: captures demand during
|
||||
target: Termite Swarming
|
||||
- source: SEO
|
||||
type: reduces long-term CAC for
|
||||
target: Pest Control Enterprise
|
||||
- source: Route Density
|
||||
type: drives revenue per technician in
|
||||
target: Pest Control Enterprise
|
||||
- source: Rollins
|
||||
type: consolidates market share in
|
||||
target: Pest Control Enterprise
|
||||
- source: Rentokil
|
||||
type: consolidates market share in
|
||||
target: Pest Control Enterprise
|
||||
- source: Ecolab
|
||||
type: consolidates market share in
|
||||
target: Pest Control Enterprise
|
||||
- source: Terminix
|
||||
type: consolidates market share in
|
||||
target: Pest Control Enterprise
|
||||
- source: Communication Failure Patterns
|
||||
type: drives churn in
|
||||
target: Pest Control Enterprise
|
||||
- source: Churn Rate
|
||||
type: correlates with customer satisfaction in
|
||||
target: Communication Failure Patterns
|
||||
- source: Offshore Customer Service
|
||||
type: contributes to
|
||||
target: Communication Failure Patterns
|
||||
- source: Commercial Pest Control
|
||||
type: represents market segment of
|
||||
target: Pest Control Enterprise
|
||||
- source: Gemini
|
||||
type: source for
|
||||
target: Pest Control Enterprise
|
||||
|
|
@ -0,0 +1,224 @@
|
|||
# --- Graphify extraction fragment ---
|
||||
note_slug: pest-control-enterprise-revenue
|
||||
|
||||
entities:
|
||||
- name: Pest Control Enterprise
|
||||
type: Domain
|
||||
facet: domain/pest-control
|
||||
- name: Recurring Revenue
|
||||
type: Concept
|
||||
- name: ACV
|
||||
type: Concept
|
||||
- name: Monthly maintenance
|
||||
type: Concept
|
||||
- name: Bi-monthly
|
||||
type: Concept
|
||||
- name: Tri-annual
|
||||
type: Concept
|
||||
- name: Quarterly
|
||||
type: Concept
|
||||
- name: Premium/Gold bundled
|
||||
type: Concept
|
||||
- name: Termite warranty renewal
|
||||
type: Concept
|
||||
- name: Initial service fee
|
||||
type: Concept
|
||||
- name: CAC
|
||||
type: Concept
|
||||
- name: LTV
|
||||
type: Concept
|
||||
- name: Termite monitoring/baiting
|
||||
type: Concept
|
||||
- name: Mosquito seasonal
|
||||
type: Concept
|
||||
- name: Rodent exclusion
|
||||
type: Concept
|
||||
- name: Wasp/hornet removal
|
||||
type: Concept
|
||||
- name: Bed bug remediation
|
||||
type: Concept
|
||||
- name: Communication Failure Patterns
|
||||
type: Concept
|
||||
- name: '"No one answered"'
|
||||
type: Concept
|
||||
- name: '"Voicemail full"'
|
||||
type: Concept
|
||||
- name: '"Dropped calls and ghosting"'
|
||||
type: Concept
|
||||
- name: Offshore agent disconnect
|
||||
type: Concept
|
||||
- name: CPL
|
||||
type: Concept
|
||||
- name: Churn
|
||||
type: Concept
|
||||
- name: Climatologically-Driven Lead Seasonality
|
||||
type: Concept
|
||||
- name: Termite Swarming
|
||||
type: Concept
|
||||
- name: Subterranean termites
|
||||
type: Concept
|
||||
- name: Growing Degree Days (GDD)
|
||||
type: Concept
|
||||
- name: PPC
|
||||
type: Concept
|
||||
- name: Heatwave-Driven Migrations
|
||||
type: Concept
|
||||
- name: First Freeze Rodent Push
|
||||
type: Concept
|
||||
- name: Northeast
|
||||
type: Concept
|
||||
- name: Mid-Atlantic
|
||||
type: Concept
|
||||
- name: Southeast
|
||||
type: Concept
|
||||
- name: Southwest
|
||||
type: Concept
|
||||
- name: Northwest
|
||||
type: Concept
|
||||
- name: Operational Unit Economics
|
||||
type: Concept
|
||||
- name: Revenue per technician
|
||||
type: Concept
|
||||
- name: COGS
|
||||
type: Concept
|
||||
- name: Direct labor
|
||||
type: Concept
|
||||
- name: Materials/chemicals
|
||||
type: Concept
|
||||
- name: Marketing/lead gen
|
||||
type: Concept
|
||||
- name: Vehicle ops
|
||||
type: Concept
|
||||
- name: Admin/management
|
||||
type: Concept
|
||||
- name: EBITDA
|
||||
type: Concept
|
||||
- name: SEO
|
||||
type: Concept
|
||||
- name: Marketing allocation model
|
||||
type: Concept
|
||||
- name: Commercial
|
||||
type: Concept
|
||||
- name: M&A
|
||||
type: Concept
|
||||
- name: Big Four
|
||||
type: Concept
|
||||
- name: Rollins
|
||||
type: Client
|
||||
- name: Rentokil
|
||||
type: Client
|
||||
- name: Ecolab
|
||||
type: Client
|
||||
- name: Terminix
|
||||
type: Client
|
||||
- name: Revenue multiples
|
||||
type: Concept
|
||||
- name: SDE multiples
|
||||
type: Concept
|
||||
- name: Route density
|
||||
type: Concept
|
||||
- name: Gemini
|
||||
type: Tool
|
||||
- name: SOPs
|
||||
type: Concept
|
||||
|
||||
relationships:
|
||||
- source: Recurring Revenue
|
||||
type: comprises_share_of
|
||||
target: Pest Control Enterprise
|
||||
- source: Recurring Revenue
|
||||
type: determines
|
||||
target: M&A
|
||||
- source: Initial service fee
|
||||
type: offsets
|
||||
target: CAC
|
||||
- source: Initial service fee
|
||||
type: inflates
|
||||
target: ACV
|
||||
- source: Termite warranty renewal
|
||||
type: is_a
|
||||
target: ACV
|
||||
- source: Monthly maintenance
|
||||
type: is_a
|
||||
target: ACV
|
||||
- source: Communication Failure Patterns
|
||||
type: drives
|
||||
target: Churn
|
||||
- source: '"No one answered"'
|
||||
type: wastes
|
||||
target: CPL
|
||||
- source: Offshore agent disconnect
|
||||
type: misidentifies
|
||||
target: Subterranean termites
|
||||
- source: Termite Swarming
|
||||
type: is_event_of
|
||||
target: Climatologically-Driven Lead Seasonality
|
||||
- source: Subterranean termites
|
||||
type: cause
|
||||
target: Termite Swarming
|
||||
- source: Growing Degree Days (GDD)
|
||||
type: predicts
|
||||
target: Termite Swarming
|
||||
- source: PPC
|
||||
type: increases_with
|
||||
target: Growing Degree Days (GDD)
|
||||
- source: Heatwave-Driven Migrations
|
||||
type: is_event_of
|
||||
target: Climatologically-Driven Lead Seasonality
|
||||
- source: First Freeze Rodent Push
|
||||
type: is_event_of
|
||||
target: Climatologically-Driven Lead Seasonality
|
||||
- source: First Freeze Rodent Push
|
||||
type: triggers
|
||||
target: Rodent exclusion
|
||||
- source: Northeast
|
||||
type: experiences
|
||||
target: Subterranean termites
|
||||
- source: Mid-Atlantic
|
||||
type: experiences
|
||||
target: Subterranean termites
|
||||
- source: Direct labor
|
||||
type: component_of
|
||||
target: COGS
|
||||
- source: Materials/chemicals
|
||||
type: component_of
|
||||
target: COGS
|
||||
- source: Marketing/lead gen
|
||||
type: component_of
|
||||
target: COGS
|
||||
- source: Vehicle ops
|
||||
type: component_of
|
||||
target: COGS
|
||||
- source: Admin/management
|
||||
type: component_of
|
||||
target: COGS
|
||||
- source: Marketing allocation model
|
||||
type: allocates_to
|
||||
target: PPC
|
||||
- source: Marketing allocation model
|
||||
type: allocates_to
|
||||
target: SEO
|
||||
- source: Big Four
|
||||
type: includes
|
||||
target: Rollins
|
||||
- source: Big Four
|
||||
type: includes
|
||||
target: Rentokil
|
||||
- source: Big Four
|
||||
type: includes
|
||||
target: Ecolab
|
||||
- source: Big Four
|
||||
type: includes
|
||||
target: Terminix
|
||||
- source: Big Four
|
||||
type: consolidates
|
||||
target: Pest Control Enterprise
|
||||
- source: SOPs
|
||||
type: increases
|
||||
target: SDE multiples
|
||||
- source: Route density
|
||||
type: drives
|
||||
target: M&A
|
||||
- source: Gemini
|
||||
type: sourced
|
||||
target: Pest Control Enterprise
|
||||
|
|
@ -0,0 +1,206 @@
|
|||
# --- Graphify extraction fragment ---
|
||||
note_slug: pest-control-enterprise-revenue
|
||||
entities:
|
||||
- name: Pest Control Enterprise
|
||||
type: business-segment
|
||||
facet: domain/pest-control
|
||||
- name: Recurring Revenue
|
||||
type: financial-concept
|
||||
facet: domain/pest-control
|
||||
- name: Monthly Maintenance Program
|
||||
type: service-program
|
||||
facet: domain/pest-control
|
||||
- name: Bi-monthly Program
|
||||
type: service-program
|
||||
facet: domain/pest-control
|
||||
- name: Tri-annual Program
|
||||
type: service-program
|
||||
facet: domain/pest-control
|
||||
- name: Quarterly Program
|
||||
type: service-program
|
||||
facet: domain/pest-control
|
||||
- name: Premium/Gold Bundled Program
|
||||
type: service-program
|
||||
facet: domain/pest-control
|
||||
- name: Termite Warranty Renewal
|
||||
type: service-program
|
||||
facet: domain/pest-control
|
||||
- name: Initial Service Fee
|
||||
type: pricing-mechanism
|
||||
facet: domain/pest-control
|
||||
- name: Termite Monitoring/Baiting
|
||||
type: specialty-service
|
||||
facet: domain/pest-control
|
||||
- name: Mosquito Seasonal Service
|
||||
type: specialty-service
|
||||
facet: domain/pest-control
|
||||
- name: Rodent Exclusion
|
||||
type: specialty-service
|
||||
facet: domain/pest-control
|
||||
- name: Wasp/Hornet Removal
|
||||
type: specialty-service
|
||||
facet: domain/pest-control
|
||||
- name: Bed Bug Remediation
|
||||
type: specialty-service
|
||||
facet: domain/pest-control
|
||||
- name: Missed Call Failure
|
||||
type: communication-failure-pattern
|
||||
facet: domain/pest-control
|
||||
- name: Voicemail Full Failure
|
||||
type: communication-failure-pattern
|
||||
facet: domain/pest-control
|
||||
- name: Dropped Calls and Ghosting
|
||||
type: communication-failure-pattern
|
||||
facet: domain/pest-control
|
||||
- name: Offshore Agent Disconnect
|
||||
type: communication-failure-pattern
|
||||
facet: domain/pest-control
|
||||
- name: Annual Churn
|
||||
type: operational-metric
|
||||
facet: domain/pest-control
|
||||
- name: Termite Swarming Season
|
||||
type: seasonal-lead-trigger
|
||||
facet: domain/pest-control
|
||||
- name: Heatwave-Driven Migrations
|
||||
type: seasonal-lead-trigger
|
||||
facet: domain/pest-control
|
||||
- name: First Freeze Rodent Push
|
||||
type: seasonal-lead-trigger
|
||||
facet: domain/pest-control
|
||||
- name: Growing Degree Days
|
||||
type: forecasting-concept
|
||||
facet: domain/pest-control
|
||||
- name: Revenue per Technician
|
||||
type: operational-metric
|
||||
facet: domain/pest-control
|
||||
- name: COGS Structure
|
||||
type: financial-concept
|
||||
facet: domain/pest-control
|
||||
- name: Marketing Allocation Model
|
||||
type: operational-framework
|
||||
facet: domain/pest-control
|
||||
- name: Commercial Pest Control
|
||||
type: business-segment
|
||||
facet: domain/pest-control
|
||||
- name: Rollins
|
||||
type: company
|
||||
facet: domain/pest-control
|
||||
- name: Rentokil
|
||||
type: company
|
||||
facet: domain/pest-control
|
||||
- name: Ecolab
|
||||
type: company
|
||||
facet: domain/pest-control
|
||||
- name: Terminix
|
||||
type: company
|
||||
facet: domain/pest-control
|
||||
- name: Big Four
|
||||
type: company-group
|
||||
facet: domain/pest-control
|
||||
- name: M&A Valuation
|
||||
type: financial-concept
|
||||
facet: domain/pest-control
|
||||
- name: SDE Multiple
|
||||
type: financial-metric
|
||||
facet: domain/pest-control
|
||||
- name: Revenue Multiple
|
||||
type: financial-metric
|
||||
facet: domain/pest-control
|
||||
- name: Route Density
|
||||
type: operational-metric
|
||||
facet: domain/pest-control
|
||||
- name: Customer Retention
|
||||
type: operational-metric
|
||||
facet: domain/pest-control
|
||||
relationships:
|
||||
- source: Pest Control Enterprise
|
||||
type: generates
|
||||
target: Recurring Revenue
|
||||
- source: Recurring Revenue
|
||||
type: determines
|
||||
target: M&A Valuation
|
||||
- source: Monthly Maintenance Program
|
||||
type: is-tier-of
|
||||
target: Recurring Revenue
|
||||
- source: Bi-monthly Program
|
||||
type: is-tier-of
|
||||
target: Recurring Revenue
|
||||
- source: Tri-annual Program
|
||||
type: is-tier-of
|
||||
target: Recurring Revenue
|
||||
- source: Quarterly Program
|
||||
type: is-tier-of
|
||||
target: Recurring Revenue
|
||||
- source: Premium/Gold Bundled Program
|
||||
type: is-tier-of
|
||||
target: Recurring Revenue
|
||||
- source: Termite Warranty Renewal
|
||||
type: is-tier-of
|
||||
target: Recurring Revenue
|
||||
- source: Initial Service Fee
|
||||
type: offsets-CAC-for
|
||||
target: Pest Control Enterprise
|
||||
- source: Missed Call Failure
|
||||
type: increases
|
||||
target: Annual Churn
|
||||
confidence: INFERRED
|
||||
- source: Voicemail Full Failure
|
||||
type: increases
|
||||
target: Annual Churn
|
||||
confidence: INFERRED
|
||||
- source: Dropped Calls and Ghosting
|
||||
type: increases
|
||||
target: Annual Churn
|
||||
confidence: INFERRED
|
||||
- source: Offshore Agent Disconnect
|
||||
type: causes
|
||||
target: Dropped Calls and Ghosting
|
||||
confidence: INFERRED
|
||||
- source: Termite Swarming Season
|
||||
type: drives-demand-for
|
||||
target: Termite Monitoring/Baiting
|
||||
- source: Heatwave-Driven Migrations
|
||||
type: drives-demand-for
|
||||
target: Mosquito Seasonal Service
|
||||
- source: First Freeze Rodent Push
|
||||
type: drives-demand-for
|
||||
target: Rodent Exclusion
|
||||
- source: Growing Degree Days
|
||||
type: used-to-predict
|
||||
target: Termite Swarming Season
|
||||
- source: Marketing Allocation Model
|
||||
type: allocates-spend-toward
|
||||
target: Termite Swarming Season
|
||||
- source: Rollins
|
||||
type: member-of
|
||||
target: Big Four
|
||||
- source: Rentokil
|
||||
type: member-of
|
||||
target: Big Four
|
||||
- source: Ecolab
|
||||
type: member-of
|
||||
target: Big Four
|
||||
- source: Terminix
|
||||
type: member-of
|
||||
target: Big Four
|
||||
- source: Big Four
|
||||
type: consolidates
|
||||
target: Pest Control Enterprise
|
||||
- source: M&A Valuation
|
||||
type: measured-by
|
||||
target: SDE Multiple
|
||||
- source: M&A Valuation
|
||||
type: measured-by
|
||||
target: Revenue Multiple
|
||||
- source: Route Density
|
||||
type: premium-driver-for
|
||||
target: M&A Valuation
|
||||
- source: Customer Retention
|
||||
type: premium-driver-for
|
||||
target: M&A Valuation
|
||||
- source: Commercial Pest Control
|
||||
type: segment-of
|
||||
target: Pest Control Enterprise
|
||||
- source: Annual Churn
|
||||
type: inversely-affects
|
||||
target: Customer Retention
|
||||
|
|
@ -0,0 +1,136 @@
|
|||
note_slug: pest-control-sms-market-research-stats
|
||||
|
||||
entities:
|
||||
- name: MIT/InsideSales.com Lead Response Management Study
|
||||
type: research study
|
||||
- name: Harvard Business Review
|
||||
type: publication
|
||||
facet: domain/sales-operations
|
||||
- name: Velocify Ultimate Contact Strategy Study
|
||||
type: research study
|
||||
- name: InsideSales.com 2021
|
||||
type: research study
|
||||
- name: HomeAdvisor
|
||||
type: company
|
||||
facet: tool/homeadvisor
|
||||
- name: Angi
|
||||
type: company
|
||||
facet: tool/angi
|
||||
- name: CallRail
|
||||
type: company
|
||||
facet: tool/callrail
|
||||
- name: PATLive
|
||||
type: company
|
||||
- name: Forbes
|
||||
type: publication
|
||||
- name: SellCell
|
||||
type: publication
|
||||
- name: Driven Results
|
||||
type: research organization
|
||||
- name: HubSpot
|
||||
type: company
|
||||
facet: tool/hubspot
|
||||
- name: Google Ads
|
||||
type: tool/google-ads
|
||||
facet: tool/google-ads
|
||||
- name: Google Local Services Ads
|
||||
type: tool/google-local-services-ads
|
||||
facet: tool/google-local-services-ads
|
||||
- name: Thumbtack
|
||||
type: company
|
||||
facet: tool/thumbtack
|
||||
- name: Coalmarch
|
||||
type: company
|
||||
- name: Valve+Meter
|
||||
type: research organization
|
||||
- name: Invoca
|
||||
type: company
|
||||
- name: PPMA
|
||||
type: organization
|
||||
- name: Scorpion
|
||||
type: company
|
||||
- name: NPMA
|
||||
type: organization
|
||||
- name: PCO Bookkeepers
|
||||
type: organization
|
||||
- name: Gartner
|
||||
type: company
|
||||
- name: D7 Networks
|
||||
type: company
|
||||
- name: CTIA
|
||||
type: organization
|
||||
- name: TransUnion
|
||||
type: company
|
||||
- name: Avochato
|
||||
type: company
|
||||
- name: Leads360
|
||||
type: company
|
||||
- name: Briostack
|
||||
type: company
|
||||
- name: Cube Creative Design
|
||||
type: company
|
||||
- name: ServiceTitan
|
||||
type: company
|
||||
facet: tool/servicetitan
|
||||
- name: Point Loma Electric & Plumbing
|
||||
type: company
|
||||
- name: Hatch
|
||||
type: company
|
||||
- name: Shafer Services
|
||||
type: company
|
||||
- name: Aruza Pest Control
|
||||
type: company
|
||||
- name: After-Hours SMS Lead Capture
|
||||
type: product concept
|
||||
facet: project/after-hours-sms-lead-capture
|
||||
- name: pest control industry
|
||||
type: market segment
|
||||
facet: domain/pest-control
|
||||
|
||||
relationships:
|
||||
- source: MIT/InsideSales.com Lead Response Management Study
|
||||
type: documents lead response time decay
|
||||
target: After-Hours SMS Lead Capture
|
||||
confidence: INFERRED
|
||||
- source: Harvard Business Review
|
||||
type: documents lead qualification speed
|
||||
target: After-Hours SMS Lead Capture
|
||||
confidence: INFERRED
|
||||
- source: Velocify Ultimate Contact Strategy Study
|
||||
type: documents conversion decay curve
|
||||
target: After-Hours SMS Lead Capture
|
||||
confidence: INFERRED
|
||||
- source: HomeAdvisor
|
||||
type: shows after-hours call volume in
|
||||
target: pest control industry
|
||||
- source: Angi
|
||||
type: shows after-hours call volume in
|
||||
target: pest control industry
|
||||
- source: CallRail
|
||||
type: documents call volume peaks in
|
||||
target: pest control industry
|
||||
- source: PATLive
|
||||
type: measures caller abandonment in
|
||||
target: pest control industry
|
||||
- source: Driven Results
|
||||
type: measured real-world conversion outcomes for
|
||||
target: After-Hours SMS Lead Capture
|
||||
confidence: INFERRED
|
||||
- source: ServiceTitan
|
||||
type: tracks adoption gap in
|
||||
target: pest control industry
|
||||
- source: NPMA
|
||||
type: published gross margin data for
|
||||
target: pest control industry
|
||||
- source: Point Loma Electric & Plumbing
|
||||
type: achieved booking rate boost with
|
||||
target: After-Hours SMS Lead Capture
|
||||
confidence: INFERRED
|
||||
- source: Shafer Services
|
||||
type: achieved lead booking increase with
|
||||
target: After-Hours SMS Lead Capture
|
||||
confidence: INFERRED
|
||||
- source: Aruza Pest Control
|
||||
type: achieved cost reduction with
|
||||
target: After-Hours SMS Lead Capture
|
||||
confidence: INFERRED
|
||||
|
|
@ -0,0 +1,134 @@
|
|||
# --- Graphify extraction fragment ---
|
||||
note_slug: pest-control-sms-market-research-stats
|
||||
|
||||
entities:
|
||||
- name: After-Hours SMS Lead Capture & Qualification
|
||||
type: product
|
||||
facet: project/niche-automation-prospecting
|
||||
- name: pest control companies
|
||||
type: industry
|
||||
facet: domain/pest-control
|
||||
- name: niche-automation-prospecting
|
||||
type: initiative
|
||||
facet: project/niche-automation-prospecting
|
||||
- name: pest-control-spring-2026
|
||||
type: campaign
|
||||
facet: project/pest-control-spring-2026
|
||||
- name: MIT/InsideSales.com Lead Response Management Study (2007)
|
||||
type: study
|
||||
- name: Harvard Business Review (March 2011)
|
||||
type: study
|
||||
- name: Velocify "Ultimate Contact Strategy" Study (2012)
|
||||
type: study
|
||||
- name: InsideSales.com 2021
|
||||
type: study
|
||||
- name: HomeAdvisor
|
||||
type: company
|
||||
- name: Angi
|
||||
type: company
|
||||
- name: CallRail
|
||||
type: company
|
||||
- name: PATLive
|
||||
type: company
|
||||
- name: Forbes
|
||||
type: publication
|
||||
- name: SellCell 2024
|
||||
type: study
|
||||
- name: Driven Results
|
||||
type: company
|
||||
- name: HubSpot 2023
|
||||
type: study
|
||||
- name: Google Ads (PPC)
|
||||
type: channel
|
||||
facet: tool/google-ads
|
||||
- name: Google Local Services Ads
|
||||
type: channel
|
||||
facet: tool/google-local-services-ads
|
||||
- name: Thumbtack
|
||||
type: company
|
||||
- name: Coalmarch
|
||||
type: company
|
||||
- name: Valve+Meter
|
||||
type: company
|
||||
- name: Invoca
|
||||
type: company
|
||||
- name: NPMA & PCO Bookkeepers 2025 Cost Study
|
||||
type: study
|
||||
- name: PPMA survey
|
||||
type: study
|
||||
- name: Scorpion
|
||||
type: company
|
||||
- name: Gartner 2016
|
||||
type: study
|
||||
- name: D7 Networks
|
||||
type: company
|
||||
- name: CTIA
|
||||
type: organization
|
||||
- name: TransUnion
|
||||
type: company
|
||||
- name: Avochato
|
||||
type: company
|
||||
- name: Velocify/Leads360
|
||||
type: study
|
||||
- name: Briostack
|
||||
type: company
|
||||
- name: Cube Creative Design
|
||||
type: company
|
||||
- name: ServiceTitan 2025
|
||||
type: study
|
||||
- name: Point Loma Electric & Plumbing
|
||||
type: company
|
||||
- name: Hatch
|
||||
type: company
|
||||
- name: Shafer Services
|
||||
type: company
|
||||
- name: Aruza Pest Control
|
||||
type: company
|
||||
- name: SMS
|
||||
type: channel
|
||||
facet: domain/sms
|
||||
- name: NPMA
|
||||
type: organization
|
||||
- name: ServiceTitan
|
||||
type: company
|
||||
|
||||
relationships:
|
||||
- source: After-Hours SMS Lead Capture & Qualification
|
||||
type: is sold to
|
||||
target: pest control companies
|
||||
- source: After-Hours SMS Lead Capture & Qualification
|
||||
type: uses channel
|
||||
target: SMS
|
||||
- source: MIT/InsideSales.com Lead Response Management Study (2007)
|
||||
type: measured conversion decay for
|
||||
target: pest control companies
|
||||
confidence: INFERRED
|
||||
- source: Velocify/Leads360
|
||||
type: found texted prospects convert 40% higher for
|
||||
target: SMS
|
||||
confidence: INFERRED
|
||||
- source: Aruza Pest Control
|
||||
type: is a customer outcome case studied by
|
||||
target: Cube Creative Design
|
||||
- source: Point Loma Electric & Plumbing
|
||||
type: is a case study by
|
||||
target: Hatch
|
||||
- source: Shafer Services
|
||||
type: is a case study by
|
||||
target: Hatch
|
||||
- source: NPMA & PCO Bookkeepers 2025 Cost Study
|
||||
type: reports gross margin for
|
||||
target: pest control companies
|
||||
- source: Coalmarch
|
||||
type: provides cost-per-lead average for
|
||||
target: pest control companies
|
||||
- source: ServiceTitan 2025
|
||||
type: surveyed
|
||||
target: ServiceTitan
|
||||
confidence: AMBIGUOUS
|
||||
- source: Briostack
|
||||
type: reports automation adoption gap for
|
||||
target: pest control companies
|
||||
- source: Gartner 2016
|
||||
type: reports open and response rates for
|
||||
target: SMS
|
||||
|
|
@ -0,0 +1,142 @@
|
|||
# --- Graphify extraction fragment ---
|
||||
note_slug: pest-control-sms-market-research-stats
|
||||
|
||||
entities:
|
||||
- name: MIT/InsideSales.com Lead Response Management Study
|
||||
type: research-study
|
||||
facet: domain/lead-response
|
||||
- name: Harvard Business Review Lead Response Study
|
||||
type: research-study
|
||||
facet: domain/lead-response
|
||||
- name: Velocify Ultimate Contact Strategy Study
|
||||
type: research-study
|
||||
facet: domain/lead-response
|
||||
- name: InsideSales.com 2021 Study
|
||||
type: research-study
|
||||
facet: domain/lead-response
|
||||
- name: HomeAdvisor
|
||||
type: organization
|
||||
- name: Angi
|
||||
type: organization
|
||||
- name: CallRail
|
||||
type: organization
|
||||
- name: PATLive
|
||||
type: organization
|
||||
- name: Driven Results
|
||||
type: organization
|
||||
- name: HubSpot
|
||||
type: organization
|
||||
- name: Valve+Meter
|
||||
type: organization
|
||||
- name: Invoca
|
||||
type: organization
|
||||
- name: Coalmarch
|
||||
type: organization
|
||||
- name: NPMA
|
||||
type: organization
|
||||
- name: PCO Bookkeepers
|
||||
type: organization
|
||||
- name: PPMA
|
||||
type: organization
|
||||
- name: Scorpion
|
||||
type: organization
|
||||
- name: Briostack
|
||||
type: organization
|
||||
- name: Cube Creative Design
|
||||
type: organization
|
||||
- name: ServiceTitan
|
||||
type: organization
|
||||
- name: Gartner
|
||||
type: organization
|
||||
- name: TransUnion
|
||||
type: organization
|
||||
- name: Avochato
|
||||
type: organization
|
||||
- name: Hatch
|
||||
type: organization
|
||||
- name: Google Ads
|
||||
type: tool
|
||||
facet: tool/paid-search
|
||||
- name: Google Local Services Ads
|
||||
type: tool
|
||||
facet: tool/paid-search
|
||||
- name: Thumbtack
|
||||
type: tool
|
||||
facet: tool/lead-marketplace
|
||||
- name: Point Loma Electric & Plumbing
|
||||
type: organization
|
||||
- name: Shafer Services
|
||||
type: organization
|
||||
- name: Aruza Pest Control
|
||||
type: organization
|
||||
- name: After-Hours SMS Lead Capture & Qualification
|
||||
type: product-concept
|
||||
facet: project/niche-automation-prospecting
|
||||
- name: pest control
|
||||
type: industry
|
||||
facet: domain/pest-control
|
||||
- name: home services
|
||||
type: industry
|
||||
facet: domain/field-service
|
||||
- name: lead response time
|
||||
type: concept
|
||||
facet: domain/lead-response
|
||||
- name: SMS marketing
|
||||
type: concept
|
||||
facet: domain/lead-capture
|
||||
- name: lead conversion
|
||||
type: concept
|
||||
facet: domain/lead-response
|
||||
- name: after-hours call handling
|
||||
type: concept
|
||||
facet: domain/field-service
|
||||
- name: niche-automation-prospecting
|
||||
type: project
|
||||
facet: project/niche-automation-prospecting
|
||||
|
||||
relationships:
|
||||
- source: MIT/InsideSales.com Lead Response Management Study
|
||||
type: measures decay of
|
||||
target: lead conversion
|
||||
- source: Harvard Business Review Lead Response Study
|
||||
type: measures decay of
|
||||
target: lead conversion
|
||||
- source: Velocify Ultimate Contact Strategy Study
|
||||
type: measures decay of
|
||||
target: lead conversion
|
||||
- source: InsideSales.com 2021 Study
|
||||
type: measures decay of
|
||||
target: lead conversion
|
||||
- source: After-Hours SMS Lead Capture & Qualification
|
||||
type: targets industry
|
||||
target: pest control
|
||||
- source: After-Hours SMS Lead Capture & Qualification
|
||||
type: addresses problem of
|
||||
target: after-hours call handling
|
||||
- source: After-Hours SMS Lead Capture & Qualification
|
||||
type: leverages
|
||||
target: SMS marketing
|
||||
- source: Aruza Pest Control
|
||||
type: is customer in case study by
|
||||
target: Cube Creative Design
|
||||
- source: Point Loma Electric & Plumbing
|
||||
type: is customer in case study by
|
||||
target: Hatch
|
||||
- source: Shafer Services
|
||||
type: is customer in case study by
|
||||
target: Hatch
|
||||
- source: Driven Results
|
||||
type: sourced after-hours data for
|
||||
target: home services
|
||||
- source: Valve+Meter
|
||||
type: conducted secret-shopper study of
|
||||
target: home services
|
||||
- source: NPMA
|
||||
type: co-published cost study with
|
||||
target: PCO Bookkeepers
|
||||
- source: ServiceTitan
|
||||
type: published industry survey on
|
||||
target: home services
|
||||
- source: niche-automation-prospecting
|
||||
type: contains research for
|
||||
target: After-Hours SMS Lead Capture & Qualification
|
||||
|
|
@ -0,0 +1,31 @@
|
|||
# Benchmark Run Findings — 2026-06-04 (Step 2c reference set)
|
||||
|
||||
_Run: 2026-06-04 — reference set generated, verified, gate satisfied._
|
||||
|
||||
This document records the outcomes, surprises, and lessons from the Step 2c reference-extraction benchmark run. It is the authoritative findings record for this run date and serves as input to future benchmark comparisons when the Graphify/Ollama production pipeline is evaluated.
|
||||
|
||||
## What was run
|
||||
|
||||
- 6 cross-domain fixture notes hand-selected from ~/Documents/SecondBrain (4 pest-control / niche-automation-prospecting, 2 software-design), covering all five fixture categories (tool, client/project, convention, domain, relationship-dense). Fixtures are listed in dispatch-prompt.md.
|
||||
- Each note extracted by 3 Claude tiers (claude-haiku-4-5, claude-sonnet-4-6, claude-opus-4-8) via subagent dispatch with model overrides = 18 reference fragments in reference-outputs/.
|
||||
- Notes were run AS-IS (no vault frontmatter modification); the SecondBrain vault was backed up first to ~/Documents/SecondBrain-backup-20260604-114749.
|
||||
- The .opus.md files are the gold-standard scoring rubric; haiku/sonnet are kept to characterize the tier quality gradient.
|
||||
|
||||
## Key finding: fairness-contract contamination (the gate working as designed)
|
||||
|
||||
- The first-pass haiku extraction of the `ai-coding-conventions-synthesis` note leaked three entities — `Graphify`, `Milvus Lite`, `SecondBrain vault` — that do NOT appear in the note text. They were injected from the cc-os CLAUDE.md / project context that Claude Code auto-loads into subagents. Sonnet and opus did not leak on the same note.
|
||||
- Lesson 1: the soft "ignore any injected context" instruction in the fairness contract is NOT reliable on the haiku tier — re-running clean required an explicit forbidden-class hint (do not introduce knowledge-graph/vector-db/vault tooling unless it appears in the note text).
|
||||
- Lesson 2 (important, prevents a wrong conclusion): this is largely a HARNESS ARTIFACT, not a property of the production pipeline. In the real Graphify→Ollama pipeline, the SLM receives only the prompt Graphify constructs — there is no CLAUDE.md injection path. So the benchmark's contamination exposure is HIGHER than production; the cleaned reference set is conservative. Do NOT conclude "haiku extracts poorly," and do NOT carry the anti-contamination hint into the production Graphify extraction prompt — it is unneeded there.
|
||||
|
||||
## Spec gaps found and fixed
|
||||
|
||||
- The original extraction-spec.md did not require a sub-value after a facet namespace, producing degenerate bare-slash facets (`domain/`, `tool/`) in several sonnet/haiku outputs. It also did not state strict referential integrity, producing relationships pointing at undeclared entities.
|
||||
- Both rules were added to extraction-spec.md on 2026-06-04, and the affected files regenerated so the whole 18-file set conforms to the final spec.
|
||||
|
||||
## Tier richness gradient (observed)
|
||||
|
||||
- Opus is reliably the richest extractor (e.g. the enterprise-revenue note: opus ~59-61 entities vs haiku ~25). Sonnet is sometimes terser than haiku. On stat-dense notes all three tiers saturate to similar counts. Treat entity/relationship counts as indicative, not exact (extractor self-counts diverged from verified counts).
|
||||
|
||||
## Status
|
||||
|
||||
- All 18 files verified: valid YAML, conformant schema, referential integrity, no contamination. Step 2c gate satisfied 2026-06-04.
|
||||
|
|
@ -0,0 +1,246 @@
|
|||
# Local-Model Extraction Scoring — Results (2026-06-04)
|
||||
|
||||
_Last updated: 2026-06-05_ | _Status: UNBLOCKED — qwen2.5-coder:7b selected; scoring (3.4), speed (3.3), and model selection (3.5/3.6) complete. Remaining: build vault graph (4), update docs (5)._
|
||||
|
||||
---
|
||||
|
||||
## Toolchain (Step 2a)
|
||||
|
||||
- graphify 0.8.31 (PyPI `graphifyy`) installed at `~/.local/bin/graphify`; `graphify --version` → `graphify 0.8.31`. [verified: local shell]
|
||||
- ollama 0.30.3, systemd service `/usr/local/bin/ollama serve`, host 127.0.0.1:11434. [verified: local shell]
|
||||
- GPU: RTX 3060, 12GB VRAM. All extraction ran 100% on GPU (no CPU spill) per `ollama ps`. [verified: local shell]
|
||||
|
||||
---
|
||||
|
||||
## Candidate models — true bases (resolves the e4b/e2b naming question)
|
||||
|
||||
Pinned via `ollama show <tag>` info cards. These are genuine next-gen families, NOT gemma3n/qwen2.5 renames: [verified: ollama show output]
|
||||
|
||||
| Local tag | Architecture | Params | Quant | Native ctx |
|
||||
|---------------|--------------|---------------------------------|--------|------------|
|
||||
| gemma4:e4b | gemma4 | 8.0B (MatFormer "effective 4B") | Q4_K_M | 131072 |
|
||||
| gemma4:e2b | gemma4 | 5.1B (effective 2B) | Q4_K_M | 131072 |
|
||||
| qwen3.5:2b | qwen35 | 2.3B | Q8_0 | 262144 |
|
||||
|
||||
---
|
||||
|
||||
## Critical config finding — num_ctx does not propagate through graphify (Step 2b)
|
||||
|
||||
- graphify posts to ollama's OpenAI-compatible `/v1/chat/completions` endpoint. That endpoint **silently ignores** the per-request `options.num_ctx` graphify sends. Proven by A/B: `POST /v1/chat/completions` with num_ctx=8192 → `ollama ps` CONTEXT=4096; `POST /api/chat` with num_ctx=8192 → CONTEXT=8192. [verified: local A/B test]
|
||||
- Therefore `GRAPHIFY_OLLAMA_NUM_CTX` (env var) has **NO EFFECT** through graphify; context pins at ollama's /v1 default 4096.
|
||||
- At 4096, graphify's extraction output JSON is truncated mid-response (finish_reason=length) and the chunk is **DISCARDED** → empty graph, regardless of (small) input note size.
|
||||
- **WORKAROUND (validated):** bake context into a Modelfile variant — `FROM <model>` + `PARAMETER num_ctx 8192`, then `ollama create <name>-8k`. The /v1 endpoint **does** honor the model's baked default. Verified `ollama ps` CONTEXT=8192 (and 16384), 100% GPU. This is the same lever production (vault build) will need. Non-invasive: no sudo, no systemd restart, no shell-profile/dotfiles edit; reversible via `ollama rm`. [verified: local shell]
|
||||
- Two other operational notes: graphify's `OLLAMA_BASE_URL` must **end in `/v1`** (e.g. `http://127.0.0.1:11434/v1`) or every call 404s; and the CLI requires `OLLAMA_API_KEY` set to any non-empty value unless the host is loopback (127.0.0.1/localhost/::1).
|
||||
|
||||
---
|
||||
|
||||
## Benchmark methodology
|
||||
|
||||
- Each of the 6 Step-2c fixtures was isolated in its own temp directory (graphify `extract` does NOT accept a single file path — it walks a directory; isolation also avoids conflating notes in one token-budget chunk and excludes `.obsidian/` config noise). [verified: local testing]
|
||||
- Command shape: `OLLAMA_BASE_URL=http://127.0.0.1:11434/v1 graphify extract <fixture-dir> --backend ollama --model <8k-variant> --max-concurrency 1 --out <dir>`. Models run sequentially (single GPU), one resident at a time.
|
||||
- Scoring was scoped to dimensions graphify's output and the `.opus` gold references share: entity correctness, relationship plausibility, and edge-level INFERRED/AMBIGUOUS confidence. **Excluded as non-comparable:** the six-facet taxonomy and per-entity type/confidence (graphify's built-in extraction prompt emits neither — per ADR-011 facets are note frontmatter metadata, never Graphify's extraction job), and free-text relationship TYPE strings (graphify uses a fixed relation vocabulary: `references`/`conceptually_related_to`/`semantically_similar_to`/etc., vs the references' free-text verbs).
|
||||
|
||||
---
|
||||
|
||||
## Results through graphify's real extraction path
|
||||
|
||||
All three tested as 8192-context Modelfile variants. **Scoring blocked** because outputs were degenerate:
|
||||
|
||||
| Model | Result | Failure mode |
|
||||
|--------------------|-------------------------------------------|--------------|
|
||||
| gemma4:e4b (8k) | EMPTY graphs (multiple fixtures, deterministic 5/5) | Hollow, invalid JSON (~5039 output tokens of unparseable content); graphify discards chunk. graphify's own warning: "model too small for JSON instruction following — try a larger model." |
|
||||
| qwen3.5:2b (8k) | EMPTY graph | Output truncated at max_completion_tokens; qwen3.5 "thinking" mode appears to consume the output budget before emitting the JSON graph. |
|
||||
| gemma4:e2b (8k) | 15 nodes, 0 edges (valid JSON) on the oo-principles fixture | Extracted plausible entities (OO Principles, Single Responsibility Principle, Law of Demeter, Dependency Injection, the four design approaches, etc.) but emitted ZERO relationships → unusable as a knowledge graph. |
|
||||
|
||||
**Caveat on variance:** model Modelfile defaults are temperature 1 (poor for structured extraction); one PRE-fix run of gemma4:e4b on oo-principles yielded 4 nodes/3 edges, indicating high run-to-run variance.
|
||||
|
||||
---
|
||||
|
||||
## Conclusion
|
||||
|
||||
No specified candidate (gemma4:e4b, gemma4:e2b, qwen3.5:2b) produces a usable entity+relationship graph through Graphify's **default extraction prompt** on this hardware. Per the change proposal's risk clause ("no candidate scores acceptably → surface as finding, don't force selection"), model selection (task 3.6) and the initial vault graph build (task 4) are **DEFERRED** pending a decision.
|
||||
|
||||
**NOTE:** The earlier gut-check (`local-llm-findings-2026-06-04.md`) reported gemma4:e4b doing "6/6 clean parse, 13–34 entities" — but that used a **direct ollama call** with a simpler custom prompt, **NOT** graphify's actual extraction prompt. Through graphify's real path the result is far worse. The graphify-path numbers are the authoritative ones for production.
|
||||
|
||||
---
|
||||
|
||||
## Untested levers (candidate next steps, not yet attempted)
|
||||
|
||||
1. **Disable model "thinking" mode** — gemma4 and qwen3.5 are thinking-capable; thinking tokens likely burn the output budget, causing the hollow/truncated JSON. Most promising cheap lever, but exposing a think-toggle through graphify's /v1 path is unverified.
|
||||
2. **`graphify extract --mode deep`** to elicit INFERRED edges (addresses e2b's 0-edge result).
|
||||
3. **Override sampling temperature to 0** for structured extraction (Modelfile defaults are temp 1).
|
||||
4. **Pull a stronger model that fits 12GB VRAM with 8k context** — e.g. a 7–8B at Q4 (~5GB, leaves headroom for context); graphify itself suggests a ~14B (~9GB, tighter). This expands the candidate set beyond the three the change specified (a download + scope decision).
|
||||
|
||||
---
|
||||
|
||||
## Raw artifacts (ephemeral, /tmp — will not survive reboot)
|
||||
|
||||
- e4b first benchmark (pre-fix, 4096): `/tmp/graphify-bench/out/e4b/`
|
||||
- 8k-variant runs: `/tmp/graphify-bench/out2/`, `/tmp/graphify-bench/out3/`
|
||||
- Created reversible ollama variants: `gemma4-e4b-8k`, `gemma4-e4b-16k`, `gemma4-e2b-8k`, `qwen35-2b-8k`
|
||||
|
||||
---
|
||||
|
||||
## Round 2 (2026-06-05): thinking-disable patch + edge gate
|
||||
|
||||
### The thinking-mode investigation (user-directed)
|
||||
- GitHub issue safishamsi/graphify#792 turned out to be about CPU scaling, local API timeouts, and the OLLAMA_API_KEY auth gate — NOT thinking/reasoning. (Documents the same auth friction we hit.)
|
||||
- Ollama's `/v1` (OpenAI-compat) endpoint, which Graphify uses, IGNORES `think:false` and `chat_template_kwargs:{enable_thinking:false}`, but DOES honor top-level `reasoning_effort:"none"`. The native `/api/chat` honors `think:false`. (Same /v1-drops-nonstandard-options pattern as num_ctx.)
|
||||
- Graphify sends NO thinking control for the ollama backend by default; there is no env var / CLI flag / config file to inject it (confirmed by source trace of llm.py). `PARAMETER think false` in a Modelfile is rejected by `ollama create`; a Modelfile `SYSTEM /no_think` does not survive because Graphify sends its own system message and ollama's /v1 REPLACES (not merges) the system message.
|
||||
- WORKING FIX (applied to the installed package): one-line patch adding `"reasoning_effort": "none"` to the ollama backend config dict in `~/.local/lib/python3.14/site-packages/graphify/llm.py` (~line 71). Graphify already applies `reasoning_effort` as a top-level kwarg if present, and it survives the extra_body overwrite. The author already uses the equivalent pattern (`extra_body={"thinking":{"type":"disabled"}}`) for the Kimi/moonshot backend — just not for ollama. PRODUCTION IMPLICATION: this is a local patch lost on `pip install --upgrade`; production needs an upstream PR (likely welcome) or a maintained patch/wrapper.
|
||||
- Verified effect: with the patch, every model's raw output begins directly with `{"nodes":[...` — no reasoning preamble. Thinking is genuinely suppressed.
|
||||
|
||||
### The edge gate — the criterion that actually matters
|
||||
A knowledge graph needs RELATIONSHIPS (edges), not just entities (nodes). Across ~10+ runs this whole investigation, LLM-extracted relationships appeared essentially once (e4b pre-patch: 3 edges, since non-reproducible). All other non-empty results were nodes-only. So the gate for selection is: does a config emit a plausible EDGE-bearing graph on one fixture?
|
||||
|
||||
Bounded test on the `oo-principles-plugin-concept-design-recommendations` fixture (a note that clearly contains relationships), thinking-disable confound checked by running both ON and OFF:
|
||||
|
||||
| Config | Exit | Nodes | Edges | Outcome |
|
||||
|---|---|---|---|---|
|
||||
| qwen3.5:2b @16k, thinking-OFF | 1 | — | empty | output truncated mid-JSON → invalid → discarded |
|
||||
| qwen3.5:2b @16k, thinking-ON | 1 | — | empty | same truncation; 16k did not rescue it |
|
||||
| gemma4:e2b @16k, thinking-ON | 0 | 17 | 0 | clean valid JSON, coherent on-topic nodes, ZERO edges |
|
||||
| gemma4:e2b @16k, thinking-ON, --mode deep | 0 | 10 | 0 | deep mode (explicitly requests INFERRED edges) made it WORSE: fewer nodes, still 0 edges |
|
||||
|
||||
`hyperedges` was also empty `[]` in the clean runs — no relationships hid in the alternate key.
|
||||
|
||||
### Confound resolved: thinking-off is NOT strictly better
|
||||
gemma4:e2b @8k: thinking-ON gave 15 nodes / 3307 output tokens (valid); thinking-OFF gave 1 node / 395 tokens. For e2b the thinking tokens were doing extraction work; the binding constraint is output room, not thinking. So the user's "thinking wastes the budget" premise holds for qwen (truncation) but inverts for e2b. Thinking-disable is a useful lever, not a universal win.
|
||||
|
||||
### Two distinct failure modes
|
||||
- qwen3.5:2b (2.3B, Q8_0): capacity/truncation — runs out of output room emitting node JSON before reaching edges. 16k context insufficient; thinking on/off irrelevant.
|
||||
- gemma4:e2b (5.1B, Q4_K_M): clean completion, extracts plausible NODES but emits zero RELATIONSHIPS even when deep mode explicitly asks for them. The model understands the doc (node labels: OO Principles, Single Responsibility Principle, Law of Demeter, Dependency Injection, Replace Conditional with Polymorphism, the 4 approaches, the 3 layers) — it just won't populate `links`.
|
||||
- gemma4:e4b (8.0B, Q4_K_M): hollow/malformed JSON (dropped — not a context problem).
|
||||
|
||||
### Round-2 conclusion
|
||||
The cheap-lever space is exhausted (thinking ON/OFF, 8k/16k context, --mode deep). No local 2–5B candidate produces a relationship-bearing graph through Graphify's extraction prompt on a 12GB GPU. Relationship extraction (not entity extraction) is the wall. Recommended path: a stronger model that fits 12GB with ~16k context — e.g. qwen2.5-coder:7b (~5GB Q4, non-thinking, strong at structured JSON+edges). This expands the candidate set beyond the three the change specified (a multi-GB download + scope decision) — pending user go-ahead.
|
||||
|
||||
### Round-2 ollama variants created (reversible via `ollama rm`)
|
||||
gemma4-e4b-8k, gemma4-e4b-16k, gemma4-e2b-8k, gemma4-e2b-16k, qwen35-2b-8k, qwen35-2b-16k. Installed graphify llm.py currently carries the thinking-OFF patch (backup of original at /tmp/graphify-bench/llm.py.orig).
|
||||
|
||||
---
|
||||
|
||||
## Round 3 (2026-06-05): qwen2.5-coder:7b clears the edge gate — full benchmark
|
||||
|
||||
### Config that works
|
||||
- Model: `qwen2.5-coder:7b` (graphify's hardcoded ollama default, llm.py:67), run as a Modelfile variant `qwen25-coder-7b-16k` (FROM qwen2.5-coder:7b + PARAMETER num_ctx 16384).
|
||||
- Command: `OLLAMA_BASE_URL=http://127.0.0.1:11434/v1 graphify extract <fixture-dir> --backend ollama --model qwen25-coder-7b-16k --max-concurrency 1 --out <dir>`. The thinking-OFF llm.py patch is in place (no-op for qwen2.5-coder, which has no thinking mode).
|
||||
- VRAM: 5.6GB resident, 100% GPU, 16384 context, no CPU spill on the 12GB RTX 3060.
|
||||
- Speed (task 3.3): aggregate throughput ~59.4 tok/s (substantive runs cluster 57–61 tok/s). Total wall-clock all 6 = 253.7s (4.2 min); avg 42.3s/fixture (49.9s excluding the degenerate 10dlc run). Use tok/s not wall-clock (wall-clock tracks output length).
|
||||
|
||||
### Per-fixture results (edge gate PASSED: 5/6 produce relationships)
|
||||
| Fixture | Wall | Nodes | Edges | Out tok | Notes |
|
||||
|---|---|---|---|---|---|
|
||||
| oo-principles-plugin-concept | 44.9s | 11 | 10 | 2553 | gate fixture; cold-load incl. |
|
||||
| 10dlc-isv-setup-guide-oncadence | 4.4s | 1 | 0 | 159 | under-extraction on smallest note (502 words), exit 0 valid JSON — not a crash |
|
||||
| pest-control-enterprise-revenue | 64.2s | 23 | 17 | 3932 | only fixture with INFERRED edges (7) |
|
||||
| ai-coding-conventions-synthesis | 37.9s | 10 | 9 | 2284 | |
|
||||
| pest-control-after-hours-sms | 76.8s | 21 | 20 | 4641 | |
|
||||
| pest-control-email-a-b-c-hub | 25.5s | 7 | 6 | 1489 | |
|
||||
|
||||
All exit 0, no truncation/hollow/invalid-JSON warnings. Confidence tags are provenance method tags: EXTRACTED (from text) vs INFERRED (model-inferred); INFERRED appeared only in enterprise-revenue.
|
||||
|
||||
### Full edge lists (for scoring vs .opus references next session)
|
||||
**oo-principles (10, all EXTRACTED):** Process Layer→shares_data_with→Mechanic Layer; Process Layer→shares_data_with→Theory Layer; Mechanic Layer→shares_data_with→Cards Layer; Annotated Process Graph Layer→shares_data_with→Cards Layer; Bundles Layer→shares_data_with→Cards Layer; CLAUDE.md→shares_data_with→Cards Layer; NotebookLM Notebook→shares_data_with→Cards Layer; Phase Bundled Context Packs Layer→shares_data_with→Cards Layer; Refactoring Layer→shares_data_with→Cards Layer; Situational Trigger Files Layer→shares_data_with→Cards Layer.
|
||||
**10dlc:** none.
|
||||
**enterprise-revenue (17):** Marketing Allocation Model→references→Unit Economics; ACV Tiers→references→Unit Economics; M&A Context→references→ACV Tiers; Communication Failure Patterns→references→Unit Economics; Dropped calls and ghosting→conceptually_related_to→Communication Failure Patterns [INFERRED]; No one answered→conceptually_related_to→Communication Failure Patterns [INFERRED]; Offshore agent disconnect→conceptually_related_to→Communication Failure Patterns [INFERRED]; Voicemail full→conceptually_related_to→Communication Failure Patterns [INFERRED]; M&A Context→references→Communication Failure Patterns; Climatologically-Driven Lead Seasonality→references→Unit Economics; Fall First Freeze Rodent Push→conceptually_related_to→Climatologically-Driven Lead Seasonality [INFERRED]; Spring Termite Swarming→conceptually_related_to→Climatologically-Driven Lead Seasonality [INFERRED]; Summer Heatwave-Driven Migrations→conceptually_related_to→Climatologically-Driven Lead Seasonality [INFERRED]; M&A Context→references→Climatologically-Driven Lead Seasonality; M&A Context→references→Operational Unit Economics; Operational Unit Economics→references→COGS Structure; M&A Context→references→Unit Economics.
|
||||
**ai-coding-conventions (9, all EXTRACTED):** Core Finding→references→{Context Linking Patterns, Enforcement Mechanisms, Key Tradeoffs, Notable Public References, Organizational Patterns, Process vs. Reference Design, See Also, The Three-Tier Architecture (arXiv 2602.20478), Token Efficiency Strategies}.
|
||||
**after-hours-sms (20, all EXTRACTED):** document node→cites→ each of 20 study/source nodes (MIT/InsideSales 2007, HBR 2011, Velocify 2012, InsideSales 2021, HomeAdvisor/Angi 2024, Thumbtack 2024, Coalmarch 2025, Invoca 2024, Driven Results 2025, HubSpot 2023, Gartner 2016, D7 Networks/CTIA 2024, TransUnion 2024, Avochato 2019, Briostack 2024, ServiceTitan 2025, Point Loma Hatch, Shafer HVAC Hatch, Aruza Cube, ServiceTitan Data 2025).
|
||||
**email-a-b-c-hub (6, all EXTRACTED):** Experiment Hub→references→{Blind A/B/C Methodology Guide, Per Group Copy and Scores, What Owners Respond to/Reject, V4 Copywriter Draft, V4 Golden Framework Draft, V4 Golden Framework Revised}.
|
||||
|
||||
### Round-3 takeaway
|
||||
qwen2.5-coder:7b is the de-facto selected model pending the formal scoring pass (3.4). It's graphify's own default, fits 12GB with 16k context at ~59 tok/s, and produces typed, mostly-EXTRACTED edges. The earlier gut-check's gemma4:e4b pick was an artifact of using a simpler direct prompt, not graphify's real path. Source graph.json (ephemeral): /tmp/graphify-bench/out6/coder7b/<slug>/graphify-out/graph.json.
|
||||
|
||||
---
|
||||
|
||||
## 3.4 Scoring — qwen2.5-coder:7b vs Opus Gold Standard
|
||||
|
||||
**Sources used:**
|
||||
- Candidate output: `scoring-results-2026-06-04.md`, Round 3 section (per-fixture node/edge lists)
|
||||
- Gold-standard references: `reference-outputs/*.opus.md` (all 6 `.opus.md` files)
|
||||
- Fixture mapping (from Round 3 table): `oo-principles-plugin-concept`, `10dlc-isv-setup-guide-oncadence`, `pest-control-enterprise-revenue`, `ai-coding-conventions-synthesis`, `pest-control-after-hours-sms`, `pest-control-email-a-b-c-hub`
|
||||
|
||||
### Dimensions Excluded (N/A)
|
||||
|
||||
**Facets:** The `.opus.md` references include facet annotations (e.g., `facet: tool/twilio`, `facet: convention/oo-principles`). These are note-level metadata per ADR-011 — they are not part of graphify's extraction output at all, so there is nothing to compare. N/A.
|
||||
|
||||
**Entity-type labels:** The opus references annotate entities with types (`type: layer`, `type: principle`, `type: Tool`, etc.). Graphify's extraction output produces a node list with names only — no type labels are emitted. N/A.
|
||||
|
||||
**Free-text relationship typing:** The opus references use descriptive, domain-specific verbs (`comprises_share_of`, `is_event_of`, `measured conversion decay for`, `tested for campaign`, etc.). Graphify uses a fixed relation vocabulary (`references`, `conceptually_related_to`, `semantically_similar_to`, `shares_data_with`, `cites`). These are structurally incommensurable — comparing relation-type strings would score graphify's vocabulary design, not extraction quality. N/A.
|
||||
|
||||
### Rubric
|
||||
|
||||
**Entity Correctness (a):** 1–5. How many of the entities graphify extracted correspond to real entities in the reference? Penalizes under-extraction (missing key nodes) and over-compression (collapsing distinct entities). Does not penalize naming style differences.
|
||||
|
||||
**Relationship Plausibility (b):** 1–5. Are the edges graphify drew present or derivable from the reference's relationship set? Penalizes phantom edges (no basis in reference), missing major structural relationships, and systematically wrong pairing patterns. Does not penalize relation-verb style.
|
||||
|
||||
**Confidence Accuracy (c):** 1–5. Where graphify tagged EXTRACTED vs INFERRED vs AMBIGUOUS, does that align with what the reference treated as explicit (no `confidence:` tag) vs inferred (`confidence: INFERRED`) vs ambiguous (`confidence: AMBIGUOUS`)? N/A on fixtures where graphify emitted no edges at all, or no INFERRED/AMBIGUOUS tags appeared in either output.
|
||||
|
||||
### Per-Fixture Scoring Table
|
||||
|
||||
| Fixture | (a) Entity Correctness | (b) Relationship Plausibility | (c) Confidence Accuracy | Notes |
|
||||
|---|---|---|---|---|
|
||||
| oo-principles-plugin-concept | 2 | 1 | N/A† | |
|
||||
| 10dlc-isv-setup-guide | 1 | N/A (0 edges) | N/A | |
|
||||
| pest-control-enterprise-revenue | 3 | 3 | 3 | |
|
||||
| ai-coding-conventions-synthesis | 2 | 2 | N/A† | |
|
||||
| pest-control-after-hours-sms | 3 | 2 | N/A† | |
|
||||
| pest-control-email-a-b-c-hub | 2 | 2 | N/A† | |
|
||||
|
||||
†Graphify emitted no INFERRED/AMBIGUOUS tags on this fixture (all EXTRACTED), so there is no confidence signal to compare against the reference's INFERRED annotations. N/A does not mean graphify was wrong; it means the dimension is unobservable.
|
||||
|
||||
### Per-Fixture Justifications
|
||||
|
||||
**oo-principles-plugin-concept**
|
||||
- (a) **2/5.** Captured the layer decomposition (Process, Mechanic, Theory + 4 named layers, CLAUDE.md, NotebookLM Notebook = 11 nodes) but missed the plugin as a named entity, all OO principle entities (TDD, SRP, LoD, DI, Shameless Green, Flocking Rules), the book/paper sources, and the 4-phase development lifecycle. ~30% of the ~30-entity reference set.
|
||||
- (b) **1/5.** All 10 edges are `shares_data_with` → "Cards Layer". The reference relationships are structural/directional (includes, encodes, references, retains, alternative to, evolves from, modeled on). Homogenization collapses the architecture into a flat data-sharing star with no analog in the reference.
|
||||
- (c) **N/A.** All graphify edges EXTRACTED; reference has 3 INFERRED. No INFERRED signal to compare.
|
||||
|
||||
**10dlc-isv-setup-guide**
|
||||
- (a) **1/5.** Extracted 1 node (whole document as a single entity) vs 20 distinct reference entities (OnCadence, 10DLC, Twilio, TrustHub API, ISV, CSP, EIN, IRS, T-Mobile, subaccount-per-client architecture, etc.). Near-total extraction failure on a 502-word note.
|
||||
- (b) **N/A.** 0 edges produced vs 17 reference relationships.
|
||||
- (c) **N/A.** No edges.
|
||||
|
||||
**pest-control-enterprise-revenue**
|
||||
- (a) **3/5.** 23 nodes capturing the major conceptual clusters (Marketing Allocation Model, ACV Tiers, M&A Context, Unit Economics, Communication Failure Patterns + 4 sub-patterns, Seasonality + 3 seasonal events, COGS Structure) but at shallower granularity than the ~55-entity reference (misses CAC/LTV/CPL/Churn, COGS components, regions, Big Four integrators, GDD, multiples).
|
||||
- (b) **3/5.** 17 edges incl. 7 INFERRED. The INFERRED membership cluster (comm sub-patterns → Communication Failure Patterns; seasonal events → Seasonality) maps closely onto the reference's `is_event_of`/`component_of` structure. ~50% of reference edges missed (COGS breakdown, Big Four, causal chains).
|
||||
- (c) **3/5.** The 7 INFERRED tags are defensible group-membership edges; not systematically miscalibrated, though it under-fires on EXTRACTED edges the reference treats as explicit, and never modeled the reference's `EIN → issued_by → IRS` background-knowledge INFERRED edge.
|
||||
|
||||
**ai-coding-conventions-synthesis**
|
||||
- (a) **2/5.** Extracted the document's section-heading outline (10 nodes: Context Linking Patterns, Enforcement Mechanisms, Three-Tier Architecture, etc.) rather than the ~40 domain entities (Claude, Cursor, Copilot, Constitution, Domain Specialist Agents, .cursorrules, AGENTS.md, MCP, named projects). Only Three-Tier Architecture overlaps in substance.
|
||||
- (b) **2/5.** All 9 edges are `Core Finding → references → {section heading}` — a hub-and-spoke from a synthetic aggregate node. Zero structural resolution vs the reference's compositional/directional relationships. 2 not 1 because no phantom entities introduced.
|
||||
- (c) **N/A.** All EXTRACTED; reference's 2 INFERRED edges not modeled.
|
||||
|
||||
**pest-control-after-hours-sms**
|
||||
- (a) **3/5.** 21 nodes (document + 20 study/source nodes) covering ~half the reference's company/study entities, but missing the product/project/campaign entities (After-Hours SMS product, niche-automation-prospecting initiative, pest-control-spring-2026 campaign) and several companies (Google Ads/LSA, NPMA, CallRail, PATLive, etc.).
|
||||
- (b) **2/5.** All 20 edges are `document → cites → {study}` — a citation list. Defensible but reductive; the reference's product–customer, product–channel, and study–finding semantic edges are entirely missing.
|
||||
- (c) **N/A.** All EXTRACTED; reference has 2 INFERRED + 1 AMBIGUOUS, none modeled.
|
||||
|
||||
**pest-control-email-a-b-c-hub**
|
||||
- (a) **2/5.** Extracted the hub's table-of-contents (7 nodes: Experiment Hub + 6 linked sub-documents) but missed the experiment's analytic entities (3 personas, 9 email groups, 5 hooks, structural/retired patterns, 3 versions) — the primary conceptual content. ~45-entity reference.
|
||||
- (b) **2/5.** All 6 edges are `Experiment Hub → references → {sub-document}` — the index, not the findings. Version outcomes, recommended groups, retired patterns all absent.
|
||||
- (c) **N/A.** All EXTRACTED; reference's 2 INFERRED edges not modeled.
|
||||
|
||||
### Overall Summary
|
||||
|
||||
**Entity extraction: weak to moderate (avg ~2.2/5).** Consistently fewer/coarser entities than the opus reference. On hub-structured notes it extracts the structural outline (section headings, linked sub-docs) rather than conceptual entities; on dense domain notes it captures ~40–55% of the reference entity set; on the smallest note (10dlc) it nearly fails (1 node vs 20).
|
||||
|
||||
**Relationship extraction: poor to moderate (avg ~1.9/5, excluding the N/A fixture).** Dominant failure mode is **homogenization**: nearly all relationship semantics collapse into one or two relation types (`references`, `cites`, `shares_data_with`) in a hub-and-spoke topology. enterprise-revenue is the only fixture with meaningfully differentiated edges — and the only one where INFERRED edges appeared — suggesting extraction quality correlates with note density.
|
||||
|
||||
**Confidence calibration: partially defensible where observable (3/5 on the one applicable fixture).** INFERRED tags only appeared on enterprise-revenue (7/17 edges) and were defensible group-membership edges. On the other five fixtures all edges were EXTRACTED, precluding comparison.
|
||||
|
||||
**Net assessment:** qwen2.5-coder:7b passes the edge gate (5/6 fixtures produce relationships) and extracts credible entities on content-dense notes, confirming it as a viable baseline over the tested alternatives. Its relationship extraction is systemically shallow: the relation flattening means the graph encodes "these things are related" but not "how," limiting downstream query quality. For a production vault graph the primary gap is relationship semantic resolution, not entity recall.
|
||||
|
||||
---
|
||||
|
||||
## 3.6 Model Selection
|
||||
|
||||
**Selected model: qwen2.5-coder:7b** (run as the Modelfile variant `qwen25-coder-7b-16k`, num_ctx 16384).
|
||||
|
||||
The selection is not a choice among viable candidates — it is the only candidate that cleared the relationship/edge gate. Every smaller candidate tested (gemma4:e4b 8.0B, gemma4:e2b 5.1B, qwen3.5:2b 2.3B) failed to produce relationship-bearing graphs through graphify's prompt on this hardware. The scoring in section 3.4 therefore characterizes qwen2.5-coder:7b's extraction quality against the opus gold standard rather than differentiating among alternatives; there was no viable alternative to differentiate.
|
||||
|
||||
This model is also graphify's own hardcoded ollama default (llm.py:67), which is a strong signal that it is the intended extraction model for this backend. It fits the 12GB RTX 3060 comfortably at 5.6GB VRAM resident with 16384 context and runs at ~59 tok/s, completing a full 6-fixture pass in ~4 minutes.
|
||||
|
||||
**Known limitations (carried forward honestly):** shallow relationship semantics — graphify's fixed relation vocabulary combined with qwen2.5-coder:7b's extraction behavior produces hub-and-spoke topologies dominated by `references`, `cites`, and `shares_data_with`, collapsing domain-specific structural relationships. Entity recall is weak on hub-structured notes (extracts the structural outline instead of conceptual entities) and near-failing on very short notes (10dlc: 1 node vs 20 in reference). These limitations are accepted for the initial build. Relationship semantic resolution is the known primary gap for future improvement.
|
||||
|
||||
**Speed:** ~59 tok/s on the 12GB GPU at max-concurrency 1.
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
schema: spec-driven
|
||||
created: 2026-06-04
|
||||
|
|
@ -0,0 +1,42 @@
|
|||
## Context
|
||||
|
||||
`cc-os` is a design-only repository; the build runbook is `docs/memory-system/05-implementation-process.md` and Step 2c is its documented gate. The hard constraint driving this change: only the Claude tiers are reachable here (via Claude Code subagents) — local Ollama models cannot be run in this benchmarking environment. The vault is `~/Documents/SecondBrain` (ADR-012) under the six-facet taxonomy (ADR-011). Graphify extracts entities plus typed edges plus confidence tags (`INFERRED`/`AMBIGUOUS`) from documents via a local SLM, and extracts code via tree-sitter AST (free, no model). The benchmark exists to choose the local doc-extraction model; this change does not make that choice, it produces the reference set that choice will be measured against.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- A runnable benchmark that produces a reusable, diffable reference set across Claude tiers.
|
||||
- A fairness contract that keeps the comparison as close to apples-to-apples as the environment allows.
|
||||
- An incremental, observable build-and-migration path that validates the system before committing the whole vault to it.
|
||||
|
||||
**Non-Goals:**
|
||||
- Choosing the final extraction model now (the reference set feeds that later decision).
|
||||
- Pivoting away from local Ollama doc extraction (architecture stays intact per the existing ADRs).
|
||||
- Running Ollama models within this change.
|
||||
- Bulk-migrating the vault now.
|
||||
|
||||
## Decisions
|
||||
|
||||
- **Reference-set, not model selection.** Claude subagents produce gold-standard outputs, not a final pick. Rejected alternative: treating this as a "Claude-as-extraction-backend" pivot — that is an ADR-level architectural shift (cost, privacy) and is out of scope.
|
||||
- **Mimic-extraction task with an explicit embedded output schema.** Subagents reason directly to the Graphify-shaped fragment rather than invoking `graphify extract`, because Ollama backends cannot run here and the goal is tier-vs-tier signal. The output schema (entities, typed relationships, `INFERRED`/`AMBIGUOUS` confidence) is embedded verbatim in the prompt so per-model files are diffable against each other now and against Ollama output later. Each subagent writes to its own per-model file.
|
||||
- **Fairness contract: minimal context only.** Each subagent receives only the raw note text and the shared extraction spec. It is explicitly instructed not to read repository files (`CLAUDE.md`, design docs) or pull project context. This is the core property that makes the comparison meaningful.
|
||||
- **Speed dropped for the Claude run.** Wall-clock per note is untrackable across dispatched subagents here, so quality is the only metric for the reference run: entity correctness, relationship plausibility/typing, and confidence-tag accuracy. Speed re-enters when local Ollama models are timed against the references.
|
||||
- **Build-first / migrate-incrementally.** A 5–10 note fixture set (the variety already called for in Step 1c) feeds the gate immediately; bulk vault migration is deferred to last; first end-to-end validation runs against one small project containing both code and documents. The build-order inversion is recorded in ADR-013, and `CLAUDE.md`'s "Decisions locked" pointer is updated.
|
||||
- **Migration-unit granularity, surfaced not hidden.** The first migration unit is named "one small project with both code and documents." Vault notes (local-SLM extraction path) and project code (tree-sitter path) are different extraction paths; design.md surfaces this so the user can react at proposal review rather than discovering it mid-migration.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- Fixtures unrepresentative of the real vault → choose deliberate variety per Step 1c (tool note, client/project note, convention note, domain note, one relationship-dense note) and keep them as living fixtures.
|
||||
- Reference set encodes Claude-tier idiosyncrasies and biases later Ollama scoring → treat the references as a quality ceiling / scoring rubric, not literal ground truth; keep a human in the loop reviewing god-nodes.
|
||||
- Deferring bulk migration delays real-world validation of the schema at scale → mitigated by the early end-to-end test on one small mixed project before broad rollout.
|
||||
- Prompt context leakage breaks the fairness contract → the prompt enumerates the only allowed inputs and explicitly forbids reading repo/project files.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
No code migration. The change edits docs and adds ADR-013; the only new artifact is the benchmark prompt file plus a directory for reference outputs (proposed: `docs/memory-system/benchmark/`). Rollback is a straight revert of the doc edits.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Migration-unit granularity (whole project repo vs. a vault-note cluster) — first unit is named now; refine after observing the first real migration.
|
||||
- Where the prompt file and per-model reference outputs live (proposed default: `docs/memory-system/benchmark/`).
|
||||
- Which specific small project is the initial mixed code+docs validation target — the user picks this at apply time (candidates under `~/projects/` and `~/dev/`).
|
||||
|
|
@ -0,0 +1,29 @@
|
|||
## Why
|
||||
|
||||
Two gaps block executing the build runbook (`docs/memory-system/05-implementation-process.md`). First, Step 2c's model benchmark assumes head-to-head runs across local Ollama models and Claude API models, but the only models reachable from this environment are the Claude tiers via Claude Code subagents — so the benchmark cannot be run as written, and nothing captures a gold-standard to score local models against later. Second, the runbook front-loads bulk vault migration (all ~20 notes and all projects) as Step 1, committing to a schema and workflow before the system exists to validate them.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Redesign Step 2c as a **reference-set benchmark**: a copy/paste-able Claude Code prompt — written as a committed file — that dispatches one subagent per Claude tier (`claude-haiku-4-5`, `claude-sonnet-4-6`, `claude-opus-4-8`) to mimic Graphify doc extraction on the fixture notes, each emitting the same Graphify-shaped structured fragment (entities, typed relationships, `INFERRED`/`AMBIGUOUS` confidence tags) to a per-model file. These per-model files become the gold-standard reference set that local Ollama models are scored against in a later step.
|
||||
- Establish a **fairness contract** for the prompt: each subagent receives only the note text plus a shared minimal extraction spec — no `CLAUDE.md`, no design docs, no project context.
|
||||
- Drop wall-clock speed as a metric for the Claude reference run (untrackable in this environment; quality only). Speed re-enters later when Ollama models are benchmarked against the reference set.
|
||||
- **Resequence the build order**: build the full system against a small fixture set first; defer bulk vault migration and multi-project onboarding until the system is built and validated end-to-end on one small project that contains both code and documents; then onboard remaining projects one at a time, observing and adjusting per project.
|
||||
- Keep four things explicitly distinct so the runbook does not blur them: benchmark fixtures (needed now, for the gate), bulk vault migration (deferred), the initial mixed code+docs validation project (post-build), and project-by-project rollout.
|
||||
- Add **ADR-013** recording the build-order inversion (vault-migration-first → build-first / migrate-incrementally) and amend the "Decisions locked" pointer in `CLAUDE.md`.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
|
||||
- `reference-extraction-benchmark`: the procedure, fairness contract, output schema, and deliverable prompt file for producing a reusable gold-standard Claude reference extraction set that local models are later scored against.
|
||||
- `incremental-migration`: the resequenced build order — build and validate on fixtures plus one small mixed code+docs project before any bulk migration, then onboard projects one at a time.
|
||||
|
||||
### Modified Capabilities
|
||||
|
||||
<!-- None. No prior specs exist in openspec/specs/. -->
|
||||
|
||||
## Impact
|
||||
|
||||
- Docs: `docs/memory-system/05-implementation-process.md` (Step 1 sequencing + Step 2c rewrites), `docs/memory-system/03-architecture-decisions.md` (new ADR-013), `docs/memory-system/04-build-plan.md` (reconcile if it states the old order), `CLAUDE.md` ("Decisions locked" line).
|
||||
- New deliverable: a committed, copy/paste-able benchmark dispatch prompt file, plus a home for the per-model reference outputs.
|
||||
- No application code. This change touches design, runbook, and process only.
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
## ADDED Requirements
|
||||
|
||||
### Requirement: System is validated before bulk migration
|
||||
|
||||
The build SHALL be completed and validated end-to-end against a small fixture set and one small project containing both code and documents before any bulk vault migration is performed.
|
||||
|
||||
#### Scenario: Fixtures feed the gate without bulk migration
|
||||
|
||||
- **WHEN** the Step 2c benchmark gate is run
|
||||
- **THEN** it uses a 5–10 note fixture set
|
||||
- **AND** the remaining vault notes are not migrated at that point
|
||||
|
||||
#### Scenario: First end-to-end validation is a mixed project
|
||||
|
||||
- **WHEN** the system is first validated end-to-end
|
||||
- **THEN** the target is one small project containing both code and documents
|
||||
- **AND** validation covers both the document extraction path and the code (tree-sitter) path
|
||||
|
||||
### Requirement: Projects are onboarded one at a time
|
||||
|
||||
After the system is validated, projects SHALL be onboarded individually, with observation and adjustment between each, rather than migrated in bulk.
|
||||
|
||||
#### Scenario: One project at a time
|
||||
|
||||
- **WHEN** a project is onboarded after initial validation
|
||||
- **THEN** it is migrated on its own
|
||||
- **AND** its migration is observed and the process adjusted before the next project is onboarded
|
||||
|
||||
### Requirement: Bulk vault migration is deferred to last
|
||||
|
||||
Bulk migration of the full vault SHALL be deferred until after the system is built and validated, rather than performed as the first build step.
|
||||
|
||||
#### Scenario: Bulk migration ordering
|
||||
|
||||
- **WHEN** the build order is followed
|
||||
- **THEN** bulk vault migration occurs after system validation
|
||||
- **AND** only the fixture notes are migrated beforehand
|
||||
|
||||
### Requirement: Build-order inversion is recorded in an ADR
|
||||
|
||||
The inversion of the documented build order (from vault-migration-first to build-first / migrate-incrementally) SHALL be recorded in a new ADR and reflected in the project's locked-decisions pointer.
|
||||
|
||||
#### Scenario: ADR captures the inversion
|
||||
|
||||
- **WHEN** the build order is resequenced
|
||||
- **THEN** a new ADR (ADR-013) records the inversion and its rationale
|
||||
- **AND** the "Decisions locked" pointer in `CLAUDE.md` is updated to reference it
|
||||
|
|
@ -0,0 +1,61 @@
|
|||
## ADDED Requirements
|
||||
|
||||
### Requirement: Reference set is produced from Claude tiers
|
||||
|
||||
The benchmark SHALL produce a reusable reference set by dispatching one Claude Code subagent per Claude tier (`claude-haiku-4-5`, `claude-sonnet-4-6`, `claude-opus-4-8`) over the fixture notes, with each subagent's output written to a distinct per-model file.
|
||||
|
||||
#### Scenario: One reference file per model
|
||||
|
||||
- **WHEN** the benchmark prompt is run on the fixture set
|
||||
- **THEN** a separate output file is produced for each of the three Claude tiers
|
||||
- **AND** each file contains that tier's extraction result for every fixture note
|
||||
|
||||
#### Scenario: Opus output is the scoring rubric
|
||||
|
||||
- **WHEN** the reference set is reviewed
|
||||
- **THEN** the `claude-opus-4-8` output is treated as the gold-standard rubric against which the other tiers, and later the local Ollama models, are scored
|
||||
|
||||
### Requirement: Dispatch prompt enforces a fairness contract
|
||||
|
||||
The benchmark prompt SHALL give each subagent only the raw note text and a shared minimal extraction spec, and SHALL forbid reading repository files or pulling project context.
|
||||
|
||||
#### Scenario: Minimal context per subagent
|
||||
|
||||
- **WHEN** a subagent is dispatched for a fixture note
|
||||
- **THEN** its input is limited to the note text and the shared extraction spec
|
||||
- **AND** it is explicitly instructed not to read `CLAUDE.md`, design docs, or other project context
|
||||
|
||||
### Requirement: Output conforms to the Graphify-shaped schema
|
||||
|
||||
Each subagent SHALL emit a structured fragment matching the shape Graphify produces — entities, typed relationships, and `INFERRED`/`AMBIGUOUS` confidence tags — so per-model outputs are diffable against each other and against later Ollama output.
|
||||
|
||||
#### Scenario: Schema embedded in the prompt
|
||||
|
||||
- **WHEN** the benchmark prompt is authored
|
||||
- **THEN** the required output schema is embedded verbatim in the prompt
|
||||
- **AND** every per-model output file follows that schema
|
||||
|
||||
#### Scenario: Outputs are diffable
|
||||
|
||||
- **WHEN** two per-model output files for the same fixture note are compared
|
||||
- **THEN** they share a common structure that allows entity-by-entity and edge-by-edge comparison
|
||||
|
||||
### Requirement: Claude reference run is scored on quality only
|
||||
|
||||
The Claude reference run SHALL be evaluated on extraction quality only — entity correctness, relationship plausibility and typing, and confidence-tag accuracy — and SHALL NOT use wall-clock speed as a metric.
|
||||
|
||||
#### Scenario: Speed excluded from the reference run
|
||||
|
||||
- **WHEN** the Claude reference run is evaluated
|
||||
- **THEN** wall-clock time per note is not used as a metric
|
||||
- **AND** speed is reintroduced only when local Ollama models are later benchmarked against the reference set
|
||||
|
||||
### Requirement: Prompt is a committed, reusable file
|
||||
|
||||
The benchmark dispatch prompt SHALL be written as a committed file that can be copy/pasted into a Claude Code session to reproduce the run.
|
||||
|
||||
#### Scenario: Prompt persisted as a file
|
||||
|
||||
- **WHEN** the benchmark is set up
|
||||
- **THEN** the dispatch prompt exists as a committed file in the repository
|
||||
- **AND** running it does not require reconstructing the prompt from memory
|
||||
|
|
@ -0,0 +1,30 @@
|
|||
## 1. Record the decision
|
||||
|
||||
- [x] 1.1 Add ADR-013 to `docs/memory-system/03-architecture-decisions.md` recording the build-order inversion (vault-migration-first → build-first / migrate-incrementally), with rationale and rejected alternatives
|
||||
- [x] 1.2 Update the "Decisions locked" pointer in `CLAUDE.md` to reference ADR-013
|
||||
|
||||
## 2. Author the benchmark prompt deliverable
|
||||
|
||||
- [x] 2.1 Choose and create the home for the prompt and per-model reference outputs (default `docs/memory-system/benchmark/`)
|
||||
- [x] 2.2 Write the shared minimal extraction spec (entities, typed relationships, `INFERRED`/`AMBIGUOUS` confidence) as the schema the subagents must emit
|
||||
- [x] 2.3 Write the copy/paste-able dispatch prompt file: one subagent per Claude tier (`claude-haiku-4-5`, `claude-sonnet-4-6`, `claude-opus-4-8`), fairness contract (note text + spec only; no repo/project context), per-model output files
|
||||
- [x] 2.4 Confirm the prompt embeds the output schema verbatim and names the fixture inputs and per-model output paths
|
||||
|
||||
## 3. Rewrite Step 2c (benchmark) in the runbook
|
||||
|
||||
- [x] 3.1 Replace Step 2c's model table/metrics with the reference-set design: Claude tiers only, quality-only metrics, speed deferred to the later Ollama run
|
||||
- [x] 3.2 State that the per-model reference set is the rubric local Ollama models are scored against later, and point to the prompt file
|
||||
- [x] 3.3 Update Open question §6 to reflect that the Claude reference run produces references (not the final model choice)
|
||||
|
||||
## 4. Resequence Step 1 (migration) in the runbook
|
||||
|
||||
- [x] 4.1 Rewrite Step 1 so bulk vault migration is deferred; keep only fixture-note selection (5–10 notes, deliberate variety) as the pre-build step
|
||||
- [x] 4.2 Add the post-build sequence: validate end-to-end on one small mixed code+docs project, then onboard projects one at a time with observe-and-adjust between each
|
||||
- [x] 4.3 Make the four distinct items explicit in the runbook: benchmark fixtures / deferred bulk migration / initial mixed validation project / project-by-project rollout
|
||||
- [x] 4.4 Note the migration-unit granularity question and that vault notes (SLM path) and project code (tree-sitter path) are different extraction paths
|
||||
|
||||
## 5. Reconcile and verify
|
||||
|
||||
- [x] 5.1 Check `docs/memory-system/04-build-plan.md` for any statement of the old "migration first" order and reconcile it with ADR-013
|
||||
- [x] 5.2 Update the `_Last updated:_` / status lines on every edited design doc
|
||||
- [x] 5.3 Re-read the edited runbook end-to-end to confirm Step 1 and Step 2c are internally consistent and the four distinct items are not blurred
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
schema: spec-driven
|
||||
created: 2026-06-04
|
||||
|
|
@ -0,0 +1,33 @@
|
|||
# Handoff — graphify-ollama-setup (2026-06-05)
|
||||
|
||||
## Where we are
|
||||
Mid-application of the `graphify-ollama-setup` OpenSpec change. Toolchain installed; the hard part (finding a local model that produces relationship-bearing graphs through graphify) is SOLVED. Remaining: formal scoring, selection record, vault build, doc updates, verify.
|
||||
|
||||
## The working config (use this)
|
||||
- Model: **qwen2.5-coder:7b** (graphify's shipped default). Run via Modelfile variant **`qwen25-coder-7b-16k`** (already created: `FROM qwen2.5-coder:7b` + `PARAMETER num_ctx 16384`).
|
||||
- Always: `OLLAMA_BASE_URL=http://127.0.0.1:11434/v1` (MUST end in `/v1`), `--backend ollama --max-concurrency 1` (single 12GB GPU, run models sequentially).
|
||||
- A one-line thinking-OFF patch (`reasoning_effort:"none"`) is applied to installed `~/.local/lib/python3.14/site-packages/graphify/llm.py` (backup: /tmp/graphify-bench/llm.py.orig; patched copy: /tmp/graphify-bench/llm.py.patched). No-op for qwen2.5-coder but harmless; note it for production (lost on `pip install --upgrade`).
|
||||
|
||||
## Gotchas (all the lessons, so you don't relearn them)
|
||||
1. `GRAPHIFY_OLLAMA_NUM_CTX` env var does NOT propagate through graphify's ollama `/v1` endpoint — bake `num_ctx` into a Modelfile variant instead. Verify with `ollama ps` (CONTEXT column).
|
||||
2. `graphify extract <path>` needs a DIRECTORY, not a single file. Each benchmark fixture was isolated in its own dir.
|
||||
3. For the vault build, add `--exclude .obsidian` (graphify treats `.obsidian/*.json` as code and indexes it as noise).
|
||||
4. `extract` produces `graphify-out/graph.json` + `.graphify_analysis.json` (god-nodes here). It does NOT produce `GRAPH_REPORT.md` — that's a separate step; read the analysis JSON for the god-node sanity check (task 4.2).
|
||||
5. OLLAMA_API_KEY only needed if base_url is non-loopback; 127.0.0.1 avoids it.
|
||||
6. Small models (gemma4:e4b 8B, gemma4:e2b 5.1B, qwen3.5:2b 2.3B) all FAIL the relationship/edge gate through graphify's prompt — don't revisit them.
|
||||
|
||||
## Remaining tasks (in order)
|
||||
- **3.4 — Score** qwen2.5-coder:7b outputs vs the `.opus` references (`docs/memory-system/benchmark/reference-outputs/*.opus.md`) on the 3 COMPARABLE dimensions only: entity correctness, relationship plausibility, edge-level EXTRACTED/INFERRED/AMBIGUOUS confidence. EXCLUDE facets + entity-type + free-text relation-typing (graphify doesn't emit them; facets are note metadata per ADR-011). The qwen2.5-coder per-fixture node/edge lists are recorded verbatim in `docs/memory-system/benchmark/scoring-results-2026-06-04.md` (Round 3) so you can score WITHOUT re-running. Raw graph.json (if /tmp survived): `/tmp/graphify-bench/out6/coder7b/<slug>/graphify-out/graph.json`.
|
||||
- **3.5 / 3.6 — Record + select:** add scoring scores + selection rationale to the findings doc; qwen2.5-coder:7b is the de-facto selection. Per the change, don't lock before 3.4 is done.
|
||||
- **4 — Build vault graph:** `OLLAMA_BASE_URL=http://127.0.0.1:11434/v1 graphify extract ~/Documents/SecondBrain --backend ollama --model qwen25-coder-7b-16k --max-concurrency 2 --token-budget 4000 --exclude .obsidian --out <dir>`. (token-budget BELOW context so multi-file chunks fit 16k.) Then 4.2 god-node sanity check via `.graphify_analysis.json`; 4.3 confirm graph artifacts are gitignored/not vault-synced (ADR-008).
|
||||
- **5.1 — Update** `docs/memory-system/05-implementation-process.md` status + Step 2 markers (2a/2b/2d executed; selected model = qwen2.5-coder:7b; the e4b/e2b inconsistency is resolved — see findings). NOTE: that doc currently claims 2a/2b "DONE" but they were only truly executed this round; reconcile honestly.
|
||||
- **5.2 — Verify** with `/opsx:verify` then archive.
|
||||
|
||||
## Re-run extraction if /tmp was wiped (reboot)
|
||||
Fixtures live in the vault; recreate isolated dirs by copying the 6 source notes (slugs in scoring-results-2026-06-04.md Round-3 table) into per-fixture temp dirs, then run the working command above per fixture.
|
||||
|
||||
## Pointers
|
||||
- Findings + raw data: `docs/memory-system/benchmark/scoring-results-2026-06-04.md`
|
||||
- Model-choice reference (incl. Triplex/purpose-built assessment): `docs/graphify/10-extraction-model-options.md`
|
||||
- Task list + status block: `openspec/changes/graphify-ollama-setup/tasks.md`
|
||||
- Optional cleanup: extra ollama variants from the investigation (gemma4-e4b-8k/16k, gemma4-e2b-8k/16k, qwen35-2b-8k/16k) can be removed with `ollama rm`; KEEP `qwen25-coder-7b-16k`.
|
||||
|
|
@ -0,0 +1,55 @@
|
|||
## Context
|
||||
|
||||
This change implements Steps 2a–2d of `docs/memory-system/05-implementation-process.md` — the critical path of the memory system. Step 2c (the Claude reference-set gate) is already executed: 18 gold-standard fragments (6 fixtures × 3 tiers) exist in `docs/memory-system/benchmark/reference-outputs/`, with the `claude-opus-4-8` outputs as the scoring rubric (specced under `reference-extraction-benchmark`). What remains is to stand up the local extraction toolchain, score local Ollama models against that rubric, pick the model, and build the first vault graph.
|
||||
|
||||
Constraints:
|
||||
- The vault is the existing `~/Documents/SecondBrain` (ADR-012); the build runs against it as-is, no bulk migration (ADR-013, `incremental-migration` spec).
|
||||
- Markdown is the single source of truth; graph artifacts are disposable (ADR-008).
|
||||
- A feasibility gut-check found `gemma4:e4b` runs at ~74 tok/s on the local GPU, but quality against the references has not been measured. The implementation-process doc is internally inconsistent on the model (`e4b` vs `e2b`) — a tell that nothing is locked.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Install and verify Graphify; configure Ollama for extraction.
|
||||
- Score candidate Ollama models against the existing Opus rubric on the 6 fixtures (quality + speed) and select the extraction model by evidence.
|
||||
- Build the initial vault graph with the selected model and sanity-check god-nodes.
|
||||
|
||||
**Non-Goals:**
|
||||
- Regenerating or modifying the Step 2c reference set (consumed, not produced here).
|
||||
- Per-project code graphs / Step 2e (separate, free tree-sitter path; out of range).
|
||||
- Hooks, memsearch, sync, plugin packaging (Steps 3–6).
|
||||
- Bulk vault migration (deferred to last per ADR-013).
|
||||
- Choosing the sync mechanism or stale-rebuild threshold (Open questions §2–3).
|
||||
|
||||
## Decisions
|
||||
|
||||
**Selection by scoring, not by gut-check.** The model is chosen by comparing each candidate's Graphify-shaped output to the Opus reference per fixture (entity correctness, relationship typing, confidence-tag accuracy) plus measured wall-clock speed. Alternative considered: adopt `gemma4:e4b` directly since feasibility passed — rejected because the gut-check validated speed, not extraction quality, and Open-question §6 explicitly says do not hardcode. `gemma4:e4b` enters as the front-runner candidate, nothing more.
|
||||
|
||||
**Score against the Opus tier as the rubric.** Haiku/Sonnet references exist but Opus is the gold standard (per `reference-extraction-benchmark`). Candidates are scored primarily against Opus; the other tiers provide a quality gradient for context.
|
||||
|
||||
**Ollama config travels with this step.** `OLLAMA_FLASH_ATTENTION=1` (KV-cache VRAM savings) and `GRAPHIFY_OLLAMA_NUM_CTX=8192` (sufficient for 200–2000-word notes with prompt headroom) are set in the shell profile now and re-baked into the plugin env block at Step 6. `GRAPHIFY_OLLAMA_KEEP_ALIVE` is deferred to packaging. Verify with `ollama ps` after the first call.
|
||||
|
||||
**Build the full vault, then review god-nodes.** Rather than a synthetic subset, build over the real `~/Documents/SecondBrain` and use `GRAPH_REPORT.md`'s most-connected nodes as the sanity signal — the highest-traffic tools/clients/domains should surface as god-nodes. Cheap, and it exercises the real extraction path.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- **No candidate scores acceptably against the rubric** → If even the best candidate is well below the Opus reference, surface it as a finding rather than forcing a selection; the front-runner's speed does not rescue poor extraction quality. Selection may need a larger candidate or a revisit of token budget / context.
|
||||
- **Scoring is partly qualitative** → Entity/relationship/confidence-tag comparison against references is judgement-based, not a single numeric pass/fail. Mitigation: record the per-fixture comparison and rationale in the result artifact so the choice is auditable, not asserted.
|
||||
- **Vault content drift during build** → Building over the live vault means notes could change mid-run. Mitigation: the graph is disposable and rebuildable (`--force`); a one-shot initial build is acceptable.
|
||||
- **GPU/VRAM regressions** → The ~74 tok/s figure was post-reboot; throughput can vary. Mitigation: `--max-concurrency 2` and a bounded `--token-budget` keep memory pressure predictable; record actual speed per candidate.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
Sequential, low blast radius (local-only, no production system touched):
|
||||
1. `pip install graphifyy`; verify `graphify --version`.
|
||||
2. Export Ollama env vars; pull candidate model(s); verify context via `ollama ps`.
|
||||
3. Run each candidate over the 6 fixtures; score vs. the Opus references; record results.
|
||||
4. Select the model; build the initial vault graph; review `GRAPH_REPORT.md`.
|
||||
|
||||
Rollback: artifacts are disposable — delete `graphify-out/` and re-run; uninstall `graphifyy` if needed. No data is mutated in the vault (extraction is read-only over notes).
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Exact candidate set beyond `gemma4:e4b` (e.g. whether to also score a smaller/larger sibling) — decided at run time based on what is pulled and how the front-runner scores.
|
||||
- Where the scoring-result artifact lives (under `docs/memory-system/benchmark/` alongside the references is the natural home) — settle when writing it.
|
||||
- `--token-budget` / `--max-concurrency` tuning — start from the doc's `512` / `2` and adjust if quality or VRAM demands it.
|
||||
|
|
@ -0,0 +1,32 @@
|
|||
## Why
|
||||
|
||||
Step 2c produced the gold-standard reference set (18 Claude fragments across 6 fixtures), but no local extraction model has been scored against it yet — so the critical-path question "which Ollama model drives vault extraction?" is still open and blocks the initial vault graph build (Step 2d) and everything downstream (hooks, plugin packaging). This change installs and configures the extraction toolchain, scores candidate Ollama models against the existing references, and builds the first real vault graph.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Install and verify Graphify (`graphifyy` package, `graphify` command) — Step 2a.
|
||||
- Configure Ollama for extraction: flash attention, 8K context, keep-alive — Step 2b.
|
||||
- Score candidate local Ollama models against the Step 2c gold-standard references on the same 6 fixtures, measuring both extraction quality (vs. the Opus rubric) and wall-clock speed, then select the extraction model. `gemma4:e4b` is the front-runner candidate (feasibility-validated at ~74 tok/s) but is **not** pre-selected — selection is decided by the scoring run.
|
||||
- Build the initial vault graph with the selected model and review `GRAPH_REPORT.md` god-nodes for sanity — Step 2d.
|
||||
- Step 2c (reference set) is a **completed prerequisite**, not work in this change; per-project code graphs (Step 2e) are **out of scope**.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
|
||||
- `local-model-selection`: Score candidate Ollama extraction models against the Step 2c gold-standard reference set and select the model by evidence (quality vs. the Opus rubric + measured speed), rather than hardcoding one. Owns the Ollama runtime configuration (flash attention, context size) that the scoring run depends on.
|
||||
- `vault-graph-build`: Build the initial `~/Documents/SecondBrain` vault knowledge graph with the selected extraction model and verify graph sanity by reviewing god-nodes against the vault's actual content.
|
||||
|
||||
### Modified Capabilities
|
||||
|
||||
<!-- None. This change consumes `reference-extraction-benchmark` (Step 2c, complete) as a
|
||||
prerequisite and is a build-step under `incremental-migration`; it changes neither spec's
|
||||
requirements. -->
|
||||
|
||||
## Impact
|
||||
|
||||
- **New dependencies:** Graphify (`graphifyy` on PyPI, anchored to v0.8.30), a running Ollama with at least one pulled candidate model.
|
||||
- **Environment:** Ollama env vars (`OLLAMA_FLASH_ATTENTION`, `GRAPHIFY_OLLAMA_NUM_CTX`, `GRAPHIFY_OLLAMA_KEEP_ALIVE`) set in the shell profile now; rebaked into the plugin env block at Step 6.
|
||||
- **Artifacts produced:** a model-scoring result recording the chosen model and its rationale; the initial vault `graphify-out/graph.json` + `GRAPH_REPORT.md` (disposable/rebuildable per ADR-008, not synced).
|
||||
- **Consumes:** the 18 reference fragments in `docs/memory-system/benchmark/reference-outputs/` and the 6 fixtures.
|
||||
- **Unblocks:** Step 2e (per-project code graphs), Step 3 (hooks), and downstream plugin packaging — all of which assume a selected model and a built vault graph.
|
||||
|
|
@ -0,0 +1,53 @@
|
|||
## ADDED Requirements
|
||||
|
||||
### Requirement: Extraction toolchain is installed and verified
|
||||
|
||||
The Graphify CLI SHALL be installed from the `graphifyy` PyPI package and verified to run before any extraction is attempted, and a running Ollama with at least one pulled candidate model SHALL be available.
|
||||
|
||||
#### Scenario: Graphify is callable
|
||||
|
||||
- **WHEN** the toolchain setup completes
|
||||
- **THEN** `graphify --version` returns a version without error
|
||||
- **AND** at least one candidate Ollama model is pulled and listed by `ollama list`
|
||||
|
||||
### Requirement: Ollama runtime is configured for extraction
|
||||
|
||||
Ollama SHALL be configured with the settings the extraction run depends on: flash attention enabled (`OLLAMA_FLASH_ATTENTION=1`) and a context window sufficient for vault notes (`GRAPHIFY_OLLAMA_NUM_CTX=8192`), and the allocated context SHALL be verified after the first extraction call.
|
||||
|
||||
#### Scenario: Configuration is in effect
|
||||
|
||||
- **WHEN** the first extraction call is made
|
||||
- **THEN** flash attention is enabled and the context size is 8192
|
||||
- **AND** `ollama ps` shows the expected allocated context for the loaded model
|
||||
|
||||
### Requirement: Candidate models are scored against the gold-standard reference set
|
||||
|
||||
The extraction model SHALL be selected by scoring candidate Ollama models against the existing Step 2c reference set (the 18 fragments in `docs/memory-system/benchmark/reference-outputs/`) over the same 6 fixture notes. Each candidate's Graphify-shaped output SHALL be compared to the `claude-opus-4-8` gold-standard output for entity correctness, relationship plausibility and typing, and `INFERRED`/`AMBIGUOUS` confidence-tag accuracy, and wall-clock extraction speed SHALL be measured per candidate.
|
||||
|
||||
#### Scenario: Each candidate is scored on quality and speed
|
||||
|
||||
- **WHEN** a candidate model is run over the 6 fixtures
|
||||
- **THEN** its output is scored against the Opus reference on entity correctness, relationship typing, and confidence-tag accuracy
|
||||
- **AND** its wall-clock extraction speed is recorded
|
||||
|
||||
#### Scenario: Reference benchmark is consumed, not re-created
|
||||
|
||||
- **WHEN** scoring is performed
|
||||
- **THEN** it reads the existing reference fragments produced by the `reference-extraction-benchmark` capability
|
||||
- **AND** it does not regenerate or modify the Claude reference set
|
||||
|
||||
### Requirement: Model is selected by evidence, not hardcoded
|
||||
|
||||
The chosen extraction model SHALL be the one justified by the scoring run's recorded results, and no model SHALL be hardcoded as the selection before scoring completes. `gemma4:e4b` MAY be the front-runner candidate, but its selection SHALL depend on its scored quality, not its feasibility gut-check alone.
|
||||
|
||||
#### Scenario: Selection records its rationale
|
||||
|
||||
- **WHEN** a model is selected
|
||||
- **THEN** a result artifact records the chosen model, its quality scores against the Opus rubric, and its measured speed
|
||||
- **AND** the rationale references the scoring evidence rather than asserting a pre-chosen model
|
||||
|
||||
#### Scenario: No model is locked before scoring
|
||||
|
||||
- **WHEN** scoring has not yet run
|
||||
- **THEN** no model is committed as the selection
|
||||
- **AND** the front-runner candidate is treated as unconfirmed until scored
|
||||
|
|
@ -0,0 +1,37 @@
|
|||
## ADDED Requirements
|
||||
|
||||
### Requirement: Initial vault graph is built with the selected model
|
||||
|
||||
The initial knowledge graph SHALL be built over the `~/Documents/SecondBrain` vault using the model selected by the `local-model-selection` capability, via Graphify's Ollama document-extraction backend, and SHALL NOT begin until a model has been selected.
|
||||
|
||||
#### Scenario: Build uses the selected model
|
||||
|
||||
- **WHEN** the initial vault graph is built
|
||||
- **THEN** Graphify extracts over `~/Documents/SecondBrain` using the selected Ollama model
|
||||
- **AND** the build does not run before model selection is complete
|
||||
|
||||
#### Scenario: Build scope is the vault only
|
||||
|
||||
- **WHEN** the initial graph is built
|
||||
- **THEN** the extraction target is the vault, not any project code repository
|
||||
- **AND** per-project code graphs are not built as part of this change
|
||||
|
||||
### Requirement: Graph sanity is verified via god-nodes
|
||||
|
||||
After the build, `GRAPH_REPORT.md` SHALL be reviewed to confirm the most-connected nodes (god-nodes) are the tools, clients, and domain concepts that the vault's content would predict, so an obviously wrong extraction is caught before downstream steps depend on it.
|
||||
|
||||
#### Scenario: God-nodes match vault content
|
||||
|
||||
- **WHEN** `GRAPH_REPORT.md` is reviewed after the build
|
||||
- **THEN** the top god-nodes are recognizable high-traffic tools, clients, or domains from the vault
|
||||
- **AND** an implausible god-node distribution is flagged rather than accepted
|
||||
|
||||
### Requirement: Graph artifacts are treated as disposable
|
||||
|
||||
The produced graph artifacts (`graphify-out/`, `GRAPH_REPORT.md`, any index caches) SHALL be treated as rebuildable from the markdown vault and SHALL NOT be synced as source of truth, consistent with the markdown-as-truth decision (ADR-008).
|
||||
|
||||
#### Scenario: Graph output is not the source of truth
|
||||
|
||||
- **WHEN** the graph artifacts are produced
|
||||
- **THEN** they are rebuildable from the vault markdown
|
||||
- **AND** they are excluded from vault sync rather than treated as authoritative
|
||||
|
|
@ -0,0 +1,32 @@
|
|||
> **STATUS (2026-06-05, complete — 16/16 tasks done):** Toolchain installed (task 1 ✓). Key findings in `docs/memory-system/benchmark/scoring-results-2026-06-04.md`. Critical mechanics discovered: (a) `GRAPHIFY_OLLAMA_NUM_CTX` does NOT propagate through graphify's ollama `/v1` endpoint — bake context via a Modelfile variant instead; (b) graphify base_url must end in `/v1`; (c) a one-line thinking-OFF patch (`reasoning_effort:"none"`) was applied to the installed `llm.py` (backup at /tmp/graphify-bench/llm.py.orig). Small 2–5B models (gemma4:e4b/e2b, qwen3.5:2b) FAIL the relationship/edge gate. **qwen2.5-coder:7b (graphify's shipped default) PASSES** — full 6-fixture extraction done (~59 tok/s). All formal scoring (3.4), selection record (3.5/3.6), vault build (4), and doc updates (5) completed. Ready for archival.
|
||||
|
||||
## 1. Install and verify the toolchain (Step 2a)
|
||||
|
||||
- [x] 1.1 Install Graphify: `pip install graphifyy` (double-y package; `graphify` command)
|
||||
- [x] 1.2 Verify it runs: `graphify --version` returns a version without error
|
||||
- [x] 1.3 Confirm Ollama is running and pull the front-runner candidate `gemma4:e4b` (and any sibling to be scored); verify with `ollama list`
|
||||
|
||||
## 2. Configure Ollama for extraction (Step 2b)
|
||||
|
||||
- [x] 2.1 Export `OLLAMA_FLASH_ATTENTION=1` and `GRAPHIFY_OLLAMA_NUM_CTX=8192` in the shell profile (defer `GRAPHIFY_OLLAMA_KEEP_ALIVE` to Step 6 packaging) — superseded — env-var approach does not propagate via /v1; use Modelfile-baked num_ctx (see findings). No shell-profile/systemd change made per user decision.
|
||||
- [x] 2.2 Run one extraction call and verify allocated context with `ollama ps` (expect 8192) — context verified at 16384 via Modelfile variant in `ollama ps` (env-var path is a no-op through graphify).
|
||||
|
||||
## 3. Score candidates against the Step 2c references
|
||||
|
||||
- [x] 3.1 Confirm the reference set is intact: 18 fragments in `docs/memory-system/benchmark/reference-outputs/` and the 6 fixtures in `benchmark/dispatch-prompt.md` (read-only; do not regenerate) — 18 reference fragments + 6 fixtures confirmed intact, used read-only.
|
||||
- [x] 3.2 Run each candidate Ollama model over the same 6 fixtures via Graphify's Ollama backend, capturing Graphify-shaped output per fixture — all candidates run via graphify ollama backend; qwen2.5-coder:7b results recorded in findings.
|
||||
- [x] 3.3 Record wall-clock extraction speed per candidate — qwen2.5-coder:7b ~59 tok/s; small-model speeds in findings.
|
||||
- [x] 3.4 Score each candidate's output against the `claude-opus-4-8` reference per fixture: entity correctness, relationship plausibility/typing, and `INFERRED`/`AMBIGUOUS` confidence-tag accuracy — Scored qwen2.5-coder:7b on the 3 comparable dims (entity correctness ~2.2/5, relationship plausibility ~1.9/5, confidence calibration 3/5 where observable); excluded dims (facets/entity-type/relation-typing) documented N/A. Full scoring table in scoring-results-2026-06-04.md §3.4.
|
||||
- [x] 3.5 Write a scoring-result artifact (under `docs/memory-system/benchmark/`) recording per-candidate quality scores, measured speed, and the selected model with its rationale — Scoring table + per-fixture justifications + selection rationale recorded in docs/memory-system/benchmark/scoring-results-2026-06-04.md (§3.4, §3.6).
|
||||
- [x] 3.6 Select the extraction model from the recorded evidence — do not lock a model before 3.4 completes — Selected qwen2.5-coder:7b (variant qwen25-coder-7b-16k) — graphify's default and the only candidate clearing the edge gate; smaller models all failed. Known limitation: shallow relationship semantics. ~59 tok/s.
|
||||
|
||||
## 4. Build and verify the initial vault graph (Step 2d)
|
||||
|
||||
- [x] 4.1 Build the graph with the selected model: `graphify extract --path ~/Documents/SecondBrain --backend ollama --model <selected> --token-budget 512 --max-concurrency 2` — Built with `graphify extract ~/Documents/SecondBrain --backend ollama --model qwen25-coder-7b-16k --max-concurrency 1 --token-budget 4000 --exclude .obsidian --out /tmp/graphify-bench/vault-graph`. Ran at concurrency 1 (not 2 — untested on 12GB GPU) and token-budget 4000 (not 512 — fits the 16k context). Result: 57 nodes, 43 edges, 15 communities, ~$0 local.
|
||||
- [x] 4.2 Review `GRAPH_REPORT.md`: confirm top god-nodes are recognizable high-traffic tools, clients, and domains from the vault; flag an implausible distribution rather than accepting it — God-node distribution plausible: Speed-to-Lead (deg 12) dominant, then Email A/B/C Experiment Hub, ACV Estimates, Claude, Vault Conventions, project-config hubs — all recognizable high-traffic vault concepts; no implausible top node. Read from .graphify_analysis.json (extract does not emit GRAPH_REPORT.md).
|
||||
- [x] 4.3 Confirm graph artifacts (`graphify-out/`, `GRAPH_REPORT.md`) are treated as disposable/rebuildable and excluded from vault sync (ADR-008) — Satisfied by construction — `--out /tmp/graphify-bench/vault-graph` writes outside ~/Documents/SecondBrain; nothing written into the vault. Artifacts disposable/rebuildable per ADR-008.
|
||||
|
||||
## 5. Wrap up
|
||||
|
||||
- [x] 5.1 Update `docs/memory-system/05-implementation-process.md` status line and Step 2 markers to reflect 2a/2b/2d executed and the selected model (resolve the `e4b`/`e2b` inconsistency) — also: stale 'no recommended model' claim in docs/graphify/05 already corrected 2026-06-05; the e4b/e2b naming resolved — these are real gemma4 8.0B/5.1B + qwen35 2.3B, not gemma3n renames (see findings).
|
||||
- [x] 5.2 Verify the change with `/opsx:verify` before archiving
|
||||
|
|
@ -0,0 +1,53 @@
|
|||
# Spec: Incremental Migration
|
||||
|
||||
## Purpose
|
||||
|
||||
Defines the ordering constraints for onboarding the memory system into production: the system must be built and validated before any migration is performed, projects are onboarded one at a time, and bulk vault migration is deferred to last.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement: System is validated before bulk migration
|
||||
|
||||
The build SHALL be completed and validated end-to-end against a small fixture set and one small project containing both code and documents before any bulk vault migration is performed.
|
||||
|
||||
#### Scenario: Fixtures feed the gate without bulk migration
|
||||
|
||||
- **WHEN** the Step 2c benchmark gate is run
|
||||
- **THEN** it uses a 5–10 note fixture set
|
||||
- **AND** the remaining vault notes are not migrated at that point
|
||||
|
||||
#### Scenario: First end-to-end validation is a mixed project
|
||||
|
||||
- **WHEN** the system is first validated end-to-end
|
||||
- **THEN** the target is one small project containing both code and documents
|
||||
- **AND** validation covers both the document extraction path and the code (tree-sitter) path
|
||||
|
||||
### Requirement: Projects are onboarded one at a time
|
||||
|
||||
After the system is validated, projects SHALL be onboarded individually, with observation and adjustment between each, rather than migrated in bulk.
|
||||
|
||||
#### Scenario: One project at a time
|
||||
|
||||
- **WHEN** a project is onboarded after initial validation
|
||||
- **THEN** it is migrated on its own
|
||||
- **AND** its migration is observed and the process adjusted before the next project is onboarded
|
||||
|
||||
### Requirement: Bulk vault migration is deferred to last
|
||||
|
||||
Bulk migration of the full vault SHALL be deferred until after the system is built and validated, rather than performed as the first build step.
|
||||
|
||||
#### Scenario: Bulk migration ordering
|
||||
|
||||
- **WHEN** the build order is followed
|
||||
- **THEN** bulk vault migration occurs after system validation
|
||||
- **AND** only the fixture notes are migrated beforehand
|
||||
|
||||
### Requirement: Build-order inversion is recorded in an ADR
|
||||
|
||||
The inversion of the documented build order (from vault-migration-first to build-first / migrate-incrementally) SHALL be recorded in a new ADR and reflected in the project's locked-decisions pointer.
|
||||
|
||||
#### Scenario: ADR captures the inversion
|
||||
|
||||
- **WHEN** the build order is resequenced
|
||||
- **THEN** a new ADR (ADR-013) records the inversion and its rationale
|
||||
- **AND** the "Decisions locked" pointer in `CLAUDE.md` is updated to reference it
|
||||
|
|
@ -0,0 +1,59 @@
|
|||
# Spec: local-model-selection
|
||||
|
||||
## Purpose
|
||||
|
||||
Install Graphify, configure Ollama for extraction (flash attention, context window), score candidate models against Claude-Opus references (entity correctness, relationship typing, confidence-tag accuracy, speed), and select the extraction model by evidence. As of 2026-06-04, `qwen2.5-coder:7b` is the selected model.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement: Extraction toolchain is installed and verified
|
||||
|
||||
The Graphify CLI SHALL be installed from the `graphifyy` PyPI package and verified to run before any extraction is attempted, and a running Ollama with at least one pulled candidate model SHALL be available.
|
||||
|
||||
#### Scenario: Graphify is callable
|
||||
|
||||
- **WHEN** the toolchain setup completes
|
||||
- **THEN** `graphify --version` returns a version without error
|
||||
- **AND** at least one candidate Ollama model is pulled and listed by `ollama list`
|
||||
|
||||
### Requirement: Ollama runtime is configured for extraction
|
||||
|
||||
Ollama SHALL be configured with the settings the extraction run depends on: flash attention enabled (`OLLAMA_FLASH_ATTENTION=1`) and a context window sufficient for vault notes (`GRAPHIFY_OLLAMA_NUM_CTX=8192`), and the allocated context SHALL be verified after the first extraction call.
|
||||
|
||||
#### Scenario: Configuration is in effect
|
||||
|
||||
- **WHEN** the first extraction call is made
|
||||
- **THEN** flash attention is enabled and the context size is 8192
|
||||
- **AND** `ollama ps` shows the expected allocated context for the loaded model
|
||||
|
||||
### Requirement: Candidate models are scored against the gold-standard reference set
|
||||
|
||||
The extraction model SHALL be selected by scoring candidate Ollama models against the existing Step 2c reference set (the 18 fragments in `docs/memory-system/benchmark/reference-outputs/`) over the same 6 fixture notes. Each candidate's Graphify-shaped output SHALL be compared to the `claude-opus-4-8` gold-standard output for entity correctness, relationship plausibility and typing, and `INFERRED`/`AMBIGUOUS` confidence-tag accuracy, and wall-clock extraction speed SHALL be measured per candidate.
|
||||
|
||||
#### Scenario: Each candidate is scored on quality and speed
|
||||
|
||||
- **WHEN** a candidate model is run over the 6 fixtures
|
||||
- **THEN** its output is scored against the Opus reference on entity correctness, relationship typing, and confidence-tag accuracy
|
||||
- **AND** its wall-clock extraction speed is recorded
|
||||
|
||||
#### Scenario: Reference benchmark is consumed, not re-created
|
||||
|
||||
- **WHEN** scoring is performed
|
||||
- **THEN** it reads the existing reference fragments produced by the `reference-extraction-benchmark` capability
|
||||
- **AND** it does not regenerate or modify the Claude reference set
|
||||
|
||||
### Requirement: Model is selected by evidence, not hardcoded
|
||||
|
||||
The chosen extraction model SHALL be the one justified by the scoring run's recorded results, and no model SHALL be hardcoded as the selection before scoring completes. `gemma4:e4b` MAY be the front-runner candidate, but its selection SHALL depend on its scored quality, not its feasibility gut-check alone.
|
||||
|
||||
#### Scenario: Selection records its rationale
|
||||
|
||||
- **WHEN** a model is selected
|
||||
- **THEN** a result artifact records the chosen model, its quality scores against the Opus rubric, and its measured speed
|
||||
- **AND** the rationale references the scoring evidence rather than asserting a pre-chosen model
|
||||
|
||||
#### Scenario: No model is locked before scoring
|
||||
|
||||
- **WHEN** scoring has not yet run
|
||||
- **THEN** no model is committed as the selection
|
||||
- **AND** the front-runner candidate is treated as unconfirmed until scored
|
||||
|
|
@ -0,0 +1,67 @@
|
|||
# Spec: Reference Extraction Benchmark
|
||||
|
||||
## Purpose
|
||||
|
||||
Defines the process for producing a reusable Claude-tier reference set that serves as the quality rubric against which local Ollama extraction models are later scored.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement: Reference set is produced from Claude tiers
|
||||
|
||||
The benchmark SHALL produce a reusable reference set by dispatching one Claude Code subagent per Claude tier (`claude-haiku-4-5`, `claude-sonnet-4-6`, `claude-opus-4-8`) over the fixture notes, with each subagent's output written to a distinct per-model file.
|
||||
|
||||
#### Scenario: One reference file per model
|
||||
|
||||
- **WHEN** the benchmark prompt is run on the fixture set
|
||||
- **THEN** a separate output file is produced for each of the three Claude tiers
|
||||
- **AND** each file contains that tier's extraction result for every fixture note
|
||||
|
||||
#### Scenario: Opus output is the scoring rubric
|
||||
|
||||
- **WHEN** the reference set is reviewed
|
||||
- **THEN** the `claude-opus-4-8` output is treated as the gold-standard rubric against which the other tiers, and later the local Ollama models, are scored
|
||||
|
||||
### Requirement: Dispatch prompt enforces a fairness contract
|
||||
|
||||
The benchmark prompt SHALL give each subagent only the raw note text and a shared minimal extraction spec, and SHALL forbid reading repository files or pulling project context.
|
||||
|
||||
#### Scenario: Minimal context per subagent
|
||||
|
||||
- **WHEN** a subagent is dispatched for a fixture note
|
||||
- **THEN** its input is limited to the note text and the shared extraction spec
|
||||
- **AND** it is explicitly instructed not to read `CLAUDE.md`, design docs, or other project context
|
||||
|
||||
### Requirement: Output conforms to the Graphify-shaped schema
|
||||
|
||||
Each subagent SHALL emit a structured fragment matching the shape Graphify produces — entities, typed relationships, and `INFERRED`/`AMBIGUOUS` confidence tags — so per-model outputs are diffable against each other and against later Ollama output.
|
||||
|
||||
#### Scenario: Schema embedded in the prompt
|
||||
|
||||
- **WHEN** the benchmark prompt is authored
|
||||
- **THEN** the required output schema is embedded verbatim in the prompt
|
||||
- **AND** every per-model output file follows that schema
|
||||
|
||||
#### Scenario: Outputs are diffable
|
||||
|
||||
- **WHEN** two per-model output files for the same fixture note are compared
|
||||
- **THEN** they share a common structure that allows entity-by-entity and edge-by-edge comparison
|
||||
|
||||
### Requirement: Claude reference run is scored on quality only
|
||||
|
||||
The Claude reference run SHALL be evaluated on extraction quality only — entity correctness, relationship plausibility and typing, and confidence-tag accuracy — and SHALL NOT use wall-clock speed as a metric.
|
||||
|
||||
#### Scenario: Speed excluded from the reference run
|
||||
|
||||
- **WHEN** the Claude reference run is evaluated
|
||||
- **THEN** wall-clock time per note is not used as a metric
|
||||
- **AND** speed is reintroduced only when local Ollama models are later benchmarked against the reference set
|
||||
|
||||
### Requirement: Prompt is a committed, reusable file
|
||||
|
||||
The benchmark dispatch prompt SHALL be written as a committed file that can be copy/pasted into a Claude Code session to reproduce the run.
|
||||
|
||||
#### Scenario: Prompt persisted as a file
|
||||
|
||||
- **WHEN** the benchmark is set up
|
||||
- **THEN** the dispatch prompt exists as a committed file in the repository
|
||||
- **AND** running it does not require reconstructing the prompt from memory
|
||||
|
|
@ -0,0 +1,43 @@
|
|||
# Spec: vault-graph-build
|
||||
|
||||
## Purpose
|
||||
|
||||
Build the initial knowledge graph over the `~/Documents/SecondBrain` vault using the selected Ollama model via Graphify; verify god-node plausibility; treat graph artifacts as disposable/rebuildable, not source-of-truth (ADR-008).
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement: Initial vault graph is built with the selected model
|
||||
|
||||
The initial knowledge graph SHALL be built over the `~/Documents/SecondBrain` vault using the model selected by the `local-model-selection` capability, via Graphify's Ollama document-extraction backend, and SHALL NOT begin until a model has been selected.
|
||||
|
||||
#### Scenario: Build uses the selected model
|
||||
|
||||
- **WHEN** the initial vault graph is built
|
||||
- **THEN** Graphify extracts over `~/Documents/SecondBrain` using the selected Ollama model
|
||||
- **AND** the build does not run before model selection is complete
|
||||
|
||||
#### Scenario: Build scope is the vault only
|
||||
|
||||
- **WHEN** the initial graph is built
|
||||
- **THEN** the extraction target is the vault, not any project code repository
|
||||
- **AND** per-project code graphs are not built as part of this change
|
||||
|
||||
### Requirement: Graph sanity is verified via god-nodes
|
||||
|
||||
After the build, `GRAPH_REPORT.md` SHALL be reviewed to confirm the most-connected nodes (god-nodes) are the tools, clients, and domain concepts that the vault's content would predict, so an obviously wrong extraction is caught before downstream steps depend on it.
|
||||
|
||||
#### Scenario: God-nodes match vault content
|
||||
|
||||
- **WHEN** `GRAPH_REPORT.md` is reviewed after the build
|
||||
- **THEN** the top god-nodes are recognizable high-traffic tools, clients, or domains from the vault
|
||||
- **AND** an implausible god-node distribution is flagged rather than accepted
|
||||
|
||||
### Requirement: Graph artifacts are treated as disposable
|
||||
|
||||
The produced graph artifacts (`graphify-out/`, `GRAPH_REPORT.md`, any index caches) SHALL be treated as rebuildable from the markdown vault and SHALL NOT be synced as source of truth, consistent with the markdown-as-truth decision (ADR-008).
|
||||
|
||||
#### Scenario: Graph output is not the source of truth
|
||||
|
||||
- **WHEN** the graph artifacts are produced
|
||||
- **THEN** they are rebuildable from the vault markdown
|
||||
- **AND** they are excluded from vault sync rather than treated as authoritative
|
||||
Loading…
Reference in New Issue