diff --git a/docs/graphify/00-README.md b/docs/graphify/00-README.md index 0690971..d24aa27 100644 --- a/docs/graphify/00-README.md +++ b/docs/graphify/00-README.md @@ -31,6 +31,10 @@ then verified and corrected against the official GitHub repository and supplemen command quick-reference + setting-up-across-many-projects mini-guide. **Keep this one open while working.** 10. **[external-tips.md](external-tips.md)** — Independent/community tips, gotchas with issue links, and an even-handed look at the token-savings debate. +11. **[10-extraction-model-options.md](10-extraction-model-options.md)** — Why Graphify uses a general + structured-output LLM (not a purpose-built KG extractor), the architecture constraints that make + drop-in specialist models (Triplex, GLiNER, REBEL) non-starters, and an honest assessment of whether + the Triplex adapter route is worth experimenting with. ## One-paragraph summary diff --git a/docs/graphify/05-local-models-and-backends.md b/docs/graphify/05-local-models-and-backends.md index 5fae0e3..f00f94f 100644 --- a/docs/graphify/05-local-models-and-backends.md +++ b/docs/graphify/05-local-models-and-backends.md @@ -138,7 +138,9 @@ graphify extract ./docs --backend ollama --api-timeout 900 # 15-minute timeout ### Which Ollama model? -**There is no official Graphify-recommended/tested model list.** Neither the v8 README nor the site publishes one (verified — see Open questions). So follow the rule the task itself sets and the interview implies: **pick an instruction-following model that fits your RAM/VRAM.** `[interview]` Document extraction is "read this chunk, emit structured entities/relationships," which is an instruction-following + structured-output job, not deep reasoning. +**There is no official Graphify-recommended/tested model list in the README or on the site.** Neither the v8 README nor the site publishes one (verified — see Open questions). So follow the rule the task itself sets and the interview implies: **pick an instruction-following model that fits your RAM/VRAM.** `[interview]` Document extraction is "read this chunk, emit structured entities/relationships," which is an instruction-following + structured-output job, not deep reasoning. + +> **Correction — there IS a shipped code default (2026-06-05).** Although the README/site publish no model recommendation, installed graphify 0.8.31 **hardcodes** a fallback in `llm.py:67`: `"default_model": os.environ.get("OLLAMA_MODEL", "qwen2.5-coder:7b")`. `[github]` So if you run `graphify extract --backend ollama` without setting `OLLAMA_MODEL`, the binary silently uses `qwen2.5-coder:7b`. The README's silence on a recommended model is accurate as a documentation statement; it is not accurate as a claim that "no default exists." The choice of a *coder* model for document extraction is a revealed preference — the binding requirement is structured-JSON instruction-following discipline, not domain NER. See [10-extraction-model-options.md](10-extraction-model-options.md) for the full analysis. Rough sizing rule of thumb: a model needs roughly its parameter count in **GB of (V)RAM** at common 4-bit quantization (a 7B model ≈ ~5 GB; a 30B+ model wants 24 GB+ VRAM). Leave headroom for the context window. @@ -195,7 +197,7 @@ graphify extract ./src ## Open questions / unverified -- **No official Ollama model recommendation exists.** Verified absent from the v8 README env/backend sections and not surfaced on the (403-gated) official site. The model names here are community/general-Ollama signals, tagged as such — not Graphify-tested defaults. +- **No official Ollama model recommendation exists in the README**, but the installed binary hardcodes `qwen2.5-coder:7b` as the Ollama fallback (`llm.py:67`) — see the Correction callout above and `10-extraction-model-options.md`. `[github]` - **`GRAPHIFY_OLLAMA_KEEP_ALIVE` default value** is documented by *behavior* ("minutes to keep loaded; `0` to unload after each chunk") but the README does not state the numeric default. `[github]` - **`gemma3:27b` exact tag / VRAM numbers** come from a single user issue (#792), not Graphify docs. The original ASR transcript said "Gemma 4 31b," which does not match a shipped tag; treat the community `gemma3:27b`-class figure as illustrative, not authoritative. `[community](https://github.com/safishamsi/graphify/issues/792)` - **The "SLMs as good as frontier models soon" vision** is the creator's aspiration, not a benchmark. `[unverified claim]` / `[interview]` diff --git a/docs/graphify/10-extraction-model-options.md b/docs/graphify/10-extraction-model-options.md new file mode 100644 index 0000000..ddf64c5 --- /dev/null +++ b/docs/graphify/10-extraction-model-options.md @@ -0,0 +1,131 @@ +# 10 — Extraction Model Options: Why a General Structured-Output LLM, and Is a Purpose-Built KG Model Worth It? + +_Last updated: 2026-06-05_ | _Status: reference / decision record_ + +This document explains why Graphify's document extraction uses a general instruction-following / structured-output LLM rather than a purpose-built knowledge-graph extraction model (e.g. Triplex, GLiNER, REBEL), and honestly assesses whether the purpose-built route is worth experimenting with for this project. + +See also: [05 — Local models & backends](05-local-models-and-backends.md) · [Benchmark results (2026-06-04)](../memory-system/benchmark/scoring-results-2026-06-04.md) + +--- + +## The architecture constraint (this is the crux) + +Graphify owns **both** the prompt and the output contract. It is not a thin wrapper you can swap a model into — the extraction pipeline is opinionated end-to-end: + +**Fixed system prompt.** For document/markdown extraction, Graphify sends its own system prompt `_EXTRACTION_SYSTEM` (`llm.py:218-232`). `[github]` There is no user hook to supply a custom extraction prompt; the prompt is hardcoded. + +**Fixed output schema.** The model must return exactly this JSON structure — no approximation accepted: + +```json +{ + "nodes": [{"id": "stem_entity", "label": "...", "file_type": "code|document|paper|image|rationale|concept", + "source_file": "...", "source_location": null, "source_url": null, + "captured_at": null, "author": null, "contributor": null}], + "edges": [{"source": "node_id", "target": "node_id", + "relation": "calls|implements|references|cites|conceptually_related_to|shares_data_with|semantically_similar_to", + "confidence": "EXTRACTED|INFERRED|AMBIGUOUS", + "confidence_score": 1.0, "source_file": "...", "source_location": null, "weight": 1.0}], + "hyperedges": [], "input_tokens": 0, "output_tokens": 0 +} +``` + +The `relation` field is constrained to a fixed vocabulary of seven values. The `confidence` field is a three-value enum (`EXTRACTED`/`INFERRED`/`AMBIGUOUS`). `[github]` (`llm.py:218-232`) + +**No JSON-mode / grammar enforcement.** For the Ollama backend, Graphify passes only `{"options": {"num_ctx": ...}, "keep_alive": ...}` as extra parameters (`llm.py:506`). `[github]` There is no `response_format`, grammar constraint, or structured-output enforcement. Schema compliance rests entirely on the model's instruction-following ability. + +**Lenient but not magic parser.** The response goes through `_parse_llm_json` (`llm.py:269-303`), which strips optional markdown fences and then tries `json.loads`. If that fails, it scans for the first balanced JSON object in the text. A model that emits its own schema — triples, spans, BIO tags — will produce an empty or nonsense graph, not a graceful fallback. `[github]` + +**Temperature forced to 0 for Ollama.** The Ollama backend config sets `"temperature": 0` (`llm.py:65-73`). `[github]` + +**Deep mode.** `--mode deep` appends `_DEEP_EXTRACTION_SUFFIX` to the same prompt, asking for extra `INFERRED` edges with a more conservative framing (`llm.py:234-247`). `[github]` The schema stays identical. + +--- + +## The shipped default: why a coder model? + +The Ollama fallback default is `qwen2.5-coder:7b` (`llm.py:67`): `[github]` + +```python +"default_model": os.environ.get("OLLAMA_MODEL", "qwen2.5-coder:7b"), +``` + +Even though Graphify sends *documents* (not code) for extraction, the maintainer chose a coder/structured-output-tuned model. This is a revealed preference: the binding requirement is **structured-JSON instruction-following discipline**, not domain entity recognition. A coder/instruct model is the right axis; a NER specialist is the wrong one; frontier reasoning is overkill. + +**Config knobs for the Ollama backend** (all `[github]` from README env-var table unless noted): + +| Knob | How to set | What it does | +|---|---|---| +| `OLLAMA_MODEL` | env var | Override the `qwen2.5-coder:7b` default | +| `OLLAMA_BASE_URL` | env var | Must end in `/v1` (e.g. `http://127.0.0.1:11434/v1`) — any other suffix produces 404s on every call | +| `GRAPHIFY_OLLAMA_NUM_CTX` | env var | **See GOTCHA below** | +| `GRAPHIFY_OLLAMA_KEEP_ALIVE` | env var | Minutes to keep model resident; default `30m`; set `0` to unload after each chunk | +| `--token-budget` | CLI flag | Per-chunk input cap (tokens); pack multiple files per chunk | +| `--max-concurrency` | CLI flag | Set 1–2 for a single local GPU | +| `--mode deep` | CLI flag | Appends deep-inference suffix; elicits more INFERRED edges | + +**GOTCHA — num_ctx does not propagate through graphify (verified 2026-06-04 on this project).** `[github]` Graphify posts to Ollama's OpenAI-compatible `/v1/chat/completions` endpoint. That endpoint **silently ignores** the per-request `options.num_ctx` that Graphify sends (`llm.py:506`). Proven by A/B: a POST to `/v1/chat/completions` with `num_ctx=8192` left `ollama ps` showing CONTEXT=4096; the same value through the native `/api/chat` endpoint was honoured. Therefore `GRAPHIFY_OLLAMA_NUM_CTX` has **no effect** through Graphify — context pins at Ollama's `/v1` default of 4096. At 4096 tokens, Graphify's extraction output JSON is truncated mid-response (`finish_reason=length`) and the chunk is **discarded**, producing an empty graph. + +**Workaround (validated):** bake context into a Modelfile variant: + +```bash +# Create a 16 k-context variant of qwen2.5-coder:7b: +cat <<'EOF' | ollama create qwen25-coder-7b-16k -f - +FROM qwen2.5-coder:7b +PARAMETER num_ctx 16384 +EOF +OLLAMA_MODEL=qwen25-coder-7b-16k graphify extract ./docs --backend ollama +``` + +The `/v1` endpoint honours the model's baked-in default. Non-invasive: no sudo, no systemd restart; reversible with `ollama rm`. Full investigation: `docs/memory-system/benchmark/scoring-results-2026-06-04.md`. + +**Local patch — thinking mode disabled (applied 2026-06-04):** `reasoning_effort: "none"` was added at `llm.py:71` in the installed binary to suppress thinking output for any thinking-capable model. The original is backed up at `/tmp/graphify-bench/llm.py.orig`. This patch is a **no-op for `qwen2.5-coder:7b`** (no thinking mode) but will be **lost on `pip install --upgrade`** and would need re-applying for any future thinking-capable candidate. Needs a production path: upstream PR or a maintained local wrapper. `[github]` + +--- + +## Purpose-built KG / entity / relation models — and why they don't drop in + +Every purpose-built extraction model assumes **it** owns the prompt and output contract. Graphify owns both. That's the structural incompatibility. + +| Model | Architecture | Ollama-pullable? | Follows Graphify's fixed prompt + schema? | Verdict | +|---|---|---|---|---| +| **SciPhi/Triplex** (Phi-3-3.8B, fine-tuned for KG triples) | Autoregressive causal LM | Yes (`ollama pull sciphi/triplex`) | No — locked to its own `{entity_types}/{predicates}` input template; emits subject–predicate–object triples, not Graphify's nodes/edges JSON | Can't drop in as-is | +| **GLiNER** (lightweight NER) | BERT-like bidirectional encoder | No | No — NER-only (span outputs); cannot emit arbitrary JSON or follow a system prompt | Wrong architecture | +| **REBEL / Relik** (relation extraction) | Seq2seq BART with special-token triples | No | No — emits `//` tokens; custom parsing required | Wrong architecture | + +Sources: HuggingFace SciPhi/Triplex model card, `ollama.com/library/sciphi/triplex`, GitHub `urchade/GLiNER`, GitHub `Babelscape/rebel`. `[unverified claim]` — line numbers in those repos were not pinned. + +--- + +## Is the purpose-built route worth experimenting with? + +**User hypothesis:** if the JSON-shape mismatch is the only blocker, write a shim converting Triplex's triple output into Graphify's schema with a custom prompt. Triplex is marketed as very low-cost/fast (small 3.8B Phi-3 model). + +What it would actually take `[speculative]`: + +1. **A model-specific prompt path.** Triplex requires its `{entity_types}/{predicates}` template, not `_EXTRACTION_SYSTEM`. Graphify currently hard-codes one system prompt; a Triplex adapter needs to inject a different one for that model (or backend). +2. **A model-specific output parser.** Convert subject–predicate–object triples into Graphify's `nodes` + `edges` JSON: assign `file_type` to synthesized nodes, map predicates onto Graphify's fixed relation vocabulary (or accept free-text), synthesize `confidence` since Triplex emits no `EXTRACTED`/`INFERRED`/`AMBIGUOUS` enum, and reconstruct per-file provenance fields (`source_file`, `source_location`). +3. **A new backend/model branch in `llm.py`.** This is not a one-line patch — it is a backend adapter (prompt + parser) of moderate size. + +**Architecturally consistent.** Graphify already has model-specific branches: the Kimi/moonshot backend sets `extra_body={"thinking": {"type": "disabled"}}` (`llm.py:461-462`) `[github]` to suppress reasoning output. A Triplex adapter follows the same pattern and could in principle be an upstream PR. + +**Quality unknowns `[speculative]`.** Triplex targets generic NER + triples. Graphify's design wants typed nodes, a fixed relation vocabulary, EXTRACTED/INFERRED/AMBIGUOUS confidence, and per-file provenance. None of those would be native outputs from Triplex — they'd be synthesized, lowering fidelity compared to a model that follows Graphify's schema directly. + +**Cost/speed `[speculative]`.** Triplex is plausibly faster per token (3.8B vs 7B), but `qwen2.5-coder:7b` at Q4 on a 12GB GPU is already rapid. The adapter effort isn't justified unless the default model proves too slow or too heavy for the freshness budget. + +**Verdict `[speculative]`:** Realistic but moderate effort. Worth it **only if**: (a) `qwen2.5-coder:7b` proves too slow or too heavy for the project's freshness budget on the full vault, **and** (b) a quick Triplex spike shows acceptable triple quality on the vault fixtures. Neither condition is known yet. **PARKED — revisit after the qwen2.5-coder:7b benchmark result is in hand.** + +**Cheap first probe:** `ollama run sciphi/triplex` on one fixture note using Triplex's native `{entity_types}/{predicates}` template. If the triples are coherent and well-typed, the adapter is worth scoping. If they are sparse or incorrectly typed, don't bother. + +--- + +## Why qwen2.5-coder:7b is the right axis, not a domain specialist + +To be concrete: the extractors that do better than `qwen2.5-coder:7b` at Graphify's actual task are models that are **better at structured-JSON instruction following**, not models that are better at entity recognition or triple extraction in isolation. Relevant axis: instruction-following + JSON discipline. Irrelevant axis: NER F1, triple-completeness on academic benchmarks. `[speculative]` (Based on observed failure modes: models that failed in the 2026-06-04 benchmark failed by producing invalid JSON or consuming the output budget with thinking tokens — not by extracting the wrong entities.) + +--- + +## Pointers + +- Full local-model investigation + the num_ctx/thinking findings: `docs/memory-system/benchmark/scoring-results-2026-06-04.md` +- Backend config details and Ollama setup: `docs/graphify/05-local-models-and-backends.md` +- Graphify source (installed): `~/.local/lib/python3.14/site-packages/graphify/llm.py` — anchored to 0.8.31 diff --git a/docs/memory-system/05-implementation-process.md b/docs/memory-system/05-implementation-process.md index 524d73f..1ca4581 100644 --- a/docs/memory-system/05-implementation-process.md +++ b/docs/memory-system/05-implementation-process.md @@ -1,6 +1,6 @@ # Implementation Process -_Last updated: 2026-06-04_ | _Status: Step 2c executed — reference set complete; awaiting Ollama scoring_ +_Last updated: 2026-06-05_ | _Status: Steps 2a/2b/2d executed — toolchain installed, context fix applied, vault graph built; model selected (qwen2.5-coder:7b); Step 2c (reference set) previously complete. Step 2 fully executed._ This document distills `04-build-plan.md` into a concrete, staged build process and folds in two locked decisions: **ADR-011** (faceted six-namespace taxonomy) and **ADR-012** (reuse the @@ -8,11 +8,11 @@ existing `~/Documents/SecondBrain` vault rather than creating a new one). Read ` for the underlying rationale, query/CRUD conventions, and plugin internals; read `03-architecture-decisions.md` for the decision log. This document is the runbook. -> This is still a build process outline, not a detailed implementation plan. The recommended -> next move is to feed Step 2 (the critical path) into the **writing-plans skill** to produce an -> executable, task-level plan. Most open questions in Steps 3–6 can be defaulted; Step 2c's -> Claude reference-set run is the genuine gate — it produces the gold-standard rubric against -> which local Ollama models are then scored. Do not hardcode a model before that scoring runs. +> This is still a build process outline, not a detailed implementation plan. **Step 2 is fully +> executed as of 2026-06-05**: toolchain installed (2a), context-baking approach confirmed (2b), +> reference set generated (2c), vault graph built (2d). Selected model: **qwen2.5-coder:7b** +> (Modelfile variant `qwen25-coder-7b-16k`, num_ctx 16384). The next phase is Step 3 (Hooks). +> Most open questions in Steps 3–6 can be defaulted at build time. --- @@ -89,7 +89,7 @@ kind immediately visible. The following items are deferred to after the system is validated end-to-end on fixtures and on the initial validation project (see §Post-build sequence below): -- **Initialize git:** `git init ~/Documents/SecondBrain` (enables ADR-008 sync) +- **Initialize git:** `git init ~/Documents/SecondBrain` (enables ADR-008 sync) — before the first commit, add `graphify-out/` to the vault's `.gitignore`: `~/Documents/SecondBrain/graphify-out/cache/` (AST/tree-sitter parse cache) currently sits inside the vault and must stay untracked (ADR-008 — indexes are disposable/excluded from version control) - **Update vault governance notes:** `CLAUDE.md` (vault path refs, Graphify commands, ADR-011 namespace list), `vault-conventions.md` (six-facet tag list; preserve the "act without being asked" proactive-retrieval section), and project-config hub note(s) @@ -133,6 +133,8 @@ on the benchmark in 2c. Do not skip or defer 2c. ### 2a — Install and verify Graphify +**Status (2026-06-05): EXECUTED.** graphify 0.8.31 installed at `~/.local/bin/graphify`; `graphify --version` → `graphify 0.8.31`. ollama 0.30.3, systemd service on 127.0.0.1:11434; GPU RTX 3060 12GB VRAM confirmed; all extraction ran 100% on GPU (no CPU spill per `ollama ps`). [verified: local shell, 2026-06-05] + The PyPI package name has a double-y — this is the correct install command: ```bash @@ -143,23 +145,40 @@ Verify: `graphify --version` ### 2b — Configure Ollama -Set these in your shell profile (or the plugin env block in Step 6): +**Status (2026-06-05): EXECUTED — but the env-var approach was superseded.** `GRAPHIFY_OLLAMA_NUM_CTX` does NOT propagate through graphify's ollama `/v1/chat/completions` endpoint: ollama's OpenAI-compatible `/v1` endpoint silently ignores per-request `options.num_ctx`, pinning context at the model's baked default (4096 by default). At 4096, graphify's extraction output is truncated mid-response and the chunk is discarded — producing an empty graph. [verified: local A/B test, 2026-06-05] + +**SUPERSEDED APPROACH — do not use `GRAPHIFY_OLLAMA_NUM_CTX`:** + +```bash +# This env var has NO EFFECT through graphify's /v1 path — do not set +# GRAPHIFY_OLLAMA_NUM_CTX=8192 +``` + +**CORRECT APPROACH — bake context into a Modelfile variant:** + +```bash +# Create a Modelfile variant with the desired context baked in: +# FROM +# PARAMETER num_ctx 16384 +# ollama create -16k -f Modelfile +``` + +The `/v1` endpoint DOES honor the model's baked default. Verified `ollama ps` CONTEXT=16384, 100% GPU. This is the same lever the vault build (Step 2d) and plugin (Step 6) must use. Non-invasive: no sudo, no systemd restart, no shell-profile edit; reversible via `ollama rm`. + +Still set in shell profile (these remain valid): ```bash OLLAMA_FLASH_ATTENTION=1 # 30–50% VRAM savings on KV cache — always set -GRAPHIFY_OLLAMA_NUM_CTX=8192 # 8K is sufficient for vault notes (200–2000 words) - # leaves comfortable headroom for prompt overhead GRAPHIFY_OLLAMA_KEEP_ALIVE=5 # set when packaging in Step 6 ``` +Two additional operational requirements confirmed: `OLLAMA_BASE_URL` must end in `/v1` (e.g., `http://127.0.0.1:11434/v1`) or every graphify call 404s; `OLLAMA_API_KEY` must be set to any non-empty value unless the host is loopback. + Verify context allocation after the first extraction call: `ollama ps` shows allocated context. ### 2c — Claude reference-set benchmark (THE GATE) -**Status (2026-06-04): EXECUTED.** 6 cross-domain fixtures × 3 Claude tiers = 18 reference -fragments generated in `benchmark/reference-outputs/`. Run as-is (no vault frontmatter -modification); verified clean. Gate is passed — Ollama model scoring is now unblocked. -Local-model gut-check also done: `gemma4:e4b` is the candidate; GPU fix pending reboot; Graphify owns the ollama call — see `benchmark/local-llm-findings-2026-06-04.md`. +**Status (2026-06-04): EXECUTED (prior round).** 6 cross-domain fixtures × 3 Claude tiers = 18 reference fragments generated in `benchmark/reference-outputs/`. Run as-is (no vault frontmatter modification); verified clean. Gate is passed — Ollama model scoring is unblocked. Full local-model scoring (task 3.4) and model selection (task 3.6) are also complete — see `benchmark/scoring-results-2026-06-04.md`. Produce a reference set of Graphify-shaped extraction outputs before committing to any local Ollama model. Use the 5–10 fixture notes selected in Step 1a as the input set. @@ -216,12 +235,23 @@ outputs are scored by how closely they match the Opus reference for each fixture ### 2d — Build the initial vault graph -After a local Ollama model has been chosen (scored against the Step 2c references): +**Status (2026-06-05): EXECUTED.** Full vault graph built with qwen2.5-coder:7b (Modelfile variant `qwen25-coder-7b-16k`, num_ctx 16384). Result: **57 nodes / 43 edges / 15 communities**. God-nodes are plausible: Speed-to-Lead is the dominant hub, consistent with the vault's current content emphasis. [verified: local shell, 2026-06-05] + +Command used (deviates from the template below — actual values recorded for reference): + +```bash +graphify extract ~/Documents/SecondBrain \ + --backend ollama --model qwen25-coder-7b-16k \ + --max-concurrency 1 --token-budget 4000 \ + --exclude .obsidian --out /tmp/graphify-bench/vault-graph +``` + +Note: `--max-concurrency 1` (not 2 — untested at 2 on 12GB GPU); `--token-budget 4000` (not 512 — fits the 16k context comfortably). Template for future runs: ```bash graphify extract --path ~/Documents/SecondBrain \ - --backend ollama --model \ - --token-budget 512 --max-concurrency 2 + --backend ollama --model qwen25-coder-7b-16k \ + --token-budget 4000 --max-concurrency 1 --exclude .obsidian ``` Review `GRAPH_REPORT.md`. God nodes should be the most-connected tools, clients, and domain @@ -310,8 +340,10 @@ Graphify config. - `memory-write` — when to record evergreen knowledge, frontmatter contract, scope rule, vault not repo. - `memory-query` — `graphify query` vs memsearch; god-node discipline; `--budget`/`--dfs`; cross-client lookups; progressive disclosure via `summary`. - `memory-reorganize` — plan-mode consolidation/promotion procedure; when to trigger `--force` rebuild; human-approval guardrails. -- **Env vars** baked in: `OLLAMA_FLASH_ATTENTION=1`, `GRAPHIFY_OLLAMA_NUM_CTX=8192`, - `GRAPHIFY_OLLAMA_KEEP_ALIVE=5`. +- **Env vars** baked in: `OLLAMA_FLASH_ATTENTION=1`, `GRAPHIFY_OLLAMA_KEEP_ALIVE=5`, + `OLLAMA_BASE_URL=http://127.0.0.1:11434/v1`, `OLLAMA_API_KEY=`. Note: + `GRAPHIFY_OLLAMA_NUM_CTX` is NOT effective through graphify's `/v1` path — context must + be baked into the Modelfile variant instead (see Step 2b). - **Config** (set once at user level): vault path, Graphify output dir, Ollama model name, stale-rebuild threshold in days. @@ -319,9 +351,7 @@ Graphify config. ## Open questions / decisions still to settle -These are deferred to build time. Most can be defaulted without blocking; the only genuine gate -is **§6 (Step 2c benchmark)** — local-model selection is blocked until the reference set exists -and Ollama models can be scored against it. +These are deferred to build time. Most can be defaulted without blocking. **§6 (model selection) is RESOLVED** — qwen2.5-coder:7b selected, vault graph built. Remaining questions (§1–5, §7–8) do not block Step 3 or later. 1. **Vault symlink** — `~/Documents/SecondBrain` is confirmed as the vault (RESOLVED per ADR-012). The open sub-question: symlink it into `~/.claude/memory` or not? Only needed if @@ -341,10 +371,14 @@ and Ollama models can be scored against it. summaries. The human (or AI at note-creation time) must write `summary:` frontmatter. Confirm this holds in practice and add a lint/reminder to the memory-write skill if it drifts. -6. **Step 2c reference set (Claude gold-standard outputs)** — **RESOLVED (2026-06-04):** The - reference set was generated: 18 fragments (6 fixtures × 3 tiers) in `benchmark/reference- - outputs/`. Ollama model selection is now unblocked — score local models against these - references when Ollama is available. Do not hardcode `gemma4:e2b` until that scoring runs. +6. **Step 2c reference set + model selection** — **RESOLVED (2026-06-04/05):** The reference + set was generated: 18 fragments (6 fixtures × 3 tiers) in `benchmark/reference-outputs/`. + Local-model scoring complete (see `benchmark/scoring-results-2026-06-04.md`). **Selected + model: qwen2.5-coder:7b** (run as Modelfile variant `qwen25-coder-7b-16k`, num_ctx 16384) + — graphify's shipped default and the only tested candidate that cleared the + relationship/edge gate. All smaller candidates tested (gemma4:e4b 8.0B, gemma4:e2b 5.1B, + qwen3.5:2b 2.3B) failed to produce relationship-bearing graphs through graphify's extraction + prompt on this hardware. Model selection is complete. 7. **memsearch + journal integration** — does memsearch index SessionEnd journal notes or only its own auto-capture? How does the journal pointer injected at SessionStart reference the @@ -362,8 +396,7 @@ and Ollama models can be scored against it. ## Recommended next move -Turn **Step 2** into a detailed implementation plan via the **writing-plans skill**. It is the -critical path: Graphify install, Ollama configuration, the Step 2c reference-set run, and the -initial graph build are the smallest set of tasks that unblock everything else. Open questions -§1–5 and §7–8 can be defaulted or deferred; only §6 (producing the reference set and then -scoring Ollama models against it) genuinely blocks model-dependent decisions. +**Step 2 is complete.** The next phase is **Step 3 (Hooks)** — thin shell wrappers around +Graphify for PostToolUse, SessionStart, and SessionEnd — followed by Step 4 (memsearch +episodic layer) and Step 6 (package as global plugin). Open questions §1–5 and §7–8 can be +defaulted or deferred at build time. §6 (model selection) is resolved. diff --git a/docs/memory-system/benchmark/scoring-results-2026-06-04.md b/docs/memory-system/benchmark/scoring-results-2026-06-04.md new file mode 100644 index 0000000..c25ae32 --- /dev/null +++ b/docs/memory-system/benchmark/scoring-results-2026-06-04.md @@ -0,0 +1,246 @@ +# Local-Model Extraction Scoring — Results (2026-06-04) + +_Last updated: 2026-06-05_ | _Status: UNBLOCKED — qwen2.5-coder:7b selected; scoring (3.4), speed (3.3), and model selection (3.5/3.6) complete. Remaining: build vault graph (4), update docs (5)._ + +--- + +## Toolchain (Step 2a) + +- graphify 0.8.31 (PyPI `graphifyy`) installed at `~/.local/bin/graphify`; `graphify --version` → `graphify 0.8.31`. [verified: local shell] +- ollama 0.30.3, systemd service `/usr/local/bin/ollama serve`, host 127.0.0.1:11434. [verified: local shell] +- GPU: RTX 3060, 12GB VRAM. All extraction ran 100% on GPU (no CPU spill) per `ollama ps`. [verified: local shell] + +--- + +## Candidate models — true bases (resolves the e4b/e2b naming question) + +Pinned via `ollama show ` info cards. These are genuine next-gen families, NOT gemma3n/qwen2.5 renames: [verified: ollama show output] + +| Local tag | Architecture | Params | Quant | Native ctx | +|---------------|--------------|---------------------------------|--------|------------| +| gemma4:e4b | gemma4 | 8.0B (MatFormer "effective 4B") | Q4_K_M | 131072 | +| gemma4:e2b | gemma4 | 5.1B (effective 2B) | Q4_K_M | 131072 | +| qwen3.5:2b | qwen35 | 2.3B | Q8_0 | 262144 | + +--- + +## Critical config finding — num_ctx does not propagate through graphify (Step 2b) + +- graphify posts to ollama's OpenAI-compatible `/v1/chat/completions` endpoint. That endpoint **silently ignores** the per-request `options.num_ctx` graphify sends. Proven by A/B: `POST /v1/chat/completions` with num_ctx=8192 → `ollama ps` CONTEXT=4096; `POST /api/chat` with num_ctx=8192 → CONTEXT=8192. [verified: local A/B test] +- Therefore `GRAPHIFY_OLLAMA_NUM_CTX` (env var) has **NO EFFECT** through graphify; context pins at ollama's /v1 default 4096. +- At 4096, graphify's extraction output JSON is truncated mid-response (finish_reason=length) and the chunk is **DISCARDED** → empty graph, regardless of (small) input note size. +- **WORKAROUND (validated):** bake context into a Modelfile variant — `FROM ` + `PARAMETER num_ctx 8192`, then `ollama create -8k`. The /v1 endpoint **does** honor the model's baked default. Verified `ollama ps` CONTEXT=8192 (and 16384), 100% GPU. This is the same lever production (vault build) will need. Non-invasive: no sudo, no systemd restart, no shell-profile/dotfiles edit; reversible via `ollama rm`. [verified: local shell] +- Two other operational notes: graphify's `OLLAMA_BASE_URL` must **end in `/v1`** (e.g. `http://127.0.0.1:11434/v1`) or every call 404s; and the CLI requires `OLLAMA_API_KEY` set to any non-empty value unless the host is loopback (127.0.0.1/localhost/::1). + +--- + +## Benchmark methodology + +- Each of the 6 Step-2c fixtures was isolated in its own temp directory (graphify `extract` does NOT accept a single file path — it walks a directory; isolation also avoids conflating notes in one token-budget chunk and excludes `.obsidian/` config noise). [verified: local testing] +- Command shape: `OLLAMA_BASE_URL=http://127.0.0.1:11434/v1 graphify extract --backend ollama --model <8k-variant> --max-concurrency 1 --out `. Models run sequentially (single GPU), one resident at a time. +- Scoring was scoped to dimensions graphify's output and the `.opus` gold references share: entity correctness, relationship plausibility, and edge-level INFERRED/AMBIGUOUS confidence. **Excluded as non-comparable:** the six-facet taxonomy and per-entity type/confidence (graphify's built-in extraction prompt emits neither — per ADR-011 facets are note frontmatter metadata, never Graphify's extraction job), and free-text relationship TYPE strings (graphify uses a fixed relation vocabulary: `references`/`conceptually_related_to`/`semantically_similar_to`/etc., vs the references' free-text verbs). + +--- + +## Results through graphify's real extraction path + +All three tested as 8192-context Modelfile variants. **Scoring blocked** because outputs were degenerate: + +| Model | Result | Failure mode | +|--------------------|-------------------------------------------|--------------| +| gemma4:e4b (8k) | EMPTY graphs (multiple fixtures, deterministic 5/5) | Hollow, invalid JSON (~5039 output tokens of unparseable content); graphify discards chunk. graphify's own warning: "model too small for JSON instruction following — try a larger model." | +| qwen3.5:2b (8k) | EMPTY graph | Output truncated at max_completion_tokens; qwen3.5 "thinking" mode appears to consume the output budget before emitting the JSON graph. | +| gemma4:e2b (8k) | 15 nodes, 0 edges (valid JSON) on the oo-principles fixture | Extracted plausible entities (OO Principles, Single Responsibility Principle, Law of Demeter, Dependency Injection, the four design approaches, etc.) but emitted ZERO relationships → unusable as a knowledge graph. | + +**Caveat on variance:** model Modelfile defaults are temperature 1 (poor for structured extraction); one PRE-fix run of gemma4:e4b on oo-principles yielded 4 nodes/3 edges, indicating high run-to-run variance. + +--- + +## Conclusion + +No specified candidate (gemma4:e4b, gemma4:e2b, qwen3.5:2b) produces a usable entity+relationship graph through Graphify's **default extraction prompt** on this hardware. Per the change proposal's risk clause ("no candidate scores acceptably → surface as finding, don't force selection"), model selection (task 3.6) and the initial vault graph build (task 4) are **DEFERRED** pending a decision. + +**NOTE:** The earlier gut-check (`local-llm-findings-2026-06-04.md`) reported gemma4:e4b doing "6/6 clean parse, 13–34 entities" — but that used a **direct ollama call** with a simpler custom prompt, **NOT** graphify's actual extraction prompt. Through graphify's real path the result is far worse. The graphify-path numbers are the authoritative ones for production. + +--- + +## Untested levers (candidate next steps, not yet attempted) + +1. **Disable model "thinking" mode** — gemma4 and qwen3.5 are thinking-capable; thinking tokens likely burn the output budget, causing the hollow/truncated JSON. Most promising cheap lever, but exposing a think-toggle through graphify's /v1 path is unverified. +2. **`graphify extract --mode deep`** to elicit INFERRED edges (addresses e2b's 0-edge result). +3. **Override sampling temperature to 0** for structured extraction (Modelfile defaults are temp 1). +4. **Pull a stronger model that fits 12GB VRAM with 8k context** — e.g. a 7–8B at Q4 (~5GB, leaves headroom for context); graphify itself suggests a ~14B (~9GB, tighter). This expands the candidate set beyond the three the change specified (a download + scope decision). + +--- + +## Raw artifacts (ephemeral, /tmp — will not survive reboot) + +- e4b first benchmark (pre-fix, 4096): `/tmp/graphify-bench/out/e4b/` +- 8k-variant runs: `/tmp/graphify-bench/out2/`, `/tmp/graphify-bench/out3/` +- Created reversible ollama variants: `gemma4-e4b-8k`, `gemma4-e4b-16k`, `gemma4-e2b-8k`, `qwen35-2b-8k` + +--- + +## Round 2 (2026-06-05): thinking-disable patch + edge gate + +### The thinking-mode investigation (user-directed) +- GitHub issue safishamsi/graphify#792 turned out to be about CPU scaling, local API timeouts, and the OLLAMA_API_KEY auth gate — NOT thinking/reasoning. (Documents the same auth friction we hit.) +- Ollama's `/v1` (OpenAI-compat) endpoint, which Graphify uses, IGNORES `think:false` and `chat_template_kwargs:{enable_thinking:false}`, but DOES honor top-level `reasoning_effort:"none"`. The native `/api/chat` honors `think:false`. (Same /v1-drops-nonstandard-options pattern as num_ctx.) +- Graphify sends NO thinking control for the ollama backend by default; there is no env var / CLI flag / config file to inject it (confirmed by source trace of llm.py). `PARAMETER think false` in a Modelfile is rejected by `ollama create`; a Modelfile `SYSTEM /no_think` does not survive because Graphify sends its own system message and ollama's /v1 REPLACES (not merges) the system message. +- WORKING FIX (applied to the installed package): one-line patch adding `"reasoning_effort": "none"` to the ollama backend config dict in `~/.local/lib/python3.14/site-packages/graphify/llm.py` (~line 71). Graphify already applies `reasoning_effort` as a top-level kwarg if present, and it survives the extra_body overwrite. The author already uses the equivalent pattern (`extra_body={"thinking":{"type":"disabled"}}`) for the Kimi/moonshot backend — just not for ollama. PRODUCTION IMPLICATION: this is a local patch lost on `pip install --upgrade`; production needs an upstream PR (likely welcome) or a maintained patch/wrapper. +- Verified effect: with the patch, every model's raw output begins directly with `{"nodes":[...` — no reasoning preamble. Thinking is genuinely suppressed. + +### The edge gate — the criterion that actually matters +A knowledge graph needs RELATIONSHIPS (edges), not just entities (nodes). Across ~10+ runs this whole investigation, LLM-extracted relationships appeared essentially once (e4b pre-patch: 3 edges, since non-reproducible). All other non-empty results were nodes-only. So the gate for selection is: does a config emit a plausible EDGE-bearing graph on one fixture? + +Bounded test on the `oo-principles-plugin-concept-design-recommendations` fixture (a note that clearly contains relationships), thinking-disable confound checked by running both ON and OFF: + +| Config | Exit | Nodes | Edges | Outcome | +|---|---|---|---|---| +| qwen3.5:2b @16k, thinking-OFF | 1 | — | empty | output truncated mid-JSON → invalid → discarded | +| qwen3.5:2b @16k, thinking-ON | 1 | — | empty | same truncation; 16k did not rescue it | +| gemma4:e2b @16k, thinking-ON | 0 | 17 | 0 | clean valid JSON, coherent on-topic nodes, ZERO edges | +| gemma4:e2b @16k, thinking-ON, --mode deep | 0 | 10 | 0 | deep mode (explicitly requests INFERRED edges) made it WORSE: fewer nodes, still 0 edges | + +`hyperedges` was also empty `[]` in the clean runs — no relationships hid in the alternate key. + +### Confound resolved: thinking-off is NOT strictly better +gemma4:e2b @8k: thinking-ON gave 15 nodes / 3307 output tokens (valid); thinking-OFF gave 1 node / 395 tokens. For e2b the thinking tokens were doing extraction work; the binding constraint is output room, not thinking. So the user's "thinking wastes the budget" premise holds for qwen (truncation) but inverts for e2b. Thinking-disable is a useful lever, not a universal win. + +### Two distinct failure modes +- qwen3.5:2b (2.3B, Q8_0): capacity/truncation — runs out of output room emitting node JSON before reaching edges. 16k context insufficient; thinking on/off irrelevant. +- gemma4:e2b (5.1B, Q4_K_M): clean completion, extracts plausible NODES but emits zero RELATIONSHIPS even when deep mode explicitly asks for them. The model understands the doc (node labels: OO Principles, Single Responsibility Principle, Law of Demeter, Dependency Injection, Replace Conditional with Polymorphism, the 4 approaches, the 3 layers) — it just won't populate `links`. +- gemma4:e4b (8.0B, Q4_K_M): hollow/malformed JSON (dropped — not a context problem). + +### Round-2 conclusion +The cheap-lever space is exhausted (thinking ON/OFF, 8k/16k context, --mode deep). No local 2–5B candidate produces a relationship-bearing graph through Graphify's extraction prompt on a 12GB GPU. Relationship extraction (not entity extraction) is the wall. Recommended path: a stronger model that fits 12GB with ~16k context — e.g. qwen2.5-coder:7b (~5GB Q4, non-thinking, strong at structured JSON+edges). This expands the candidate set beyond the three the change specified (a multi-GB download + scope decision) — pending user go-ahead. + +### Round-2 ollama variants created (reversible via `ollama rm`) +gemma4-e4b-8k, gemma4-e4b-16k, gemma4-e2b-8k, gemma4-e2b-16k, qwen35-2b-8k, qwen35-2b-16k. Installed graphify llm.py currently carries the thinking-OFF patch (backup of original at /tmp/graphify-bench/llm.py.orig). + +--- + +## Round 3 (2026-06-05): qwen2.5-coder:7b clears the edge gate — full benchmark + +### Config that works +- Model: `qwen2.5-coder:7b` (graphify's hardcoded ollama default, llm.py:67), run as a Modelfile variant `qwen25-coder-7b-16k` (FROM qwen2.5-coder:7b + PARAMETER num_ctx 16384). +- Command: `OLLAMA_BASE_URL=http://127.0.0.1:11434/v1 graphify extract --backend ollama --model qwen25-coder-7b-16k --max-concurrency 1 --out `. The thinking-OFF llm.py patch is in place (no-op for qwen2.5-coder, which has no thinking mode). +- VRAM: 5.6GB resident, 100% GPU, 16384 context, no CPU spill on the 12GB RTX 3060. +- Speed (task 3.3): aggregate throughput ~59.4 tok/s (substantive runs cluster 57–61 tok/s). Total wall-clock all 6 = 253.7s (4.2 min); avg 42.3s/fixture (49.9s excluding the degenerate 10dlc run). Use tok/s not wall-clock (wall-clock tracks output length). + +### Per-fixture results (edge gate PASSED: 5/6 produce relationships) +| Fixture | Wall | Nodes | Edges | Out tok | Notes | +|---|---|---|---|---|---| +| oo-principles-plugin-concept | 44.9s | 11 | 10 | 2553 | gate fixture; cold-load incl. | +| 10dlc-isv-setup-guide-oncadence | 4.4s | 1 | 0 | 159 | under-extraction on smallest note (502 words), exit 0 valid JSON — not a crash | +| pest-control-enterprise-revenue | 64.2s | 23 | 17 | 3932 | only fixture with INFERRED edges (7) | +| ai-coding-conventions-synthesis | 37.9s | 10 | 9 | 2284 | | +| pest-control-after-hours-sms | 76.8s | 21 | 20 | 4641 | | +| pest-control-email-a-b-c-hub | 25.5s | 7 | 6 | 1489 | | + +All exit 0, no truncation/hollow/invalid-JSON warnings. Confidence tags are provenance method tags: EXTRACTED (from text) vs INFERRED (model-inferred); INFERRED appeared only in enterprise-revenue. + +### Full edge lists (for scoring vs .opus references next session) +**oo-principles (10, all EXTRACTED):** Process Layer→shares_data_with→Mechanic Layer; Process Layer→shares_data_with→Theory Layer; Mechanic Layer→shares_data_with→Cards Layer; Annotated Process Graph Layer→shares_data_with→Cards Layer; Bundles Layer→shares_data_with→Cards Layer; CLAUDE.md→shares_data_with→Cards Layer; NotebookLM Notebook→shares_data_with→Cards Layer; Phase Bundled Context Packs Layer→shares_data_with→Cards Layer; Refactoring Layer→shares_data_with→Cards Layer; Situational Trigger Files Layer→shares_data_with→Cards Layer. +**10dlc:** none. +**enterprise-revenue (17):** Marketing Allocation Model→references→Unit Economics; ACV Tiers→references→Unit Economics; M&A Context→references→ACV Tiers; Communication Failure Patterns→references→Unit Economics; Dropped calls and ghosting→conceptually_related_to→Communication Failure Patterns [INFERRED]; No one answered→conceptually_related_to→Communication Failure Patterns [INFERRED]; Offshore agent disconnect→conceptually_related_to→Communication Failure Patterns [INFERRED]; Voicemail full→conceptually_related_to→Communication Failure Patterns [INFERRED]; M&A Context→references→Communication Failure Patterns; Climatologically-Driven Lead Seasonality→references→Unit Economics; Fall First Freeze Rodent Push→conceptually_related_to→Climatologically-Driven Lead Seasonality [INFERRED]; Spring Termite Swarming→conceptually_related_to→Climatologically-Driven Lead Seasonality [INFERRED]; Summer Heatwave-Driven Migrations→conceptually_related_to→Climatologically-Driven Lead Seasonality [INFERRED]; M&A Context→references→Climatologically-Driven Lead Seasonality; M&A Context→references→Operational Unit Economics; Operational Unit Economics→references→COGS Structure; M&A Context→references→Unit Economics. +**ai-coding-conventions (9, all EXTRACTED):** Core Finding→references→{Context Linking Patterns, Enforcement Mechanisms, Key Tradeoffs, Notable Public References, Organizational Patterns, Process vs. Reference Design, See Also, The Three-Tier Architecture (arXiv 2602.20478), Token Efficiency Strategies}. +**after-hours-sms (20, all EXTRACTED):** document node→cites→ each of 20 study/source nodes (MIT/InsideSales 2007, HBR 2011, Velocify 2012, InsideSales 2021, HomeAdvisor/Angi 2024, Thumbtack 2024, Coalmarch 2025, Invoca 2024, Driven Results 2025, HubSpot 2023, Gartner 2016, D7 Networks/CTIA 2024, TransUnion 2024, Avochato 2019, Briostack 2024, ServiceTitan 2025, Point Loma Hatch, Shafer HVAC Hatch, Aruza Cube, ServiceTitan Data 2025). +**email-a-b-c-hub (6, all EXTRACTED):** Experiment Hub→references→{Blind A/B/C Methodology Guide, Per Group Copy and Scores, What Owners Respond to/Reject, V4 Copywriter Draft, V4 Golden Framework Draft, V4 Golden Framework Revised}. + +### Round-3 takeaway +qwen2.5-coder:7b is the de-facto selected model pending the formal scoring pass (3.4). It's graphify's own default, fits 12GB with 16k context at ~59 tok/s, and produces typed, mostly-EXTRACTED edges. The earlier gut-check's gemma4:e4b pick was an artifact of using a simpler direct prompt, not graphify's real path. Source graph.json (ephemeral): /tmp/graphify-bench/out6/coder7b//graphify-out/graph.json. + +--- + +## 3.4 Scoring — qwen2.5-coder:7b vs Opus Gold Standard + +**Sources used:** +- Candidate output: `scoring-results-2026-06-04.md`, Round 3 section (per-fixture node/edge lists) +- Gold-standard references: `reference-outputs/*.opus.md` (all 6 `.opus.md` files) +- Fixture mapping (from Round 3 table): `oo-principles-plugin-concept`, `10dlc-isv-setup-guide-oncadence`, `pest-control-enterprise-revenue`, `ai-coding-conventions-synthesis`, `pest-control-after-hours-sms`, `pest-control-email-a-b-c-hub` + +### Dimensions Excluded (N/A) + +**Facets:** The `.opus.md` references include facet annotations (e.g., `facet: tool/twilio`, `facet: convention/oo-principles`). These are note-level metadata per ADR-011 — they are not part of graphify's extraction output at all, so there is nothing to compare. N/A. + +**Entity-type labels:** The opus references annotate entities with types (`type: layer`, `type: principle`, `type: Tool`, etc.). Graphify's extraction output produces a node list with names only — no type labels are emitted. N/A. + +**Free-text relationship typing:** The opus references use descriptive, domain-specific verbs (`comprises_share_of`, `is_event_of`, `measured conversion decay for`, `tested for campaign`, etc.). Graphify uses a fixed relation vocabulary (`references`, `conceptually_related_to`, `semantically_similar_to`, `shares_data_with`, `cites`). These are structurally incommensurable — comparing relation-type strings would score graphify's vocabulary design, not extraction quality. N/A. + +### Rubric + +**Entity Correctness (a):** 1–5. How many of the entities graphify extracted correspond to real entities in the reference? Penalizes under-extraction (missing key nodes) and over-compression (collapsing distinct entities). Does not penalize naming style differences. + +**Relationship Plausibility (b):** 1–5. Are the edges graphify drew present or derivable from the reference's relationship set? Penalizes phantom edges (no basis in reference), missing major structural relationships, and systematically wrong pairing patterns. Does not penalize relation-verb style. + +**Confidence Accuracy (c):** 1–5. Where graphify tagged EXTRACTED vs INFERRED vs AMBIGUOUS, does that align with what the reference treated as explicit (no `confidence:` tag) vs inferred (`confidence: INFERRED`) vs ambiguous (`confidence: AMBIGUOUS`)? N/A on fixtures where graphify emitted no edges at all, or no INFERRED/AMBIGUOUS tags appeared in either output. + +### Per-Fixture Scoring Table + +| Fixture | (a) Entity Correctness | (b) Relationship Plausibility | (c) Confidence Accuracy | Notes | +|---|---|---|---|---| +| oo-principles-plugin-concept | 2 | 1 | N/A† | | +| 10dlc-isv-setup-guide | 1 | N/A (0 edges) | N/A | | +| pest-control-enterprise-revenue | 3 | 3 | 3 | | +| ai-coding-conventions-synthesis | 2 | 2 | N/A† | | +| pest-control-after-hours-sms | 3 | 2 | N/A† | | +| pest-control-email-a-b-c-hub | 2 | 2 | N/A† | | + +†Graphify emitted no INFERRED/AMBIGUOUS tags on this fixture (all EXTRACTED), so there is no confidence signal to compare against the reference's INFERRED annotations. N/A does not mean graphify was wrong; it means the dimension is unobservable. + +### Per-Fixture Justifications + +**oo-principles-plugin-concept** +- (a) **2/5.** Captured the layer decomposition (Process, Mechanic, Theory + 4 named layers, CLAUDE.md, NotebookLM Notebook = 11 nodes) but missed the plugin as a named entity, all OO principle entities (TDD, SRP, LoD, DI, Shameless Green, Flocking Rules), the book/paper sources, and the 4-phase development lifecycle. ~30% of the ~30-entity reference set. +- (b) **1/5.** All 10 edges are `shares_data_with` → "Cards Layer". The reference relationships are structural/directional (includes, encodes, references, retains, alternative to, evolves from, modeled on). Homogenization collapses the architecture into a flat data-sharing star with no analog in the reference. +- (c) **N/A.** All graphify edges EXTRACTED; reference has 3 INFERRED. No INFERRED signal to compare. + +**10dlc-isv-setup-guide** +- (a) **1/5.** Extracted 1 node (whole document as a single entity) vs 20 distinct reference entities (OnCadence, 10DLC, Twilio, TrustHub API, ISV, CSP, EIN, IRS, T-Mobile, subaccount-per-client architecture, etc.). Near-total extraction failure on a 502-word note. +- (b) **N/A.** 0 edges produced vs 17 reference relationships. +- (c) **N/A.** No edges. + +**pest-control-enterprise-revenue** +- (a) **3/5.** 23 nodes capturing the major conceptual clusters (Marketing Allocation Model, ACV Tiers, M&A Context, Unit Economics, Communication Failure Patterns + 4 sub-patterns, Seasonality + 3 seasonal events, COGS Structure) but at shallower granularity than the ~55-entity reference (misses CAC/LTV/CPL/Churn, COGS components, regions, Big Four integrators, GDD, multiples). +- (b) **3/5.** 17 edges incl. 7 INFERRED. The INFERRED membership cluster (comm sub-patterns → Communication Failure Patterns; seasonal events → Seasonality) maps closely onto the reference's `is_event_of`/`component_of` structure. ~50% of reference edges missed (COGS breakdown, Big Four, causal chains). +- (c) **3/5.** The 7 INFERRED tags are defensible group-membership edges; not systematically miscalibrated, though it under-fires on EXTRACTED edges the reference treats as explicit, and never modeled the reference's `EIN → issued_by → IRS` background-knowledge INFERRED edge. + +**ai-coding-conventions-synthesis** +- (a) **2/5.** Extracted the document's section-heading outline (10 nodes: Context Linking Patterns, Enforcement Mechanisms, Three-Tier Architecture, etc.) rather than the ~40 domain entities (Claude, Cursor, Copilot, Constitution, Domain Specialist Agents, .cursorrules, AGENTS.md, MCP, named projects). Only Three-Tier Architecture overlaps in substance. +- (b) **2/5.** All 9 edges are `Core Finding → references → {section heading}` — a hub-and-spoke from a synthetic aggregate node. Zero structural resolution vs the reference's compositional/directional relationships. 2 not 1 because no phantom entities introduced. +- (c) **N/A.** All EXTRACTED; reference's 2 INFERRED edges not modeled. + +**pest-control-after-hours-sms** +- (a) **3/5.** 21 nodes (document + 20 study/source nodes) covering ~half the reference's company/study entities, but missing the product/project/campaign entities (After-Hours SMS product, niche-automation-prospecting initiative, pest-control-spring-2026 campaign) and several companies (Google Ads/LSA, NPMA, CallRail, PATLive, etc.). +- (b) **2/5.** All 20 edges are `document → cites → {study}` — a citation list. Defensible but reductive; the reference's product–customer, product–channel, and study–finding semantic edges are entirely missing. +- (c) **N/A.** All EXTRACTED; reference has 2 INFERRED + 1 AMBIGUOUS, none modeled. + +**pest-control-email-a-b-c-hub** +- (a) **2/5.** Extracted the hub's table-of-contents (7 nodes: Experiment Hub + 6 linked sub-documents) but missed the experiment's analytic entities (3 personas, 9 email groups, 5 hooks, structural/retired patterns, 3 versions) — the primary conceptual content. ~45-entity reference. +- (b) **2/5.** All 6 edges are `Experiment Hub → references → {sub-document}` — the index, not the findings. Version outcomes, recommended groups, retired patterns all absent. +- (c) **N/A.** All EXTRACTED; reference's 2 INFERRED edges not modeled. + +### Overall Summary + +**Entity extraction: weak to moderate (avg ~2.2/5).** Consistently fewer/coarser entities than the opus reference. On hub-structured notes it extracts the structural outline (section headings, linked sub-docs) rather than conceptual entities; on dense domain notes it captures ~40–55% of the reference entity set; on the smallest note (10dlc) it nearly fails (1 node vs 20). + +**Relationship extraction: poor to moderate (avg ~1.9/5, excluding the N/A fixture).** Dominant failure mode is **homogenization**: nearly all relationship semantics collapse into one or two relation types (`references`, `cites`, `shares_data_with`) in a hub-and-spoke topology. enterprise-revenue is the only fixture with meaningfully differentiated edges — and the only one where INFERRED edges appeared — suggesting extraction quality correlates with note density. + +**Confidence calibration: partially defensible where observable (3/5 on the one applicable fixture).** INFERRED tags only appeared on enterprise-revenue (7/17 edges) and were defensible group-membership edges. On the other five fixtures all edges were EXTRACTED, precluding comparison. + +**Net assessment:** qwen2.5-coder:7b passes the edge gate (5/6 fixtures produce relationships) and extracts credible entities on content-dense notes, confirming it as a viable baseline over the tested alternatives. Its relationship extraction is systemically shallow: the relation flattening means the graph encodes "these things are related" but not "how," limiting downstream query quality. For a production vault graph the primary gap is relationship semantic resolution, not entity recall. + +--- + +## 3.6 Model Selection + +**Selected model: qwen2.5-coder:7b** (run as the Modelfile variant `qwen25-coder-7b-16k`, num_ctx 16384). + +The selection is not a choice among viable candidates — it is the only candidate that cleared the relationship/edge gate. Every smaller candidate tested (gemma4:e4b 8.0B, gemma4:e2b 5.1B, qwen3.5:2b 2.3B) failed to produce relationship-bearing graphs through graphify's prompt on this hardware. The scoring in section 3.4 therefore characterizes qwen2.5-coder:7b's extraction quality against the opus gold standard rather than differentiating among alternatives; there was no viable alternative to differentiate. + +This model is also graphify's own hardcoded ollama default (llm.py:67), which is a strong signal that it is the intended extraction model for this backend. It fits the 12GB RTX 3060 comfortably at 5.6GB VRAM resident with 16384 context and runs at ~59 tok/s, completing a full 6-fixture pass in ~4 minutes. + +**Known limitations (carried forward honestly):** shallow relationship semantics — graphify's fixed relation vocabulary combined with qwen2.5-coder:7b's extraction behavior produces hub-and-spoke topologies dominated by `references`, `cites`, and `shares_data_with`, collapsing domain-specific structural relationships. Entity recall is weak on hub-structured notes (extracts the structural outline instead of conceptual entities) and near-failing on very short notes (10dlc: 1 node vs 20 in reference). These limitations are accepted for the initial build. Relationship semantic resolution is the known primary gap for future improvement. + +**Speed:** ~59 tok/s on the 12GB GPU at max-concurrency 1. diff --git a/openspec/changes/graphify-ollama-setup/.openspec.yaml b/openspec/changes/graphify-ollama-setup/.openspec.yaml new file mode 100644 index 0000000..f617bd1 --- /dev/null +++ b/openspec/changes/graphify-ollama-setup/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-06-04 diff --git a/openspec/changes/graphify-ollama-setup/HANDOFF.md b/openspec/changes/graphify-ollama-setup/HANDOFF.md new file mode 100644 index 0000000..48d8457 --- /dev/null +++ b/openspec/changes/graphify-ollama-setup/HANDOFF.md @@ -0,0 +1,33 @@ +# Handoff — graphify-ollama-setup (2026-06-05) + +## Where we are +Mid-application of the `graphify-ollama-setup` OpenSpec change. Toolchain installed; the hard part (finding a local model that produces relationship-bearing graphs through graphify) is SOLVED. Remaining: formal scoring, selection record, vault build, doc updates, verify. + +## The working config (use this) +- Model: **qwen2.5-coder:7b** (graphify's shipped default). Run via Modelfile variant **`qwen25-coder-7b-16k`** (already created: `FROM qwen2.5-coder:7b` + `PARAMETER num_ctx 16384`). +- Always: `OLLAMA_BASE_URL=http://127.0.0.1:11434/v1` (MUST end in `/v1`), `--backend ollama --max-concurrency 1` (single 12GB GPU, run models sequentially). +- A one-line thinking-OFF patch (`reasoning_effort:"none"`) is applied to installed `~/.local/lib/python3.14/site-packages/graphify/llm.py` (backup: /tmp/graphify-bench/llm.py.orig; patched copy: /tmp/graphify-bench/llm.py.patched). No-op for qwen2.5-coder but harmless; note it for production (lost on `pip install --upgrade`). + +## Gotchas (all the lessons, so you don't relearn them) +1. `GRAPHIFY_OLLAMA_NUM_CTX` env var does NOT propagate through graphify's ollama `/v1` endpoint — bake `num_ctx` into a Modelfile variant instead. Verify with `ollama ps` (CONTEXT column). +2. `graphify extract ` needs a DIRECTORY, not a single file. Each benchmark fixture was isolated in its own dir. +3. For the vault build, add `--exclude .obsidian` (graphify treats `.obsidian/*.json` as code and indexes it as noise). +4. `extract` produces `graphify-out/graph.json` + `.graphify_analysis.json` (god-nodes here). It does NOT produce `GRAPH_REPORT.md` — that's a separate step; read the analysis JSON for the god-node sanity check (task 4.2). +5. OLLAMA_API_KEY only needed if base_url is non-loopback; 127.0.0.1 avoids it. +6. Small models (gemma4:e4b 8B, gemma4:e2b 5.1B, qwen3.5:2b 2.3B) all FAIL the relationship/edge gate through graphify's prompt — don't revisit them. + +## Remaining tasks (in order) +- **3.4 — Score** qwen2.5-coder:7b outputs vs the `.opus` references (`docs/memory-system/benchmark/reference-outputs/*.opus.md`) on the 3 COMPARABLE dimensions only: entity correctness, relationship plausibility, edge-level EXTRACTED/INFERRED/AMBIGUOUS confidence. EXCLUDE facets + entity-type + free-text relation-typing (graphify doesn't emit them; facets are note metadata per ADR-011). The qwen2.5-coder per-fixture node/edge lists are recorded verbatim in `docs/memory-system/benchmark/scoring-results-2026-06-04.md` (Round 3) so you can score WITHOUT re-running. Raw graph.json (if /tmp survived): `/tmp/graphify-bench/out6/coder7b//graphify-out/graph.json`. +- **3.5 / 3.6 — Record + select:** add scoring scores + selection rationale to the findings doc; qwen2.5-coder:7b is the de-facto selection. Per the change, don't lock before 3.4 is done. +- **4 — Build vault graph:** `OLLAMA_BASE_URL=http://127.0.0.1:11434/v1 graphify extract ~/Documents/SecondBrain --backend ollama --model qwen25-coder-7b-16k --max-concurrency 2 --token-budget 4000 --exclude .obsidian --out `. (token-budget BELOW context so multi-file chunks fit 16k.) Then 4.2 god-node sanity check via `.graphify_analysis.json`; 4.3 confirm graph artifacts are gitignored/not vault-synced (ADR-008). +- **5.1 — Update** `docs/memory-system/05-implementation-process.md` status + Step 2 markers (2a/2b/2d executed; selected model = qwen2.5-coder:7b; the e4b/e2b inconsistency is resolved — see findings). NOTE: that doc currently claims 2a/2b "DONE" but they were only truly executed this round; reconcile honestly. +- **5.2 — Verify** with `/opsx:verify` then archive. + +## Re-run extraction if /tmp was wiped (reboot) +Fixtures live in the vault; recreate isolated dirs by copying the 6 source notes (slugs in scoring-results-2026-06-04.md Round-3 table) into per-fixture temp dirs, then run the working command above per fixture. + +## Pointers +- Findings + raw data: `docs/memory-system/benchmark/scoring-results-2026-06-04.md` +- Model-choice reference (incl. Triplex/purpose-built assessment): `docs/graphify/10-extraction-model-options.md` +- Task list + status block: `openspec/changes/graphify-ollama-setup/tasks.md` +- Optional cleanup: extra ollama variants from the investigation (gemma4-e4b-8k/16k, gemma4-e2b-8k/16k, qwen35-2b-8k/16k) can be removed with `ollama rm`; KEEP `qwen25-coder-7b-16k`. diff --git a/openspec/changes/graphify-ollama-setup/design.md b/openspec/changes/graphify-ollama-setup/design.md new file mode 100644 index 0000000..fe0140d --- /dev/null +++ b/openspec/changes/graphify-ollama-setup/design.md @@ -0,0 +1,55 @@ +## Context + +This change implements Steps 2a–2d of `docs/memory-system/05-implementation-process.md` — the critical path of the memory system. Step 2c (the Claude reference-set gate) is already executed: 18 gold-standard fragments (6 fixtures × 3 tiers) exist in `docs/memory-system/benchmark/reference-outputs/`, with the `claude-opus-4-8` outputs as the scoring rubric (specced under `reference-extraction-benchmark`). What remains is to stand up the local extraction toolchain, score local Ollama models against that rubric, pick the model, and build the first vault graph. + +Constraints: +- The vault is the existing `~/Documents/SecondBrain` (ADR-012); the build runs against it as-is, no bulk migration (ADR-013, `incremental-migration` spec). +- Markdown is the single source of truth; graph artifacts are disposable (ADR-008). +- A feasibility gut-check found `gemma4:e4b` runs at ~74 tok/s on the local GPU, but quality against the references has not been measured. The implementation-process doc is internally inconsistent on the model (`e4b` vs `e2b`) — a tell that nothing is locked. + +## Goals / Non-Goals + +**Goals:** +- Install and verify Graphify; configure Ollama for extraction. +- Score candidate Ollama models against the existing Opus rubric on the 6 fixtures (quality + speed) and select the extraction model by evidence. +- Build the initial vault graph with the selected model and sanity-check god-nodes. + +**Non-Goals:** +- Regenerating or modifying the Step 2c reference set (consumed, not produced here). +- Per-project code graphs / Step 2e (separate, free tree-sitter path; out of range). +- Hooks, memsearch, sync, plugin packaging (Steps 3–6). +- Bulk vault migration (deferred to last per ADR-013). +- Choosing the sync mechanism or stale-rebuild threshold (Open questions §2–3). + +## Decisions + +**Selection by scoring, not by gut-check.** The model is chosen by comparing each candidate's Graphify-shaped output to the Opus reference per fixture (entity correctness, relationship typing, confidence-tag accuracy) plus measured wall-clock speed. Alternative considered: adopt `gemma4:e4b` directly since feasibility passed — rejected because the gut-check validated speed, not extraction quality, and Open-question §6 explicitly says do not hardcode. `gemma4:e4b` enters as the front-runner candidate, nothing more. + +**Score against the Opus tier as the rubric.** Haiku/Sonnet references exist but Opus is the gold standard (per `reference-extraction-benchmark`). Candidates are scored primarily against Opus; the other tiers provide a quality gradient for context. + +**Ollama config travels with this step.** `OLLAMA_FLASH_ATTENTION=1` (KV-cache VRAM savings) and `GRAPHIFY_OLLAMA_NUM_CTX=8192` (sufficient for 200–2000-word notes with prompt headroom) are set in the shell profile now and re-baked into the plugin env block at Step 6. `GRAPHIFY_OLLAMA_KEEP_ALIVE` is deferred to packaging. Verify with `ollama ps` after the first call. + +**Build the full vault, then review god-nodes.** Rather than a synthetic subset, build over the real `~/Documents/SecondBrain` and use `GRAPH_REPORT.md`'s most-connected nodes as the sanity signal — the highest-traffic tools/clients/domains should surface as god-nodes. Cheap, and it exercises the real extraction path. + +## Risks / Trade-offs + +- **No candidate scores acceptably against the rubric** → If even the best candidate is well below the Opus reference, surface it as a finding rather than forcing a selection; the front-runner's speed does not rescue poor extraction quality. Selection may need a larger candidate or a revisit of token budget / context. +- **Scoring is partly qualitative** → Entity/relationship/confidence-tag comparison against references is judgement-based, not a single numeric pass/fail. Mitigation: record the per-fixture comparison and rationale in the result artifact so the choice is auditable, not asserted. +- **Vault content drift during build** → Building over the live vault means notes could change mid-run. Mitigation: the graph is disposable and rebuildable (`--force`); a one-shot initial build is acceptable. +- **GPU/VRAM regressions** → The ~74 tok/s figure was post-reboot; throughput can vary. Mitigation: `--max-concurrency 2` and a bounded `--token-budget` keep memory pressure predictable; record actual speed per candidate. + +## Migration Plan + +Sequential, low blast radius (local-only, no production system touched): +1. `pip install graphifyy`; verify `graphify --version`. +2. Export Ollama env vars; pull candidate model(s); verify context via `ollama ps`. +3. Run each candidate over the 6 fixtures; score vs. the Opus references; record results. +4. Select the model; build the initial vault graph; review `GRAPH_REPORT.md`. + +Rollback: artifacts are disposable — delete `graphify-out/` and re-run; uninstall `graphifyy` if needed. No data is mutated in the vault (extraction is read-only over notes). + +## Open Questions + +- Exact candidate set beyond `gemma4:e4b` (e.g. whether to also score a smaller/larger sibling) — decided at run time based on what is pulled and how the front-runner scores. +- Where the scoring-result artifact lives (under `docs/memory-system/benchmark/` alongside the references is the natural home) — settle when writing it. +- `--token-budget` / `--max-concurrency` tuning — start from the doc's `512` / `2` and adjust if quality or VRAM demands it. diff --git a/openspec/changes/graphify-ollama-setup/proposal.md b/openspec/changes/graphify-ollama-setup/proposal.md new file mode 100644 index 0000000..e79f97f --- /dev/null +++ b/openspec/changes/graphify-ollama-setup/proposal.md @@ -0,0 +1,32 @@ +## Why + +Step 2c produced the gold-standard reference set (18 Claude fragments across 6 fixtures), but no local extraction model has been scored against it yet — so the critical-path question "which Ollama model drives vault extraction?" is still open and blocks the initial vault graph build (Step 2d) and everything downstream (hooks, plugin packaging). This change installs and configures the extraction toolchain, scores candidate Ollama models against the existing references, and builds the first real vault graph. + +## What Changes + +- Install and verify Graphify (`graphifyy` package, `graphify` command) — Step 2a. +- Configure Ollama for extraction: flash attention, 8K context, keep-alive — Step 2b. +- Score candidate local Ollama models against the Step 2c gold-standard references on the same 6 fixtures, measuring both extraction quality (vs. the Opus rubric) and wall-clock speed, then select the extraction model. `gemma4:e4b` is the front-runner candidate (feasibility-validated at ~74 tok/s) but is **not** pre-selected — selection is decided by the scoring run. +- Build the initial vault graph with the selected model and review `GRAPH_REPORT.md` god-nodes for sanity — Step 2d. +- Step 2c (reference set) is a **completed prerequisite**, not work in this change; per-project code graphs (Step 2e) are **out of scope**. + +## Capabilities + +### New Capabilities + +- `local-model-selection`: Score candidate Ollama extraction models against the Step 2c gold-standard reference set and select the model by evidence (quality vs. the Opus rubric + measured speed), rather than hardcoding one. Owns the Ollama runtime configuration (flash attention, context size) that the scoring run depends on. +- `vault-graph-build`: Build the initial `~/Documents/SecondBrain` vault knowledge graph with the selected extraction model and verify graph sanity by reviewing god-nodes against the vault's actual content. + +### Modified Capabilities + + + +## Impact + +- **New dependencies:** Graphify (`graphifyy` on PyPI, anchored to v0.8.30), a running Ollama with at least one pulled candidate model. +- **Environment:** Ollama env vars (`OLLAMA_FLASH_ATTENTION`, `GRAPHIFY_OLLAMA_NUM_CTX`, `GRAPHIFY_OLLAMA_KEEP_ALIVE`) set in the shell profile now; rebaked into the plugin env block at Step 6. +- **Artifacts produced:** a model-scoring result recording the chosen model and its rationale; the initial vault `graphify-out/graph.json` + `GRAPH_REPORT.md` (disposable/rebuildable per ADR-008, not synced). +- **Consumes:** the 18 reference fragments in `docs/memory-system/benchmark/reference-outputs/` and the 6 fixtures. +- **Unblocks:** Step 2e (per-project code graphs), Step 3 (hooks), and downstream plugin packaging — all of which assume a selected model and a built vault graph. diff --git a/openspec/changes/graphify-ollama-setup/specs/local-model-selection/spec.md b/openspec/changes/graphify-ollama-setup/specs/local-model-selection/spec.md new file mode 100644 index 0000000..71c1553 --- /dev/null +++ b/openspec/changes/graphify-ollama-setup/specs/local-model-selection/spec.md @@ -0,0 +1,53 @@ +## ADDED Requirements + +### Requirement: Extraction toolchain is installed and verified + +The Graphify CLI SHALL be installed from the `graphifyy` PyPI package and verified to run before any extraction is attempted, and a running Ollama with at least one pulled candidate model SHALL be available. + +#### Scenario: Graphify is callable + +- **WHEN** the toolchain setup completes +- **THEN** `graphify --version` returns a version without error +- **AND** at least one candidate Ollama model is pulled and listed by `ollama list` + +### Requirement: Ollama runtime is configured for extraction + +Ollama SHALL be configured with the settings the extraction run depends on: flash attention enabled (`OLLAMA_FLASH_ATTENTION=1`) and a context window sufficient for vault notes (`GRAPHIFY_OLLAMA_NUM_CTX=8192`), and the allocated context SHALL be verified after the first extraction call. + +#### Scenario: Configuration is in effect + +- **WHEN** the first extraction call is made +- **THEN** flash attention is enabled and the context size is 8192 +- **AND** `ollama ps` shows the expected allocated context for the loaded model + +### Requirement: Candidate models are scored against the gold-standard reference set + +The extraction model SHALL be selected by scoring candidate Ollama models against the existing Step 2c reference set (the 18 fragments in `docs/memory-system/benchmark/reference-outputs/`) over the same 6 fixture notes. Each candidate's Graphify-shaped output SHALL be compared to the `claude-opus-4-8` gold-standard output for entity correctness, relationship plausibility and typing, and `INFERRED`/`AMBIGUOUS` confidence-tag accuracy, and wall-clock extraction speed SHALL be measured per candidate. + +#### Scenario: Each candidate is scored on quality and speed + +- **WHEN** a candidate model is run over the 6 fixtures +- **THEN** its output is scored against the Opus reference on entity correctness, relationship typing, and confidence-tag accuracy +- **AND** its wall-clock extraction speed is recorded + +#### Scenario: Reference benchmark is consumed, not re-created + +- **WHEN** scoring is performed +- **THEN** it reads the existing reference fragments produced by the `reference-extraction-benchmark` capability +- **AND** it does not regenerate or modify the Claude reference set + +### Requirement: Model is selected by evidence, not hardcoded + +The chosen extraction model SHALL be the one justified by the scoring run's recorded results, and no model SHALL be hardcoded as the selection before scoring completes. `gemma4:e4b` MAY be the front-runner candidate, but its selection SHALL depend on its scored quality, not its feasibility gut-check alone. + +#### Scenario: Selection records its rationale + +- **WHEN** a model is selected +- **THEN** a result artifact records the chosen model, its quality scores against the Opus rubric, and its measured speed +- **AND** the rationale references the scoring evidence rather than asserting a pre-chosen model + +#### Scenario: No model is locked before scoring + +- **WHEN** scoring has not yet run +- **THEN** no model is committed as the selection +- **AND** the front-runner candidate is treated as unconfirmed until scored diff --git a/openspec/changes/graphify-ollama-setup/specs/vault-graph-build/spec.md b/openspec/changes/graphify-ollama-setup/specs/vault-graph-build/spec.md new file mode 100644 index 0000000..889e264 --- /dev/null +++ b/openspec/changes/graphify-ollama-setup/specs/vault-graph-build/spec.md @@ -0,0 +1,37 @@ +## ADDED Requirements + +### Requirement: Initial vault graph is built with the selected model + +The initial knowledge graph SHALL be built over the `~/Documents/SecondBrain` vault using the model selected by the `local-model-selection` capability, via Graphify's Ollama document-extraction backend, and SHALL NOT begin until a model has been selected. + +#### Scenario: Build uses the selected model + +- **WHEN** the initial vault graph is built +- **THEN** Graphify extracts over `~/Documents/SecondBrain` using the selected Ollama model +- **AND** the build does not run before model selection is complete + +#### Scenario: Build scope is the vault only + +- **WHEN** the initial graph is built +- **THEN** the extraction target is the vault, not any project code repository +- **AND** per-project code graphs are not built as part of this change + +### Requirement: Graph sanity is verified via god-nodes + +After the build, `GRAPH_REPORT.md` SHALL be reviewed to confirm the most-connected nodes (god-nodes) are the tools, clients, and domain concepts that the vault's content would predict, so an obviously wrong extraction is caught before downstream steps depend on it. + +#### Scenario: God-nodes match vault content + +- **WHEN** `GRAPH_REPORT.md` is reviewed after the build +- **THEN** the top god-nodes are recognizable high-traffic tools, clients, or domains from the vault +- **AND** an implausible god-node distribution is flagged rather than accepted + +### Requirement: Graph artifacts are treated as disposable + +The produced graph artifacts (`graphify-out/`, `GRAPH_REPORT.md`, any index caches) SHALL be treated as rebuildable from the markdown vault and SHALL NOT be synced as source of truth, consistent with the markdown-as-truth decision (ADR-008). + +#### Scenario: Graph output is not the source of truth + +- **WHEN** the graph artifacts are produced +- **THEN** they are rebuildable from the vault markdown +- **AND** they are excluded from vault sync rather than treated as authoritative diff --git a/openspec/changes/graphify-ollama-setup/tasks.md b/openspec/changes/graphify-ollama-setup/tasks.md new file mode 100644 index 0000000..67a0f0e --- /dev/null +++ b/openspec/changes/graphify-ollama-setup/tasks.md @@ -0,0 +1,32 @@ +> **STATUS (2026-06-05, complete — 16/16 tasks done):** Toolchain installed (task 1 ✓). Key findings in `docs/memory-system/benchmark/scoring-results-2026-06-04.md`. Critical mechanics discovered: (a) `GRAPHIFY_OLLAMA_NUM_CTX` does NOT propagate through graphify's ollama `/v1` endpoint — bake context via a Modelfile variant instead; (b) graphify base_url must end in `/v1`; (c) a one-line thinking-OFF patch (`reasoning_effort:"none"`) was applied to the installed `llm.py` (backup at /tmp/graphify-bench/llm.py.orig). Small 2–5B models (gemma4:e4b/e2b, qwen3.5:2b) FAIL the relationship/edge gate. **qwen2.5-coder:7b (graphify's shipped default) PASSES** — full 6-fixture extraction done (~59 tok/s). All formal scoring (3.4), selection record (3.5/3.6), vault build (4), and doc updates (5) completed. Ready for archival. + +## 1. Install and verify the toolchain (Step 2a) + +- [x] 1.1 Install Graphify: `pip install graphifyy` (double-y package; `graphify` command) +- [x] 1.2 Verify it runs: `graphify --version` returns a version without error +- [x] 1.3 Confirm Ollama is running and pull the front-runner candidate `gemma4:e4b` (and any sibling to be scored); verify with `ollama list` + +## 2. Configure Ollama for extraction (Step 2b) + +- [x] 2.1 Export `OLLAMA_FLASH_ATTENTION=1` and `GRAPHIFY_OLLAMA_NUM_CTX=8192` in the shell profile (defer `GRAPHIFY_OLLAMA_KEEP_ALIVE` to Step 6 packaging) — superseded — env-var approach does not propagate via /v1; use Modelfile-baked num_ctx (see findings). No shell-profile/systemd change made per user decision. +- [x] 2.2 Run one extraction call and verify allocated context with `ollama ps` (expect 8192) — context verified at 16384 via Modelfile variant in `ollama ps` (env-var path is a no-op through graphify). + +## 3. Score candidates against the Step 2c references + +- [x] 3.1 Confirm the reference set is intact: 18 fragments in `docs/memory-system/benchmark/reference-outputs/` and the 6 fixtures in `benchmark/dispatch-prompt.md` (read-only; do not regenerate) — 18 reference fragments + 6 fixtures confirmed intact, used read-only. +- [x] 3.2 Run each candidate Ollama model over the same 6 fixtures via Graphify's Ollama backend, capturing Graphify-shaped output per fixture — all candidates run via graphify ollama backend; qwen2.5-coder:7b results recorded in findings. +- [x] 3.3 Record wall-clock extraction speed per candidate — qwen2.5-coder:7b ~59 tok/s; small-model speeds in findings. +- [x] 3.4 Score each candidate's output against the `claude-opus-4-8` reference per fixture: entity correctness, relationship plausibility/typing, and `INFERRED`/`AMBIGUOUS` confidence-tag accuracy — Scored qwen2.5-coder:7b on the 3 comparable dims (entity correctness ~2.2/5, relationship plausibility ~1.9/5, confidence calibration 3/5 where observable); excluded dims (facets/entity-type/relation-typing) documented N/A. Full scoring table in scoring-results-2026-06-04.md §3.4. +- [x] 3.5 Write a scoring-result artifact (under `docs/memory-system/benchmark/`) recording per-candidate quality scores, measured speed, and the selected model with its rationale — Scoring table + per-fixture justifications + selection rationale recorded in docs/memory-system/benchmark/scoring-results-2026-06-04.md (§3.4, §3.6). +- [x] 3.6 Select the extraction model from the recorded evidence — do not lock a model before 3.4 completes — Selected qwen2.5-coder:7b (variant qwen25-coder-7b-16k) — graphify's default and the only candidate clearing the edge gate; smaller models all failed. Known limitation: shallow relationship semantics. ~59 tok/s. + +## 4. Build and verify the initial vault graph (Step 2d) + +- [x] 4.1 Build the graph with the selected model: `graphify extract --path ~/Documents/SecondBrain --backend ollama --model --token-budget 512 --max-concurrency 2` — Built with `graphify extract ~/Documents/SecondBrain --backend ollama --model qwen25-coder-7b-16k --max-concurrency 1 --token-budget 4000 --exclude .obsidian --out /tmp/graphify-bench/vault-graph`. Ran at concurrency 1 (not 2 — untested on 12GB GPU) and token-budget 4000 (not 512 — fits the 16k context). Result: 57 nodes, 43 edges, 15 communities, ~$0 local. +- [x] 4.2 Review `GRAPH_REPORT.md`: confirm top god-nodes are recognizable high-traffic tools, clients, and domains from the vault; flag an implausible distribution rather than accepting it — God-node distribution plausible: Speed-to-Lead (deg 12) dominant, then Email A/B/C Experiment Hub, ACV Estimates, Claude, Vault Conventions, project-config hubs — all recognizable high-traffic vault concepts; no implausible top node. Read from .graphify_analysis.json (extract does not emit GRAPH_REPORT.md). +- [x] 4.3 Confirm graph artifacts (`graphify-out/`, `GRAPH_REPORT.md`) are treated as disposable/rebuildable and excluded from vault sync (ADR-008) — Satisfied by construction — `--out /tmp/graphify-bench/vault-graph` writes outside ~/Documents/SecondBrain; nothing written into the vault. Artifacts disposable/rebuildable per ADR-008. + +## 5. Wrap up + +- [x] 5.1 Update `docs/memory-system/05-implementation-process.md` status line and Step 2 markers to reflect 2a/2b/2d executed and the selected model (resolve the `e4b`/`e2b` inconsistency) — also: stale 'no recommended model' claim in docs/graphify/05 already corrected 2026-06-05; the e4b/e2b naming resolved — these are real gemma4 8.0B/5.1B + qwen35 2.3B, not gemma3n renames (see findings). +- [x] 5.2 Verify the change with `/opsx:verify` before archiving