33 lines
5.4 KiB
Markdown
33 lines
5.4 KiB
Markdown
> **STATUS (2026-06-05, complete — 16/16 tasks done):** Toolchain installed (task 1 ✓). Key findings in `docs/memory-system/benchmark/scoring-results-2026-06-04.md`. Critical mechanics discovered: (a) `GRAPHIFY_OLLAMA_NUM_CTX` does NOT propagate through graphify's ollama `/v1` endpoint — bake context via a Modelfile variant instead; (b) graphify base_url must end in `/v1`; (c) a one-line thinking-OFF patch (`reasoning_effort:"none"`) was applied to the installed `llm.py` (backup at /tmp/graphify-bench/llm.py.orig). Small 2–5B models (gemma4:e4b/e2b, qwen3.5:2b) FAIL the relationship/edge gate. **qwen2.5-coder:7b (graphify's shipped default) PASSES** — full 6-fixture extraction done (~59 tok/s). All formal scoring (3.4), selection record (3.5/3.6), vault build (4), and doc updates (5) completed. Ready for archival.
|
||
|
||
## 1. Install and verify the toolchain (Step 2a)
|
||
|
||
- [x] 1.1 Install Graphify: `pip install graphifyy` (double-y package; `graphify` command)
|
||
- [x] 1.2 Verify it runs: `graphify --version` returns a version without error
|
||
- [x] 1.3 Confirm Ollama is running and pull the front-runner candidate `gemma4:e4b` (and any sibling to be scored); verify with `ollama list`
|
||
|
||
## 2. Configure Ollama for extraction (Step 2b)
|
||
|
||
- [x] 2.1 Export `OLLAMA_FLASH_ATTENTION=1` and `GRAPHIFY_OLLAMA_NUM_CTX=8192` in the shell profile (defer `GRAPHIFY_OLLAMA_KEEP_ALIVE` to Step 6 packaging) — superseded — env-var approach does not propagate via /v1; use Modelfile-baked num_ctx (see findings). No shell-profile/systemd change made per user decision.
|
||
- [x] 2.2 Run one extraction call and verify allocated context with `ollama ps` (expect 8192) — context verified at 16384 via Modelfile variant in `ollama ps` (env-var path is a no-op through graphify).
|
||
|
||
## 3. Score candidates against the Step 2c references
|
||
|
||
- [x] 3.1 Confirm the reference set is intact: 18 fragments in `docs/memory-system/benchmark/reference-outputs/` and the 6 fixtures in `benchmark/dispatch-prompt.md` (read-only; do not regenerate) — 18 reference fragments + 6 fixtures confirmed intact, used read-only.
|
||
- [x] 3.2 Run each candidate Ollama model over the same 6 fixtures via Graphify's Ollama backend, capturing Graphify-shaped output per fixture — all candidates run via graphify ollama backend; qwen2.5-coder:7b results recorded in findings.
|
||
- [x] 3.3 Record wall-clock extraction speed per candidate — qwen2.5-coder:7b ~59 tok/s; small-model speeds in findings.
|
||
- [x] 3.4 Score each candidate's output against the `claude-opus-4-8` reference per fixture: entity correctness, relationship plausibility/typing, and `INFERRED`/`AMBIGUOUS` confidence-tag accuracy — Scored qwen2.5-coder:7b on the 3 comparable dims (entity correctness ~2.2/5, relationship plausibility ~1.9/5, confidence calibration 3/5 where observable); excluded dims (facets/entity-type/relation-typing) documented N/A. Full scoring table in scoring-results-2026-06-04.md §3.4.
|
||
- [x] 3.5 Write a scoring-result artifact (under `docs/memory-system/benchmark/`) recording per-candidate quality scores, measured speed, and the selected model with its rationale — Scoring table + per-fixture justifications + selection rationale recorded in docs/memory-system/benchmark/scoring-results-2026-06-04.md (§3.4, §3.6).
|
||
- [x] 3.6 Select the extraction model from the recorded evidence — do not lock a model before 3.4 completes — Selected qwen2.5-coder:7b (variant qwen25-coder-7b-16k) — graphify's default and the only candidate clearing the edge gate; smaller models all failed. Known limitation: shallow relationship semantics. ~59 tok/s.
|
||
|
||
## 4. Build and verify the initial vault graph (Step 2d)
|
||
|
||
- [x] 4.1 Build the graph with the selected model: `graphify extract --path ~/Documents/SecondBrain --backend ollama --model <selected> --token-budget 512 --max-concurrency 2` — Built with `graphify extract ~/Documents/SecondBrain --backend ollama --model qwen25-coder-7b-16k --max-concurrency 1 --token-budget 4000 --exclude .obsidian --out /tmp/graphify-bench/vault-graph`. Ran at concurrency 1 (not 2 — untested on 12GB GPU) and token-budget 4000 (not 512 — fits the 16k context). Result: 57 nodes, 43 edges, 15 communities, ~$0 local.
|
||
- [x] 4.2 Review `GRAPH_REPORT.md`: confirm top god-nodes are recognizable high-traffic tools, clients, and domains from the vault; flag an implausible distribution rather than accepting it — God-node distribution plausible: Speed-to-Lead (deg 12) dominant, then Email A/B/C Experiment Hub, ACV Estimates, Claude, Vault Conventions, project-config hubs — all recognizable high-traffic vault concepts; no implausible top node. Read from .graphify_analysis.json (extract does not emit GRAPH_REPORT.md).
|
||
- [x] 4.3 Confirm graph artifacts (`graphify-out/`, `GRAPH_REPORT.md`) are treated as disposable/rebuildable and excluded from vault sync (ADR-008) — Satisfied by construction — `--out /tmp/graphify-bench/vault-graph` writes outside ~/Documents/SecondBrain; nothing written into the vault. Artifacts disposable/rebuildable per ADR-008.
|
||
|
||
## 5. Wrap up
|
||
|
||
- [x] 5.1 Update `docs/memory-system/05-implementation-process.md` status line and Step 2 markers to reflect 2a/2b/2d executed and the selected model (resolve the `e4b`/`e2b` inconsistency) — also: stale 'no recommended model' claim in docs/graphify/05 already corrected 2026-06-05; the e4b/e2b naming resolved — these are real gemma4 8.0B/5.1B + qwen35 2.3B, not gemma3n renames (see findings).
|
||
- [x] 5.2 Verify the change with `/opsx:verify` before archiving
|