From 65595c39955caafd6833d5d1973094ff9e32e621 Mon Sep 17 00:00:00 2001 From: jared Date: Thu, 4 Jun 2026 14:37:39 -0400 Subject: [PATCH] Consolidate local-LLM gut-check findings and update pointers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add new doc docs/memory-system/benchmark/local-llm-findings-2026-06-04.md consolidating the local-LLM doc-extraction gut-check run. Key findings: gemma4:e4b is the best installed model and adequate for the extraction role (below frontier haiku but acceptable, catching ~45–60% of Opus's high-value entities); GPU went unused due to NVIDIA driver version mismatch (fix = reboot, pending); and Graphify owns the ollama call (HTTP API, prompt/chunking/context/parsing internal) so raw-ollama tuning is not production config. Add progressive-disclosure pointers from 04-build-plan.md and 05-implementation-process.md. --- docs/memory-system/04-build-plan.md | 2 +- .../05-implementation-process.md | 1 + .../local-llm-findings-2026-06-04.md | 253 ++++++++++++++++++ 3 files changed, 255 insertions(+), 1 deletion(-) create mode 100644 docs/memory-system/benchmark/local-llm-findings-2026-06-04.md diff --git a/docs/memory-system/04-build-plan.md b/docs/memory-system/04-build-plan.md index b4c3f95..1caff3e 100644 --- a/docs/memory-system/04-build-plan.md +++ b/docs/memory-system/04-build-plan.md @@ -108,7 +108,7 @@ re-enters and the final model is chosen. Do not hardcode a model before that run fragments generated in `benchmark/reference-outputs/`. Run as-is (no vault frontmatter modification); verified clean. Fixtures listed in `benchmark/dispatch-prompt.md`. -Authoritative detail lives in `docs/memory-system/05-implementation-process.md` §2c. +Authoritative detail lives in `docs/memory-system/05-implementation-process.md` §2c. Local-model gut-check done (2026-06-04): `gemma4:e4b` is the candidate; GPU fix pending reboot; Graphify owns the ollama call — see `docs/memory-system/benchmark/local-llm-findings-2026-06-04.md`. #### 2d — Initial fixture graph build (ADR-013: small-first) Run the initial build against the small fixture set (5–10 notes from Step 1/2c), not the diff --git a/docs/memory-system/05-implementation-process.md b/docs/memory-system/05-implementation-process.md index 17b74a8..524d73f 100644 --- a/docs/memory-system/05-implementation-process.md +++ b/docs/memory-system/05-implementation-process.md @@ -159,6 +159,7 @@ Verify context allocation after the first extraction call: `ollama ps` shows all **Status (2026-06-04): EXECUTED.** 6 cross-domain fixtures × 3 Claude tiers = 18 reference fragments generated in `benchmark/reference-outputs/`. Run as-is (no vault frontmatter modification); verified clean. Gate is passed — Ollama model scoring is now unblocked. +Local-model gut-check also done: `gemma4:e4b` is the candidate; GPU fix pending reboot; Graphify owns the ollama call — see `benchmark/local-llm-findings-2026-06-04.md`. Produce a reference set of Graphify-shaped extraction outputs before committing to any local Ollama model. Use the 5–10 fixture notes selected in Step 1a as the input set. diff --git a/docs/memory-system/benchmark/local-llm-findings-2026-06-04.md b/docs/memory-system/benchmark/local-llm-findings-2026-06-04.md new file mode 100644 index 0000000..31a93ad --- /dev/null +++ b/docs/memory-system/benchmark/local-llm-findings-2026-06-04.md @@ -0,0 +1,253 @@ +# Local-LLM Doc-Extraction Gut-Check — 2026-06-04 + +_Last updated: 2026-06-04_ | _Status: gut-check complete; gemma4:e4b is the candidate; GPU fix pending reboot; Graphify validation is the next step._ + +This document consolidates the local-LLM doc-extraction gut-check run on 2026-06-04. It is +the authoritative findings record for that session and feeds forward into the Graphify-ollama +validation step. + +**Four things this document keeps strictly separate:** + +- **(A) Model-capability signal** — what carries forward; the durable conclusion. +- **(B) Raw-ollama harness gotchas** — benchmark-process notes only; explicitly NOT production config. +- **(C) GPU driver fix** — real diagnosis; timing estimate, not a measured result. +- **(D) Graphify config surface** — the actual production guidance. + +Cross-references: +- Claude-tier reference run: `docs/memory-system/benchmark/run-findings-2026-06-04.md` +- 18 Claude reference fragments: `docs/memory-system/benchmark/reference-outputs/` (`.haiku.md`, `.sonnet.md`, `.opus.md` files only) +- Authoritative Graphify-ollama section: `docs/graphify/05-local-models-and-backends.md` + +--- + +## TL;DR — What worked + +- The local model class is **good-enough** for the doc-extraction role. Best installed model: + **`gemma4:e4b`** (~9.6 GB, fits the RTX 3060's 12 GB). It produced valid schema on 6/6 + fixtures with clean, well-namespaced facets (0 bad facets out of 6) and note-grounded + entities (no contamination). +- It catches the high-value entities (~45–60% of Opus's entity recall), missing Opus's long + tail — acceptable for the role. +- **Critical reframe:** in production we will NOT hand-invoke ollama. Graphify owns the ollama + call (prompt, chunking, context window, parsing) over its HTTP API. The hand-rolled benchmark + validated *model capability*, not the production pipeline. The raw-ollama tuning discovered + during this run is Graphify-internal, not our config surface. + +--- + +## (A) Model-capability signal — measured via a hand-rolled prompt/parser (NOT Graphify) + +This signal was gathered through a custom prompt and a custom YAML parser, NOT Graphify's +extraction path. It says "the model class is good-enough." It does NOT say "this is what +Graphify will output" — Graphify uses a different prompt, schema, and parser. + +### Results across the 3 installed models (all reasoning models; only these 3 are on the machine) + +| Model | Size | Parse rate | Entities/note | Facet quality | Verdict | +|---|---|---|---|---|---| +| `gemma4:e4b` | ~9.6 GB | 6/6 (2 needed trivial colon-quote repair) | 13–34 (~45–60% of Opus) | 0 bad/6 | **Usable — best pick** | +| `gemma4:e2b` | ~7.2 GB | 4/6 (2 hard YAML breaks — dropped `source:` key) | 23–31 when valid | 0 bad/6 | Usable only behind a tolerant parser + retry | +| `qwen3.5:2b` | ~2.7 GB | parses but collapses to ~1 entity/note | 1 | n/a | Not usable as tested | + +All three land below haiku (the weakest Claude frontier tier: 19–37 entities, 6/6 clean parse, +clean referential integrity). "Below frontier-floor" does not mean "unusable" — `gemma4:e4b`'s +misses are the long tail; the high-value entities are present. + +### gemma4:e4b quirks observed + +- Occasional top-level key rename (e.g. `entity_relationships` instead of the spec's key). +- A `target: Concept` placeholder leak on the longest note — it echoed the schema's example + value rather than a concrete entity. Seen once across 6 fixtures. + +These quirks were caught by the hand-rolled parser and are noted as model-class behavior. +Whether Graphify's parser handles them gracefully is a separate question (see section D). + +--- + +## (B) Raw-ollama harness gotchas — benchmark-process notes ONLY + +> **EXPLICIT CAVEAT:** This section records why the first benchmark pass produced junk and +> what was done to fix the harness. These knobs are NOT our production settings. They are +> Graphify-internal concerns (see section D). Do not treat this section as production config. + +### Problem: default context truncation produced garbage output + +Bare `ollama run` with the default (~4K) context truncated output mid-stream. Reasoning models +burn their token budget on chain-of-thought first; the model got guillotined mid-output, leaving +malformed YAML. + +### Harness fix (benchmark only) + +- Called the HTTP API at `/api/generate` instead of `ollama run`. +- Set `num_ctx=8192` (per-request via the API body). +- Set `think=false` to suppress chain-of-thought tokens and keep the output budget for the + extraction result. +- Added a tolerant YAML parser that retries on soft parse errors and auto-quotes bare scalars + containing `:`. + +These changes produced the results in section A. They explain the benchmark outcome; they do +not define the production path, which Graphify handles internally. + +--- + +## (C) GPU not used during this run — root cause and fix + +### Root cause: NVIDIA driver version mismatch + +| Item | Value | +|---|---| +| In-kernel module | 580.142 | +| On-disk / userspace driver | 580.159.03 | + +The driver was updated on disk on 2026-06-03 but the stale module stayed resident. Result: +`nvidia-smi` fails — "Failed to initialize NVML: Driver/library version mismatch." Because +NVML/CUDA cannot initialize, ollama finds no CUDA device (`total_vram="0 B"`) and runs 100% +on CPU. + +Same-machine proof: the ollama log at 07:57 on 2026-06-03 DID find the RTX 3060 +(`compute=8.6 ... 12.0 GiB`); the 16:12 service instance the benchmark ran under found no CUDA. + +**CPU eval rate observed:** ~10.6 tok/s → ~187–313 s/note. The 9.6 GB model fits the 12 GB +card; VRAM is not the bottleneck — the driver mismatch is. + +### Fix (user must run — non-interactive sudo was blocked during this session) + +The running kernel already matches the on-disk 580.159.03 module, so a reboot loads the correct +module: + +```bash +sudo reboot +# then verify: +nvidia-smi # should print the RTX 3060 table, no NVML error + +# only if CUDA still fails after reboot: +sudo dnf reinstall 'akmod-nvidia*' 'xorg-x11-drv-nvidia*' # then reboot again +``` + +### ESTIMATE — post-reboot GPU timing (NOT measured; blocked by sudo this session) + +With the RTX 3060 and the 4-bit `gemma4:e4b` model fully resident, ~40–60 tok/s plus faster +prompt processing should drop extraction to roughly single-digit to ~20 s/note — i.e., +plausibly within a "well under a minute" target. + +**This is an estimate to be confirmed post-reboot using the curl command in the appendix below.** +It is not a measured result. + +--- + +## (D) Graphify config surface — production guidance + +> This is the only section that contains production configuration guidance. Sections B and C +> are process notes; section A is a model-capability signal only. + +### How Graphify talks to Ollama + +Graphify talks to Ollama over its **HTTP API** at `http://localhost:11434`, not by shelling out +to `ollama run`. `[github-inferred]` from the env-var/timeout/num_ctx design; the exact endpoint +(`/api/generate` vs `/api/chat`) is `[unverified]` — package internals are not browseable on +GitHub. + +Graphify **owns** the prompt template, chunking, context sizing (auto-sized `num_ctx` by +default), and output parsing. `[github]` for the env-var knobs; prompt and parser specifics are +`[github-inferred]` / `[unverified]`. Whether Graphify sets `think=false` and whether it does +tolerant-YAML / retry are `[unverified]`. + +Implication: the hand-rolled extraction-spec prompt and raw-ollama settings from section B +likely do **not** apply to Graphify's ollama path. Our levers are the documented external knobs +only. + +### Text-vs-code split + +Confirmed `[github]` (verbatim README): code is extracted locally via tree-sitter AST with zero +LLM calls; docs/PDFs/images go through the LLM backend (Ollama here). Ollama is invoked **for +text documents only.** + +### Model selection + +No official Graphify-recommended Ollama model. `[github]` — absence verified at v0.8.30. +Community starting points: `qwen2.5:7b`, `phi4:14b`. `[community]` Default behavior: auto-detect +the running model. Override via `OLLAMA_MODEL`. + +Per this benchmark, `gemma4:e4b` is the best installed candidate — but this must be validated +through Graphify's own extraction path, not the hand-rolled benchmark (see Deferred +follow-ups §1 below). + +### Environment variables and CLI flags (all `[github]` from v0.8.30 README) + +| Knob | Default | Notes | +|---|---|---| +| `OLLAMA_BASE_URL` | `http://localhost:11434` | Ollama endpoint | +| `OLLAMA_MODEL` | auto-detect | Override to pin a model | +| `GRAPHIFY_OLLAMA_NUM_CTX` | auto-sized | Override context window | +| `GRAPHIFY_OLLAMA_KEEP_ALIVE` | unspecified | Minutes to keep model loaded; `0` = unload after each chunk | +| `--token-budget` | — | Chunk size (tokens) | +| `--max-concurrency` | — | Parallel extraction workers | +| `--api-timeout` | 600 s | Increase for slower hardware | + +### Known sharp edge + +Context saturation across consecutive chunks on the Ollama backend can exhaust VRAM after a +few chunks. `[community, issue #798]` Mitigate by lowering `GRAPHIFY_OLLAMA_NUM_CTX` and/or +setting `GRAPHIFY_OLLAMA_KEEP_ALIVE=0`. + +--- + +## Appendix — Exact commands + +### Graphify Ollama path `[github]` + +```bash +# Install (package is graphifyy — double-y; command is graphify): +uv tool install "graphifyy[ollama]" + +# Pull the candidate model (gemma4:e4b is the local best pick per this benchmark): +ollama pull gemma4:e4b + +# Start Ollama if not already running as a service: +ollama serve + +# Basic extraction: +graphify extract ./docs --backend ollama + +# Constrained single-GPU tuning: +GRAPHIFY_OLLAMA_NUM_CTX=8192 graphify extract ./docs --backend ollama \ + --token-budget 4000 --max-concurrency 2 --api-timeout 900 + +# Free VRAM between chunks (mitigates issue #798): +GRAPHIFY_OLLAMA_KEEP_ALIVE=0 graphify extract ./docs --backend ollama +``` + +### Post-reboot GPU re-timing (benchmark verification only — NOT production) + +Run this after rebooting to confirm the GPU is now used and to measure actual tok/s: + +```bash +F="$HOME/Documents/SecondBrain/2026-03-31-10dlc-isv-setup-guide-oncadence.md" +PROMPT=$(python3 -c "import json;print(json.dumps('Extract entities from this note:\n\n'+open('$F').read()))") +time curl -s http://127.0.0.1:11434/api/generate \ + -d "{\"model\":\"gemma4:e4b\",\"prompt\":$PROMPT,\"stream\":false,\"think\":false,\"options\":{\"num_ctx\":8192}}" \ + | python3 -c "import json,sys;d=json.load(sys.stdin);ec=d['eval_count'];ed=d['eval_duration'];print('eval tok/s:',round(ec/(ed/1e9),2),'| eval_count:',ec,'| total wall (s):',round(d['total_duration']/1e9,2))" +``` + +This measures raw model tok/s via the hand-rolled harness — it does NOT measure Graphify's +production extraction performance. + +--- + +## Deferred follow-ups + +1. **Definitive validation: run Graphify's own ollama path over the fixtures** — NOT re-scoring + the hand-rolled benchmark. Do NOT conclude the next step is "score Graphify output against + the 18 Opus references": that is apples-to-oranges (different prompt/schema/output). The 18 + Opus references validated model capability; they are not a scoring rubric for Graphify's + graph output. + +2. **Resolve `[unverified]` Graphify internals** (HTTP endpoint, think-mode, retry/parsing): + after `uv tool install "graphifyy[ollama]"`, read the installed client source under the + tool's site-packages. This does not require the GPU — a cheap deferred follow-up. + +3. **Confirm GPU timing post-reboot** using the curl command in the appendix above. The + ~40–60 tok/s estimate in section C is currently unverified. + +4. **Raw per-model benchmark outputs from this run** were throwaway scratch (wrong harness, + hand-rolled prompt) and are not committed.