13 KiB

Raw Blame History

Local-LLM Doc-Extraction Gut-Check — 2026-06-04

Last updated: 2026-06-04 | Status: gut-check complete; gemma4:e4b is the candidate; GPU confirmed post-reboot (~74 tok/s, see appendix); Graphify validation is the next step.

This document consolidates the local-LLM doc-extraction gut-check run on 2026-06-04. It is the authoritative findings record for that session and feeds forward into the Graphify-ollama validation step.

Four things this document keeps strictly separate:

(A) Model-capability signal — what carries forward; the durable conclusion.
(B) Raw-ollama harness gotchas — benchmark-process notes only; explicitly NOT production config.
(C) GPU driver fix — real diagnosis; timing estimate, not a measured result.
(D) Graphify config surface — the actual production guidance.

Cross-references:

Claude-tier reference run: docs/memory-system/benchmark/run-findings-2026-06-04.md
18 Claude reference fragments: docs/memory-system/benchmark/reference-outputs/ (.haiku.md, .sonnet.md, .opus.md files only)
Authoritative Graphify-ollama section: docs/graphify/05-local-models-and-backends.md

TL;DR — What worked

The local model class is good-enough for the doc-extraction role. Best installed model: gemma4:e4b (~9.6 GB, fits the RTX 3060's 12 GB). It produced valid schema on 6/6 fixtures with clean, well-namespaced facets (0 bad facets out of 6) and note-grounded entities (no contamination).
It catches the high-value entities (~45–60% of Opus's entity recall), missing Opus's long tail — acceptable for the role.
Critical reframe: in production we will NOT hand-invoke ollama. Graphify owns the ollama call (prompt, chunking, context window, parsing) over its HTTP API. The hand-rolled benchmark validated model capability, not the production pipeline. The raw-ollama tuning discovered during this run is Graphify-internal, not our config surface.

(A) Model-capability signal — measured via a hand-rolled prompt/parser (NOT Graphify)

This signal was gathered through a custom prompt and a custom YAML parser, NOT Graphify's extraction path. It says "the model class is good-enough." It does NOT say "this is what Graphify will output" — Graphify uses a different prompt, schema, and parser.

Results across the 3 installed models (all reasoning models; only these 3 are on the machine)

Model	Size	Parse rate	Entities/note	Facet quality	Verdict
`gemma4:e4b`	~9.6 GB	6/6 (2 needed trivial colon-quote repair)	13–34 (~45–60% of Opus)	0 bad/6	Usable — best pick
`gemma4:e2b`	~7.2 GB	4/6 (2 hard YAML breaks — dropped `source:` key)	23–31 when valid	0 bad/6	Usable only behind a tolerant parser + retry
`qwen3.5:2b`	~2.7 GB	parses but collapses to ~1 entity/note	1	n/a	Not usable as tested

All three land below haiku (the weakest Claude frontier tier: 19–37 entities, 6/6 clean parse, clean referential integrity). "Below frontier-floor" does not mean "unusable" — gemma4:e4b's misses are the long tail; the high-value entities are present.

gemma4:e4b quirks observed

Occasional top-level key rename (e.g. entity_relationships instead of the spec's key).
A target: Concept placeholder leak on the longest note — it echoed the schema's example value rather than a concrete entity. Seen once across 6 fixtures.

These quirks were caught by the hand-rolled parser and are noted as model-class behavior. Whether Graphify's parser handles them gracefully is a separate question (see section D).

(B) Raw-ollama harness gotchas — benchmark-process notes ONLY

EXPLICIT CAVEAT: This section records why the first benchmark pass produced junk and what was done to fix the harness. These knobs are NOT our production settings. They are Graphify-internal concerns (see section D). Do not treat this section as production config.

Problem: default context truncation produced garbage output

Bare ollama run with the default (~4K) context truncated output mid-stream. Reasoning models burn their token budget on chain-of-thought first; the model got guillotined mid-output, leaving malformed YAML.

Harness fix (benchmark only)

Called the HTTP API at /api/generate instead of ollama run.
Set num_ctx=8192 (per-request via the API body).
Set think=false to suppress chain-of-thought tokens and keep the output budget for the extraction result.
Added a tolerant YAML parser that retries on soft parse errors and auto-quotes bare scalars containing :.

These changes produced the results in section A. They explain the benchmark outcome; they do not define the production path, which Graphify handles internally.

(C) GPU not used during this run — root cause and fix

Root cause: NVIDIA driver version mismatch

Item	Value
In-kernel module	580.142
On-disk / userspace driver	580.159.03

The driver was updated on disk on 2026-06-03 but the stale module stayed resident. Result: nvidia-smi fails — "Failed to initialize NVML: Driver/library version mismatch." Because NVML/CUDA cannot initialize, ollama finds no CUDA device (total_vram="0 B") and runs 100% on CPU.

Same-machine proof: the ollama log at 07:57 on 2026-06-03 DID find the RTX 3060 (compute=8.6 ... 12.0 GiB); the 16:12 service instance the benchmark ran under found no CUDA.

CPU eval rate observed: ~10.6 tok/s → ~187–313 s/note. The 9.6 GB model fits the 12 GB card; VRAM is not the bottleneck — the driver mismatch is.

Fix (user must run — non-interactive sudo was blocked during this session)

The running kernel already matches the on-disk 580.159.03 module, so a reboot loads the correct module:

sudo reboot
# then verify:
nvidia-smi          # should print the RTX 3060 table, no NVML error

# only if CUDA still fails after reboot:
sudo dnf reinstall 'akmod-nvidia*' 'xorg-x11-drv-nvidia*'   # then reboot again

ESTIMATE — post-reboot GPU timing (NOT measured; blocked by sudo this session)

With the RTX 3060 and the 4-bit gemma4:e4b model fully resident, ~40–60 tok/s plus faster prompt processing should drop extraction to roughly single-digit to ~20 s/note — i.e., plausibly within a "well under a minute" target.

This is an estimate to be confirmed post-reboot using the curl command in the appendix below. It is not a measured result.

(D) Graphify config surface — production guidance

This is the only section that contains production configuration guidance. Sections B and C are process notes; section A is a model-capability signal only.

How Graphify talks to Ollama

Graphify talks to Ollama over its HTTP API at http://localhost:11434, not by shelling out to ollama run. [github-inferred] from the env-var/timeout/num_ctx design; the exact endpoint (/api/generate vs /api/chat) is [unverified] — package internals are not browseable on GitHub.

Graphify owns the prompt template, chunking, context sizing (auto-sized num_ctx by default), and output parsing. [github] for the env-var knobs; prompt and parser specifics are [github-inferred] / [unverified]. Whether Graphify sets think=false and whether it does tolerant-YAML / retry are [unverified].

Implication: the hand-rolled extraction-spec prompt and raw-ollama settings from section B likely do not apply to Graphify's ollama path. Our levers are the documented external knobs only.

Text-vs-code split

Confirmed [github] (verbatim README): code is extracted locally via tree-sitter AST with zero LLM calls; docs/PDFs/images go through the LLM backend (Ollama here). Ollama is invoked for text documents only.

Model selection

No official Graphify-recommended Ollama model. [github] — absence verified at v0.8.30. Community starting points: qwen2.5:7b, phi4:14b. [community] Default behavior: auto-detect the running model. Override via OLLAMA_MODEL.

Per this benchmark, gemma4:e4b is the best installed candidate — but this must be validated through Graphify's own extraction path, not the hand-rolled benchmark (see Deferred follow-ups §1 below).

Environment variables and CLI flags (all `[github]` from v0.8.30 README)

Knob	Default	Notes
`OLLAMA_BASE_URL`	`http://localhost:11434`	Ollama endpoint
`OLLAMA_MODEL`	auto-detect	Override to pin a model
`GRAPHIFY_OLLAMA_NUM_CTX`	auto-sized	Override context window
`GRAPHIFY_OLLAMA_KEEP_ALIVE`	unspecified	Minutes to keep model loaded; `0` = unload after each chunk
`--token-budget`	—	Chunk size (tokens)
`--max-concurrency`	—	Parallel extraction workers
`--api-timeout`	600 s	Increase for slower hardware

Known sharp edge

Context saturation across consecutive chunks on the Ollama backend can exhaust VRAM after a few chunks. [community, issue #798] Mitigate by lowering GRAPHIFY_OLLAMA_NUM_CTX and/or setting GRAPHIFY_OLLAMA_KEEP_ALIVE=0.

Appendix — Exact commands

Graphify Ollama path `[github]`

# Install (package is graphifyy — double-y; command is graphify):
uv tool install "graphifyy[ollama]"

# Pull the candidate model (gemma4:e4b is the local best pick per this benchmark):
ollama pull gemma4:e4b

# Start Ollama if not already running as a service:
ollama serve

# Basic extraction:
graphify extract ./docs --backend ollama

# Constrained single-GPU tuning:
GRAPHIFY_OLLAMA_NUM_CTX=8192 graphify extract ./docs --backend ollama \
  --token-budget 4000 --max-concurrency 2 --api-timeout 900

# Free VRAM between chunks (mitigates issue #798):
GRAPHIFY_OLLAMA_KEEP_ALIVE=0 graphify extract ./docs --backend ollama

Post-reboot GPU re-timing (benchmark verification only — NOT production)

Run this after rebooting to confirm the GPU is now used and to measure actual tok/s:

F="$HOME/Documents/SecondBrain/2026-03-31-10dlc-isv-setup-guide-oncadence.md"
PROMPT=$(python3 -c "import json;print(json.dumps('Extract entities from this note:\n\n'+open('$F').read()))")
time curl -s http://127.0.0.1:11434/api/generate \
  -d "{\"model\":\"gemma4:e4b\",\"prompt\":$PROMPT,\"stream\":false,\"think\":false,\"options\":{\"num_ctx\":8192}}" \
  | python3 -c "import json,sys;d=json.load(sys.stdin);ec=d['eval_count'];ed=d['eval_duration'];print('eval tok/s:',round(ec/(ed/1e9),2),'| eval_count:',ec,'| total wall (s):',round(d['total_duration']/1e9,2))"

This measures raw model tok/s via the hand-rolled harness — it does NOT measure Graphify's production extraction performance.

Measured results (run 2026-06-04, post-reboot)

GPU is now fully active and resident — the driver mismatch diagnosed in section C is RESOLVED. nvidia-smi is healthy (Driver 580.159.03, RTX 3060, no NVML error). The gemma4:e4b model loaded fully GPU-resident: /api/ps reports size_vram 3.29 GB == size 3.29 GB (no CPU fallback), with 4462 MiB used on the card.

Pass	eval tok/s	eval_count	total wall (s)	load (s)	prompt_eval (s)	prompt_eval_count
Cold	74.27	701	76.04	38.15	28.44	905
Warm	74.25	717	10.13	0.43	0.04	—

Conclusion: steady-state generation is ~74 tok/s — at or above the section C estimate of 40–60 tok/s. Warm extraction lands at ~10 s/note; the cold pass is ~76 s, dominated by the one-time 38 s model load plus a 28 s cold prompt-eval. This confirms section C's "well under a minute once the model is resident" prediction.

Caveat retained: these are raw model tok/s from the hand-rolled curl harness, NOT Graphify production extraction performance (see Deferred follow-up §1).

Deferred follow-ups

Definitive validation: run Graphify's own ollama path over the fixtures — NOT re-scoring the hand-rolled benchmark. Do NOT conclude the next step is "score Graphify output against the 18 Opus references": that is apples-to-oranges (different prompt/schema/output). The 18 Opus references validated model capability; they are not a scoring rubric for Graphify's graph output.
Resolve [unverified] Graphify internals (HTTP endpoint, think-mode, retry/parsing): after uv tool install "graphifyy[ollama]", read the installed client source under the tool's site-packages. This does not require the GPU — a cheap deferred follow-up.
Confirm GPU timing post-reboot — RESOLVED (2026-06-04). GPU re-timing ran post-reboot; measured ~74 tok/s steady-state (at/above the section C estimate). See the "Measured results" table under the Post-reboot GPU re-timing appendix above.
Raw per-model benchmark outputs from this run were throwaway scratch (wrong harness, hand-rolled prompt) and are not committed.

13 KiB Raw Blame History Unescape Escape