cc-os/docs/memory-system/benchmark/local-llm-findings-2026-06-04.md

# Local-LLM Doc-Extraction Gut-Check — 2026-06-04

_Last updated: 2026-06-04_ | _Status: gut-check complete; gemma4:e4b is the candidate; GPU confirmed post-reboot (~74 tok/s, see appendix); Graphify validation is the next step._

This document consolidates the local-LLM doc-extraction gut-check run on 2026-06-04. It is
the authoritative findings record for that session and feeds forward into the Graphify-ollama
validation step.

**Four things this document keeps strictly separate:**

- **(A) Model-capability signal** — what carries forward; the durable conclusion.
- **(B) Raw-ollama harness gotchas** — benchmark-process notes only; explicitly NOT production config.
- **(C) GPU driver fix** — real diagnosis; timing estimate, not a measured result.
- **(D) Graphify config surface** — the actual production guidance.

Cross-references:
- Claude-tier reference run: `docs/memory-system/benchmark/run-findings-2026-06-04.md`
- 18 Claude reference fragments: `docs/memory-system/benchmark/reference-outputs/` (`.haiku.md`, `.sonnet.md`, `.opus.md` files only)
- Authoritative Graphify-ollama section: `docs/graphify/05-local-models-and-backends.md`

---

## TL;DR — What worked

- The local model class is **good-enough** for the doc-extraction role. Best installed model:
  **`gemma4:e4b`** (~9.6 GB, fits the RTX 3060's 12 GB). It produced valid schema on 6/6
  fixtures with clean, well-namespaced facets (0 bad facets out of 6) and note-grounded
  entities (no contamination).
- It catches the high-value entities (~45–60% of Opus's entity recall), missing Opus's long
  tail — acceptable for the role.
- **Critical reframe:** in production we will NOT hand-invoke ollama. Graphify owns the ollama
  call (prompt, chunking, context window, parsing) over its HTTP API. The hand-rolled benchmark
  validated *model capability*, not the production pipeline. The raw-ollama tuning discovered
  during this run is Graphify-internal, not our config surface.

---

## (A) Model-capability signal — measured via a hand-rolled prompt/parser (NOT Graphify)

This signal was gathered through a custom prompt and a custom YAML parser, NOT Graphify's
extraction path. It says "the model class is good-enough." It does NOT say "this is what
Graphify will output" — Graphify uses a different prompt, schema, and parser.

### Results across the 3 installed models (all reasoning models; only these 3 are on the machine)

| Model | Size | Parse rate | Entities/note | Facet quality | Verdict |
|---|---|---|---|---|---|
| `gemma4:e4b` | ~9.6 GB | 6/6 (2 needed trivial colon-quote repair) | 13–34 (~45–60% of Opus) | 0 bad/6 | **Usable — best pick** |
| `gemma4:e2b` | ~7.2 GB | 4/6 (2 hard YAML breaks — dropped `source:` key) | 23–31 when valid | 0 bad/6 | Usable only behind a tolerant parser + retry |
| `qwen3.5:2b` | ~2.7 GB | parses but collapses to ~1 entity/note | 1 | n/a | Not usable as tested |

All three land below haiku (the weakest Claude frontier tier: 19–37 entities, 6/6 clean parse,
clean referential integrity). "Below frontier-floor" does not mean "unusable" — `gemma4:e4b`'s
misses are the long tail; the high-value entities are present.

### gemma4:e4b quirks observed

- Occasional top-level key rename (e.g. `entity_relationships` instead of the spec's key).
- A `target: Concept` placeholder leak on the longest note — it echoed the schema's example
  value rather than a concrete entity. Seen once across 6 fixtures.

These quirks were caught by the hand-rolled parser and are noted as model-class behavior.
Whether Graphify's parser handles them gracefully is a separate question (see section D).

---

## (B) Raw-ollama harness gotchas — benchmark-process notes ONLY

> **EXPLICIT CAVEAT:** This section records why the first benchmark pass produced junk and
> what was done to fix the harness. These knobs are NOT our production settings. They are
> Graphify-internal concerns (see section D). Do not treat this section as production config.

### Problem: default context truncation produced garbage output

Bare `ollama run` with the default (~4K) context truncated output mid-stream. Reasoning models
burn their token budget on chain-of-thought first; the model got guillotined mid-output, leaving
malformed YAML.

### Harness fix (benchmark only)

- Called the HTTP API at `/api/generate` instead of `ollama run`.
- Set `num_ctx=8192` (per-request via the API body).
- Set `think=false` to suppress chain-of-thought tokens and keep the output budget for the
  extraction result.
- Added a tolerant YAML parser that retries on soft parse errors and auto-quotes bare scalars
  containing `:`.

These changes produced the results in section A. They explain the benchmark outcome; they do
not define the production path, which Graphify handles internally.

---

## (C) GPU not used during this run — root cause and fix

### Root cause: NVIDIA driver version mismatch

| Item | Value |
|---|---|
| In-kernel module | 580.142 |
| On-disk / userspace driver | 580.159.03 |

The driver was updated on disk on 2026-06-03 but the stale module stayed resident. Result:
`nvidia-smi` fails — "Failed to initialize NVML: Driver/library version mismatch." Because
NVML/CUDA cannot initialize, ollama finds no CUDA device (`total_vram="0 B"`) and runs 100%
on CPU.

Same-machine proof: the ollama log at 07:57 on 2026-06-03 DID find the RTX 3060
(`compute=8.6 ... 12.0 GiB`); the 16:12 service instance the benchmark ran under found no CUDA.

**CPU eval rate observed:** ~10.6 tok/s → ~187–313 s/note. The 9.6 GB model fits the 12 GB
card; VRAM is not the bottleneck — the driver mismatch is.

### Fix (user must run — non-interactive sudo was blocked during this session)

The running kernel already matches the on-disk 580.159.03 module, so a reboot loads the correct
module:

```bash
sudo reboot
# then verify:
nvidia-smi          # should print the RTX 3060 table, no NVML error

# only if CUDA still fails after reboot:
sudo dnf reinstall 'akmod-nvidia*' 'xorg-x11-drv-nvidia*'   # then reboot again
```

### ESTIMATE — post-reboot GPU timing (NOT measured; blocked by sudo this session)

With the RTX 3060 and the 4-bit `gemma4:e4b` model fully resident, ~40–60 tok/s plus faster
prompt processing should drop extraction to roughly single-digit to ~20 s/note — i.e.,
plausibly within a "well under a minute" target.

**This is an estimate to be confirmed post-reboot using the curl command in the appendix below.**
It is not a measured result.

---

## (D) Graphify config surface — production guidance

> This is the only section that contains production configuration guidance. Sections B and C
> are process notes; section A is a model-capability signal only.

### How Graphify talks to Ollama

Graphify talks to Ollama over its **HTTP API** at `http://localhost:11434`, not by shelling out
to `ollama run`. `[github-inferred]` from the env-var/timeout/num_ctx design; the exact endpoint
(`/api/generate` vs `/api/chat`) is `[unverified]` — package internals are not browseable on
GitHub.

Graphify **owns** the prompt template, chunking, context sizing (auto-sized `num_ctx` by
default), and output parsing. `[github]` for the env-var knobs; prompt and parser specifics are
`[github-inferred]` / `[unverified]`. Whether Graphify sets `think=false` and whether it does
tolerant-YAML / retry are `[unverified]`.

Implication: the hand-rolled extraction-spec prompt and raw-ollama settings from section B
likely do **not** apply to Graphify's ollama path. Our levers are the documented external knobs
only.

### Text-vs-code split

Confirmed `[github]` (verbatim README): code is extracted locally via tree-sitter AST with zero
LLM calls; docs/PDFs/images go through the LLM backend (Ollama here). Ollama is invoked **for
text documents only.**

### Model selection

No official Graphify-recommended Ollama model. `[github]` — absence verified at v0.8.30.
Community starting points: `qwen2.5:7b`, `phi4:14b`. `[community]` Default behavior: auto-detect
the running model. Override via `OLLAMA_MODEL`.

Per this benchmark, `gemma4:e4b` is the best installed candidate — but this must be validated
through Graphify's own extraction path, not the hand-rolled benchmark (see Deferred
follow-ups §1 below).

### Environment variables and CLI flags (all `[github]` from v0.8.30 README)

| Knob | Default | Notes |
|---|---|---|
| `OLLAMA_BASE_URL` | `http://localhost:11434` | Ollama endpoint |
| `OLLAMA_MODEL` | auto-detect | Override to pin a model |
| `GRAPHIFY_OLLAMA_NUM_CTX` | auto-sized | Override context window |
| `GRAPHIFY_OLLAMA_KEEP_ALIVE` | unspecified | Minutes to keep model loaded; `0` = unload after each chunk |
| `--token-budget` | — | Chunk size (tokens) |
| `--max-concurrency` | — | Parallel extraction workers |
| `--api-timeout` | 600 s | Increase for slower hardware |

### Known sharp edge

Context saturation across consecutive chunks on the Ollama backend can exhaust VRAM after a
few chunks. `[community, issue #798]` Mitigate by lowering `GRAPHIFY_OLLAMA_NUM_CTX` and/or
setting `GRAPHIFY_OLLAMA_KEEP_ALIVE=0`.

---

## Appendix — Exact commands

### Graphify Ollama path `[github]`

```bash
# Install (package is graphifyy — double-y; command is graphify):
uv tool install "graphifyy[ollama]"

# Pull the candidate model (gemma4:e4b is the local best pick per this benchmark):
ollama pull gemma4:e4b

# Start Ollama if not already running as a service:
ollama serve

# Basic extraction:
graphify extract ./docs --backend ollama

# Constrained single-GPU tuning:
GRAPHIFY_OLLAMA_NUM_CTX=8192 graphify extract ./docs --backend ollama \
  --token-budget 4000 --max-concurrency 2 --api-timeout 900

# Free VRAM between chunks (mitigates issue #798):
GRAPHIFY_OLLAMA_KEEP_ALIVE=0 graphify extract ./docs --backend ollama
```

### Post-reboot GPU re-timing (benchmark verification only — NOT production)

Run this after rebooting to confirm the GPU is now used and to measure actual tok/s:

```bash
F="$HOME/Documents/SecondBrain/2026-03-31-10dlc-isv-setup-guide-oncadence.md"
PROMPT=$(python3 -c "import json;print(json.dumps('Extract entities from this note:\n\n'+open('$F').read()))")
time curl -s http://127.0.0.1:11434/api/generate \
  -d "{\"model\":\"gemma4:e4b\",\"prompt\":$PROMPT,\"stream\":false,\"think\":false,\"options\":{\"num_ctx\":8192}}" \
  | python3 -c "import json,sys;d=json.load(sys.stdin);ec=d['eval_count'];ed=d['eval_duration'];print('eval tok/s:',round(ec/(ed/1e9),2),'| eval_count:',ec,'| total wall (s):',round(d['total_duration']/1e9,2))"
```

This measures raw model tok/s via the hand-rolled harness — it does NOT measure Graphify's
production extraction performance.

#### Measured results (run 2026-06-04, post-reboot)

GPU is now fully active and resident — the driver mismatch diagnosed in section C is
**RESOLVED**. `nvidia-smi` is healthy (Driver 580.159.03, RTX 3060, no NVML error). The
`gemma4:e4b` model loaded fully GPU-resident: `/api/ps` reports `size_vram` 3.29 GB ==
`size` 3.29 GB (no CPU fallback), with 4462 MiB used on the card.

| Pass | eval tok/s | eval_count | total wall (s) | load (s) | prompt_eval (s) | prompt_eval_count |
|---|---|---|---|---|---|---|
| Cold | 74.27 | 701 | 76.04 | 38.15 | 28.44 | 905 |
| Warm | 74.25 | 717 | 10.13 | 0.43 | 0.04 | — |

Conclusion: steady-state generation is **~74 tok/s** — at or above the section C estimate of
40–60 tok/s. Warm extraction lands at **~10 s/note**; the cold pass is ~76 s, dominated by the
one-time 38 s model load plus a 28 s cold prompt-eval. This confirms section C's "well under a
minute once the model is resident" prediction.

> **Caveat retained:** these are raw model tok/s from the hand-rolled curl harness, NOT
> Graphify production extraction performance (see Deferred follow-up §1).

---

## Deferred follow-ups

1. **Definitive validation: run Graphify's own ollama path over the fixtures** — NOT re-scoring
   the hand-rolled benchmark. Do NOT conclude the next step is "score Graphify output against
   the 18 Opus references": that is apples-to-oranges (different prompt/schema/output). The 18
   Opus references validated model capability; they are not a scoring rubric for Graphify's
   graph output.

2. **Resolve `[unverified]` Graphify internals** (HTTP endpoint, think-mode, retry/parsing):
   after `uv tool install "graphifyy[ollama]"`, read the installed client source under the
   tool's site-packages. This does not require the GPU — a cheap deferred follow-up.

3. **Confirm GPU timing post-reboot** — **RESOLVED (2026-06-04).** GPU re-timing ran post-reboot;
   measured ~74 tok/s steady-state (at/above the section C estimate). See the "Measured results"
   table under the Post-reboot GPU re-timing appendix above.

4. **Raw per-model benchmark outputs from this run** were throwaway scratch (wrong harness,
   hand-rolled prompt) and are not committed.