Consolidate local-LLM gut-check findings and update pointers
Add new doc docs/memory-system/benchmark/local-llm-findings-2026-06-04.md consolidating the local-LLM doc-extraction gut-check run. Key findings: gemma4:e4b is the best installed model and adequate for the extraction role (below frontier haiku but acceptable, catching ~45–60% of Opus's high-value entities); GPU went unused due to NVIDIA driver version mismatch (fix = reboot, pending); and Graphify owns the ollama call (HTTP API, prompt/chunking/context/parsing internal) so raw-ollama tuning is not production config. Add progressive-disclosure pointers from 04-build-plan.md and 05-implementation-process.md.
This commit is contained in:
jared
parent
77355b4b44
commit
65595c3995
|
|
@ -108,7 +108,7 @@ re-enters and the final model is chosen. Do not hardcode a model before that run
|
|||
fragments generated in `benchmark/reference-outputs/`. Run as-is (no vault frontmatter
|
||||
modification); verified clean. Fixtures listed in `benchmark/dispatch-prompt.md`.
|
||||
|
||||
Authoritative detail lives in `docs/memory-system/05-implementation-process.md` §2c.
|
||||
Authoritative detail lives in `docs/memory-system/05-implementation-process.md` §2c. Local-model gut-check done (2026-06-04): `gemma4:e4b` is the candidate; GPU fix pending reboot; Graphify owns the ollama call — see `docs/memory-system/benchmark/local-llm-findings-2026-06-04.md`.
|
||||
|
||||
#### 2d — Initial fixture graph build (ADR-013: small-first)
|
||||
Run the initial build against the small fixture set (5–10 notes from Step 1/2c), not the
|
||||
|
|
|
|||
|
|
@ -159,6 +159,7 @@ Verify context allocation after the first extraction call: `ollama ps` shows all
|
|||
**Status (2026-06-04): EXECUTED.** 6 cross-domain fixtures × 3 Claude tiers = 18 reference
|
||||
fragments generated in `benchmark/reference-outputs/`. Run as-is (no vault frontmatter
|
||||
modification); verified clean. Gate is passed — Ollama model scoring is now unblocked.
|
||||
Local-model gut-check also done: `gemma4:e4b` is the candidate; GPU fix pending reboot; Graphify owns the ollama call — see `benchmark/local-llm-findings-2026-06-04.md`.
|
||||
|
||||
Produce a reference set of Graphify-shaped extraction outputs before committing to any local
|
||||
Ollama model. Use the 5–10 fixture notes selected in Step 1a as the input set.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,253 @@
|
|||
# Local-LLM Doc-Extraction Gut-Check — 2026-06-04
|
||||
|
||||
_Last updated: 2026-06-04_ | _Status: gut-check complete; gemma4:e4b is the candidate; GPU fix pending reboot; Graphify validation is the next step._
|
||||
|
||||
This document consolidates the local-LLM doc-extraction gut-check run on 2026-06-04. It is
|
||||
the authoritative findings record for that session and feeds forward into the Graphify-ollama
|
||||
validation step.
|
||||
|
||||
**Four things this document keeps strictly separate:**
|
||||
|
||||
- **(A) Model-capability signal** — what carries forward; the durable conclusion.
|
||||
- **(B) Raw-ollama harness gotchas** — benchmark-process notes only; explicitly NOT production config.
|
||||
- **(C) GPU driver fix** — real diagnosis; timing estimate, not a measured result.
|
||||
- **(D) Graphify config surface** — the actual production guidance.
|
||||
|
||||
Cross-references:
|
||||
- Claude-tier reference run: `docs/memory-system/benchmark/run-findings-2026-06-04.md`
|
||||
- 18 Claude reference fragments: `docs/memory-system/benchmark/reference-outputs/` (`.haiku.md`, `.sonnet.md`, `.opus.md` files only)
|
||||
- Authoritative Graphify-ollama section: `docs/graphify/05-local-models-and-backends.md`
|
||||
|
||||
---
|
||||
|
||||
## TL;DR — What worked
|
||||
|
||||
- The local model class is **good-enough** for the doc-extraction role. Best installed model:
|
||||
**`gemma4:e4b`** (~9.6 GB, fits the RTX 3060's 12 GB). It produced valid schema on 6/6
|
||||
fixtures with clean, well-namespaced facets (0 bad facets out of 6) and note-grounded
|
||||
entities (no contamination).
|
||||
- It catches the high-value entities (~45–60% of Opus's entity recall), missing Opus's long
|
||||
tail — acceptable for the role.
|
||||
- **Critical reframe:** in production we will NOT hand-invoke ollama. Graphify owns the ollama
|
||||
call (prompt, chunking, context window, parsing) over its HTTP API. The hand-rolled benchmark
|
||||
validated *model capability*, not the production pipeline. The raw-ollama tuning discovered
|
||||
during this run is Graphify-internal, not our config surface.
|
||||
|
||||
---
|
||||
|
||||
## (A) Model-capability signal — measured via a hand-rolled prompt/parser (NOT Graphify)
|
||||
|
||||
This signal was gathered through a custom prompt and a custom YAML parser, NOT Graphify's
|
||||
extraction path. It says "the model class is good-enough." It does NOT say "this is what
|
||||
Graphify will output" — Graphify uses a different prompt, schema, and parser.
|
||||
|
||||
### Results across the 3 installed models (all reasoning models; only these 3 are on the machine)
|
||||
|
||||
| Model | Size | Parse rate | Entities/note | Facet quality | Verdict |
|
||||
|---|---|---|---|---|---|
|
||||
| `gemma4:e4b` | ~9.6 GB | 6/6 (2 needed trivial colon-quote repair) | 13–34 (~45–60% of Opus) | 0 bad/6 | **Usable — best pick** |
|
||||
| `gemma4:e2b` | ~7.2 GB | 4/6 (2 hard YAML breaks — dropped `source:` key) | 23–31 when valid | 0 bad/6 | Usable only behind a tolerant parser + retry |
|
||||
| `qwen3.5:2b` | ~2.7 GB | parses but collapses to ~1 entity/note | 1 | n/a | Not usable as tested |
|
||||
|
||||
All three land below haiku (the weakest Claude frontier tier: 19–37 entities, 6/6 clean parse,
|
||||
clean referential integrity). "Below frontier-floor" does not mean "unusable" — `gemma4:e4b`'s
|
||||
misses are the long tail; the high-value entities are present.
|
||||
|
||||
### gemma4:e4b quirks observed
|
||||
|
||||
- Occasional top-level key rename (e.g. `entity_relationships` instead of the spec's key).
|
||||
- A `target: Concept` placeholder leak on the longest note — it echoed the schema's example
|
||||
value rather than a concrete entity. Seen once across 6 fixtures.
|
||||
|
||||
These quirks were caught by the hand-rolled parser and are noted as model-class behavior.
|
||||
Whether Graphify's parser handles them gracefully is a separate question (see section D).
|
||||
|
||||
---
|
||||
|
||||
## (B) Raw-ollama harness gotchas — benchmark-process notes ONLY
|
||||
|
||||
> **EXPLICIT CAVEAT:** This section records why the first benchmark pass produced junk and
|
||||
> what was done to fix the harness. These knobs are NOT our production settings. They are
|
||||
> Graphify-internal concerns (see section D). Do not treat this section as production config.
|
||||
|
||||
### Problem: default context truncation produced garbage output
|
||||
|
||||
Bare `ollama run` with the default (~4K) context truncated output mid-stream. Reasoning models
|
||||
burn their token budget on chain-of-thought first; the model got guillotined mid-output, leaving
|
||||
malformed YAML.
|
||||
|
||||
### Harness fix (benchmark only)
|
||||
|
||||
- Called the HTTP API at `/api/generate` instead of `ollama run`.
|
||||
- Set `num_ctx=8192` (per-request via the API body).
|
||||
- Set `think=false` to suppress chain-of-thought tokens and keep the output budget for the
|
||||
extraction result.
|
||||
- Added a tolerant YAML parser that retries on soft parse errors and auto-quotes bare scalars
|
||||
containing `:`.
|
||||
|
||||
These changes produced the results in section A. They explain the benchmark outcome; they do
|
||||
not define the production path, which Graphify handles internally.
|
||||
|
||||
---
|
||||
|
||||
## (C) GPU not used during this run — root cause and fix
|
||||
|
||||
### Root cause: NVIDIA driver version mismatch
|
||||
|
||||
| Item | Value |
|
||||
|---|---|
|
||||
| In-kernel module | 580.142 |
|
||||
| On-disk / userspace driver | 580.159.03 |
|
||||
|
||||
The driver was updated on disk on 2026-06-03 but the stale module stayed resident. Result:
|
||||
`nvidia-smi` fails — "Failed to initialize NVML: Driver/library version mismatch." Because
|
||||
NVML/CUDA cannot initialize, ollama finds no CUDA device (`total_vram="0 B"`) and runs 100%
|
||||
on CPU.
|
||||
|
||||
Same-machine proof: the ollama log at 07:57 on 2026-06-03 DID find the RTX 3060
|
||||
(`compute=8.6 ... 12.0 GiB`); the 16:12 service instance the benchmark ran under found no CUDA.
|
||||
|
||||
**CPU eval rate observed:** ~10.6 tok/s → ~187–313 s/note. The 9.6 GB model fits the 12 GB
|
||||
card; VRAM is not the bottleneck — the driver mismatch is.
|
||||
|
||||
### Fix (user must run — non-interactive sudo was blocked during this session)
|
||||
|
||||
The running kernel already matches the on-disk 580.159.03 module, so a reboot loads the correct
|
||||
module:
|
||||
|
||||
```bash
|
||||
sudo reboot
|
||||
# then verify:
|
||||
nvidia-smi # should print the RTX 3060 table, no NVML error
|
||||
|
||||
# only if CUDA still fails after reboot:
|
||||
sudo dnf reinstall 'akmod-nvidia*' 'xorg-x11-drv-nvidia*' # then reboot again
|
||||
```
|
||||
|
||||
### ESTIMATE — post-reboot GPU timing (NOT measured; blocked by sudo this session)
|
||||
|
||||
With the RTX 3060 and the 4-bit `gemma4:e4b` model fully resident, ~40–60 tok/s plus faster
|
||||
prompt processing should drop extraction to roughly single-digit to ~20 s/note — i.e.,
|
||||
plausibly within a "well under a minute" target.
|
||||
|
||||
**This is an estimate to be confirmed post-reboot using the curl command in the appendix below.**
|
||||
It is not a measured result.
|
||||
|
||||
---
|
||||
|
||||
## (D) Graphify config surface — production guidance
|
||||
|
||||
> This is the only section that contains production configuration guidance. Sections B and C
|
||||
> are process notes; section A is a model-capability signal only.
|
||||
|
||||
### How Graphify talks to Ollama
|
||||
|
||||
Graphify talks to Ollama over its **HTTP API** at `http://localhost:11434`, not by shelling out
|
||||
to `ollama run`. `[github-inferred]` from the env-var/timeout/num_ctx design; the exact endpoint
|
||||
(`/api/generate` vs `/api/chat`) is `[unverified]` — package internals are not browseable on
|
||||
GitHub.
|
||||
|
||||
Graphify **owns** the prompt template, chunking, context sizing (auto-sized `num_ctx` by
|
||||
default), and output parsing. `[github]` for the env-var knobs; prompt and parser specifics are
|
||||
`[github-inferred]` / `[unverified]`. Whether Graphify sets `think=false` and whether it does
|
||||
tolerant-YAML / retry are `[unverified]`.
|
||||
|
||||
Implication: the hand-rolled extraction-spec prompt and raw-ollama settings from section B
|
||||
likely do **not** apply to Graphify's ollama path. Our levers are the documented external knobs
|
||||
only.
|
||||
|
||||
### Text-vs-code split
|
||||
|
||||
Confirmed `[github]` (verbatim README): code is extracted locally via tree-sitter AST with zero
|
||||
LLM calls; docs/PDFs/images go through the LLM backend (Ollama here). Ollama is invoked **for
|
||||
text documents only.**
|
||||
|
||||
### Model selection
|
||||
|
||||
No official Graphify-recommended Ollama model. `[github]` — absence verified at v0.8.30.
|
||||
Community starting points: `qwen2.5:7b`, `phi4:14b`. `[community]` Default behavior: auto-detect
|
||||
the running model. Override via `OLLAMA_MODEL`.
|
||||
|
||||
Per this benchmark, `gemma4:e4b` is the best installed candidate — but this must be validated
|
||||
through Graphify's own extraction path, not the hand-rolled benchmark (see Deferred
|
||||
follow-ups §1 below).
|
||||
|
||||
### Environment variables and CLI flags (all `[github]` from v0.8.30 README)
|
||||
|
||||
| Knob | Default | Notes |
|
||||
|---|---|---|
|
||||
| `OLLAMA_BASE_URL` | `http://localhost:11434` | Ollama endpoint |
|
||||
| `OLLAMA_MODEL` | auto-detect | Override to pin a model |
|
||||
| `GRAPHIFY_OLLAMA_NUM_CTX` | auto-sized | Override context window |
|
||||
| `GRAPHIFY_OLLAMA_KEEP_ALIVE` | unspecified | Minutes to keep model loaded; `0` = unload after each chunk |
|
||||
| `--token-budget` | — | Chunk size (tokens) |
|
||||
| `--max-concurrency` | — | Parallel extraction workers |
|
||||
| `--api-timeout` | 600 s | Increase for slower hardware |
|
||||
|
||||
### Known sharp edge
|
||||
|
||||
Context saturation across consecutive chunks on the Ollama backend can exhaust VRAM after a
|
||||
few chunks. `[community, issue #798]` Mitigate by lowering `GRAPHIFY_OLLAMA_NUM_CTX` and/or
|
||||
setting `GRAPHIFY_OLLAMA_KEEP_ALIVE=0`.
|
||||
|
||||
---
|
||||
|
||||
## Appendix — Exact commands
|
||||
|
||||
### Graphify Ollama path `[github]`
|
||||
|
||||
```bash
|
||||
# Install (package is graphifyy — double-y; command is graphify):
|
||||
uv tool install "graphifyy[ollama]"
|
||||
|
||||
# Pull the candidate model (gemma4:e4b is the local best pick per this benchmark):
|
||||
ollama pull gemma4:e4b
|
||||
|
||||
# Start Ollama if not already running as a service:
|
||||
ollama serve
|
||||
|
||||
# Basic extraction:
|
||||
graphify extract ./docs --backend ollama
|
||||
|
||||
# Constrained single-GPU tuning:
|
||||
GRAPHIFY_OLLAMA_NUM_CTX=8192 graphify extract ./docs --backend ollama \
|
||||
--token-budget 4000 --max-concurrency 2 --api-timeout 900
|
||||
|
||||
# Free VRAM between chunks (mitigates issue #798):
|
||||
GRAPHIFY_OLLAMA_KEEP_ALIVE=0 graphify extract ./docs --backend ollama
|
||||
```
|
||||
|
||||
### Post-reboot GPU re-timing (benchmark verification only — NOT production)
|
||||
|
||||
Run this after rebooting to confirm the GPU is now used and to measure actual tok/s:
|
||||
|
||||
```bash
|
||||
F="$HOME/Documents/SecondBrain/2026-03-31-10dlc-isv-setup-guide-oncadence.md"
|
||||
PROMPT=$(python3 -c "import json;print(json.dumps('Extract entities from this note:\n\n'+open('$F').read()))")
|
||||
time curl -s http://127.0.0.1:11434/api/generate \
|
||||
-d "{\"model\":\"gemma4:e4b\",\"prompt\":$PROMPT,\"stream\":false,\"think\":false,\"options\":{\"num_ctx\":8192}}" \
|
||||
| python3 -c "import json,sys;d=json.load(sys.stdin);ec=d['eval_count'];ed=d['eval_duration'];print('eval tok/s:',round(ec/(ed/1e9),2),'| eval_count:',ec,'| total wall (s):',round(d['total_duration']/1e9,2))"
|
||||
```
|
||||
|
||||
This measures raw model tok/s via the hand-rolled harness — it does NOT measure Graphify's
|
||||
production extraction performance.
|
||||
|
||||
---
|
||||
|
||||
## Deferred follow-ups
|
||||
|
||||
1. **Definitive validation: run Graphify's own ollama path over the fixtures** — NOT re-scoring
|
||||
the hand-rolled benchmark. Do NOT conclude the next step is "score Graphify output against
|
||||
the 18 Opus references": that is apples-to-oranges (different prompt/schema/output). The 18
|
||||
Opus references validated model capability; they are not a scoring rubric for Graphify's
|
||||
graph output.
|
||||
|
||||
2. **Resolve `[unverified]` Graphify internals** (HTTP endpoint, think-mode, retry/parsing):
|
||||
after `uv tool install "graphifyy[ollama]"`, read the installed client source under the
|
||||
tool's site-packages. This does not require the GPU — a cheap deferred follow-up.
|
||||
|
||||
3. **Confirm GPU timing post-reboot** using the curl command in the appendix above. The
|
||||
~40–60 tok/s estimate in section C is currently unverified.
|
||||
|
||||
4. **Raw per-model benchmark outputs from this run** were throwaway scratch (wrong harness,
|
||||
hand-rolled prompt) and are not committed.
|
||||
Loading…
Reference in New Issue