jared
/
cc-os
1
0
Fork 0
Code Issues 18 Pull Requests Packages Projects Releases Wiki Activity

Consolidate local-LLM gut-check findings and update pointers 

Add new doc docs/memory-system/benchmark/local-llm-findings-2026-06-04.md
consolidating the local-LLM doc-extraction gut-check run. Key findings:
gemma4:e4b is the best installed model and adequate for the extraction
role (below frontier haiku but acceptable, catching ~45–60% of Opus's
high-value entities); GPU went unused due to NVIDIA driver version
mismatch (fix = reboot, pending); and Graphify owns the ollama call
(HTTP API, prompt/chunking/context/parsing internal) so raw-ollama
tuning is not production config. Add progressive-disclosure pointers
from 04-build-plan.md and 05-implementation-process.md.
This commit is contained in:
jared 2026-06-04 14:37:39 -04:00
parent 77355b4b44
commit 65595c3995
3 changed files with 255 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
docs/memory-system/04-build-plan.md
View File

@ -108,7 +108,7 @@ re-enters and the final model is chosen. Do not hardcode a model before that run
fragments generated in `benchmark/reference-outputs/`. Run as-is (no vault frontmatter
modification); verified clean. Fixtures listed in `benchmark/dispatch-prompt.md`.
Authoritative detail lives in `docs/memory-system/05-implementation-process.md` §2c.
Authoritative detail lives in `docs/memory-system/05-implementation-process.md` §2c. Local-model gut-check done (2026-06-04): `gemma4:e4b` is the candidate; GPU fix pending reboot; Graphify owns the ollama call — see `docs/memory-system/benchmark/local-llm-findings-2026-06-04.md`.
#### 2d — Initial fixture graph build (ADR-013: small-first)
Run the initial build against the small fixture set (5–10 notes from Step 1/2c), not the

1
docs/memory-system/05-implementation-process.md
View File

@ -159,6 +159,7 @@ Verify context allocation after the first extraction call: `ollama ps` shows all
**Status (2026-06-04): EXECUTED.** 6 cross-domain fixtures × 3 Claude tiers = 18 reference
fragments generated in `benchmark/reference-outputs/`. Run as-is (no vault frontmatter
modification); verified clean. Gate is passed — Ollama model scoring is now unblocked.
Local-model gut-check also done: `gemma4:e4b` is the candidate; GPU fix pending reboot; Graphify owns the ollama call — see `benchmark/local-llm-findings-2026-06-04.md`.
Produce a reference set of Graphify-shaped extraction outputs before committing to any local
Ollama model. Use the 5–10 fixture notes selected in Step 1a as the input set.

253
docs/memory-system/benchmark/local-llm-findings-2026-06-04.md Normal file
View File

@ -0,0 +1,253 @@
# Local-LLM Doc-Extraction Gut-Check — 2026-06-04
_Last updated: 2026-06-04_ | _Status: gut-check complete; gemma4:e4b is the candidate; GPU fix pending reboot; Graphify validation is the next step._
This document consolidates the local-LLM doc-extraction gut-check run on 2026-06-04. It is
the authoritative findings record for that session and feeds forward into the Graphify-ollama
validation step.
**Four things this document keeps strictly separate:**
- **(A) Model-capability signal** — what carries forward; the durable conclusion.
- **(B) Raw-ollama harness gotchas** — benchmark-process notes only; explicitly NOT production config.
- **(C) GPU driver fix** — real diagnosis; timing estimate, not a measured result.
- **(D) Graphify config surface** — the actual production guidance.
Cross-references:
- Claude-tier reference run: `docs/memory-system/benchmark/run-findings-2026-06-04.md`
- 18 Claude reference fragments: `docs/memory-system/benchmark/reference-outputs/` (`.haiku.md`, `.sonnet.md`, `.opus.md` files only)
- Authoritative Graphify-ollama section: `docs/graphify/05-local-models-and-backends.md`
---
## TL;DR — What worked
- The local model class is **good-enough** for the doc-extraction role. Best installed model:
**`gemma4:e4b`** (~9.6 GB, fits the RTX 3060's 12 GB). It produced valid schema on 6/6
fixtures with clean, well-namespaced facets (0 bad facets out of 6) and note-grounded
entities (no contamination).
- It catches the high-value entities (~45–60% of Opus's entity recall), missing Opus's long
tail — acceptable for the role.
- **Critical reframe:** in production we will NOT hand-invoke ollama. Graphify owns the ollama
call (prompt, chunking, context window, parsing) over its HTTP API. The hand-rolled benchmark
validated *model capability*, not the production pipeline. The raw-ollama tuning discovered
during this run is Graphify-internal, not our config surface.
---
## (A) Model-capability signal — measured via a hand-rolled prompt/parser (NOT Graphify)
This signal was gathered through a custom prompt and a custom YAML parser, NOT Graphify's
extraction path. It says "the model class is good-enough." It does NOT say "this is what
Graphify will output" — Graphify uses a different prompt, schema, and parser.
### Results across the 3 installed models (all reasoning models; only these 3 are on the machine)
| Model | Size | Parse rate | Entities/note | Facet quality | Verdict |
|---|---|---|---|---|---|
| `gemma4:e4b` | ~9.6 GB | 6/6 (2 needed trivial colon-quote repair) | 13–34 (~45–60% of Opus) | 0 bad/6 | **Usable — best pick** |
| `gemma4:e2b` | ~7.2 GB | 4/6 (2 hard YAML breaks — dropped `source:` key) | 23–31 when valid | 0 bad/6 | Usable only behind a tolerant parser + retry |
| `qwen3.5:2b` | ~2.7 GB | parses but collapses to ~1 entity/note | 1 | n/a | Not usable as tested |
All three land below haiku (the weakest Claude frontier tier: 19–37 entities, 6/6 clean parse,
clean referential integrity). "Below frontier-floor" does not mean "unusable" — `gemma4:e4b`'s
misses are the long tail; the high-value entities are present.
### gemma4:e4b quirks observed
- Occasional top-level key rename (e.g. `entity_relationships` instead of the spec's key).
- A `target: Concept` placeholder leak on the longest note — it echoed the schema's example
value rather than a concrete entity. Seen once across 6 fixtures.
These quirks were caught by the hand-rolled parser and are noted as model-class behavior.
Whether Graphify's parser handles them gracefully is a separate question (see section D).
---
## (B) Raw-ollama harness gotchas — benchmark-process notes ONLY
> **EXPLICIT CAVEAT:** This section records why the first benchmark pass produced junk and
> what was done to fix the harness. These knobs are NOT our production settings. They are
> Graphify-internal concerns (see section D). Do not treat this section as production config.
### Problem: default context truncation produced garbage output
Bare `ollama run` with the default (~4K) context truncated output mid-stream. Reasoning models
burn their token budget on chain-of-thought first; the model got guillotined mid-output, leaving
malformed YAML.
### Harness fix (benchmark only)
- Called the HTTP API at `/api/generate` instead of `ollama run`.
- Set `num_ctx=8192` (per-request via the API body).
- Set `think=false` to suppress chain-of-thought tokens and keep the output budget for the
extraction result.
- Added a tolerant YAML parser that retries on soft parse errors and auto-quotes bare scalars
containing `:`.
These changes produced the results in section A. They explain the benchmark outcome; they do
not define the production path, which Graphify handles internally.
---
## (C) GPU not used during this run — root cause and fix
### Root cause: NVIDIA driver version mismatch
| Item | Value |
|---|---|
| In-kernel module | 580.142 |
| On-disk / userspace driver | 580.159.03 |
The driver was updated on disk on 2026-06-03 but the stale module stayed resident. Result:
`nvidia-smi` fails — "Failed to initialize NVML: Driver/library version mismatch." Because
NVML/CUDA cannot initialize, ollama finds no CUDA device (`total_vram="0 B"`) and runs 100%
on CPU.
Same-machine proof: the ollama log at 07:57 on 2026-06-03 DID find the RTX 3060
(`compute=8.6 ... 12.0 GiB`); the 16:12 service instance the benchmark ran under found no CUDA.
**CPU eval rate observed:** ~10.6 tok/s → ~187–313 s/note. The 9.6 GB model fits the 12 GB
card; VRAM is not the bottleneck — the driver mismatch is.
### Fix (user must run — non-interactive sudo was blocked during this session)
The running kernel already matches the on-disk 580.159.03 module, so a reboot loads the correct
module:
```bash
sudo reboot
# then verify:
nvidia-smi # should print the RTX 3060 table, no NVML error
# only if CUDA still fails after reboot:
sudo dnf reinstall 'akmod-nvidia*' 'xorg-x11-drv-nvidia*' # then reboot again
```
### ESTIMATE — post-reboot GPU timing (NOT measured; blocked by sudo this session)
With the RTX 3060 and the 4-bit `gemma4:e4b` model fully resident, ~40–60 tok/s plus faster
prompt processing should drop extraction to roughly single-digit to ~20 s/note — i.e.,
plausibly within a "well under a minute" target.
**This is an estimate to be confirmed post-reboot using the curl command in the appendix below.**
It is not a measured result.
---
## (D) Graphify config surface — production guidance
> This is the only section that contains production configuration guidance. Sections B and C
> are process notes; section A is a model-capability signal only.
### How Graphify talks to Ollama
Graphify talks to Ollama over its **HTTP API** at `http://localhost:11434`, not by shelling out
to `ollama run`. `[github-inferred]` from the env-var/timeout/num_ctx design; the exact endpoint
(`/api/generate` vs `/api/chat`) is `[unverified]` — package internals are not browseable on
GitHub.
Graphify **owns** the prompt template, chunking, context sizing (auto-sized `num_ctx` by
default), and output parsing. `[github]` for the env-var knobs; prompt and parser specifics are
`[github-inferred]` / `[unverified]`. Whether Graphify sets `think=false` and whether it does
tolerant-YAML / retry are `[unverified]`.
Implication: the hand-rolled extraction-spec prompt and raw-ollama settings from section B
likely do **not** apply to Graphify's ollama path. Our levers are the documented external knobs
only.
### Text-vs-code split
Confirmed `[github]` (verbatim README): code is extracted locally via tree-sitter AST with zero
LLM calls; docs/PDFs/images go through the LLM backend (Ollama here). Ollama is invoked **for
text documents only.**
### Model selection
No official Graphify-recommended Ollama model. `[github]` — absence verified at v0.8.30.
Community starting points: `qwen2.5:7b`, `phi4:14b`. `[community]` Default behavior: auto-detect
the running model. Override via `OLLAMA_MODEL`.
Per this benchmark, `gemma4:e4b` is the best installed candidate — but this must be validated
through Graphify's own extraction path, not the hand-rolled benchmark (see Deferred
follow-ups §1 below).
### Environment variables and CLI flags (all `[github]` from v0.8.30 README)
| Knob | Default | Notes |
|---|---|---|
| `OLLAMA_BASE_URL` | `http://localhost:11434` | Ollama endpoint |
| `OLLAMA_MODEL` | auto-detect | Override to pin a model |
| `GRAPHIFY_OLLAMA_NUM_CTX` | auto-sized | Override context window |
| `GRAPHIFY_OLLAMA_KEEP_ALIVE` | unspecified | Minutes to keep model loaded; `0` = unload after each chunk |
| `--token-budget` | — | Chunk size (tokens) |
| `--max-concurrency` | — | Parallel extraction workers |
| `--api-timeout` | 600 s | Increase for slower hardware |
### Known sharp edge
Context saturation across consecutive chunks on the Ollama backend can exhaust VRAM after a
few chunks. `[community, issue #798]` Mitigate by lowering `GRAPHIFY_OLLAMA_NUM_CTX` and/or
setting `GRAPHIFY_OLLAMA_KEEP_ALIVE=0`.
---
## Appendix — Exact commands
### Graphify Ollama path `[github]`
```bash
# Install (package is graphifyy — double-y; command is graphify):
uv tool install "graphifyy[ollama]"
# Pull the candidate model (gemma4:e4b is the local best pick per this benchmark):
ollama pull gemma4:e4b
# Start Ollama if not already running as a service:
ollama serve
# Basic extraction:
graphify extract ./docs --backend ollama
# Constrained single-GPU tuning:
GRAPHIFY_OLLAMA_NUM_CTX=8192 graphify extract ./docs --backend ollama \
--token-budget 4000 --max-concurrency 2 --api-timeout 900
# Free VRAM between chunks (mitigates issue #798):
GRAPHIFY_OLLAMA_KEEP_ALIVE=0 graphify extract ./docs --backend ollama
```
### Post-reboot GPU re-timing (benchmark verification only — NOT production)
Run this after rebooting to confirm the GPU is now used and to measure actual tok/s:
```bash
F="$HOME/Documents/SecondBrain/2026-03-31-10dlc-isv-setup-guide-oncadence.md"
PROMPT=$(python3 -c "import json;print(json.dumps('Extract entities from this note:\n\n'+open('$F').read()))")
time curl -s http://127.0.0.1:11434/api/generate \
-d "{\"model\":\"gemma4:e4b\",\"prompt\":$PROMPT,\"stream\":false,\"think\":false,\"options\":{\"num_ctx\":8192}}" \
| python3 -c "import json,sys;d=json.load(sys.stdin);ec=d['eval_count'];ed=d['eval_duration'];print('eval tok/s:',round(ec/(ed/1e9),2),'| eval_count:',ec,'| total wall (s):',round(d['total_duration']/1e9,2))"
```
This measures raw model tok/s via the hand-rolled harness — it does NOT measure Graphify's
production extraction performance.
---
## Deferred follow-ups
1. **Definitive validation: run Graphify's own ollama path over the fixtures** — NOT re-scoring
the hand-rolled benchmark. Do NOT conclude the next step is "score Graphify output against
the 18 Opus references": that is apples-to-oranges (different prompt/schema/output). The 18
Opus references validated model capability; they are not a scoring rubric for Graphify's
graph output.
2. **Resolve `[unverified]` Graphify internals** (HTTP endpoint, think-mode, retry/parsing):
after `uv tool install "graphifyy[ollama]"`, read the installed client source under the
tool's site-packages. This does not require the GPU — a cheap deferred follow-up.
3. **Confirm GPU timing post-reboot** using the curl command in the appendix above. The
~40–60 tok/s estimate in section C is currently unverified.
4. **Raw per-model benchmark outputs from this run** were throwaway scratch (wrong harness,
hand-rolled prompt) and are not committed.