14 KiB

Raw Blame History

05 — Local Models & Backends

How Graphify decides what (if anything) talks to an LLM, the full menu of backends, and how to run document extraction fully local with Ollama. The short version: code never needs a model; only documents do — so backend choice is purely a documents/knowledge-base question.

The one rule that drives everything: code is free, docs cost a model

Graphify splits its inputs in two:

Code files — processed locally via tree-sitter (AST). No API calls, nothing leaves your machine. [github] (README: "Code is extracted locally with no API calls (AST via tree-sitter)" and "Code files — processed locally via tree-sitter. Nothing leaves your machine.") This is also the creator's repeated point in the interview: "The a call is free of cost. There's no API call included for an LLM... You have to call the LLM back end only for your documents." [interview]
Docs, PDFs, images — sent to a model for semantic extraction. [github] (README: "Docs, PDFs, images — sent to your AI assistant for semantic extraction").

So if you only ingest code, you never pick a backend and never spend a token on extraction. Backend selection below matters only for documents and knowledge bases.

SLM vs LLM — for a non-expert

The interview leans hard on SLMs (small language models) as the local-first future. [interview] In plain terms:

An LLM (large language model — e.g. Claude, GPT, Gemini) is huge. It generally runs in the cloud because it needs a lot of memory / GPU to hold.
An SLM is a smaller model (a few billion parameters) that fits in ordinary RAM, or in a consumer GPU's VRAM, and runs on your own machine. Because it runs locally, your files never go to a cloud provider. [interview]

The creator's framing: "large language models do not fit in smaller RAM size. So you need larger RAM or ... GPUs ... to fit them in. So rather than ... LLMs, you can just go for SLMs ... so that their data isn't shared to cloud-based LLMs." [interview] He describes Graphify's direction as a "local-first AI memory system" where "none of your file will be shared with cloud-based LLMs." [interview]

Reality check: that "as good as a frontier model" claim is the creator's aspiration, not a measured fact. [unverified claim] For document extraction (chunk → entities/relationships), a competent local instruction-following model is usually good enough; for the highest-quality graphs on messy corpora, a cloud model still tends to win. Treat local as the privacy/cost choice, cloud as the max-quality choice.

All set per run with --backend <name> on graphify extract. [github] (README backend list, verbatim: "gemini, kimi, claude, openai, deepseek, ollama, bedrock, or claude-cli".) Each non-local backend reads its credential from an env var:

Backend	Flag	Auth	Where data goes
Ollama (local)	`--backend ollama`	none for loopback	Your machine only
AWS Bedrock	`--backend bedrock`	IAM via standard AWS credential chain (`AWS_` / `~/.aws/credentials`) — no API key*	AWS account
Claude (API)	`--backend claude`	`ANTHROPIC_API_KEY`	Anthropic
Claude Code CLI	`--backend claude-cli`	none — uses your Claude subscription	Anthropic
Gemini	`--backend gemini`	`GEMINI_API_KEY` or `GOOGLE_API_KEY`	Google
OpenAI / compatible	`--backend openai`	`OPENAI_API_KEY`	OpenAI (or your compatible endpoint)
DeepSeek	`--backend deepseek`	`DEEPSEEK_API_KEY`	DeepSeek
Kimi (Moonshot)	`--backend kimi`	`MOONSHOT_API_KEY`	Moonshot AI servers in China `[github]`

All flag values, env-var names, and auth notes above are quoted from the v8 README env-var table and backend list. [github]

In-platform vs headless. When you run Graphify inside a coding assistant via the /graphify skill, document extraction uses whatever model your IDE session already runs (Claude / Gemini / etc.) — no separate key needed. [github] (README: "using whatever model your IDE session runs".) Headless graphify extract is where you must supply a backend + credential.

Auto-detect priority. If you don't pass --backend, headless graphify extract picks one based on which key is set, in this order: Gemini → Kimi → Claude → OpenAI → DeepSeek → Bedrock → Ollama. [github] Pass --backend explicitly if you care which one runs (you usually do, for privacy).

How to choose

Privacy / offline / zero per-token cost → --backend ollama. Nothing leaves the machine. The interview's recommended default for documents when you want to "save cost." [interview]
Enterprise cloud, no API keys to manage → --backend bedrock (uses your existing IAM). [github] [interview]
Already paying for a coding-assistant subscription → in-platform /graphify, or --backend claude-cli to reuse that subscription with no extra key. [github]
Max extraction quality, simplest setup → a frontier cloud backend (claude / gemini / openai) with the matching key.

Setting up Ollama (runnable)

1. Install Ollama and pull a model

Ollama is a separate program that runs models locally and exposes an HTTP API on http://localhost:11434. Install it, then pull an instruction-following model:

# Install Ollama: see https://ollama.com/download
# Then pull a model that fits your RAM/VRAM (see "Which model?" below):
ollama pull qwen2.5:7b      # example only — pick one that fits your hardware
ollama serve                # if not already running as a service

2. Install Graphify with the Ollama extra

uv tool install "graphifyy[ollama]"

Note the package name is graphifyy (double-y) and the optional extra is [ollama]. [github] (README install table.)

3. Run document extraction against Ollama

# Local Ollama — no API key needed for loopback:
graphify extract ./docs --backend ollama

That's it for code, too — but remember, code uses AST regardless, so the backend only kicks in for the documents in ./docs.

Ollama environment variables

All four below are confirmed verbatim in the v8 README env-var table. [github]

Env var	Purpose	Default
`OLLAMA_BASE_URL`	Ollama server URL	`http://localhost:11434`
`OLLAMA_MODEL`	Model name to use	auto-detect
`GRAPHIFY_OLLAMA_NUM_CTX`	Override Ollama KV-cache (context) window size	auto-sized
`GRAPHIFY_OLLAMA_KEEP_ALIVE`	Minutes to keep the model loaded in memory; set `0` to unload after each chunk	(model stays loaded)

# Point at a remote Ollama box and pin a specific model:
OLLAMA_BASE_URL=http://192.168.1.50:11434 \
OLLAMA_MODEL=qwen2.5:7b \
graphify extract ./docs --backend ollama

Tuning `GRAPHIFY_OLLAMA_NUM_CTX` and `KEEP_ALIVE`

These two are the knobs the README's own troubleshooting section reaches for when "Ollama runs out of VRAM / context window exceeded." [github]

GRAPHIFY_OLLAMA_NUM_CTX — the KV-cache window. Auto-sized by default; override it down if you hit VRAM limits, or up if your chunks are large. Both directions appear verbatim in the README:

# Shrink the context window to survive a small GPU (from the VRAM troubleshooting section):
GRAPHIFY_OLLAMA_NUM_CTX=8192 graphify extract ./docs --backend ollama --token-budget 4000

# Or raise it for big chunks on a big GPU:
GRAPHIFY_OLLAMA_NUM_CTX=32768 graphify extract ./docs --backend ollama

GRAPHIFY_OLLAMA_KEEP_ALIVE=0 — unload the model between chunks. Frees VRAM on small GPUs at the cost of reload time per chunk:

GRAPHIFY_OLLAMA_KEEP_ALIVE=0 graphify extract ./docs --backend ollama   # saves VRAM on small GPUs

Slow local models timing out? Raise the HTTP timeout (default 600s) — README confirms both the env var GRAPHIFY_API_TIMEOUT and the --api-timeout flag:

graphify extract ./docs --backend ollama --api-timeout 900   # 15-minute timeout

Known sharp edge: community bug reports flag context-window saturation across consecutive chunks on the Ollama backend (the model's session not resetting between chunks, exhausting VRAM after a few chunks). If you see degrading or "hollow" responses partway through a large run, lower GRAPHIFY_OLLAMA_NUM_CTX and/or set GRAPHIFY_OLLAMA_KEEP_ALIVE=0. [community](https://github.com/safishamsi/graphify/issues/798)

Which Ollama model?

There is no official Graphify-recommended/tested model list in the README or on the site. Neither the v8 README nor the site publishes one (verified — see Open questions). So follow the rule the task itself sets and the interview implies: pick an instruction-following model that fits your RAM/VRAM. [interview] Document extraction is "read this chunk, emit structured entities/relationships," which is an instruction-following + structured-output job, not deep reasoning.

Correction — there IS a shipped code default (2026-06-05). Although the README/site publish no model recommendation, installed graphify 0.8.31 hardcodes a fallback in llm.py:67: "default_model": os.environ.get("OLLAMA_MODEL", "qwen2.5-coder:7b"). [github] So if you run graphify extract --backend ollama without setting OLLAMA_MODEL, the binary silently uses qwen2.5-coder:7b. The README's silence on a recommended model is accurate as a documentation statement; it is not accurate as a claim that "no default exists." The choice of a coder model for document extraction is a revealed preference — the binding requirement is structured-JSON instruction-following discipline, not domain NER. See 10-extraction-model-options.md for the full analysis.

Rough sizing rule of thumb: a model needs roughly its parameter count in GB of (V)RAM at common 4-bit quantization (a 7B model ≈ ~5 GB; a 30B+ model wants 24 GB+ VRAM). Leave headroom for the context window.

Community signals (not Graphify-verified):

A user feature-request reports running gemma3:27b-class models on a 24 GB+ VRAM GPU with custom num_ctx/concurrency settings for high GPU saturation. Treat the specific tag and numbers as one user's setup, not a spec. [community](https://github.com/safishamsi/graphify/issues/792)
General Ollama guidance points to Qwen2.5 (e.g. qwen2.5:7b) and Phi-4 (phi4:14b) as strong instruction-following / structured-output models in their size classes. Good starting points to test, not Graphify endorsements. [community](https://ollama.com/library)

Start with the largest instruction-following model that comfortably fits your hardware, run a small extract, then check the god nodes to judge whether the graph quality is acceptable before committing to a big run.

Controlling spend & local load: `--token-budget` and `--max-concurrency`

These two govern how document chunks are sent to whatever backend you chose. Both are confirmed in the v8 README. [github]

--token-budget — size of each semantic chunk sent to the model. Smaller budget = smaller chunks, which is the recommended setting for local/small models (less context per call, fits small windows). [github] (README: "smaller semantic chunks for local/small models".)
--max-concurrency — number of parallel LLM calls. Lower it for local inference so you don't overwhelm a single GPU/CPU. [github] (README: "fewer parallel LLM calls (useful for local inference)".)

# Gentle on a single local GPU: small chunks, low concurrency, generous timeout
GRAPHIFY_OLLAMA_NUM_CTX=8192 \
graphify extract ./docs \
  --backend ollama \
  --token-budget 4000 \
  --max-concurrency 2 \
  --api-timeout 900

For cloud backends, the same two flags are your cost dials: smaller --token-budget and the graphify update incremental flow keep token spend down. The interview's repeated cost advice: extract code with AST (free), only spend model tokens on documents, and always graphify update rather than re-ingesting from scratch. [interview]

Quick reference

# Fully local document extraction (privacy / zero per-token cost)
graphify extract ./docs --backend ollama

# Enterprise cloud via existing AWS IAM (no API key)
graphify extract ./docs --backend bedrock

# Reuse your Claude subscription, no extra key
graphify extract ./docs --backend claude-cli

# Explicit cloud key
ANTHROPIC_API_KEY=sk-... graphify extract ./docs --backend claude

# Code only → no backend needed at all (AST is free)
graphify extract ./src

Open questions / unverified

No official Ollama model recommendation exists in the README, but the installed binary hardcodes qwen2.5-coder:7b as the Ollama fallback (llm.py:67) — see the Correction callout above and 10-extraction-model-options.md. [github]
GRAPHIFY_OLLAMA_KEEP_ALIVE default value is documented by behavior ("minutes to keep loaded; 0 to unload after each chunk") but the README does not state the numeric default. [github]
gemma3:27b exact tag / VRAM numbers come from a single user issue (#792), not Graphify docs. The original ASR transcript said "Gemma 4 31b," which does not match a shipped tag; treat the community gemma3:27b-class figure as illustrative, not authoritative. [community](https://github.com/safishamsi/graphify/issues/792)
The "SLMs as good as frontier models soon" vision is the creator's aspiration, not a benchmark. [unverified claim] / [interview]
The 70x/90x token-savings headline numbers are corpus-dependent sales claims (the creator himself says "there is no ceiling or floor"); not used as a basis for any guidance here. [unverified claim] / [interview]
Could not confirm any graphify.yaml config-file equivalents for these knobs against the raw README; the graphify.yaml keys mentioned in issue #792 are a feature request, not shipped config. [community](https://github.com/safishamsi/graphify/issues/792)

14 KiB Raw Blame History