cc-os/docs/graphify/05-local-models-and-backends.md

# 05 — Local Models & Backends

How Graphify decides what (if anything) talks to an LLM, the full menu of backends, and how to run document extraction fully local with Ollama. The short version: **code never needs a model; only documents do** — so backend choice is purely a documents/knowledge-base question.

See also: [03 — Ingesting code (AST)](03-ingesting-code-ast.md) · [04 — Ingesting docs & knowledge](04-ingesting-docs-knowledge.md) · [07 — Token economics & updates](07-token-economics-and-updates.md).

---

## The one rule that drives everything: code is free, docs cost a model

Graphify splits its inputs in two:

- **Code files — processed locally via tree-sitter (AST). No API calls, nothing leaves your machine.** `[github]` (README: *"Code is extracted locally with no API calls (AST via tree-sitter)"* and *"Code files — processed locally via tree-sitter. Nothing leaves your machine."*) This is also the creator's repeated point in the interview: *"The a call is free of cost. There's no API call included for an LLM... You have to call the LLM back end only for your documents."* `[interview]`
- **Docs, PDFs, images — sent to a model for semantic extraction.** `[github]` (README: *"Docs, PDFs, images — sent to your AI assistant for semantic extraction"*).

So if you only ingest code, **you never pick a backend and never spend a token on extraction.** Backend selection below matters *only* for documents and knowledge bases.

---

## SLM vs LLM — for a non-expert

The interview leans hard on **SLMs (small language models)** as the local-first future. `[interview]` In plain terms:

- An **LLM** (large language model — e.g. Claude, GPT, Gemini) is huge. It generally runs in the cloud because it needs a lot of memory / GPU to hold.
- An **SLM** is a smaller model (a few billion parameters) that **fits in ordinary RAM, or in a consumer GPU's VRAM, and runs on your own machine.** Because it runs locally, **your files never go to a cloud provider.** `[interview]`

The creator's framing: *"large language models do not fit in smaller RAM size. So you need larger RAM or ... GPUs ... to fit them in. So rather than ... LLMs, you can just go for SLMs ... so that their data isn't shared to cloud-based LLMs."* `[interview]` He describes Graphify's direction as a *"local-first AI memory system"* where *"none of your file will be shared with cloud-based LLMs."* `[interview]`

Reality check: that "as good as a frontier model" claim is the creator's **aspiration**, not a measured fact. `[unverified claim]` For document extraction (chunk → entities/relationships), a competent local instruction-following model is usually *good enough*; for the highest-quality graphs on messy corpora, a cloud model still tends to win. Treat local as the **privacy/cost** choice, cloud as the **max-quality** choice.

---

## The backend menu

All set per run with `--backend <name>` on `graphify extract`. `[github]` (README backend list, verbatim: *"gemini, kimi, claude, openai, deepseek, ollama, bedrock, or claude-cli"*.) Each non-local backend reads its credential from an env var:

| Backend | Flag | Auth | Where data goes |
|---|---|---|---|
| **Ollama (local)** | `--backend ollama` | none for loopback | **Your machine only** |
| **AWS Bedrock** | `--backend bedrock` | IAM via standard AWS credential chain (`AWS_*` / `~/.aws/credentials`) — **no API key** | AWS account |
| **Claude (API)** | `--backend claude` | `ANTHROPIC_API_KEY` | Anthropic |
| **Claude Code CLI** | `--backend claude-cli` | none — uses your Claude subscription | Anthropic |
| **Gemini** | `--backend gemini` | `GEMINI_API_KEY` or `GOOGLE_API_KEY` | Google |
| **OpenAI / compatible** | `--backend openai` | `OPENAI_API_KEY` | OpenAI (or your compatible endpoint) |
| **DeepSeek** | `--backend deepseek` | `DEEPSEEK_API_KEY` | DeepSeek |
| **Kimi (Moonshot)** | `--backend kimi` | `MOONSHOT_API_KEY` | **Moonshot AI servers in China** `[github]` |

All flag values, env-var names, and auth notes above are quoted from the v8 README env-var table and backend list. `[github]`

**In-platform vs headless.** When you run Graphify *inside* a coding assistant via the `/graphify` skill, document extraction uses **whatever model your IDE session already runs** (Claude / Gemini / etc.) — no separate key needed. `[github]` (README: *"using whatever model your IDE session runs"*.) Headless `graphify extract` is where you must supply a backend + credential.

**Auto-detect priority.** If you don't pass `--backend`, headless `graphify extract` picks one based on which key is set, in this order: **Gemini → Kimi → Claude → OpenAI → DeepSeek → Bedrock → Ollama.** `[github]` Pass `--backend` explicitly if you care which one runs (you usually do, for privacy).

### How to choose

- **Privacy / offline / zero per-token cost → `--backend ollama`.** Nothing leaves the machine. The interview's recommended default for documents when you want to *"save cost."* `[interview]`
- **Enterprise cloud, no API keys to manage → `--backend bedrock`** (uses your existing IAM). `[github]` `[interview]`
- **Already paying for a coding-assistant subscription → in-platform `/graphify`, or `--backend claude-cli`** to reuse that subscription with no extra key. `[github]`
- **Max extraction quality, simplest setup → a frontier cloud backend** (`claude` / `gemini` / `openai`) with the matching key.

---

## Setting up Ollama (runnable)

### 1. Install Ollama and pull a model

[Ollama](https://ollama.com) is a separate program that runs models locally and exposes an HTTP API on `http://localhost:11434`. Install it, then pull an instruction-following model:

```bash
# Install Ollama: see https://ollama.com/download
# Then pull a model that fits your RAM/VRAM (see "Which model?" below):
ollama pull qwen2.5:7b      # example only — pick one that fits your hardware
ollama serve                # if not already running as a service
```

### 2. Install Graphify with the Ollama extra

```bash
uv tool install "graphifyy[ollama]"
```

> Note the package name is **`graphifyy`** (double-y) and the optional extra is `[ollama]`. `[github]` (README install table.)

### 3. Run document extraction against Ollama

```bash
# Local Ollama — no API key needed for loopback:
graphify extract ./docs --backend ollama
```

That's it for code, too — but remember, **code uses AST regardless**, so the backend only kicks in for the documents in `./docs`.

### Ollama environment variables

All four below are confirmed verbatim in the v8 README env-var table. `[github]`

| Env var | Purpose | Default |
|---|---|---|
| `OLLAMA_BASE_URL` | Ollama server URL | `http://localhost:11434` |
| `OLLAMA_MODEL` | Model name to use | auto-detect |
| `GRAPHIFY_OLLAMA_NUM_CTX` | Override Ollama KV-cache (context) window size | auto-sized |
| `GRAPHIFY_OLLAMA_KEEP_ALIVE` | Minutes to keep the model loaded in memory; **set `0` to unload after each chunk** | (model stays loaded) |

```bash
# Point at a remote Ollama box and pin a specific model:
OLLAMA_BASE_URL=http://192.168.1.50:11434 \
OLLAMA_MODEL=qwen2.5:7b \
graphify extract ./docs --backend ollama
```

### Tuning `GRAPHIFY_OLLAMA_NUM_CTX` and `KEEP_ALIVE`

These two are the knobs the README's own troubleshooting section reaches for when *"Ollama runs out of VRAM / context window exceeded."* `[github]`

**`GRAPHIFY_OLLAMA_NUM_CTX`** — the KV-cache window. Auto-sized by default; override it down if you hit VRAM limits, or up if your chunks are large. Both directions appear verbatim in the README:

```bash
# Shrink the context window to survive a small GPU (from the VRAM troubleshooting section):
GRAPHIFY_OLLAMA_NUM_CTX=8192 graphify extract ./docs --backend ollama --token-budget 4000

# Or raise it for big chunks on a big GPU:
GRAPHIFY_OLLAMA_NUM_CTX=32768 graphify extract ./docs --backend ollama
```

**`GRAPHIFY_OLLAMA_KEEP_ALIVE=0`** — unload the model between chunks. Frees VRAM on small GPUs at the cost of reload time per chunk:

```bash
GRAPHIFY_OLLAMA_KEEP_ALIVE=0 graphify extract ./docs --backend ollama   # saves VRAM on small GPUs
```

**Slow local models timing out?** Raise the HTTP timeout (default 600s) — README confirms both the env var `GRAPHIFY_API_TIMEOUT` and the `--api-timeout` flag:

```bash
graphify extract ./docs --backend ollama --api-timeout 900   # 15-minute timeout
```

> **Known sharp edge:** community bug reports flag context-window saturation across consecutive chunks on the Ollama backend (the model's session not resetting between chunks, exhausting VRAM after a few chunks). If you see degrading or "hollow" responses partway through a large run, lower `GRAPHIFY_OLLAMA_NUM_CTX` and/or set `GRAPHIFY_OLLAMA_KEEP_ALIVE=0`. `[community](https://github.com/safishamsi/graphify/issues/798)`

### Which Ollama model?

**There is no official Graphify-recommended/tested model list.** Neither the v8 README nor the site publishes one (verified — see Open questions). So follow the rule the task itself sets and the interview implies: **pick an instruction-following model that fits your RAM/VRAM.** `[interview]` Document extraction is "read this chunk, emit structured entities/relationships," which is an instruction-following + structured-output job, not deep reasoning.

Rough sizing rule of thumb: a model needs roughly its parameter count in **GB of (V)RAM** at common 4-bit quantization (a 7B model ≈ ~5 GB; a 30B+ model wants 24 GB+ VRAM). Leave headroom for the context window.

Community signals (not Graphify-verified):

- A user feature-request reports running **`gemma3:27b`-class models on a 24 GB+ VRAM GPU** with custom `num_ctx`/concurrency settings for high GPU saturation. Treat the specific tag and numbers as one user's setup, not a spec. `[community](https://github.com/safishamsi/graphify/issues/792)`
- General Ollama guidance points to **Qwen2.5 (e.g. `qwen2.5:7b`)** and **Phi-4 (`phi4:14b`)** as strong instruction-following / structured-output models in their size classes. Good *starting points* to test, not Graphify endorsements. `[community](https://ollama.com/library)`

Start with the largest instruction-following model that comfortably fits your hardware, run a small `extract`, then check the [god nodes](06-querying-and-god-nodes.md) to judge whether the graph quality is acceptable before committing to a big run.

---

## Controlling spend & local load: `--token-budget` and `--max-concurrency`

These two govern *how* document chunks are sent to whatever backend you chose. Both are confirmed in the v8 README. `[github]`

- **`--token-budget`** — size of each semantic chunk sent to the model. **Smaller budget = smaller chunks**, which is the recommended setting for local/small models (less context per call, fits small windows). `[github]` (README: *"smaller semantic chunks for local/small models"*.)
- **`--max-concurrency`** — number of parallel LLM calls. **Lower it for local inference** so you don't overwhelm a single GPU/CPU. `[github]` (README: *"fewer parallel LLM calls (useful for local inference)"*.)

```bash
# Gentle on a single local GPU: small chunks, low concurrency, generous timeout
GRAPHIFY_OLLAMA_NUM_CTX=8192 \
graphify extract ./docs \
  --backend ollama \
  --token-budget 4000 \
  --max-concurrency 2 \
  --api-timeout 900
```

For **cloud** backends, the same two flags are your cost dials: smaller `--token-budget` and the [`graphify update`](07-token-economics-and-updates.md) incremental flow keep token spend down. The interview's repeated cost advice: extract **code with AST (free)**, only spend model tokens on **documents**, and always **`graphify update`** rather than re-ingesting from scratch. `[interview]`

---

## Quick reference

```bash
# Fully local document extraction (privacy / zero per-token cost)
graphify extract ./docs --backend ollama

# Enterprise cloud via existing AWS IAM (no API key)
graphify extract ./docs --backend bedrock

# Reuse your Claude subscription, no extra key
graphify extract ./docs --backend claude-cli

# Explicit cloud key
ANTHROPIC_API_KEY=sk-... graphify extract ./docs --backend claude

# Code only → no backend needed at all (AST is free)
graphify extract ./src
```

---

## Open questions / unverified

- **No official Ollama model recommendation exists.** Verified absent from the v8 README env/backend sections and not surfaced on the (403-gated) official site. The model names here are community/general-Ollama signals, tagged as such — not Graphify-tested defaults.
- **`GRAPHIFY_OLLAMA_KEEP_ALIVE` default value** is documented by *behavior* ("minutes to keep loaded; `0` to unload after each chunk") but the README does not state the numeric default. `[github]`
- **`gemma3:27b` exact tag / VRAM numbers** come from a single user issue (#792), not Graphify docs. The original ASR transcript said "Gemma 4 31b," which does not match a shipped tag; treat the community `gemma3:27b`-class figure as illustrative, not authoritative. `[community](https://github.com/safishamsi/graphify/issues/792)`
- **The "SLMs as good as frontier models soon" vision** is the creator's aspiration, not a benchmark. `[unverified claim]` / `[interview]`
- **The 70x/90x token-savings headline numbers** are corpus-dependent sales claims (the creator himself says *"there is no ceiling or floor"*); not used as a basis for any guidance here. `[unverified claim]` / `[interview]`
- Could not confirm any `graphify.yaml` config-file equivalents for these knobs against the raw README; the `graphify.yaml` keys mentioned in issue #792 are a **feature request**, not shipped config. `[community](https://github.com/safishamsi/graphify/issues/792)`