11 KiB

Raw Blame History

10 — Extraction Model Options: Why a General Structured-Output LLM, and Is a Purpose-Built KG Model Worth It?

Last updated: 2026-06-05 | Status: reference / decision record

This document explains why Graphify's document extraction uses a general instruction-following / structured-output LLM rather than a purpose-built knowledge-graph extraction model (e.g. Triplex, GLiNER, REBEL), and honestly assesses whether the purpose-built route is worth experimenting with for this project.

The architecture constraint (this is the crux)

Graphify owns both the prompt and the output contract. It is not a thin wrapper you can swap a model into — the extraction pipeline is opinionated end-to-end:

Fixed system prompt. For document/markdown extraction, Graphify sends its own system prompt _EXTRACTION_SYSTEM (llm.py:218-232). [github] There is no user hook to supply a custom extraction prompt; the prompt is hardcoded.

Fixed output schema. The model must return exactly this JSON structure — no approximation accepted:

{
  "nodes": [{"id": "stem_entity", "label": "...", "file_type": "code|document|paper|image|rationale|concept",
             "source_file": "...", "source_location": null, "source_url": null,
             "captured_at": null, "author": null, "contributor": null}],
  "edges": [{"source": "node_id", "target": "node_id",
             "relation": "calls|implements|references|cites|conceptually_related_to|shares_data_with|semantically_similar_to",
             "confidence": "EXTRACTED|INFERRED|AMBIGUOUS",
             "confidence_score": 1.0, "source_file": "...", "source_location": null, "weight": 1.0}],
  "hyperedges": [], "input_tokens": 0, "output_tokens": 0
}

The relation field is constrained to a fixed vocabulary of seven values. The confidence field is a three-value enum (EXTRACTED/INFERRED/AMBIGUOUS). [github] (llm.py:218-232)

No JSON-mode / grammar enforcement. For the Ollama backend, Graphify passes only {"options": {"num_ctx": ...}, "keep_alive": ...} as extra parameters (llm.py:506). [github] There is no response_format, grammar constraint, or structured-output enforcement. Schema compliance rests entirely on the model's instruction-following ability.

Lenient but not magic parser. The response goes through _parse_llm_json (llm.py:269-303), which strips optional markdown fences and then tries json.loads. If that fails, it scans for the first balanced JSON object in the text. A model that emits its own schema — triples, spans, BIO tags — will produce an empty or nonsense graph, not a graceful fallback. [github]

Temperature forced to 0 for Ollama. The Ollama backend config sets "temperature": 0 (llm.py:65-73). [github]

Deep mode. --mode deep appends _DEEP_EXTRACTION_SUFFIX to the same prompt, asking for extra INFERRED edges with a more conservative framing (llm.py:234-247). [github] The schema stays identical.

The shipped default: why a coder model?

The Ollama fallback default is qwen2.5-coder:7b (llm.py:67): [github]

"default_model": os.environ.get("OLLAMA_MODEL", "qwen2.5-coder:7b"),

Even though Graphify sends documents (not code) for extraction, the maintainer chose a coder/structured-output-tuned model. This is a revealed preference: the binding requirement is structured-JSON instruction-following discipline, not domain entity recognition. A coder/instruct model is the right axis; a NER specialist is the wrong one; frontier reasoning is overkill.

Config knobs for the Ollama backend (all [github] from README env-var table unless noted):

Knob	How to set	What it does
`OLLAMA_MODEL`	env var	Override the `qwen2.5-coder:7b` default
`OLLAMA_BASE_URL`	env var	Must end in `/v1` (e.g. `http://127.0.0.1:11434/v1`) — any other suffix produces 404s on every call
`GRAPHIFY_OLLAMA_NUM_CTX`	env var	See GOTCHA below
`GRAPHIFY_OLLAMA_KEEP_ALIVE`	env var	Minutes to keep model resident; default `30m`; set `0` to unload after each chunk
`--token-budget`	CLI flag	Per-chunk input cap (tokens); pack multiple files per chunk
`--max-concurrency`	CLI flag	Set 1–2 for a single local GPU
`--mode deep`	CLI flag	Appends deep-inference suffix; elicits more INFERRED edges

GOTCHA — num_ctx does not propagate through graphify (verified 2026-06-04 on this project). [github] Graphify posts to Ollama's OpenAI-compatible /v1/chat/completions endpoint. That endpoint silently ignores the per-request options.num_ctx that Graphify sends (llm.py:506). Proven by A/B: a POST to /v1/chat/completions with num_ctx=8192 left ollama ps showing CONTEXT=4096; the same value through the native /api/chat endpoint was honoured. Therefore GRAPHIFY_OLLAMA_NUM_CTX has no effect through Graphify — context pins at Ollama's /v1 default of 4096. At 4096 tokens, Graphify's extraction output JSON is truncated mid-response (finish_reason=length) and the chunk is discarded, producing an empty graph.

Workaround (validated): bake context into a Modelfile variant:

# Create a 16 k-context variant of qwen2.5-coder:7b:
cat <<'EOF' | ollama create qwen25-coder-7b-16k -f -
FROM qwen2.5-coder:7b
PARAMETER num_ctx 16384
EOF
OLLAMA_MODEL=qwen25-coder-7b-16k graphify extract ./docs --backend ollama

The /v1 endpoint honours the model's baked-in default. Non-invasive: no sudo, no systemd restart; reversible with ollama rm. Full investigation: docs/memory-system/benchmark/scoring-results-2026-06-04.md.

Local patch — thinking mode disabled (applied 2026-06-04): reasoning_effort: "none" was added at llm.py:71 in the installed binary to suppress thinking output for any thinking-capable model. The original is backed up at /tmp/graphify-bench/llm.py.orig. This patch is a no-op for qwen2.5-coder:7b (no thinking mode) but will be lost on pip install --upgrade and would need re-applying for any future thinking-capable candidate. Needs a production path: upstream PR or a maintained local wrapper. [github]

Purpose-built KG / entity / relation models — and why they don't drop in

Every purpose-built extraction model assumes it owns the prompt and output contract. Graphify owns both. That's the structural incompatibility.

Model	Architecture	Ollama-pullable?	Follows Graphify's fixed prompt + schema?	Verdict
SciPhi/Triplex (Phi-3-3.8B, fine-tuned for KG triples)	Autoregressive causal LM	Yes (`ollama pull sciphi/triplex`)	No — locked to its own `{entity_types}/{predicates}` input template; emits subject–predicate–object triples, not Graphify's nodes/edges JSON	Can't drop in as-is
GLiNER (lightweight NER)	BERT-like bidirectional encoder	No	No — NER-only (span outputs); cannot emit arbitrary JSON or follow a system prompt	Wrong architecture
REBEL / Relik (relation extraction)	Seq2seq BART with special-token triples	No	No — emits `<triplet>/<subj>/<obj>` tokens; custom parsing required	Wrong architecture

Sources: HuggingFace SciPhi/Triplex model card, ollama.com/library/sciphi/triplex, GitHub urchade/GLiNER, GitHub Babelscape/rebel. [unverified claim] — line numbers in those repos were not pinned.

Is the purpose-built route worth experimenting with?

User hypothesis: if the JSON-shape mismatch is the only blocker, write a shim converting Triplex's triple output into Graphify's schema with a custom prompt. Triplex is marketed as very low-cost/fast (small 3.8B Phi-3 model).

What it would actually take [speculative]:

A model-specific prompt path. Triplex requires its {entity_types}/{predicates} template, not _EXTRACTION_SYSTEM. Graphify currently hard-codes one system prompt; a Triplex adapter needs to inject a different one for that model (or backend).
A model-specific output parser. Convert subject–predicate–object triples into Graphify's nodes + edges JSON: assign file_type to synthesized nodes, map predicates onto Graphify's fixed relation vocabulary (or accept free-text), synthesize confidence since Triplex emits no EXTRACTED/INFERRED/AMBIGUOUS enum, and reconstruct per-file provenance fields (source_file, source_location).
A new backend/model branch in llm.py. This is not a one-line patch — it is a backend adapter (prompt + parser) of moderate size.

Architecturally consistent. Graphify already has model-specific branches: the Kimi/moonshot backend sets extra_body={"thinking": {"type": "disabled"}} (llm.py:461-462) [github] to suppress reasoning output. A Triplex adapter follows the same pattern and could in principle be an upstream PR.

Quality unknowns [speculative]. Triplex targets generic NER + triples. Graphify's design wants typed nodes, a fixed relation vocabulary, EXTRACTED/INFERRED/AMBIGUOUS confidence, and per-file provenance. None of those would be native outputs from Triplex — they'd be synthesized, lowering fidelity compared to a model that follows Graphify's schema directly.

Cost/speed [speculative]. Triplex is plausibly faster per token (3.8B vs 7B), but qwen2.5-coder:7b at Q4 on a 12GB GPU is already rapid. The adapter effort isn't justified unless the default model proves too slow or too heavy for the freshness budget.

Verdict [speculative]: Realistic but moderate effort. Worth it only if: (a) qwen2.5-coder:7b proves too slow or too heavy for the project's freshness budget on the full vault, and (b) a quick Triplex spike shows acceptable triple quality on the vault fixtures. Neither condition is known yet. PARKED — revisit after the qwen2.5-coder:7b benchmark result is in hand.

Cheap first probe: ollama run sciphi/triplex on one fixture note using Triplex's native {entity_types}/{predicates} template. If the triples are coherent and well-typed, the adapter is worth scoping. If they are sparse or incorrectly typed, don't bother.

Why qwen2.5-coder:7b is the right axis, not a domain specialist

To be concrete: the extractors that do better than qwen2.5-coder:7b at Graphify's actual task are models that are better at structured-JSON instruction following, not models that are better at entity recognition or triple extraction in isolation. Relevant axis: instruction-following + JSON discipline. Irrelevant axis: NER F1, triple-completeness on academic benchmarks. [speculative] (Based on observed failure modes: models that failed in the 2026-06-04 benchmark failed by producing invalid JSON or consuming the output budget with thinking tokens — not by extracting the wrong entities.)

Pointers

Full local-model investigation + the num_ctx/thinking findings: docs/memory-system/benchmark/scoring-results-2026-06-04.md
Backend config details and Ollama setup: docs/graphify/05-local-models-and-backends.md
Graphify source (installed): ~/.local/lib/python3.14/site-packages/graphify/llm.py — anchored to 0.8.31

11 KiB Raw Blame History Unescape Escape