132 lines
11 KiB
Markdown
132 lines
11 KiB
Markdown
|
# 10 — Extraction Model Options: Why a General Structured-Output LLM, and Is a Purpose-Built KG Model Worth It?
|
|||
|
|
|
|||
|
|
_Last updated: 2026-06-05_ | _Status: reference / decision record_
|
|||
|
|
|
|||
|
|
This document explains why Graphify's document extraction uses a general instruction-following / structured-output LLM rather than a purpose-built knowledge-graph extraction model (e.g. Triplex, GLiNER, REBEL), and honestly assesses whether the purpose-built route is worth experimenting with for this project.
|
|||
|
|
|
|||
|
|
See also: [05 — Local models & backends](05-local-models-and-backends.md) · [Benchmark results (2026-06-04)](../memory-system/benchmark/scoring-results-2026-06-04.md)
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## The architecture constraint (this is the crux)
|
|||
|
|
|
|||
|
|
Graphify owns **both** the prompt and the output contract. It is not a thin wrapper you can swap a model into — the extraction pipeline is opinionated end-to-end:
|
|||
|
|
|
|||
|
|
**Fixed system prompt.** For document/markdown extraction, Graphify sends its own system prompt `_EXTRACTION_SYSTEM` (`llm.py:218-232`). `[github]` There is no user hook to supply a custom extraction prompt; the prompt is hardcoded.
|
|||
|
|
|
|||
|
|
**Fixed output schema.** The model must return exactly this JSON structure — no approximation accepted:
|
|||
|
|
|
|||
|
|
```json
|
|||
|
|
{
|
|||
|
|
"nodes": [{"id": "stem_entity", "label": "...", "file_type": "code|document|paper|image|rationale|concept",
|
|||
|
|
"source_file": "...", "source_location": null, "source_url": null,
|
|||
|
|
"captured_at": null, "author": null, "contributor": null}],
|
|||
|
|
"edges": [{"source": "node_id", "target": "node_id",
|
|||
|
|
"relation": "calls|implements|references|cites|conceptually_related_to|shares_data_with|semantically_similar_to",
|
|||
|
|
"confidence": "EXTRACTED|INFERRED|AMBIGUOUS",
|
|||
|
|
"confidence_score": 1.0, "source_file": "...", "source_location": null, "weight": 1.0}],
|
|||
|
|
"hyperedges": [], "input_tokens": 0, "output_tokens": 0
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
The `relation` field is constrained to a fixed vocabulary of seven values. The `confidence` field is a three-value enum (`EXTRACTED`/`INFERRED`/`AMBIGUOUS`). `[github]` (`llm.py:218-232`)
|
|||
|
|
|
|||
|
|
**No JSON-mode / grammar enforcement.** For the Ollama backend, Graphify passes only `{"options": {"num_ctx": ...}, "keep_alive": ...}` as extra parameters (`llm.py:506`). `[github]` There is no `response_format`, grammar constraint, or structured-output enforcement. Schema compliance rests entirely on the model's instruction-following ability.
|
|||
|
|
|
|||
|
|
**Lenient but not magic parser.** The response goes through `_parse_llm_json` (`llm.py:269-303`), which strips optional markdown fences and then tries `json.loads`. If that fails, it scans for the first balanced JSON object in the text. A model that emits its own schema — triples, spans, BIO tags — will produce an empty or nonsense graph, not a graceful fallback. `[github]`
|
|||
|
|
|
|||
|
|
**Temperature forced to 0 for Ollama.** The Ollama backend config sets `"temperature": 0` (`llm.py:65-73`). `[github]`
|
|||
|
|
|
|||
|
|
**Deep mode.** `--mode deep` appends `_DEEP_EXTRACTION_SUFFIX` to the same prompt, asking for extra `INFERRED` edges with a more conservative framing (`llm.py:234-247`). `[github]` The schema stays identical.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## The shipped default: why a coder model?
|
|||
|
|
|
|||
|
|
The Ollama fallback default is `qwen2.5-coder:7b` (`llm.py:67`): `[github]`
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
"default_model": os.environ.get("OLLAMA_MODEL", "qwen2.5-coder:7b"),
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Even though Graphify sends *documents* (not code) for extraction, the maintainer chose a coder/structured-output-tuned model. This is a revealed preference: the binding requirement is **structured-JSON instruction-following discipline**, not domain entity recognition. A coder/instruct model is the right axis; a NER specialist is the wrong one; frontier reasoning is overkill.
|
|||
|
|
|
|||
|
|
**Config knobs for the Ollama backend** (all `[github]` from README env-var table unless noted):
|
|||
|
|
|
|||
|
|
| Knob | How to set | What it does |
|
|||
|
|
|---|---|---|
|
|||
|
|
| `OLLAMA_MODEL` | env var | Override the `qwen2.5-coder:7b` default |
|
|||
|
|
| `OLLAMA_BASE_URL` | env var | Must end in `/v1` (e.g. `http://127.0.0.1:11434/v1`) — any other suffix produces 404s on every call |
|
|||
|
|
| `GRAPHIFY_OLLAMA_NUM_CTX` | env var | **See GOTCHA below** |
|
|||
|
|
| `GRAPHIFY_OLLAMA_KEEP_ALIVE` | env var | Minutes to keep model resident; default `30m`; set `0` to unload after each chunk |
|
|||
|
|
| `--token-budget` | CLI flag | Per-chunk input cap (tokens); pack multiple files per chunk |
|
|||
|
|
| `--max-concurrency` | CLI flag | Set 1–2 for a single local GPU |
|
|||
|
|
| `--mode deep` | CLI flag | Appends deep-inference suffix; elicits more INFERRED edges |
|
|||
|
|
|
|||
|
|
**GOTCHA — num_ctx does not propagate through graphify (verified 2026-06-04 on this project).** `[github]` Graphify posts to Ollama's OpenAI-compatible `/v1/chat/completions` endpoint. That endpoint **silently ignores** the per-request `options.num_ctx` that Graphify sends (`llm.py:506`). Proven by A/B: a POST to `/v1/chat/completions` with `num_ctx=8192` left `ollama ps` showing CONTEXT=4096; the same value through the native `/api/chat` endpoint was honoured. Therefore `GRAPHIFY_OLLAMA_NUM_CTX` has **no effect** through Graphify — context pins at Ollama's `/v1` default of 4096. At 4096 tokens, Graphify's extraction output JSON is truncated mid-response (`finish_reason=length`) and the chunk is **discarded**, producing an empty graph.
|
|||
|
|
|
|||
|
|
**Workaround (validated):** bake context into a Modelfile variant:
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
# Create a 16 k-context variant of qwen2.5-coder:7b:
|
|||
|
|
cat <<'EOF' | ollama create qwen25-coder-7b-16k -f -
|
|||
|
|
FROM qwen2.5-coder:7b
|
|||
|
|
PARAMETER num_ctx 16384
|
|||
|
|
EOF
|
|||
|
|
OLLAMA_MODEL=qwen25-coder-7b-16k graphify extract ./docs --backend ollama
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
The `/v1` endpoint honours the model's baked-in default. Non-invasive: no sudo, no systemd restart; reversible with `ollama rm`. Full investigation: `docs/memory-system/benchmark/scoring-results-2026-06-04.md`.
|
|||
|
|
|
|||
|
|
**Local patch — thinking mode disabled (applied 2026-06-04):** `reasoning_effort: "none"` was added at `llm.py:71` in the installed binary to suppress thinking output for any thinking-capable model. The original is backed up at `/tmp/graphify-bench/llm.py.orig`. This patch is a **no-op for `qwen2.5-coder:7b`** (no thinking mode) but will be **lost on `pip install --upgrade`** and would need re-applying for any future thinking-capable candidate. Needs a production path: upstream PR or a maintained local wrapper. `[github]`
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Purpose-built KG / entity / relation models — and why they don't drop in
|
|||
|
|
|
|||
|
|
Every purpose-built extraction model assumes **it** owns the prompt and output contract. Graphify owns both. That's the structural incompatibility.
|
|||
|
|
|
|||
|
|
| Model | Architecture | Ollama-pullable? | Follows Graphify's fixed prompt + schema? | Verdict |
|
|||
|
|
|---|---|---|---|---|
|
|||
|
|
| **SciPhi/Triplex** (Phi-3-3.8B, fine-tuned for KG triples) | Autoregressive causal LM | Yes (`ollama pull sciphi/triplex`) | No — locked to its own `{entity_types}/{predicates}` input template; emits subject–predicate–object triples, not Graphify's nodes/edges JSON | Can't drop in as-is |
|
|||
|
|
| **GLiNER** (lightweight NER) | BERT-like bidirectional encoder | No | No — NER-only (span outputs); cannot emit arbitrary JSON or follow a system prompt | Wrong architecture |
|
|||
|
|
| **REBEL / Relik** (relation extraction) | Seq2seq BART with special-token triples | No | No — emits `<triplet>/<subj>/<obj>` tokens; custom parsing required | Wrong architecture |
|
|||
|
|
|
|||
|
|
Sources: HuggingFace SciPhi/Triplex model card, `ollama.com/library/sciphi/triplex`, GitHub `urchade/GLiNER`, GitHub `Babelscape/rebel`. `[unverified claim]` — line numbers in those repos were not pinned.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Is the purpose-built route worth experimenting with?
|
|||
|
|
|
|||
|
|
**User hypothesis:** if the JSON-shape mismatch is the only blocker, write a shim converting Triplex's triple output into Graphify's schema with a custom prompt. Triplex is marketed as very low-cost/fast (small 3.8B Phi-3 model).
|
|||
|
|
|
|||
|
|
What it would actually take `[speculative]`:
|
|||
|
|
|
|||
|
|
1. **A model-specific prompt path.** Triplex requires its `{entity_types}/{predicates}` template, not `_EXTRACTION_SYSTEM`. Graphify currently hard-codes one system prompt; a Triplex adapter needs to inject a different one for that model (or backend).
|
|||
|
|
2. **A model-specific output parser.** Convert subject–predicate–object triples into Graphify's `nodes` + `edges` JSON: assign `file_type` to synthesized nodes, map predicates onto Graphify's fixed relation vocabulary (or accept free-text), synthesize `confidence` since Triplex emits no `EXTRACTED`/`INFERRED`/`AMBIGUOUS` enum, and reconstruct per-file provenance fields (`source_file`, `source_location`).
|
|||
|
|
3. **A new backend/model branch in `llm.py`.** This is not a one-line patch — it is a backend adapter (prompt + parser) of moderate size.
|
|||
|
|
|
|||
|
|
**Architecturally consistent.** Graphify already has model-specific branches: the Kimi/moonshot backend sets `extra_body={"thinking": {"type": "disabled"}}` (`llm.py:461-462`) `[github]` to suppress reasoning output. A Triplex adapter follows the same pattern and could in principle be an upstream PR.
|
|||
|
|
|
|||
|
|
**Quality unknowns `[speculative]`.** Triplex targets generic NER + triples. Graphify's design wants typed nodes, a fixed relation vocabulary, EXTRACTED/INFERRED/AMBIGUOUS confidence, and per-file provenance. None of those would be native outputs from Triplex — they'd be synthesized, lowering fidelity compared to a model that follows Graphify's schema directly.
|
|||
|
|
|
|||
|
|
**Cost/speed `[speculative]`.** Triplex is plausibly faster per token (3.8B vs 7B), but `qwen2.5-coder:7b` at Q4 on a 12GB GPU is already rapid. The adapter effort isn't justified unless the default model proves too slow or too heavy for the freshness budget.
|
|||
|
|
|
|||
|
|
**Verdict `[speculative]`:** Realistic but moderate effort. Worth it **only if**: (a) `qwen2.5-coder:7b` proves too slow or too heavy for the project's freshness budget on the full vault, **and** (b) a quick Triplex spike shows acceptable triple quality on the vault fixtures. Neither condition is known yet. **PARKED — revisit after the qwen2.5-coder:7b benchmark result is in hand.**
|
|||
|
|
|
|||
|
|
**Cheap first probe:** `ollama run sciphi/triplex` on one fixture note using Triplex's native `{entity_types}/{predicates}` template. If the triples are coherent and well-typed, the adapter is worth scoping. If they are sparse or incorrectly typed, don't bother.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Why qwen2.5-coder:7b is the right axis, not a domain specialist
|
|||
|
|
|
|||
|
|
To be concrete: the extractors that do better than `qwen2.5-coder:7b` at Graphify's actual task are models that are **better at structured-JSON instruction following**, not models that are better at entity recognition or triple extraction in isolation. Relevant axis: instruction-following + JSON discipline. Irrelevant axis: NER F1, triple-completeness on academic benchmarks. `[speculative]` (Based on observed failure modes: models that failed in the 2026-06-04 benchmark failed by producing invalid JSON or consuming the output budget with thinking tokens — not by extracting the wrong entities.)
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Pointers
|
|||
|
|
|
|||
|
|
- Full local-model investigation + the num_ctx/thinking findings: `docs/memory-system/benchmark/scoring-results-2026-06-04.md`
|
|||
|
|
- Backend config details and Ollama setup: `docs/graphify/05-local-models-and-backends.md`
|
|||
|
|
- Graphify source (installed): `~/.local/lib/python3.14/site-packages/graphify/llm.py` — anchored to 0.8.31
|