cc-os/docs/memory-system/benchmark/scoring-results-2026-06-04.md

# Local-Model Extraction Scoring — Results (2026-06-04)

_Last updated: 2026-06-05_ | _Status: UNBLOCKED — qwen2.5-coder:7b selected; scoring (3.4), speed (3.3), and model selection (3.5/3.6) complete. Remaining: build vault graph (4), update docs (5)._

---

## Toolchain (Step 2a)

- graphify 0.8.31 (PyPI `graphifyy`) installed at `~/.local/bin/graphify`; `graphify --version` → `graphify 0.8.31`. [verified: local shell]
- ollama 0.30.3, systemd service `/usr/local/bin/ollama serve`, host 127.0.0.1:11434. [verified: local shell]
- GPU: RTX 3060, 12GB VRAM. All extraction ran 100% on GPU (no CPU spill) per `ollama ps`. [verified: local shell]

---

## Candidate models — true bases (resolves the e4b/e2b naming question)

Pinned via `ollama show <tag>` info cards. These are genuine next-gen families, NOT gemma3n/qwen2.5 renames: [verified: ollama show output]

| Local tag     | Architecture | Params                          | Quant  | Native ctx |
|---------------|--------------|---------------------------------|--------|------------|
| gemma4:e4b    | gemma4       | 8.0B (MatFormer "effective 4B") | Q4_K_M | 131072     |
| gemma4:e2b    | gemma4       | 5.1B (effective 2B)             | Q4_K_M | 131072     |
| qwen3.5:2b    | qwen35       | 2.3B                            | Q8_0   | 262144     |

---

## Critical config finding — num_ctx does not propagate through graphify (Step 2b)

- graphify posts to ollama's OpenAI-compatible `/v1/chat/completions` endpoint. That endpoint **silently ignores** the per-request `options.num_ctx` graphify sends. Proven by A/B: `POST /v1/chat/completions` with num_ctx=8192 → `ollama ps` CONTEXT=4096; `POST /api/chat` with num_ctx=8192 → CONTEXT=8192. [verified: local A/B test]
- Therefore `GRAPHIFY_OLLAMA_NUM_CTX` (env var) has **NO EFFECT** through graphify; context pins at ollama's /v1 default 4096.
- At 4096, graphify's extraction output JSON is truncated mid-response (finish_reason=length) and the chunk is **DISCARDED** → empty graph, regardless of (small) input note size.
- **WORKAROUND (validated):** bake context into a Modelfile variant — `FROM <model>` + `PARAMETER num_ctx 8192`, then `ollama create <name>-8k`. The /v1 endpoint **does** honor the model's baked default. Verified `ollama ps` CONTEXT=8192 (and 16384), 100% GPU. This is the same lever production (vault build) will need. Non-invasive: no sudo, no systemd restart, no shell-profile/dotfiles edit; reversible via `ollama rm`. [verified: local shell]
- Two other operational notes: graphify's `OLLAMA_BASE_URL` must **end in `/v1`** (e.g. `http://127.0.0.1:11434/v1`) or every call 404s; and the CLI requires `OLLAMA_API_KEY` set to any non-empty value unless the host is loopback (127.0.0.1/localhost/::1).

---

## Benchmark methodology

- Each of the 6 Step-2c fixtures was isolated in its own temp directory (graphify `extract` does NOT accept a single file path — it walks a directory; isolation also avoids conflating notes in one token-budget chunk and excludes `.obsidian/` config noise). [verified: local testing]
- Command shape: `OLLAMA_BASE_URL=http://127.0.0.1:11434/v1 graphify extract <fixture-dir> --backend ollama --model <8k-variant> --max-concurrency 1 --out <dir>`. Models run sequentially (single GPU), one resident at a time.
- Scoring was scoped to dimensions graphify's output and the `.opus` gold references share: entity correctness, relationship plausibility, and edge-level INFERRED/AMBIGUOUS confidence. **Excluded as non-comparable:** the six-facet taxonomy and per-entity type/confidence (graphify's built-in extraction prompt emits neither — per ADR-011 facets are note frontmatter metadata, never Graphify's extraction job), and free-text relationship TYPE strings (graphify uses a fixed relation vocabulary: `references`/`conceptually_related_to`/`semantically_similar_to`/etc., vs the references' free-text verbs).

---

## Results through graphify's real extraction path

All three tested as 8192-context Modelfile variants. **Scoring blocked** because outputs were degenerate:

| Model              | Result                                    | Failure mode |
|--------------------|-------------------------------------------|--------------|
| gemma4:e4b (8k)    | EMPTY graphs (multiple fixtures, deterministic 5/5) | Hollow, invalid JSON (~5039 output tokens of unparseable content); graphify discards chunk. graphify's own warning: "model too small for JSON instruction following — try a larger model." |
| qwen3.5:2b (8k)    | EMPTY graph                               | Output truncated at max_completion_tokens; qwen3.5 "thinking" mode appears to consume the output budget before emitting the JSON graph. |
| gemma4:e2b (8k)    | 15 nodes, 0 edges (valid JSON) on the oo-principles fixture | Extracted plausible entities (OO Principles, Single Responsibility Principle, Law of Demeter, Dependency Injection, the four design approaches, etc.) but emitted ZERO relationships → unusable as a knowledge graph. |

**Caveat on variance:** model Modelfile defaults are temperature 1 (poor for structured extraction); one PRE-fix run of gemma4:e4b on oo-principles yielded 4 nodes/3 edges, indicating high run-to-run variance.

---

## Conclusion

No specified candidate (gemma4:e4b, gemma4:e2b, qwen3.5:2b) produces a usable entity+relationship graph through Graphify's **default extraction prompt** on this hardware. Per the change proposal's risk clause ("no candidate scores acceptably → surface as finding, don't force selection"), model selection (task 3.6) and the initial vault graph build (task 4) are **DEFERRED** pending a decision.

**NOTE:** The earlier gut-check (`local-llm-findings-2026-06-04.md`) reported gemma4:e4b doing "6/6 clean parse, 13–34 entities" — but that used a **direct ollama call** with a simpler custom prompt, **NOT** graphify's actual extraction prompt. Through graphify's real path the result is far worse. The graphify-path numbers are the authoritative ones for production.

---

## Untested levers (candidate next steps, not yet attempted)

1. **Disable model "thinking" mode** — gemma4 and qwen3.5 are thinking-capable; thinking tokens likely burn the output budget, causing the hollow/truncated JSON. Most promising cheap lever, but exposing a think-toggle through graphify's /v1 path is unverified.
2. **`graphify extract --mode deep`** to elicit INFERRED edges (addresses e2b's 0-edge result).
3. **Override sampling temperature to 0** for structured extraction (Modelfile defaults are temp 1).
4. **Pull a stronger model that fits 12GB VRAM with 8k context** — e.g. a 7–8B at Q4 (~5GB, leaves headroom for context); graphify itself suggests a ~14B (~9GB, tighter). This expands the candidate set beyond the three the change specified (a download + scope decision).

---

## Raw artifacts (ephemeral, /tmp — will not survive reboot)

- e4b first benchmark (pre-fix, 4096): `/tmp/graphify-bench/out/e4b/`
- 8k-variant runs: `/tmp/graphify-bench/out2/`, `/tmp/graphify-bench/out3/`
- Created reversible ollama variants: `gemma4-e4b-8k`, `gemma4-e4b-16k`, `gemma4-e2b-8k`, `qwen35-2b-8k`

---

## Round 2 (2026-06-05): thinking-disable patch + edge gate

### The thinking-mode investigation (user-directed)
- GitHub issue safishamsi/graphify#792 turned out to be about CPU scaling, local API timeouts, and the OLLAMA_API_KEY auth gate — NOT thinking/reasoning. (Documents the same auth friction we hit.)
- Ollama's `/v1` (OpenAI-compat) endpoint, which Graphify uses, IGNORES `think:false` and `chat_template_kwargs:{enable_thinking:false}`, but DOES honor top-level `reasoning_effort:"none"`. The native `/api/chat` honors `think:false`. (Same /v1-drops-nonstandard-options pattern as num_ctx.)
- Graphify sends NO thinking control for the ollama backend by default; there is no env var / CLI flag / config file to inject it (confirmed by source trace of llm.py). `PARAMETER think false` in a Modelfile is rejected by `ollama create`; a Modelfile `SYSTEM /no_think` does not survive because Graphify sends its own system message and ollama's /v1 REPLACES (not merges) the system message.
- WORKING FIX (applied to the installed package): one-line patch adding `"reasoning_effort": "none"` to the ollama backend config dict in `~/.local/lib/python3.14/site-packages/graphify/llm.py` (~line 71). Graphify already applies `reasoning_effort` as a top-level kwarg if present, and it survives the extra_body overwrite. The author already uses the equivalent pattern (`extra_body={"thinking":{"type":"disabled"}}`) for the Kimi/moonshot backend — just not for ollama. PRODUCTION IMPLICATION: this is a local patch lost on `pip install --upgrade`; production needs an upstream PR (likely welcome) or a maintained patch/wrapper.
- Verified effect: with the patch, every model's raw output begins directly with `{"nodes":[...` — no reasoning preamble. Thinking is genuinely suppressed.

### The edge gate — the criterion that actually matters
A knowledge graph needs RELATIONSHIPS (edges), not just entities (nodes). Across ~10+ runs this whole investigation, LLM-extracted relationships appeared essentially once (e4b pre-patch: 3 edges, since non-reproducible). All other non-empty results were nodes-only. So the gate for selection is: does a config emit a plausible EDGE-bearing graph on one fixture?

Bounded test on the `oo-principles-plugin-concept-design-recommendations` fixture (a note that clearly contains relationships), thinking-disable confound checked by running both ON and OFF:

| Config | Exit | Nodes | Edges | Outcome |
|---|---|---|---|---|
| qwen3.5:2b @16k, thinking-OFF | 1 | — | empty | output truncated mid-JSON → invalid → discarded |
| qwen3.5:2b @16k, thinking-ON | 1 | — | empty | same truncation; 16k did not rescue it |
| gemma4:e2b @16k, thinking-ON | 0 | 17 | 0 | clean valid JSON, coherent on-topic nodes, ZERO edges |
| gemma4:e2b @16k, thinking-ON, --mode deep | 0 | 10 | 0 | deep mode (explicitly requests INFERRED edges) made it WORSE: fewer nodes, still 0 edges |

`hyperedges` was also empty `[]` in the clean runs — no relationships hid in the alternate key.

### Confound resolved: thinking-off is NOT strictly better
gemma4:e2b @8k: thinking-ON gave 15 nodes / 3307 output tokens (valid); thinking-OFF gave 1 node / 395 tokens. For e2b the thinking tokens were doing extraction work; the binding constraint is output room, not thinking. So the user's "thinking wastes the budget" premise holds for qwen (truncation) but inverts for e2b. Thinking-disable is a useful lever, not a universal win.

### Two distinct failure modes
- qwen3.5:2b (2.3B, Q8_0): capacity/truncation — runs out of output room emitting node JSON before reaching edges. 16k context insufficient; thinking on/off irrelevant.
- gemma4:e2b (5.1B, Q4_K_M): clean completion, extracts plausible NODES but emits zero RELATIONSHIPS even when deep mode explicitly asks for them. The model understands the doc (node labels: OO Principles, Single Responsibility Principle, Law of Demeter, Dependency Injection, Replace Conditional with Polymorphism, the 4 approaches, the 3 layers) — it just won't populate `links`.
- gemma4:e4b (8.0B, Q4_K_M): hollow/malformed JSON (dropped — not a context problem).

### Round-2 conclusion
The cheap-lever space is exhausted (thinking ON/OFF, 8k/16k context, --mode deep). No local 2–5B candidate produces a relationship-bearing graph through Graphify's extraction prompt on a 12GB GPU. Relationship extraction (not entity extraction) is the wall. Recommended path: a stronger model that fits 12GB with ~16k context — e.g. qwen2.5-coder:7b (~5GB Q4, non-thinking, strong at structured JSON+edges). This expands the candidate set beyond the three the change specified (a multi-GB download + scope decision) — pending user go-ahead.

### Round-2 ollama variants created (reversible via `ollama rm`)
gemma4-e4b-8k, gemma4-e4b-16k, gemma4-e2b-8k, gemma4-e2b-16k, qwen35-2b-8k, qwen35-2b-16k. Installed graphify llm.py currently carries the thinking-OFF patch (backup of original at /tmp/graphify-bench/llm.py.orig).

---

## Round 3 (2026-06-05): qwen2.5-coder:7b clears the edge gate — full benchmark

### Config that works
- Model: `qwen2.5-coder:7b` (graphify's hardcoded ollama default, llm.py:67), run as a Modelfile variant `qwen25-coder-7b-16k` (FROM qwen2.5-coder:7b + PARAMETER num_ctx 16384).
- Command: `OLLAMA_BASE_URL=http://127.0.0.1:11434/v1 graphify extract <fixture-dir> --backend ollama --model qwen25-coder-7b-16k --max-concurrency 1 --out <dir>`. The thinking-OFF llm.py patch is in place (no-op for qwen2.5-coder, which has no thinking mode).
- VRAM: 5.6GB resident, 100% GPU, 16384 context, no CPU spill on the 12GB RTX 3060.
- Speed (task 3.3): aggregate throughput ~59.4 tok/s (substantive runs cluster 57–61 tok/s). Total wall-clock all 6 = 253.7s (4.2 min); avg 42.3s/fixture (49.9s excluding the degenerate 10dlc run). Use tok/s not wall-clock (wall-clock tracks output length).

### Per-fixture results (edge gate PASSED: 5/6 produce relationships)
| Fixture | Wall | Nodes | Edges | Out tok | Notes |
|---|---|---|---|---|---|
| oo-principles-plugin-concept | 44.9s | 11 | 10 | 2553 | gate fixture; cold-load incl. |
| 10dlc-isv-setup-guide-oncadence | 4.4s | 1 | 0 | 159 | under-extraction on smallest note (502 words), exit 0 valid JSON — not a crash |
| pest-control-enterprise-revenue | 64.2s | 23 | 17 | 3932 | only fixture with INFERRED edges (7) |
| ai-coding-conventions-synthesis | 37.9s | 10 | 9 | 2284 | |
| pest-control-after-hours-sms | 76.8s | 21 | 20 | 4641 | |
| pest-control-email-a-b-c-hub | 25.5s | 7 | 6 | 1489 | |

All exit 0, no truncation/hollow/invalid-JSON warnings. Confidence tags are provenance method tags: EXTRACTED (from text) vs INFERRED (model-inferred); INFERRED appeared only in enterprise-revenue.

### Full edge lists (for scoring vs .opus references next session)
**oo-principles (10, all EXTRACTED):** Process Layer→shares_data_with→Mechanic Layer; Process Layer→shares_data_with→Theory Layer; Mechanic Layer→shares_data_with→Cards Layer; Annotated Process Graph Layer→shares_data_with→Cards Layer; Bundles Layer→shares_data_with→Cards Layer; CLAUDE.md→shares_data_with→Cards Layer; NotebookLM Notebook→shares_data_with→Cards Layer; Phase Bundled Context Packs Layer→shares_data_with→Cards Layer; Refactoring Layer→shares_data_with→Cards Layer; Situational Trigger Files Layer→shares_data_with→Cards Layer.
**10dlc:** none.
**enterprise-revenue (17):** Marketing Allocation Model→references→Unit Economics; ACV Tiers→references→Unit Economics; M&A Context→references→ACV Tiers; Communication Failure Patterns→references→Unit Economics; Dropped calls and ghosting→conceptually_related_to→Communication Failure Patterns [INFERRED]; No one answered→conceptually_related_to→Communication Failure Patterns [INFERRED]; Offshore agent disconnect→conceptually_related_to→Communication Failure Patterns [INFERRED]; Voicemail full→conceptually_related_to→Communication Failure Patterns [INFERRED]; M&A Context→references→Communication Failure Patterns; Climatologically-Driven Lead Seasonality→references→Unit Economics; Fall First Freeze Rodent Push→conceptually_related_to→Climatologically-Driven Lead Seasonality [INFERRED]; Spring Termite Swarming→conceptually_related_to→Climatologically-Driven Lead Seasonality [INFERRED]; Summer Heatwave-Driven Migrations→conceptually_related_to→Climatologically-Driven Lead Seasonality [INFERRED]; M&A Context→references→Climatologically-Driven Lead Seasonality; M&A Context→references→Operational Unit Economics; Operational Unit Economics→references→COGS Structure; M&A Context→references→Unit Economics.
**ai-coding-conventions (9, all EXTRACTED):** Core Finding→references→{Context Linking Patterns, Enforcement Mechanisms, Key Tradeoffs, Notable Public References, Organizational Patterns, Process vs. Reference Design, See Also, The Three-Tier Architecture (arXiv 2602.20478), Token Efficiency Strategies}.
**after-hours-sms (20, all EXTRACTED):** document node→cites→ each of 20 study/source nodes (MIT/InsideSales 2007, HBR 2011, Velocify 2012, InsideSales 2021, HomeAdvisor/Angi 2024, Thumbtack 2024, Coalmarch 2025, Invoca 2024, Driven Results 2025, HubSpot 2023, Gartner 2016, D7 Networks/CTIA 2024, TransUnion 2024, Avochato 2019, Briostack 2024, ServiceTitan 2025, Point Loma Hatch, Shafer HVAC Hatch, Aruza Cube, ServiceTitan Data 2025).
**email-a-b-c-hub (6, all EXTRACTED):** Experiment Hub→references→{Blind A/B/C Methodology Guide, Per Group Copy and Scores, What Owners Respond to/Reject, V4 Copywriter Draft, V4 Golden Framework Draft, V4 Golden Framework Revised}.

### Round-3 takeaway
qwen2.5-coder:7b is the de-facto selected model pending the formal scoring pass (3.4). It's graphify's own default, fits 12GB with 16k context at ~59 tok/s, and produces typed, mostly-EXTRACTED edges. The earlier gut-check's gemma4:e4b pick was an artifact of using a simpler direct prompt, not graphify's real path. Source graph.json (ephemeral): /tmp/graphify-bench/out6/coder7b/<slug>/graphify-out/graph.json.

---

## 3.4 Scoring — qwen2.5-coder:7b vs Opus Gold Standard

**Sources used:**
- Candidate output: `scoring-results-2026-06-04.md`, Round 3 section (per-fixture node/edge lists)
- Gold-standard references: `reference-outputs/*.opus.md` (all 6 `.opus.md` files)
- Fixture mapping (from Round 3 table): `oo-principles-plugin-concept`, `10dlc-isv-setup-guide-oncadence`, `pest-control-enterprise-revenue`, `ai-coding-conventions-synthesis`, `pest-control-after-hours-sms`, `pest-control-email-a-b-c-hub`

### Dimensions Excluded (N/A)

**Facets:** The `.opus.md` references include facet annotations (e.g., `facet: tool/twilio`, `facet: convention/oo-principles`). These are note-level metadata per ADR-011 — they are not part of graphify's extraction output at all, so there is nothing to compare. N/A.

**Entity-type labels:** The opus references annotate entities with types (`type: layer`, `type: principle`, `type: Tool`, etc.). Graphify's extraction output produces a node list with names only — no type labels are emitted. N/A.

**Free-text relationship typing:** The opus references use descriptive, domain-specific verbs (`comprises_share_of`, `is_event_of`, `measured conversion decay for`, `tested for campaign`, etc.). Graphify uses a fixed relation vocabulary (`references`, `conceptually_related_to`, `semantically_similar_to`, `shares_data_with`, `cites`). These are structurally incommensurable — comparing relation-type strings would score graphify's vocabulary design, not extraction quality. N/A.

### Rubric

**Entity Correctness (a):** 1–5. How many of the entities graphify extracted correspond to real entities in the reference? Penalizes under-extraction (missing key nodes) and over-compression (collapsing distinct entities). Does not penalize naming style differences.

**Relationship Plausibility (b):** 1–5. Are the edges graphify drew present or derivable from the reference's relationship set? Penalizes phantom edges (no basis in reference), missing major structural relationships, and systematically wrong pairing patterns. Does not penalize relation-verb style.

**Confidence Accuracy (c):** 1–5. Where graphify tagged EXTRACTED vs INFERRED vs AMBIGUOUS, does that align with what the reference treated as explicit (no `confidence:` tag) vs inferred (`confidence: INFERRED`) vs ambiguous (`confidence: AMBIGUOUS`)? N/A on fixtures where graphify emitted no edges at all, or no INFERRED/AMBIGUOUS tags appeared in either output.

### Per-Fixture Scoring Table

| Fixture | (a) Entity Correctness | (b) Relationship Plausibility | (c) Confidence Accuracy | Notes |
|---|---|---|---|---|
| oo-principles-plugin-concept | 2 | 1 | N/A† | |
| 10dlc-isv-setup-guide | 1 | N/A (0 edges) | N/A | |
| pest-control-enterprise-revenue | 3 | 3 | 3 | |
| ai-coding-conventions-synthesis | 2 | 2 | N/A† | |
| pest-control-after-hours-sms | 3 | 2 | N/A† | |
| pest-control-email-a-b-c-hub | 2 | 2 | N/A† | |

†Graphify emitted no INFERRED/AMBIGUOUS tags on this fixture (all EXTRACTED), so there is no confidence signal to compare against the reference's INFERRED annotations. N/A does not mean graphify was wrong; it means the dimension is unobservable.

### Per-Fixture Justifications

**oo-principles-plugin-concept**
- (a) **2/5.** Captured the layer decomposition (Process, Mechanic, Theory + 4 named layers, CLAUDE.md, NotebookLM Notebook = 11 nodes) but missed the plugin as a named entity, all OO principle entities (TDD, SRP, LoD, DI, Shameless Green, Flocking Rules), the book/paper sources, and the 4-phase development lifecycle. ~30% of the ~30-entity reference set.
- (b) **1/5.** All 10 edges are `shares_data_with` → "Cards Layer". The reference relationships are structural/directional (includes, encodes, references, retains, alternative to, evolves from, modeled on). Homogenization collapses the architecture into a flat data-sharing star with no analog in the reference.
- (c) **N/A.** All graphify edges EXTRACTED; reference has 3 INFERRED. No INFERRED signal to compare.

**10dlc-isv-setup-guide**
- (a) **1/5.** Extracted 1 node (whole document as a single entity) vs 20 distinct reference entities (OnCadence, 10DLC, Twilio, TrustHub API, ISV, CSP, EIN, IRS, T-Mobile, subaccount-per-client architecture, etc.). Near-total extraction failure on a 502-word note.
- (b) **N/A.** 0 edges produced vs 17 reference relationships.
- (c) **N/A.** No edges.

**pest-control-enterprise-revenue**
- (a) **3/5.** 23 nodes capturing the major conceptual clusters (Marketing Allocation Model, ACV Tiers, M&A Context, Unit Economics, Communication Failure Patterns + 4 sub-patterns, Seasonality + 3 seasonal events, COGS Structure) but at shallower granularity than the ~55-entity reference (misses CAC/LTV/CPL/Churn, COGS components, regions, Big Four integrators, GDD, multiples).
- (b) **3/5.** 17 edges incl. 7 INFERRED. The INFERRED membership cluster (comm sub-patterns → Communication Failure Patterns; seasonal events → Seasonality) maps closely onto the reference's `is_event_of`/`component_of` structure. ~50% of reference edges missed (COGS breakdown, Big Four, causal chains).
- (c) **3/5.** The 7 INFERRED tags are defensible group-membership edges; not systematically miscalibrated, though it under-fires on EXTRACTED edges the reference treats as explicit, and never modeled the reference's `EIN → issued_by → IRS` background-knowledge INFERRED edge.

**ai-coding-conventions-synthesis**
- (a) **2/5.** Extracted the document's section-heading outline (10 nodes: Context Linking Patterns, Enforcement Mechanisms, Three-Tier Architecture, etc.) rather than the ~40 domain entities (Claude, Cursor, Copilot, Constitution, Domain Specialist Agents, .cursorrules, AGENTS.md, MCP, named projects). Only Three-Tier Architecture overlaps in substance.
- (b) **2/5.** All 9 edges are `Core Finding → references → {section heading}` — a hub-and-spoke from a synthetic aggregate node. Zero structural resolution vs the reference's compositional/directional relationships. 2 not 1 because no phantom entities introduced.
- (c) **N/A.** All EXTRACTED; reference's 2 INFERRED edges not modeled.

**pest-control-after-hours-sms**
- (a) **3/5.** 21 nodes (document + 20 study/source nodes) covering ~half the reference's company/study entities, but missing the product/project/campaign entities (After-Hours SMS product, niche-automation-prospecting initiative, pest-control-spring-2026 campaign) and several companies (Google Ads/LSA, NPMA, CallRail, PATLive, etc.).
- (b) **2/5.** All 20 edges are `document → cites → {study}` — a citation list. Defensible but reductive; the reference's product–customer, product–channel, and study–finding semantic edges are entirely missing.
- (c) **N/A.** All EXTRACTED; reference has 2 INFERRED + 1 AMBIGUOUS, none modeled.

**pest-control-email-a-b-c-hub**
- (a) **2/5.** Extracted the hub's table-of-contents (7 nodes: Experiment Hub + 6 linked sub-documents) but missed the experiment's analytic entities (3 personas, 9 email groups, 5 hooks, structural/retired patterns, 3 versions) — the primary conceptual content. ~45-entity reference.
- (b) **2/5.** All 6 edges are `Experiment Hub → references → {sub-document}` — the index, not the findings. Version outcomes, recommended groups, retired patterns all absent.
- (c) **N/A.** All EXTRACTED; reference's 2 INFERRED edges not modeled.

### Overall Summary

**Entity extraction: weak to moderate (avg ~2.2/5).** Consistently fewer/coarser entities than the opus reference. On hub-structured notes it extracts the structural outline (section headings, linked sub-docs) rather than conceptual entities; on dense domain notes it captures ~40–55% of the reference entity set; on the smallest note (10dlc) it nearly fails (1 node vs 20).

**Relationship extraction: poor to moderate (avg ~1.9/5, excluding the N/A fixture).** Dominant failure mode is **homogenization**: nearly all relationship semantics collapse into one or two relation types (`references`, `cites`, `shares_data_with`) in a hub-and-spoke topology. enterprise-revenue is the only fixture with meaningfully differentiated edges — and the only one where INFERRED edges appeared — suggesting extraction quality correlates with note density.

**Confidence calibration: partially defensible where observable (3/5 on the one applicable fixture).** INFERRED tags only appeared on enterprise-revenue (7/17 edges) and were defensible group-membership edges. On the other five fixtures all edges were EXTRACTED, precluding comparison.

**Net assessment:** qwen2.5-coder:7b passes the edge gate (5/6 fixtures produce relationships) and extracts credible entities on content-dense notes, confirming it as a viable baseline over the tested alternatives. Its relationship extraction is systemically shallow: the relation flattening means the graph encodes "these things are related" but not "how," limiting downstream query quality. For a production vault graph the primary gap is relationship semantic resolution, not entity recall.

---

## 3.6 Model Selection

**Selected model: qwen2.5-coder:7b** (run as the Modelfile variant `qwen25-coder-7b-16k`, num_ctx 16384).

The selection is not a choice among viable candidates — it is the only candidate that cleared the relationship/edge gate. Every smaller candidate tested (gemma4:e4b 8.0B, gemma4:e2b 5.1B, qwen3.5:2b 2.3B) failed to produce relationship-bearing graphs through graphify's prompt on this hardware. The scoring in section 3.4 therefore characterizes qwen2.5-coder:7b's extraction quality against the opus gold standard rather than differentiating among alternatives; there was no viable alternative to differentiate.

This model is also graphify's own hardcoded ollama default (llm.py:67), which is a strong signal that it is the intended extraction model for this backend. It fits the 12GB RTX 3060 comfortably at 5.6GB VRAM resident with 16384 context and runs at ~59 tok/s, completing a full 6-fixture pass in ~4 minutes.

**Known limitations (carried forward honestly):** shallow relationship semantics — graphify's fixed relation vocabulary combined with qwen2.5-coder:7b's extraction behavior produces hub-and-spoke topologies dominated by `references`, `cites`, and `shares_data_with`, collapsing domain-specific structural relationships. Entity recall is weak on hub-structured notes (extracts the structural outline instead of conceptual entities) and near-failing on very short notes (10dlc: 1 node vs 20 in reference). These limitations are accepted for the initial build. Relationship semantic resolution is the known primary gap for future improvement.

**Speed:** ~59 tok/s on the 12GB GPU at max-concurrency 1.