cc-os/openspec/specs/local-model-selection/spec.md

Spec: local-model-selection

Purpose

Install Graphify, configure Ollama for extraction (flash attention, context window), score candidate models against Claude-Opus references (entity correctness, relationship typing, confidence-tag accuracy, speed), and select the extraction model by evidence. As of 2026-06-04, qwen2.5-coder:7b is the selected model.

Requirements

Requirement: Extraction toolchain is installed and verified

The Graphify CLI SHALL be installed from the graphifyy PyPI package and verified to run before any extraction is attempted, and a running Ollama with at least one pulled candidate model SHALL be available.

Scenario: Graphify is callable

• WHEN the toolchain setup completes
• THEN graphify --version returns a version without error
• AND at least one candidate Ollama model is pulled and listed by ollama list

Requirement: Ollama runtime is configured for extraction

Ollama SHALL be configured with the settings the extraction run depends on: flash attention enabled (OLLAMA_FLASH_ATTENTION=1) and a context window sufficient for vault notes (GRAPHIFY_OLLAMA_NUM_CTX=8192), and the allocated context SHALL be verified after the first extraction call.

Scenario: Configuration is in effect

• WHEN the first extraction call is made
• THEN flash attention is enabled and the context size is 8192
• AND ollama ps shows the expected allocated context for the loaded model

Requirement: Candidate models are scored against the gold-standard reference set

The extraction model SHALL be selected by scoring candidate Ollama models against the existing Step 2c reference set (the 18 fragments in docs/memory-system/benchmark/reference-outputs/) over the same 6 fixture notes. Each candidate's Graphify-shaped output SHALL be compared to the claude-opus-4-8 gold-standard output for entity correctness, relationship plausibility and typing, and INFERRED/AMBIGUOUS confidence-tag accuracy, and wall-clock extraction speed SHALL be measured per candidate.

Scenario: Each candidate is scored on quality and speed

• WHEN a candidate model is run over the 6 fixtures
• THEN its output is scored against the Opus reference on entity correctness, relationship typing, and confidence-tag accuracy
• AND its wall-clock extraction speed is recorded

Scenario: Reference benchmark is consumed, not re-created

• WHEN scoring is performed
• THEN it reads the existing reference fragments produced by the reference-extraction-benchmark capability
• AND it does not regenerate or modify the Claude reference set

Requirement: Model is selected by evidence, not hardcoded

The chosen extraction model SHALL be the one justified by the scoring run's recorded results, and no model SHALL be hardcoded as the selection before scoring completes. gemma4:e4b MAY be the front-runner candidate, but its selection SHALL depend on its scored quality, not its feasibility gut-check alone.

Scenario: Selection records its rationale

• WHEN a model is selected
• THEN a result artifact records the chosen model, its quality scores against the Opus rubric, and its measured speed
• AND the rationale references the scoring evidence rather than asserting a pre-chosen model

Scenario: No model is locked before scoring

• WHEN scoring has not yet run
• THEN no model is committed as the selection
• AND the front-runner candidate is treated as unconfirmed until scored