60 lines
3.4 KiB
Markdown
60 lines
3.4 KiB
Markdown
# Spec: local-model-selection
|
|
|
|
## Purpose
|
|
|
|
Install Graphify, configure Ollama for extraction (flash attention, context window), score candidate models against Claude-Opus references (entity correctness, relationship typing, confidence-tag accuracy, speed), and select the extraction model by evidence. As of 2026-06-04, `qwen2.5-coder:7b` is the selected model.
|
|
|
|
## Requirements
|
|
|
|
### Requirement: Extraction toolchain is installed and verified
|
|
|
|
The Graphify CLI SHALL be installed from the `graphifyy` PyPI package and verified to run before any extraction is attempted, and a running Ollama with at least one pulled candidate model SHALL be available.
|
|
|
|
#### Scenario: Graphify is callable
|
|
|
|
- **WHEN** the toolchain setup completes
|
|
- **THEN** `graphify --version` returns a version without error
|
|
- **AND** at least one candidate Ollama model is pulled and listed by `ollama list`
|
|
|
|
### Requirement: Ollama runtime is configured for extraction
|
|
|
|
Ollama SHALL be configured with the settings the extraction run depends on: flash attention enabled (`OLLAMA_FLASH_ATTENTION=1`) and a context window sufficient for vault notes (`GRAPHIFY_OLLAMA_NUM_CTX=8192`), and the allocated context SHALL be verified after the first extraction call.
|
|
|
|
#### Scenario: Configuration is in effect
|
|
|
|
- **WHEN** the first extraction call is made
|
|
- **THEN** flash attention is enabled and the context size is 8192
|
|
- **AND** `ollama ps` shows the expected allocated context for the loaded model
|
|
|
|
### Requirement: Candidate models are scored against the gold-standard reference set
|
|
|
|
The extraction model SHALL be selected by scoring candidate Ollama models against the existing Step 2c reference set (the 18 fragments in `docs/memory-system/benchmark/reference-outputs/`) over the same 6 fixture notes. Each candidate's Graphify-shaped output SHALL be compared to the `claude-opus-4-8` gold-standard output for entity correctness, relationship plausibility and typing, and `INFERRED`/`AMBIGUOUS` confidence-tag accuracy, and wall-clock extraction speed SHALL be measured per candidate.
|
|
|
|
#### Scenario: Each candidate is scored on quality and speed
|
|
|
|
- **WHEN** a candidate model is run over the 6 fixtures
|
|
- **THEN** its output is scored against the Opus reference on entity correctness, relationship typing, and confidence-tag accuracy
|
|
- **AND** its wall-clock extraction speed is recorded
|
|
|
|
#### Scenario: Reference benchmark is consumed, not re-created
|
|
|
|
- **WHEN** scoring is performed
|
|
- **THEN** it reads the existing reference fragments produced by the `reference-extraction-benchmark` capability
|
|
- **AND** it does not regenerate or modify the Claude reference set
|
|
|
|
### Requirement: Model is selected by evidence, not hardcoded
|
|
|
|
The chosen extraction model SHALL be the one justified by the scoring run's recorded results, and no model SHALL be hardcoded as the selection before scoring completes. `gemma4:e4b` MAY be the front-runner candidate, but its selection SHALL depend on its scored quality, not its feasibility gut-check alone.
|
|
|
|
#### Scenario: Selection records its rationale
|
|
|
|
- **WHEN** a model is selected
|
|
- **THEN** a result artifact records the chosen model, its quality scores against the Opus rubric, and its measured speed
|
|
- **AND** the rationale references the scoring evidence rather than asserting a pre-chosen model
|
|
|
|
#### Scenario: No model is locked before scoring
|
|
|
|
- **WHEN** scoring has not yet run
|
|
- **THEN** no model is committed as the selection
|
|
- **AND** the front-runner candidate is treated as unconfirmed until scored
|