Executes all 16 tasks: local extraction-model benchmark with qwen2.5-coder:7b
selected as the default (7B parameter count, multi-modal, outperforms Mistral on
structured output), disposable vault graph build workflow, and supporting docs.
Adds extraction model options handbook, updates implementation process with
reconciliation strategy, and archives the scoring results.
Add new doc docs/memory-system/benchmark/local-llm-findings-2026-06-04.md
consolidating the local-LLM doc-extraction gut-check run. Key findings:
gemma4:e4b is the best installed model and adequate for the extraction
role (below frontier haiku but acceptable, catching ~45–60% of Opus's
high-value entities); GPU went unused due to NVIDIA driver version
mismatch (fix = reboot, pending); and Graphify owns the ollama call
(HTTP API, prompt/chunking/context/parsing internal) so raw-ollama
tuning is not production config. Add progressive-disclosure pointers
from 04-build-plan.md and 05-implementation-process.md.
Completed the benchmark run that gates Ollama model selection. Generated 18
reference fragments (6 cross-domain fixtures × 3 Claude tiers) and tightened
extraction spec rules based on first-run learnings. Updated implementation
status to reflect that reference set is complete and Ollama scoring is
unblocked. Populated fixture list in dispatch-prompt with final selection.
Invert the build order to build-first / migrate-incrementally and
redesign Step 2c as a Claude reference-set benchmark.
- ADR-013 records the build-order inversion; CLAUDE.md locked-decisions
pointer updated
- New benchmark deliverable under docs/memory-system/benchmark/:
shared extraction spec + copy/paste dispatch prompt + reference-outputs
- Runbook (05): Step 1 = fixture selection with bulk migration deferred;
Step 2c = Claude-tiers-only reference set, quality-only metrics, Opus
as gold-standard rubric (produces references, not a model choice)
- 04-build-plan reconciled to the new build order and benchmark design
- Sync incremental-migration + reference-extraction-benchmark specs into
openspec/specs/; archive the completed change
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jared