cc-os/openspec/changes/archive/2026-06-04-incremental-migration-and-reference-benchmark/proposal.md

Why

Two gaps block executing the build runbook (docs/memory-system/05-implementation-process.md). First, Step 2c's model benchmark assumes head-to-head runs across local Ollama models and Claude API models, but the only models reachable from this environment are the Claude tiers via Claude Code subagents — so the benchmark cannot be run as written, and nothing captures a gold-standard to score local models against later. Second, the runbook front-loads bulk vault migration (all ~20 notes and all projects) as Step 1, committing to a schema and workflow before the system exists to validate them.

What Changes

• Redesign Step 2c as a reference-set benchmark: a copy/paste-able Claude Code prompt — written as a committed file — that dispatches one subagent per Claude tier (claude-haiku-4-5, claude-sonnet-4-6, claude-opus-4-8) to mimic Graphify doc extraction on the fixture notes, each emitting the same Graphify-shaped structured fragment (entities, typed relationships, INFERRED/AMBIGUOUS confidence tags) to a per-model file. These per-model files become the gold-standard reference set that local Ollama models are scored against in a later step.
• Establish a fairness contract for the prompt: each subagent receives only the note text plus a shared minimal extraction spec — no CLAUDE.md, no design docs, no project context.
• Drop wall-clock speed as a metric for the Claude reference run (untrackable in this environment; quality only). Speed re-enters later when Ollama models are benchmarked against the reference set.
• Resequence the build order: build the full system against a small fixture set first; defer bulk vault migration and multi-project onboarding until the system is built and validated end-to-end on one small project that contains both code and documents; then onboard remaining projects one at a time, observing and adjusting per project.
• Keep four things explicitly distinct so the runbook does not blur them: benchmark fixtures (needed now, for the gate), bulk vault migration (deferred), the initial mixed code+docs validation project (post-build), and project-by-project rollout.
• Add ADR-013 recording the build-order inversion (vault-migration-first → build-first / migrate-incrementally) and amend the "Decisions locked" pointer in CLAUDE.md.

Capabilities

New Capabilities

• reference-extraction-benchmark: the procedure, fairness contract, output schema, and deliverable prompt file for producing a reusable gold-standard Claude reference extraction set that local models are later scored against.
• incremental-migration: the resequenced build order — build and validate on fixtures plus one small mixed code+docs project before any bulk migration, then onboard projects one at a time.

Modified Capabilities

Impact

• Docs: docs/memory-system/05-implementation-process.md (Step 1 sequencing + Step 2c rewrites), docs/memory-system/03-architecture-decisions.md (new ADR-013), docs/memory-system/04-build-plan.md (reconcile if it states the old order), CLAUDE.md ("Decisions locked" line).
• New deliverable: a committed, copy/paste-able benchmark dispatch prompt file, plus a home for the per-model reference outputs.
• No application code. This change touches design, runbook, and process only.