cc-os/openspec/changes/archive/2026-06-04-incremental-migration-and-reference-benchmark/design.md

Context

cc-os is a design-only repository; the build runbook is docs/memory-system/05-implementation-process.md and Step 2c is its documented gate. The hard constraint driving this change: only the Claude tiers are reachable here (via Claude Code subagents) — local Ollama models cannot be run in this benchmarking environment. The vault is ~/Documents/SecondBrain (ADR-012) under the six-facet taxonomy (ADR-011). Graphify extracts entities plus typed edges plus confidence tags (INFERRED/AMBIGUOUS) from documents via a local SLM, and extracts code via tree-sitter AST (free, no model). The benchmark exists to choose the local doc-extraction model; this change does not make that choice, it produces the reference set that choice will be measured against.

Goals / Non-Goals

Goals:

• A runnable benchmark that produces a reusable, diffable reference set across Claude tiers.
• A fairness contract that keeps the comparison as close to apples-to-apples as the environment allows.
• An incremental, observable build-and-migration path that validates the system before committing the whole vault to it.

Non-Goals:

• Choosing the final extraction model now (the reference set feeds that later decision).
• Pivoting away from local Ollama doc extraction (architecture stays intact per the existing ADRs).
• Running Ollama models within this change.
• Bulk-migrating the vault now.

Decisions

• Reference-set, not model selection. Claude subagents produce gold-standard outputs, not a final pick. Rejected alternative: treating this as a "Claude-as-extraction-backend" pivot — that is an ADR-level architectural shift (cost, privacy) and is out of scope.
• Mimic-extraction task with an explicit embedded output schema. Subagents reason directly to the Graphify-shaped fragment rather than invoking graphify extract, because Ollama backends cannot run here and the goal is tier-vs-tier signal. The output schema (entities, typed relationships, INFERRED/AMBIGUOUS confidence) is embedded verbatim in the prompt so per-model files are diffable against each other now and against Ollama output later. Each subagent writes to its own per-model file.
• Fairness contract: minimal context only. Each subagent receives only the raw note text and the shared extraction spec. It is explicitly instructed not to read repository files (CLAUDE.md, design docs) or pull project context. This is the core property that makes the comparison meaningful.
• Speed dropped for the Claude run. Wall-clock per note is untrackable across dispatched subagents here, so quality is the only metric for the reference run: entity correctness, relationship plausibility/typing, and confidence-tag accuracy. Speed re-enters when local Ollama models are timed against the references.
• Build-first / migrate-incrementally. A 5–10 note fixture set (the variety already called for in Step 1c) feeds the gate immediately; bulk vault migration is deferred to last; first end-to-end validation runs against one small project containing both code and documents. The build-order inversion is recorded in ADR-013, and CLAUDE.md's "Decisions locked" pointer is updated.
• Migration-unit granularity, surfaced not hidden. The first migration unit is named "one small project with both code and documents." Vault notes (local-SLM extraction path) and project code (tree-sitter path) are different extraction paths; design.md surfaces this so the user can react at proposal review rather than discovering it mid-migration.

Risks / Trade-offs

• Fixtures unrepresentative of the real vault → choose deliberate variety per Step 1c (tool note, client/project note, convention note, domain note, one relationship-dense note) and keep them as living fixtures.
• Reference set encodes Claude-tier idiosyncrasies and biases later Ollama scoring → treat the references as a quality ceiling / scoring rubric, not literal ground truth; keep a human in the loop reviewing god-nodes.
• Deferring bulk migration delays real-world validation of the schema at scale → mitigated by the early end-to-end test on one small mixed project before broad rollout.
• Prompt context leakage breaks the fairness contract → the prompt enumerates the only allowed inputs and explicitly forbids reading repo/project files.

Migration Plan

No code migration. The change edits docs and adds ADR-013; the only new artifact is the benchmark prompt file plus a directory for reference outputs (proposed: docs/memory-system/benchmark/). Rollback is a straight revert of the doc edits.

Open Questions

• Migration-unit granularity (whole project repo vs. a vault-note cluster) — first unit is named now; refine after observing the first real migration.
• Where the prompt file and per-model reference outputs live (proposed default: docs/memory-system/benchmark/).
• Which specific small project is the initial mixed code+docs validation target — the user picks this at apply time (candidates under ~/projects/ and ~/dev/).