cc-os/openspec/changes/archive/2026-06-04-incremental-migr.../design.md

## Context

`cc-os` is a design-only repository; the build runbook is `docs/memory-system/05-implementation-process.md` and Step 2c is its documented gate. The hard constraint driving this change: only the Claude tiers are reachable here (via Claude Code subagents) — local Ollama models cannot be run in this benchmarking environment. The vault is `~/Documents/SecondBrain` (ADR-012) under the six-facet taxonomy (ADR-011). Graphify extracts entities plus typed edges plus confidence tags (`INFERRED`/`AMBIGUOUS`) from documents via a local SLM, and extracts code via tree-sitter AST (free, no model). The benchmark exists to choose the local doc-extraction model; this change does not make that choice, it produces the reference set that choice will be measured against.

## Goals / Non-Goals

**Goals:**
- A runnable benchmark that produces a reusable, diffable reference set across Claude tiers.
- A fairness contract that keeps the comparison as close to apples-to-apples as the environment allows.
- An incremental, observable build-and-migration path that validates the system before committing the whole vault to it.

**Non-Goals:**
- Choosing the final extraction model now (the reference set feeds that later decision).
- Pivoting away from local Ollama doc extraction (architecture stays intact per the existing ADRs).
- Running Ollama models within this change.
- Bulk-migrating the vault now.

## Decisions

- **Reference-set, not model selection.** Claude subagents produce gold-standard outputs, not a final pick. Rejected alternative: treating this as a "Claude-as-extraction-backend" pivot — that is an ADR-level architectural shift (cost, privacy) and is out of scope.
- **Mimic-extraction task with an explicit embedded output schema.** Subagents reason directly to the Graphify-shaped fragment rather than invoking `graphify extract`, because Ollama backends cannot run here and the goal is tier-vs-tier signal. The output schema (entities, typed relationships, `INFERRED`/`AMBIGUOUS` confidence) is embedded verbatim in the prompt so per-model files are diffable against each other now and against Ollama output later. Each subagent writes to its own per-model file.
- **Fairness contract: minimal context only.** Each subagent receives only the raw note text and the shared extraction spec. It is explicitly instructed not to read repository files (`CLAUDE.md`, design docs) or pull project context. This is the core property that makes the comparison meaningful.
- **Speed dropped for the Claude run.** Wall-clock per note is untrackable across dispatched subagents here, so quality is the only metric for the reference run: entity correctness, relationship plausibility/typing, and confidence-tag accuracy. Speed re-enters when local Ollama models are timed against the references.
- **Build-first / migrate-incrementally.** A 5–10 note fixture set (the variety already called for in Step 1c) feeds the gate immediately; bulk vault migration is deferred to last; first end-to-end validation runs against one small project containing both code and documents. The build-order inversion is recorded in ADR-013, and `CLAUDE.md`'s "Decisions locked" pointer is updated.
- **Migration-unit granularity, surfaced not hidden.** The first migration unit is named "one small project with both code and documents." Vault notes (local-SLM extraction path) and project code (tree-sitter path) are different extraction paths; design.md surfaces this so the user can react at proposal review rather than discovering it mid-migration.

## Risks / Trade-offs

- Fixtures unrepresentative of the real vault → choose deliberate variety per Step 1c (tool note, client/project note, convention note, domain note, one relationship-dense note) and keep them as living fixtures.
- Reference set encodes Claude-tier idiosyncrasies and biases later Ollama scoring → treat the references as a quality ceiling / scoring rubric, not literal ground truth; keep a human in the loop reviewing god-nodes.
- Deferring bulk migration delays real-world validation of the schema at scale → mitigated by the early end-to-end test on one small mixed project before broad rollout.
- Prompt context leakage breaks the fairness contract → the prompt enumerates the only allowed inputs and explicitly forbids reading repo/project files.

## Migration Plan

No code migration. The change edits docs and adds ADR-013; the only new artifact is the benchmark prompt file plus a directory for reference outputs (proposed: `docs/memory-system/benchmark/`). Rollback is a straight revert of the doc edits.

## Open Questions

- Migration-unit granularity (whole project repo vs. a vault-note cluster) — first unit is named now; refine after observing the first real migration.
- Where the prompt file and per-model reference outputs live (proposed default: `docs/memory-system/benchmark/`).
- Which specific small project is the initial mixed code+docs validation target — the user picks this at apply time (candidates under `~/projects/` and `~/dev/`).
-												Add incremental-migration + reference-benchmark change (ADR-013)

Invert the build order to build-first / migrate-incrementally and
redesign Step 2c as a Claude reference-set benchmark.

- ADR-013 records the build-order inversion; CLAUDE.md locked-decisions
  pointer updated
- New benchmark deliverable under docs/memory-system/benchmark/:
  shared extraction spec + copy/paste dispatch prompt + reference-outputs
- Runbook (05): Step 1 = fixture selection with bulk migration deferred;
  Step 2c = Claude-tiers-only reference set, quality-only metrics, Opus
  as gold-standard rubric (produces references, not a model choice)
- 04-build-plan reconciled to the new build order and benchmark design
- Sync incremental-migration + reference-extraction-benchmark specs into
  openspec/specs/; archive the completed change

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

											
										
										
											2026-06-04 15:17:32 +00:00
+								## Context
 								`cc-os` is a design-only repository; the build runbook is `docs/memory-system/05-implementation-process.md` and Step 2c is its documented gate. The hard constraint driving this change: only the Claude tiers are reachable here (via Claude Code subagents) — local Ollama models cannot be run in this benchmarking environment. The vault is `~/Documents/SecondBrain` (ADR-012) under the six-facet taxonomy (ADR-011). Graphify extracts entities plus typed edges plus confidence tags (`INFERRED`/`AMBIGUOUS`) from documents via a local SLM, and extracts code via tree-sitter AST (free, no model). The benchmark exists to choose the local doc-extraction model; this change does not make that choice, it produces the reference set that choice will be measured against.
 								## Goals / Non-Goals
 								**Goals:**
 								- A runnable benchmark that produces a reusable, diffable reference set across Claude tiers.
 								- A fairness contract that keeps the comparison as close to apples-to-apples as the environment allows.
 								- An incremental, observable build-and-migration path that validates the system before committing the whole vault to it.
 								**Non-Goals:**
 								- Choosing the final extraction model now (the reference set feeds that later decision).
 								- Pivoting away from local Ollama doc extraction (architecture stays intact per the existing ADRs).
 								- Running Ollama models within this change.
 								- Bulk-migrating the vault now.
 								## Decisions
 								- **Reference-set, not model selection.** Claude subagents produce gold-standard outputs, not a final pick. Rejected alternative: treating this as a "Claude-as-extraction-backend" pivot — that is an ADR-level architectural shift (cost, privacy) and is out of scope.
 								- **Mimic-extraction task with an explicit embedded output schema.** Subagents reason directly to the Graphify-shaped fragment rather than invoking `graphify extract`, because Ollama backends cannot run here and the goal is tier-vs-tier signal. The output schema (entities, typed relationships, `INFERRED`/`AMBIGUOUS` confidence) is embedded verbatim in the prompt so per-model files are diffable against each other now and against Ollama output later. Each subagent writes to its own per-model file.
 								- **Fairness contract: minimal context only.** Each subagent receives only the raw note text and the shared extraction spec. It is explicitly instructed not to read repository files (`CLAUDE.md`, design docs) or pull project context. This is the core property that makes the comparison meaningful.
 								- **Speed dropped for the Claude run.** Wall-clock per note is untrackable across dispatched subagents here, so quality is the only metric for the reference run: entity correctness, relationship plausibility/typing, and confidence-tag accuracy. Speed re-enters when local Ollama models are timed against the references.
 								- **Build-first / migrate-incrementally.** A 5–10 note fixture set (the variety already called for in Step 1c) feeds the gate immediately; bulk vault migration is deferred to last; first end-to-end validation runs against one small project containing both code and documents. The build-order inversion is recorded in ADR-013, and `CLAUDE.md`'s "Decisions locked" pointer is updated.
 								- **Migration-unit granularity, surfaced not hidden.** The first migration unit is named "one small project with both code and documents." Vault notes (local-SLM extraction path) and project code (tree-sitter path) are different extraction paths; design.md surfaces this so the user can react at proposal review rather than discovering it mid-migration.
 								## Risks / Trade-offs
 								- Fixtures unrepresentative of the real vault → choose deliberate variety per Step 1c (tool note, client/project note, convention note, domain note, one relationship-dense note) and keep them as living fixtures.
 								- Reference set encodes Claude-tier idiosyncrasies and biases later Ollama scoring → treat the references as a quality ceiling / scoring rubric, not literal ground truth; keep a human in the loop reviewing god-nodes.
 								- Deferring bulk migration delays real-world validation of the schema at scale → mitigated by the early end-to-end test on one small mixed project before broad rollout.
 								- Prompt context leakage breaks the fairness contract → the prompt enumerates the only allowed inputs and explicitly forbids reading repo/project files.
 								## Migration Plan
 								No code migration. The change edits docs and adds ADR-013; the only new artifact is the benchmark prompt file plus a directory for reference outputs (proposed: `docs/memory-system/benchmark/`). Rollback is a straight revert of the doc edits.
 								## Open Questions
 								- Migration-unit granularity (whole project repo vs. a vault-note cluster) — first unit is named now; refine after observing the first real migration.
 								- Where the prompt file and per-model reference outputs live (proposed default: `docs/memory-system/benchmark/`).
 								- Which specific small project is the initial mixed code+docs validation target — the user picks this at apply time (candidates under `~/projects/` and `~/dev/`).