From 77febc8aca6dbf6cbb1b1c071eeddca3a2be2794 Mon Sep 17 00:00:00 2001 From: jared Date: Thu, 4 Jun 2026 11:17:32 -0400 Subject: [PATCH] Add incremental-migration + reference-benchmark change (ADR-013) Invert the build order to build-first / migrate-incrementally and redesign Step 2c as a Claude reference-set benchmark. - ADR-013 records the build-order inversion; CLAUDE.md locked-decisions pointer updated - New benchmark deliverable under docs/memory-system/benchmark/: shared extraction spec + copy/paste dispatch prompt + reference-outputs - Runbook (05): Step 1 = fixture selection with bulk migration deferred; Step 2c = Claude-tiers-only reference set, quality-only metrics, Opus as gold-standard rubric (produces references, not a model choice) - 04-build-plan reconciled to the new build order and benchmark design - Sync incremental-migration + reference-extraction-benchmark specs into openspec/specs/; archive the completed change Co-Authored-By: Claude Opus 4.8 --- CLAUDE.md | 2 +- .../03-architecture-decisions.md | 39 ++++ docs/memory-system/04-build-plan.md | 75 ++++--- .../05-implementation-process.md | 199 +++++++++++------ .../benchmark/dispatch-prompt.md | 207 ++++++++++++++++++ .../benchmark/extraction-spec.md | 84 +++++++ .../benchmark/reference-outputs/README.md | 26 +++ .../.openspec.yaml | 2 + .../design.md | 42 ++++ .../proposal.md | 29 +++ .../specs/incremental-migration/spec.md | 47 ++++ .../reference-extraction-benchmark/spec.md | 61 ++++++ .../tasks.md | 30 +++ openspec/specs/incremental-migration/spec.md | 53 +++++ .../reference-extraction-benchmark/spec.md | 67 ++++++ 15 files changed, 864 insertions(+), 99 deletions(-) create mode 100644 docs/memory-system/benchmark/dispatch-prompt.md create mode 100644 docs/memory-system/benchmark/extraction-spec.md create mode 100644 docs/memory-system/benchmark/reference-outputs/README.md create mode 100644 openspec/changes/archive/2026-06-04-incremental-migration-and-reference-benchmark/.openspec.yaml create mode 100644 openspec/changes/archive/2026-06-04-incremental-migration-and-reference-benchmark/design.md create mode 100644 openspec/changes/archive/2026-06-04-incremental-migration-and-reference-benchmark/proposal.md create mode 100644 openspec/changes/archive/2026-06-04-incremental-migration-and-reference-benchmark/specs/incremental-migration/spec.md create mode 100644 openspec/changes/archive/2026-06-04-incremental-migration-and-reference-benchmark/specs/reference-extraction-benchmark/spec.md create mode 100644 openspec/changes/archive/2026-06-04-incremental-migration-and-reference-benchmark/tasks.md create mode 100644 openspec/specs/incremental-migration/spec.md create mode 100644 openspec/specs/reference-extraction-benchmark/spec.md diff --git a/CLAUDE.md b/CLAUDE.md index 1704d41..a6d2158 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -58,7 +58,7 @@ tag-index CLI and also covers the deferred QMD semantic layer. `04-build-plan.md `06-graphify-evaluation.md` reflect this; if an older doc still describes the Ruby CLI, defer to those two and fix the stale doc. -**Decisions locked (2026-06-04):** Six-facet tag taxonomy + `scope/` (ADR-011); reuse `~/Documents/SecondBrain` vault rather than creating a new one (ADR-012). +**Decisions locked (2026-06-04):** Six-facet tag taxonomy + `scope/` (ADR-011); reuse `~/Documents/SecondBrain` vault rather than creating a new one (ADR-012); build-first / migrate-incrementally — build full system against a fixture set first, defer bulk vault migration to last, onboard projects one at a time (ADR-013). ## OpenSpec workflow diff --git a/docs/memory-system/03-architecture-decisions.md b/docs/memory-system/03-architecture-decisions.md index 5f9e084..9e9be79 100644 --- a/docs/memory-system/03-architecture-decisions.md +++ b/docs/memory-system/03-architecture-decisions.md @@ -1,5 +1,7 @@ # Architecture Decision Records +_Last updated: 2026-06-04_ + A running log of decisions and *why*. Format per entry: Context · Decision · Rationale · Alternatives rejected · Status. Newest decisions extend the log; supersede rather than delete. @@ -256,6 +258,43 @@ _Date: 2026-06-04_ remaining work is mechanical migration. - **Status**: Accepted. +## ADR-013 — Build-first / migrate-incrementally (build-order inversion) + +_Date: 2026-06-04_ + +- **Context**: The build runbook (`05-implementation-process.md`) originally front-loaded bulk + vault migration as Step 1 — migrating all ~20 existing SecondBrain notes and all projects to + the ADR-011 six-facet taxonomy before the system existed to validate them. This committed to + a schema and workflow (the tag taxonomy from ADR-011, the vault-reuse choice from ADR-012, + and Graphify extraction behavior) before any end-to-end path had been exercised. The risk: + locking in an approach that fails at scale, with no feedback loop until the entire vault has + been touched. +- **Decision**: **Invert the build order.** The full system is built and validated against a + small **5–10 note fixture set** first. Bulk vault migration is deferred to the final stage. + The first real-data validation uses **one small project that contains both code AND + documents**, exercising both the local-SLM doc-extraction path and the tree-sitter code path + in the same run. After that single project validates end-to-end, remaining projects are + onboarded **one at a time** with an observe-and-adjust step between each. +- **Rationale**: Validates the ADR-011 taxonomy and ADR-012 vault conventions against the real + Graphify extraction pipeline before the entire vault is committed. The first mixed code+docs + project surfaces both extraction paths (SLM for docs, tree-sitter for code) early, when + corrections are cheap. Per-project rollout keeps the blast radius of any schema or workflow + correction small; each project is an opportunity to observe and adjust rather than discover + problems across 20 notes at once. This is consistent with the "markdown-as-truth, indexes are + disposable" principle (ADR-008): the vault notes are durable, but the extraction schema should + be validated before it shapes all of them. +- **Alternatives rejected**: + - **Keep migration-first (status quo)**: Front-loads all ~20 notes and all projects before + any end-to-end validation exists. Commits to ADR-011's taxonomy and ADR-012's migration + steps against the full vault without a feedback loop — exactly the gap this decision closes. + - **Big-bang migrate everything after build**: Build against fixtures, then migrate all notes + and all projects in one batch at the end. Avoids the pre-build commitment problem but still + risks a single large irreversible migration with no observe-and-adjust loop between units. + Per-project rollout with intermediate checkpoints is strictly safer. +- **Cross-references**: ADR-011 (six-facet tag taxonomy — the schema being validated); + ADR-012 (SecondBrain vault reuse — the migration steps this order defers). +- **Status**: Accepted (updates `05-implementation-process.md` build order). + ## Rejected tools (summary) | Tool | Why rejected for our use | diff --git a/docs/memory-system/04-build-plan.md b/docs/memory-system/04-build-plan.md index 2323584..97742e4 100644 --- a/docs/memory-system/04-build-plan.md +++ b/docs/memory-system/04-build-plan.md @@ -1,5 +1,7 @@ # Build Plan +_Last updated: 2026-06-04_ + How a human builds this system, step by step, and answers to the operational questions: which scripts and hooks, how the AI knows when to write and what conventions to follow, how and when it queries, the CRUD hooks, and how it's packaged as a global Claude Code plugin with @@ -14,6 +16,12 @@ vault and free AST-based code graphs per project. The deferred QMD semantic laye is also skipped — Graphify covers it without vectors. See `06-graphify-evaluation.md` for the full rationale. +**Architecture decision (2026-06-04):** Build order inverted (ADR-013). Build and validate +the full system against a small 5–10 note fixture set first; defer bulk vault migration to last. +Validate end-to-end on one small pilot project containing both code and documents before +onboarding any others. Onboard remaining projects one at a time, with observe-and-adjust between +each. Steps 2d and 2e below are updated accordingly. + --- ## Part A — Build order (human builder's path) @@ -61,39 +69,46 @@ Available models (as of 2026-06-03, in order of interest): - `qwen3.5:2b` — 2.7 GB, smallest, good fallback if VRAM is constrained - `gemma4:e4b` — 9.6 GB, more capable, slower -#### 2c — Model benchmarking (before committing to a model) +#### 2c — Claude reference-set benchmark (THE GATE, before committing to a local model) -Run a small extraction test across all local models plus the three Claude API models. The goal -is to find the best speed/accuracy tradeoff for entity+relationship extraction from vault notes. +This step produces the **gold-standard reference set** — one structured extraction output per +fixture note per Claude tier. It does **not** choose the final extraction model, and it does +**not** measure speed. -**Test set:** 5–10 representative vault notes spanning different note types (tool note, client -note, convention note, domain note). Include one note that is dense with relationships. +**Input:** the 5–10 fixture notes selected in Step 1 (from the runbook). -**What to measure per model:** -1. Extraction speed (tokens/sec or wall-clock time per note) -2. Entity quality: are the right concepts extracted? Any hallucinated entities? -3. Relationship quality: are edges plausible and correctly typed? Missing relationships? -4. Confidence tag accuracy: are `INFERRED` vs `AMBIGUOUS` edges appropriately flagged? +**What to run:** dispatch one Claude Code subagent per tier (Claude-tiers only — Ollama models +are not reachable in this environment): -**Models to test:** -| Model | Backend | Notes | +| Tier | Model ID | Role | |---|---|---| -| `gemma4:e2b` | Ollama local | Primary candidate — fast, 7.2 GB | -| `qwen3.5:2b` | Ollama local | Smallest, fastest | -| `gemma4:e4b` | Ollama local | Highest local quality | -| `claude-haiku-4-5` | Claude API | Baseline API option | -| `claude-sonnet-4-6` | Claude API | Mid-tier reference | -| `claude-opus-4-8` | Claude API | **Gold standard** — judge quality against this | +| Haiku | `claude-haiku-4-5` | Lightweight reference | +| Sonnet | `claude-sonnet-4-6` | Mid-tier reference | +| Opus | `claude-opus-4-8` | **Gold standard** (scoring rubric) | -Run using subagents in Claude Code: dispatch one agent per model, each extracting the same -test set, return structured JSON of entities+relationships. Review god-node quality in -`GRAPH_REPORT.md` after each run. Use Opus output as the scoring rubric for the local models. +Each subagent receives only the fixture note text plus the shared extraction spec — no design +docs, no project context (fairness contract). See ADR-013. -**Decision rule:** Choose the fastest local model whose entity/relationship quality is -"close enough" to Opus (subjective; likely Gemma4:e2b or Gemma4:e4b based on the description). -API models are a fallback option for high-stakes notes, not the default. +**Metrics — quality only (wall-clock speed is explicitly out of scope here):** +1. Entity correctness — right concepts extracted, no hallucinated entities +2. Relationship plausibility and typing — edges plausible, correctly typed, no missing key edges +3. Confidence-tag accuracy — `INFERRED` vs `AMBIGUOUS` applied appropriately + +**Deliverables** (produced by this step): +- Dispatch prompt (copy/paste-able): `docs/memory-system/benchmark/dispatch-prompt.md` +- Shared extraction schema: `docs/memory-system/benchmark/extraction-spec.md` +- Per-tier outputs: `docs/memory-system/benchmark/reference-outputs/..md` + +Opus output is the rubric. **Deferred later step:** local Ollama models (gemma4:e2b, qwen3.5:2b, +gemma4:e4b) are timed AND scored against these references — that scoring run is where speed +re-enters and the final model is chosen. Do not hardcode a model before that run completes. + +Authoritative detail lives in `docs/memory-system/05-implementation-process.md` §2c. + +#### 2d — Initial fixture graph build (ADR-013: small-first) +Run the initial build against the small fixture set (5–10 notes from Step 1/2c), not the +full vault. The bulk vault build is deferred to after the system is validated end-to-end. -#### 2d — Initial vault graph build ```bash graphify extract --path ~/brain --backend ollama --model gemma4:e2b \ --token-budget 512 --max-concurrency 2 @@ -102,8 +117,16 @@ Tune `--token-budget` (semantic chunk size) and `--max-concurrency` based on VRA Review `GRAPH_REPORT.md` — check god nodes make sense (they should be your most-connected tools, clients, and domain concepts). +**Full vault migration** (the `~/brain` build above run over all notes) is the final step — +deferred to after end-to-end validation on the pilot project. Do not bulk-migrate the vault +until the system is verified working on the fixture set and pilot project. + #### 2e — Per-project code graphs (free, no model needed) -For each client project: +**ADR-013 order:** start with ONE pilot project that contains both code and documents; validate +end-to-end before onboarding others. Onboard remaining projects one at a time, with +observe-and-adjust between each. Do not batch all projects at once. + +For the pilot project (and each subsequent project, one at a time): ```bash graphify extract --path ~/projects// --no-docs ``` diff --git a/docs/memory-system/05-implementation-process.md b/docs/memory-system/05-implementation-process.md index bd68b69..d82d385 100644 --- a/docs/memory-system/05-implementation-process.md +++ b/docs/memory-system/05-implementation-process.md @@ -1,6 +1,6 @@ # Implementation Process -_Last updated: 2026-06-04_ | _Status: Ready to plan — awaiting Step 2c benchmark results_ +_Last updated: 2026-06-04_ | _Status: Ready to plan — awaiting Step 2c reference set_ This document distills `04-build-plan.md` into a concrete, staged build process and folds in two locked decisions: **ADR-011** (faceted six-namespace taxonomy) and **ADR-012** (reuse the @@ -11,44 +11,57 @@ for the underlying rationale, query/CRUD conventions, and plugin internals; read > This is still a build process outline, not a detailed implementation plan. The recommended > next move is to feed Step 2 (the critical path) into the **writing-plans skill** to produce an > executable, task-level plan. Most open questions in Steps 3–6 can be defaulted; Step 2c's -> model benchmark is the genuine gate that the rest depends on — do not hardcode a model before -> it runs. +> Claude reference-set run is the genuine gate — it produces the gold-standard rubric against +> which local Ollama models are then scored. Do not hardcode a model before that scoring runs. --- ## Build order -Bottom-up: vault migration first (usable standalone), then Graphify, then hooks, then plugin. +Bottom-up: fixture selection first (feeds the Step 2c gate), then Graphify setup, then hooks, then plugin. Bulk vault migration is deferred to after the system is validated end-to-end. --- -## Step 1 — Adopt and migrate the SecondBrain vault +## Step 1 — Select benchmark fixtures (and defer bulk migration) -**Per ADR-012:** reuse `~/Documents/SecondBrain` as the knowledge vault rather than creating a -new `~/brain`. The vault is already flat, already scoped to durable knowledge, and already -articulates the correct governance philosophy. Migration cost is mechanical. +**Per ADR-012:** `~/Documents/SecondBrain` is the vault. The vault is already flat, already +scoped to durable knowledge, and already articulates the correct governance philosophy. -### 1a — Initialize git +**Bulk vault migration is DEFERRED.** Do not migrate all notes now. The only pre-build activity +in this step is selecting a small fixture set that feeds the Step 2c benchmark gate. Everything +else — tagging ~20 existing notes, updating governance docs, initializing git, fixing broken +references — happens after the system is validated end-to-end. -The vault has no `.git` yet. Initialize it so ADR-008's sync strategy works: +> **Four sequencing concepts to keep distinct — they are not interchangeable:** +> +> 1. **Benchmark fixtures** — 5–10 notes selected NOW, in this step. They feed the Step 2c +> extraction benchmark gate. They are living fixtures, kept and reused throughout development. +> 2. **Bulk vault migration** — converting all ~20 existing notes to the six-facet tag schema, +> updating governance docs, initializing git. DEFERRED to last, after the system is fully +> validated. +> 3. **Initial validation project** — the first real end-to-end test after the system is built: +> one small project that contains BOTH code AND documents. POST-BUILD, before any rollout. +> 4. **Project-by-project rollout** — onboarding remaining projects one at a time, observing +> and adjusting between each. Follows the initial validation project. -```bash -git init ~/Documents/SecondBrain -``` +### 1a — Select fixture notes -### 1b — Update vault governance notes +Choose 5–10 notes from `~/Documents/SecondBrain` with deliberate variety. The selection must +include at minimum: -Update three files to the new schema: +- A **tool note** (e.g., a note about a CLI tool or SaaS product) +- A **client/project note** (scoped, not global) +- A **convention note** (a `type/convention` or workflow note) +- A **domain note** (a topic or subject-area note) +- At least one **relationship-dense note** — a note where several concepts interrelate and + Graphify should emit multiple typed edges and confidence tags -- **`CLAUDE.md`** — update vault path references, Graphify commands, ADR-011 namespace list. -- **`vault-conventions.md`** — update tag-namespace list to the six-facet form; keep the - existing "act without being asked" proactive-retrieval section intact (it's better than - what cc-os had — adopt it, don't overwrite it). -- **Project-config hub note(s)** — update the tag-inference table values to namespaced form - (e.g. `semrush-work → tool/semrush`, `pest-control → domain/pest-control`). Keep the - table; it operationalizes auto-tagging by path pattern and should be preserved. +These notes must be usable as-is by a subagent that receives only the note text plus the shared +extraction spec (see Step 2c). Apply `summary:` frontmatter and the six-facet tag schema to +each fixture note now; do not wait for bulk migration. -**Frontmatter contract** (apply to all notes, new and migrated): +**Frontmatter contract** (required for fixture notes immediately; apply to all notes at bulk +migration time): ```yaml --- @@ -67,34 +80,45 @@ tags: `type/` listed first preserves the vault's existing type-first ordering habit and makes note kind immediately visible. -### 1c — Migrate existing notes (~20) +### 1b — Deferred bulk migration work (do not start until post-validation) -For each existing note: +The following items are deferred to after the system is validated end-to-end on fixtures and +on the initial validation project (see §Post-build sequence below): -1. Add `summary:` frontmatter (one line; write it now — it's the human-authored router hint). -2. Convert flat unnamespaced tags to the six-facet namespaced form. Examples: - - `research` → `type/research` - - `pest-control` → `domain/pest-control` - - `niche-automation-prospecting` → `domain/niche-automation-prospecting` (or split across - `domain/` + `project/` if the note is project-specific) - - `semrush` → `tool/semrush` -3. Add `scope/global` or `scope/project` to each note. +- **Initialize git:** `git init ~/Documents/SecondBrain` (enables ADR-008 sync) +- **Update vault governance notes:** `CLAUDE.md` (vault path refs, Graphify commands, + ADR-011 namespace list), `vault-conventions.md` (six-facet tag list; preserve the + "act without being asked" proactive-retrieval section), and project-config hub note(s) + (update tag-inference table to namespaced form, e.g. `semrush-work → tool/semrush`) +- **Migrate remaining notes (~20):** add `summary:` frontmatter and convert unnamespaced tags + to six-facet form (e.g. `research → type/research`, `pest-control → domain/pest-control`, + `semrush → tool/semrush`). Add `scope/global` or `scope/project` to each note. +- **Create `_templates` for core note types:** `research`, `howto`, `adr`, `hub`. The long + tail stays freeform until a pattern earns a template (per ADR-011). +- **Fix the broken vault search reference:** `vault-conventions.md` (or `CLAUDE.md`) currently + references `~/.claude/scripts/vault_search.rb`, which does not exist. Replace every reference + with `graphify query` (per ADR-010). -Keep 5–10 migrated notes aside as representative test cases for Step 2c's extraction benchmark. -Include variety: a tool note, a client/project note, a convention note, a domain note, and at -least one note dense with relationships between concepts. +### Post-build sequence (after Steps 2–6 are complete) -### 1d — Create `_templates` for core note types only +After the system is built and validated on fixtures: -Add templates for: `research`, `howto`, `adr`, `hub`. The long tail stays freeform until a -pattern earns a template (per ADR-011). Consistent per-type structure improves Graphify's -local-SLM extraction reliability. +**First:** Validate end-to-end on **one small project that contains BOTH code AND documents.** +This project is the initial validation target — not a production migration. It exercises both +extraction paths simultaneously: vault notes use the local-SLM extraction path; project code +uses the tree-sitter AST path. These are different extraction paths, which is precisely why the +first validation project must contain both. Observe how the two paths compose before proceeding. -### 1e — Fix the broken vault search reference +**Then:** Onboard remaining projects **one at a time.** Observe extraction quality, god-node +shape, and hook behavior between each project before onboarding the next. Adjust configuration +as needed. -`vault-conventions.md` (or `CLAUDE.md`) currently references -`~/.claude/scripts/vault_search.rb`, which does not exist. Replace every reference to that -script with `graphify query` (per ADR-010). +**Finally:** Execute bulk vault migration (items in 1b above) and run full vault graph build. + +**Migration-unit granularity open question:** the right unit — a whole project repo, a +vault-note cluster, a client boundary — is not yet established. The first validation project +("one small project with both code and documents") is the anchor; refine the granularity +definition after observing that first real migration. --- @@ -126,37 +150,64 @@ GRAPHIFY_OLLAMA_KEEP_ALIVE=5 # set when packaging in Step 6 Verify context allocation after the first extraction call: `ollama ps` shows allocated context. -### 2c — Model benchmark (THE GATE) +### 2c — Claude reference-set benchmark (THE GATE) -Run a small extraction test before committing to any model. Use the 5–10 representative -migrated notes from Step 1c as the test set. +Produce a reference set of Graphify-shaped extraction outputs before committing to any local +Ollama model. Use the 5–10 fixture notes selected in Step 1a as the input set. -**Models to benchmark:** +**What this step produces — and what it does NOT decide:** -| Model | Backend | Size | Notes | -|---|---|---|---| -| `gemma4:e2b` | Ollama local | 7.2 GB | Primary candidate — fast, large context | -| `qwen3.5:2b` | Ollama local | 2.7 GB | Smallest; fallback if VRAM is constrained | -| `gemma4:e4b` | Ollama local | 9.6 GB | Highest local quality; slower | -| `claude-haiku-4-5` | Claude API | — | API baseline | -| `claude-sonnet-4-6` | Claude API | — | Mid-tier reference | -| `claude-opus-4-8` | Claude API | — | **Gold standard — score local models against this** | +This step produces the **gold-standard reference set**: one structured extraction output per +fixture note per Claude tier. The reference set is the scoring rubric against which local Ollama +models are scored in a later step. This step does NOT choose the final extraction model — +local-model selection happens when Ollama models are timed and scored against these references. -**What to measure per model:** +**Why Claude-only here:** local Ollama models cannot be run in this benchmarking environment. +Only Claude tiers are reachable via dispatched subagents. The Claude outputs serve as quality +anchors; Ollama speed and quality are measured separately, against these anchors. -1. Extraction speed (wall-clock time per note) -2. Entity quality: right concepts extracted, no hallucinated entities -3. Relationship quality: edges plausible and correctly typed, no missing key edges -4. Confidence tag accuracy: `INFERRED` vs `AMBIGUOUS` appropriately flagged +**Dispatch one Claude Code subagent per tier:** -**Decision rule:** choose the fastest local model whose entity/relationship quality is "close -enough" to Opus output. Do not hardcode `gemma4:e2b` — the benchmark decides. +| Tier | Model ID | Role | +|---|---|---| +| Haiku | `claude-haiku-4-5` | Lightweight reference | +| Sonnet | `claude-sonnet-4-6` | Mid-tier reference | +| Opus | `claude-opus-4-8` | **Gold standard** | -Review god-node quality in `GRAPH_REPORT.md` after each run. Opus output is the scoring rubric. +Each subagent receives only the fixture note text plus the shared extraction spec at +`docs/memory-system/benchmark/extraction-spec.md`. It must not read `CLAUDE.md`, design docs, +or pull any project context — each subagent operates with the note and spec alone (fairness +contract). + +Each subagent emits a Graphify-shaped structured fragment containing: +- Extracted entities (named concepts, tools, people, projects) +- Typed relationships between entities +- Confidence tags: `INFERRED` (inferred but not stated) and `AMBIGUOUS` (could be interpreted + multiple ways) applied where appropriate + +**Metrics — quality only (wall-clock speed is explicitly out of scope here):** + +Wall-clock timing is untrackable across dispatched subagents and is not measured in this step. +Speed re-enters the picture in the later Ollama-scoring step, where local models are timed +against these reference outputs. For this step, measure quality only: + +1. **Entity correctness** — right concepts extracted, no hallucinated entities +2. **Relationship plausibility and typing** — edges plausible, correctly typed, no missing key edges +3. **Confidence-tag accuracy** — `INFERRED` vs `AMBIGUOUS` applied appropriately + +**Deliverable files:** + +- Dispatch prompt (copy/paste-able): `docs/memory-system/benchmark/dispatch-prompt.md` +- Shared extraction schema: `docs/memory-system/benchmark/extraction-spec.md` +- Per-model outputs: `docs/memory-system/benchmark/reference-outputs/..md` + where `tier` is one of `haiku`, `sonnet`, or `opus` + +`claude-opus-4-8` output is the gold standard. When Ollama models are benchmarked later, their +outputs are scored by how closely they match the Opus reference for each fixture note. ### 2d — Build the initial vault graph -After 2c picks a model: +After a local Ollama model has been chosen (scored against the Step 2c references): ```bash graphify extract --path ~/Documents/SecondBrain \ @@ -260,7 +311,8 @@ Graphify config. ## Open questions / decisions still to settle These are deferred to build time. Most can be defaulted without blocking; the only genuine gate -is **§6 (Step 2c benchmark)** — model selection is blocked until it runs. +is **§6 (Step 2c benchmark)** — local-model selection is blocked until the reference set exists +and Ollama models can be scored against it. 1. **Vault symlink** — `~/Documents/SecondBrain` is confirmed as the vault (RESOLVED per ADR-012). The open sub-question: symlink it into `~/.claude/memory` or not? Only needed if @@ -280,8 +332,11 @@ is **§6 (Step 2c benchmark)** — model selection is blocked until it runs. summaries. The human (or AI at note-creation time) must write `summary:` frontmatter. Confirm this holds in practice and add a lint/reminder to the memory-write skill if it drifts. -6. **Model benchmark results (Step 2c)** — all model choices are deferred until the benchmark - runs. Do not hardcode `gemma4:e2b` (it's the primary candidate, not the confirmed pick). +6. **Step 2c reference set (Claude gold-standard outputs)** — Step 2c produces the scoring + rubric (Opus output as gold standard, Haiku/Sonnet as lighter-tier references), not the final + model choice. Local Ollama model selection is deferred until those references exist and Ollama + models can be timed and quality-scored against them. Do not hardcode `gemma4:e2b` — the Ollama + scoring step decides. The Step 2c outcome is the reference set, not the chosen model. 7. **memsearch + journal integration** — does memsearch index SessionEnd journal notes or only its own auto-capture? How does the journal pointer injected at SessionStart reference the @@ -300,7 +355,7 @@ is **§6 (Step 2c benchmark)** — model selection is blocked until it runs. ## Recommended next move Turn **Step 2** into a detailed implementation plan via the **writing-plans skill**. It is the -critical path: Graphify install, Ollama configuration, the model benchmark (2c), and the initial -graph build are the smallest set of tasks that unblock everything else. Open questions §1–5 and -§7–8 can be defaulted or deferred; only §6 (the benchmark) genuinely blocks model-dependent -decisions. +critical path: Graphify install, Ollama configuration, the Step 2c reference-set run, and the +initial graph build are the smallest set of tasks that unblock everything else. Open questions +§1–5 and §7–8 can be defaulted or deferred; only §6 (producing the reference set and then +scoring Ollama models against it) genuinely blocks model-dependent decisions. diff --git a/docs/memory-system/benchmark/dispatch-prompt.md b/docs/memory-system/benchmark/dispatch-prompt.md new file mode 100644 index 0000000..a6d6734 --- /dev/null +++ b/docs/memory-system/benchmark/dispatch-prompt.md @@ -0,0 +1,207 @@ +# Claude-Tier Reference-Set Benchmark — Dispatch Prompt + +_Last updated: 2026-06-04_ +_Status: active — copy-paste this into a Claude Code session to generate the reference set_ + +## Overview + +This prompt dispatches **one subagent per Claude tier** (`claude-haiku-4-5`, `claude-sonnet-4-6`, +`claude-opus-4-8`) over each fixture note selected in Step 1c of the build runbook. Each subagent +emits a Graphify-shaped structured fragment. The per-model output files become the **gold-standard +reference set** against which local Ollama doc-extraction models are scored in a later step. + +**`claude-opus-4-8`'s output is the gold-standard rubric.** + +Evaluation is on **quality only**: entity correctness, relationship plausibility and typing, and +confidence-tag accuracy. Wall-clock speed is NOT a metric for this Claude reference run — speed +re-enters only when local Ollama models are timed against this reference set. + +--- + +## Step 1 — Select Fixture Notes (operator action, before dispatching) + +Fixture notes are selected at build time from `~/Documents/SecondBrain` per Step 1c of the +runbook. Choose **5–10 notes** with deliberate variety: + +- One **tool note** (documents a specific tool or library) +- One **client/project note** (describes a client engagement or project) +- One **convention note** (captures a working convention or practice) +- One **domain note** (covers a knowledge or technical domain) +- One **relationship-dense note** (many named entities or cross-references) +- Additional notes as needed for coverage + +Assign each note a **kebab-case slug** (used in output filenames). Populate the placeholder list +below before dispatching: + +``` +# FIXTURES — populate before running +# Format: : +FIXTURES=( + # : + # : + # ... (5-10 entries, one per selected note) +) +``` + +--- + +## Step 2 — Output File Convention + +For each fixture note, the run produces **one output file per Claude tier** — three files per note, +`3 × N` files total (where N is the number of fixture notes). + +Output path pattern: + +``` +docs/memory-system/benchmark/reference-outputs/..md +``` + +Where `` is one of: `haiku`, `sonnet`, `opus` + +Examples (using a hypothetical slug `graphify-tool-overview`): +- `reference-outputs/graphify-tool-overview.haiku.md` +- `reference-outputs/graphify-tool-overview.sonnet.md` +- `reference-outputs/graphify-tool-overview.opus.md` + +Each file contains that tier's extraction fragment for **one fixture note** — the `note_slug:` +block for that note only. Do not combine multiple notes into one file. + +--- + +## Step 3 — Dispatch (copy-paste this block into a Claude Code session) + +> **Operator:** replace ``, ``, and model names as needed. Dispatch all +> three subagents for each fixture note. They can run in parallel across tiers for the same note. + +--- + +### Dispatch template (repeat for each fixture note × each tier) + +``` +Dispatch a subagent using model to perform the following task. + +=== FAIRNESS CONTRACT === +You will receive exactly two inputs: + 1. The raw text of one vault note (below). + 2. The shared extraction spec and output schema (below). + +You MUST NOT read any repository files (CLAUDE.md, design docs, specs, tasks), access the +vault directory structure, or use any project or system context. If your environment has +injected any such context automatically, you must ignore it entirely — treat it as if it +does not exist. The only allowed inputs are the note text and the extraction spec below. +=== END FAIRNESS CONTRACT === + +=== NOTE TEXT === + +=== END NOTE TEXT === + +=== EXTRACTION SPEC AND OUTPUT SCHEMA === + +## Facet Vocabulary (closed) + +Notes carry six flat, namespaced facets plus one scope tag. These are the ONLY valid values +for the optional `facet` field on entities. Do not invent new namespaces. + +| Prefix | Meaning | +|--------------|-----------------------------------------| +| `type/` | What kind of thing the note is about | +| `client/` | A client identity | +| `project/` | A project name | +| `domain/` | A knowledge or technical domain | +| `tool/` | A specific tool, library, or CLI | +| `convention/`| A working convention or practice | +| `scope/` | Applicability scope (cross-cutting tag) | + +An entity should carry a `facet` value only when the note text directly supports mapping it +to one of the above. Absence of a `facet` field is correct when no mapping is warranted. + +## Output Schema + +Emit ONLY the following YAML structure. No other keys, no prose, no summary. + +```yaml +# --- Graphify extraction fragment --- +# One block per fixture note. Repeat this structure for each note. + +note_slug: # operator fills this in at run time + +entities: + - name: # exact or near-exact name as it appears in the note + type: # e.g. Person, Tool, Project, Concept, Convention, Client, Domain + facet: # OPTIONAL — must be one of the seven prefixes above, e.g. "tool/graphify" + confidence: # OPTIONAL — omit if the entity is directly stated + # Values: INFERRED | AMBIGUOUS + +relationships: + - source: # must match a name in the entities list above + type: # free-text verb phrase, e.g. "uses", "depends_on", "implements", "replaces" + target: # must match a name in the entities list above + confidence: # OPTIONAL — omit if the relationship is directly stated + # Values: INFERRED | AMBIGUOUS +``` + +## Confidence tag semantics + +| Tag | Meaning | +|------------|--------------------------------------------------------------------------------------| +| _(absent)_ | The entity or relationship is **directly stated** in the note text. | +| `INFERRED` | Not stated literally but **reasonably deducible** from the note text alone. | +| `AMBIGUOUS`| Supportable by the text but **uncertain or admits multiple readings**. | + +## Rules + +- **Entities only:** extract entities that are named (not vague category references). +- **No invented names:** entity names must be grounded in the note text. +- **Relationship types:** free-text verb phrases; do not normalize to a fixed vocabulary. +- **Facet mapping:** use the closed vocabulary above; only assign when the note text supports it. +- **No extra keys:** do not add summaries, scores, embeddings, or metadata fields. +- **One block per note:** if processing multiple fixture notes, emit one `note_slug:` block per + note, separated by `---`. + +=== END EXTRACTION SPEC === + +Write your output to: + docs/memory-system/benchmark/reference-outputs/..md + +where is the kebab-case slug for this note and is one of: haiku, sonnet, opus. + +The file must contain ONLY the YAML extraction fragment — no preamble, no explanation. +``` + +**MODEL and TIER substitutions:** + +| Dispatch | MODEL | TIER | +|----------|------------------------|---------| +| 1st | `claude-haiku-4-5` | `haiku` | +| 2nd | `claude-sonnet-4-6` | `sonnet`| +| 3rd | `claude-opus-4-8` | `opus` | + +--- + +## Step 4 — After All Subagents Complete + +1. Verify `3 × N` files exist in `docs/memory-system/benchmark/reference-outputs/`. +2. For each fixture note, diff the three tier files side by side to understand tier-level + disagreements in entity recognition, relationship typing, and confidence tagging. +3. `claude-opus-4-8`'s `.opus.md` file for each note is the **gold-standard rubric** against + which local Ollama extraction will later be scored. +4. Do NOT run Ollama models in this step — that is a separate later step that uses these files + as the scoring reference. + +--- + +## Notes on the Fairness Contract + +The fairness contract requires that each subagent reasons from **note text + extraction spec +only**. Two mechanisms can break this: + +1. **Deliberate reads:** the subagent reads repository files or the vault. The dispatch template + forbids this explicitly. +2. **Injected context:** Claude Code automatically injects `CLAUDE.md` and project context into + subagent sessions. The dispatch template explicitly instructs the subagent to **ignore any + injected context** and treat it as non-existent. This is the critical instruction — "don't + read files" is not sufficient on its own. + +If you observe tier outputs that appear to reflect knowledge of the vault structure or system +design (beyond what the note text contains), treat that output as contaminated and re-run that +subagent with stronger isolation instructions. diff --git a/docs/memory-system/benchmark/extraction-spec.md b/docs/memory-system/benchmark/extraction-spec.md new file mode 100644 index 0000000..cf7978b --- /dev/null +++ b/docs/memory-system/benchmark/extraction-spec.md @@ -0,0 +1,84 @@ +# Graphify Document-Extraction Benchmark — Shared Extraction Spec + +_Last updated: 2026-06-04_ +_Status: active — used as the shared schema for the Claude-tier reference-set benchmark_ + +## Purpose + +This spec defines the **minimal structured fragment** that every benchmarking subagent (one per +Claude tier) must emit when given a raw vault note. The fragment mimics what Graphify's local SLM +doc-extraction step produces: a list of **entities**, a list of **typed relationships** (edges +between entities), and per-item **confidence tags** where applicable. + +The schema is intentionally minimal. It exists so that: +1. Per-model output files are **diffable entity-by-entity and edge-by-edge** across tiers. +2. The same schema is used unchanged when local Ollama models are later scored against this + reference set — making the comparison apples-to-apples. + +Subagents receive only the raw note text plus this spec. No repository files, no vault structure, +no project context. If any such context is injected by the environment, it must be ignored. + +--- + +## Facet Vocabulary (closed) + +Notes in the SecondBrain vault carry six flat, namespaced facets plus one scope tag. These are the +**only** valid values for the optional `facet` field on entities. Do not invent new namespaces. + +| Prefix | Meaning | +|--------------|-----------------------------------------| +| `type/` | What kind of thing the note is about | +| `client/` | A client identity | +| `project/` | A project name | +| `domain/` | A knowledge or technical domain | +| `tool/` | A specific tool, library, or CLI | +| `convention/`| A working convention or practice | +| `scope/` | Applicability scope (cross-cutting tag) | + +An entity should carry a `facet` value only when the note text directly supports mapping it to one +of the above. Absence of a `facet` field is correct when no mapping is warranted. + +--- + +## Output Schema + +The required output format. Every subagent MUST emit this structure and no other top-level keys. + +```yaml +# --- Graphify extraction fragment --- +# One block per fixture note. Repeat this structure for each note. + +note_slug: # operator fills this in at run time + +entities: + - name: # exact or near-exact name as it appears in the note + type: # e.g. Person, Tool, Project, Concept, Convention, Client, Domain + facet: # OPTIONAL — must be one of the seven prefixes above, e.g. "tool/graphify" + confidence: # OPTIONAL — omit if the entity is directly stated + # Values: INFERRED | AMBIGUOUS + +relationships: + - source: # must match a name in the entities list above + type: # free-text verb phrase, e.g. "uses", "depends_on", "implements", "replaces" + target: # must match a name in the entities list above + confidence: # OPTIONAL — omit if the relationship is directly stated + # Values: INFERRED | AMBIGUOUS +``` + +### Confidence tag semantics + +| Tag | Meaning | +|------------|--------------------------------------------------------------------------------------| +| _(absent)_ | The entity or relationship is **directly stated** in the note text. | +| `INFERRED` | Not stated literally but **reasonably deducible** from the note text alone. | +| `AMBIGUOUS`| Supportable by the text but **uncertain or admits multiple readings**. | + +### Rules + +- **Entities only:** extract entities that are named (not vague category references). +- **No invented names:** entity names must be grounded in the note text. +- **Relationship types:** free-text verb phrases; do not normalize to a fixed vocabulary. +- **Facet mapping:** use the closed vocabulary above; only assign when the note text supports it. +- **No extra keys:** do not add summaries, scores, embeddings, or metadata fields. +- **One block per note:** if processing multiple fixture notes, emit one `note_slug:` block per note, + separated by `---`. diff --git a/docs/memory-system/benchmark/reference-outputs/README.md b/docs/memory-system/benchmark/reference-outputs/README.md new file mode 100644 index 0000000..2ee3ce9 --- /dev/null +++ b/docs/memory-system/benchmark/reference-outputs/README.md @@ -0,0 +1,26 @@ +# Reference Outputs + +_Last updated: 2026-06-04_ + +This directory holds the per-model Graphify-shaped extraction fragments generated by the +Claude-tier reference-set benchmark (see `../dispatch-prompt.md`). + +## File naming convention + +``` +..md +``` + +Where: +- `` is the kebab-case identifier for the fixture note (assigned in Step 1c of the + build runbook) +- `` is one of: `haiku`, `sonnet`, `opus` + +## Contents + +Each file contains the YAML extraction fragment for one fixture note as produced by one Claude +tier. Files are generated by running the dispatch prompt — they do not exist until that step +is executed. + +`claude-opus-4-8`'s `.opus.md` files are the **gold-standard rubric** against which local Ollama +doc-extraction models are scored in a later step. diff --git a/openspec/changes/archive/2026-06-04-incremental-migration-and-reference-benchmark/.openspec.yaml b/openspec/changes/archive/2026-06-04-incremental-migration-and-reference-benchmark/.openspec.yaml new file mode 100644 index 0000000..f617bd1 --- /dev/null +++ b/openspec/changes/archive/2026-06-04-incremental-migration-and-reference-benchmark/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-06-04 diff --git a/openspec/changes/archive/2026-06-04-incremental-migration-and-reference-benchmark/design.md b/openspec/changes/archive/2026-06-04-incremental-migration-and-reference-benchmark/design.md new file mode 100644 index 0000000..b54a679 --- /dev/null +++ b/openspec/changes/archive/2026-06-04-incremental-migration-and-reference-benchmark/design.md @@ -0,0 +1,42 @@ +## Context + +`cc-os` is a design-only repository; the build runbook is `docs/memory-system/05-implementation-process.md` and Step 2c is its documented gate. The hard constraint driving this change: only the Claude tiers are reachable here (via Claude Code subagents) — local Ollama models cannot be run in this benchmarking environment. The vault is `~/Documents/SecondBrain` (ADR-012) under the six-facet taxonomy (ADR-011). Graphify extracts entities plus typed edges plus confidence tags (`INFERRED`/`AMBIGUOUS`) from documents via a local SLM, and extracts code via tree-sitter AST (free, no model). The benchmark exists to choose the local doc-extraction model; this change does not make that choice, it produces the reference set that choice will be measured against. + +## Goals / Non-Goals + +**Goals:** +- A runnable benchmark that produces a reusable, diffable reference set across Claude tiers. +- A fairness contract that keeps the comparison as close to apples-to-apples as the environment allows. +- An incremental, observable build-and-migration path that validates the system before committing the whole vault to it. + +**Non-Goals:** +- Choosing the final extraction model now (the reference set feeds that later decision). +- Pivoting away from local Ollama doc extraction (architecture stays intact per the existing ADRs). +- Running Ollama models within this change. +- Bulk-migrating the vault now. + +## Decisions + +- **Reference-set, not model selection.** Claude subagents produce gold-standard outputs, not a final pick. Rejected alternative: treating this as a "Claude-as-extraction-backend" pivot — that is an ADR-level architectural shift (cost, privacy) and is out of scope. +- **Mimic-extraction task with an explicit embedded output schema.** Subagents reason directly to the Graphify-shaped fragment rather than invoking `graphify extract`, because Ollama backends cannot run here and the goal is tier-vs-tier signal. The output schema (entities, typed relationships, `INFERRED`/`AMBIGUOUS` confidence) is embedded verbatim in the prompt so per-model files are diffable against each other now and against Ollama output later. Each subagent writes to its own per-model file. +- **Fairness contract: minimal context only.** Each subagent receives only the raw note text and the shared extraction spec. It is explicitly instructed not to read repository files (`CLAUDE.md`, design docs) or pull project context. This is the core property that makes the comparison meaningful. +- **Speed dropped for the Claude run.** Wall-clock per note is untrackable across dispatched subagents here, so quality is the only metric for the reference run: entity correctness, relationship plausibility/typing, and confidence-tag accuracy. Speed re-enters when local Ollama models are timed against the references. +- **Build-first / migrate-incrementally.** A 5–10 note fixture set (the variety already called for in Step 1c) feeds the gate immediately; bulk vault migration is deferred to last; first end-to-end validation runs against one small project containing both code and documents. The build-order inversion is recorded in ADR-013, and `CLAUDE.md`'s "Decisions locked" pointer is updated. +- **Migration-unit granularity, surfaced not hidden.** The first migration unit is named "one small project with both code and documents." Vault notes (local-SLM extraction path) and project code (tree-sitter path) are different extraction paths; design.md surfaces this so the user can react at proposal review rather than discovering it mid-migration. + +## Risks / Trade-offs + +- Fixtures unrepresentative of the real vault → choose deliberate variety per Step 1c (tool note, client/project note, convention note, domain note, one relationship-dense note) and keep them as living fixtures. +- Reference set encodes Claude-tier idiosyncrasies and biases later Ollama scoring → treat the references as a quality ceiling / scoring rubric, not literal ground truth; keep a human in the loop reviewing god-nodes. +- Deferring bulk migration delays real-world validation of the schema at scale → mitigated by the early end-to-end test on one small mixed project before broad rollout. +- Prompt context leakage breaks the fairness contract → the prompt enumerates the only allowed inputs and explicitly forbids reading repo/project files. + +## Migration Plan + +No code migration. The change edits docs and adds ADR-013; the only new artifact is the benchmark prompt file plus a directory for reference outputs (proposed: `docs/memory-system/benchmark/`). Rollback is a straight revert of the doc edits. + +## Open Questions + +- Migration-unit granularity (whole project repo vs. a vault-note cluster) — first unit is named now; refine after observing the first real migration. +- Where the prompt file and per-model reference outputs live (proposed default: `docs/memory-system/benchmark/`). +- Which specific small project is the initial mixed code+docs validation target — the user picks this at apply time (candidates under `~/projects/` and `~/dev/`). diff --git a/openspec/changes/archive/2026-06-04-incremental-migration-and-reference-benchmark/proposal.md b/openspec/changes/archive/2026-06-04-incremental-migration-and-reference-benchmark/proposal.md new file mode 100644 index 0000000..698b8d9 --- /dev/null +++ b/openspec/changes/archive/2026-06-04-incremental-migration-and-reference-benchmark/proposal.md @@ -0,0 +1,29 @@ +## Why + +Two gaps block executing the build runbook (`docs/memory-system/05-implementation-process.md`). First, Step 2c's model benchmark assumes head-to-head runs across local Ollama models and Claude API models, but the only models reachable from this environment are the Claude tiers via Claude Code subagents — so the benchmark cannot be run as written, and nothing captures a gold-standard to score local models against later. Second, the runbook front-loads bulk vault migration (all ~20 notes and all projects) as Step 1, committing to a schema and workflow before the system exists to validate them. + +## What Changes + +- Redesign Step 2c as a **reference-set benchmark**: a copy/paste-able Claude Code prompt — written as a committed file — that dispatches one subagent per Claude tier (`claude-haiku-4-5`, `claude-sonnet-4-6`, `claude-opus-4-8`) to mimic Graphify doc extraction on the fixture notes, each emitting the same Graphify-shaped structured fragment (entities, typed relationships, `INFERRED`/`AMBIGUOUS` confidence tags) to a per-model file. These per-model files become the gold-standard reference set that local Ollama models are scored against in a later step. +- Establish a **fairness contract** for the prompt: each subagent receives only the note text plus a shared minimal extraction spec — no `CLAUDE.md`, no design docs, no project context. +- Drop wall-clock speed as a metric for the Claude reference run (untrackable in this environment; quality only). Speed re-enters later when Ollama models are benchmarked against the reference set. +- **Resequence the build order**: build the full system against a small fixture set first; defer bulk vault migration and multi-project onboarding until the system is built and validated end-to-end on one small project that contains both code and documents; then onboard remaining projects one at a time, observing and adjusting per project. +- Keep four things explicitly distinct so the runbook does not blur them: benchmark fixtures (needed now, for the gate), bulk vault migration (deferred), the initial mixed code+docs validation project (post-build), and project-by-project rollout. +- Add **ADR-013** recording the build-order inversion (vault-migration-first → build-first / migrate-incrementally) and amend the "Decisions locked" pointer in `CLAUDE.md`. + +## Capabilities + +### New Capabilities + +- `reference-extraction-benchmark`: the procedure, fairness contract, output schema, and deliverable prompt file for producing a reusable gold-standard Claude reference extraction set that local models are later scored against. +- `incremental-migration`: the resequenced build order — build and validate on fixtures plus one small mixed code+docs project before any bulk migration, then onboard projects one at a time. + +### Modified Capabilities + + + +## Impact + +- Docs: `docs/memory-system/05-implementation-process.md` (Step 1 sequencing + Step 2c rewrites), `docs/memory-system/03-architecture-decisions.md` (new ADR-013), `docs/memory-system/04-build-plan.md` (reconcile if it states the old order), `CLAUDE.md` ("Decisions locked" line). +- New deliverable: a committed, copy/paste-able benchmark dispatch prompt file, plus a home for the per-model reference outputs. +- No application code. This change touches design, runbook, and process only. diff --git a/openspec/changes/archive/2026-06-04-incremental-migration-and-reference-benchmark/specs/incremental-migration/spec.md b/openspec/changes/archive/2026-06-04-incremental-migration-and-reference-benchmark/specs/incremental-migration/spec.md new file mode 100644 index 0000000..a7b9254 --- /dev/null +++ b/openspec/changes/archive/2026-06-04-incremental-migration-and-reference-benchmark/specs/incremental-migration/spec.md @@ -0,0 +1,47 @@ +## ADDED Requirements + +### Requirement: System is validated before bulk migration + +The build SHALL be completed and validated end-to-end against a small fixture set and one small project containing both code and documents before any bulk vault migration is performed. + +#### Scenario: Fixtures feed the gate without bulk migration + +- **WHEN** the Step 2c benchmark gate is run +- **THEN** it uses a 5–10 note fixture set +- **AND** the remaining vault notes are not migrated at that point + +#### Scenario: First end-to-end validation is a mixed project + +- **WHEN** the system is first validated end-to-end +- **THEN** the target is one small project containing both code and documents +- **AND** validation covers both the document extraction path and the code (tree-sitter) path + +### Requirement: Projects are onboarded one at a time + +After the system is validated, projects SHALL be onboarded individually, with observation and adjustment between each, rather than migrated in bulk. + +#### Scenario: One project at a time + +- **WHEN** a project is onboarded after initial validation +- **THEN** it is migrated on its own +- **AND** its migration is observed and the process adjusted before the next project is onboarded + +### Requirement: Bulk vault migration is deferred to last + +Bulk migration of the full vault SHALL be deferred until after the system is built and validated, rather than performed as the first build step. + +#### Scenario: Bulk migration ordering + +- **WHEN** the build order is followed +- **THEN** bulk vault migration occurs after system validation +- **AND** only the fixture notes are migrated beforehand + +### Requirement: Build-order inversion is recorded in an ADR + +The inversion of the documented build order (from vault-migration-first to build-first / migrate-incrementally) SHALL be recorded in a new ADR and reflected in the project's locked-decisions pointer. + +#### Scenario: ADR captures the inversion + +- **WHEN** the build order is resequenced +- **THEN** a new ADR (ADR-013) records the inversion and its rationale +- **AND** the "Decisions locked" pointer in `CLAUDE.md` is updated to reference it diff --git a/openspec/changes/archive/2026-06-04-incremental-migration-and-reference-benchmark/specs/reference-extraction-benchmark/spec.md b/openspec/changes/archive/2026-06-04-incremental-migration-and-reference-benchmark/specs/reference-extraction-benchmark/spec.md new file mode 100644 index 0000000..3eac838 --- /dev/null +++ b/openspec/changes/archive/2026-06-04-incremental-migration-and-reference-benchmark/specs/reference-extraction-benchmark/spec.md @@ -0,0 +1,61 @@ +## ADDED Requirements + +### Requirement: Reference set is produced from Claude tiers + +The benchmark SHALL produce a reusable reference set by dispatching one Claude Code subagent per Claude tier (`claude-haiku-4-5`, `claude-sonnet-4-6`, `claude-opus-4-8`) over the fixture notes, with each subagent's output written to a distinct per-model file. + +#### Scenario: One reference file per model + +- **WHEN** the benchmark prompt is run on the fixture set +- **THEN** a separate output file is produced for each of the three Claude tiers +- **AND** each file contains that tier's extraction result for every fixture note + +#### Scenario: Opus output is the scoring rubric + +- **WHEN** the reference set is reviewed +- **THEN** the `claude-opus-4-8` output is treated as the gold-standard rubric against which the other tiers, and later the local Ollama models, are scored + +### Requirement: Dispatch prompt enforces a fairness contract + +The benchmark prompt SHALL give each subagent only the raw note text and a shared minimal extraction spec, and SHALL forbid reading repository files or pulling project context. + +#### Scenario: Minimal context per subagent + +- **WHEN** a subagent is dispatched for a fixture note +- **THEN** its input is limited to the note text and the shared extraction spec +- **AND** it is explicitly instructed not to read `CLAUDE.md`, design docs, or other project context + +### Requirement: Output conforms to the Graphify-shaped schema + +Each subagent SHALL emit a structured fragment matching the shape Graphify produces — entities, typed relationships, and `INFERRED`/`AMBIGUOUS` confidence tags — so per-model outputs are diffable against each other and against later Ollama output. + +#### Scenario: Schema embedded in the prompt + +- **WHEN** the benchmark prompt is authored +- **THEN** the required output schema is embedded verbatim in the prompt +- **AND** every per-model output file follows that schema + +#### Scenario: Outputs are diffable + +- **WHEN** two per-model output files for the same fixture note are compared +- **THEN** they share a common structure that allows entity-by-entity and edge-by-edge comparison + +### Requirement: Claude reference run is scored on quality only + +The Claude reference run SHALL be evaluated on extraction quality only — entity correctness, relationship plausibility and typing, and confidence-tag accuracy — and SHALL NOT use wall-clock speed as a metric. + +#### Scenario: Speed excluded from the reference run + +- **WHEN** the Claude reference run is evaluated +- **THEN** wall-clock time per note is not used as a metric +- **AND** speed is reintroduced only when local Ollama models are later benchmarked against the reference set + +### Requirement: Prompt is a committed, reusable file + +The benchmark dispatch prompt SHALL be written as a committed file that can be copy/pasted into a Claude Code session to reproduce the run. + +#### Scenario: Prompt persisted as a file + +- **WHEN** the benchmark is set up +- **THEN** the dispatch prompt exists as a committed file in the repository +- **AND** running it does not require reconstructing the prompt from memory diff --git a/openspec/changes/archive/2026-06-04-incremental-migration-and-reference-benchmark/tasks.md b/openspec/changes/archive/2026-06-04-incremental-migration-and-reference-benchmark/tasks.md new file mode 100644 index 0000000..c63b51e --- /dev/null +++ b/openspec/changes/archive/2026-06-04-incremental-migration-and-reference-benchmark/tasks.md @@ -0,0 +1,30 @@ +## 1. Record the decision + +- [x] 1.1 Add ADR-013 to `docs/memory-system/03-architecture-decisions.md` recording the build-order inversion (vault-migration-first → build-first / migrate-incrementally), with rationale and rejected alternatives +- [x] 1.2 Update the "Decisions locked" pointer in `CLAUDE.md` to reference ADR-013 + +## 2. Author the benchmark prompt deliverable + +- [x] 2.1 Choose and create the home for the prompt and per-model reference outputs (default `docs/memory-system/benchmark/`) +- [x] 2.2 Write the shared minimal extraction spec (entities, typed relationships, `INFERRED`/`AMBIGUOUS` confidence) as the schema the subagents must emit +- [x] 2.3 Write the copy/paste-able dispatch prompt file: one subagent per Claude tier (`claude-haiku-4-5`, `claude-sonnet-4-6`, `claude-opus-4-8`), fairness contract (note text + spec only; no repo/project context), per-model output files +- [x] 2.4 Confirm the prompt embeds the output schema verbatim and names the fixture inputs and per-model output paths + +## 3. Rewrite Step 2c (benchmark) in the runbook + +- [x] 3.1 Replace Step 2c's model table/metrics with the reference-set design: Claude tiers only, quality-only metrics, speed deferred to the later Ollama run +- [x] 3.2 State that the per-model reference set is the rubric local Ollama models are scored against later, and point to the prompt file +- [x] 3.3 Update Open question §6 to reflect that the Claude reference run produces references (not the final model choice) + +## 4. Resequence Step 1 (migration) in the runbook + +- [x] 4.1 Rewrite Step 1 so bulk vault migration is deferred; keep only fixture-note selection (5–10 notes, deliberate variety) as the pre-build step +- [x] 4.2 Add the post-build sequence: validate end-to-end on one small mixed code+docs project, then onboard projects one at a time with observe-and-adjust between each +- [x] 4.3 Make the four distinct items explicit in the runbook: benchmark fixtures / deferred bulk migration / initial mixed validation project / project-by-project rollout +- [x] 4.4 Note the migration-unit granularity question and that vault notes (SLM path) and project code (tree-sitter path) are different extraction paths + +## 5. Reconcile and verify + +- [x] 5.1 Check `docs/memory-system/04-build-plan.md` for any statement of the old "migration first" order and reconcile it with ADR-013 +- [x] 5.2 Update the `_Last updated:_` / status lines on every edited design doc +- [x] 5.3 Re-read the edited runbook end-to-end to confirm Step 1 and Step 2c are internally consistent and the four distinct items are not blurred diff --git a/openspec/specs/incremental-migration/spec.md b/openspec/specs/incremental-migration/spec.md new file mode 100644 index 0000000..ff727d5 --- /dev/null +++ b/openspec/specs/incremental-migration/spec.md @@ -0,0 +1,53 @@ +# Spec: Incremental Migration + +## Purpose + +Defines the ordering constraints for onboarding the memory system into production: the system must be built and validated before any migration is performed, projects are onboarded one at a time, and bulk vault migration is deferred to last. + +## Requirements + +### Requirement: System is validated before bulk migration + +The build SHALL be completed and validated end-to-end against a small fixture set and one small project containing both code and documents before any bulk vault migration is performed. + +#### Scenario: Fixtures feed the gate without bulk migration + +- **WHEN** the Step 2c benchmark gate is run +- **THEN** it uses a 5–10 note fixture set +- **AND** the remaining vault notes are not migrated at that point + +#### Scenario: First end-to-end validation is a mixed project + +- **WHEN** the system is first validated end-to-end +- **THEN** the target is one small project containing both code and documents +- **AND** validation covers both the document extraction path and the code (tree-sitter) path + +### Requirement: Projects are onboarded one at a time + +After the system is validated, projects SHALL be onboarded individually, with observation and adjustment between each, rather than migrated in bulk. + +#### Scenario: One project at a time + +- **WHEN** a project is onboarded after initial validation +- **THEN** it is migrated on its own +- **AND** its migration is observed and the process adjusted before the next project is onboarded + +### Requirement: Bulk vault migration is deferred to last + +Bulk migration of the full vault SHALL be deferred until after the system is built and validated, rather than performed as the first build step. + +#### Scenario: Bulk migration ordering + +- **WHEN** the build order is followed +- **THEN** bulk vault migration occurs after system validation +- **AND** only the fixture notes are migrated beforehand + +### Requirement: Build-order inversion is recorded in an ADR + +The inversion of the documented build order (from vault-migration-first to build-first / migrate-incrementally) SHALL be recorded in a new ADR and reflected in the project's locked-decisions pointer. + +#### Scenario: ADR captures the inversion + +- **WHEN** the build order is resequenced +- **THEN** a new ADR (ADR-013) records the inversion and its rationale +- **AND** the "Decisions locked" pointer in `CLAUDE.md` is updated to reference it diff --git a/openspec/specs/reference-extraction-benchmark/spec.md b/openspec/specs/reference-extraction-benchmark/spec.md new file mode 100644 index 0000000..f3802e2 --- /dev/null +++ b/openspec/specs/reference-extraction-benchmark/spec.md @@ -0,0 +1,67 @@ +# Spec: Reference Extraction Benchmark + +## Purpose + +Defines the process for producing a reusable Claude-tier reference set that serves as the quality rubric against which local Ollama extraction models are later scored. + +## Requirements + +### Requirement: Reference set is produced from Claude tiers + +The benchmark SHALL produce a reusable reference set by dispatching one Claude Code subagent per Claude tier (`claude-haiku-4-5`, `claude-sonnet-4-6`, `claude-opus-4-8`) over the fixture notes, with each subagent's output written to a distinct per-model file. + +#### Scenario: One reference file per model + +- **WHEN** the benchmark prompt is run on the fixture set +- **THEN** a separate output file is produced for each of the three Claude tiers +- **AND** each file contains that tier's extraction result for every fixture note + +#### Scenario: Opus output is the scoring rubric + +- **WHEN** the reference set is reviewed +- **THEN** the `claude-opus-4-8` output is treated as the gold-standard rubric against which the other tiers, and later the local Ollama models, are scored + +### Requirement: Dispatch prompt enforces a fairness contract + +The benchmark prompt SHALL give each subagent only the raw note text and a shared minimal extraction spec, and SHALL forbid reading repository files or pulling project context. + +#### Scenario: Minimal context per subagent + +- **WHEN** a subagent is dispatched for a fixture note +- **THEN** its input is limited to the note text and the shared extraction spec +- **AND** it is explicitly instructed not to read `CLAUDE.md`, design docs, or other project context + +### Requirement: Output conforms to the Graphify-shaped schema + +Each subagent SHALL emit a structured fragment matching the shape Graphify produces — entities, typed relationships, and `INFERRED`/`AMBIGUOUS` confidence tags — so per-model outputs are diffable against each other and against later Ollama output. + +#### Scenario: Schema embedded in the prompt + +- **WHEN** the benchmark prompt is authored +- **THEN** the required output schema is embedded verbatim in the prompt +- **AND** every per-model output file follows that schema + +#### Scenario: Outputs are diffable + +- **WHEN** two per-model output files for the same fixture note are compared +- **THEN** they share a common structure that allows entity-by-entity and edge-by-edge comparison + +### Requirement: Claude reference run is scored on quality only + +The Claude reference run SHALL be evaluated on extraction quality only — entity correctness, relationship plausibility and typing, and confidence-tag accuracy — and SHALL NOT use wall-clock speed as a metric. + +#### Scenario: Speed excluded from the reference run + +- **WHEN** the Claude reference run is evaluated +- **THEN** wall-clock time per note is not used as a metric +- **AND** speed is reintroduced only when local Ollama models are later benchmarked against the reference set + +### Requirement: Prompt is a committed, reusable file + +The benchmark dispatch prompt SHALL be written as a committed file that can be copy/pasted into a Claude Code session to reproduce the run. + +#### Scenario: Prompt persisted as a file + +- **WHEN** the benchmark is set up +- **THEN** the dispatch prompt exists as a committed file in the repository +- **AND** running it does not require reconstructing the prompt from memory