jared
/
cc-os
1
0
Fork 0
Code Issues 18 Pull Requests Packages Projects Releases Wiki Activity

Add incremental-migration + reference-benchmark change (ADR-013) 

Invert the build order to build-first / migrate-incrementally and
redesign Step 2c as a Claude reference-set benchmark.

- ADR-013 records the build-order inversion; CLAUDE.md locked-decisions
  pointer updated
- New benchmark deliverable under docs/memory-system/benchmark/:
  shared extraction spec + copy/paste dispatch prompt + reference-outputs
- Runbook (05): Step 1 = fixture selection with bulk migration deferred;
  Step 2c = Claude-tiers-only reference set, quality-only metrics, Opus
  as gold-standard rubric (produces references, not a model choice)
- 04-build-plan reconciled to the new build order and benchmark design
- Sync incremental-migration + reference-extraction-benchmark specs into
  openspec/specs/; archive the completed change

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
jared 2026-06-04 11:17:32 -04:00
parent 235fd45ff7
commit 77febc8aca
15 changed files with 864 additions and 99 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
CLAUDE.md
View File

@ -58,7 +58,7 @@ tag-index CLI and also covers the deferred QMD semantic layer. `04-build-plan.md
`06-graphify-evaluation.md` reflect this; if an older doc still describes the Ruby CLI, defer `06-graphify-evaluation.md` reflect this; if an older doc still describes the Ruby CLI, defer
to those two and fix the stale doc. to those two and fix the stale doc.
**Decisions locked (2026-06-04):** Six-facet tag taxonomy + `scope/` (ADR-011); reuse `~/Documents/SecondBrain` vault rather than creating a new one (ADR-012). **Decisions locked (2026-06-04):** Six-facet tag taxonomy + `scope/` (ADR-011); reuse `~/Documents/SecondBrain` vault rather than creating a new one (ADR-012); build-first / migrate-incrementally — build full system against a fixture set first, defer bulk vault migration to last, onboard projects one at a time (ADR-013).
## OpenSpec workflow ## OpenSpec workflow

39
docs/memory-system/03-architecture-decisions.md
View File

@ -1,5 +1,7 @@
# Architecture Decision Records # Architecture Decision Records
_Last updated: 2026-06-04_
A running log of decisions and *why*. Format per entry: Context · Decision · Rationale · A running log of decisions and *why*. Format per entry: Context · Decision · Rationale ·
Alternatives rejected · Status. Newest decisions extend the log; supersede rather than delete. Alternatives rejected · Status. Newest decisions extend the log; supersede rather than delete.
@ -256,6 +258,43 @@ _Date: 2026-06-04_
remaining work is mechanical migration. remaining work is mechanical migration.
- **Status**: Accepted. - **Status**: Accepted.
## ADR-013 — Build-first / migrate-incrementally (build-order inversion)
_Date: 2026-06-04_
- **Context**: The build runbook (`05-implementation-process.md`) originally front-loaded bulk
vault migration as Step 1 — migrating all ~20 existing SecondBrain notes and all projects to
the ADR-011 six-facet taxonomy before the system existed to validate them. This committed to
a schema and workflow (the tag taxonomy from ADR-011, the vault-reuse choice from ADR-012,
and Graphify extraction behavior) before any end-to-end path had been exercised. The risk:
locking in an approach that fails at scale, with no feedback loop until the entire vault has
been touched.
- **Decision**: **Invert the build order.** The full system is built and validated against a
small **5–10 note fixture set** first. Bulk vault migration is deferred to the final stage.
The first real-data validation uses **one small project that contains both code AND
documents**, exercising both the local-SLM doc-extraction path and the tree-sitter code path
in the same run. After that single project validates end-to-end, remaining projects are
onboarded **one at a time** with an observe-and-adjust step between each.
- **Rationale**: Validates the ADR-011 taxonomy and ADR-012 vault conventions against the real
Graphify extraction pipeline before the entire vault is committed. The first mixed code+docs
project surfaces both extraction paths (SLM for docs, tree-sitter for code) early, when
corrections are cheap. Per-project rollout keeps the blast radius of any schema or workflow
correction small; each project is an opportunity to observe and adjust rather than discover
problems across 20 notes at once. This is consistent with the "markdown-as-truth, indexes are
disposable" principle (ADR-008): the vault notes are durable, but the extraction schema should
be validated before it shapes all of them.
- **Alternatives rejected**:
- **Keep migration-first (status quo)**: Front-loads all ~20 notes and all projects before
any end-to-end validation exists. Commits to ADR-011's taxonomy and ADR-012's migration
steps against the full vault without a feedback loop — exactly the gap this decision closes.
- **Big-bang migrate everything after build**: Build against fixtures, then migrate all notes
and all projects in one batch at the end. Avoids the pre-build commitment problem but still
risks a single large irreversible migration with no observe-and-adjust loop between units.
Per-project rollout with intermediate checkpoints is strictly safer.
- **Cross-references**: ADR-011 (six-facet tag taxonomy — the schema being validated);
ADR-012 (SecondBrain vault reuse — the migration steps this order defers).
- **Status**: Accepted (updates `05-implementation-process.md` build order).
## Rejected tools (summary) ## Rejected tools (summary)
| Tool | Why rejected for our use | | Tool | Why rejected for our use |

75
docs/memory-system/04-build-plan.md
View File

@ -1,5 +1,7 @@
# Build Plan # Build Plan
_Last updated: 2026-06-04_
How a human builds this system, step by step, and answers to the operational questions: How a human builds this system, step by step, and answers to the operational questions:
which scripts and hooks, how the AI knows when to write and what conventions to follow, how and which scripts and hooks, how the AI knows when to write and what conventions to follow, how and
when it queries, the CRUD hooks, and how it's packaged as a global Claude Code plugin with when it queries, the CRUD hooks, and how it's packaged as a global Claude Code plugin with
@ -14,6 +16,12 @@ vault and free AST-based code graphs per project. The deferred QMD semantic laye
is also skipped — Graphify covers it without vectors. See `06-graphify-evaluation.md` for is also skipped — Graphify covers it without vectors. See `06-graphify-evaluation.md` for
the full rationale. the full rationale.
**Architecture decision (2026-06-04):** Build order inverted (ADR-013). Build and validate
the full system against a small 5–10 note fixture set first; defer bulk vault migration to last.
Validate end-to-end on one small pilot project containing both code and documents before
onboarding any others. Onboard remaining projects one at a time, with observe-and-adjust between
each. Steps 2d and 2e below are updated accordingly.
--- ---
## Part A — Build order (human builder's path) ## Part A — Build order (human builder's path)
@ -61,39 +69,46 @@ Available models (as of 2026-06-03, in order of interest):
- `qwen3.5:2b` — 2.7 GB, smallest, good fallback if VRAM is constrained - `qwen3.5:2b` — 2.7 GB, smallest, good fallback if VRAM is constrained
- `gemma4:e4b` — 9.6 GB, more capable, slower - `gemma4:e4b` — 9.6 GB, more capable, slower
#### 2c — Model benchmarking (before committing to a model) #### 2c — Claude reference-set benchmark (THE GATE, before committing to a local model)
Run a small extraction test across all local models plus the three Claude API models. The goal This step produces the **gold-standard reference set** — one structured extraction output per
is to find the best speed/accuracy tradeoff for entity+relationship extraction from vault notes. fixture note per Claude tier. It does **not** choose the final extraction model, and it does
**not** measure speed.
**Test set:** 5–10 representative vault notes spanning different note types (tool note, client **Input:** the 5–10 fixture notes selected in Step 1 (from the runbook).
note, convention note, domain note). Include one note that is dense with relationships.
**What to measure per model:** **What to run:** dispatch one Claude Code subagent per tier (Claude-tiers only — Ollama models
1. Extraction speed (tokens/sec or wall-clock time per note) are not reachable in this environment):
2. Entity quality: are the right concepts extracted? Any hallucinated entities?
3. Relationship quality: are edges plausible and correctly typed? Missing relationships?
4. Confidence tag accuracy: are `INFERRED` vs `AMBIGUOUS` edges appropriately flagged?
**Models to test:** | Tier | Model ID | Role |
| Model | Backend | Notes |
|---|---|---| |---|---|---|
| `gemma4:e2b` | Ollama local | Primary candidate — fast, 7.2 GB | | Haiku | `claude-haiku-4-5` | Lightweight reference |
| `qwen3.5:2b` | Ollama local | Smallest, fastest | | Sonnet | `claude-sonnet-4-6` | Mid-tier reference |
| `gemma4:e4b` | Ollama local | Highest local quality | | Opus | `claude-opus-4-8` | **Gold standard** (scoring rubric) |
| `claude-haiku-4-5` | Claude API | Baseline API option |
| `claude-sonnet-4-6` | Claude API | Mid-tier reference |
| `claude-opus-4-8` | Claude API | **Gold standard** — judge quality against this |
Run using subagents in Claude Code: dispatch one agent per model, each extracting the same Each subagent receives only the fixture note text plus the shared extraction spec — no design
test set, return structured JSON of entities+relationships. Review god-node quality in docs, no project context (fairness contract). See ADR-013.
`GRAPH_REPORT.md` after each run. Use Opus output as the scoring rubric for the local models.
**Decision rule:** Choose the fastest local model whose entity/relationship quality is **Metrics — quality only (wall-clock speed is explicitly out of scope here):**
"close enough" to Opus (subjective; likely Gemma4:e2b or Gemma4:e4b based on the description). 1. Entity correctness — right concepts extracted, no hallucinated entities
API models are a fallback option for high-stakes notes, not the default. 2. Relationship plausibility and typing — edges plausible, correctly typed, no missing key edges
3. Confidence-tag accuracy — `INFERRED` vs `AMBIGUOUS` applied appropriately
**Deliverables** (produced by this step):
- Dispatch prompt (copy/paste-able): `docs/memory-system/benchmark/dispatch-prompt.md`
- Shared extraction schema: `docs/memory-system/benchmark/extraction-spec.md`
- Per-tier outputs: `docs/memory-system/benchmark/reference-outputs/<note-slug>.<tier>.md`
Opus output is the rubric. **Deferred later step:** local Ollama models (gemma4:e2b, qwen3.5:2b,
gemma4:e4b) are timed AND scored against these references — that scoring run is where speed
re-enters and the final model is chosen. Do not hardcode a model before that run completes.
Authoritative detail lives in `docs/memory-system/05-implementation-process.md` §2c.
#### 2d — Initial fixture graph build (ADR-013: small-first)
Run the initial build against the small fixture set (5–10 notes from Step 1/2c), not the
full vault. The bulk vault build is deferred to after the system is validated end-to-end.
#### 2d — Initial vault graph build
```bash ```bash
graphify extract --path ~/brain --backend ollama --model gemma4:e2b \ graphify extract --path ~/brain --backend ollama --model gemma4:e2b \
--token-budget 512 --max-concurrency 2 --token-budget 512 --max-concurrency 2
@ -102,8 +117,16 @@ Tune `--token-budget` (semantic chunk size) and `--max-concurrency` based on VRA
Review `GRAPH_REPORT.md` — check god nodes make sense (they should be your most-connected Review `GRAPH_REPORT.md` — check god nodes make sense (they should be your most-connected
tools, clients, and domain concepts). tools, clients, and domain concepts).
**Full vault migration** (the `~/brain` build above run over all notes) is the final step —
deferred to after end-to-end validation on the pilot project. Do not bulk-migrate the vault
until the system is verified working on the fixture set and pilot project.
#### 2e — Per-project code graphs (free, no model needed) #### 2e — Per-project code graphs (free, no model needed)
For each client project: **ADR-013 order:** start with ONE pilot project that contains both code and documents; validate
end-to-end before onboarding others. Onboard remaining projects one at a time, with
observe-and-adjust between each. Do not batch all projects at once.
For the pilot project (and each subsequent project, one at a time):
```bash ```bash
graphify extract --path ~/projects/<client>/<project> --no-docs graphify extract --path ~/projects/<client>/<project> --no-docs
``` ```

199
docs/memory-system/05-implementation-process.md
View File

@ -1,6 +1,6 @@
# Implementation Process # Implementation Process
_Last updated: 2026-06-04_ | _Status: Ready to plan — awaiting Step 2c benchmark results_ _Last updated: 2026-06-04_ | _Status: Ready to plan — awaiting Step 2c reference set_
This document distills `04-build-plan.md` into a concrete, staged build process and folds in This document distills `04-build-plan.md` into a concrete, staged build process and folds in
two locked decisions: **ADR-011** (faceted six-namespace taxonomy) and **ADR-012** (reuse the two locked decisions: **ADR-011** (faceted six-namespace taxonomy) and **ADR-012** (reuse the
@ -11,44 +11,57 @@ for the underlying rationale, query/CRUD conventions, and plugin internals; read
> This is still a build process outline, not a detailed implementation plan. The recommended > This is still a build process outline, not a detailed implementation plan. The recommended
> next move is to feed Step 2 (the critical path) into the **writing-plans skill** to produce an > next move is to feed Step 2 (the critical path) into the **writing-plans skill** to produce an
> executable, task-level plan. Most open questions in Steps 3–6 can be defaulted; Step 2c's > executable, task-level plan. Most open questions in Steps 3–6 can be defaulted; Step 2c's
> model benchmark is the genuine gate that the rest depends on — do not hardcode a model before > Claude reference-set run is the genuine gate — it produces the gold-standard rubric against
> it runs. > which local Ollama models are then scored. Do not hardcode a model before that scoring runs.
--- ---
## Build order ## Build order
Bottom-up: vault migration first (usable standalone), then Graphify, then hooks, then plugin. Bottom-up: fixture selection first (feeds the Step 2c gate), then Graphify setup, then hooks, then plugin. Bulk vault migration is deferred to after the system is validated end-to-end.
--- ---
## Step 1 — Adopt and migrate the SecondBrain vault ## Step 1 — Select benchmark fixtures (and defer bulk migration)
**Per ADR-012:** reuse `~/Documents/SecondBrain` as the knowledge vault rather than creating a **Per ADR-012:** `~/Documents/SecondBrain` is the vault. The vault is already flat, already
new `~/brain`. The vault is already flat, already scoped to durable knowledge, and already scoped to durable knowledge, and already articulates the correct governance philosophy.
articulates the correct governance philosophy. Migration cost is mechanical.
### 1a — Initialize git **Bulk vault migration is DEFERRED.** Do not migrate all notes now. The only pre-build activity
in this step is selecting a small fixture set that feeds the Step 2c benchmark gate. Everything
else — tagging ~20 existing notes, updating governance docs, initializing git, fixing broken
references — happens after the system is validated end-to-end.
The vault has no `.git` yet. Initialize it so ADR-008's sync strategy works: > **Four sequencing concepts to keep distinct — they are not interchangeable:**
>
> 1. **Benchmark fixtures** — 5–10 notes selected NOW, in this step. They feed the Step 2c
> extraction benchmark gate. They are living fixtures, kept and reused throughout development.
> 2. **Bulk vault migration** — converting all ~20 existing notes to the six-facet tag schema,
> updating governance docs, initializing git. DEFERRED to last, after the system is fully
> validated.
> 3. **Initial validation project** — the first real end-to-end test after the system is built:
> one small project that contains BOTH code AND documents. POST-BUILD, before any rollout.
> 4. **Project-by-project rollout** — onboarding remaining projects one at a time, observing
> and adjusting between each. Follows the initial validation project.
```bash ### 1a — Select fixture notes
git init ~/Documents/SecondBrain
```
### 1b — Update vault governance notes Choose 5–10 notes from `~/Documents/SecondBrain` with deliberate variety. The selection must
include at minimum:
Update three files to the new schema: - A **tool note** (e.g., a note about a CLI tool or SaaS product)
- A **client/project note** (scoped, not global)
- A **convention note** (a `type/convention` or workflow note)
- A **domain note** (a topic or subject-area note)
- At least one **relationship-dense note** — a note where several concepts interrelate and
Graphify should emit multiple typed edges and confidence tags
- **`CLAUDE.md`** — update vault path references, Graphify commands, ADR-011 namespace list. These notes must be usable as-is by a subagent that receives only the note text plus the shared
- **`vault-conventions.md`** — update tag-namespace list to the six-facet form; keep the extraction spec (see Step 2c). Apply `summary:` frontmatter and the six-facet tag schema to
existing "act without being asked" proactive-retrieval section intact (it's better than each fixture note now; do not wait for bulk migration.
what cc-os had — adopt it, don't overwrite it).
- **Project-config hub note(s)** — update the tag-inference table values to namespaced form
(e.g. `semrush-work → tool/semrush`, `pest-control → domain/pest-control`). Keep the
table; it operationalizes auto-tagging by path pattern and should be preserved.
**Frontmatter contract** (apply to all notes, new and migrated): **Frontmatter contract** (required for fixture notes immediately; apply to all notes at bulk
migration time):
```yaml ```yaml
--- ---
@ -67,34 +80,45 @@ tags:
`type/` listed first preserves the vault's existing type-first ordering habit and makes note `type/` listed first preserves the vault's existing type-first ordering habit and makes note
kind immediately visible. kind immediately visible.
### 1c — Migrate existing notes (~20) ### 1b — Deferred bulk migration work (do not start until post-validation)
For each existing note: The following items are deferred to after the system is validated end-to-end on fixtures and
on the initial validation project (see §Post-build sequence below):
1. Add `summary:` frontmatter (one line; write it now — it's the human-authored router hint). - **Initialize git:** `git init ~/Documents/SecondBrain` (enables ADR-008 sync)
2. Convert flat unnamespaced tags to the six-facet namespaced form. Examples: - **Update vault governance notes:** `CLAUDE.md` (vault path refs, Graphify commands,
- `research` → `type/research` ADR-011 namespace list), `vault-conventions.md` (six-facet tag list; preserve the
- `pest-control` → `domain/pest-control` "act without being asked" proactive-retrieval section), and project-config hub note(s)
- `niche-automation-prospecting` → `domain/niche-automation-prospecting` (or split across (update tag-inference table to namespaced form, e.g. `semrush-work → tool/semrush`)
`domain/` + `project/` if the note is project-specific) - **Migrate remaining notes (~20):** add `summary:` frontmatter and convert unnamespaced tags
- `semrush` → `tool/semrush` to six-facet form (e.g. `research → type/research`, `pest-control → domain/pest-control`,
3. Add `scope/global` or `scope/project` to each note. `semrush → tool/semrush`). Add `scope/global` or `scope/project` to each note.
- **Create `_templates` for core note types:** `research`, `howto`, `adr`, `hub`. The long
tail stays freeform until a pattern earns a template (per ADR-011).
- **Fix the broken vault search reference:** `vault-conventions.md` (or `CLAUDE.md`) currently
references `~/.claude/scripts/vault_search.rb`, which does not exist. Replace every reference
with `graphify query` (per ADR-010).
Keep 5–10 migrated notes aside as representative test cases for Step 2c's extraction benchmark. ### Post-build sequence (after Steps 2–6 are complete)
Include variety: a tool note, a client/project note, a convention note, a domain note, and at
least one note dense with relationships between concepts.
### 1d — Create `_templates` for core note types only After the system is built and validated on fixtures:
Add templates for: `research`, `howto`, `adr`, `hub`. The long tail stays freeform until a **First:** Validate end-to-end on **one small project that contains BOTH code AND documents.**
pattern earns a template (per ADR-011). Consistent per-type structure improves Graphify's This project is the initial validation target — not a production migration. It exercises both
local-SLM extraction reliability. extraction paths simultaneously: vault notes use the local-SLM extraction path; project code
uses the tree-sitter AST path. These are different extraction paths, which is precisely why the
first validation project must contain both. Observe how the two paths compose before proceeding.
### 1e — Fix the broken vault search reference **Then:** Onboard remaining projects **one at a time.** Observe extraction quality, god-node
shape, and hook behavior between each project before onboarding the next. Adjust configuration
as needed.
`vault-conventions.md` (or `CLAUDE.md`) currently references **Finally:** Execute bulk vault migration (items in 1b above) and run full vault graph build.
`~/.claude/scripts/vault_search.rb`, which does not exist. Replace every reference to that
script with `graphify query` (per ADR-010). **Migration-unit granularity open question:** the right unit — a whole project repo, a
vault-note cluster, a client boundary — is not yet established. The first validation project
("one small project with both code and documents") is the anchor; refine the granularity
definition after observing that first real migration.
--- ---
@ -126,37 +150,64 @@ GRAPHIFY_OLLAMA_KEEP_ALIVE=5 # set when packaging in Step 6
Verify context allocation after the first extraction call: `ollama ps` shows allocated context. Verify context allocation after the first extraction call: `ollama ps` shows allocated context.
### 2c — Model benchmark (THE GATE) ### 2c — Claude reference-set benchmark (THE GATE)
Run a small extraction test before committing to any model. Use the 5–10 representative Produce a reference set of Graphify-shaped extraction outputs before committing to any local
migrated notes from Step 1c as the test set. Ollama model. Use the 5–10 fixture notes selected in Step 1a as the input set.
**Models to benchmark:** **What this step produces — and what it does NOT decide:**
| Model | Backend | Size | Notes | This step produces the **gold-standard reference set**: one structured extraction output per
|---|---|---|---| fixture note per Claude tier. The reference set is the scoring rubric against which local Ollama
| `gemma4:e2b` | Ollama local | 7.2 GB | Primary candidate — fast, large context | models are scored in a later step. This step does NOT choose the final extraction model —
| `qwen3.5:2b` | Ollama local | 2.7 GB | Smallest; fallback if VRAM is constrained | local-model selection happens when Ollama models are timed and scored against these references.
| `gemma4:e4b` | Ollama local | 9.6 GB | Highest local quality; slower |
| `claude-haiku-4-5` | Claude API | — | API baseline |
| `claude-sonnet-4-6` | Claude API | — | Mid-tier reference |
| `claude-opus-4-8` | Claude API | — | **Gold standard — score local models against this** |
**What to measure per model:** **Why Claude-only here:** local Ollama models cannot be run in this benchmarking environment.
Only Claude tiers are reachable via dispatched subagents. The Claude outputs serve as quality
anchors; Ollama speed and quality are measured separately, against these anchors.
1. Extraction speed (wall-clock time per note) **Dispatch one Claude Code subagent per tier:**
2. Entity quality: right concepts extracted, no hallucinated entities
3. Relationship quality: edges plausible and correctly typed, no missing key edges
4. Confidence tag accuracy: `INFERRED` vs `AMBIGUOUS` appropriately flagged
**Decision rule:** choose the fastest local model whose entity/relationship quality is "close | Tier | Model ID | Role |
enough" to Opus output. Do not hardcode `gemma4:e2b` — the benchmark decides. |---|---|---|
| Haiku | `claude-haiku-4-5` | Lightweight reference |
| Sonnet | `claude-sonnet-4-6` | Mid-tier reference |
| Opus | `claude-opus-4-8` | **Gold standard** |
Review god-node quality in `GRAPH_REPORT.md` after each run. Opus output is the scoring rubric. Each subagent receives only the fixture note text plus the shared extraction spec at
`docs/memory-system/benchmark/extraction-spec.md`. It must not read `CLAUDE.md`, design docs,
or pull any project context — each subagent operates with the note and spec alone (fairness
contract).
Each subagent emits a Graphify-shaped structured fragment containing:
- Extracted entities (named concepts, tools, people, projects)
- Typed relationships between entities
- Confidence tags: `INFERRED` (inferred but not stated) and `AMBIGUOUS` (could be interpreted
multiple ways) applied where appropriate
**Metrics — quality only (wall-clock speed is explicitly out of scope here):**
Wall-clock timing is untrackable across dispatched subagents and is not measured in this step.
Speed re-enters the picture in the later Ollama-scoring step, where local models are timed
against these reference outputs. For this step, measure quality only:
1. **Entity correctness** — right concepts extracted, no hallucinated entities
2. **Relationship plausibility and typing** — edges plausible, correctly typed, no missing key edges
3. **Confidence-tag accuracy** — `INFERRED` vs `AMBIGUOUS` applied appropriately
**Deliverable files:**
- Dispatch prompt (copy/paste-able): `docs/memory-system/benchmark/dispatch-prompt.md`
- Shared extraction schema: `docs/memory-system/benchmark/extraction-spec.md`
- Per-model outputs: `docs/memory-system/benchmark/reference-outputs/<note-slug>.<tier>.md`
where `tier` is one of `haiku`, `sonnet`, or `opus`
`claude-opus-4-8` output is the gold standard. When Ollama models are benchmarked later, their
outputs are scored by how closely they match the Opus reference for each fixture note.
### 2d — Build the initial vault graph ### 2d — Build the initial vault graph
After 2c picks a model: After a local Ollama model has been chosen (scored against the Step 2c references):
```bash ```bash
graphify extract --path ~/Documents/SecondBrain \ graphify extract --path ~/Documents/SecondBrain \
@ -260,7 +311,8 @@ Graphify config.
## Open questions / decisions still to settle ## Open questions / decisions still to settle
These are deferred to build time. Most can be defaulted without blocking; the only genuine gate These are deferred to build time. Most can be defaulted without blocking; the only genuine gate
is **§6 (Step 2c benchmark)** — model selection is blocked until it runs. is **§6 (Step 2c benchmark)** — local-model selection is blocked until the reference set exists
and Ollama models can be scored against it.
1. **Vault symlink** — `~/Documents/SecondBrain` is confirmed as the vault (RESOLVED per 1. **Vault symlink** — `~/Documents/SecondBrain` is confirmed as the vault (RESOLVED per
ADR-012). The open sub-question: symlink it into `~/.claude/memory` or not? Only needed if ADR-012). The open sub-question: symlink it into `~/.claude/memory` or not? Only needed if
@ -280,8 +332,11 @@ is **§6 (Step 2c benchmark)** — model selection is blocked until it runs.
summaries. The human (or AI at note-creation time) must write `summary:` frontmatter. Confirm summaries. The human (or AI at note-creation time) must write `summary:` frontmatter. Confirm
this holds in practice and add a lint/reminder to the memory-write skill if it drifts. this holds in practice and add a lint/reminder to the memory-write skill if it drifts.
6. **Model benchmark results (Step 2c)** — all model choices are deferred until the benchmark 6. **Step 2c reference set (Claude gold-standard outputs)** — Step 2c produces the scoring
runs. Do not hardcode `gemma4:e2b` (it's the primary candidate, not the confirmed pick). rubric (Opus output as gold standard, Haiku/Sonnet as lighter-tier references), not the final
model choice. Local Ollama model selection is deferred until those references exist and Ollama
models can be timed and quality-scored against them. Do not hardcode `gemma4:e2b` — the Ollama
scoring step decides. The Step 2c outcome is the reference set, not the chosen model.
7. **memsearch + journal integration** — does memsearch index SessionEnd journal notes or only 7. **memsearch + journal integration** — does memsearch index SessionEnd journal notes or only
its own auto-capture? How does the journal pointer injected at SessionStart reference the its own auto-capture? How does the journal pointer injected at SessionStart reference the
@ -300,7 +355,7 @@ is **§6 (Step 2c benchmark)** — model selection is blocked until it runs.
## Recommended next move ## Recommended next move
Turn **Step 2** into a detailed implementation plan via the **writing-plans skill**. It is the Turn **Step 2** into a detailed implementation plan via the **writing-plans skill**. It is the
critical path: Graphify install, Ollama configuration, the model benchmark (2c), and the initial critical path: Graphify install, Ollama configuration, the Step 2c reference-set run, and the
graph build are the smallest set of tasks that unblock everything else. Open questions §1–5 and initial graph build are the smallest set of tasks that unblock everything else. Open questions
§7–8 can be defaulted or deferred; only §6 (the benchmark) genuinely blocks model-dependent §1–5 and §7–8 can be defaulted or deferred; only §6 (producing the reference set and then
decisions. scoring Ollama models against it) genuinely blocks model-dependent decisions.

207
docs/memory-system/benchmark/dispatch-prompt.md Normal file
View File

@ -0,0 +1,207 @@
# Claude-Tier Reference-Set Benchmark — Dispatch Prompt
_Last updated: 2026-06-04_
_Status: active — copy-paste this into a Claude Code session to generate the reference set_
## Overview
This prompt dispatches **one subagent per Claude tier** (`claude-haiku-4-5`, `claude-sonnet-4-6`,
`claude-opus-4-8`) over each fixture note selected in Step 1c of the build runbook. Each subagent
emits a Graphify-shaped structured fragment. The per-model output files become the **gold-standard
reference set** against which local Ollama doc-extraction models are scored in a later step.
**`claude-opus-4-8`'s output is the gold-standard rubric.**
Evaluation is on **quality only**: entity correctness, relationship plausibility and typing, and
confidence-tag accuracy. Wall-clock speed is NOT a metric for this Claude reference run — speed
re-enters only when local Ollama models are timed against this reference set.
---
## Step 1 — Select Fixture Notes (operator action, before dispatching)
Fixture notes are selected at build time from `~/Documents/SecondBrain` per Step 1c of the
runbook. Choose **5–10 notes** with deliberate variety:
- One **tool note** (documents a specific tool or library)
- One **client/project note** (describes a client engagement or project)
- One **convention note** (captures a working convention or practice)
- One **domain note** (covers a knowledge or technical domain)
- One **relationship-dense note** (many named entities or cross-references)
- Additional notes as needed for coverage
Assign each note a **kebab-case slug** (used in output filenames). Populate the placeholder list
below before dispatching:
```
# FIXTURES — populate before running
# Format: <note-slug>:<absolute-path-to-note>
FIXTURES=(
# <note-slug>:<path>
# <note-slug>:<path>
# ... (5-10 entries, one per selected note)
)
```
---
## Step 2 — Output File Convention
For each fixture note, the run produces **one output file per Claude tier** — three files per note,
`3 × N` files total (where N is the number of fixture notes).
Output path pattern:
```
docs/memory-system/benchmark/reference-outputs/<note-slug>.<tier>.md
```
Where `<tier>` is one of: `haiku`, `sonnet`, `opus`
Examples (using a hypothetical slug `graphify-tool-overview`):
- `reference-outputs/graphify-tool-overview.haiku.md`
- `reference-outputs/graphify-tool-overview.sonnet.md`
- `reference-outputs/graphify-tool-overview.opus.md`
Each file contains that tier's extraction fragment for **one fixture note** — the `note_slug:`
block for that note only. Do not combine multiple notes into one file.
---
## Step 3 — Dispatch (copy-paste this block into a Claude Code session)
> **Operator:** replace `<NOTE-SLUG>`, `<NOTE-TEXT>`, and model names as needed. Dispatch all
> three subagents for each fixture note. They can run in parallel across tiers for the same note.
---
### Dispatch template (repeat for each fixture note × each tier)
```
Dispatch a subagent using model <MODEL> to perform the following task.
=== FAIRNESS CONTRACT ===
You will receive exactly two inputs:
1. The raw text of one vault note (below).
2. The shared extraction spec and output schema (below).
You MUST NOT read any repository files (CLAUDE.md, design docs, specs, tasks), access the
vault directory structure, or use any project or system context. If your environment has
injected any such context automatically, you must ignore it entirely — treat it as if it
does not exist. The only allowed inputs are the note text and the extraction spec below.
=== END FAIRNESS CONTRACT ===
=== NOTE TEXT ===
<NOTE-TEXT>
=== END NOTE TEXT ===
=== EXTRACTION SPEC AND OUTPUT SCHEMA ===
## Facet Vocabulary (closed)
Notes carry six flat, namespaced facets plus one scope tag. These are the ONLY valid values
for the optional `facet` field on entities. Do not invent new namespaces.
| Prefix | Meaning |
|--------------|-----------------------------------------|
| `type/` | What kind of thing the note is about |
| `client/` | A client identity |
| `project/` | A project name |
| `domain/` | A knowledge or technical domain |
| `tool/` | A specific tool, library, or CLI |
| `convention/`| A working convention or practice |
| `scope/` | Applicability scope (cross-cutting tag) |
An entity should carry a `facet` value only when the note text directly supports mapping it
to one of the above. Absence of a `facet` field is correct when no mapping is warranted.
## Output Schema
Emit ONLY the following YAML structure. No other keys, no prose, no summary.
```yaml
# --- Graphify extraction fragment ---
# One block per fixture note. Repeat this structure for each note.
note_slug: <kebab-case-identifier-for-the-note> # operator fills this in at run time
entities:
- name: <string> # exact or near-exact name as it appears in the note
type: <string> # e.g. Person, Tool, Project, Concept, Convention, Client, Domain
facet: <string> # OPTIONAL — must be one of the seven prefixes above, e.g. "tool/graphify"
confidence: <string> # OPTIONAL — omit if the entity is directly stated
# Values: INFERRED | AMBIGUOUS
relationships:
- source: <entity name> # must match a name in the entities list above
type: <string> # free-text verb phrase, e.g. "uses", "depends_on", "implements", "replaces"
target: <entity name> # must match a name in the entities list above
confidence: <string> # OPTIONAL — omit if the relationship is directly stated
# Values: INFERRED | AMBIGUOUS
```
## Confidence tag semantics
| Tag | Meaning |
|------------|--------------------------------------------------------------------------------------|
| _(absent)_ | The entity or relationship is **directly stated** in the note text. |
| `INFERRED` | Not stated literally but **reasonably deducible** from the note text alone. |
| `AMBIGUOUS`| Supportable by the text but **uncertain or admits multiple readings**. |
## Rules
- **Entities only:** extract entities that are named (not vague category references).
- **No invented names:** entity names must be grounded in the note text.
- **Relationship types:** free-text verb phrases; do not normalize to a fixed vocabulary.
- **Facet mapping:** use the closed vocabulary above; only assign when the note text supports it.
- **No extra keys:** do not add summaries, scores, embeddings, or metadata fields.
- **One block per note:** if processing multiple fixture notes, emit one `note_slug:` block per
note, separated by `---`.
=== END EXTRACTION SPEC ===
Write your output to:
docs/memory-system/benchmark/reference-outputs/<NOTE-SLUG>.<TIER>.md
where <NOTE-SLUG> is the kebab-case slug for this note and <TIER> is one of: haiku, sonnet, opus.
The file must contain ONLY the YAML extraction fragment — no preamble, no explanation.
```
**MODEL and TIER substitutions:**
| Dispatch | MODEL | TIER |
|----------|------------------------|---------|
| 1st | `claude-haiku-4-5` | `haiku` |
| 2nd | `claude-sonnet-4-6` | `sonnet`|
| 3rd | `claude-opus-4-8` | `opus` |
---
## Step 4 — After All Subagents Complete
1. Verify `3 × N` files exist in `docs/memory-system/benchmark/reference-outputs/`.
2. For each fixture note, diff the three tier files side by side to understand tier-level
disagreements in entity recognition, relationship typing, and confidence tagging.
3. `claude-opus-4-8`'s `.opus.md` file for each note is the **gold-standard rubric** against
which local Ollama extraction will later be scored.
4. Do NOT run Ollama models in this step — that is a separate later step that uses these files
as the scoring reference.
---
## Notes on the Fairness Contract
The fairness contract requires that each subagent reasons from **note text + extraction spec
only**. Two mechanisms can break this:
1. **Deliberate reads:** the subagent reads repository files or the vault. The dispatch template
forbids this explicitly.
2. **Injected context:** Claude Code automatically injects `CLAUDE.md` and project context into
subagent sessions. The dispatch template explicitly instructs the subagent to **ignore any
injected context** and treat it as non-existent. This is the critical instruction — "don't
read files" is not sufficient on its own.
If you observe tier outputs that appear to reflect knowledge of the vault structure or system
design (beyond what the note text contains), treat that output as contaminated and re-run that
subagent with stronger isolation instructions.

84
docs/memory-system/benchmark/extraction-spec.md Normal file
View File

@ -0,0 +1,84 @@
# Graphify Document-Extraction Benchmark — Shared Extraction Spec
_Last updated: 2026-06-04_
_Status: active — used as the shared schema for the Claude-tier reference-set benchmark_
## Purpose
This spec defines the **minimal structured fragment** that every benchmarking subagent (one per
Claude tier) must emit when given a raw vault note. The fragment mimics what Graphify's local SLM
doc-extraction step produces: a list of **entities**, a list of **typed relationships** (edges
between entities), and per-item **confidence tags** where applicable.
The schema is intentionally minimal. It exists so that:
1. Per-model output files are **diffable entity-by-entity and edge-by-edge** across tiers.
2. The same schema is used unchanged when local Ollama models are later scored against this
reference set — making the comparison apples-to-apples.
Subagents receive only the raw note text plus this spec. No repository files, no vault structure,
no project context. If any such context is injected by the environment, it must be ignored.
---
## Facet Vocabulary (closed)
Notes in the SecondBrain vault carry six flat, namespaced facets plus one scope tag. These are the
**only** valid values for the optional `facet` field on entities. Do not invent new namespaces.
| Prefix | Meaning |
|--------------|-----------------------------------------|
| `type/` | What kind of thing the note is about |
| `client/` | A client identity |
| `project/` | A project name |
| `domain/` | A knowledge or technical domain |
| `tool/` | A specific tool, library, or CLI |
| `convention/`| A working convention or practice |
| `scope/` | Applicability scope (cross-cutting tag) |
An entity should carry a `facet` value only when the note text directly supports mapping it to one
of the above. Absence of a `facet` field is correct when no mapping is warranted.
---
## Output Schema
The required output format. Every subagent MUST emit this structure and no other top-level keys.
```yaml
# --- Graphify extraction fragment ---
# One block per fixture note. Repeat this structure for each note.
note_slug: <kebab-case-identifier-for-the-note> # operator fills this in at run time
entities:
- name: <string> # exact or near-exact name as it appears in the note
type: <string> # e.g. Person, Tool, Project, Concept, Convention, Client, Domain
facet: <string> # OPTIONAL — must be one of the seven prefixes above, e.g. "tool/graphify"
confidence: <string> # OPTIONAL — omit if the entity is directly stated
# Values: INFERRED | AMBIGUOUS
relationships:
- source: <entity name> # must match a name in the entities list above
type: <string> # free-text verb phrase, e.g. "uses", "depends_on", "implements", "replaces"
target: <entity name> # must match a name in the entities list above
confidence: <string> # OPTIONAL — omit if the relationship is directly stated
# Values: INFERRED | AMBIGUOUS
```
### Confidence tag semantics
| Tag | Meaning |
|------------|--------------------------------------------------------------------------------------|
| _(absent)_ | The entity or relationship is **directly stated** in the note text. |
| `INFERRED` | Not stated literally but **reasonably deducible** from the note text alone. |
| `AMBIGUOUS`| Supportable by the text but **uncertain or admits multiple readings**. |
### Rules
- **Entities only:** extract entities that are named (not vague category references).
- **No invented names:** entity names must be grounded in the note text.
- **Relationship types:** free-text verb phrases; do not normalize to a fixed vocabulary.
- **Facet mapping:** use the closed vocabulary above; only assign when the note text supports it.
- **No extra keys:** do not add summaries, scores, embeddings, or metadata fields.
- **One block per note:** if processing multiple fixture notes, emit one `note_slug:` block per note,
separated by `---`.

26
docs/memory-system/benchmark/reference-outputs/README.md Normal file
View File

@ -0,0 +1,26 @@
# Reference Outputs
_Last updated: 2026-06-04_
This directory holds the per-model Graphify-shaped extraction fragments generated by the
Claude-tier reference-set benchmark (see `../dispatch-prompt.md`).
## File naming convention
```
<note-slug>.<tier>.md
```
Where:
- `<note-slug>` is the kebab-case identifier for the fixture note (assigned in Step 1c of the
build runbook)
- `<tier>` is one of: `haiku`, `sonnet`, `opus`
## Contents
Each file contains the YAML extraction fragment for one fixture note as produced by one Claude
tier. Files are generated by running the dispatch prompt — they do not exist until that step
is executed.
`claude-opus-4-8`'s `.opus.md` files are the **gold-standard rubric** against which local Ollama
doc-extraction models are scored in a later step.

2
openspec/changes/archive/2026-06-04-incremental-migration-and-reference-benchmark/.openspec.yaml Normal file
View File

@ -0,0 +1,2 @@
schema: spec-driven
created: 2026-06-04

42
openspec/changes/archive/2026-06-04-incremental-migration-and-reference-benchmark/design.md Normal file
View File

@ -0,0 +1,42 @@
## Context
`cc-os` is a design-only repository; the build runbook is `docs/memory-system/05-implementation-process.md` and Step 2c is its documented gate. The hard constraint driving this change: only the Claude tiers are reachable here (via Claude Code subagents) — local Ollama models cannot be run in this benchmarking environment. The vault is `~/Documents/SecondBrain` (ADR-012) under the six-facet taxonomy (ADR-011). Graphify extracts entities plus typed edges plus confidence tags (`INFERRED`/`AMBIGUOUS`) from documents via a local SLM, and extracts code via tree-sitter AST (free, no model). The benchmark exists to choose the local doc-extraction model; this change does not make that choice, it produces the reference set that choice will be measured against.
## Goals / Non-Goals
**Goals:**
- A runnable benchmark that produces a reusable, diffable reference set across Claude tiers.
- A fairness contract that keeps the comparison as close to apples-to-apples as the environment allows.
- An incremental, observable build-and-migration path that validates the system before committing the whole vault to it.
**Non-Goals:**
- Choosing the final extraction model now (the reference set feeds that later decision).
- Pivoting away from local Ollama doc extraction (architecture stays intact per the existing ADRs).
- Running Ollama models within this change.
- Bulk-migrating the vault now.
## Decisions
- **Reference-set, not model selection.** Claude subagents produce gold-standard outputs, not a final pick. Rejected alternative: treating this as a "Claude-as-extraction-backend" pivot — that is an ADR-level architectural shift (cost, privacy) and is out of scope.
- **Mimic-extraction task with an explicit embedded output schema.** Subagents reason directly to the Graphify-shaped fragment rather than invoking `graphify extract`, because Ollama backends cannot run here and the goal is tier-vs-tier signal. The output schema (entities, typed relationships, `INFERRED`/`AMBIGUOUS` confidence) is embedded verbatim in the prompt so per-model files are diffable against each other now and against Ollama output later. Each subagent writes to its own per-model file.
- **Fairness contract: minimal context only.** Each subagent receives only the raw note text and the shared extraction spec. It is explicitly instructed not to read repository files (`CLAUDE.md`, design docs) or pull project context. This is the core property that makes the comparison meaningful.
- **Speed dropped for the Claude run.** Wall-clock per note is untrackable across dispatched subagents here, so quality is the only metric for the reference run: entity correctness, relationship plausibility/typing, and confidence-tag accuracy. Speed re-enters when local Ollama models are timed against the references.
- **Build-first / migrate-incrementally.** A 5–10 note fixture set (the variety already called for in Step 1c) feeds the gate immediately; bulk vault migration is deferred to last; first end-to-end validation runs against one small project containing both code and documents. The build-order inversion is recorded in ADR-013, and `CLAUDE.md`'s "Decisions locked" pointer is updated.
- **Migration-unit granularity, surfaced not hidden.** The first migration unit is named "one small project with both code and documents." Vault notes (local-SLM extraction path) and project code (tree-sitter path) are different extraction paths; design.md surfaces this so the user can react at proposal review rather than discovering it mid-migration.
## Risks / Trade-offs
- Fixtures unrepresentative of the real vault → choose deliberate variety per Step 1c (tool note, client/project note, convention note, domain note, one relationship-dense note) and keep them as living fixtures.
- Reference set encodes Claude-tier idiosyncrasies and biases later Ollama scoring → treat the references as a quality ceiling / scoring rubric, not literal ground truth; keep a human in the loop reviewing god-nodes.
- Deferring bulk migration delays real-world validation of the schema at scale → mitigated by the early end-to-end test on one small mixed project before broad rollout.
- Prompt context leakage breaks the fairness contract → the prompt enumerates the only allowed inputs and explicitly forbids reading repo/project files.
## Migration Plan
No code migration. The change edits docs and adds ADR-013; the only new artifact is the benchmark prompt file plus a directory for reference outputs (proposed: `docs/memory-system/benchmark/`). Rollback is a straight revert of the doc edits.
## Open Questions
- Migration-unit granularity (whole project repo vs. a vault-note cluster) — first unit is named now; refine after observing the first real migration.
- Where the prompt file and per-model reference outputs live (proposed default: `docs/memory-system/benchmark/`).
- Which specific small project is the initial mixed code+docs validation target — the user picks this at apply time (candidates under `~/projects/` and `~/dev/`).

29
openspec/changes/archive/2026-06-04-incremental-migration-and-reference-benchmark/proposal.md Normal file
View File

@ -0,0 +1,29 @@
## Why
Two gaps block executing the build runbook (`docs/memory-system/05-implementation-process.md`). First, Step 2c's model benchmark assumes head-to-head runs across local Ollama models and Claude API models, but the only models reachable from this environment are the Claude tiers via Claude Code subagents — so the benchmark cannot be run as written, and nothing captures a gold-standard to score local models against later. Second, the runbook front-loads bulk vault migration (all ~20 notes and all projects) as Step 1, committing to a schema and workflow before the system exists to validate them.
## What Changes
- Redesign Step 2c as a **reference-set benchmark**: a copy/paste-able Claude Code prompt — written as a committed file — that dispatches one subagent per Claude tier (`claude-haiku-4-5`, `claude-sonnet-4-6`, `claude-opus-4-8`) to mimic Graphify doc extraction on the fixture notes, each emitting the same Graphify-shaped structured fragment (entities, typed relationships, `INFERRED`/`AMBIGUOUS` confidence tags) to a per-model file. These per-model files become the gold-standard reference set that local Ollama models are scored against in a later step.
- Establish a **fairness contract** for the prompt: each subagent receives only the note text plus a shared minimal extraction spec — no `CLAUDE.md`, no design docs, no project context.
- Drop wall-clock speed as a metric for the Claude reference run (untrackable in this environment; quality only). Speed re-enters later when Ollama models are benchmarked against the reference set.
- **Resequence the build order**: build the full system against a small fixture set first; defer bulk vault migration and multi-project onboarding until the system is built and validated end-to-end on one small project that contains both code and documents; then onboard remaining projects one at a time, observing and adjusting per project.
- Keep four things explicitly distinct so the runbook does not blur them: benchmark fixtures (needed now, for the gate), bulk vault migration (deferred), the initial mixed code+docs validation project (post-build), and project-by-project rollout.
- Add **ADR-013** recording the build-order inversion (vault-migration-first → build-first / migrate-incrementally) and amend the "Decisions locked" pointer in `CLAUDE.md`.
## Capabilities
### New Capabilities
- `reference-extraction-benchmark`: the procedure, fairness contract, output schema, and deliverable prompt file for producing a reusable gold-standard Claude reference extraction set that local models are later scored against.
- `incremental-migration`: the resequenced build order — build and validate on fixtures plus one small mixed code+docs project before any bulk migration, then onboard projects one at a time.
### Modified Capabilities
<!-- None. No prior specs exist in openspec/specs/. -->
## Impact
- Docs: `docs/memory-system/05-implementation-process.md` (Step 1 sequencing + Step 2c rewrites), `docs/memory-system/03-architecture-decisions.md` (new ADR-013), `docs/memory-system/04-build-plan.md` (reconcile if it states the old order), `CLAUDE.md` ("Decisions locked" line).
- New deliverable: a committed, copy/paste-able benchmark dispatch prompt file, plus a home for the per-model reference outputs.
- No application code. This change touches design, runbook, and process only.

47
openspec/changes/archive/2026-06-04-incremental-migration-and-reference-benchmark/specs/incremental-migration/spec.md Normal file
View File

@ -0,0 +1,47 @@
## ADDED Requirements
### Requirement: System is validated before bulk migration
The build SHALL be completed and validated end-to-end against a small fixture set and one small project containing both code and documents before any bulk vault migration is performed.
#### Scenario: Fixtures feed the gate without bulk migration
- **WHEN** the Step 2c benchmark gate is run
- **THEN** it uses a 5–10 note fixture set
- **AND** the remaining vault notes are not migrated at that point
#### Scenario: First end-to-end validation is a mixed project
- **WHEN** the system is first validated end-to-end
- **THEN** the target is one small project containing both code and documents
- **AND** validation covers both the document extraction path and the code (tree-sitter) path
### Requirement: Projects are onboarded one at a time
After the system is validated, projects SHALL be onboarded individually, with observation and adjustment between each, rather than migrated in bulk.
#### Scenario: One project at a time
- **WHEN** a project is onboarded after initial validation
- **THEN** it is migrated on its own
- **AND** its migration is observed and the process adjusted before the next project is onboarded
### Requirement: Bulk vault migration is deferred to last
Bulk migration of the full vault SHALL be deferred until after the system is built and validated, rather than performed as the first build step.
#### Scenario: Bulk migration ordering
- **WHEN** the build order is followed
- **THEN** bulk vault migration occurs after system validation
- **AND** only the fixture notes are migrated beforehand
### Requirement: Build-order inversion is recorded in an ADR
The inversion of the documented build order (from vault-migration-first to build-first / migrate-incrementally) SHALL be recorded in a new ADR and reflected in the project's locked-decisions pointer.
#### Scenario: ADR captures the inversion
- **WHEN** the build order is resequenced
- **THEN** a new ADR (ADR-013) records the inversion and its rationale
- **AND** the "Decisions locked" pointer in `CLAUDE.md` is updated to reference it

61
openspec/changes/archive/2026-06-04-incremental-migration-and-reference-benchmark/specs/reference-extraction-benchmark/spec.md Normal file
View File

@ -0,0 +1,61 @@
## ADDED Requirements
### Requirement: Reference set is produced from Claude tiers
The benchmark SHALL produce a reusable reference set by dispatching one Claude Code subagent per Claude tier (`claude-haiku-4-5`, `claude-sonnet-4-6`, `claude-opus-4-8`) over the fixture notes, with each subagent's output written to a distinct per-model file.
#### Scenario: One reference file per model
- **WHEN** the benchmark prompt is run on the fixture set
- **THEN** a separate output file is produced for each of the three Claude tiers
- **AND** each file contains that tier's extraction result for every fixture note
#### Scenario: Opus output is the scoring rubric
- **WHEN** the reference set is reviewed
- **THEN** the `claude-opus-4-8` output is treated as the gold-standard rubric against which the other tiers, and later the local Ollama models, are scored
### Requirement: Dispatch prompt enforces a fairness contract
The benchmark prompt SHALL give each subagent only the raw note text and a shared minimal extraction spec, and SHALL forbid reading repository files or pulling project context.
#### Scenario: Minimal context per subagent
- **WHEN** a subagent is dispatched for a fixture note
- **THEN** its input is limited to the note text and the shared extraction spec
- **AND** it is explicitly instructed not to read `CLAUDE.md`, design docs, or other project context
### Requirement: Output conforms to the Graphify-shaped schema
Each subagent SHALL emit a structured fragment matching the shape Graphify produces — entities, typed relationships, and `INFERRED`/`AMBIGUOUS` confidence tags — so per-model outputs are diffable against each other and against later Ollama output.
#### Scenario: Schema embedded in the prompt
- **WHEN** the benchmark prompt is authored
- **THEN** the required output schema is embedded verbatim in the prompt
- **AND** every per-model output file follows that schema
#### Scenario: Outputs are diffable
- **WHEN** two per-model output files for the same fixture note are compared
- **THEN** they share a common structure that allows entity-by-entity and edge-by-edge comparison
### Requirement: Claude reference run is scored on quality only
The Claude reference run SHALL be evaluated on extraction quality only — entity correctness, relationship plausibility and typing, and confidence-tag accuracy — and SHALL NOT use wall-clock speed as a metric.
#### Scenario: Speed excluded from the reference run
- **WHEN** the Claude reference run is evaluated
- **THEN** wall-clock time per note is not used as a metric
- **AND** speed is reintroduced only when local Ollama models are later benchmarked against the reference set
### Requirement: Prompt is a committed, reusable file
The benchmark dispatch prompt SHALL be written as a committed file that can be copy/pasted into a Claude Code session to reproduce the run.
#### Scenario: Prompt persisted as a file
- **WHEN** the benchmark is set up
- **THEN** the dispatch prompt exists as a committed file in the repository
- **AND** running it does not require reconstructing the prompt from memory

30
openspec/changes/archive/2026-06-04-incremental-migration-and-reference-benchmark/tasks.md Normal file
View File

@ -0,0 +1,30 @@
## 1. Record the decision
- [x] 1.1 Add ADR-013 to `docs/memory-system/03-architecture-decisions.md` recording the build-order inversion (vault-migration-first → build-first / migrate-incrementally), with rationale and rejected alternatives
- [x] 1.2 Update the "Decisions locked" pointer in `CLAUDE.md` to reference ADR-013
## 2. Author the benchmark prompt deliverable
- [x] 2.1 Choose and create the home for the prompt and per-model reference outputs (default `docs/memory-system/benchmark/`)
- [x] 2.2 Write the shared minimal extraction spec (entities, typed relationships, `INFERRED`/`AMBIGUOUS` confidence) as the schema the subagents must emit
- [x] 2.3 Write the copy/paste-able dispatch prompt file: one subagent per Claude tier (`claude-haiku-4-5`, `claude-sonnet-4-6`, `claude-opus-4-8`), fairness contract (note text + spec only; no repo/project context), per-model output files
- [x] 2.4 Confirm the prompt embeds the output schema verbatim and names the fixture inputs and per-model output paths
## 3. Rewrite Step 2c (benchmark) in the runbook
- [x] 3.1 Replace Step 2c's model table/metrics with the reference-set design: Claude tiers only, quality-only metrics, speed deferred to the later Ollama run
- [x] 3.2 State that the per-model reference set is the rubric local Ollama models are scored against later, and point to the prompt file
- [x] 3.3 Update Open question §6 to reflect that the Claude reference run produces references (not the final model choice)
## 4. Resequence Step 1 (migration) in the runbook
- [x] 4.1 Rewrite Step 1 so bulk vault migration is deferred; keep only fixture-note selection (5–10 notes, deliberate variety) as the pre-build step
- [x] 4.2 Add the post-build sequence: validate end-to-end on one small mixed code+docs project, then onboard projects one at a time with observe-and-adjust between each
- [x] 4.3 Make the four distinct items explicit in the runbook: benchmark fixtures / deferred bulk migration / initial mixed validation project / project-by-project rollout
- [x] 4.4 Note the migration-unit granularity question and that vault notes (SLM path) and project code (tree-sitter path) are different extraction paths
## 5. Reconcile and verify
- [x] 5.1 Check `docs/memory-system/04-build-plan.md` for any statement of the old "migration first" order and reconcile it with ADR-013
- [x] 5.2 Update the `_Last updated:_` / status lines on every edited design doc
- [x] 5.3 Re-read the edited runbook end-to-end to confirm Step 1 and Step 2c are internally consistent and the four distinct items are not blurred

53
openspec/specs/incremental-migration/spec.md Normal file
View File

@ -0,0 +1,53 @@
# Spec: Incremental Migration
## Purpose
Defines the ordering constraints for onboarding the memory system into production: the system must be built and validated before any migration is performed, projects are onboarded one at a time, and bulk vault migration is deferred to last.
## Requirements
### Requirement: System is validated before bulk migration
The build SHALL be completed and validated end-to-end against a small fixture set and one small project containing both code and documents before any bulk vault migration is performed.
#### Scenario: Fixtures feed the gate without bulk migration
- **WHEN** the Step 2c benchmark gate is run
- **THEN** it uses a 5–10 note fixture set
- **AND** the remaining vault notes are not migrated at that point
#### Scenario: First end-to-end validation is a mixed project
- **WHEN** the system is first validated end-to-end
- **THEN** the target is one small project containing both code and documents
- **AND** validation covers both the document extraction path and the code (tree-sitter) path
### Requirement: Projects are onboarded one at a time
After the system is validated, projects SHALL be onboarded individually, with observation and adjustment between each, rather than migrated in bulk.
#### Scenario: One project at a time
- **WHEN** a project is onboarded after initial validation
- **THEN** it is migrated on its own
- **AND** its migration is observed and the process adjusted before the next project is onboarded
### Requirement: Bulk vault migration is deferred to last
Bulk migration of the full vault SHALL be deferred until after the system is built and validated, rather than performed as the first build step.
#### Scenario: Bulk migration ordering
- **WHEN** the build order is followed
- **THEN** bulk vault migration occurs after system validation
- **AND** only the fixture notes are migrated beforehand
### Requirement: Build-order inversion is recorded in an ADR
The inversion of the documented build order (from vault-migration-first to build-first / migrate-incrementally) SHALL be recorded in a new ADR and reflected in the project's locked-decisions pointer.
#### Scenario: ADR captures the inversion
- **WHEN** the build order is resequenced
- **THEN** a new ADR (ADR-013) records the inversion and its rationale
- **AND** the "Decisions locked" pointer in `CLAUDE.md` is updated to reference it

67
openspec/specs/reference-extraction-benchmark/spec.md Normal file
View File

@ -0,0 +1,67 @@
# Spec: Reference Extraction Benchmark
## Purpose
Defines the process for producing a reusable Claude-tier reference set that serves as the quality rubric against which local Ollama extraction models are later scored.
## Requirements
### Requirement: Reference set is produced from Claude tiers
The benchmark SHALL produce a reusable reference set by dispatching one Claude Code subagent per Claude tier (`claude-haiku-4-5`, `claude-sonnet-4-6`, `claude-opus-4-8`) over the fixture notes, with each subagent's output written to a distinct per-model file.
#### Scenario: One reference file per model
- **WHEN** the benchmark prompt is run on the fixture set
- **THEN** a separate output file is produced for each of the three Claude tiers
- **AND** each file contains that tier's extraction result for every fixture note
#### Scenario: Opus output is the scoring rubric
- **WHEN** the reference set is reviewed
- **THEN** the `claude-opus-4-8` output is treated as the gold-standard rubric against which the other tiers, and later the local Ollama models, are scored
### Requirement: Dispatch prompt enforces a fairness contract
The benchmark prompt SHALL give each subagent only the raw note text and a shared minimal extraction spec, and SHALL forbid reading repository files or pulling project context.
#### Scenario: Minimal context per subagent
- **WHEN** a subagent is dispatched for a fixture note
- **THEN** its input is limited to the note text and the shared extraction spec
- **AND** it is explicitly instructed not to read `CLAUDE.md`, design docs, or other project context
### Requirement: Output conforms to the Graphify-shaped schema
Each subagent SHALL emit a structured fragment matching the shape Graphify produces — entities, typed relationships, and `INFERRED`/`AMBIGUOUS` confidence tags — so per-model outputs are diffable against each other and against later Ollama output.
#### Scenario: Schema embedded in the prompt
- **WHEN** the benchmark prompt is authored
- **THEN** the required output schema is embedded verbatim in the prompt
- **AND** every per-model output file follows that schema
#### Scenario: Outputs are diffable
- **WHEN** two per-model output files for the same fixture note are compared
- **THEN** they share a common structure that allows entity-by-entity and edge-by-edge comparison
### Requirement: Claude reference run is scored on quality only
The Claude reference run SHALL be evaluated on extraction quality only — entity correctness, relationship plausibility and typing, and confidence-tag accuracy — and SHALL NOT use wall-clock speed as a metric.
#### Scenario: Speed excluded from the reference run
- **WHEN** the Claude reference run is evaluated
- **THEN** wall-clock time per note is not used as a metric
- **AND** speed is reintroduced only when local Ollama models are later benchmarked against the reference set
### Requirement: Prompt is a committed, reusable file
The benchmark dispatch prompt SHALL be written as a committed file that can be copy/pasted into a Claude Code session to reproduce the run.
#### Scenario: Prompt persisted as a file
- **WHEN** the benchmark is set up
- **THEN** the dispatch prompt exists as a committed file in the repository
- **AND** running it does not require reconstructing the prompt from memory