30 lines
3.3 KiB
Markdown
30 lines
3.3 KiB
Markdown
## Why
|
|
|
|
Two gaps block executing the build runbook (`docs/memory-system/05-implementation-process.md`). First, Step 2c's model benchmark assumes head-to-head runs across local Ollama models and Claude API models, but the only models reachable from this environment are the Claude tiers via Claude Code subagents — so the benchmark cannot be run as written, and nothing captures a gold-standard to score local models against later. Second, the runbook front-loads bulk vault migration (all ~20 notes and all projects) as Step 1, committing to a schema and workflow before the system exists to validate them.
|
|
|
|
## What Changes
|
|
|
|
- Redesign Step 2c as a **reference-set benchmark**: a copy/paste-able Claude Code prompt — written as a committed file — that dispatches one subagent per Claude tier (`claude-haiku-4-5`, `claude-sonnet-4-6`, `claude-opus-4-8`) to mimic Graphify doc extraction on the fixture notes, each emitting the same Graphify-shaped structured fragment (entities, typed relationships, `INFERRED`/`AMBIGUOUS` confidence tags) to a per-model file. These per-model files become the gold-standard reference set that local Ollama models are scored against in a later step.
|
|
- Establish a **fairness contract** for the prompt: each subagent receives only the note text plus a shared minimal extraction spec — no `CLAUDE.md`, no design docs, no project context.
|
|
- Drop wall-clock speed as a metric for the Claude reference run (untrackable in this environment; quality only). Speed re-enters later when Ollama models are benchmarked against the reference set.
|
|
- **Resequence the build order**: build the full system against a small fixture set first; defer bulk vault migration and multi-project onboarding until the system is built and validated end-to-end on one small project that contains both code and documents; then onboard remaining projects one at a time, observing and adjusting per project.
|
|
- Keep four things explicitly distinct so the runbook does not blur them: benchmark fixtures (needed now, for the gate), bulk vault migration (deferred), the initial mixed code+docs validation project (post-build), and project-by-project rollout.
|
|
- Add **ADR-013** recording the build-order inversion (vault-migration-first → build-first / migrate-incrementally) and amend the "Decisions locked" pointer in `CLAUDE.md`.
|
|
|
|
## Capabilities
|
|
|
|
### New Capabilities
|
|
|
|
- `reference-extraction-benchmark`: the procedure, fairness contract, output schema, and deliverable prompt file for producing a reusable gold-standard Claude reference extraction set that local models are later scored against.
|
|
- `incremental-migration`: the resequenced build order — build and validate on fixtures plus one small mixed code+docs project before any bulk migration, then onboard projects one at a time.
|
|
|
|
### Modified Capabilities
|
|
|
|
<!-- None. No prior specs exist in openspec/specs/. -->
|
|
|
|
## Impact
|
|
|
|
- Docs: `docs/memory-system/05-implementation-process.md` (Step 1 sequencing + Step 2c rewrites), `docs/memory-system/03-architecture-decisions.md` (new ADR-013), `docs/memory-system/04-build-plan.md` (reconcile if it states the old order), `CLAUDE.md` ("Decisions locked" line).
|
|
- New deliverable: a committed, copy/paste-able benchmark dispatch prompt file, plus a home for the per-model reference outputs.
|
|
- No application code. This change touches design, runbook, and process only.
|