56 lines
5.3 KiB
Markdown
56 lines
5.3 KiB
Markdown
## Context
|
||
|
||
This change implements Steps 2a–2d of `docs/memory-system/05-implementation-process.md` — the critical path of the memory system. Step 2c (the Claude reference-set gate) is already executed: 18 gold-standard fragments (6 fixtures × 3 tiers) exist in `docs/memory-system/benchmark/reference-outputs/`, with the `claude-opus-4-8` outputs as the scoring rubric (specced under `reference-extraction-benchmark`). What remains is to stand up the local extraction toolchain, score local Ollama models against that rubric, pick the model, and build the first vault graph.
|
||
|
||
Constraints:
|
||
- The vault is the existing `~/Documents/SecondBrain` (ADR-012); the build runs against it as-is, no bulk migration (ADR-013, `incremental-migration` spec).
|
||
- Markdown is the single source of truth; graph artifacts are disposable (ADR-008).
|
||
- A feasibility gut-check found `gemma4:e4b` runs at ~74 tok/s on the local GPU, but quality against the references has not been measured. The implementation-process doc is internally inconsistent on the model (`e4b` vs `e2b`) — a tell that nothing is locked.
|
||
|
||
## Goals / Non-Goals
|
||
|
||
**Goals:**
|
||
- Install and verify Graphify; configure Ollama for extraction.
|
||
- Score candidate Ollama models against the existing Opus rubric on the 6 fixtures (quality + speed) and select the extraction model by evidence.
|
||
- Build the initial vault graph with the selected model and sanity-check god-nodes.
|
||
|
||
**Non-Goals:**
|
||
- Regenerating or modifying the Step 2c reference set (consumed, not produced here).
|
||
- Per-project code graphs / Step 2e (separate, free tree-sitter path; out of range).
|
||
- Hooks, memsearch, sync, plugin packaging (Steps 3–6).
|
||
- Bulk vault migration (deferred to last per ADR-013).
|
||
- Choosing the sync mechanism or stale-rebuild threshold (Open questions §2–3).
|
||
|
||
## Decisions
|
||
|
||
**Selection by scoring, not by gut-check.** The model is chosen by comparing each candidate's Graphify-shaped output to the Opus reference per fixture (entity correctness, relationship typing, confidence-tag accuracy) plus measured wall-clock speed. Alternative considered: adopt `gemma4:e4b` directly since feasibility passed — rejected because the gut-check validated speed, not extraction quality, and Open-question §6 explicitly says do not hardcode. `gemma4:e4b` enters as the front-runner candidate, nothing more.
|
||
|
||
**Score against the Opus tier as the rubric.** Haiku/Sonnet references exist but Opus is the gold standard (per `reference-extraction-benchmark`). Candidates are scored primarily against Opus; the other tiers provide a quality gradient for context.
|
||
|
||
**Ollama config travels with this step.** `OLLAMA_FLASH_ATTENTION=1` (KV-cache VRAM savings) and `GRAPHIFY_OLLAMA_NUM_CTX=8192` (sufficient for 200–2000-word notes with prompt headroom) are set in the shell profile now and re-baked into the plugin env block at Step 6. `GRAPHIFY_OLLAMA_KEEP_ALIVE` is deferred to packaging. Verify with `ollama ps` after the first call.
|
||
|
||
**Build the full vault, then review god-nodes.** Rather than a synthetic subset, build over the real `~/Documents/SecondBrain` and use `GRAPH_REPORT.md`'s most-connected nodes as the sanity signal — the highest-traffic tools/clients/domains should surface as god-nodes. Cheap, and it exercises the real extraction path.
|
||
|
||
## Risks / Trade-offs
|
||
|
||
- **No candidate scores acceptably against the rubric** → If even the best candidate is well below the Opus reference, surface it as a finding rather than forcing a selection; the front-runner's speed does not rescue poor extraction quality. Selection may need a larger candidate or a revisit of token budget / context.
|
||
- **Scoring is partly qualitative** → Entity/relationship/confidence-tag comparison against references is judgement-based, not a single numeric pass/fail. Mitigation: record the per-fixture comparison and rationale in the result artifact so the choice is auditable, not asserted.
|
||
- **Vault content drift during build** → Building over the live vault means notes could change mid-run. Mitigation: the graph is disposable and rebuildable (`--force`); a one-shot initial build is acceptable.
|
||
- **GPU/VRAM regressions** → The ~74 tok/s figure was post-reboot; throughput can vary. Mitigation: `--max-concurrency 2` and a bounded `--token-budget` keep memory pressure predictable; record actual speed per candidate.
|
||
|
||
## Migration Plan
|
||
|
||
Sequential, low blast radius (local-only, no production system touched):
|
||
1. `pip install graphifyy`; verify `graphify --version`.
|
||
2. Export Ollama env vars; pull candidate model(s); verify context via `ollama ps`.
|
||
3. Run each candidate over the 6 fixtures; score vs. the Opus references; record results.
|
||
4. Select the model; build the initial vault graph; review `GRAPH_REPORT.md`.
|
||
|
||
Rollback: artifacts are disposable — delete `graphify-out/` and re-run; uninstall `graphifyy` if needed. No data is mutated in the vault (extraction is read-only over notes).
|
||
|
||
## Open Questions
|
||
|
||
- Exact candidate set beyond `gemma4:e4b` (e.g. whether to also score a smaller/larger sibling) — decided at run time based on what is pulled and how the front-runner scores.
|
||
- Where the scoring-result artifact lives (under `docs/memory-system/benchmark/` alongside the references is the natural home) — settle when writing it.
|
||
- `--token-budget` / `--max-concurrency` tuning — start from the doc's `512` / `2` and adjust if quality or VRAM demands it.
|