Document ADR-015: memsearch episodic sync and resolved commingling design
memsearch episodic memory (Step 4) is now live and synced via git. Record the final sync design: dedicated private Forgejo repo at forgejo.swansoncloud.com/jared/memsearch, auto-commit+push via cc-os session-end hook, and the resolved decision to use a single global cross-client store (not per-client repos). Commingling is an explicit design choice, not a tolerated risk. Update build plan to split Step 5 into 5a (episodic sync, done) and 5b (vault sync, remaining). Update CLAUDE.md implementation status, USER-GUIDE to reflect memsearch now operational, and add full ADR-015 decision record with rationale and alternatives.
This commit is contained in:
jared
parent
accae31fba
commit
53a3c4c2c2
|
|
@ -61,7 +61,9 @@ to those two and fix the stale doc.
|
|||
|
||||
**Empirical finding locked (2026-06-05):** Graphify is a structure extractor, not a topic clusterer — no emergent hub nodes appear even at `--mode deep`; hub notes + wikilinks must be author-provided during migration (not deferred). Migration scaffolding is now a first-class deliverable. Open question: do facet tags create graph edges? (ADR-014; findings: `docs/memory-system/07-graph-connectivity-findings.md`).
|
||||
|
||||
**Implementation status (2026-06-09):** The global Claude Code plugin is live (`~/.claude/plugins/memory/`). Steps 1, 2a, 2b, 3, and 6 of the build plan are complete (including the `memory-project` skill, previously TODO under Step 6/Part D). Step 1 completed 2026-06-09: vault-conventions.md exists, all 6 fixture notes seeded, and 14 Graphify handbook + memory-system design notes migrated to the vault as migration scaffolding. Step 4 (memsearch) completed 2026-06-09: plugin installed via marketplace, MEMSEARCH_DIR set global (~/.memsearch), Stop hook verified producing daily memory files, search confirmed working. Step 5 (sync) remains. See `docs/memory-system/04-build-plan.md` for full step status.
|
||||
**Implementation status (2026-06-09):** The global Claude Code plugin is live (`~/.claude/plugins/memory/`). Steps 1, 2a, 2b, 3, and 6 of the build plan are complete (including the `memory-project` skill, previously TODO under Step 6/Part D). Step 1 completed 2026-06-09: vault-conventions.md exists, all 6 fixture notes seeded, and 14 Graphify handbook + memory-system design notes migrated to the vault as migration scaffolding. Step 4 (memsearch) completed 2026-06-09: plugin installed via marketplace, MEMSEARCH_DIR set global (~/.memsearch), Stop hook verified producing daily memory files, search confirmed working. Step 5a (memsearch episodic git sync) completed 2026-06-09: dedicated private Forgejo repo (`forgejo.swansoncloud.com/jared/memsearch`), whitelist `.gitignore` (memory/*.md only), auto-commit+push via cc-os `session-end.sh` hook (ADR-015). Step 5b (Obsidian vault → VPS sync) remains. See `docs/memory-system/04-build-plan.md` for full step status.
|
||||
|
||||
**Decision (2026-06-09):** Single global memsearch store for ALL clients — cross-client commingling is the accepted, intended design (ADR-015 resolved). One private Forgejo repo (`forgejo.swansoncloud.com/jared/memsearch`), not per-client repos. Forward direction: minimize/eliminate dedicated per-client working directories; work projects locally or use a single general `clients/` dir for non-local client work — memory is captured globally regardless of cwd. No `clients/` directory structure designed yet; open item.
|
||||
|
||||
## Implemented Components
|
||||
|
||||
|
|
@ -78,12 +80,13 @@ to those two and fix the stale doc.
|
|||
|
||||
**memsearch** — v0.4.6 via Claude Code plugin marketplace (`memsearch@memsearch-plugins`)
|
||||
- Hooks: Stop, SessionStart, UserPromptSubmit, SessionEnd (ship with plugin)
|
||||
- Memory store: `~/.memsearch/memory/YYYY-MM-DD.md` daily files (global, cross-project)
|
||||
- Memory store: `~/.memsearch/memory/YYYY-MM-DD.md` daily files (global, cross-project, all clients — one store by design)
|
||||
- Index: `~/.memsearch/milvus.db` (Milvus Lite, local); embeddings via ONNX bge-m3
|
||||
- Config: `MEMSEARCH_DIR=~/.memsearch` in `~/.zshrc` for global scope
|
||||
- Skills: `/memory-recall`, `/memory-config` (ship with plugin)
|
||||
- Git sync: `~/.memsearch` is a dedicated private Forgejo repo (`forgejo.swansoncloud.com/jared/memsearch`); whitelist `.gitignore` commits only `memory/*.md` (excludes rebuildable `milvus.db`, model, config); auto-commit+push wired into the cc-os memory plugin's own `session-end.sh` hook (see ADR-015)
|
||||
|
||||
**Not yet implemented:** VPS sync (Step 5)
|
||||
**Not yet implemented:** Obsidian vault → VPS sync (Step 5b). Memsearch episodic git sync is done (Step 5a, 2026-06-09).
|
||||
|
||||
## OpenSpec workflow
|
||||
|
||||
|
|
|
|||
|
|
@ -159,6 +159,10 @@ tags:
|
|||
projects — decisions, patterns, conventions, reference facts. Ephemeral session notes belong
|
||||
in the journal, not the vault.
|
||||
|
||||
Note the contrast with episodic memory: while vault notes use `client/` and `scope/` facets to
|
||||
scope individual notes, memsearch episodic memory is a **single global store spanning all
|
||||
clients and projects** — there is no per-client partitioning at the store level (by design).
|
||||
|
||||
Hub notes (type/hub) tie a domain together with wikilinks. If you find a cluster of notes
|
||||
that share a theme but have no hub, create one — Graphify does not create hub nodes
|
||||
automatically (see Known limitations).
|
||||
|
|
@ -194,10 +198,13 @@ At session start, Claude receives the path to `<project>/graphify-out/graph.json
|
|||
the content. To actually query the project graph, invoke `memory-project query <question>`.
|
||||
Claude will not automatically read the project graph without being asked.
|
||||
|
||||
**memsearch (episodic memory) is not built yet.**
|
||||
The design calls for a second memory tier — episodic memory of what happened across sessions,
|
||||
stored in Milvus Lite. This does not exist. Only semantic/knowledge memory via Graphify is
|
||||
live. Session journals (written manually) are the current substitute.
|
||||
**memsearch (episodic memory) is live as of 2026-06-09.**
|
||||
The second memory tier — episodic memory of what happened across sessions, stored in Milvus
|
||||
Lite — is installed and operational. Memory is captured globally across all clients and
|
||||
projects in one store (`~/.memsearch`), synced to a private self-hosted Forgejo repo
|
||||
(`forgejo.swansoncloud.com/jared/memsearch`). This is the intended design: a single
|
||||
commingled global store, not per-client stores. Use `/memory-recall` to search episodic memory;
|
||||
use `/memory-config` to inspect configuration. See ADR-015 for the rationale and sync details.
|
||||
|
||||
**Vault graph rebuild is fire-and-forget.**
|
||||
The `SessionStart` rebuild runs in the background with no progress indicator. If the rebuild
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# Architecture Decision Records
|
||||
|
||||
_Last updated: 2026-06-05_
|
||||
_Last updated: 2026-06-09_
|
||||
|
||||
A running log of decisions and *why*. Format per entry: Context · Decision · Rationale ·
|
||||
Alternatives rejected · Status. Newest decisions extend the log; supersede rather than delete.
|
||||
|
|
@ -353,6 +353,82 @@ _Date: 2026-06-05_
|
|||
validates the hub-notes/wikilinks mechanism of ADR-011 while flagging its facet-tag half as
|
||||
an open question.
|
||||
|
||||
## ADR-015 — memsearch episodic memory version-controlled in a dedicated private repo, auto-synced via cc-os SessionEnd hook
|
||||
|
||||
_Date: 2026-06-09_
|
||||
|
||||
- **Context**: memsearch's memory store at `~/.memsearch` accumulates daily session-summary
|
||||
files (`memory/YYYY-MM-DD.md`) that are the only irreplaceable data in the episodic layer —
|
||||
the Milvus Lite index (`milvus.db`) and the bge-m3 ONNX embeddings model are derived/
|
||||
disposable and can be rebuilt at any time via `memsearch index`. With Step 4 (episodic layer)
|
||||
now live, preserving episodic memory across machines and protecting against local disk loss
|
||||
required a sync strategy. The question was: what venue, what scope, and who owns the sync?
|
||||
memsearch's own stated design philosophy is "markdown files are the canonical data store; the
|
||||
vector database is a derived index" and notes that markdown is "git-friendly" and the index
|
||||
rebuildable from markdown — making git a natural fit for the markdown layer.
|
||||
- **Decision**: `~/.memsearch` is a **dedicated git repo** on branch `main` with remote `origin`
|
||||
pointing to a **private self-hosted Forgejo repo**
|
||||
(`ssh://git@forgejo.swansoncloud.com:2222/jared/memsearch.git`;
|
||||
web: `https://forgejo.swansoncloud.com/jared/memsearch`). A whitelist `.gitignore` tracks
|
||||
**only** `memory/*.md` (the daily session files) and `.gitignore` itself. Excluded as
|
||||
derived/disposable: `milvus.db` (Milvus Lite index — rebuildable any time via
|
||||
`memsearch index`), `config.toml`, and the bge-m3 ONNX embeddings model (lives in
|
||||
`~/.cache/huggingface`, not in the store). Auto-commit and push are wired into the **cc-os
|
||||
memory plugin's own `session-end.sh`** hook, not the marketplace plugin. The appended block
|
||||
guards on `~/.memsearch/.git` existing, runs `git add -A` (whitelist makes it safe), commits
|
||||
only when something is staged (message: `memsearch: session memory <date>`), and pushes with
|
||||
`timeout 30 ... || true`. The entire block is wrapped in a subshell with a trailing `|| true`
|
||||
so it can **never fail session shutdown**.
|
||||
- **Rationale**:
|
||||
- **Dedicated repo (not folded into the vault or a project repo):** ADR-001 established
|
||||
episodic and semantic/knowledge as separate systems by design, and `~/.memsearch` is a
|
||||
global, cross-project store with no natural home in any single project repo. Committing
|
||||
episodic session logs into the Obsidian vault repo would conflate the two systems and
|
||||
violate the separation of concerns that ADR-001 is built on.
|
||||
- **Whitelist — commit markdown, exclude rebuildable index/model/config:** consistent with
|
||||
memsearch's own design philosophy ("markdown is canonical; index is derived") and with the
|
||||
cc-os principle from ADR-008 ("sync the vault, not the indexes"). The 544 MB bge-m3 ONNX
|
||||
model is not even in the store; the Milvus Lite DB rebuilds in one command. Committing them
|
||||
would bloat the repo for zero durability gain.
|
||||
- **Sync lives in cc-os hook, not the marketplace plugin:** the marketplace plugin's hook
|
||||
scripts (`stop.sh`, `session-start.sh`, `user-prompt-submit.sh`, `session-end.sh`) perform
|
||||
no git operations — only a read-only `git rev-parse --show-toplevel` for scoping. Adding
|
||||
sync to a marketplace plugin that can be clobbered by an upstream update is fragile; owning
|
||||
it in the cc-os `session-end.sh` keeps the sync logic under our control and version-tracked
|
||||
in this repo.
|
||||
- **Fail-safe contract:** the `|| true` wrap and `timeout 30` on push ensure that a network
|
||||
outage, an unreachable Forgejo instance, or any git error cannot prevent a session from
|
||||
closing cleanly. The SessionEnd hook's harness timeout is ~10 s, so push is effectively
|
||||
capped; commits land locally first, and any unpushed commit is carried forward by the next
|
||||
session's push (daily files are append-only, so no conflict risk).
|
||||
- **Commingling — resolved 2026-06-09**: `~/.memsearch` aggregates session summaries across
|
||||
**all clients** into one global store. This is an **explicit design choice**, not a tolerated
|
||||
risk: the user deliberately accepts a single commingled global store across all clients.
|
||||
Private self-hosted Forgejo (user-owned infrastructure) is the chosen sync venue; client-
|
||||
agreement compliance is the user's knowing responsibility. The concern was previously recorded
|
||||
as user-owned and unresolved; that is now closed: single global store is the intended design.
|
||||
This also aligns with a forward direction of minimizing dedicated per-client working
|
||||
directories — since memsearch captures memory globally regardless of cwd, a single
|
||||
`clients/` area for non-project client work becomes viable (direction stated; not yet designed).
|
||||
- **Alternatives rejected**:
|
||||
- **Fold into the Obsidian vault repo:** violates ADR-001's episodic/semantic separation.
|
||||
Episodic logs have different lifecycle (accrete and decay), different write patterns
|
||||
(auto-captured every session), and would clutter a vault intended for curated durable
|
||||
knowledge.
|
||||
- **Auto-commit in the marketplace plugin:** the marketplace plugin can be overwritten on
|
||||
update, losing the sync logic silently. Out-of-band ownership in cc-os is safer.
|
||||
- **Commit the Milvus Lite index or the embeddings model:** both are large binaries, derived
|
||||
from the markdown source, and rebuildable. Committing them wastes space and provides no
|
||||
additional durability. The markdown files are the canonical source; the index follows from
|
||||
them.
|
||||
- **Syncthing or rsync instead of git:** git provides both version history and conflict-free
|
||||
daily-append semantics; Syncthing is continuous-async (suitable for the vault where changes
|
||||
are sparse); git's push-on-session-end cadence matches how memsearch produces data (one
|
||||
daily file per day, append-only). Git was already chosen for ADR-008's vault sync rationale;
|
||||
applying the same mechanism to the episodic store is consistent.
|
||||
- **Status**: Accepted (commingling resolved 2026-06-09). Initial commit `106cebc` in the
|
||||
Forgejo repo; git repo initialized 2026-06-09 via the `git-context:repo-init` skill.
|
||||
|
||||
## Rejected tools (summary)
|
||||
|
||||
| Tool | Why rejected for our use |
|
||||
|
|
|
|||
|
|
@ -195,7 +195,28 @@ older than N days (7 is a reasonable starting point).
|
|||
- Graphify does **not** replace memsearch here. Time-anchored semantic queries ("what was I
|
||||
working on last Tuesday?") are better served by vector similarity over session logs.
|
||||
|
||||
### Step 5 — Sync — NOT STARTED
|
||||
### Step 5 — Sync — PARTIAL
|
||||
|
||||
Step 5 covers two distinct sync lines; they are at different statuses:
|
||||
|
||||
#### 5a — memsearch episodic memory git sync — **DONE (2026-06-09)**
|
||||
`~/.memsearch` is now a git repo on branch `main` with remote `origin` pointing to a private
|
||||
self-hosted Forgejo repo (`ssh://git@forgejo.swansoncloud.com:2222/jared/memsearch.git`;
|
||||
web: `https://forgejo.swansoncloud.com/jared/memsearch`). Initial commit `106cebc`.
|
||||
|
||||
A whitelist `.gitignore` tracks **only** `memory/*.md` (the daily session files) and
|
||||
`.gitignore`. Excluded as derived/disposable: `milvus.db` (Milvus Lite index — rebuildable via
|
||||
`memsearch index`), `config.toml`, and the bge-m3 ONNX model (lives in `~/.cache/huggingface`).
|
||||
|
||||
Auto-commit and push are wired into the cc-os memory plugin's **own `session-end.sh`** hook
|
||||
(not the marketplace plugin, which can be clobbered on update). The block guards on
|
||||
`~/.memsearch/.git` existing, commits only when staged content exists (message:
|
||||
`memsearch: session memory <date>`), and pushes with `timeout 30 ... || true`. Wrapped in a
|
||||
subshell with trailing `|| true` — cannot fail session shutdown. See **ADR-015** for rationale
|
||||
and the cross-client commingling, which is an accepted design choice (single global store
|
||||
across all clients by intent — see ADR-015, resolved 2026-06-09).
|
||||
|
||||
#### 5b — Obsidian vault → VPS sync — **NOT STARTED**
|
||||
- Pick **git** (versioned, hourly push/pull) or **Syncthing** (continuous, zero-thought) for
|
||||
the vault → VPS. Configure on each machine.
|
||||
- **Do not sync** the Graphify `graphify-out/` directories, Milvus caches, or the Ollama
|
||||
|
|
|
|||
Loading…
Reference in New Issue