cc-os/docs/adr/0015-memsearch-episodic-mem...

95 lines
6.7 KiB
Markdown
Raw Normal View History

---
id: "0015"
date: 2026-06-09
status: Accepted
supersedes:
superseded-by:
affected-paths: []
affected-components: []
migration_confidence: medium
migration_source: "docs/memory-system/03-architecture-decisions.md### ADR-015 — memsearch episodic memory version-controlled in a dedicated private repo, auto-synced via cc-os SessionEnd hook"
---
# 0015 — memsearch episodic memory version-controlled in a dedicated private repo, auto-synced via cc-os SessionEnd hook
## Context
memsearch's memory store at `~/.memsearch` accumulates daily session-summary
files (`memory/YYYY-MM-DD.md`) that are the only irreplaceable data in the episodic layer —
the Milvus Lite index (`milvus.db`) and the bge-m3 ONNX embeddings model are derived/
disposable and can be rebuilt at any time via `memsearch index`. With Step 4 (episodic layer)
now live, preserving episodic memory across machines and protecting against local disk loss
required a sync strategy. The question was: what venue, what scope, and who owns the sync?
memsearch's own stated design philosophy is "markdown files are the canonical data store; the
vector database is a derived index" and notes that markdown is "git-friendly" and the index
rebuildable from markdown — making git a natural fit for the markdown layer.
## Decision
`~/.memsearch` is a **dedicated git repo** on branch `main` with remote `origin`
pointing to a **private self-hosted Forgejo repo**
(`ssh://git@forgejo.swansoncloud.com:2222/jared/memsearch.git`;
web: `https://forgejo.swansoncloud.com/jared/memsearch`). A whitelist `.gitignore` tracks
**only** `memory/*.md` (the daily session files) and `.gitignore` itself. Excluded as
derived/disposable: `milvus.db` (Milvus Lite index — rebuildable any time via
`memsearch index`), `config.toml`, and the bge-m3 ONNX embeddings model (lives in
`~/.cache/huggingface`, not in the store). Auto-commit and push are wired into the **cc-os
memory plugin's own `session-end.sh`** hook, not the marketplace plugin. The appended block
guards on `~/.memsearch/.git` existing, runs `git add -A` (whitelist makes it safe), commits
only when something is staged (message: `memsearch: session memory <date>`), and pushes with
`timeout 30 ... || true`. The entire block is wrapped in a subshell with a trailing `|| true`
so it can **never fail session shutdown**.
- **Rationale**:
- **Dedicated repo (not folded into the vault or a project repo):** ADR-001 established
episodic and semantic/knowledge as separate systems by design, and `~/.memsearch` is a
global, cross-project store with no natural home in any single project repo. Committing
episodic session logs into the Obsidian vault repo would conflate the two systems and
violate the separation of concerns that ADR-001 is built on.
- **Whitelist — commit markdown, exclude rebuildable index/model/config:** consistent with
memsearch's own design philosophy ("markdown is canonical; index is derived") and with the
cc-os principle from ADR-008 ("sync the vault, not the indexes"). The 544 MB bge-m3 ONNX
model is not even in the store; the Milvus Lite DB rebuilds in one command. Committing them
would bloat the repo for zero durability gain.
- **Sync lives in cc-os hook, not the marketplace plugin:** the marketplace plugin's hook
scripts (`stop.sh`, `session-start.sh`, `user-prompt-submit.sh`, `session-end.sh`) perform
no git operations — only a read-only `git rev-parse --show-toplevel` for scoping. Adding
sync to a marketplace plugin that can be clobbered by an upstream update is fragile; owning
it in the cc-os `session-end.sh` keeps the sync logic under our control and version-tracked
in this repo.
- **Fail-safe contract:** the `|| true` wrap and `timeout 30` on push ensure that a network
outage, an unreachable Forgejo instance, or any git error cannot prevent a session from
closing cleanly. The SessionEnd hook's harness timeout is ~10 s, so push is effectively
capped; commits land locally first, and any unpushed commit is carried forward by the next
session's push (daily files are append-only, so no conflict risk).
- **Commingling — resolved 2026-06-09**: `~/.memsearch` aggregates session summaries across
**all clients** into one global store. This is an **explicit design choice**, not a tolerated
risk: the user deliberately accepts a single commingled global store across all clients.
Private self-hosted Forgejo (user-owned infrastructure) is the chosen sync venue; client-
agreement compliance is the user's knowing responsibility. The concern was previously recorded
as user-owned and unresolved; that is now closed: single global store is the intended design.
This also aligns with a forward direction of minimizing dedicated per-client working
directories — since memsearch captures memory globally regardless of cwd, a single
`clients/` area for non-project client work becomes viable (direction stated; not yet designed).
## Consequences
~/.memsearch becomes its own git repo synced to a private self-hosted Forgejo remote, tracking only the daily markdown session files (via a whitelist .gitignore) while excluding the derived/disposable Milvus index, config, and embeddings model; sync is wired into cc-os's own SessionEnd hook rather than the marketplace plugin, wrapped in a fail-safe subshell so it can never block session shutdown. The design explicitly accepts a single global store commingling all clients' episodic memory as an intentional choice, with client-agreement compliance left as the user's responsibility.
## Alternatives rejected
- **Fold into the Obsidian vault repo:** violates ADR-001's episodic/semantic separation.
Episodic logs have different lifecycle (accrete and decay), different write patterns
(auto-captured every session), and would clutter a vault intended for curated durable
knowledge.
- **Auto-commit in the marketplace plugin:** the marketplace plugin can be overwritten on
update, losing the sync logic silently. Out-of-band ownership in cc-os is safer.
- **Commit the Milvus Lite index or the embeddings model:** both are large binaries, derived
from the markdown source, and rebuildable. Committing them wastes space and provides no
additional durability. The markdown files are the canonical source; the index follows from
them.
- **Syncthing or rsync instead of git:** git provides both version history and conflict-free
daily-append semantics; Syncthing is continuous-async (suitable for the vault where changes
are sparse); git's push-on-session-end cadence matches how memsearch produces data (one
daily file per day, append-only). Git was already chosen for ADR-008's vault sync rationale;
applying the same mechanism to the episodic store is consistent.