10 KiB

Raw Blame History

Memory Plugin Hook Tests

Golden fixture tests for the four Python hooks in ../hooks/.

Directory layout

tests/
  harness.sh             # Reusable per-scenario runner
  generate-fixtures.sh   # Runs all scenarios and writes golden/
  inputs/                # Source input JSON files (with __SANDBOX__ placeholders)
  golden/
    <scenario>/
      input.json         # Input fed to hook on stdin (with __SANDBOX__ expanded)
      stdout.txt         # Normalized hook stdout
      exit_code.txt      # Hook exit code
      sideeffects/       # Files and flags created by the hook (normalized)

Sandbox design

Each scenario gets a fresh SANDBOX=$(mktemp -d) directory (path-canonicalized via pwd -P to prevent symlink leaks). The harness sets HOME="$SANDBOX" so all hook code that resolves ~/... paths lands inside the sandbox, not in the real home directory.

Key sandbox contents created before each hook run:

$SANDBOX/.claude/plugins/memory/config.yaml — points vault_path and graphify_output_dir at sandbox-local dirs
$SANDBOX/vault/journal/ — journal target for session_end.py
$SANDBOX/.cache/graphify/ — stamp/lock/log directory for session_start.py
$SANDBOX/bin/graphify — shim that logs calls to graphify-shim.log, exits 0
$SANDBOX/bin/git — safety shim (see below)

PATH="$SANDBOX/bin:$PATH" is exported so shims take priority over system binaries.

Git shim safety

memsearch_sync.py reads MEMSEARCH_DIR from Config.memsearch_dir (no longer hardcoded) — but the resolved path is still absolute and bypasses HOME=$SANDBOX. The git shim is the only protection against the hook committing or pushing to the real memsearch repository.

The shim at $SANDBOX/bin/git:

Scans every argument for the string /home/jared/.memsearch
If found: logs the blocked call to $SANDBOX/git-shim.log and exits 0
If not found: exec /usr/bin/git "$@" (passthrough)

Critical: the log path is baked into the shim file at write-time (absolute literal), not passed only via env var. This ensures the log is captured even inside session-end's ( ... ) || true subshell.

The shim emits nothing to stdout or stderr — only writes to the log file. This matters because memsearch_sync.py runs git diff --cached --quiet without redirecting stdout, and any shim chatter would appear in captured stdout and corrupt golden fixtures.

Normalization rules

Applied in this order (order matters: timestamp before date-only avoids partial corruption):

ISO 8601 timestamps YYYY-MM-DDTHH:MM:SSZ → __TIMESTAMP__
Date-only YYYY-MM-DD → __DATE__
Sandbox temp path $SANDBOX → __SANDBOX__
cwd=<path> log values → cwd=__CWD__ (varies by machine and harness invocation dir)
Real home path /home/jared → __REAL_HOME__

Scenarios

session-start group

Scenario	Setup	What it tests
`session-start-fresh`	Stamp file exists (fresh mtime)	Rebuild NOT triggered; log says "rebuild not needed"
`session-start-stale`	No stamp file	Rebuild triggered via nohup; graphify shim called with vault path

Side effects captured: memory-hook-log-delta.txt (lines added to /tmp/memory-hook.log), graphify-shim-args.txt (stale only — polled up to 5 seconds for background call).

session-context group

Scenario	Setup	What it tests
`session-context-first-with-graph`	Real git repo + `graphify-out/graph.json`	First prompt injects graph path in additionalContext
`session-context-reentry`	Flag file pre-existing	Returns `{}` immediately (no-op on re-entry)
`session-context-no-graph`	Real git repo, no `graphify-out/graph.json`	Returns empty additionalContext

Side effects captured: flag-file-created.txt (yes/no for all session-context scenarios).

post-tool-use-write group

Scenario	Setup	What it tests
`post-tool-use-vault-md`	Stamp exists, vault/notes dir	Touch file written, stamp deleted, log entry created
`post-tool-use-non-md`	vault/notes dir	Non-.md file → all guards fire, no side effects
`post-tool-use-outside-vault`	(none)	Outside-vault path → no side effects

Side effects captured: touch-file.txt, post-tool-use-log.txt, stamp-deleted.txt (all post-tool-use scenarios; absent/empty for non-md and outside-vault); noops.txt confirmation for non-md and outside-vault.

session-end group

Scenario	Setup	What it tests
`session-end-with-touches`	Touch file with two paths	Journal written, touched paths listed, touch file deleted, memsearch git ops blocked
`session-end-no-touches`	No touch file	Journal written with "(none)", memsearch git ops blocked

Side effects captured: journal.txt (found by glob, not date recomputation), git-shim-log.txt, touch-file-deleted.txt (all session-end scenarios).

Expected absence: the shim causes git diff --cached --quiet to exit 0 (reads as "no staged changes"), so session_end.py / memsearch_sync.py never reaches the commit branch. The fixture stdout therefore does NOT contain [session-end] memsearch memory committed. This is correct behaviour — the shim is protecting the real repo, not a real commit being skipped.

Project field: session_end.py runs git rev-parse --show-toplevel from the hook's cwd. Scenarios run from /tmp with CLAUDE_PROJECT_DIR unset, so **Project:** unknown in the journal fixture is expected.

Running all scenarios

The bash hooks (hooks/*.sh) were removed after parity was verified — they live in git history only. The only supported invocation uses the Python hooks via tests/python-wrappers/:

cd /home/jared/dev/cc-os/plugins/memory/tests
HOOKS_DIR="$(pwd)/python-wrappers" bash generate-fixtures.sh

The script:

Records the real memsearch HEAD before starting
Cleans /tmp flag files before each scenario
After EACH session-end scenario: immediately verifies HEAD is unchanged
Final check: prints SAFETY CHECK PASSED or exits non-zero

Replaying against the Python hooks

The Python hooks are canonical. The bash .sh files were removed after parity was verified (2026-06-12) and exist only in git history. To validate the Python hooks against golden fixtures (e.g. after modifying a hook):

Ensure the Python hooks are executable in ../hooks/ and the wrappers are in tests/python-wrappers/.

Run generate-fixtures.sh via python-wrappers with a replay dir:

HOOKS_DIR=/home/jared/dev/cc-os/plugins/memory/tests/python-wrappers \
  bash tests/generate-fixtures.sh --replay-dir /tmp/replay-out

This writes new fixtures to /tmp/replay-out using the Python hooks.

Diff against the committed golden directory:
```
diff -r tests/golden/ /tmp/replay-out/
```
An empty diff means the Python port is behaviorally identical. Any diff is a parity failure.
Clean up: rm -rf /tmp/replay-out

Note: --replay-dir skips the strict pre-run HEAD equality check (the hash may have advanced since golden fixtures were generated). The before==after safety invariant is always enforced.

Parity caveats and known nondeterminism

session-start-stale is inherently racy. The hook spawns graphify via nohup ... & and exits immediately. The harness polls for the shim log up to 5 seconds. On heavily loaded systems the poll may time out; if so, graphify-shim-args.txt will say (graphify not invoked within 5s) — re-run in that case. The Python port uses the same detached spawn pattern.
session-start cwd in log lines is normalized to __CWD__. The hook logs cwd=$(pwd) (shell invocation directory, not the JSON cwd field). This varies by machine and harness run location, so it is masked. The Python port should produce the same log format with the actual cwd; the harness normalization will mask it identically.
session-context stdout JSON must match exactly. The fixture captures output of json.dumps(...) with no extra whitespace. The Python port must serialize with the same key order and no trailing newline inside the JSON value. Diff will fail on whitespace differences even if the data is semantically identical.
Journal filename is date-based (YYYY-MM-DD.md). Masked to __DATE__ in normalized content. The harness finds the journal file by glob rather than recomputing today's date, so it works across midnight boundaries.
session-end **Project:** unknown — session-end hooks run from /tmp with no git repo and CLAUDE_PROJECT_DIR unset. The Python port should produce identical output.

Replaying session-end scenarios safely

memsearch_sync.py reads the memsearch path from Config.memsearch_dir (not hardcoded), but the resolved path is still absolute and bypasses the sandbox. The sequence to follow:

# 1. Record HEAD before
EXPECTED=$(git -C /home/jared/.memsearch rev-parse HEAD)
echo "Expect HEAD: $EXPECTED"

# 2. Shim preflight (make_sandbox populates $SANDBOX/bin/git already)
"$SANDBOX/bin/git" -C /home/jared/.memsearch status 2>/dev/null
cat "$SANDBOX/git-shim.log"   # must show BLOCKED line

# 3. Run scenario (generate-fixtures.sh does this automatically)

# 4. Verify HEAD immediately after
AFTER=$(git -C /home/jared/.memsearch rev-parse HEAD)
[ "$AFTER" = "$EXPECTED" ] && echo "PASS" || echo "BREACH: $AFTER"

Never run session-end scenarios with the real git binary on PATH before confirming the shim intercepts the memsearch path.

Adding new scenarios

Add input JSON to tests/inputs/<scenario-name>.json
- Use __SANDBOX__ as a placeholder where the sandbox temp path is needed
Add a run_<scenario_name>() function in generate-fixtures.sh following the existing pattern: make_sandbox → setup → expand input → run hook → write_fixture → capture side effects → rm -rf "$sb"
Add the call to the bottom of generate-fixtures.sh (before the final safety check)
Re-run bash generate-fixtures.sh to generate the golden files
Review the normalized output, then commit tests/golden/<scenario-name>/ to source control

Side effect files that must be stable across machines: use normalize() on all paths, dates, and timestamps before writing. If a side effect contains machine-specific data that cannot be normalized, omit it from the golden fixture and note the omission in a sideeffects/NOTES.txt.

10 KiB Raw Blame History