From 98f829063189a9aa26953a5f22ac41412eceb793 Mon Sep 17 00:00:00 2001 From: jared Date: Wed, 17 Jun 2026 10:57:06 -0400 Subject: [PATCH] Refine memory-project onboarding: type-based exclude taxonomy + 16k-context model Assessment now reasons about ignorable content by category (fetched deps, build output, caches, tooling/AI-assistant dirs, lockfiles, bulk data, binaries) so it generalizes across stacks. Onboarding switches to the config.yaml model qwen25-coder-7b-16k: inference is GPU-bound at ~65 tok/s, so the speed lever is fewer chunks per doc via a larger context window (GRAPHIFY_OLLAMA_NUM_CTX does not propagate through the OpenAI-compatible path; context must come from the modelfile). Extends ADR-017. Co-Authored-By: Claude Opus 4.8 --- CLAUDE.md | 4 +-- .../03-architecture-decisions.md | 6 +++- plugins/memory/skills/memory-project/SKILL.md | 31 +++++++++++++++---- 3 files changed, 32 insertions(+), 9 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 036cd4a..df341fd 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -69,14 +69,14 @@ to those two and fix the stale doc. **Implementation status (2026-06-15):** Step 5b done and automated — vault (`~/Documents/SecondBrain`) initialized as git repo, pushed to private Forgejo (`ssh://git@forgejo.swansoncloud.com:2222/jared/SecondBrain.git`); 52 files, git chosen over Syncthing. Auto-commit+push wired via `vault_sync.py` SessionEnd hook (third SessionEnd hook, runs after `session_end.py` so the daily journal note is included; push-only — SessionStart pull is an optional future item for multi-machine). Step 2e pilot done — llf-schema (`/home/jared/dev/llf-schema`, PHP 8.2 WordPress plugin) onboarded via `memory-project` skill: 605 nodes / 930 edges / 52 communities; Ollama doc pass lossy on WordPress docs but AST pass solid. Step 2d closed — live vault `graphify-out/` is the baseline; fixture-only build superseded. -**Implementation status (2026-06-17):** `memory-project` skill now uses assessment-first onboarding: surveys the repo structure, generates a per-project `.graphifyignore` file to exclude non-code directories (node_modules/, etc.), confirms the ignore rules with the user, then extracts the graph. Fixes issue where repos with large dependency trees routed non-code files through the Ollama doc pass (see ADR-017). +**Implementation status (2026-06-17):** `memory-project` skill now uses assessment-first onboarding: surveys the repo structure, classifies exclude candidates by TYPE (11 categories: fetched deps, build output, caches, VCS internals, editor/AI-tooling dirs, lockfiles, coverage/logs, bulk data, binaries, secrets, graphify-out/ — illustrative names, not fixed templates), generates a per-project `.graphifyignore`, confirms with the user, then extracts the graph using `qwen25-coder-7b-16k` (config.yaml `ollama_model`; 16k-context build of qwen2.5-coder:7b — larger context window cuts chunk count per doc, the main speed lever). Fixes issue where repos with large dependency trees routed non-code files through the Ollama doc pass (see ADR-017). ## Implemented Components **Global memory plugin** — `cc-os/plugins/memory/` (git-tracked, 2026-06-12); symlinked into `~/.claude/plugins/memory` - Hooks: `hooks/` — `session_start.py`, `session_context.py` (project graph path only), `post_tool_use_write.py`, `session_end.py` (vault journal), `memsearch_sync.py` (second SessionEnd hook; memsearch auto-commit+push, 30s timeout), `vault_sync.py` (third SessionEnd hook; vault auto-commit+push to forgejo.swansoncloud.com/jared/SecondBrain, 30s timeout; mirrors memsearch_sync.py) - Shared modules: `config.py` (load_config → frozen Config dataclass), `hook_io.py` (read_input → HookInput dataclass), `session_state.py` (record_touch/read_touches; encapsulates `/tmp/claude-vault-touched-$SESSION_ID` contract) -- Skills: `skills/` — memory-vault, memory-write, memory-reorganize, memory-project (assessment-first onboarding: surveys repo, writes per-project `.graphifyignore`, confirms with user, then extracts — per ADR-017) +- Skills: `skills/` — memory-vault, memory-write, memory-reorganize, memory-project (assessment-first onboarding: surveys repo, classifies excludes by type (11 categories), writes per-project `.graphifyignore`, confirms with user, then extracts using `qwen25-coder-7b-16k` — per ADR-017) - Config: `config.yaml` — vault path, Ollama model (qwen25-coder-7b-16k), env vars - Hook wiring: `~/.claude/settings.json` (hook entries invoke `/usr/bin/python3` with absolute paths into cc-os) diff --git a/docs/memory-system/03-architecture-decisions.md b/docs/memory-system/03-architecture-decisions.md index c5fb028..b2e721a 100644 --- a/docs/memory-system/03-architecture-decisions.md +++ b/docs/memory-system/03-architecture-decisions.md @@ -1,6 +1,6 @@ # Architecture Decision Records -_Last updated: 2026-06-12_ +_Last updated: 2026-06-17_ A running log of decisions and *why*. Format per entry: Context · Decision · Rationale · Alternatives rejected · Status. Newest decisions extend the log; supersede rather than delete. @@ -533,6 +533,10 @@ _Date: 2026-06-17_ - **Alternatives rejected**: **Static `.graphifyignore` template** — rejected; dependency/build/cache dirs vary by project stack; a template would both over-exclude (suppressing wanted source) and under-exclude (missing stack-specific dirs) for any given project. **Auto-write without user confirmation** — rejected; borderline directories (e.g. migrations, seeds, fixtures) require human judgment; mechanical exclusion would silently drop legitimate content. **AST-only mode (skip doc pass entirely)** — deferred; the `--backend ollama` flag stays; the ignore list is the correct cost lever, not backend switching. - **Status**: Accepted 2026-06-17. +**Addendum — 2026-06-17:** +- **Default-exclude taxonomy is type-based, not name-based.** The skill's assessment step organizes exclude candidates into 11 categories by KIND (fetched deps, build/generated output, caches, VCS internals, editor/AI-tooling dirs, lockfiles, coverage/logs, bulk data/databases, binaries, secrets, graphify-out/) with a meta-principle: index what a human authored; exclude anything fetched, generated, cached, tooling config, bulk data/binary, or secret. Example names within each category are illustrative, not exhaustive — the goal is to recognize the kind even in an unfamiliar stack. See SKILL.md for the full table. +- **Onboarding uses the 16k-context model.** The `graphify extract` command now passes `--model qwen25-coder-7b-16k` (the `ollama_model` from config.yaml), a 16k-context build of qwen2.5-coder:7b. Inference is GPU-bound (~65 tok/s); the main speed lever is chunk-count × generation — fewer chunks per doc via a larger context window. `GRAPHIFY_OLLAMA_NUM_CTX` does not propagate through graphify's OpenAI-compatible endpoint; the larger context must come from the model's Modelfile. + ## Rejected tools (summary) | Tool | Why rejected for our use | diff --git a/plugins/memory/skills/memory-project/SKILL.md b/plugins/memory/skills/memory-project/SKILL.md index e91264e..ceb8461 100644 --- a/plugins/memory/skills/memory-project/SKILL.md +++ b/plugins/memory/skills/memory-project/SKILL.md @@ -8,13 +8,30 @@ description: Manage per-project Graphify knowledge graphs — onboard, update, r Assessment-first onboarding per ADR-017. Do not run `graphify extract` bare — graphify does not honor `.gitignore`; it uses a separate `.graphifyignore` (same syntax). Without one, repos with `node_modules/` or other dependency/build/cache dirs will walk every file and route all non-code content through the slow Ollama doc pass. Only non-code files hit the LLM; code uses the free tree-sitter AST pass. -**Step 1 — Assess the repo.** Survey before touching graphify. List top-level dirs and get a file-type/size profile. Identify: +**Step 1 — Assess the repo.** Survey before touching graphify. List top-level dirs and get a file-type/size profile. Identify what to exclude using the taxonomy below, then surface borderline cases (migrations, seeds, fixtures, sample data) for a human call — do not make that judgment unilaterally. + +**Meta-principle:** index what a human authored as this project's source and knowledge; exclude anything that is (a) fetched, (b) generated/derived/compiled, (c) cached, (d) tooling/environment config, (e) bulk data or binary, or (f) secret. Recognize the KIND even in an unfamiliar stack — the example names are illustrative, not exhaustive. Weight remaining cost by **non-code file count**: only non-code files go through the Ollama doc pass (~15–25s each), so a stray docs-heavy tooling dir or a large lockfile is what blows up runtime, not source-file count. + +**Default-exclude categories:** + +| # | Category | Illustrative names | +|---|----------|--------------------| +| 1 | Fetched dependencies | `node_modules/`, `vendor/`, `.venv/`/`venv/`, `target/`, `Pods/`, `site-packages/` | +| 2 | Build / compiled / generated output | `dist/`, `build/`, `out/`, `.next/`, `.svelte-kit/`, `bin/`, `obj/`, `*.min.js`, source maps, generated dirs | +| 3 | Caches | `.cache/`, `.vite/`, `.turbo/`, `__pycache__/`, `.pytest_cache/`, `.gradle/`, `.terraform/` | +| 4 | VCS internals | `.git/`, `.hg/`, `.svn/` | +| 5 | Editor / IDE & AI-assistant tooling dirs | `.vscode/`, `.idea/`, `.claude/`, `.codex/`, `.cursor/`, `.pi/`, `.impeccable/`, `.husky/` — these are full of markdown that floods the doc pass | +| 6 | Lockfiles (machine-generated, hit the LLM) | `package-lock.json`, `yarn.lock`, `pnpm-lock.yaml`, `poetry.lock`, `Cargo.lock`, `composer.lock`, `Gemfile.lock` | +| 7 | Coverage / reports / logs | `coverage/`, `htmlcov/`, `test-results/`, `playwright-report/`, `*.log`, `logs/` | +| 8 | Data, databases & bulk dumps | `*.db`, `*.sqlite`, `pb_data/`, large `*.csv`/`*.json`/`*.parquet`, snapshots — keep small schema/seed files that define structure; drop bulk data | +| 9 | Binary & media | images, audio, video, fonts, archives, PDFs/Office docs unless they ARE the knowledge | +| 10 | Secrets / env | `.env`, `*.pem`, credentials | +| 11 | graphify-out/ | graphify's own output — avoid self-ingestion | + +For non-coding/writing/research projects the same principle holds but offenders shift: exported duplicates of source notes (PDF/HTML/DOCX exports of the same markdown — index the source, not the export), attachment/media folders, app config (`.obsidian/`), and archived old-version folders. -- **Exclude candidates:** dependency dirs (`node_modules/`, `vendor/`, `venv/`, `.venv/`); build/generated output (`dist/`, `build/`, `.next/`, `target/`, `pb_public/`); caches (`.vite/`, `coverage/`, `__pycache__/`); VCS internals (`.git/`); lockfiles; databases and logs (`*.db`, `*.log`); binary/media blobs; and `graphify-out/` itself. - **Include candidates:** source code (free AST pass) and genuine docs (README, design docs, ADRs). -Weigh cost by non-code file count, not directory size. Surface borderline cases (migrations, seeds, fixtures, sample data) for a human call — do not make that judgment unilaterally. - **Step 2 — Draft `.graphifyignore`.** Write `.gitignore`-syntax rules from the assessment. Add a one-line rationale comment above each rule so the intent is clear later. **Step 3 — Confirm with the user.** Present the proposed ignore list and rationale; adjust per feedback before writing anything. This is a judgment step, not a mechanical one — do not skip it. @@ -24,10 +41,12 @@ Weigh cost by non-code file count, not directory size. Surface borderline cases **Step 5 — Build the graph.** ```bash -graphify extract . --backend ollama --model qwen2.5-coder:7b +graphify extract . --backend ollama --model qwen25-coder-7b-16k graphify cluster-only . ``` +`qwen25-coder-7b-16k` is the `ollama_model` in `config.yaml` — a 16k-context build of qwen2.5-coder:7b. Its larger context window reduces the number of chunks per document, which is the main onboarding-speed lever: inference is GPU-bound (~65 tok/s), so the cost is chunk-count × generation, not hardware. Note: `GRAPHIFY_OLLAMA_NUM_CTX` does not propagate through graphify's OpenAI-compatible path; the larger context comes from the model's own Modelfile. + Confirm `./graphify-out/graph.json` exists before reporting done. ## Incremental update @@ -61,5 +80,5 @@ For traversal mechanics (`graphify path`, `graphify explain`, `--budget`, `--dfs ## Constraints - **Never commit `graphify-out/`** — it is local, disposable, and rebuildable. -- Model is `qwen2.5-coder:7b` — do not substitute. +- Model is `qwen25-coder-7b-16k` (config.yaml `ollama_model`) — do not substitute. This is a 16k-context build of qwen2.5-coder:7b; using the smaller base model silently reduces context and increases chunk count. - The `.gitignore` entry for `graphify-out/` must survive even if the directory is deleted.