From a722a2db6501e10b1547aed70fbf7b374b2e9f86 Mon Sep 17 00:00:00 2001 From: jared Date: Thu, 2 Jul 2026 13:02:40 -0400 Subject: [PATCH] Add gitignore, RAM incident diagnosis, and new docs Confirms bad RAM (memtest86+ single-bit flips) as root cause of the 2026-06-24 Btrfs corruption incident, adds fedora-upgrade.md and other pending docs, and gitignores the doc-hygiene state directory. Co-Authored-By: Claude Sonnet 5 --- .gitignore | 1 + CLAUDE.md | 1 + docs/cc-tool-audit.md | 268 ++++++++++++++++++ docs/fedora-upgrade.md | 70 +++++ docs/incidents.md | 69 +++++ docs/orchestration-audit-prd.md | 97 +++++++ .../plans/2026-05-07-n8n-backup-monitor.md | 48 ++-- 7 files changed, 530 insertions(+), 24 deletions(-) create mode 100644 .gitignore create mode 100644 docs/cc-tool-audit.md create mode 100644 docs/fedora-upgrade.md create mode 100644 docs/orchestration-audit-prd.md diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..0d13ce0 --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +.dochygiene/ diff --git a/CLAUDE.md b/CLAUDE.md index 5256b1e..21a1095 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -35,6 +35,7 @@ Do NOT use `^(...)$` wrapping — Hyprland 0.51 rejects it as "Invalid rulev2". - `docs/window-manager/` — migration plan, phased progress log, keybindings cheatsheet - `docs/hyprland/` — version-aware Hyprland config reference; update these when Hyprland is upgraded +- `docs/fedora-upgrade.md` — distro release-upgrade procedure; F43→44 deferred (solopasha COPR aquamarine not rebuilt vs F44 libdisplay-info) - `docs/synology.md` — DS218plus SSH alias, `/volume1/backup` map, restic repo commands, recovery notes - `docs/vps-inventory.md` — OVH VPS (`ovh-vps`) access notes; old Hyperthrive DO VPS deleted 2026-06-24 (historical notes retained) - `docs/incidents.md` — dated incident log (diagnosed-and-fixed issues): symptom, root cause, ruled-out, fix, follow-up diff --git a/docs/cc-tool-audit.md b/docs/cc-tool-audit.md new file mode 100644 index 0000000..6e69bf0 --- /dev/null +++ b/docs/cc-tool-audit.md @@ -0,0 +1,268 @@ +# Claude Code tool & context audit + +**Date:** 2026-06-30 +**Question:** Are too many built-in CC tools loaded into context, slowing sessions and +hurting focus? Which can be excluded globally vs. situationally? + +**Method:** Mechanically parsed every tool-call (`tool_use`) across **224 main +sessions + 1,946 subagent transcripts = 2,170 transcripts** in `~/.claude/projects/`, +cross-referenced against the full tools reference +(https://code.claude.com/docs/en/tools-reference.md), and measured the actual +bytes loaded into one representative session's startup context. + +--- + +## TL;DR — the hypothesis is mostly outdated + +Excluding **built-in tools** used to meaningfully shrink the prompt. It no longer +does, because Claude Code now **defers** rarely-used tool schemas (tool search): +they load on demand via `ToolSearch`, not at startup. + +- Total *static* startup context in a typical session here ≈ **6,300 tokens**. +- Built-in tool schemas are a small slice of that; **MCP cost = 0** (no MCP servers + configured). **Skills (50) ≈ 3,700 tok** are the single largest contributor. +- Of **24 built-in tools never used in 2,170 transcripts**, only **one** (`Workflow`) + is both always-loaded *and* unused. The other 23 are already deferred, platform- + impossible, or trivially small. + +So: a confident global *tool* exclusion list exists, but its payoff is small. The +real levers for "feels slow / unfocused" are elsewhere (see §5). + +> ⚠️ Honest caveat: trimming context lowers prefill latency and cost and may sharpen +> focus, but it does **not** change Opus's per-token generation speed. Opus is simply +> slower to generate than Sonnet/Haiku; no amount of tool pruning fixes that. + +--- + +## 1. What tools actually get used (2,170 transcripts) + +| Tool | Calls | Transcripts using it | +|---|---:|---:| +| Bash | 9,168 | 1,248 | +| Read | 5,955 | 1,185 | +| Edit | 1,983 | 501 | +| Agent | 1,948 | 541 | +| Write | 981 | 382 | +| WebFetch | 288 | 72 | +| ToolSearch | 169 | 139 | +| WebSearch | 124 | 36 | +| AskUserQuestion | 108 | 70 | +| Skill | 104 | 84 | +| TaskUpdate / TaskCreate | 54 / 31 | 10 / 7 | +| SendMessage | 40 | 27 | +| Monitor | 21 | 12 | +| TaskStop / ScheduleWakeup | 8 / 4 | 5 / 3 | +| RemoteTrigger / ReportFindings | 1 / 1 | 1 / 1 | + +**Never used anywhere (0 calls):** `Artifact`, `CronCreate/Delete/List`, +`EnterPlanMode/ExitPlanMode`, `EnterWorktree/ExitWorktree`, `Glob`, `Grep`, +`ListMcpResourcesTool`, `LSP`, `NotebookEdit`, `PowerShell`, `PushNotification`, +`ReadMcpResourceTool`, `SendUserFile`, `ShareOnboardingGuide`, +`TaskGet/TaskList/TaskOutput`, `TodoWrite`, `WaitForMcpServers`, `Workflow`. + +> Note `Grep`/`Glob` show **0** — search runs through `Bash` (ripgrep/find directly, +> reinforced by the RTK proxy). They are still worth keeping (see §4). + +--- + +## 2. The decisive cross-reference: never-used ∩ actually-loaded + +The "never used" list conflates three very different things. Only the first bucket +gives back real context when excluded. + +| Bucket | Tools | Exclusion payoff | +|---|---|---| +| **Always-loaded & unused** | `Workflow` | **Real** — large schema | +| **Already deferred** (loaded on demand, not at startup) | `CronCreate/Delete/List`, `EnterPlanMode/ExitPlanMode`, `EnterWorktree/ExitWorktree`, `LSP`, `NotebookEdit`, `PushNotification`, `TaskGet/List/Output` | **~0** — not in the prompt anyway | +| **Platform/plan-impossible or not loaded here** | `PowerShell` (Linux), `Artifact`/`ShareOnboardingGuide` (need Team/Ent.), `SendUserFile`/`PushNotification` (need Remote Control), `TodoWrite` (disabled by default), `WaitForMcpServers` (only when tool-search is OFF), `Glob`/`Grep`, MCP-resource tools | **Cosmetic** — either absent or trivially small | + +**This is why manual exclusion mattered more in the past:** tool-search deferral now +does the trimming automatically. + +--- + +## 3. Where the startup context actually goes (~6,300 tok measured) + +| Contributor | Est. tokens | # items | +|---|---:|---:| +| **Skills** (name + description) | ~3,700 | 50 | +| **Subagent types** | ~917 | 18 | +| CLAUDE.md (project) | ~842 | 1 | +| MEMORY.md auto-index | ~427 | 1 | +| RTK.md | ~241 | 1 | +| CLAUDE.md (global) | ~148 | 1 | +| **MCP tool schemas** | **0** | 0 servers | +| **Total** | **~6,300** | | + +Skills + agents = ~74% of static context. Heaviest single items: the built-in +`claude-api` skill (~269 tok), personal `the-humanizer` (~239), `impeccable` (~226), +`memsearch:memory-recall` (~185). + +--- + +## 4. Recommendation A — global tool exclusions (small but safe) + +Add to `~/.claude/settings.json` `permissions.deny`. The mechanism: a denied tool is +removed from the advertised set (its schema isn't loaded), per the tools reference +("to disable a tool entirely, add its name to the deny array"). Verify by checking the +session token count before/after if you want proof. + +```jsonc +{ + "permissions": { + "deny": [ + "Workflow", // 0/2170 uses; largest always-loaded built-in schema. + // KEEP it ONLY if you use /workflows or "ultracode". + "PowerShell", // Linux host — never available. Cosmetic. + "Artifact", // needs Team/Enterprise plan. Cosmetic. + "ShareOnboardingGuide" // paid team feature. Cosmetic. + ] + } +} +``` + +- **`Workflow` is the only entry with real payoff.** It is the multi-agent + orchestration tool (the big JS-script one) — *not* `Agent`. You use `Agent` + constantly (1,948 calls) and `Workflow` never. Keep `Agent`; drop `Workflow` + unless/until you adopt `/workflows`. +- **Do NOT deny** `Glob`/`Grep` (tiny; the sanctioned search path — denying pushes + everything to `Bash grep`/`find`, which can add permission prompts), anything + **deferred** (no savings), or plan-mode/worktree/Task tools (deferred *and* + UI-driven — denying `ExitPlanMode` would break Shift-Tab plan mode). + +Net context saved: roughly the `Workflow` schema. Honest, but modest. + +--- + +## 5. Recommendation B — the real lever: situational plugins/skills + +Skills/agents (~4,600 tok) dwarf the built-in-tool headroom, and they load *every* +session regardless of relevance. These are controlled per-machine via +`enabledPlugins` in `~/.claude/settings.json` (not easily per-project), so the +practical move is toggling plugins by what you're doing. + +Currently enabled: `superpowers` (already disabled ✓), `ruby-lsp`, `perspectives`, +`git-context`, `invoice-ninja`, `memory`, `memsearch`, `doc-hygiene`, `codex`. + +| Plugin / skill group | Relevant to | Drop when not… | +|---|---|---| +| `codex:*` (5 skills + agent) | coding handoffs to Codex | …coding | +| `ruby-lsp` | Ruby projects | …in a Ruby repo | +| `perspectives:*` (9 agents) | design/plan critique | …planning/designing | +| `invoice-ninja` | freelance billing | …doing client billing | +| `deep-research`, web tools | research | …researching | +| personal writing/UI: `the-humanizer`, `impeccable`, `edit-article`, `prototype` | content/frontend | …writing or building UI | +| `doc-hygiene:*` | docs repos | …maintaining docs | +| `memory:*` + `memsearch:*` | cross-session recall | rarely — but see below | + +**Suggested situational profiles** (toggle `enabledPlugins`): + +- **Coding repo:** keep `codex`, `ruby-lsp` (if Ruby), `git-context`. Drop + `invoice-ninja`, writing/UI skills, `deep-research`, `perspectives`. +- **Freelance / PM / client:** keep `invoice-ninja`, `git-context`. Drop `codex`, + `ruby-lsp`, `perspectives`, coding skills (`tdd`/`diagnose`/`run`/`verify`). +- **Research / writing:** keep `deep-research`, writing skills. Drop `codex`, + `ruby-lsp`, `invoice-ninja`. +- **Infra / sysadmin (this repo):** Bash/Read/Edit-heavy. Drop writing/UI, + `invoice-ninja`, `codex`, coding frameworks; keep `doc-hygiene`, `git-context`. + +> Reality check: even the full ~6.3k static context is small next to a working +> session that fills 50k–150k+ tokens. Trimming skills is good hygiene and helps +> focus, but it is not where most of a slow session's time goes. + +--- + +## 6. What is probably actually slowing sessions + +Ranked by likely impact, from the data: + +1. **Opus generation speed.** Inherent; switch to Sonnet/Haiku for mechanical work, + or use `/fast` (faster Opus output) for interactive coding. +2. **Heavy `Agent` delegation.** 67% of sessions spawn subagents; 541 transcripts + used `Agent`. Each subagent is a fresh context with its own prefill + sequential + wall-clock wait. Great for big fan-out; pure latency for small lookups you could + do inline. +3. **Per-prompt / per-tool hooks.** The `memory` plugin registers all 5 hook types, + incl. **PostToolUse (runs after every tool call)** and SessionStart/UserPromptSubmit + (the large memory dump at session start). `memsearch` adds more. This is a + per-operation tax that tool-count pruning won't touch — worth timing/disabling to + test. **Most promising thing to measure next.** +4. **Long sessions** → context summarization passes. Use `/clear` between unrelated + tasks rather than carrying one giant session. +5. Static tool/skill context (this audit) — real but the **smallest** of these. + +--- + +## Appendix — four representative sessions (loaded vs. used) + +1. **Coding** (`llf-schema/19320d96`) — implement an OpenSpec change. 191 calls, only + **Edit/Read/Write/Bash**. No web, no `Agent`, no skills. Pure local code loop. +2. **Infra, wide** (`proxmox-hermes-01/c24faa0a`) — plan a Proxmox agent setup. + Used nearly the whole belt: Bash/Agent/WebFetch/WebSearch/Read/Edit/Write/ + AskUserQuestion/Skill(`git-context:repo-init`). The legit case for keeping web + tools loaded. +3. **Infra, ops** (`systems-admin/ffed234f`) — "how did the backup do this week?" + 50 calls, only **Bash/Read/Edit/Write**. Textbook shell-only profile. +4. **PM / orchestration** (`ovh-prod-01/991913d9`) — decide next server-consolidation + phase. Lead agent used only **Agent + AskUserQuestion**; 360+ tool calls happened + in 40 subagents. Never touched a file or shell itself. + +Three of four would have been unaffected by stripping every web tool, every skill, +and (for #1/#3) all delegation tooling. + +--- + +## Addendum (2026-06-30) — follow-up: skills, hooks, and orchestration + +### Skill loading — there IS a native lever (no "dispatcher" needed) + +Per the docs: a skill's **body** loads only on invocation ("costs almost nothing +until you need it"); only **name + description** load at startup, and that listing is +**budget-capped** (`skillListingBudgetFraction`, ~1% of context; `skillListingMaxDescChars` +default 1,536/skill). There is **no** skill-level `ToolSearch` equivalent. + +To reduce always-loaded skill weight, use **`skillOverrides`** in settings rather than +a community dispatcher skill: +- `"name-only"` — keep the skill available but drop its description from context. +- `"off"` — hide it entirely. +Plus `enabledPlugins`, `disableBundledSkills: true`. The dispatcher/index pattern is +**not** officially recommended — the budget + `skillOverrides` already do it. + +### Hooks are synchronous but cheap (measured on this machine) + +Hooks block the turn until exit (up to `timeout`); `"async": true` makes one +fire-and-forget. The `memory` plugin's hooks are synchronous but fast: +`post_tool_use_write` ~38 ms (Write|Edit only, early-exits off-vault), +`session_context` ~33 ms/prompt (injects ~0 tokens normally), +`rtk hook` ~24 ms/Bash. Negligible vs. multi-second model turns. **Not a latency +source.** The big memory dump is SessionStart-only (once). + +### Orchestration audit — subagent model mismatch (1,962 transcripts) + +Model is recorded per subagent transcript. Distribution: **Sonnet 36.7% · Haiku 36.5% +· Opus 26.6%**. By work profile: + +| Subagent model | n | % that did only file-edit/shell work | +|---|---:|---:| +| Opus | 521 | 53% (+38% spawned their own subagents) | +| Sonnet | 720 | 64% | +| Haiku | 716 | 83% (well-matched) | + +**Finding:** 27% of subagents run on **Opus**, most doing mechanical edits. Root cause: +when an `Agent` spawn omits `model`, the subagent **inherits the parent model (Opus)** — +so the CLAUDE.md routing table is advisory and leaks. + +**Subagent model resolution order** (docs): `CLAUDE_CODE_SUBAGENT_MODEL` env → +per-invocation `model` param → subagent frontmatter `model:` → inherited main model. + +**Enforcement fix:** set **`CLAUDE_CODE_SUBAGENT_MODEL=haiku`** (or `sonnet`) so +subagents default cheap instead of inheriting Opus; pin per-type `model:` frontmatter +for specialized agents; escalate by explicit criteria only. + +**Instruction tuning (pilot on one project first):** +1. Drop "no minimum complexity threshold" — direct-operate trivial single-file ops; + delegate only when parallelizable, large, or context-heavy. Mandatory delegation of + one-line edits is a net latency *loss*. +2. Make routing enforced, not advisory (env var above + "always pass model on spawn"). +3. Relax "never read before delegating" — allow a cheap orienting read when paths/ + structure are uncertain, to avoid blind-spec retry loops. diff --git a/docs/fedora-upgrade.md b/docs/fedora-upgrade.md new file mode 100644 index 0000000..dead26a --- /dev/null +++ b/docs/fedora-upgrade.md @@ -0,0 +1,70 @@ +# Fedora release upgrades + +Notes and decisions for distro version upgrades on this machine. The Hyprland +stack comes from the **solopasha COPR**, which is the recurring complication — +it must be rebuilt against the new Fedora release before the system upgrade can +resolve. + +## Standard upgrade procedure + +```bash +sudo dnf upgrade --refresh +sudo dnf install dnf-plugin-system-upgrade # usually already present +sudo dnf system-upgrade download --releasever=NN +sudo dnf system-upgrade reboot +``` + +## Status: Fedora 43 → 44 — DEFERRED (decided 2026-06-11) + +**Decision: stay on Fedora 43 and wait. Do not upgrade to 44 yet.** + +### Why deferred + +`sudo dnf system-upgrade download --releasever=44` fails to resolve because the +solopasha COPR has **not rebuilt `aquamarine` against Fedora 44**: + +- Fedora 44 ships `libdisplay-info-0.3.0`, which provides only `libdisplay-info.so.3`. +- The COPR's F44 build `aquamarine-0.9.5-2.fc44` still links against the old + `libdisplay-info.so.2` (verified — its `requires` list still names `.so.2`). + So it is uninstallable on F44. +- `hyprland` requires `aquamarine`, so it fails too. +- The old `libdisplay-info-0.2.0` cannot be kept — same package name (file + conflict), and `mutter` / `mesa-vulkan-drivers` / `wlroots` / `sway` all move + to `.so.3` versions on F44. + +This is **the sole blocker**. Everything else in the transaction resolves +cleanly. It is a stale COPR rebuild, not a local misconfiguration. There is no +native Hyprland/aquamarine in Fedora 44's repos, so the COPR is the only source. + +No strong reason to rush: F43 is fully supported and the current GNOME/Hyprland +setup works. Staying put until the COPR catches up. + +### How to proceed next time (re-checking F44) + +1. Refresh and re-attempt the resolve (no commitment — this only downloads): + ```bash + sudo dnf upgrade --refresh + sudo dnf system-upgrade download --releasever=44 + ``` +2. **If it resolves** (no `libdisplay-info.so.2` / aquamarine error): the COPR + has rebuilt. Proceed with `sudo dnf system-upgrade reboot`. +3. **If it still errors on aquamarine / `libdisplay-info.so.2`:** the COPR is + still behind. Quick check of the dep: + ```bash + dnf repoquery --releasever=44 --repo='copr:*solopasha*' --requires aquamarine | grep display + ``` + `libdisplay-info.so.2` = still stale, keep waiting. `libdisplay-info.so.3` = + ready to upgrade. + +### Override path (only if F44 becomes urgent) + +Proceed now by temporarily dropping the Hyprland stack and running on the GNOME +fallback. Reinstall Hyprland from the COPR once it is rebuilt. + +```bash +sudo dnf remove aquamarine hyprland hyprland-uwsm xdg-desktop-portal-hyprland +sudo dnf system-upgrade download --releasever=44 +sudo dnf system-upgrade reboot +# after reboot into F44, log into GNOME; reinstall when COPR is ready: +# sudo dnf install hyprland hyprland-uwsm xdg-desktop-portal-hyprland +``` diff --git a/docs/incidents.md b/docs/incidents.md index 137dc14..180e1d8 100644 --- a/docs/incidents.md +++ b/docs/incidents.md @@ -4,6 +4,75 @@ Dated entries for diagnosed-and-fixed system incidents on this machine. Newest first. Keep entries factual: symptom, root cause, what was ruled out, fix applied, and any follow-up. +## 2026-06-26 — memtest86+ confirms bad RAM (root cause of the 2026-06-24 Btrfs corruption) + +**Result: FAILED.** Ran memtest86+ v8.10 from the GRUB boot entry on the +Ryzen 5 5600G / 95.3 GB DDR4-2666 (CAS 20-19-19-43) workstation. + +- **Status: Failed!, Errors: 7** at only **0:00:51** elapsed — still in Pass 0, + Test #3 (Moving inversions, 1s & 0s). Errored almost immediately. +- Both captured failures are **single-bit flips** (signature of a failing + module, not a controller-wide / addressing fault): + - `0x000bc0115dd0` (47 GB): expected `f…`, found `bf…` → **bit 62 flipped 1→0** + - `0x001005f391e0` (64 GB): expected `0…`, found `…01000000` → **bit 24 flipped 0→1** +- Temps 58/65 °C (fine). DDR4-2666 is JEDEC-standard, **not** an aggressive + XMP/EXPO overclock — so this is not merely an unstable memory profile. + +**Conclusion** + +Confirms the 2026-06-24 hypothesis: scattered multi-file Btrfs csum corruption +with zero drive I/O errors and clean NVMe SMART = **bad RAM** writing garbage to +otherwise-good flash. Root cause: **confirmed**. + +**RAM layout (from `dmidecode -t memory`, 2026-06-26)** + +96 GB total = **two complete G.Skill Ripjaws V kits**, both DDR4-3200 CL16 +running at 2666 (XMP/DOCP off): + +| Slot | Size | Part number | Kit | +|------|------|-------------|-----| +| Channel A · DIMM 0 | 32 GB | F4-3200C16-32GVK | 2×32 GB pair → 64 GB | +| Channel B · DIMM 0 | 32 GB | F4-3200C16-32GVK | (matched cross-channel) | +| Channel A · DIMM 1 | 16 GB | F4-3200C16-16GVK | 2×16 GB pair → 32 GB | +| Channel B · DIMM 1 | 16 GB | F4-3200C16-16GVK | (matched cross-channel) | + +Two valid matched dual-channel pairs → whichever stick fails, drop to the other +complete pair and keep dual-channel. **Fallbacks need zero purchase:** if a +16 GB stick is bad → run the 2×32 = 64 GB; if a 32 GB stick is bad → run the +2×16 = 32 GB. + +**Booting memtest (GRUB hotkeys unreliable on this box)** + +memtest86+ 8.10 is installed (`/boot/memtest86+x64.efi`); GRUB uses +`GRUB_DEFAULT=saved`, so one-shot boot needs no keypresses: + +```bash +sudo grub2-mkconfig -o /boot/grub2/grub.cfg +sudo grep -o "menuentry '[^']*emtest[^']*'" /boot/grub2/grub.cfg # get exact title +sudo grub2-reboot "" # one-shot; reverts after next boot +sudo systemctl poweroff # pull sticks, then power on → boots memtest +``` + +Exit memtest with **Esc** (reboots back to Fedora). + +**Follow-up / next steps** + +- **Treat the machine as untrustworthy until the RAM is replaced** — any data + written (including backups) since the corruption began may be silently bad. + Be cautious about pruning known-good restic snapshots. +- **Isolate the bad DIMM** — test **by kit first** (the 2×32, then the 2×16); + a passing pair is immediately a runnable config. Only split a *failing* pair + to find the single bad stick. Errors at 47 GB / 64 GB are interleaved logical + addresses and do NOT map to one physical stick. +- **Don't discard the bad stick** — G.Skill Ripjaws V carry a limited lifetime + warranty; RMA for a free replacement (restores the matched pair at no cost). + Keep its good partner as a spare or for the RMA-restored pair. Avoid buying a + single stick to pair (DDR4 matching is finicky) unless RMA fails. +- After fixing, **re-run memtest to a clean multi-pass**, then **re-scrub + `/home`** and watch for *new* csum errors. None appearing = corruption was + historical and is now resolved. Defer `restic check --read-data` until the RAM + is known-good (a bad-RAM box throws false integrity errors). + ## 2026-06-24 — Chrome repeatedly crashing (SIGBUS), corrupt Favicons block on single-copy Btrfs **Symptom** diff --git a/docs/orchestration-audit-prd.md b/docs/orchestration-audit-prd.md new file mode 100644 index 0000000..2238872 --- /dev/null +++ b/docs/orchestration-audit-prd.md @@ -0,0 +1,97 @@ +# PRD — Deeper orchestration audit + +**Status:** ready to run (new session) · **Created:** 2026-06-30 +**Predecessor:** `docs/cc-tool-audit.md` (§ Addendum — orchestration audit) +**Note:** No issue tracker/remote on this repo, so this PRD lives as a doc. The +`/to-prd` "publish + `ready-for-agent` label" step was adapted to a file. + +--- + +## Problem Statement + +Sessions feel slow and over-spend on model cost. A prior audit +(`docs/cc-tool-audit.md`) traced this less to loaded tools/skills/hooks (all small) +and more to the **orchestrator-subagent delegation pattern**. A first-pass audit of +1,962 subagent transcripts found that **27% of subagents run on Opus**, the majority +of them doing mechanical file-edit/shell work that Haiku/Sonnet would handle faster +and cheaper. The CLAUDE.md "Session Orchestration" routing table is **advisory** — +when an `Agent` spawn omits `model`, the subagent inherits the parent (Opus). The +rule also mandates delegation with **no minimum complexity threshold**, so trivial +one-line edits pay a full subagent prefill + sequential wait. + +An interim fix is already applied (`CLAUDE_CODE_SUBAGENT_MODEL=haiku` in global +settings). What's missing is the **quantified evidence** needed to safely revise the +orchestration *instructions* across many CLAUDE.md files — not just the default model. + +## Solution + +A deeper, repeatable audit of the existing session transcripts that produces the +numbers to ground a **pilot** revision of the Session Orchestration rules on one +project, before any broad rollout. The audit answers: how often is the wrong model +chosen, is delegation itself wasteful for small tasks, and is work being decomposed +well before hand-off. Output is a short findings report + a proposed pilot rule diff. + +## User Stories + +1. As the operator, I want the subagent model distribution broken down per project, so that I can see which repos leak Opus most. +2. As the operator, I want the "file-edit/shell" subagent bucket split into *trivial* (1–2 calls, single file) vs *substantial* (multi-file, many calls), so that I can tell genuine Sonnet/Opus work from Haiku-appropriate work instead of relying on a coarse bucket. +3. As the operator, I want each subagent's spawn traced to whether its model was **explicitly passed** by the orchestrator or **inherited** from the parent, so that I can measure the routing-leak rate directly. +4. As the operator, I want a count of subagents that did near-zero work (0–1 tool calls, or returned trivially), so that I can quantify wasted delegations caused by the "no threshold" rule. +5. As the operator, I want to see how many orchestrator turns spawned a single subagent for a single small op (delegation that a direct op would beat on latency), so that I can justify a complexity threshold. +6. As the operator, I want parallelism measured — how often independent subagents were batched concurrently vs. spawned sequentially — so that I can see if decomposition exploits the parallel-batch guidance. +7. As the operator, I want examples of good and bad decomposition (a clean parallel batch vs. a needless single-op delegation), so that the pilot rule change is illustrated, not just asserted. +8. As the operator, I want the audit limited to delegation-heavy projects (cc-os, llf-schema, proxmox, philly-seo, ovh-prod), so that it runs on the relevant sample, not every repo. +9. As the operator, I want the interim `CLAUDE_CODE_SUBAGENT_MODEL=haiku` effect measurable in future sessions (a before/after marker), so that I can confirm the env var actually shifted the distribution. +10. As the operator, I want a proposed pilot diff to one project's "Session Orchestration" section (threshold + enforced routing + allow orienting reads), so that I can try it on a single repo and measure wall-clock/cost before rolling out. +11. As the operator, I want the audit scripted/repeatable (saved under the repo), so that I can re-run it after the pilot to compare. + +## Implementation Decisions + +- **Data source:** existing JSONL transcripts in `~/.claude/projects/*/` (main) and + `~/.claude/projects/*//subagents/agent-*.jsonl` (subagents). Model is in + each subagent message's `model` field; tool calls are `type:"tool_use"` blocks + (walk recursively). Project dir names start with `-` — use python `glob`, never + shell `find`/`ls`. +- **Trivial vs substantial classifier:** trivial = ≤2 tool calls and ≤1 distinct file + touched; substantial = otherwise. Tune threshold against spot-checked examples. +- **Inherited vs explicit model trace:** match parent `Agent` tool_use input (does it + carry a `model`/`subagent_type`?) to the child transcript via agent id, then compare + requested vs actual model. Where the parent omitted model and child==parent model, + classify as "inherited (leak)." +- **Parallelism metric:** within a main session, detect `Agent` tool_use blocks issued + in the same assistant turn (concurrent) vs separate turns (sequential). +- **Scope:** the 5 delegation-heavy projects above; report per-project and pooled. +- **Output:** append a "Deep audit" section to `docs/cc-tool-audit.md` (or a sibling + doc) with tables + 3–5 examples, plus a proposed pilot rule diff for ONE project. +- **No production change** beyond the already-applied env var until the pilot is + reviewed. + +## Testing Decisions + +- This is an analysis task, not a shipped feature — "tests" = sanity checks on the + parser: total subagent count matches the prior audit (~1,962 with a model), model + totals reconcile, and the trivial/substantial split sums to the file-edit bucket. +- Spot-verify the classifier on 5 hand-read transcripts (one per project) before + trusting the aggregate. +- Re-run the same script post-pilot to confirm the distribution shifted (regression + baseline = today's numbers). + +## Out of Scope + +- Editing every CLAUDE.md (this is assessment → single-project pilot only). +- Changing tool/skill/hook config (prior audit showed negligible payoff). +- Building any new tooling, MCP, or harness changes. +- Touching the live orchestration behavior beyond the env var already set. + +## Further Notes + +- Interim change already live: `env.CLAUDE_CODE_SUBAGENT_MODEL=haiku` in + `~/.claude/settings.json` (applies to sessions launched after 2026-06-30). +- Prior-audit baseline to beat: Opus 26.6% / Sonnet 36.7% / Haiku 36.5% of subagents; + Opus subagents 53% file-edit/shell. +- Reusable scratch scripts from the first pass: + `/tmp/claude-1000/.../scratchpad/{full,cat,orch}.py` (ephemeral — recreate or + promote into the repo when running the deep audit). +- Open instruction-design questions for the pilot: where to set the complexity + threshold; whether to allow N orienting reads before delegating; how to phrase + "always pass model on spawn" given the env var now provides a cheap default. diff --git a/docs/superpowers/plans/2026-05-07-n8n-backup-monitor.md b/docs/superpowers/plans/2026-05-07-n8n-backup-monitor.md index 835eb0c..84bc2ae 100644 --- a/docs/superpowers/plans/2026-05-07-n8n-backup-monitor.md +++ b/docs/superpowers/plans/2026-05-07-n8n-backup-monitor.md @@ -36,7 +36,7 @@ - [ ] Create the `scripts/` directory: ```bash -mkdir -p /home/jared/systems-admin/scripts +mkdir -p /home/jared/servers/desktop/scripts ``` - [ ] Create `scripts/notify_backup.rb` with the full header comment and empty class stubs. The `if __FILE__ == $PROGRAM_NAME` guard keeps the CLI from running during tests: @@ -114,13 +114,13 @@ require_relative 'notify_backup' - [ ] Run the test file to confirm the skeleton loads without errors: ```bash -ruby /home/jared/systems-admin/scripts/notify_backup_test.rb +ruby /home/jared/servers/desktop/scripts/notify_backup_test.rb ``` Expected output: `0 runs, 0 assertions, 0 failures, 0 errors` - [ ] Commit: ```bash -cd /home/jared/systems-admin +cd /home/jared/servers/desktop git add scripts/ git commit -m "feat(backup-monitor): add notify_backup script skeleton and test infrastructure" ``` @@ -165,7 +165,7 @@ end - [ ] Run to confirm 6 failures: ```bash -ruby /home/jared/systems-admin/scripts/notify_backup_test.rb +ruby /home/jared/servers/desktop/scripts/notify_backup_test.rb ``` Expected: `6 failures` (NoMethodError on ByteFormatter.format) @@ -189,13 +189,13 @@ end - [ ] Run tests to confirm they pass: ```bash -ruby /home/jared/systems-admin/scripts/notify_backup_test.rb +ruby /home/jared/servers/desktop/scripts/notify_backup_test.rb ``` Expected: `6 runs, 6 assertions, 0 failures, 0 errors` - [ ] Commit: ```bash -cd /home/jared/systems-admin +cd /home/jared/servers/desktop git add scripts/ git commit -m "feat(backup-monitor): implement ByteFormatter with tests" ``` @@ -265,7 +265,7 @@ end - [ ] Run to confirm failures: ```bash -ruby /home/jared/systems-admin/scripts/notify_backup_test.rb +ruby /home/jared/servers/desktop/scripts/notify_backup_test.rb ``` Expected: `PayloadBuilderTest` failures, `ByteFormatterTest` still passing @@ -305,13 +305,13 @@ end - [ ] Run all tests: ```bash -ruby /home/jared/systems-admin/scripts/notify_backup_test.rb +ruby /home/jared/servers/desktop/scripts/notify_backup_test.rb ``` Expected: `12 runs, 12+ assertions, 0 failures, 0 errors` - [ ] Commit: ```bash -cd /home/jared/systems-admin +cd /home/jared/servers/desktop git add scripts/ git commit -m "feat(backup-monitor): implement PayloadBuilder with tests" ``` @@ -388,7 +388,7 @@ end - [ ] Run to confirm failures: ```bash -ruby /home/jared/systems-admin/scripts/notify_backup_test.rb +ruby /home/jared/servers/desktop/scripts/notify_backup_test.rb ``` Expected: `WebhookClientTest` failures @@ -427,13 +427,13 @@ end - [ ] Run all tests: ```bash -ruby /home/jared/systems-admin/scripts/notify_backup_test.rb +ruby /home/jared/servers/desktop/scripts/notify_backup_test.rb ``` Expected: all tests pass, 0 failures - [ ] Commit: ```bash -cd /home/jared/systems-admin +cd /home/jared/servers/desktop git add scripts/ git commit -m "feat(backup-monitor): implement WebhookClient with injected HTTP adapter and tests" ``` @@ -483,7 +483,7 @@ end - [ ] Run to confirm failures: ```bash -ruby /home/jared/systems-admin/scripts/notify_backup_test.rb +ruby /home/jared/servers/desktop/scripts/notify_backup_test.rb ``` Expected: `BackupNotifierTest` failures @@ -545,13 +545,13 @@ end - [ ] Run all tests: ```bash -ruby /home/jared/systems-admin/scripts/notify_backup_test.rb +ruby /home/jared/servers/desktop/scripts/notify_backup_test.rb ``` Expected: all tests pass (roughly 16+ assertions, 0 failures) - [ ] Commit: ```bash -cd /home/jared/systems-admin +cd /home/jared/servers/desktop git add scripts/ git commit -m "feat(backup-monitor): implement BackupNotifier and CLI entry point" ``` @@ -565,7 +565,7 @@ git commit -m "feat(backup-monitor): implement BackupNotifier and CLI entry poin - [ ] Copy and make executable: ```bash -cp /home/jared/systems-admin/scripts/notify_backup.rb ~/.local/bin/notify_backup.rb +cp /home/jared/servers/desktop/scripts/notify_backup.rb ~/.local/bin/notify_backup.rb chmod +x ~/.local/bin/notify_backup.rb ``` @@ -771,15 +771,15 @@ if __name__ == "__main__": - [ ] Make it executable and test it on a known-bad payload: ```bash -chmod +x /home/jared/systems-admin/scripts/validate-n8n-workflow.py +chmod +x /home/jared/servers/desktop/scripts/validate-n8n-workflow.py echo '{"name":"test","nodes":[],"connections":{"missing-node":{"main":[[]]}}}' \ - | python3 /home/jared/systems-admin/scripts/validate-n8n-workflow.py /dev/stdin + | python3 /home/jared/servers/desktop/scripts/validate-n8n-workflow.py /dev/stdin ``` Expected: `ERROR: connections: source 'missing-node' does not match any node name.` - [ ] Commit: ```bash -cd /home/jared/systems-admin +cd /home/jared/servers/desktop git add scripts/validate-n8n-workflow.py git commit -m "feat(backup-monitor): add n8n workflow JSON validator" ``` @@ -1160,7 +1160,7 @@ puts " #{SCRIPTS_DIR}/n8n-workflow-reporter.json" - [ ] Run the build script: ```bash -cd /home/jared/systems-admin +cd /home/jared/servers/desktop ruby scripts/build-n8n-workflows.rb ``` Expected: two files created with no error output @@ -1276,7 +1276,7 @@ Expected: no error output. Verify a row appears in the `backup_runs` table in n8 - [ ] Commit the final build script with real IDs: ```bash -cd /home/jared/systems-admin +cd /home/jared/servers/desktop git add scripts/build-n8n-workflows.rb scripts/n8n-workflow-receiver.json scripts/n8n-workflow-reporter.json git commit -m "feat(backup-monitor): fill credential/datatable IDs and reimport workflows" ``` @@ -1365,12 +1365,12 @@ Expected: no output (no syntax errors) - [ ] Copy the updated backup.sh back to the source in this repo so it stays tracked: ```bash -cp ~/.local/bin/backup.sh /home/jared/systems-admin/scripts/backup.sh +cp ~/.local/bin/backup.sh /home/jared/servers/desktop/scripts/backup.sh ``` - [ ] Commit: ```bash -cd /home/jared/systems-admin +cd /home/jared/servers/desktop git add scripts/backup.sh git commit -m "feat(backup-monitor): integrate notify_backup.rb into desktop backup.sh" ``` @@ -1444,7 +1444,7 @@ Expected: no output - [ ] Exit the SSH session. Back on desktop, commit the documentation of this change: ```bash -cd /home/jared/systems-admin +cd /home/jared/servers/desktop git commit --allow-empty -m "feat(backup-monitor): deploy notify_backup.rb to OVH VPS and integrate into backup.sh" ```