Compare commits
10 Commits
3b1bfeada1
...
a722a2db65
| Author | SHA1 | Date |
|---|---|---|
jared
|
a722a2db65 | |
|
jared
|
e4048f5480 | |
|
jared
|
876006f872 | |
|
jared
|
7ee7df68ee | |
|
jared
|
309232cb26 | |
|
jared
|
4ab40b9912 | |
|
jared
|
74f1a765e9 | |
|
jared
|
8ff8088f84 | |
|
jared
|
0ac58a3cb9 | |
|
jared
|
9bb9abb438 |
|
|
@ -0,0 +1 @@
|
|||
.dochygiene/
|
||||
13
CLAUDE.md
13
CLAUDE.md
|
|
@ -15,6 +15,8 @@ Documentation and planning for a GNOME → Hyprland migration on Fedora 43. No b
|
|||
| Symlink target | `~/.config/hypr/` → symlinked via `stow hypr` |
|
||||
| Hyprland log | `~/.cache/hyprland/hyprland.log` |
|
||||
| Dotfiles remote | `git@forgejo.swansoncloud.com:jared/dotfiles.git` |
|
||||
| Synology DS218plus access + backup paths | `docs/synology.md` |
|
||||
| VPS inventory + access notes | `docs/vps-inventory.md` |
|
||||
|
||||
## Hyprland configuration
|
||||
|
||||
|
|
@ -33,14 +35,21 @@ Do NOT use `^(...)$` wrapping — Hyprland 0.51 rejects it as "Invalid rulev2".
|
|||
|
||||
- `docs/window-manager/` — migration plan, phased progress log, keybindings cheatsheet
|
||||
- `docs/hyprland/` — version-aware Hyprland config reference; update these when Hyprland is upgraded
|
||||
- `docs/fedora-upgrade.md` — distro release-upgrade procedure; F43→44 deferred (solopasha COPR aquamarine not rebuilt vs F44 libdisplay-info)
|
||||
- `docs/synology.md` — DS218plus SSH alias, `/volume1/backup` map, restic repo commands, recovery notes
|
||||
- `docs/vps-inventory.md` — OVH VPS (`ovh-vps`) access notes; old Hyperthrive DO VPS deleted 2026-06-24 (historical notes retained)
|
||||
- `docs/incidents.md` — dated incident log (diagnosed-and-fixed issues): symptom, root cause, ruled-out, fix, follow-up
|
||||
|
||||
## Project status
|
||||
|
||||
Phases 0–3 complete (install, config, dotfiles integration). Phase 4 (first successful Hyprland login and smoke tests) is in progress. Phase 5 (tuning) is deferred.
|
||||
|
||||
Known workaround in place: `hyprpolkitagent` replaced with `lxpolkit` due to Qt 6 ABI issue on Fedora 43.
|
||||
Known workarounds in place:
|
||||
- `hyprpolkitagent` replaced with `xfce-polkit` due to Qt 6.9/6.10 private ABI mismatch in solopasha COPR (affects hyprland-qt-support, hyprland-qtutils). Revisit when COPR rebuilds against Qt 6.10.
|
||||
- GDM Wayland→Hyprland cursor handoff leaves stale cursor ghost at login position. Fixed with `exec-once = sleep 1 && hyprctl dispatch movecursor 0 0` in hyprland.conf. If unreliable, fallback is `WaylandEnable=false` in /etc/gdm/custom.conf (makes GDM use X11 greeter for clean KMS handoff).
|
||||
- `cursor { no_hardware_cursors = true; use_cpu_buffer = 1 }` set in dms/cursor.conf to prevent NVIDIA DRM (akmod-nvidia loaded for RTX 3060) from interfering with AMD Vega cursor plane rendering.
|
||||
|
||||
## Hardware
|
||||
|
||||
- AMD Vega iGPU (primary display, Wayland)
|
||||
- NVIDIA RTX 3060 dGPU (unused — `akmod-nvidia` intentionally not installed)
|
||||
- NVIDIA RTX 3060 dGPU (no display connected — `akmod-nvidia` is installed and loads `nvidia_drm` at boot)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,268 @@
|
|||
# Claude Code tool & context audit
|
||||
|
||||
**Date:** 2026-06-30
|
||||
**Question:** Are too many built-in CC tools loaded into context, slowing sessions and
|
||||
hurting focus? Which can be excluded globally vs. situationally?
|
||||
|
||||
**Method:** Mechanically parsed every tool-call (`tool_use`) across **224 main
|
||||
sessions + 1,946 subagent transcripts = 2,170 transcripts** in `~/.claude/projects/`,
|
||||
cross-referenced against the full tools reference
|
||||
(https://code.claude.com/docs/en/tools-reference.md), and measured the actual
|
||||
bytes loaded into one representative session's startup context.
|
||||
|
||||
---
|
||||
|
||||
## TL;DR — the hypothesis is mostly outdated
|
||||
|
||||
Excluding **built-in tools** used to meaningfully shrink the prompt. It no longer
|
||||
does, because Claude Code now **defers** rarely-used tool schemas (tool search):
|
||||
they load on demand via `ToolSearch`, not at startup.
|
||||
|
||||
- Total *static* startup context in a typical session here ≈ **6,300 tokens**.
|
||||
- Built-in tool schemas are a small slice of that; **MCP cost = 0** (no MCP servers
|
||||
configured). **Skills (50) ≈ 3,700 tok** are the single largest contributor.
|
||||
- Of **24 built-in tools never used in 2,170 transcripts**, only **one** (`Workflow`)
|
||||
is both always-loaded *and* unused. The other 23 are already deferred, platform-
|
||||
impossible, or trivially small.
|
||||
|
||||
So: a confident global *tool* exclusion list exists, but its payoff is small. The
|
||||
real levers for "feels slow / unfocused" are elsewhere (see §5).
|
||||
|
||||
> ⚠️ Honest caveat: trimming context lowers prefill latency and cost and may sharpen
|
||||
> focus, but it does **not** change Opus's per-token generation speed. Opus is simply
|
||||
> slower to generate than Sonnet/Haiku; no amount of tool pruning fixes that.
|
||||
|
||||
---
|
||||
|
||||
## 1. What tools actually get used (2,170 transcripts)
|
||||
|
||||
| Tool | Calls | Transcripts using it |
|
||||
|---|---:|---:|
|
||||
| Bash | 9,168 | 1,248 |
|
||||
| Read | 5,955 | 1,185 |
|
||||
| Edit | 1,983 | 501 |
|
||||
| Agent | 1,948 | 541 |
|
||||
| Write | 981 | 382 |
|
||||
| WebFetch | 288 | 72 |
|
||||
| ToolSearch | 169 | 139 |
|
||||
| WebSearch | 124 | 36 |
|
||||
| AskUserQuestion | 108 | 70 |
|
||||
| Skill | 104 | 84 |
|
||||
| TaskUpdate / TaskCreate | 54 / 31 | 10 / 7 |
|
||||
| SendMessage | 40 | 27 |
|
||||
| Monitor | 21 | 12 |
|
||||
| TaskStop / ScheduleWakeup | 8 / 4 | 5 / 3 |
|
||||
| RemoteTrigger / ReportFindings | 1 / 1 | 1 / 1 |
|
||||
|
||||
**Never used anywhere (0 calls):** `Artifact`, `CronCreate/Delete/List`,
|
||||
`EnterPlanMode/ExitPlanMode`, `EnterWorktree/ExitWorktree`, `Glob`, `Grep`,
|
||||
`ListMcpResourcesTool`, `LSP`, `NotebookEdit`, `PowerShell`, `PushNotification`,
|
||||
`ReadMcpResourceTool`, `SendUserFile`, `ShareOnboardingGuide`,
|
||||
`TaskGet/TaskList/TaskOutput`, `TodoWrite`, `WaitForMcpServers`, `Workflow`.
|
||||
|
||||
> Note `Grep`/`Glob` show **0** — search runs through `Bash` (ripgrep/find directly,
|
||||
> reinforced by the RTK proxy). They are still worth keeping (see §4).
|
||||
|
||||
---
|
||||
|
||||
## 2. The decisive cross-reference: never-used ∩ actually-loaded
|
||||
|
||||
The "never used" list conflates three very different things. Only the first bucket
|
||||
gives back real context when excluded.
|
||||
|
||||
| Bucket | Tools | Exclusion payoff |
|
||||
|---|---|---|
|
||||
| **Always-loaded & unused** | `Workflow` | **Real** — large schema |
|
||||
| **Already deferred** (loaded on demand, not at startup) | `CronCreate/Delete/List`, `EnterPlanMode/ExitPlanMode`, `EnterWorktree/ExitWorktree`, `LSP`, `NotebookEdit`, `PushNotification`, `TaskGet/List/Output` | **~0** — not in the prompt anyway |
|
||||
| **Platform/plan-impossible or not loaded here** | `PowerShell` (Linux), `Artifact`/`ShareOnboardingGuide` (need Team/Ent.), `SendUserFile`/`PushNotification` (need Remote Control), `TodoWrite` (disabled by default), `WaitForMcpServers` (only when tool-search is OFF), `Glob`/`Grep`, MCP-resource tools | **Cosmetic** — either absent or trivially small |
|
||||
|
||||
**This is why manual exclusion mattered more in the past:** tool-search deferral now
|
||||
does the trimming automatically.
|
||||
|
||||
---
|
||||
|
||||
## 3. Where the startup context actually goes (~6,300 tok measured)
|
||||
|
||||
| Contributor | Est. tokens | # items |
|
||||
|---|---:|---:|
|
||||
| **Skills** (name + description) | ~3,700 | 50 |
|
||||
| **Subagent types** | ~917 | 18 |
|
||||
| CLAUDE.md (project) | ~842 | 1 |
|
||||
| MEMORY.md auto-index | ~427 | 1 |
|
||||
| RTK.md | ~241 | 1 |
|
||||
| CLAUDE.md (global) | ~148 | 1 |
|
||||
| **MCP tool schemas** | **0** | 0 servers |
|
||||
| **Total** | **~6,300** | |
|
||||
|
||||
Skills + agents = ~74% of static context. Heaviest single items: the built-in
|
||||
`claude-api` skill (~269 tok), personal `the-humanizer` (~239), `impeccable` (~226),
|
||||
`memsearch:memory-recall` (~185).
|
||||
|
||||
---
|
||||
|
||||
## 4. Recommendation A — global tool exclusions (small but safe)
|
||||
|
||||
Add to `~/.claude/settings.json` `permissions.deny`. The mechanism: a denied tool is
|
||||
removed from the advertised set (its schema isn't loaded), per the tools reference
|
||||
("to disable a tool entirely, add its name to the deny array"). Verify by checking the
|
||||
session token count before/after if you want proof.
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"permissions": {
|
||||
"deny": [
|
||||
"Workflow", // 0/2170 uses; largest always-loaded built-in schema.
|
||||
// KEEP it ONLY if you use /workflows or "ultracode".
|
||||
"PowerShell", // Linux host — never available. Cosmetic.
|
||||
"Artifact", // needs Team/Enterprise plan. Cosmetic.
|
||||
"ShareOnboardingGuide" // paid team feature. Cosmetic.
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- **`Workflow` is the only entry with real payoff.** It is the multi-agent
|
||||
orchestration tool (the big JS-script one) — *not* `Agent`. You use `Agent`
|
||||
constantly (1,948 calls) and `Workflow` never. Keep `Agent`; drop `Workflow`
|
||||
unless/until you adopt `/workflows`.
|
||||
- **Do NOT deny** `Glob`/`Grep` (tiny; the sanctioned search path — denying pushes
|
||||
everything to `Bash grep`/`find`, which can add permission prompts), anything
|
||||
**deferred** (no savings), or plan-mode/worktree/Task tools (deferred *and*
|
||||
UI-driven — denying `ExitPlanMode` would break Shift-Tab plan mode).
|
||||
|
||||
Net context saved: roughly the `Workflow` schema. Honest, but modest.
|
||||
|
||||
---
|
||||
|
||||
## 5. Recommendation B — the real lever: situational plugins/skills
|
||||
|
||||
Skills/agents (~4,600 tok) dwarf the built-in-tool headroom, and they load *every*
|
||||
session regardless of relevance. These are controlled per-machine via
|
||||
`enabledPlugins` in `~/.claude/settings.json` (not easily per-project), so the
|
||||
practical move is toggling plugins by what you're doing.
|
||||
|
||||
Currently enabled: `superpowers` (already disabled ✓), `ruby-lsp`, `perspectives`,
|
||||
`git-context`, `invoice-ninja`, `memory`, `memsearch`, `doc-hygiene`, `codex`.
|
||||
|
||||
| Plugin / skill group | Relevant to | Drop when not… |
|
||||
|---|---|---|
|
||||
| `codex:*` (5 skills + agent) | coding handoffs to Codex | …coding |
|
||||
| `ruby-lsp` | Ruby projects | …in a Ruby repo |
|
||||
| `perspectives:*` (9 agents) | design/plan critique | …planning/designing |
|
||||
| `invoice-ninja` | freelance billing | …doing client billing |
|
||||
| `deep-research`, web tools | research | …researching |
|
||||
| personal writing/UI: `the-humanizer`, `impeccable`, `edit-article`, `prototype` | content/frontend | …writing or building UI |
|
||||
| `doc-hygiene:*` | docs repos | …maintaining docs |
|
||||
| `memory:*` + `memsearch:*` | cross-session recall | rarely — but see below |
|
||||
|
||||
**Suggested situational profiles** (toggle `enabledPlugins`):
|
||||
|
||||
- **Coding repo:** keep `codex`, `ruby-lsp` (if Ruby), `git-context`. Drop
|
||||
`invoice-ninja`, writing/UI skills, `deep-research`, `perspectives`.
|
||||
- **Freelance / PM / client:** keep `invoice-ninja`, `git-context`. Drop `codex`,
|
||||
`ruby-lsp`, `perspectives`, coding skills (`tdd`/`diagnose`/`run`/`verify`).
|
||||
- **Research / writing:** keep `deep-research`, writing skills. Drop `codex`,
|
||||
`ruby-lsp`, `invoice-ninja`.
|
||||
- **Infra / sysadmin (this repo):** Bash/Read/Edit-heavy. Drop writing/UI,
|
||||
`invoice-ninja`, `codex`, coding frameworks; keep `doc-hygiene`, `git-context`.
|
||||
|
||||
> Reality check: even the full ~6.3k static context is small next to a working
|
||||
> session that fills 50k–150k+ tokens. Trimming skills is good hygiene and helps
|
||||
> focus, but it is not where most of a slow session's time goes.
|
||||
|
||||
---
|
||||
|
||||
## 6. What is probably actually slowing sessions
|
||||
|
||||
Ranked by likely impact, from the data:
|
||||
|
||||
1. **Opus generation speed.** Inherent; switch to Sonnet/Haiku for mechanical work,
|
||||
or use `/fast` (faster Opus output) for interactive coding.
|
||||
2. **Heavy `Agent` delegation.** 67% of sessions spawn subagents; 541 transcripts
|
||||
used `Agent`. Each subagent is a fresh context with its own prefill + sequential
|
||||
wall-clock wait. Great for big fan-out; pure latency for small lookups you could
|
||||
do inline.
|
||||
3. **Per-prompt / per-tool hooks.** The `memory` plugin registers all 5 hook types,
|
||||
incl. **PostToolUse (runs after every tool call)** and SessionStart/UserPromptSubmit
|
||||
(the large memory dump at session start). `memsearch` adds more. This is a
|
||||
per-operation tax that tool-count pruning won't touch — worth timing/disabling to
|
||||
test. **Most promising thing to measure next.**
|
||||
4. **Long sessions** → context summarization passes. Use `/clear` between unrelated
|
||||
tasks rather than carrying one giant session.
|
||||
5. Static tool/skill context (this audit) — real but the **smallest** of these.
|
||||
|
||||
---
|
||||
|
||||
## Appendix — four representative sessions (loaded vs. used)
|
||||
|
||||
1. **Coding** (`llf-schema/19320d96`) — implement an OpenSpec change. 191 calls, only
|
||||
**Edit/Read/Write/Bash**. No web, no `Agent`, no skills. Pure local code loop.
|
||||
2. **Infra, wide** (`proxmox-hermes-01/c24faa0a`) — plan a Proxmox agent setup.
|
||||
Used nearly the whole belt: Bash/Agent/WebFetch/WebSearch/Read/Edit/Write/
|
||||
AskUserQuestion/Skill(`git-context:repo-init`). The legit case for keeping web
|
||||
tools loaded.
|
||||
3. **Infra, ops** (`systems-admin/ffed234f`) — "how did the backup do this week?"
|
||||
50 calls, only **Bash/Read/Edit/Write**. Textbook shell-only profile.
|
||||
4. **PM / orchestration** (`ovh-prod-01/991913d9`) — decide next server-consolidation
|
||||
phase. Lead agent used only **Agent + AskUserQuestion**; 360+ tool calls happened
|
||||
in 40 subagents. Never touched a file or shell itself.
|
||||
|
||||
Three of four would have been unaffected by stripping every web tool, every skill,
|
||||
and (for #1/#3) all delegation tooling.
|
||||
|
||||
---
|
||||
|
||||
## Addendum (2026-06-30) — follow-up: skills, hooks, and orchestration
|
||||
|
||||
### Skill loading — there IS a native lever (no "dispatcher" needed)
|
||||
|
||||
Per the docs: a skill's **body** loads only on invocation ("costs almost nothing
|
||||
until you need it"); only **name + description** load at startup, and that listing is
|
||||
**budget-capped** (`skillListingBudgetFraction`, ~1% of context; `skillListingMaxDescChars`
|
||||
default 1,536/skill). There is **no** skill-level `ToolSearch` equivalent.
|
||||
|
||||
To reduce always-loaded skill weight, use **`skillOverrides`** in settings rather than
|
||||
a community dispatcher skill:
|
||||
- `"name-only"` — keep the skill available but drop its description from context.
|
||||
- `"off"` — hide it entirely.
|
||||
Plus `enabledPlugins`, `disableBundledSkills: true`. The dispatcher/index pattern is
|
||||
**not** officially recommended — the budget + `skillOverrides` already do it.
|
||||
|
||||
### Hooks are synchronous but cheap (measured on this machine)
|
||||
|
||||
Hooks block the turn until exit (up to `timeout`); `"async": true` makes one
|
||||
fire-and-forget. The `memory` plugin's hooks are synchronous but fast:
|
||||
`post_tool_use_write` ~38 ms (Write|Edit only, early-exits off-vault),
|
||||
`session_context` ~33 ms/prompt (injects ~0 tokens normally),
|
||||
`rtk hook` ~24 ms/Bash. Negligible vs. multi-second model turns. **Not a latency
|
||||
source.** The big memory dump is SessionStart-only (once).
|
||||
|
||||
### Orchestration audit — subagent model mismatch (1,962 transcripts)
|
||||
|
||||
Model is recorded per subagent transcript. Distribution: **Sonnet 36.7% · Haiku 36.5%
|
||||
· Opus 26.6%**. By work profile:
|
||||
|
||||
| Subagent model | n | % that did only file-edit/shell work |
|
||||
|---|---:|---:|
|
||||
| Opus | 521 | 53% (+38% spawned their own subagents) |
|
||||
| Sonnet | 720 | 64% |
|
||||
| Haiku | 716 | 83% (well-matched) |
|
||||
|
||||
**Finding:** 27% of subagents run on **Opus**, most doing mechanical edits. Root cause:
|
||||
when an `Agent` spawn omits `model`, the subagent **inherits the parent model (Opus)** —
|
||||
so the CLAUDE.md routing table is advisory and leaks.
|
||||
|
||||
**Subagent model resolution order** (docs): `CLAUDE_CODE_SUBAGENT_MODEL` env →
|
||||
per-invocation `model` param → subagent frontmatter `model:` → inherited main model.
|
||||
|
||||
**Enforcement fix:** set **`CLAUDE_CODE_SUBAGENT_MODEL=haiku`** (or `sonnet`) so
|
||||
subagents default cheap instead of inheriting Opus; pin per-type `model:` frontmatter
|
||||
for specialized agents; escalate by explicit criteria only.
|
||||
|
||||
**Instruction tuning (pilot on one project first):**
|
||||
1. Drop "no minimum complexity threshold" — direct-operate trivial single-file ops;
|
||||
delegate only when parallelizable, large, or context-heavy. Mandatory delegation of
|
||||
one-line edits is a net latency *loss*.
|
||||
2. Make routing enforced, not advisory (env var above + "always pass model on spawn").
|
||||
3. Relax "never read before delegating" — allow a cheap orienting read when paths/
|
||||
structure are uncertain, to avoid blind-spec retry loops.
|
||||
|
|
@ -0,0 +1,70 @@
|
|||
# Fedora release upgrades
|
||||
|
||||
Notes and decisions for distro version upgrades on this machine. The Hyprland
|
||||
stack comes from the **solopasha COPR**, which is the recurring complication —
|
||||
it must be rebuilt against the new Fedora release before the system upgrade can
|
||||
resolve.
|
||||
|
||||
## Standard upgrade procedure
|
||||
|
||||
```bash
|
||||
sudo dnf upgrade --refresh
|
||||
sudo dnf install dnf-plugin-system-upgrade # usually already present
|
||||
sudo dnf system-upgrade download --releasever=NN
|
||||
sudo dnf system-upgrade reboot
|
||||
```
|
||||
|
||||
## Status: Fedora 43 → 44 — DEFERRED (decided 2026-06-11)
|
||||
|
||||
**Decision: stay on Fedora 43 and wait. Do not upgrade to 44 yet.**
|
||||
|
||||
### Why deferred
|
||||
|
||||
`sudo dnf system-upgrade download --releasever=44` fails to resolve because the
|
||||
solopasha COPR has **not rebuilt `aquamarine` against Fedora 44**:
|
||||
|
||||
- Fedora 44 ships `libdisplay-info-0.3.0`, which provides only `libdisplay-info.so.3`.
|
||||
- The COPR's F44 build `aquamarine-0.9.5-2.fc44` still links against the old
|
||||
`libdisplay-info.so.2` (verified — its `requires` list still names `.so.2`).
|
||||
So it is uninstallable on F44.
|
||||
- `hyprland` requires `aquamarine`, so it fails too.
|
||||
- The old `libdisplay-info-0.2.0` cannot be kept — same package name (file
|
||||
conflict), and `mutter` / `mesa-vulkan-drivers` / `wlroots` / `sway` all move
|
||||
to `.so.3` versions on F44.
|
||||
|
||||
This is **the sole blocker**. Everything else in the transaction resolves
|
||||
cleanly. It is a stale COPR rebuild, not a local misconfiguration. There is no
|
||||
native Hyprland/aquamarine in Fedora 44's repos, so the COPR is the only source.
|
||||
|
||||
No strong reason to rush: F43 is fully supported and the current GNOME/Hyprland
|
||||
setup works. Staying put until the COPR catches up.
|
||||
|
||||
### How to proceed next time (re-checking F44)
|
||||
|
||||
1. Refresh and re-attempt the resolve (no commitment — this only downloads):
|
||||
```bash
|
||||
sudo dnf upgrade --refresh
|
||||
sudo dnf system-upgrade download --releasever=44
|
||||
```
|
||||
2. **If it resolves** (no `libdisplay-info.so.2` / aquamarine error): the COPR
|
||||
has rebuilt. Proceed with `sudo dnf system-upgrade reboot`.
|
||||
3. **If it still errors on aquamarine / `libdisplay-info.so.2`:** the COPR is
|
||||
still behind. Quick check of the dep:
|
||||
```bash
|
||||
dnf repoquery --releasever=44 --repo='copr:*solopasha*' --requires aquamarine | grep display
|
||||
```
|
||||
`libdisplay-info.so.2` = still stale, keep waiting. `libdisplay-info.so.3` =
|
||||
ready to upgrade.
|
||||
|
||||
### Override path (only if F44 becomes urgent)
|
||||
|
||||
Proceed now by temporarily dropping the Hyprland stack and running on the GNOME
|
||||
fallback. Reinstall Hyprland from the COPR once it is rebuilt.
|
||||
|
||||
```bash
|
||||
sudo dnf remove aquamarine hyprland hyprland-uwsm xdg-desktop-portal-hyprland
|
||||
sudo dnf system-upgrade download --releasever=44
|
||||
sudo dnf system-upgrade reboot
|
||||
# after reboot into F44, log into GNOME; reinstall when COPR is ready:
|
||||
# sudo dnf install hyprland hyprland-uwsm xdg-desktop-portal-hyprland
|
||||
```
|
||||
|
|
@ -0,0 +1,165 @@
|
|||
# Incident log
|
||||
|
||||
Dated entries for diagnosed-and-fixed system incidents on this machine.
|
||||
Newest first. Keep entries factual: symptom, root cause, what was ruled out,
|
||||
fix applied, and any follow-up.
|
||||
|
||||
## 2026-06-26 — memtest86+ confirms bad RAM (root cause of the 2026-06-24 Btrfs corruption)
|
||||
|
||||
**Result: FAILED.** Ran memtest86+ v8.10 from the GRUB boot entry on the
|
||||
Ryzen 5 5600G / 95.3 GB DDR4-2666 (CAS 20-19-19-43) workstation.
|
||||
|
||||
- **Status: Failed!, Errors: 7** at only **0:00:51** elapsed — still in Pass 0,
|
||||
Test #3 (Moving inversions, 1s & 0s). Errored almost immediately.
|
||||
- Both captured failures are **single-bit flips** (signature of a failing
|
||||
module, not a controller-wide / addressing fault):
|
||||
- `0x000bc0115dd0` (47 GB): expected `f…`, found `bf…` → **bit 62 flipped 1→0**
|
||||
- `0x001005f391e0` (64 GB): expected `0…`, found `…01000000` → **bit 24 flipped 0→1**
|
||||
- Temps 58/65 °C (fine). DDR4-2666 is JEDEC-standard, **not** an aggressive
|
||||
XMP/EXPO overclock — so this is not merely an unstable memory profile.
|
||||
|
||||
**Conclusion**
|
||||
|
||||
Confirms the 2026-06-24 hypothesis: scattered multi-file Btrfs csum corruption
|
||||
with zero drive I/O errors and clean NVMe SMART = **bad RAM** writing garbage to
|
||||
otherwise-good flash. Root cause: **confirmed**.
|
||||
|
||||
**RAM layout (from `dmidecode -t memory`, 2026-06-26)**
|
||||
|
||||
96 GB total = **two complete G.Skill Ripjaws V kits**, both DDR4-3200 CL16
|
||||
running at 2666 (XMP/DOCP off):
|
||||
|
||||
| Slot | Size | Part number | Kit |
|
||||
|------|------|-------------|-----|
|
||||
| Channel A · DIMM 0 | 32 GB | F4-3200C16-32GVK | 2×32 GB pair → 64 GB |
|
||||
| Channel B · DIMM 0 | 32 GB | F4-3200C16-32GVK | (matched cross-channel) |
|
||||
| Channel A · DIMM 1 | 16 GB | F4-3200C16-16GVK | 2×16 GB pair → 32 GB |
|
||||
| Channel B · DIMM 1 | 16 GB | F4-3200C16-16GVK | (matched cross-channel) |
|
||||
|
||||
Two valid matched dual-channel pairs → whichever stick fails, drop to the other
|
||||
complete pair and keep dual-channel. **Fallbacks need zero purchase:** if a
|
||||
16 GB stick is bad → run the 2×32 = 64 GB; if a 32 GB stick is bad → run the
|
||||
2×16 = 32 GB.
|
||||
|
||||
**Booting memtest (GRUB hotkeys unreliable on this box)**
|
||||
|
||||
memtest86+ 8.10 is installed (`/boot/memtest86+x64.efi`); GRUB uses
|
||||
`GRUB_DEFAULT=saved`, so one-shot boot needs no keypresses:
|
||||
|
||||
```bash
|
||||
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
|
||||
sudo grep -o "menuentry '[^']*emtest[^']*'" /boot/grub2/grub.cfg # get exact title
|
||||
sudo grub2-reboot "<exact title>" # one-shot; reverts after next boot
|
||||
sudo systemctl poweroff # pull sticks, then power on → boots memtest
|
||||
```
|
||||
|
||||
Exit memtest with **Esc** (reboots back to Fedora).
|
||||
|
||||
**Follow-up / next steps**
|
||||
|
||||
- **Treat the machine as untrustworthy until the RAM is replaced** — any data
|
||||
written (including backups) since the corruption began may be silently bad.
|
||||
Be cautious about pruning known-good restic snapshots.
|
||||
- **Isolate the bad DIMM** — test **by kit first** (the 2×32, then the 2×16);
|
||||
a passing pair is immediately a runnable config. Only split a *failing* pair
|
||||
to find the single bad stick. Errors at 47 GB / 64 GB are interleaved logical
|
||||
addresses and do NOT map to one physical stick.
|
||||
- **Don't discard the bad stick** — G.Skill Ripjaws V carry a limited lifetime
|
||||
warranty; RMA for a free replacement (restores the matched pair at no cost).
|
||||
Keep its good partner as a spare or for the RMA-restored pair. Avoid buying a
|
||||
single stick to pair (DDR4 matching is finicky) unless RMA fails.
|
||||
- After fixing, **re-run memtest to a clean multi-pass**, then **re-scrub
|
||||
`/home`** and watch for *new* csum errors. None appearing = corruption was
|
||||
historical and is now resolved. Defer `restic check --read-data` until the RAM
|
||||
is known-good (a bad-RAM box throws false integrity errors).
|
||||
|
||||
## 2026-06-24 — Chrome repeatedly crashing (SIGBUS), corrupt Favicons block on single-copy Btrfs
|
||||
|
||||
**Symptom**
|
||||
|
||||
Google Chrome crashed within seconds of launch, repeatedly. Three crashes in
|
||||
three minutes on 2026-06-24 (13:15, 13:16, 13:17), all signal **SIGBUS**.
|
||||
History of crashes back to April (mixed SIGTRAP / SIGILL / SIGSEGV / SIGBUS),
|
||||
18 minidumps total.
|
||||
|
||||
**Root cause**
|
||||
|
||||
A single corrupt data block at offset **942080** of
|
||||
`~/.config/google-chrome/Profile 2/Favicons` (a SQLite DB, inode 141150,
|
||||
btrfs root 257). Chrome `mmap()`s the file on startup; the kernel raises SIGBUS
|
||||
when it cannot fault in the bad page.
|
||||
|
||||
Each SIGBUS coredump correlated to the second with a kernel log line:
|
||||
|
||||
```
|
||||
BTRFS warning: csum failed root 257 ino 141150 off 942080 csum 0x0858109c expected 0x0858109e mirror 1
|
||||
```
|
||||
|
||||
Same single-bit flip every time (`9c` vs `9e`). The volume is `Data, single`
|
||||
(no mirror), so Btrfs detects but **cannot self-heal**. `corruption_errs` was
|
||||
climbing (41 → 42). Disk `read` / `write` / `flush` `io_errs` were all 0 —
|
||||
silent at-rest corruption; the NVMe is not reporting hardware faults.
|
||||
|
||||
**Ruled out**
|
||||
|
||||
- **OOM** — 87 GiB RAM free, and SIGBUS ≠ SIGKILL.
|
||||
- **Disk full** — 5% used.
|
||||
- **Chrome version / update** — install dated 2026-06-20 predates the crashes;
|
||||
`rpm -V` clean.
|
||||
- **Profile / extension corruption.**
|
||||
- **Dual-GPU / Wayland stack** — GPU libs in the dump were merely mapped; the
|
||||
`gpu_channel.cc Buffer Handle is null` line was a downstream symptom logged
|
||||
~2s after the SIGBUS.
|
||||
|
||||
**Fix applied (2026-06-24)**
|
||||
|
||||
Confirmed Chrome not running, then deleted the regenerable cache files
|
||||
`Favicons` and `Favicons-journal` from `Profile 2/`. Chrome regenerates these on
|
||||
next launch. Verified gone.
|
||||
|
||||
**Scrub result (2026-06-24 13:28)**
|
||||
|
||||
`sudo btrfs scrub start -B /home` finished in 0:55, scanned 79.57 GiB.
|
||||
**Error summary: `csum=2`, Corrected 0, Uncorrectable 2.** The corruption is
|
||||
**not** isolated to the Favicons file — two further uncorrectable blocks, in
|
||||
two different subvolumes, both regenerable/reinstallable:
|
||||
|
||||
| File | Subvol | Action |
|
||||
|------|--------|--------|
|
||||
| `~/.cache/google-chrome/Profile 1/Cache/Cache_Data/5f188fe6012a3cdc_0` | home (root 257) | Deleted (regenerates) ✅ |
|
||||
| `/usr/share/help/fr/gdm/index.docbook` | root (root 259) | Owned by `gdm-49.2-2.fc43`; restore via `sudo dnf reinstall gdm` |
|
||||
|
||||
`corruption_errs` 44; `read`/`write`/`flush` `io_errs` 0.
|
||||
|
||||
**Remediation + verification (2026-06-24 13:38–13:42)**
|
||||
|
||||
- Deleted the Chrome cache block (`rm`, regenerates).
|
||||
- Restored the gdm file: `sudo dnf reinstall gdm`.
|
||||
- **SSD health is clean** — `sudo nvme smart-log /dev/nvme0n1`:
|
||||
`critical_warning 0`, `media_errors 0`, `percentage_used 0%`. The drive has
|
||||
logged **zero** media errors at the hardware level → a *failing SSD* is
|
||||
unlikely; corruption reached good flash from upstream (RAM/bus).
|
||||
- **Second scrub (13:41)** still reports `csum=2` uncorrectable — but at the
|
||||
**exact same two physical blocks** as the first scrub (`logical 1754923008`,
|
||||
`logical 21439315968`), and this time **with no resolvable path**. The live
|
||||
files now point at fresh extents; the corrupt extents are **orphaned dead
|
||||
blocks** (not snapshot-pinned — `snapper`/`timeshift` not installed).
|
||||
|
||||
**Assessment**
|
||||
|
||||
Same two static blocks across two scrubs 13 min apart = corruption is **static,
|
||||
not actively spreading** (no new blocks appeared). Combined with the clean SSD
|
||||
SMART, this most likely stems from a **single past corruption event** rather
|
||||
than an actively-failing component. The crash-causing live files are fixed;
|
||||
the residual scrub errors affect no live file.
|
||||
|
||||
**Outstanding / follow-up**
|
||||
|
||||
- **Re-scrub in a day or two** (`sudo btrfs scrub start -B /home`). If the two
|
||||
orphaned blocks have cleared (reclaimed) and no *new* errors appear → fully
|
||||
resolved. **New** blocks appearing = corruption is ongoing → back up now and
|
||||
run memtest.
|
||||
- **Run memtest86+** (Fedora GRUB boot entry) for confidence — scattered
|
||||
multi-file corruption with zero drive I/O errors is the classic bad-RAM
|
||||
signature, even if currently static.
|
||||
- The `corruption_errs` counter will not drop on a single-copy volume — expected.
|
||||
|
|
@ -0,0 +1,97 @@
|
|||
# PRD — Deeper orchestration audit
|
||||
|
||||
**Status:** ready to run (new session) · **Created:** 2026-06-30
|
||||
**Predecessor:** `docs/cc-tool-audit.md` (§ Addendum — orchestration audit)
|
||||
**Note:** No issue tracker/remote on this repo, so this PRD lives as a doc. The
|
||||
`/to-prd` "publish + `ready-for-agent` label" step was adapted to a file.
|
||||
|
||||
---
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Sessions feel slow and over-spend on model cost. A prior audit
|
||||
(`docs/cc-tool-audit.md`) traced this less to loaded tools/skills/hooks (all small)
|
||||
and more to the **orchestrator-subagent delegation pattern**. A first-pass audit of
|
||||
1,962 subagent transcripts found that **27% of subagents run on Opus**, the majority
|
||||
of them doing mechanical file-edit/shell work that Haiku/Sonnet would handle faster
|
||||
and cheaper. The CLAUDE.md "Session Orchestration" routing table is **advisory** —
|
||||
when an `Agent` spawn omits `model`, the subagent inherits the parent (Opus). The
|
||||
rule also mandates delegation with **no minimum complexity threshold**, so trivial
|
||||
one-line edits pay a full subagent prefill + sequential wait.
|
||||
|
||||
An interim fix is already applied (`CLAUDE_CODE_SUBAGENT_MODEL=haiku` in global
|
||||
settings). What's missing is the **quantified evidence** needed to safely revise the
|
||||
orchestration *instructions* across many CLAUDE.md files — not just the default model.
|
||||
|
||||
## Solution
|
||||
|
||||
A deeper, repeatable audit of the existing session transcripts that produces the
|
||||
numbers to ground a **pilot** revision of the Session Orchestration rules on one
|
||||
project, before any broad rollout. The audit answers: how often is the wrong model
|
||||
chosen, is delegation itself wasteful for small tasks, and is work being decomposed
|
||||
well before hand-off. Output is a short findings report + a proposed pilot rule diff.
|
||||
|
||||
## User Stories
|
||||
|
||||
1. As the operator, I want the subagent model distribution broken down per project, so that I can see which repos leak Opus most.
|
||||
2. As the operator, I want the "file-edit/shell" subagent bucket split into *trivial* (1–2 calls, single file) vs *substantial* (multi-file, many calls), so that I can tell genuine Sonnet/Opus work from Haiku-appropriate work instead of relying on a coarse bucket.
|
||||
3. As the operator, I want each subagent's spawn traced to whether its model was **explicitly passed** by the orchestrator or **inherited** from the parent, so that I can measure the routing-leak rate directly.
|
||||
4. As the operator, I want a count of subagents that did near-zero work (0–1 tool calls, or returned trivially), so that I can quantify wasted delegations caused by the "no threshold" rule.
|
||||
5. As the operator, I want to see how many orchestrator turns spawned a single subagent for a single small op (delegation that a direct op would beat on latency), so that I can justify a complexity threshold.
|
||||
6. As the operator, I want parallelism measured — how often independent subagents were batched concurrently vs. spawned sequentially — so that I can see if decomposition exploits the parallel-batch guidance.
|
||||
7. As the operator, I want examples of good and bad decomposition (a clean parallel batch vs. a needless single-op delegation), so that the pilot rule change is illustrated, not just asserted.
|
||||
8. As the operator, I want the audit limited to delegation-heavy projects (cc-os, llf-schema, proxmox, philly-seo, ovh-prod), so that it runs on the relevant sample, not every repo.
|
||||
9. As the operator, I want the interim `CLAUDE_CODE_SUBAGENT_MODEL=haiku` effect measurable in future sessions (a before/after marker), so that I can confirm the env var actually shifted the distribution.
|
||||
10. As the operator, I want a proposed pilot diff to one project's "Session Orchestration" section (threshold + enforced routing + allow orienting reads), so that I can try it on a single repo and measure wall-clock/cost before rolling out.
|
||||
11. As the operator, I want the audit scripted/repeatable (saved under the repo), so that I can re-run it after the pilot to compare.
|
||||
|
||||
## Implementation Decisions
|
||||
|
||||
- **Data source:** existing JSONL transcripts in `~/.claude/projects/*/` (main) and
|
||||
`~/.claude/projects/*/<session-id>/subagents/agent-*.jsonl` (subagents). Model is in
|
||||
each subagent message's `model` field; tool calls are `type:"tool_use"` blocks
|
||||
(walk recursively). Project dir names start with `-` — use python `glob`, never
|
||||
shell `find`/`ls`.
|
||||
- **Trivial vs substantial classifier:** trivial = ≤2 tool calls and ≤1 distinct file
|
||||
touched; substantial = otherwise. Tune threshold against spot-checked examples.
|
||||
- **Inherited vs explicit model trace:** match parent `Agent` tool_use input (does it
|
||||
carry a `model`/`subagent_type`?) to the child transcript via agent id, then compare
|
||||
requested vs actual model. Where the parent omitted model and child==parent model,
|
||||
classify as "inherited (leak)."
|
||||
- **Parallelism metric:** within a main session, detect `Agent` tool_use blocks issued
|
||||
in the same assistant turn (concurrent) vs separate turns (sequential).
|
||||
- **Scope:** the 5 delegation-heavy projects above; report per-project and pooled.
|
||||
- **Output:** append a "Deep audit" section to `docs/cc-tool-audit.md` (or a sibling
|
||||
doc) with tables + 3–5 examples, plus a proposed pilot rule diff for ONE project.
|
||||
- **No production change** beyond the already-applied env var until the pilot is
|
||||
reviewed.
|
||||
|
||||
## Testing Decisions
|
||||
|
||||
- This is an analysis task, not a shipped feature — "tests" = sanity checks on the
|
||||
parser: total subagent count matches the prior audit (~1,962 with a model), model
|
||||
totals reconcile, and the trivial/substantial split sums to the file-edit bucket.
|
||||
- Spot-verify the classifier on 5 hand-read transcripts (one per project) before
|
||||
trusting the aggregate.
|
||||
- Re-run the same script post-pilot to confirm the distribution shifted (regression
|
||||
baseline = today's numbers).
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- Editing every CLAUDE.md (this is assessment → single-project pilot only).
|
||||
- Changing tool/skill/hook config (prior audit showed negligible payoff).
|
||||
- Building any new tooling, MCP, or harness changes.
|
||||
- Touching the live orchestration behavior beyond the env var already set.
|
||||
|
||||
## Further Notes
|
||||
|
||||
- Interim change already live: `env.CLAUDE_CODE_SUBAGENT_MODEL=haiku` in
|
||||
`~/.claude/settings.json` (applies to sessions launched after 2026-06-30).
|
||||
- Prior-audit baseline to beat: Opus 26.6% / Sonnet 36.7% / Haiku 36.5% of subagents;
|
||||
Opus subagents 53% file-edit/shell.
|
||||
- Reusable scratch scripts from the first pass:
|
||||
`/tmp/claude-1000/.../scratchpad/{full,cat,orch}.py` (ephemeral — recreate or
|
||||
promote into the repo when running the deep audit).
|
||||
- Open instruction-design questions for the pilot: where to set the complexity
|
||||
threshold; whether to allow N orienting reads before delegating; how to phrase
|
||||
"always pass model on spawn" given the env var now provides a cheap default.
|
||||
|
|
@ -36,7 +36,7 @@
|
|||
|
||||
- [ ] Create the `scripts/` directory:
|
||||
```bash
|
||||
mkdir -p /home/jared/systems-admin/scripts
|
||||
mkdir -p /home/jared/servers/desktop/scripts
|
||||
```
|
||||
|
||||
- [ ] Create `scripts/notify_backup.rb` with the full header comment and empty class stubs. The `if __FILE__ == $PROGRAM_NAME` guard keeps the CLI from running during tests:
|
||||
|
|
@ -114,13 +114,13 @@ require_relative 'notify_backup'
|
|||
|
||||
- [ ] Run the test file to confirm the skeleton loads without errors:
|
||||
```bash
|
||||
ruby /home/jared/systems-admin/scripts/notify_backup_test.rb
|
||||
ruby /home/jared/servers/desktop/scripts/notify_backup_test.rb
|
||||
```
|
||||
Expected output: `0 runs, 0 assertions, 0 failures, 0 errors`
|
||||
|
||||
- [ ] Commit:
|
||||
```bash
|
||||
cd /home/jared/systems-admin
|
||||
cd /home/jared/servers/desktop
|
||||
git add scripts/
|
||||
git commit -m "feat(backup-monitor): add notify_backup script skeleton and test infrastructure"
|
||||
```
|
||||
|
|
@ -165,7 +165,7 @@ end
|
|||
|
||||
- [ ] Run to confirm 6 failures:
|
||||
```bash
|
||||
ruby /home/jared/systems-admin/scripts/notify_backup_test.rb
|
||||
ruby /home/jared/servers/desktop/scripts/notify_backup_test.rb
|
||||
```
|
||||
Expected: `6 failures` (NoMethodError on ByteFormatter.format)
|
||||
|
||||
|
|
@ -189,13 +189,13 @@ end
|
|||
|
||||
- [ ] Run tests to confirm they pass:
|
||||
```bash
|
||||
ruby /home/jared/systems-admin/scripts/notify_backup_test.rb
|
||||
ruby /home/jared/servers/desktop/scripts/notify_backup_test.rb
|
||||
```
|
||||
Expected: `6 runs, 6 assertions, 0 failures, 0 errors`
|
||||
|
||||
- [ ] Commit:
|
||||
```bash
|
||||
cd /home/jared/systems-admin
|
||||
cd /home/jared/servers/desktop
|
||||
git add scripts/
|
||||
git commit -m "feat(backup-monitor): implement ByteFormatter with tests"
|
||||
```
|
||||
|
|
@ -265,7 +265,7 @@ end
|
|||
|
||||
- [ ] Run to confirm failures:
|
||||
```bash
|
||||
ruby /home/jared/systems-admin/scripts/notify_backup_test.rb
|
||||
ruby /home/jared/servers/desktop/scripts/notify_backup_test.rb
|
||||
```
|
||||
Expected: `PayloadBuilderTest` failures, `ByteFormatterTest` still passing
|
||||
|
||||
|
|
@ -305,13 +305,13 @@ end
|
|||
|
||||
- [ ] Run all tests:
|
||||
```bash
|
||||
ruby /home/jared/systems-admin/scripts/notify_backup_test.rb
|
||||
ruby /home/jared/servers/desktop/scripts/notify_backup_test.rb
|
||||
```
|
||||
Expected: `12 runs, 12+ assertions, 0 failures, 0 errors`
|
||||
|
||||
- [ ] Commit:
|
||||
```bash
|
||||
cd /home/jared/systems-admin
|
||||
cd /home/jared/servers/desktop
|
||||
git add scripts/
|
||||
git commit -m "feat(backup-monitor): implement PayloadBuilder with tests"
|
||||
```
|
||||
|
|
@ -388,7 +388,7 @@ end
|
|||
|
||||
- [ ] Run to confirm failures:
|
||||
```bash
|
||||
ruby /home/jared/systems-admin/scripts/notify_backup_test.rb
|
||||
ruby /home/jared/servers/desktop/scripts/notify_backup_test.rb
|
||||
```
|
||||
Expected: `WebhookClientTest` failures
|
||||
|
||||
|
|
@ -427,13 +427,13 @@ end
|
|||
|
||||
- [ ] Run all tests:
|
||||
```bash
|
||||
ruby /home/jared/systems-admin/scripts/notify_backup_test.rb
|
||||
ruby /home/jared/servers/desktop/scripts/notify_backup_test.rb
|
||||
```
|
||||
Expected: all tests pass, 0 failures
|
||||
|
||||
- [ ] Commit:
|
||||
```bash
|
||||
cd /home/jared/systems-admin
|
||||
cd /home/jared/servers/desktop
|
||||
git add scripts/
|
||||
git commit -m "feat(backup-monitor): implement WebhookClient with injected HTTP adapter and tests"
|
||||
```
|
||||
|
|
@ -483,7 +483,7 @@ end
|
|||
|
||||
- [ ] Run to confirm failures:
|
||||
```bash
|
||||
ruby /home/jared/systems-admin/scripts/notify_backup_test.rb
|
||||
ruby /home/jared/servers/desktop/scripts/notify_backup_test.rb
|
||||
```
|
||||
Expected: `BackupNotifierTest` failures
|
||||
|
||||
|
|
@ -545,13 +545,13 @@ end
|
|||
|
||||
- [ ] Run all tests:
|
||||
```bash
|
||||
ruby /home/jared/systems-admin/scripts/notify_backup_test.rb
|
||||
ruby /home/jared/servers/desktop/scripts/notify_backup_test.rb
|
||||
```
|
||||
Expected: all tests pass (roughly 16+ assertions, 0 failures)
|
||||
|
||||
- [ ] Commit:
|
||||
```bash
|
||||
cd /home/jared/systems-admin
|
||||
cd /home/jared/servers/desktop
|
||||
git add scripts/
|
||||
git commit -m "feat(backup-monitor): implement BackupNotifier and CLI entry point"
|
||||
```
|
||||
|
|
@ -565,7 +565,7 @@ git commit -m "feat(backup-monitor): implement BackupNotifier and CLI entry poin
|
|||
|
||||
- [ ] Copy and make executable:
|
||||
```bash
|
||||
cp /home/jared/systems-admin/scripts/notify_backup.rb ~/.local/bin/notify_backup.rb
|
||||
cp /home/jared/servers/desktop/scripts/notify_backup.rb ~/.local/bin/notify_backup.rb
|
||||
chmod +x ~/.local/bin/notify_backup.rb
|
||||
```
|
||||
|
||||
|
|
@ -771,15 +771,15 @@ if __name__ == "__main__":
|
|||
|
||||
- [ ] Make it executable and test it on a known-bad payload:
|
||||
```bash
|
||||
chmod +x /home/jared/systems-admin/scripts/validate-n8n-workflow.py
|
||||
chmod +x /home/jared/servers/desktop/scripts/validate-n8n-workflow.py
|
||||
echo '{"name":"test","nodes":[],"connections":{"missing-node":{"main":[[]]}}}' \
|
||||
| python3 /home/jared/systems-admin/scripts/validate-n8n-workflow.py /dev/stdin
|
||||
| python3 /home/jared/servers/desktop/scripts/validate-n8n-workflow.py /dev/stdin
|
||||
```
|
||||
Expected: `ERROR: connections: source 'missing-node' does not match any node name.`
|
||||
|
||||
- [ ] Commit:
|
||||
```bash
|
||||
cd /home/jared/systems-admin
|
||||
cd /home/jared/servers/desktop
|
||||
git add scripts/validate-n8n-workflow.py
|
||||
git commit -m "feat(backup-monitor): add n8n workflow JSON validator"
|
||||
```
|
||||
|
|
@ -1160,7 +1160,7 @@ puts " #{SCRIPTS_DIR}/n8n-workflow-reporter.json"
|
|||
|
||||
- [ ] Run the build script:
|
||||
```bash
|
||||
cd /home/jared/systems-admin
|
||||
cd /home/jared/servers/desktop
|
||||
ruby scripts/build-n8n-workflows.rb
|
||||
```
|
||||
Expected: two files created with no error output
|
||||
|
|
@ -1276,7 +1276,7 @@ Expected: no error output. Verify a row appears in the `backup_runs` table in n8
|
|||
|
||||
- [ ] Commit the final build script with real IDs:
|
||||
```bash
|
||||
cd /home/jared/systems-admin
|
||||
cd /home/jared/servers/desktop
|
||||
git add scripts/build-n8n-workflows.rb scripts/n8n-workflow-receiver.json scripts/n8n-workflow-reporter.json
|
||||
git commit -m "feat(backup-monitor): fill credential/datatable IDs and reimport workflows"
|
||||
```
|
||||
|
|
@ -1365,12 +1365,12 @@ Expected: no output (no syntax errors)
|
|||
|
||||
- [ ] Copy the updated backup.sh back to the source in this repo so it stays tracked:
|
||||
```bash
|
||||
cp ~/.local/bin/backup.sh /home/jared/systems-admin/scripts/backup.sh
|
||||
cp ~/.local/bin/backup.sh /home/jared/servers/desktop/scripts/backup.sh
|
||||
```
|
||||
|
||||
- [ ] Commit:
|
||||
```bash
|
||||
cd /home/jared/systems-admin
|
||||
cd /home/jared/servers/desktop
|
||||
git add scripts/backup.sh
|
||||
git commit -m "feat(backup-monitor): integrate notify_backup.rb into desktop backup.sh"
|
||||
```
|
||||
|
|
@ -1444,36 +1444,22 @@ Expected: no output
|
|||
|
||||
- [ ] Exit the SSH session. Back on desktop, commit the documentation of this change:
|
||||
```bash
|
||||
cd /home/jared/systems-admin
|
||||
cd /home/jared/servers/desktop
|
||||
git commit --allow-empty -m "feat(backup-monitor): deploy notify_backup.rb to OVH VPS and integrate into backup.sh"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 13: End-to-end smoke test
|
||||
## Task 13: End-to-end smoke test — COMPLETE (2026-05-07)
|
||||
|
||||
- [ ] **Trigger the desktop backup manually** and watch for webhook activity:
|
||||
```bash
|
||||
source ~/.credentials && bash ~/.local/bin/backup.sh
|
||||
```
|
||||
Expected: backup runs as normal, no new errors in output
|
||||
- [x] **Triggered desktop backup** — Synology succeeded (snapshot f5f25c53), B2 had stale lock from 2026-04-13 (PID 43407) causing `restic forget` to fail (exit 11). Both runs notified; rows recorded in datatable.
|
||||
- [x] **Datatable confirmed** — 3 rows visible in n8n UI after backup run.
|
||||
- [x] **Reporter triggered manually** — MailPace queued (id: 35621228, status: queued). Email sent to jaredmswanson@gmail.com.
|
||||
- [x] **Key bug fixed:** MailPace body field must be `textbody` (not `text_body`). Required explicit `Accept: application/json` and `Content-Type: application/json` headers in n8n HTTP Request node.
|
||||
|
||||
- [ ] **Check the n8n `backup_runs` datatable** — two new rows should appear: one for `desktop/synology` (success or skipped) and one for `desktop/backblaze-b2` (success)
|
||||
|
||||
- [ ] **Trigger the weekly reporter manually** in n8n (open Workflow 2 → Execute Workflow). With at least some runs in the table, you should receive an email at jaredmswanson@gmail.com.
|
||||
|
||||
- [ ] **Verify the email format:**
|
||||
- If all sources have at least one success: subject line starts with `✓`
|
||||
- If any source has no success: subject starts with `⚠` or `✗` with detail in the body
|
||||
|
||||
- [ ] **Remove any test rows** from `backup_runs` (rows with `destination=test` from Task 10)
|
||||
|
||||
- [ ] Final commit:
|
||||
```bash
|
||||
cd /home/jared/systems-admin
|
||||
git add -A
|
||||
git commit -m "docs(backup-monitor): implementation complete — smoke test passed"
|
||||
```
|
||||
**Remaining known issues (not blocking):**
|
||||
- Stale restic locks on Synology SFTP and B2 repos (from 2026-04-13) — `restic unlock` returns 0 but lock persists; `restic forget` fails, causing all runs to report `status: failure` even when snapshots save successfully. Run `restic -r <repo> unlock` manually or investigate lock file directly.
|
||||
- MailPace credential header: working config uses `MailPace-Server-Token` (not `X-Server-Token` as noted in Self-Review). Credential named `mailpace-token` in n8n (ID: `tC1vdhAHGVwAZxcg`).
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,132 @@
|
|||
# Synology DS218plus
|
||||
|
||||
Operational notes for accessing and navigating the Synology NAS without rediscovering paths.
|
||||
|
||||
## Connection
|
||||
|
||||
- SSH config alias: `synology`
|
||||
- Hostname/IP: `192.168.86.31`
|
||||
- SSH user: `jared`
|
||||
- SSH key from `~/.ssh/config`: `~/.ssh/id_ed25519_git`
|
||||
- Verified NAS hostname: `DS218plus`
|
||||
|
||||
Quick check:
|
||||
|
||||
```bash
|
||||
ssh synology 'hostname; id'
|
||||
```
|
||||
|
||||
Expected identity:
|
||||
|
||||
```text
|
||||
DS218plus
|
||||
uid=1026(jared) gid=100(users) groups=100(users),101(administrators),65536(family)
|
||||
```
|
||||
|
||||
## Volume Paths
|
||||
|
||||
**SSH and SFTP use completely different namespaces.** Do not mix them up.
|
||||
|
||||
**SSH shell** (`ssh synology`): home is `/var/services/homes/jared` (symlink to `/volume1/homes/jared`). Volume-level paths like `/volume1/...` are accessible.
|
||||
|
||||
**SFTP / restic**: shared folders appear directly under `/` as a virtual root. `/volume1` does not exist in SFTP space.
|
||||
|
||||
```bash
|
||||
# SFTP root listing
|
||||
sftp synology <<'EOF'
|
||||
ls /
|
||||
EOF
|
||||
# → archive backup home homes music proxmox_backup
|
||||
```
|
||||
|
||||
Active restic repo (SFTP path):
|
||||
|
||||
| Purpose | SFTP path |
|
||||
| --- | --- |
|
||||
| Fedora restic repo | `/homes/jared/Backups/fedora-workstation` |
|
||||
|
||||
Active restic repo (SSH path, for manual inspection):
|
||||
|
||||
| Purpose | SSH path |
|
||||
| --- | --- |
|
||||
| Fedora restic repo | `~/Backups/fedora-workstation` = `/volume1/homes/jared/Backups/fedora-workstation` |
|
||||
|
||||
## Restic Access
|
||||
|
||||
Local `restic` is installed at `/usr/bin/restic`. Credentials are sourced from `~/.credentials`; do not print the file or password.
|
||||
|
||||
The active backup script uses:
|
||||
|
||||
```bash
|
||||
source ~/.credentials
|
||||
restic -r "sftp:${SYNOLOGY_HOST}:${SYNOLOGY_PATH}" snapshots --compact
|
||||
```
|
||||
|
||||
As of 2026-06-05, `~/.credentials` has:
|
||||
|
||||
```text
|
||||
SYNOLOGY_HOST=synology
|
||||
SYNOLOGY_PATH=/homes/jared/Backups/fedora-workstation
|
||||
```
|
||||
|
||||
The leading path segment `homes` is the Synology shared folder name (visible in DSM), not a filesystem path. `/volume1/homes/...` is the SSH/filesystem equivalent but does not exist in SFTP space.
|
||||
|
||||
Repository structure:
|
||||
|
||||
```text
|
||||
config
|
||||
data/
|
||||
index/
|
||||
keys/
|
||||
locks/
|
||||
snapshots/
|
||||
```
|
||||
|
||||
## Useful Restic Commands
|
||||
|
||||
List snapshots:
|
||||
|
||||
```bash
|
||||
source ~/.credentials
|
||||
restic -r "sftp:${SYNOLOGY_HOST}:${SYNOLOGY_PATH}" snapshots --compact
|
||||
```
|
||||
|
||||
List a path in the latest snapshot:
|
||||
|
||||
```bash
|
||||
source ~/.credentials
|
||||
restic -r "sftp:${SYNOLOGY_HOST}:${SYNOLOGY_PATH}" ls latest /home/jared/.ssh
|
||||
```
|
||||
|
||||
Search snapshots for specific files:
|
||||
|
||||
```bash
|
||||
source ~/.credentials
|
||||
restic -r "sftp:${SYNOLOGY_HOST}:${SYNOLOGY_PATH}" find /home/jared/.ssh/id_rsa /home/jared/.ssh/id_rsa.pub
|
||||
```
|
||||
|
||||
Restore a specific file or directory to a temporary location:
|
||||
|
||||
```bash
|
||||
source ~/.credentials
|
||||
mkdir -p /tmp/restic-restore
|
||||
restic -r "sftp:${SYNOLOGY_HOST}:${SYNOLOGY_PATH}" restore latest \
|
||||
--target /tmp/restic-restore \
|
||||
--include /home/jared/path/to/file
|
||||
```
|
||||
|
||||
## Snapshot State on 2026-06-05
|
||||
|
||||
The Fedora restic repo was stalled since 2026-05-24 due to wrong SFTP path in `~/.credentials` (`/home/...` instead of `/homes/...`). Fixed 2026-06-05. Manual backup run confirmed new snapshot saved. Two stale locks were also cleared before the run.
|
||||
|
||||
Prior to fix, B2 had been running successfully every night; only Synology was affected.
|
||||
|
||||
## SSH Key Recovery Findings
|
||||
|
||||
Checked while investigating Hyperthrive VPS access on 2026-05-14:
|
||||
|
||||
- Latest restic snapshot `.ssh` contains only `config`, `id_ed25519_git`, `id_ed25519_git.pub`, `id_ed25519_ovh_backup`, `id_ed25519_ovh_backup.pub`, `known_hosts`, and `known_hosts.old`.
|
||||
- `restic find` did not find `/home/jared/.ssh/id_rsa`, `/home/jared/.ssh/id_rsa.pub`, `id_ed25519_forgejo`, or `id_ed25519_forgejo.pub`.
|
||||
- `/volume1/backup/jared_backup052124/.ssh` exists but is empty.
|
||||
- `/volume1/backup/fedora_backup052124/jared/.ssh` exists but is empty.
|
||||
- No `.ssh` files were found under `/volume1/backup/fedora_backup052124` or `/volume1/backup/fedora_recovery_oct2024` in the checked paths.
|
||||
|
|
@ -0,0 +1,61 @@
|
|||
# VPS Inventory
|
||||
|
||||
Operational notes for small VPS hosts. Keep secrets out of this file.
|
||||
|
||||
## Hyperthrive DigitalOcean Docker Services — DELETED 2026-06-24
|
||||
|
||||
**This VPS no longer exists.** The DigitalOcean droplet was destroyed on
|
||||
2026-06-24 after all services were migrated to the OVH VPS (below) and the box
|
||||
ran powered-off for weeks without incident. DNS verification confirmed no
|
||||
records pointed at `161.35.12.75` at deletion time. Service-by-service
|
||||
migration reconciliation lives in the `server-ovh-prod-01-main` project
|
||||
(`~/projects/server-ovh-prod-01-main/`). The notes below are retained as
|
||||
historical record only — the SSH alias and IP are dead.
|
||||
|
||||
- Former SSH alias: `hyperthrive-docker`
|
||||
- Former Host/IP: `161.35.12.75`
|
||||
- User: `jared`
|
||||
- Former local identity file: `~/.ssh/id_ed25519_git`
|
||||
- Provider label seen in console: `hyperthrive-docker-services`
|
||||
- OS observed on 2026-05-14: Ubuntu 24.04.4 LTS
|
||||
|
||||
### 2026-05-14 Recovery Notes
|
||||
|
||||
Problem:
|
||||
|
||||
- SSH initially failed with `Permission denied (publickey)`.
|
||||
- The old expected key was not recovered from local files, Synology restic snapshots, or older Synology backup folders.
|
||||
- DigitalOcean web console login as `jared` worked.
|
||||
- Root console login required a password change and disconnected during attempts.
|
||||
- `/` was at 99.8% usage on a 48.27 GiB filesystem.
|
||||
|
||||
Resolution:
|
||||
|
||||
- Freed space from journald and apt cache.
|
||||
- Started Docker.
|
||||
- `sudo docker system df` showed Docker images using 25.48 GiB with 20.91 GiB reclaimable.
|
||||
- Ran `sudo docker system prune -af`.
|
||||
- Docker reclaimed 20.68 GiB.
|
||||
- Added/confirmed `~/.ssh/id_ed25519_git.pub` in `jared`'s `authorized_keys`.
|
||||
- Updated local SSH config so `hyperthrive-docker` uses `~/.ssh/id_ed25519_git`.
|
||||
- SSH login works again with `ssh hyperthrive-docker`.
|
||||
|
||||
### Decommissioning — COMPLETE
|
||||
|
||||
All needed services were migrated from this DigitalOcean VPS to the OVH VPS,
|
||||
verified live, and the droplet was destroyed on 2026-06-24. No further action
|
||||
required. See `~/projects/server-ovh-prod-01-main/` for the full migrated
|
||||
service inventory and per-service migration records.
|
||||
|
||||
## OVH VPS
|
||||
|
||||
- Local SSH alias: `ovh-vps`
|
||||
- Host/IP: `15.204.247.153`
|
||||
- User: `jared`
|
||||
- Local identity file: `~/.ssh/id_ed25519_ovh_backup`
|
||||
|
||||
Quick access:
|
||||
|
||||
```bash
|
||||
ssh ovh-vps
|
||||
```
|
||||
|
|
@ -60,21 +60,12 @@ run_and_notify() {
|
|||
|
||||
echo "=== Backup started $(date) ===" | tee -a "$LOG"
|
||||
|
||||
# Synology: attempt only if reachable (SSH key auth, 30s timeout for HDD spin-up)
|
||||
if ssh -o ConnectTimeout=30 -o BatchMode=yes synology true 2>/dev/null; then
|
||||
run_and_notify "sftp:${SYNOLOGY_HOST}:${SYNOLOGY_PATH}" "Synology" "desktop" "synology"
|
||||
else
|
||||
echo "Synology unreachable — skipping SFTP backup" | tee -a "$LOG"
|
||||
notify_backup.rb \
|
||||
--machine desktop \
|
||||
--destination synology \
|
||||
--status skipped \
|
||||
--log-path "$LOG" \
|
||||
--notes "Synology unreachable at $(date)" || true
|
||||
fi
|
||||
# Synology SFTP: attempt directly — restic fails fast if unreachable, no pre-flight needed
|
||||
restic -r "sftp:${SYNOLOGY_HOST}:${SYNOLOGY_PATH}" unlock 2>&1 | tee -a "$LOG" || true
|
||||
run_and_notify "sftp:${SYNOLOGY_HOST}:${SYNOLOGY_PATH}" "Synology" "desktop" "synology" || true
|
||||
|
||||
# B2: clear any stale lock before attempting (safe no-op if no lock exists)
|
||||
restic -r "b2:${B2_BUCKET}:" unlock --remove-all 2>&1 | tee -a "$LOG"
|
||||
restic -r "b2:${B2_BUCKET}:" unlock 2>&1 | tee -a "$LOG" || true
|
||||
run_and_notify "b2:${B2_BUCKET}:" "Backblaze B2" "desktop" "backblaze-b2"
|
||||
|
||||
echo "=== Backup complete $(date) ===" | tee -a "$LOG"
|
||||
|
|
|
|||
|
|
@ -152,23 +152,13 @@ reporter = {
|
|||
},
|
||||
{
|
||||
parameters: {
|
||||
resource: 'row',
|
||||
operation: 'getMany',
|
||||
operation: 'getAll',
|
||||
dataTableId: {
|
||||
'__rl' => true,
|
||||
mode: 'list',
|
||||
value: 'cWeJhlybF9eV43DB',
|
||||
cachedResultName: 'backup_runs'
|
||||
},
|
||||
filters: {
|
||||
conditions: [
|
||||
{
|
||||
keyName: 'timestamp',
|
||||
condition: 'gt',
|
||||
keyValue: '={{ $now.minus({days: 7}).toISO() }}'
|
||||
}
|
||||
]
|
||||
},
|
||||
options: {}
|
||||
},
|
||||
id: SecureRandom.uuid,
|
||||
|
|
@ -207,9 +197,16 @@ reporter = {
|
|||
url: 'https://app.mailpace.com/api/v1/send',
|
||||
authentication: 'genericCredentialType',
|
||||
genericAuthType: 'httpHeaderAuth',
|
||||
sendHeaders: true,
|
||||
headerParameters: {
|
||||
parameters: [
|
||||
{ name: 'Accept', value: 'application/json' },
|
||||
{ name: 'Content-Type', value: 'application/json' }
|
||||
]
|
||||
},
|
||||
sendBody: true,
|
||||
specifyBody: 'json',
|
||||
jsonBody: '={{ { from: "backups@swansoncloud.com", to: "jaredmswanson@gmail.com", subject: $json.subject, text_body: $json.body } }}',
|
||||
jsonBody: '={{ { from: "backups@swansoncloud.com", to: "jaredmswanson@gmail.com", subject: $json.subject, textbody: $json.body } }}',
|
||||
options: {}
|
||||
},
|
||||
id: SecureRandom.uuid,
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
"responseMode": "responseNode",
|
||||
"options": {}
|
||||
},
|
||||
"id": "700da0a0-c2a5-4b9b-839a-10b7d0f37749",
|
||||
"id": "ccf460e0-6fd7-4e1f-b008-24cd45c87d8d",
|
||||
"name": "Webhook",
|
||||
"type": "n8n-nodes-base.webhook",
|
||||
"typeVersion": 2,
|
||||
|
|
@ -17,7 +17,7 @@
|
|||
240,
|
||||
300
|
||||
],
|
||||
"webhookId": "bdb7bf40-9f87-4483-b58e-5f2f6fa0c8a5",
|
||||
"webhookId": "425dfd98-23d8-4237-819b-53ee2fe50e57",
|
||||
"credentials": {
|
||||
"httpHeaderAuth": {
|
||||
"id": "qOYMJr5ENhFJZNDe",
|
||||
|
|
@ -49,7 +49,7 @@
|
|||
}
|
||||
}
|
||||
},
|
||||
"id": "3d251557-1841-466b-9ea6-895af4b6731f",
|
||||
"id": "f9ca5d27-b7fe-40cc-a6ae-3d39102e7255",
|
||||
"name": "Insert to backup_runs",
|
||||
"type": "n8n-nodes-base.dataTable",
|
||||
"typeVersion": 1,
|
||||
|
|
@ -66,7 +66,7 @@
|
|||
"responseCode": 200
|
||||
}
|
||||
},
|
||||
"id": "6448fef7-dd0f-4508-b896-4bd7c83b24bf",
|
||||
"id": "1c253ed9-fed5-4f45-a8f2-db81c731cd6f",
|
||||
"name": "Respond 200",
|
||||
"type": "n8n-nodes-base.respondToWebhook",
|
||||
"typeVersion": 1.1,
|
||||
|
|
|
|||
|
|
@ -17,7 +17,7 @@
|
|||
]
|
||||
}
|
||||
},
|
||||
"id": "ed22d29f-35f9-45ac-a7c7-b97ffa926736",
|
||||
"id": "f6fb320b-42e8-4894-ac52-d014dbab8ab1",
|
||||
"name": "Every Friday 8 AM",
|
||||
"type": "n8n-nodes-base.scheduleTrigger",
|
||||
"typeVersion": 1.2,
|
||||
|
|
@ -33,7 +33,7 @@
|
|||
"assignments": {
|
||||
"assignments": [
|
||||
{
|
||||
"id": "212c8f16-8900-4cbf-93bb-4802ad9c3f33",
|
||||
"id": "b46a89e2-f93f-4219-84cd-cf14a28d26c4",
|
||||
"name": "sources",
|
||||
"value": "={{ [{\"machine\":\"desktop\",\"destination\":\"synology\"},{\"machine\":\"desktop\",\"destination\":\"backblaze-b2\"},{\"machine\":\"vps-ovh-prod-01\",\"destination\":\"local\"}] }}",
|
||||
"type": "arrayValue"
|
||||
|
|
@ -42,7 +42,7 @@
|
|||
},
|
||||
"options": {}
|
||||
},
|
||||
"id": "3c3c5660-6075-4f73-9bb1-a7a70c91bfd6",
|
||||
"id": "251bc404-f466-4922-b2d8-1e7d8642e151",
|
||||
"name": "Set sources",
|
||||
"type": "n8n-nodes-base.set",
|
||||
"typeVersion": 3.4,
|
||||
|
|
@ -53,26 +53,16 @@
|
|||
},
|
||||
{
|
||||
"parameters": {
|
||||
"resource": "row",
|
||||
"operation": "getMany",
|
||||
"operation": "getAll",
|
||||
"dataTableId": {
|
||||
"__rl": true,
|
||||
"mode": "list",
|
||||
"value": "cWeJhlybF9eV43DB",
|
||||
"cachedResultName": "backup_runs"
|
||||
},
|
||||
"filters": {
|
||||
"conditions": [
|
||||
{
|
||||
"keyName": "timestamp",
|
||||
"condition": "gt",
|
||||
"keyValue": "={{ $now.minus({days: 7}).toISO() }}"
|
||||
}
|
||||
]
|
||||
},
|
||||
"options": {}
|
||||
},
|
||||
"id": "474f055d-08e6-4a47-8a1f-5f18818e7daf",
|
||||
"id": "7eec8f10-b3ed-4059-bd33-3bafccb56d03",
|
||||
"name": "Get backup_runs (last 7d)",
|
||||
"type": "n8n-nodes-base.dataTable",
|
||||
"typeVersion": 1,
|
||||
|
|
@ -87,7 +77,7 @@
|
|||
"language": "javaScript",
|
||||
"jsCode": "// Runs once for all items. References upstream nodes by exact name.\nconst sources = $('Set sources').first().json.sources;\nconst runs = $('Get backup_runs (last 7d)').all().map(item => item.json);\nconst weekAgo = new Date(Date.now() - 7 * 24 * 60 * 60 * 1000);\n\nconst results = sources.map(source => {\n const sourceRuns = runs.filter(r =>\n r.machine === source.machine &&\n r.destination === source.destination &&\n new Date(r.timestamp) >= weekAgo\n );\n const hasSuccess = sourceRuns.some(r => r.status === 'success');\n const hasAnyRun = sourceRuns.length > 0;\n const status = hasSuccess ? 'green' : hasAnyRun ? 'yellow' : 'red';\n const lastSuccess = sourceRuns\n .filter(r => r.status === 'success')\n .sort((a, b) => new Date(b.timestamp) - new Date(a.timestamp))[0] || null;\n return { ...source, status, runs: sourceRuns, lastSuccess };\n});\n\nconst overallStatus = results.some(r => r.status === 'red') ? 'red'\n : results.some(r => r.status === 'yellow') ? 'yellow' : 'green';\n\nreturn [{ json: { results, overallStatus } }];\n"
|
||||
},
|
||||
"id": "4211f3b0-1878-4979-a5be-eb01bdec2d63",
|
||||
"id": "2739ccac-48a4-4736-b7e6-6da384794ec9",
|
||||
"name": "Evaluate Sources",
|
||||
"type": "n8n-nodes-base.code",
|
||||
"typeVersion": 2,
|
||||
|
|
@ -102,7 +92,7 @@
|
|||
"language": "javaScript",
|
||||
"jsCode": "// Builds subject + body from Evaluate Sources output.\n// Uses an array of lines joined at the end to avoid \\n escape issues in JSON storage.\nconst { results, overallStatus } = $input.first().json;\nconst icon = { green: '✓', yellow: '⚠', red: '✗' };\nconst label = { green: 'Backups healthy', yellow: 'Backup warning', red: 'Backup failure' };\nconst weekOf = new Date().toLocaleDateString('en-US', { month: 'short', day: 'numeric' });\nconst subject = icon[overallStatus] + ' ' + label[overallStatus] + ' — week of ' + weekOf;\n\nconst fmt = ts => ts\n ? new Date(ts).toLocaleString('en-US', { month: 'short', day: 'numeric', hour: '2-digit', minute: '2-digit' })\n : 'unknown';\n\nconst L = [];\n\nif (overallStatus === 'green') {\n L.push('All backup sources ran successfully this week.');\n L.push('');\n results.forEach(r =>\n L.push('✓ ' + r.machine + ' / ' + r.destination +\n ' (last success: ' + fmt(r.lastSuccess && r.lastSuccess.timestamp) + ')')\n );\n} else {\n const problems = results.filter(r => r.status !== 'green');\n const healthy = results.filter(r => r.status === 'green');\n\n L.push(problems.length + ' source(s) need attention:');\n L.push('');\n\n problems.forEach(r => {\n L.push(icon[r.status] + ' ' + r.machine + ' / ' + r.destination);\n\n if (r.status === 'yellow') {\n L.push(' Ran this week but never succeeded.');\n r.runs.slice(0, 3).forEach(run => {\n var line = ' ' + fmt(run.timestamp) + ' [' + run.status + ']';\n if (run.log_path) line += ' log: ' + run.log_path;\n L.push(line);\n if (run.notes) L.push(' notes: ' + (run.notes || '').slice(0, 300));\n });\n L.push('');\n L.push(' Next steps:');\n L.push(' 1. Check the log at the path shown above');\n L.push(' 2. For Synology failures: verify LAN reachability (ssh synology) — disks need ~30s to spin up');\n L.push(' 3. Re-run manually: bash ~/.local/bin/backup.sh');\n L.push(' 4. One-off skip (power outage, travel) is expected — yellow is a heads-up, not an alarm');\n L.push('');\n }\n\n if (r.status === 'red') {\n L.push(' No runs reported all week — silent failure.');\n if (r.lastSuccess) {\n L.push(' Last known good run: ' + fmt(r.lastSuccess.timestamp));\n if (r.lastSuccess.log_path) L.push(' Last log: ' + r.lastSuccess.log_path);\n } else {\n L.push(' No history found in the datatable.');\n }\n L.push('');\n L.push(' Next steps:');\n L.push(' 1. Check systemd timer: systemctl --user status restic-backup.timer');\n L.push(' 2. Verify backup.sh calls notify_backup.rb at the end');\n L.push(' 3. Check n8n Workflow 1 execution log for webhook errors');\n L.push(' 4. Run manually: bash ~/.local/bin/backup.sh');\n L.push('');\n }\n });\n\n if (healthy.length > 0) {\n L.push('Healthy sources:');\n healthy.forEach(r => L.push('✓ ' + r.machine + ' / ' + r.destination));\n }\n}\n\nreturn [{ json: { subject, body: L.join('\\n') } }];\n"
|
||||
},
|
||||
"id": "99ace456-148e-493b-af7d-174ea62c1448",
|
||||
"id": "6d6e9968-2a9a-4e52-8c94-72d6ec7552b8",
|
||||
"name": "Build Email",
|
||||
"type": "n8n-nodes-base.code",
|
||||
"typeVersion": 2,
|
||||
|
|
@ -117,12 +107,25 @@
|
|||
"url": "https://app.mailpace.com/api/v1/send",
|
||||
"authentication": "genericCredentialType",
|
||||
"genericAuthType": "httpHeaderAuth",
|
||||
"sendHeaders": true,
|
||||
"headerParameters": {
|
||||
"parameters": [
|
||||
{
|
||||
"name": "Accept",
|
||||
"value": "application/json"
|
||||
},
|
||||
{
|
||||
"name": "Content-Type",
|
||||
"value": "application/json"
|
||||
}
|
||||
]
|
||||
},
|
||||
"sendBody": true,
|
||||
"specifyBody": "json",
|
||||
"jsonBody": "={{ { from: \"backups@swansoncloud.com\", to: \"jaredmswanson@gmail.com\", subject: $json.subject, text_body: $json.body } }}",
|
||||
"jsonBody": "={{ { from: \"backups@swansoncloud.com\", to: \"jaredmswanson@gmail.com\", subject: $json.subject, textbody: $json.body } }}",
|
||||
"options": {}
|
||||
},
|
||||
"id": "cab8c0f5-8ebb-468f-ac83-8032c1020913",
|
||||
"id": "a05f5f30-df7e-4df7-bef0-15f6f7f9ceca",
|
||||
"name": "Send via MailPace",
|
||||
"type": "n8n-nodes-base.httpRequest",
|
||||
"typeVersion": 4.2,
|
||||
|
|
|
|||
Loading…
Reference in New Issue