From dc74ea1206b2191f5d92d787dafa0490a6bf5ab4 Mon Sep 17 00:00:00 2001 From: jared Date: Thu, 7 May 2026 10:29:33 -0400 Subject: [PATCH] docs(backups): add n8n backup monitor design spec Two-workflow n8n system: webhook receiver records backup run results into a datatable; weekly Friday reporter compares against expected sources and sends green/yellow/red health digest via MailPace. Co-Authored-By: Claude Sonnet 4.6 --- .../2026-05-07-n8n-backup-monitor-design.md | 172 ++++++++++++++++++ 1 file changed, 172 insertions(+) create mode 100644 docs/superpowers/specs/2026-05-07-n8n-backup-monitor-design.md diff --git a/docs/superpowers/specs/2026-05-07-n8n-backup-monitor-design.md b/docs/superpowers/specs/2026-05-07-n8n-backup-monitor-design.md new file mode 100644 index 0000000..46348d3 --- /dev/null +++ b/docs/superpowers/specs/2026-05-07-n8n-backup-monitor-design.md @@ -0,0 +1,172 @@ +# n8n Backup Monitor — Design Spec + +**Date:** 2026-05-07 +**Status:** Approved + +## Overview + +A lightweight backup health monitoring system built on the existing self-hosted n8n instance at `https://n8n.swansoncloud.com`. Each backup script calls a shared Ruby notifier script after it runs. The notifier POSTs a structured result to an n8n webhook. Every Friday morning, a second n8n workflow checks the week's backup history against a defined expected list and sends a health digest email via MailPace. + +--- + +## Architecture & Data Flow + +``` +backup.sh (or any backup script) + └── calls notify_backup.rb with result args + └── POSTs JSON to n8n webhook (auth header from ENV) + +n8n Workflow 1 (Receiver) + ├── Webhook trigger (POST /backup-notify) + ├── Validate Authorization header + └── Append row to "backup_runs" datatable + +n8n Workflow 2 (Reporter) + ├── Cron trigger (Friday, 8:00 AM) + ├── Read "backup_runs" datatable (last 7 days) + ├── Load expected sources from Set node + ├── For each expected source: classify status + ├── Roll up overall status (GREEN / YELLOW / RED) + └── Send MailPace email to jaredmswanson@gmail.com +``` + +--- + +## Ruby Notifier Script (`notify_backup.rb`) + +**Location:** `~/.local/bin/notify_backup.rb` (on each machine that runs backups) + +**Called from backup scripts with named arguments:** + +```bash +notify_backup.rb \ + --machine desktop \ + --destination synology \ + --status success \ + --bytes 4831838208 \ + --snapshot a1b2c3d4 \ + --log-path ~/.local/log/restic/2026-05-07.log \ + --notes "$(tail -20 $LOG)" +``` + +**Behavior:** + +- Reads `N8N_SWANSONCLOUD_URL` and `N8N_SWANSONCLOUD_BACKUP_AUTH_TOKEN` from ENV (sourced via `~/.credentials`) +- Converts raw `--bytes` integer to human-readable string (`4.50 GB`, `823 MB`, `12 KB`) before sending +- POSTs JSON payload to `$N8N_SWANSONCLOUD_URL/webhook/backup-notify` with `Authorization: Bearer ` header +- Exits 0 whether the POST succeeds or fails — backup reporting failure must never fail the backup job itself +- Logs a warning to stderr on POST failure so it surfaces in the backup log + +**Valid `--status` values:** `success`, `failure`, `skipped` + +- `skipped` — job was intentionally not attempted (e.g. Synology unreachable) +- `failure` — job was attempted and failed +- `success` — job completed successfully + +--- + +## Datatable Schema (`backup_runs`) + +| Column | Type | Example | +|---|---|---| +| `id` | auto | — | +| `timestamp` | datetime | `2026-05-07T01:58:44Z` | +| `machine` | string | `desktop`, `vps-ovh-prod-01` | +| `destination` | string | `synology`, `backblaze-b2` | +| `status` | string | `success`, `failure`, `skipped` | +| `bytes_transferred` | string | `4.50 GB` | +| `snapshot_id` | string | `a1b2c3d4` | +| `log_path` | string | `~/.local/log/restic/2026-05-07.log` | +| `notes` | text | free-form restic output or skip reason | + +The `machine` + `destination` pair is the identity key matched against the expected list. + +--- + +## Workflow 1 — Receiver + +**Trigger:** Webhook, POST `/backup-notify` + +**Nodes:** + +1. **Webhook** — POST only, path `/backup-notify` +2. **Validate Auth** — If node: checks `Authorization` header equals `Bearer ` (stored as n8n credential). On failure: respond 401 and halt. +3. **Append to Datatable** — writes one row to `backup_runs` +4. **Respond 200** — returns `{"ok": true}` + +The receiver has no logic beyond auth + write. Malformed payloads are logged by n8n without blocking the caller. + +**Auth token:** Stored in n8n as a credential. Must match `N8N_SWANSONCLOUD_BACKUP_AUTH_TOKEN` in the sending machine's environment. + +--- + +## Workflow 2 — Reporter + +**Trigger:** Cron, every Friday at 8:00 AM + +**Nodes:** + +1. **Cron** — Friday 8:00 AM +2. **Set: Expected Sources** — hardcoded list of `{machine, destination}` pairs. Edit this node to add/remove sources. Initial list: + - `desktop / synology` + - `desktop / backblaze-b2` + - `vps-ovh-prod-01 / local` + + Note: the VPS backup chain is `OVH server → Synology pull → Hyper Backup to B2`. Only the first step (OVH server creating `latest.tar.gz`) is scriptable with `notify_backup.rb`. The Synology pull and Hyper Backup steps are Synology DSM-managed tasks with no easy hook. Monitoring those downstream steps is deferred to a future phase. +3. **Read Datatable** — fetches all `backup_runs` rows with `timestamp` in the past 7 days +4. **Evaluate Each Source** — for each expected `machine/destination` pair: + - At least one `success` this week → **green** + - No `success`, but rows exist (`failure` or `skipped`) → **yellow** + - No rows at all → **red** (silent — never reported in) +5. **Roll Up Status:** + - Any red source → overall **RED** + - No red, any yellow → overall **YELLOW** + - All green → overall **GREEN** +6. **Build Email** — see Email Format below +7. **MailPace** — sends to `jaredmswanson@gmail.com` + +--- + +## Email Format + +### GREEN + +- **Subject:** `✓ Backups healthy — week of May 4` +- **Body:** One-line confirmation. Table of all sources with their run count and last success timestamp. + +### YELLOW + +- **Subject:** `⚠ Backup warning — week of May 4` +- **Body:** + - Summary of which sources had no successful run + - For each yellow source: failure/skipped rows with timestamp, status, notes, and log path + - Next steps: check the log at the listed path, verify connectivity (for Synology: check LAN reachability, HDD spin-up), re-run manually if needed + - Note: a single skipped night (e.g. power outage) is expected — yellow just means "worth a glance" + +### RED + +- **Subject:** `✗ Backup failure — week of May 4` +- **Body:** + - Which sources reported zero runs all week + - Last known good run date for each red source (from datatable history) + - Log path from the most recent row if any exists + - Next steps: verify the backup script is still being called (check systemd timer), confirm notifier script is reachable, check n8n logs + +--- + +## Security + +- Webhook protected by `Authorization: Bearer ` header +- Token stored in n8n as a credential (not in workflow JSON) +- Token available on sending machines via `N8N_SWANSONCLOUD_BACKUP_AUTH_TOKEN` env var, sourced from `~/.credentials` +- `~/.credentials` is excluded from dotfiles git tracking + +--- + +## Adding a New Backup Source + +1. Add the `notify_backup.rb` call to the new backup script with appropriate `--machine` and `--destination` values +2. Ensure `N8N_SWANSONCLOUD_URL` and `N8N_SWANSONCLOUD_BACKUP_AUTH_TOKEN` are in the machine's environment +3. Add the new `{machine, destination}` pair to the **Set: Expected Sources** node in Workflow 2 + +No datatable schema changes required.