184 lines
7.4 KiB
Markdown
184 lines
7.4 KiB
Markdown
# n8n Backup Monitor — Design Spec
|
|
|
|
**Date:** 2026-05-07
|
|
**Status:** Approved
|
|
|
|
## Overview
|
|
|
|
A lightweight backup health monitoring system built on the existing self-hosted n8n instance at `https://n8n.swansoncloud.com`. Each backup script calls a shared Ruby notifier script after it runs. The notifier POSTs a structured result to an n8n webhook. Every Friday morning, a second n8n workflow checks the week's backup history against a defined expected list and sends a health digest email via MailPace.
|
|
|
|
---
|
|
|
|
## Architecture & Data Flow
|
|
|
|
```
|
|
backup.sh (or any backup script)
|
|
└── calls notify_backup.rb with result args
|
|
└── POSTs JSON to n8n webhook (auth header from ENV)
|
|
|
|
n8n Workflow 1 (Receiver)
|
|
├── Webhook trigger (POST /backup-notify)
|
|
├── Validate Authorization header
|
|
└── Append row to "backup_runs" datatable
|
|
|
|
n8n Workflow 2 (Reporter)
|
|
├── Cron trigger (Friday, 8:00 AM)
|
|
├── Read "backup_runs" datatable (last 7 days)
|
|
├── Load expected sources from Set node
|
|
├── For each expected source: classify status
|
|
├── Roll up overall status (GREEN / YELLOW / RED)
|
|
└── Send MailPace email to jaredmswanson@gmail.com
|
|
```
|
|
|
|
---
|
|
|
|
## Ruby Notifier Script (`notify_backup.rb`)
|
|
|
|
### Design principles
|
|
|
|
- **Sandi Metz OOP:** Small, single-purpose objects. Each class has one reason to change. Dependencies injected, not hardcoded. No class does more than one job.
|
|
- **Self-documenting:** The script's header comment explains its purpose, all accepted arguments, exit behavior, and required ENV vars — enough for any future backup process to integrate without reading the implementation.
|
|
- **Classes:**
|
|
- `BackupNotifier` — orchestrates the flow; the only public entry point
|
|
- `PayloadBuilder` — converts raw CLI args into a clean hash (formats bytes, validates status)
|
|
- `WebhookClient` — handles the HTTP POST with auth header; knows nothing about backup data
|
|
- `ByteFormatter` — single responsibility: raw integer → human-readable string
|
|
- CLI argument parsing at the top level (OptionParser), outside of classes
|
|
|
|
**Location:** `~/.local/bin/notify_backup.rb` (on each machine that runs backups)
|
|
|
|
**Called from backup scripts with named arguments:**
|
|
|
|
```bash
|
|
notify_backup.rb \
|
|
--machine desktop \
|
|
--destination synology \
|
|
--status success \
|
|
--bytes 4831838208 \
|
|
--snapshot a1b2c3d4 \
|
|
--log-path ~/.local/log/restic/2026-05-07.log \
|
|
--notes "$(tail -20 $LOG)"
|
|
```
|
|
|
|
**Behavior:**
|
|
|
|
- Reads `N8N_SWANSONCLOUD_URL` and `N8N_SWANSONCLOUD_BACKUP_AUTH_TOKEN` from ENV (sourced via `~/.credentials`)
|
|
- Converts raw `--bytes` integer to human-readable string (`4.50 GB`, `823 MB`, `12 KB`) before sending
|
|
- POSTs JSON payload to `$N8N_SWANSONCLOUD_URL/webhook/backup-notify` with `Authorization: Bearer <token>` header
|
|
- Exits 0 whether the POST succeeds or fails — backup reporting failure must never fail the backup job itself
|
|
- Logs a warning to stderr on POST failure so it surfaces in the backup log
|
|
|
|
**Valid `--status` values:** `success`, `failure`, `skipped`
|
|
|
|
- `skipped` — job was intentionally not attempted (e.g. Synology unreachable)
|
|
- `failure` — job was attempted and failed
|
|
- `success` — job completed successfully
|
|
|
|
---
|
|
|
|
## Datatable Schema (`backup_runs`)
|
|
|
|
| Column | Type | Example |
|
|
|---|---|---|
|
|
| `id` | auto | — |
|
|
| `timestamp` | datetime | `2026-05-07T01:58:44Z` |
|
|
| `machine` | string | `desktop`, `vps-ovh-prod-01` |
|
|
| `destination` | string | `synology`, `backblaze-b2` |
|
|
| `status` | string | `success`, `failure`, `skipped` |
|
|
| `bytes_transferred` | string | `4.50 GB` |
|
|
| `snapshot_id` | string | `a1b2c3d4` |
|
|
| `log_path` | string | `~/.local/log/restic/2026-05-07.log` |
|
|
| `notes` | text | free-form restic output or skip reason |
|
|
|
|
The `machine` + `destination` pair is the identity key matched against the expected list.
|
|
|
|
---
|
|
|
|
## Workflow 1 — Receiver
|
|
|
|
**Trigger:** Webhook, POST `/backup-notify`
|
|
|
|
**Nodes:**
|
|
|
|
1. **Webhook** — POST only, path `/backup-notify`
|
|
2. **Validate Auth** — If node: checks `Authorization` header equals `Bearer <token>` (stored as n8n credential). On failure: respond 401 and halt.
|
|
3. **Append to Datatable** — writes one row to `backup_runs`
|
|
4. **Respond 200** — returns `{"ok": true}`
|
|
|
|
The receiver has no logic beyond auth + write. Malformed payloads are logged by n8n without blocking the caller.
|
|
|
|
**Auth token:** Stored in n8n as a credential. Must match `N8N_SWANSONCLOUD_BACKUP_AUTH_TOKEN` in the sending machine's environment.
|
|
|
|
---
|
|
|
|
## Workflow 2 — Reporter
|
|
|
|
**Trigger:** Cron, every Friday at 8:00 AM
|
|
|
|
**Nodes:**
|
|
|
|
1. **Cron** — Friday 8:00 AM
|
|
2. **Set: Expected Sources** — hardcoded list of `{machine, destination}` pairs. Edit this node to add/remove sources. Initial list:
|
|
- `desktop / synology`
|
|
- `desktop / backblaze-b2`
|
|
- `vps-ovh-prod-01 / local`
|
|
|
|
Note: the VPS backup chain is `OVH server → Synology pull → Hyper Backup to B2`. Only the first step (OVH server creating `latest.tar.gz`) is scriptable with `notify_backup.rb`. The Synology pull and Hyper Backup steps are Synology DSM-managed tasks with no easy hook. Monitoring those downstream steps is deferred to a future phase.
|
|
3. **Read Datatable** — fetches all `backup_runs` rows with `timestamp` in the past 7 days
|
|
4. **Evaluate Each Source** — for each expected `machine/destination` pair:
|
|
- At least one `success` this week → **green**
|
|
- No `success`, but rows exist (`failure` or `skipped`) → **yellow**
|
|
- No rows at all → **red** (silent — never reported in)
|
|
5. **Roll Up Status:**
|
|
- Any red source → overall **RED**
|
|
- No red, any yellow → overall **YELLOW**
|
|
- All green → overall **GREEN**
|
|
6. **Build Email** — see Email Format below
|
|
7. **MailPace** — sends to `jaredmswanson@gmail.com`
|
|
|
|
---
|
|
|
|
## Email Format
|
|
|
|
### GREEN
|
|
|
|
- **Subject:** `✓ Backups healthy — week of May 4`
|
|
- **Body:** One-line confirmation. Table of all sources with their run count and last success timestamp.
|
|
|
|
### YELLOW
|
|
|
|
- **Subject:** `⚠ Backup warning — week of May 4`
|
|
- **Body:**
|
|
- Summary of which sources had no successful run
|
|
- For each yellow source: failure/skipped rows with timestamp, status, notes, and log path
|
|
- Next steps: check the log at the listed path, verify connectivity (for Synology: check LAN reachability, HDD spin-up), re-run manually if needed
|
|
- Note: a single skipped night (e.g. power outage) is expected — yellow just means "worth a glance"
|
|
|
|
### RED
|
|
|
|
- **Subject:** `✗ Backup failure — week of May 4`
|
|
- **Body:**
|
|
- Which sources reported zero runs all week
|
|
- Last known good run date for each red source (from datatable history)
|
|
- Log path from the most recent row if any exists
|
|
- Next steps: verify the backup script is still being called (check systemd timer), confirm notifier script is reachable, check n8n logs
|
|
|
|
---
|
|
|
|
## Security
|
|
|
|
- Webhook protected by `Authorization: Bearer <token>` header
|
|
- Token stored in n8n as a credential (not in workflow JSON)
|
|
- Token available on sending machines via `N8N_SWANSONCLOUD_BACKUP_AUTH_TOKEN` env var, sourced from `~/.credentials`
|
|
- `~/.credentials` is excluded from dotfiles git tracking
|
|
|
|
---
|
|
|
|
## Adding a New Backup Source
|
|
|
|
1. Add the `notify_backup.rb` call to the new backup script with appropriate `--machine` and `--destination` values
|
|
2. Ensure `N8N_SWANSONCLOUD_URL` and `N8N_SWANSONCLOUD_BACKUP_AUTH_TOKEN` are in the machine's environment
|
|
3. Add the new `{machine, destination}` pair to the **Set: Expected Sources** node in Workflow 2
|
|
|
|
No datatable schema changes required.
|