6.6 KiB

Raw Blame History

n8n Backup Monitor — Design Spec

Date: 2026-05-07 Status: Approved

Overview

A lightweight backup health monitoring system built on the existing self-hosted n8n instance at https://n8n.swansoncloud.com. Each backup script calls a shared Ruby notifier script after it runs. The notifier POSTs a structured result to an n8n webhook. Every Friday morning, a second n8n workflow checks the week's backup history against a defined expected list and sends a health digest email via MailPace.

Architecture & Data Flow

backup.sh (or any backup script)
    └── calls notify_backup.rb with result args
            └── POSTs JSON to n8n webhook (auth header from ENV)

n8n Workflow 1 (Receiver)
    ├── Webhook trigger (POST /backup-notify)
    ├── Validate Authorization header
    └── Append row to "backup_runs" datatable

n8n Workflow 2 (Reporter)
    ├── Cron trigger (Friday, 8:00 AM)
    ├── Read "backup_runs" datatable (last 7 days)
    ├── Load expected sources from Set node
    ├── For each expected source: classify status
    ├── Roll up overall status (GREEN / YELLOW / RED)
    └── Send MailPace email to jaredmswanson@gmail.com

Ruby Notifier Script (`notify_backup.rb`)

Location: ~/.local/bin/notify_backup.rb (on each machine that runs backups)

Called from backup scripts with named arguments:

notify_backup.rb \
  --machine desktop \
  --destination synology \
  --status success \
  --bytes 4831838208 \
  --snapshot a1b2c3d4 \
  --log-path ~/.local/log/restic/2026-05-07.log \
  --notes "$(tail -20 $LOG)"

Behavior:

Reads N8N_SWANSONCLOUD_URL and N8N_SWANSONCLOUD_BACKUP_AUTH_TOKEN from ENV (sourced via ~/.credentials)
Converts raw --bytes integer to human-readable string (4.50 GB, 823 MB, 12 KB) before sending
POSTs JSON payload to $N8N_SWANSONCLOUD_URL/webhook/backup-notify with Authorization: Bearer <token> header
Exits 0 whether the POST succeeds or fails — backup reporting failure must never fail the backup job itself
Logs a warning to stderr on POST failure so it surfaces in the backup log

Valid --status values: success, failure, skipped

skipped — job was intentionally not attempted (e.g. Synology unreachable)
failure — job was attempted and failed
success — job completed successfully

Datatable Schema (`backup_runs`)

Column	Type	Example
`id`	auto	—
`timestamp`	datetime	`2026-05-07T01:58:44Z`
`machine`	string	`desktop`, `vps-ovh-prod-01`
`destination`	string	`synology`, `backblaze-b2`
`status`	string	`success`, `failure`, `skipped`
`bytes_transferred`	string	`4.50 GB`
`snapshot_id`	string	`a1b2c3d4`
`log_path`	string	`~/.local/log/restic/2026-05-07.log`
`notes`	text	free-form restic output or skip reason

The machine + destination pair is the identity key matched against the expected list.

Workflow 1 — Receiver

Trigger: Webhook, POST /backup-notify

Nodes:

Webhook — POST only, path /backup-notify
Validate Auth — If node: checks Authorization header equals Bearer <token> (stored as n8n credential). On failure: respond 401 and halt.
Append to Datatable — writes one row to backup_runs
Respond 200 — returns {"ok": true}

The receiver has no logic beyond auth + write. Malformed payloads are logged by n8n without blocking the caller.

Auth token: Stored in n8n as a credential. Must match N8N_SWANSONCLOUD_BACKUP_AUTH_TOKEN in the sending machine's environment.

Workflow 2 — Reporter

Trigger: Cron, every Friday at 8:00 AM

Nodes:

Cron — Friday 8:00 AM
Set: Expected Sources — hardcoded list of {machine, destination} pairs. Edit this node to add/remove sources. Initial list:
- desktop / synology
- desktop / backblaze-b2
- vps-ovh-prod-01 / local
Note: the VPS backup chain is OVH server → Synology pull → Hyper Backup to B2. Only the first step (OVH server creating latest.tar.gz) is scriptable with notify_backup.rb. The Synology pull and Hyper Backup steps are Synology DSM-managed tasks with no easy hook. Monitoring those downstream steps is deferred to a future phase.
Read Datatable — fetches all backup_runs rows with timestamp in the past 7 days
Evaluate Each Source — for each expected machine/destination pair:
- At least one success this week → green
- No success, but rows exist (failure or skipped) → yellow
- No rows at all → red (silent — never reported in)
Roll Up Status:
- Any red source → overall RED
- No red, any yellow → overall YELLOW
- All green → overall GREEN
Build Email — see Email Format below
MailPace — sends to jaredmswanson@gmail.com

Email Format

GREEN

Subject: ✓ Backups healthy — week of May 4
Body: One-line confirmation. Table of all sources with their run count and last success timestamp.

YELLOW

Subject: ⚠ Backup warning — week of May 4
Body:
- Summary of which sources had no successful run
- For each yellow source: failure/skipped rows with timestamp, status, notes, and log path
- Next steps: check the log at the listed path, verify connectivity (for Synology: check LAN reachability, HDD spin-up), re-run manually if needed
- Note: a single skipped night (e.g. power outage) is expected — yellow just means "worth a glance"

RED

Subject: ✗ Backup failure — week of May 4
Body:
- Which sources reported zero runs all week
- Last known good run date for each red source (from datatable history)
- Log path from the most recent row if any exists
- Next steps: verify the backup script is still being called (check systemd timer), confirm notifier script is reachable, check n8n logs

Security

Webhook protected by Authorization: Bearer <token> header
Token stored in n8n as a credential (not in workflow JSON)
Token available on sending machines via N8N_SWANSONCLOUD_BACKUP_AUTH_TOKEN env var, sourced from ~/.credentials
~/.credentials is excluded from dotfiles git tracking

Adding a New Backup Source

Add the notify_backup.rb call to the new backup script with appropriate --machine and --destination values
Ensure N8N_SWANSONCLOUD_URL and N8N_SWANSONCLOUD_BACKUP_AUTH_TOKEN are in the machine's environment
Add the new {machine, destination} pair to the Set: Expected Sources node in Workflow 2

No datatable schema changes required.

6.6 KiB Raw Blame History