# n8n Backup Monitor — Design Spec

**Date:** 2026-05-07
**Status:** Approved

## Overview

A lightweight backup health monitoring system built on the existing self-hosted n8n instance at `https://n8n.swansoncloud.com`. Each backup script calls a shared Ruby notifier script after it runs. The notifier POSTs a structured result to an n8n webhook. Every Friday morning, a second n8n workflow checks the week's backup history against a defined expected list and sends a health digest email via MailPace.

---

## Architecture & Data Flow

```
backup.sh (or any backup script)
    └── calls notify_backup.rb with result args
            └── POSTs JSON to n8n webhook (auth header from ENV)

n8n Workflow 1 (Receiver)
    ├── Webhook trigger (POST /backup-notify)
    ├── Validate Authorization header
    └── Append row to "backup_runs" datatable

n8n Workflow 2 (Reporter)
    ├── Cron trigger (Friday, 8:00 AM)
    ├── Read "backup_runs" datatable (last 7 days)
    ├── Load expected sources from Set node
    ├── For each expected source: classify status
    ├── Roll up overall status (GREEN / YELLOW / RED)
    └── Send MailPace email to jaredmswanson@gmail.com
```

---

## Ruby Notifier Script (`notify_backup.rb`)

**Location:** `~/.local/bin/notify_backup.rb` (on each machine that runs backups)

**Called from backup scripts with named arguments:**

```bash
notify_backup.rb \
  --machine desktop \
  --destination synology \
  --status success \
  --bytes 4831838208 \
  --snapshot a1b2c3d4 \
  --log-path ~/.local/log/restic/2026-05-07.log \
  --notes "$(tail -20 $LOG)"
```

**Behavior:**

- Reads `N8N_SWANSONCLOUD_URL` and `N8N_SWANSONCLOUD_BACKUP_AUTH_TOKEN` from ENV (sourced via `~/.credentials`)
- Converts raw `--bytes` integer to human-readable string (`4.50 GB`, `823 MB`, `12 KB`) before sending
- POSTs JSON payload to `$N8N_SWANSONCLOUD_URL/webhook/backup-notify` with `Authorization: Bearer <token>` header
- Exits 0 whether the POST succeeds or fails — backup reporting failure must never fail the backup job itself
- Logs a warning to stderr on POST failure so it surfaces in the backup log

**Valid `--status` values:** `success`, `failure`, `skipped`

- `skipped` — job was intentionally not attempted (e.g. Synology unreachable)
- `failure` — job was attempted and failed
- `success` — job completed successfully

---

## Datatable Schema (`backup_runs`)

| Column | Type | Example |
|---|---|---|
| `id` | auto | — |
| `timestamp` | datetime | `2026-05-07T01:58:44Z` |
| `machine` | string | `desktop`, `vps-ovh-prod-01` |
| `destination` | string | `synology`, `backblaze-b2` |
| `status` | string | `success`, `failure`, `skipped` |
| `bytes_transferred` | string | `4.50 GB` |
| `snapshot_id` | string | `a1b2c3d4` |
| `log_path` | string | `~/.local/log/restic/2026-05-07.log` |
| `notes` | text | free-form restic output or skip reason |

The `machine` + `destination` pair is the identity key matched against the expected list.

---

## Workflow 1 — Receiver

**Trigger:** Webhook, POST `/backup-notify`

**Nodes:**

1. **Webhook** — POST only, path `/backup-notify`
2. **Validate Auth** — If node: checks `Authorization` header equals `Bearer <token>` (stored as n8n credential). On failure: respond 401 and halt.
3. **Append to Datatable** — writes one row to `backup_runs`
4. **Respond 200** — returns `{"ok": true}`

The receiver has no logic beyond auth + write. Malformed payloads are logged by n8n without blocking the caller.

**Auth token:** Stored in n8n as a credential. Must match `N8N_SWANSONCLOUD_BACKUP_AUTH_TOKEN` in the sending machine's environment.

---

## Workflow 2 — Reporter

**Trigger:** Cron, every Friday at 8:00 AM

**Nodes:**

1. **Cron** — Friday 8:00 AM
2. **Set: Expected Sources** — hardcoded list of `{machine, destination}` pairs. Edit this node to add/remove sources. Initial list:
   - `desktop / synology`
   - `desktop / backblaze-b2`
   - `vps-ovh-prod-01 / local`

   Note: the VPS backup chain is `OVH server → Synology pull → Hyper Backup to B2`. Only the first step (OVH server creating `latest.tar.gz`) is scriptable with `notify_backup.rb`. The Synology pull and Hyper Backup steps are Synology DSM-managed tasks with no easy hook. Monitoring those downstream steps is deferred to a future phase.
3. **Read Datatable** — fetches all `backup_runs` rows with `timestamp` in the past 7 days
4. **Evaluate Each Source** — for each expected `machine/destination` pair:
   - At least one `success` this week → **green**
   - No `success`, but rows exist (`failure` or `skipped`) → **yellow**
   - No rows at all → **red** (silent — never reported in)
5. **Roll Up Status:**
   - Any red source → overall **RED**
   - No red, any yellow → overall **YELLOW**
   - All green → overall **GREEN**
6. **Build Email** — see Email Format below
7. **MailPace** — sends to `jaredmswanson@gmail.com`

---

## Email Format

### GREEN

- **Subject:** `✓ Backups healthy — week of May 4`
- **Body:** One-line confirmation. Table of all sources with their run count and last success timestamp.

### YELLOW

- **Subject:** `⚠ Backup warning — week of May 4`
- **Body:**
  - Summary of which sources had no successful run
  - For each yellow source: failure/skipped rows with timestamp, status, notes, and log path
  - Next steps: check the log at the listed path, verify connectivity (for Synology: check LAN reachability, HDD spin-up), re-run manually if needed
  - Note: a single skipped night (e.g. power outage) is expected — yellow just means "worth a glance"

### RED

- **Subject:** `✗ Backup failure — week of May 4`
- **Body:**
  - Which sources reported zero runs all week
  - Last known good run date for each red source (from datatable history)
  - Log path from the most recent row if any exists
  - Next steps: verify the backup script is still being called (check systemd timer), confirm notifier script is reachable, check n8n logs

---

## Security

- Webhook protected by `Authorization: Bearer <token>` header
- Token stored in n8n as a credential (not in workflow JSON)
- Token available on sending machines via `N8N_SWANSONCLOUD_BACKUP_AUTH_TOKEN` env var, sourced from `~/.credentials`
- `~/.credentials` is excluded from dotfiles git tracking

---

## Adding a New Backup Source

1. Add the `notify_backup.rb` call to the new backup script with appropriate `--machine` and `--destination` values
2. Ensure `N8N_SWANSONCLOUD_URL` and `N8N_SWANSONCLOUD_BACKUP_AUTH_TOKEN` are in the machine's environment
3. Add the new `{machine, destination}` pair to the **Set: Expected Sources** node in Workflow 2

No datatable schema changes required.