diff --git a/docs/superpowers/plans/2026-05-07-n8n-backup-monitor.md b/docs/superpowers/plans/2026-05-07-n8n-backup-monitor.md new file mode 100644 index 0000000..ec3e9e8 --- /dev/null +++ b/docs/superpowers/plans/2026-05-07-n8n-backup-monitor.md @@ -0,0 +1,1053 @@ +# n8n Backup Monitor Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Build a webhook-based backup health monitoring system: a reusable Ruby notifier script, two n8n workflows (receiver + weekly reporter), and integrations into the desktop and VPS backup scripts. + +**Architecture:** `notify_backup.rb` is a standalone Ruby CLI script (no gems, stdlib only) that POSTs a structured JSON payload to an n8n webhook after each backup run. n8n Workflow 1 validates auth and records each run to a datatable. Every Friday at 8 AM, n8n Workflow 2 reads the week's data, compares it against a hardcoded expected-sources list, and emails a green/yellow/red health digest via MailPace. + +**Tech Stack:** Ruby stdlib (net/http, json, optparse, minitest), n8n self-hosted at https://n8n.swansoncloud.com (Webhook, Schedule, Code, HTTP Request nodes, datatables), MailPace REST API, bash + +--- + +## File Map + +| File | Action | Responsibility | +|------|--------|----------------| +| `scripts/notify_backup.rb` | Create | Source-of-record for the notifier script | +| `scripts/notify_backup_test.rb` | Create | Minitest suite for all four classes | +| `~/.local/bin/notify_backup.rb` | Deploy | Runtime location on desktop | +| `~/.local/bin/backup.sh` | Modify | Add notify calls after each restic run | +| `~/services/backup.sh` on OVH | Modify via SSH | Add notify calls after VPS backup run | + +n8n workflows are configured in the n8n UI — no local files, but each task below gives exact node settings. + +--- + +## Task 1: Script skeleton and test infrastructure + +**Files:** +- Create: `scripts/notify_backup.rb` +- Create: `scripts/notify_backup_test.rb` + +- [ ] Create the `scripts/` directory: +```bash +mkdir -p /home/jared/systems-admin/scripts +``` + +- [ ] Create `scripts/notify_backup.rb` with the full header comment and empty class stubs. The `if __FILE__ == $PROGRAM_NAME` guard keeps the CLI from running during tests: + +```ruby +#!/usr/bin/env ruby +# frozen_string_literal: true + +# notify_backup — send a backup run result to the n8n backup monitor +# +# USAGE +# notify_backup.rb [options] +# +# REQUIRED OPTIONS +# --machine NAME Identifier for this host (e.g. desktop, vps-ovh-prod-01) +# --destination NAME Backup destination (e.g. synology, backblaze-b2, local) +# --status STATUS Result: success | failure | skipped +# +# OPTIONAL OPTIONS +# --bytes N Bytes transferred (raw integer; formatted to human-readable before sending) +# --snapshot ID Restic snapshot ID or equivalent +# --log-path PATH Path to the backup log file for this run +# --notes TEXT Free-form output (restic tail, skip reason, error message, etc.) +# +# REQUIRED ENV VARS +# N8N_SWANSONCLOUD_URL Base URL of the n8n instance (no trailing slash) +# e.g. https://n8n.swansoncloud.com +# N8N_SWANSONCLOUD_BACKUP_AUTH_TOKEN Bearer token for webhook authentication +# +# EXIT BEHAVIOR +# Always exits 0. A notification failure is logged to stderr but never causes +# the calling backup process to fail — the backup ran; the log is the source of truth. +# +# ADDING A NEW BACKUP SOURCE +# 1. Ensure the two ENV vars above are available on the new host +# 2. Call this script at the end of the backup process with --machine, --destination, --status +# 3. Add the {machine, destination} pair to the Set: Expected Sources node in n8n Workflow 2 +# +# CLASSES +# ByteFormatter raw integer bytes → human-readable string (e.g. "4.50 GB") +# PayloadBuilder converts CLI option hash into a clean JSON-ready hash +# WebhookClient sends HTTP POST with auth header; knows nothing about backup data +# BackupNotifier orchestrates: build payload → send; the only public entry point + +require 'net/http' +require 'json' +require 'optparse' + +class ByteFormatter +end + +class PayloadBuilder +end + +class WebhookClient +end + +class BackupNotifier +end + +if __FILE__ == $PROGRAM_NAME + # CLI entry point — implemented in Task 5 +end +``` + +- [ ] Create `scripts/notify_backup_test.rb`: + +```ruby +#!/usr/bin/env ruby +# frozen_string_literal: true + +require 'minitest/autorun' +require_relative 'notify_backup' +``` + +- [ ] Run the test file to confirm the skeleton loads without errors: +```bash +ruby /home/jared/systems-admin/scripts/notify_backup_test.rb +``` +Expected output: `0 runs, 0 assertions, 0 failures, 0 errors` + +- [ ] Commit: +```bash +cd /home/jared/systems-admin +git add scripts/ +git commit -m "feat(backup-monitor): add notify_backup script skeleton and test infrastructure" +``` + +--- + +## Task 2: ByteFormatter + +**Files:** +- Modify: `scripts/notify_backup.rb` — implement `ByteFormatter` +- Modify: `scripts/notify_backup_test.rb` — add `ByteFormatterTest` + +- [ ] Add the failing tests to `scripts/notify_backup_test.rb`: + +```ruby +class ByteFormatterTest < Minitest::Test + def test_gigabytes + assert_equal '4.50 GB', ByteFormatter.format(4_831_838_208) + end + + def test_megabytes + assert_equal '823.00 MB', ByteFormatter.format(863_076_352) + end + + def test_kilobytes + assert_equal '12.00 KB', ByteFormatter.format(12_288) + end + + def test_bytes_below_one_kb + assert_equal '512 B', ByteFormatter.format(512) + end + + def test_nil_returns_nil + assert_nil ByteFormatter.format(nil) + end + + def test_string_integer_is_coerced + assert_equal '1.00 GB', ByteFormatter.format('1073741824') + end +end +``` + +- [ ] Run to confirm 6 failures: +```bash +ruby /home/jared/systems-admin/scripts/notify_backup_test.rb +``` +Expected: `6 failures` (NoMethodError on ByteFormatter.format) + +- [ ] Replace the empty `ByteFormatter` class stub in `scripts/notify_backup.rb` with: + +```ruby +class ByteFormatter + UNITS = [['GB', 1_073_741_824], ['MB', 1_048_576], ['KB', 1_024]].freeze + + def self.format(bytes) + return nil if bytes.nil? + + bytes = bytes.to_i + unit, divisor = UNITS.find { |_, d| bytes >= d } + return "#{bytes} B" unless unit + + '%.2f %s' % [bytes.to_f / divisor, unit] + end +end +``` + +- [ ] Run tests to confirm they pass: +```bash +ruby /home/jared/systems-admin/scripts/notify_backup_test.rb +``` +Expected: `6 runs, 6 assertions, 0 failures, 0 errors` + +- [ ] Commit: +```bash +cd /home/jared/systems-admin +git add scripts/ +git commit -m "feat(backup-monitor): implement ByteFormatter with tests" +``` + +--- + +## Task 3: PayloadBuilder + +**Files:** +- Modify: `scripts/notify_backup.rb` — implement `PayloadBuilder` +- Modify: `scripts/notify_backup_test.rb` — add `PayloadBuilderTest` + +- [ ] Add the failing tests to `scripts/notify_backup_test.rb`: + +```ruby +class PayloadBuilderTest < Minitest::Test + def base_options + { machine: 'desktop', destination: 'synology', status: 'success' } + end + + def test_builds_required_fields + payload = PayloadBuilder.new(base_options).build + assert_equal 'desktop', payload[:machine] + assert_equal 'synology', payload[:destination] + assert_equal 'success', payload[:status] + assert_match(/\A\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z\z/, payload[:timestamp]) + end + + def test_formats_bytes_to_human_readable + payload = PayloadBuilder.new(base_options.merge(bytes: 1_073_741_824)).build + assert_equal '1.00 GB', payload[:bytes_transferred] + end + + def test_omits_nil_optional_fields + payload = PayloadBuilder.new(base_options).build + refute payload.key?(:bytes_transferred) + refute payload.key?(:snapshot_id) + refute payload.key?(:log_path) + refute payload.key?(:notes) + end + + def test_includes_all_optional_fields_when_present + opts = base_options.merge( + bytes: 1_048_576, + snapshot: 'abc12345', + log_path: '/home/jared/.local/log/restic/2026-05-07.log', + notes: 'snapshot abc12345 saved' + ) + payload = PayloadBuilder.new(opts).build + assert_equal '1.00 MB', payload[:bytes_transferred] + assert_equal 'abc12345', payload[:snapshot_id] + assert_equal '/home/jared/.local/log/restic/2026-05-07.log', payload[:log_path] + assert_equal 'snapshot abc12345 saved', payload[:notes] + end + + def test_raises_on_invalid_status + assert_raises(ArgumentError) { PayloadBuilder.new(base_options.merge(status: 'unknown')).build } + end + + def test_all_valid_statuses_accepted + %w[success failure skipped].each do |s| + assert PayloadBuilder.new(base_options.merge(status: s)).build + end + end +end +``` + +- [ ] Run to confirm failures: +```bash +ruby /home/jared/systems-admin/scripts/notify_backup_test.rb +``` +Expected: `PayloadBuilderTest` failures, `ByteFormatterTest` still passing + +- [ ] Replace the empty `PayloadBuilder` stub in `scripts/notify_backup.rb` with: + +```ruby +class PayloadBuilder + VALID_STATUSES = %w[success failure skipped].freeze + + def initialize(options) + @options = options + end + + def build + validate! + { + timestamp: Time.now.utc.strftime('%Y-%m-%dT%H:%M:%SZ'), + machine: @options[:machine], + destination: @options[:destination], + status: @options[:status], + bytes_transferred: ByteFormatter.format(@options[:bytes]), + snapshot_id: @options[:snapshot], + log_path: @options[:log_path], + notes: @options[:notes] + }.compact + end + + private + + def validate! + return if VALID_STATUSES.include?(@options[:status]) + + raise ArgumentError, "status must be one of: #{VALID_STATUSES.join(', ')}" + end +end +``` + +- [ ] Run all tests: +```bash +ruby /home/jared/systems-admin/scripts/notify_backup_test.rb +``` +Expected: `12 runs, 12+ assertions, 0 failures, 0 errors` + +- [ ] Commit: +```bash +cd /home/jared/systems-admin +git add scripts/ +git commit -m "feat(backup-monitor): implement PayloadBuilder with tests" +``` + +--- + +## Task 4: WebhookClient + +Dependencies are injected so the HTTP layer can be stubbed in tests without making real network calls. + +**Files:** +- Modify: `scripts/notify_backup.rb` — implement `WebhookClient` +- Modify: `scripts/notify_backup_test.rb` — add `WebhookClientTest` + two stub helpers + +- [ ] Add the failing tests and helpers to `scripts/notify_backup_test.rb`: + +```ruby +# Stub HTTP adapter for WebhookClient tests — replaces Net::HTTP entirely. +# Sandi Metz: test the interface, inject the dependency, don't touch globals. +class FakeHttpAdapter + def initialize(response) + @response = response + end + + def start(_host, _port, use_ssl: false) + yield FakeHttpSession.new(@response) + end +end + +class FakeHttpSession + def initialize(response) + @response = response + end + + def request(_req) + @response + end +end + +class WebhookClientTest < Minitest::Test + def make_success_response + res = Net::HTTPSuccess.new('1.1', '200', 'OK') + res.instance_variable_set(:@read, true) + res + end + + def make_error_response + res = Net::HTTPServerError.new('1.1', '500', 'Internal Server Error') + res.instance_variable_set(:@body, 'something broke') + res.instance_variable_set(:@read, true) + res + end + + def make_client(response) + WebhookClient.new( + base_url: 'https://n8n.example.com', + auth_token: 'test-token', + http_adapter: FakeHttpAdapter.new(response) + ) + end + + def test_post_succeeds_on_200 + client = make_client(make_success_response) + client.post({ machine: 'desktop', status: 'success' }) # no error raised + end + + def test_post_raises_on_500 + client = make_client(make_error_response) + err = assert_raises(RuntimeError) { client.post({}) } + assert_match(/HTTP 500/, err.message) + end +end +``` + +- [ ] Run to confirm failures: +```bash +ruby /home/jared/systems-admin/scripts/notify_backup_test.rb +``` +Expected: `WebhookClientTest` failures + +- [ ] Replace the empty `WebhookClient` stub in `scripts/notify_backup.rb` with: + +```ruby +class WebhookClient + def initialize(base_url:, auth_token:, http_adapter: Net::HTTP) + @base_url = base_url + @auth_token = auth_token + @http_adapter = http_adapter + end + + def post(payload) + uri = URI("#{@base_url}/webhook/backup-notify") + request = build_request(uri, payload) + + response = @http_adapter.start(uri.hostname, uri.port, use_ssl: uri.scheme == 'https') do |http| + http.request(request) + end + + raise "HTTP #{response.code}: #{response.body}" unless response.is_a?(Net::HTTPSuccess) + end + + private + + def build_request(uri, payload) + req = Net::HTTP::Post.new(uri) + req['Content-Type'] = 'application/json' + req['Authorization'] = "Bearer #{@auth_token}" + req.body = JSON.generate(payload) + req + end +end +``` + +- [ ] Run all tests: +```bash +ruby /home/jared/systems-admin/scripts/notify_backup_test.rb +``` +Expected: all tests pass, 0 failures + +- [ ] Commit: +```bash +cd /home/jared/systems-admin +git add scripts/ +git commit -m "feat(backup-monitor): implement WebhookClient with injected HTTP adapter and tests" +``` + +--- + +## Task 5: BackupNotifier and CLI entry point + +**Files:** +- Modify: `scripts/notify_backup.rb` — implement `BackupNotifier` + CLI block +- Modify: `scripts/notify_backup_test.rb` — add `BackupNotifierTest` + +- [ ] Add the failing tests to `scripts/notify_backup_test.rb`: + +```ruby +class BackupNotifierTest < Minitest::Test + def base_options + { machine: 'desktop', destination: 'synology', status: 'success' } + end + + def test_notify_calls_client_with_a_hash_payload + fake_client = Minitest::Mock.new + fake_client.expect(:post, nil, [Hash]) + + BackupNotifier.new(options: base_options, client: fake_client).notify + + fake_client.verify + end + + def test_from_env_returns_a_backup_notifier + ENV['N8N_SWANSONCLOUD_URL'] = 'https://n8n.example.com' + ENV['N8N_SWANSONCLOUD_BACKUP_AUTH_TOKEN'] = 'test-token' + + notifier = BackupNotifier.from_env(base_options) + assert_instance_of BackupNotifier, notifier + ensure + ENV.delete('N8N_SWANSONCLOUD_URL') + ENV.delete('N8N_SWANSONCLOUD_BACKUP_AUTH_TOKEN') + end + + def test_from_env_raises_when_env_var_missing + ENV.delete('N8N_SWANSONCLOUD_URL') + assert_raises(KeyError) { BackupNotifier.from_env(base_options) } + end +end +``` + +- [ ] Run to confirm failures: +```bash +ruby /home/jared/systems-admin/scripts/notify_backup_test.rb +``` +Expected: `BackupNotifierTest` failures + +- [ ] Replace the empty `BackupNotifier` stub in `scripts/notify_backup.rb` with: + +```ruby +class BackupNotifier + def initialize(options:, client:) + @options = options + @client = client + end + + def self.from_env(options) + url = ENV.fetch('N8N_SWANSONCLOUD_URL') + token = ENV.fetch('N8N_SWANSONCLOUD_BACKUP_AUTH_TOKEN') + new(options: options, client: WebhookClient.new(base_url: url, auth_token: token)) + end + + def notify + payload = PayloadBuilder.new(@options).build + @client.post(payload) + end +end +``` + +- [ ] Replace the empty CLI block at the bottom of `scripts/notify_backup.rb` with: + +```ruby +if __FILE__ == $PROGRAM_NAME + options = {} + + OptionParser.new do |opts| + opts.banner = 'Usage: notify_backup.rb [options]' + opts.separator '' + opts.separator 'Required:' + opts.on('--machine NAME', 'Host identifier (e.g. desktop, vps-ovh-prod-01)') { |v| options[:machine] = v } + opts.on('--destination NAME', 'Backup destination (e.g. synology, backblaze-b2)') { |v| options[:destination] = v } + opts.on('--status STATUS', 'success | failure | skipped') { |v| options[:status] = v } + opts.separator '' + opts.separator 'Optional:' + opts.on('--bytes N', Integer, 'Raw bytes transferred (formatted before sending)') { |v| options[:bytes] = v } + opts.on('--snapshot ID', 'Snapshot ID (e.g. restic snapshot hash)') { |v| options[:snapshot] = v } + opts.on('--log-path PATH', 'Path to the backup log for this run') { |v| options[:log_path] = v } + opts.on('--notes TEXT', 'Free-form output: restic tail, skip reason, etc.') { |v| options[:notes] = v } + end.parse! + + %i[machine destination status].each do |required| + abort "Error: --#{required} is required. Run with --help for usage." unless options[required] + end + + begin + BackupNotifier.from_env(options).notify + rescue => e + warn "notify_backup: failed to send webhook: #{e.message}" + exit 0 + end +end +``` + +- [ ] Run all tests: +```bash +ruby /home/jared/systems-admin/scripts/notify_backup_test.rb +``` +Expected: all tests pass (roughly 16+ assertions, 0 failures) + +- [ ] Commit: +```bash +cd /home/jared/systems-admin +git add scripts/ +git commit -m "feat(backup-monitor): implement BackupNotifier and CLI entry point" +``` + +--- + +## Task 6: Deploy to `~/.local/bin/` and verify locally + +**Files:** +- Deploy: `~/.local/bin/notify_backup.rb` + +- [ ] Copy and make executable: +```bash +cp /home/jared/systems-admin/scripts/notify_backup.rb ~/.local/bin/notify_backup.rb +chmod +x ~/.local/bin/notify_backup.rb +``` + +- [ ] Verify `--help` output shows usage cleanly: +```bash +notify_backup.rb --help +``` +Expected: usage banner with Required/Optional sections visible + +- [ ] Verify missing required arg exits with a clear message: +```bash +notify_backup.rb --machine desktop --destination synology +``` +Expected: `Error: --status is required. Run with --help for usage.` + +- [ ] Verify missing ENV var exits 0 with warning (ENV vars not yet set in this test): +```bash +env -u N8N_SWANSONCLOUD_URL notify_backup.rb --machine desktop --destination synology --status success +``` +Expected: `notify_backup: failed to send webhook: key not found: "N8N_SWANSONCLOUD_URL"` and exit code 0: +```bash +echo $? # must print 0 +``` + +--- + +## Task 7: Create `backup_runs` datatable in n8n + +This is a UI task in n8n. Open https://n8n.swansoncloud.com. + +- [ ] In the left sidebar, click **"Variables"** (or **"Data"** depending on your n8n version). Look for a **"Tables"** section. + +- [ ] Create a new table named exactly: `backup_runs` + +- [ ] Add the following columns (use string type for all unless noted): + +| Column name | Type | +|---|---| +| `timestamp` | String | +| `machine` | String | +| `destination` | String | +| `status` | String | +| `bytes_transferred` | String | +| `snapshot_id` | String | +| `log_path` | String | +| `notes` | String | + +Note: n8n auto-assigns an `id` column. If the UI only lets you create columns at workflow time (not pre-create), skip this step — Workflow 1 will create them automatically on first insert. + +--- + +## Task 8: n8n Workflow 1 — Receiver + +Build this workflow in the n8n UI. Create a new workflow named **"Backup Monitor — Receiver"**. + +- [ ] **Store the auth token as an n8n credential first:** + - Go to **Credentials** → **New** + - Type: **Header Auth** + - Name: `backup-monitor-token` + - Name field: `Authorization` + - Value field: `Bearer ` + - Save + +- [ ] **Add node 1 — Webhook:** + - Node type: **Webhook** + - HTTP Method: `POST` + - Path: `backup-notify` + - Authentication: **Header Auth** → select `backup-monitor-token` + - Response Mode: **Using 'Respond to Webhook' Node** + - Save + +- [ ] **Add node 2 — Insert to datatable:** + - Node type: **n8n** (the built-in n8n node) + - Resource: **Database** + - Operation: **Insert** + - Table: `backup_runs` + - Map each field from the incoming JSON body: + - `timestamp` → `{{ $json.body.timestamp }}` + - `machine` → `{{ $json.body.machine }}` + - `destination` → `{{ $json.body.destination }}` + - `status` → `{{ $json.body.status }}` + - `bytes_transferred` → `{{ $json.body.bytes_transferred }}` + - `snapshot_id` → `{{ $json.body.snapshot_id }}` + - `log_path` → `{{ $json.body.log_path }}` + - `notes` → `{{ $json.body.notes }}` + - Connect from Webhook node + +- [ ] **Add node 3 — Respond to Webhook:** + - Node type: **Respond to Webhook** + - Response Code: `200` + - Response Body: `{"ok":true}` + - Connect from the Insert node + +- [ ] **Activate the workflow** (toggle in top-right) + +- [ ] **Test with a real webhook call** (ENV vars must be sourced): +```bash +source ~/.credentials +notify_backup.rb \ + --machine desktop \ + --destination test \ + --status success \ + --notes "smoke test $(date)" +``` +Expected: no error output from `notify_backup.rb` + +- [ ] **Verify the row appeared in the datatable:** In n8n, open the `backup_runs` table and confirm a row exists with `machine=desktop`, `destination=test`, `status=success`. + +- [ ] Delete the test row from the datatable before continuing. + +--- + +## Task 9: n8n Workflow 2 — Reporter + +Build this workflow in the n8n UI. Create a new workflow named **"Backup Monitor — Weekly Reporter"**. + +- [ ] **Store MailPace token as a credential:** + - Go to **Credentials** → **New** + - Type: **Header Auth** + - Name: `mailpace-token` + - Name field: `MailPace-Server-Token` + - Value field: your MailPace server API token (find it at app.mailpace.com → Server Settings) + - Save + +- [ ] **Add node 1 — Schedule Trigger:** + - Node type: **Schedule Trigger** + - Trigger interval: **Weeks** + - Day of week: **Friday** + - Hour: `8` + - Minute: `0` + +- [ ] **Add node 2 — Set: Expected Sources:** + - Node type: **Set** + - Mode: **Manual** + - Add one field: + - Name: `sources` + - Type: **Array** + - Value: + ```json + [ + {"machine": "desktop", "destination": "synology"}, + {"machine": "desktop", "destination": "backblaze-b2"}, + {"machine": "vps-ovh-prod-01", "destination": "local"} + ] + ``` + - Connect from Schedule Trigger + +- [ ] **Add node 3 — Read Datatable:** + - Node type: **n8n** + - Resource: **Database** + - Operation: **Get Many Rows** + - Table: `backup_runs` + - Filter: `timestamp` is greater than `{{ new Date(Date.now() - 7 * 24 * 60 * 60 * 1000).toISOString() }}` + - Connect from Set node + +- [ ] **Add node 4 — Evaluate Sources (Code node):** + - Node type: **Code** + - Language: **JavaScript** + - Connect from Read Datatable node + - Paste this code exactly: + +```javascript +const sources = $('Set: Expected Sources').first().json.sources; +const runs = $('Read Datatable').all().map(item => item.json); +const weekAgo = new Date(Date.now() - 7 * 24 * 60 * 60 * 1000); + +const results = sources.map(source => { + const sourceRuns = runs.filter(r => + r.machine === source.machine && + r.destination === source.destination && + new Date(r.timestamp) >= weekAgo + ); + + const hasSuccess = sourceRuns.some(r => r.status === 'success'); + const hasAnyRun = sourceRuns.length > 0; + + const status = hasSuccess ? 'green' : hasAnyRun ? 'yellow' : 'red'; + + const lastSuccess = sourceRuns + .filter(r => r.status === 'success') + .sort((a, b) => new Date(b.timestamp) - new Date(a.timestamp))[0] || null; + + return { ...source, status, runs: sourceRuns, lastSuccess }; +}); + +const overallStatus = results.some(r => r.status === 'red') ? 'red' + : results.some(r => r.status === 'yellow') ? 'yellow' + : 'green'; + +return [{ json: { results, overallStatus } }]; +``` + +- [ ] **Add node 5 — Build Email (Code node):** + - Node type: **Code** + - Language: **JavaScript** + - Connect from Evaluate Sources node + - Paste this code exactly: + +```javascript +const { results, overallStatus } = $input.first().json; + +const icon = { green: '✓', yellow: '⚠', red: '✗' }; +const label = { green: 'Backups healthy', yellow: 'Backup warning', red: 'Backup failure' }; + +const weekOf = new Date().toLocaleDateString('en-US', { month: 'short', day: 'numeric' }); +const subject = `${icon[overallStatus]} ${label[overallStatus]} — week of ${weekOf}`; + +const fmt = ts => ts ? new Date(ts).toLocaleString('en-US', { month: 'short', day: 'numeric', hour: '2-digit', minute: '2-digit' }) : 'unknown'; + +let body = ''; + +if (overallStatus === 'green') { + body += `All backup sources ran successfully this week.\n\n`; + body += results.map(r => + `✓ ${r.machine} / ${r.destination} (last success: ${fmt(r.lastSuccess?.timestamp)})` + ).join('\n'); + +} else { + const problems = results.filter(r => r.status !== 'green'); + const healthy = results.filter(r => r.status === 'green'); + + body += `${problems.length} source(s) need attention:\n\n`; + + for (const r of problems) { + body += `${icon[r.status]} ${r.machine} / ${r.destination}\n`; + + if (r.status === 'yellow') { + body += ` Ran this week but never succeeded.\n`; + const shown = r.runs.slice(0, 3); + for (const run of shown) { + body += ` ${fmt(run.timestamp)} [${run.status}]`; + if (run.log_path) body += ` log: ${run.log_path}`; + body += '\n'; + if (run.notes) body += ` notes: ${run.notes.slice(0, 300).replace(/\n/g, ' ')}\n`; + } + body += `\n Next steps:\n`; + body += ` 1. Check the log at the path shown above\n`; + body += ` 2. For Synology failures: verify LAN reachability (ssh synology) and give disks 30s to spin up\n`; + body += ` 3. Re-run manually: bash ~/.local/bin/backup.sh\n`; + body += ` 4. One-off skip (power outage, travel) is expected — yellow is a heads-up, not an alarm\n\n`; + } + + if (r.status === 'red') { + body += ` No runs reported all week — silent failure.\n`; + if (r.lastSuccess) { + body += ` Last known good run: ${fmt(r.lastSuccess.timestamp)}\n`; + if (r.lastSuccess.log_path) body += ` Last log: ${r.lastSuccess.log_path}\n`; + } else { + body += ` No history found in the datatable.\n`; + } + body += `\n Next steps:\n`; + body += ` 1. Check systemd timer: systemctl --user status restic-backup.timer\n`; + body += ` 2. Verify backup.sh calls notify_backup.rb at the end\n`; + body += ` 3. Check n8n Workflow 1 execution log for webhook errors\n`; + body += ` 4. Run manually: bash ~/.local/bin/backup.sh\n\n`; + } + } + + if (healthy.length > 0) { + body += `Healthy sources:\n`; + body += healthy.map(r => `✓ ${r.machine} / ${r.destination}`).join('\n'); + } +} + +return [{ json: { subject, body } }]; +``` + +- [ ] **Add node 6 — Send via MailPace (HTTP Request node):** + - Node type: **HTTP Request** + - Method: `POST` + - URL: `https://app.mailpace.com/api/v1/send` + - Authentication: **Predefined Credential Type** → **Header Auth** → `mailpace-token` + - Body Content Type: **JSON** + - Body parameters: + - `from`: `backups@swansoncloud.com` (or whichever sending address your MailPace server is configured for) + - `to`: `jaredmswanson@gmail.com` + - `subject`: `{{ $json.subject }}` + - `text_body`: `{{ $json.body }}` + - Connect from Build Email node + +- [ ] **Test the reporter by running it manually:** + - In n8n, open the workflow and click **"Test workflow"** (or **"Execute Workflow"**) + - Check that an email arrives at jaredmswanson@gmail.com + - Expect RED or YELLOW since no real backup data is in the table yet — that's correct + +- [ ] **Activate the workflow** (toggle in top-right) + +--- + +## Task 10: Integrate `notify_backup.rb` into desktop `~/.local/bin/backup.sh` + +The existing `backup.sh` uses `set -euo pipefail`. We add a helper function that temporarily disables `set -e` to capture the exit code of `run_backup`, then always re-enables it, then sends the notification. + +**File:** `~/.local/bin/backup.sh` + +Current structure for reference (key lines): +- Line 3: `set -euo pipefail` +- Line 16: `run_backup()` function definition +- Line 35: Synology conditional block +- Line 43–45: B2 unlock + run_backup + +- [ ] Add the `run_and_notify` helper function to `~/.local/bin/backup.sh` immediately after the existing `run_backup` function (after line ~32): + +```bash +run_and_notify() { + local repo="$1" + local label="$2" + local machine="$3" + local destination="$4" + + set +e + run_backup "$repo" "$label" + local exit_code=$? + set -e + + local status="failure" + [[ $exit_code -eq 0 ]] && status="success" + + local snap + snap=$(grep -oE 'snapshot [a-f0-9]+' "$LOG" 2>/dev/null | tail -1 | awk '{print $2}') + + notify_backup.rb \ + --machine "$machine" \ + --destination "$destination" \ + --status "$status" \ + ${snap:+--snapshot "$snap"} \ + --log-path "$LOG" \ + --notes "$(tail -10 "$LOG")" || true + + return $exit_code +} +``` + +- [ ] Replace the Synology block (lines ~35–40) with: + +```bash +# Synology: attempt only if reachable (SSH key auth, 30s timeout for HDD spin-up) +if ssh -o ConnectTimeout=30 -o BatchMode=yes synology true 2>/dev/null; then + run_and_notify "sftp:${SYNOLOGY_HOST}:${SYNOLOGY_PATH}" "Synology" "desktop" "synology" +else + echo "Synology unreachable — skipping SFTP backup" | tee -a "$LOG" + notify_backup.rb \ + --machine desktop \ + --destination synology \ + --status skipped \ + --log-path "$LOG" \ + --notes "Synology unreachable at $(date)" || true +fi +``` + +- [ ] Replace the B2 lines (~43–45) with: + +```bash +# B2: clear any stale lock before attempting (safe no-op if no lock exists) +restic -r "b2:${B2_BUCKET}:" unlock --remove-all 2>&1 | tee -a "$LOG" +run_and_notify "b2:${B2_BUCKET}:" "Backblaze B2" "desktop" "backblaze-b2" +``` + +- [ ] Verify the final file looks correct (no duplicate lines, indentation intact): +```bash +cat -n ~/.local/bin/backup.sh +``` + +- [ ] Run a dry-run to confirm the script is syntactically valid: +```bash +bash -n ~/.local/bin/backup.sh +``` +Expected: no output (no syntax errors) + +- [ ] Copy the updated backup.sh back to the source in this repo so it stays tracked: +```bash +cp ~/.local/bin/backup.sh /home/jared/systems-admin/scripts/backup.sh +``` + +- [ ] Commit: +```bash +cd /home/jared/systems-admin +git add scripts/backup.sh +git commit -m "feat(backup-monitor): integrate notify_backup.rb into desktop backup.sh" +``` + +--- + +## Task 11: Deploy and integrate on OVH VPS + +The VPS backup script is at `~/services/backup.sh` on `15.204.247.153`. The notifier script and ENV vars must be deployed there too. + +- [ ] **Check if Ruby is available on the VPS:** +```bash +ssh jared@15.204.247.153 'ruby --version' +``` +If Ruby is not installed: +```bash +ssh jared@15.204.247.153 'sudo apt-get install -y ruby' +``` + +- [ ] **Copy `notify_backup.rb` to the VPS:** +```bash +scp ~/.local/bin/notify_backup.rb jared@15.204.247.153:~/.local/bin/notify_backup.rb +ssh jared@15.204.247.153 'chmod +x ~/.local/bin/notify_backup.rb' +``` + +- [ ] **Add the ENV vars to the VPS credentials file.** SSH to the server and edit `~/.credentials`: +```bash +ssh jared@15.204.247.153 +``` +Then in the SSH session, append to `~/.credentials`: +```bash +export N8N_SWANSONCLOUD_URL="https://n8n.swansoncloud.com" +export N8N_SWANSONCLOUD_BACKUP_AUTH_TOKEN="" +``` +Then source it: +```bash +source ~/.credentials +``` + +- [ ] **Verify the notifier works from the VPS** (still in the SSH session): +```bash +notify_backup.rb --machine vps-ovh-prod-01 --destination local --status success --notes "manual test $(date)" +``` +Expected: no error output + +- [ ] **Verify the test row appeared** in the n8n `backup_runs` datatable, then delete it. + +- [ ] **Add the notify call to `~/services/backup.sh` on the VPS.** Open the file: +```bash +nano ~/services/backup.sh +``` +Find the final success line (likely an `echo` at the end). Before it, add: +```bash +source "$HOME/.credentials" +SNAP=$(grep -oE 'snapshot [a-f0-9]+' "${LOG_FILE}" 2>/dev/null | tail -1 | awk '{print $2}') +notify_backup.rb \ + --machine vps-ovh-prod-01 \ + --destination local \ + --status success \ + --log-path "${LOG_FILE}" \ + ${SNAP:+--snapshot "$SNAP"} \ + --notes "$(tail -10 "${LOG_FILE}")" || true +``` +Note: The VPS `backup.sh` uses `LOG_FILE` (not `LOG`) — confirm the variable name before saving. + +- [ ] Verify syntax: +```bash +bash -n ~/services/backup.sh +``` +Expected: no output + +- [ ] Exit the SSH session. Back on desktop, commit the documentation of this change: +```bash +cd /home/jared/systems-admin +git commit --allow-empty -m "feat(backup-monitor): deploy notify_backup.rb to OVH VPS and integrate into backup.sh" +``` + +--- + +## Task 12: End-to-end smoke test + +- [ ] **Trigger the desktop backup manually** and watch for webhook activity: +```bash +source ~/.credentials && bash ~/.local/bin/backup.sh +``` +Expected: backup runs as normal, no new errors in output + +- [ ] **Check the n8n `backup_runs` datatable** — two new rows should appear: one for `desktop/synology` (success or skipped) and one for `desktop/backblaze-b2` (success) + +- [ ] **Trigger the weekly reporter manually** in n8n (open Workflow 2 → Execute Workflow). With at least some runs in the table, you should receive an email at jaredmswanson@gmail.com. + +- [ ] **Verify the email format:** + - If all sources have at least one success: subject line starts with `✓` + - If any source has no success: subject starts with `⚠` or `✗` with detail in the body + +- [ ] **Remove any test rows** from `backup_runs` (rows with `destination=test` from Task 8) + +- [ ] Final commit: +```bash +cd /home/jared/systems-admin +git add -A +git commit -m "docs(backup-monitor): implementation complete — smoke test passed" +``` + +--- + +## Self-Review Notes + +- **Spec coverage:** Ruby script (Tasks 1–6) ✓ · Datatable (Task 7) ✓ · Workflow 1 (Task 8) ✓ · Workflow 2 (Task 9) ✓ · Desktop integration (Task 10) ✓ · VPS integration (Task 11) ✓ · Email format green/yellow/red (Task 9) ✓ · Auth token from ENV (Tasks 8, 10) ✓ · `log_path` field (Tasks 2–3, 10) ✓ · `skipped` status (Task 10) ✓ +- **`backup.sh` variable name:** The VPS script uses `LOG_FILE` not `LOG` — Task 11 calls this out explicitly. +- **MailPace `from` address:** Needs to match a verified sending domain in your MailPace account. Confirm before activating Workflow 2. +- **n8n node naming:** The Code node in Task 9 references `$('Set: Expected Sources')` by name — the Set node in step 2 must be named exactly `Set: Expected Sources` or update the reference to match.