33 KiB

Raw Blame History

n8n Backup Monitor Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Build a webhook-based backup health monitoring system: a reusable Ruby notifier script, two n8n workflows (receiver + weekly reporter), and integrations into the desktop and VPS backup scripts.

Architecture: notify_backup.rb is a standalone Ruby CLI script (no gems, stdlib only) that POSTs a structured JSON payload to an n8n webhook after each backup run. n8n Workflow 1 validates auth and records each run to a datatable. Every Friday at 8 AM, n8n Workflow 2 reads the week's data, compares it against a hardcoded expected-sources list, and emails a green/yellow/red health digest via MailPace.

Tech Stack: Ruby stdlib (net/http, json, optparse, minitest), n8n self-hosted at https://n8n.swansoncloud.com (Webhook, Schedule, Code, HTTP Request nodes, datatables), MailPace REST API, bash

File Map

File	Action	Responsibility
`scripts/notify_backup.rb`	Create	Source-of-record for the notifier script
`scripts/notify_backup_test.rb`	Create	Minitest suite for all four classes
`~/.local/bin/notify_backup.rb`	Deploy	Runtime location on desktop
`~/.local/bin/backup.sh`	Modify	Add notify calls after each restic run
`~/services/backup.sh` on OVH	Modify via SSH	Add notify calls after VPS backup run

n8n workflows are configured in the n8n UI — no local files, but each task below gives exact node settings.

Task 1: Script skeleton and test infrastructure

Files:

Create: scripts/notify_backup.rb
Create: scripts/notify_backup_test.rb
Create the scripts/ directory:

mkdir -p /home/jared/systems-admin/scripts

Create scripts/notify_backup.rb with the full header comment and empty class stubs. The if __FILE__ == $PROGRAM_NAME guard keeps the CLI from running during tests:

#!/usr/bin/env ruby
# frozen_string_literal: true

# notify_backup — send a backup run result to the n8n backup monitor
#
# USAGE
#   notify_backup.rb [options]
#
# REQUIRED OPTIONS
#   --machine NAME        Identifier for this host (e.g. desktop, vps-ovh-prod-01)
#   --destination NAME    Backup destination (e.g. synology, backblaze-b2, local)
#   --status STATUS       Result: success | failure | skipped
#
# OPTIONAL OPTIONS
#   --bytes N             Bytes transferred (raw integer; formatted to human-readable before sending)
#   --snapshot ID         Restic snapshot ID or equivalent
#   --log-path PATH       Path to the backup log file for this run
#   --notes TEXT          Free-form output (restic tail, skip reason, error message, etc.)
#
# REQUIRED ENV VARS
#   N8N_SWANSONCLOUD_URL                Base URL of the n8n instance (no trailing slash)
#                                       e.g. https://n8n.swansoncloud.com
#   N8N_SWANSONCLOUD_BACKUP_AUTH_TOKEN  Bearer token for webhook authentication
#
# EXIT BEHAVIOR
#   Always exits 0. A notification failure is logged to stderr but never causes
#   the calling backup process to fail — the backup ran; the log is the source of truth.
#
# ADDING A NEW BACKUP SOURCE
#   1. Ensure the two ENV vars above are available on the new host
#   2. Call this script at the end of the backup process with --machine, --destination, --status
#   3. Add the {machine, destination} pair to the Set: Expected Sources node in n8n Workflow 2
#
# CLASSES
#   ByteFormatter    raw integer bytes → human-readable string (e.g. "4.50 GB")
#   PayloadBuilder   converts CLI option hash into a clean JSON-ready hash
#   WebhookClient    sends HTTP POST with auth header; knows nothing about backup data
#   BackupNotifier   orchestrates: build payload → send; the only public entry point

require 'net/http'
require 'json'
require 'optparse'

class ByteFormatter
end

class PayloadBuilder
end

class WebhookClient
end

class BackupNotifier
end

if __FILE__ == $PROGRAM_NAME
  # CLI entry point — implemented in Task 5
end

Create scripts/notify_backup_test.rb:

#!/usr/bin/env ruby
# frozen_string_literal: true

require 'minitest/autorun'
require_relative 'notify_backup'

Run the test file to confirm the skeleton loads without errors:

ruby /home/jared/systems-admin/scripts/notify_backup_test.rb

Expected output: 0 runs, 0 assertions, 0 failures, 0 errors

Commit:

cd /home/jared/systems-admin
git add scripts/
git commit -m "feat(backup-monitor): add notify_backup script skeleton and test infrastructure"

Task 2: ByteFormatter

Files:

Modify: scripts/notify_backup.rb — implement ByteFormatter
Modify: scripts/notify_backup_test.rb — add ByteFormatterTest
Add the failing tests to scripts/notify_backup_test.rb:

class ByteFormatterTest < Minitest::Test
  def test_gigabytes
    assert_equal '4.50 GB', ByteFormatter.format(4_831_838_208)
  end

  def test_megabytes
    assert_equal '823.00 MB', ByteFormatter.format(863_076_352)
  end

  def test_kilobytes
    assert_equal '12.00 KB', ByteFormatter.format(12_288)
  end

  def test_bytes_below_one_kb
    assert_equal '512 B', ByteFormatter.format(512)
  end

  def test_nil_returns_nil
    assert_nil ByteFormatter.format(nil)
  end

  def test_string_integer_is_coerced
    assert_equal '1.00 GB', ByteFormatter.format('1073741824')
  end
end

Run to confirm 6 failures:

ruby /home/jared/systems-admin/scripts/notify_backup_test.rb

Expected: 6 failures (NoMethodError on ByteFormatter.format)

Replace the empty ByteFormatter class stub in scripts/notify_backup.rb with:

class ByteFormatter
  UNITS = [['GB', 1_073_741_824], ['MB', 1_048_576], ['KB', 1_024]].freeze

  def self.format(bytes)
    return nil if bytes.nil?

    bytes = bytes.to_i
    unit, divisor = UNITS.find { |_, d| bytes >= d }
    return "#{bytes} B" unless unit

    '%.2f %s' % [bytes.to_f / divisor, unit]
  end
end

Run tests to confirm they pass:

ruby /home/jared/systems-admin/scripts/notify_backup_test.rb

Expected: 6 runs, 6 assertions, 0 failures, 0 errors

Commit:

cd /home/jared/systems-admin
git add scripts/
git commit -m "feat(backup-monitor): implement ByteFormatter with tests"

Task 3: PayloadBuilder

Files:

Modify: scripts/notify_backup.rb — implement PayloadBuilder
Modify: scripts/notify_backup_test.rb — add PayloadBuilderTest
Add the failing tests to scripts/notify_backup_test.rb:

class PayloadBuilderTest < Minitest::Test
  def base_options
    { machine: 'desktop', destination: 'synology', status: 'success' }
  end

  def test_builds_required_fields
    payload = PayloadBuilder.new(base_options).build
    assert_equal 'desktop', payload[:machine]
    assert_equal 'synology', payload[:destination]
    assert_equal 'success', payload[:status]
    assert_match(/\A\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z\z/, payload[:timestamp])
  end

  def test_formats_bytes_to_human_readable
    payload = PayloadBuilder.new(base_options.merge(bytes: 1_073_741_824)).build
    assert_equal '1.00 GB', payload[:bytes_transferred]
  end

  def test_omits_nil_optional_fields
    payload = PayloadBuilder.new(base_options).build
    refute payload.key?(:bytes_transferred)
    refute payload.key?(:snapshot_id)
    refute payload.key?(:log_path)
    refute payload.key?(:notes)
  end

  def test_includes_all_optional_fields_when_present
    opts = base_options.merge(
      bytes: 1_048_576,
      snapshot: 'abc12345',
      log_path: '/home/jared/.local/log/restic/2026-05-07.log',
      notes: 'snapshot abc12345 saved'
    )
    payload = PayloadBuilder.new(opts).build
    assert_equal '1.00 MB',    payload[:bytes_transferred]
    assert_equal 'abc12345',   payload[:snapshot_id]
    assert_equal '/home/jared/.local/log/restic/2026-05-07.log', payload[:log_path]
    assert_equal 'snapshot abc12345 saved', payload[:notes]
  end

  def test_raises_on_invalid_status
    assert_raises(ArgumentError) { PayloadBuilder.new(base_options.merge(status: 'unknown')).build }
  end

  def test_all_valid_statuses_accepted
    %w[success failure skipped].each do |s|
      assert PayloadBuilder.new(base_options.merge(status: s)).build
    end
  end
end

Run to confirm failures:

ruby /home/jared/systems-admin/scripts/notify_backup_test.rb

Expected: PayloadBuilderTest failures, ByteFormatterTest still passing

Replace the empty PayloadBuilder stub in scripts/notify_backup.rb with:

class PayloadBuilder
  VALID_STATUSES = %w[success failure skipped].freeze

  def initialize(options)
    @options = options
  end

  def build
    validate!
    {
      timestamp:         Time.now.utc.strftime('%Y-%m-%dT%H:%M:%SZ'),
      machine:           @options[:machine],
      destination:       @options[:destination],
      status:            @options[:status],
      bytes_transferred: ByteFormatter.format(@options[:bytes]),
      snapshot_id:       @options[:snapshot],
      log_path:          @options[:log_path],
      notes:             @options[:notes]
    }.compact
  end

  private

  def validate!
    return if VALID_STATUSES.include?(@options[:status])

    raise ArgumentError, "status must be one of: #{VALID_STATUSES.join(', ')}"
  end
end

Run all tests:

ruby /home/jared/systems-admin/scripts/notify_backup_test.rb

Expected: 12 runs, 12+ assertions, 0 failures, 0 errors

Commit:

cd /home/jared/systems-admin
git add scripts/
git commit -m "feat(backup-monitor): implement PayloadBuilder with tests"

Task 4: WebhookClient

Dependencies are injected so the HTTP layer can be stubbed in tests without making real network calls.

Files:

Modify: scripts/notify_backup.rb — implement WebhookClient
Modify: scripts/notify_backup_test.rb — add WebhookClientTest + two stub helpers
Add the failing tests and helpers to scripts/notify_backup_test.rb:

# Stub HTTP adapter for WebhookClient tests — replaces Net::HTTP entirely.
# Sandi Metz: test the interface, inject the dependency, don't touch globals.
class FakeHttpAdapter
  def initialize(response)
    @response = response
  end

  def start(_host, _port, use_ssl: false)
    yield FakeHttpSession.new(@response)
  end
end

class FakeHttpSession
  def initialize(response)
    @response = response
  end

  def request(_req)
    @response
  end
end

class WebhookClientTest < Minitest::Test
  def make_success_response
    res = Net::HTTPSuccess.new('1.1', '200', 'OK')
    res.instance_variable_set(:@read, true)
    res
  end

  def make_error_response
    res = Net::HTTPServerError.new('1.1', '500', 'Internal Server Error')
    res.instance_variable_set(:@body, 'something broke')
    res.instance_variable_set(:@read, true)
    res
  end

  def make_client(response)
    WebhookClient.new(
      base_url:     'https://n8n.example.com',
      auth_token:   'test-token',
      http_adapter: FakeHttpAdapter.new(response)
    )
  end

  def test_post_succeeds_on_200
    client = make_client(make_success_response)
    client.post({ machine: 'desktop', status: 'success' })  # no error raised
  end

  def test_post_raises_on_500
    client = make_client(make_error_response)
    err = assert_raises(RuntimeError) { client.post({}) }
    assert_match(/HTTP 500/, err.message)
  end
end

Run to confirm failures:

ruby /home/jared/systems-admin/scripts/notify_backup_test.rb

Expected: WebhookClientTest failures

Replace the empty WebhookClient stub in scripts/notify_backup.rb with:

class WebhookClient
  def initialize(base_url:, auth_token:, http_adapter: Net::HTTP)
    @base_url     = base_url
    @auth_token   = auth_token
    @http_adapter = http_adapter
  end

  def post(payload)
    uri     = URI("#{@base_url}/webhook/backup-notify")
    request = build_request(uri, payload)

    response = @http_adapter.start(uri.hostname, uri.port, use_ssl: uri.scheme == 'https') do |http|
      http.request(request)
    end

    raise "HTTP #{response.code}: #{response.body}" unless response.is_a?(Net::HTTPSuccess)
  end

  private

  def build_request(uri, payload)
    req = Net::HTTP::Post.new(uri)
    req['Content-Type']  = 'application/json'
    req['Authorization'] = "Bearer #{@auth_token}"
    req.body             = JSON.generate(payload)
    req
  end
end

Run all tests:

ruby /home/jared/systems-admin/scripts/notify_backup_test.rb

Expected: all tests pass, 0 failures

Commit:

cd /home/jared/systems-admin
git add scripts/
git commit -m "feat(backup-monitor): implement WebhookClient with injected HTTP adapter and tests"

Task 5: BackupNotifier and CLI entry point

Files:

Modify: scripts/notify_backup.rb — implement BackupNotifier + CLI block
Modify: scripts/notify_backup_test.rb — add BackupNotifierTest
Add the failing tests to scripts/notify_backup_test.rb:

class BackupNotifierTest < Minitest::Test
  def base_options
    { machine: 'desktop', destination: 'synology', status: 'success' }
  end

  def test_notify_calls_client_with_a_hash_payload
    fake_client = Minitest::Mock.new
    fake_client.expect(:post, nil, [Hash])

    BackupNotifier.new(options: base_options, client: fake_client).notify

    fake_client.verify
  end

  def test_from_env_returns_a_backup_notifier
    ENV['N8N_SWANSONCLOUD_URL']                = 'https://n8n.example.com'
    ENV['N8N_SWANSONCLOUD_BACKUP_AUTH_TOKEN']  = 'test-token'

    notifier = BackupNotifier.from_env(base_options)
    assert_instance_of BackupNotifier, notifier
  ensure
    ENV.delete('N8N_SWANSONCLOUD_URL')
    ENV.delete('N8N_SWANSONCLOUD_BACKUP_AUTH_TOKEN')
  end

  def test_from_env_raises_when_env_var_missing
    ENV.delete('N8N_SWANSONCLOUD_URL')
    assert_raises(KeyError) { BackupNotifier.from_env(base_options) }
  end
end

Run to confirm failures:

ruby /home/jared/systems-admin/scripts/notify_backup_test.rb

Expected: BackupNotifierTest failures

Replace the empty BackupNotifier stub in scripts/notify_backup.rb with:

class BackupNotifier
  def initialize(options:, client:)
    @options = options
    @client  = client
  end

  def self.from_env(options)
    url   = ENV.fetch('N8N_SWANSONCLOUD_URL')
    token = ENV.fetch('N8N_SWANSONCLOUD_BACKUP_AUTH_TOKEN')
    new(options: options, client: WebhookClient.new(base_url: url, auth_token: token))
  end

  def notify
    payload = PayloadBuilder.new(@options).build
    @client.post(payload)
  end
end

Replace the empty CLI block at the bottom of scripts/notify_backup.rb with:

if __FILE__ == $PROGRAM_NAME
  options = {}

  OptionParser.new do |opts|
    opts.banner = 'Usage: notify_backup.rb [options]'
    opts.separator ''
    opts.separator 'Required:'
    opts.on('--machine NAME',     'Host identifier (e.g. desktop, vps-ovh-prod-01)')   { |v| options[:machine] = v }
    opts.on('--destination NAME', 'Backup destination (e.g. synology, backblaze-b2)')  { |v| options[:destination] = v }
    opts.on('--status STATUS',    'success | failure | skipped')                        { |v| options[:status] = v }
    opts.separator ''
    opts.separator 'Optional:'
    opts.on('--bytes N',     Integer, 'Raw bytes transferred (formatted before sending)')  { |v| options[:bytes] = v }
    opts.on('--snapshot ID',         'Snapshot ID (e.g. restic snapshot hash)')            { |v| options[:snapshot] = v }
    opts.on('--log-path PATH',       'Path to the backup log for this run')                { |v| options[:log_path] = v }
    opts.on('--notes TEXT',          'Free-form output: restic tail, skip reason, etc.')   { |v| options[:notes] = v }
  end.parse!

  %i[machine destination status].each do |required|
    abort "Error: --#{required} is required. Run with --help for usage." unless options[required]
  end

  begin
    BackupNotifier.from_env(options).notify
  rescue => e
    warn "notify_backup: failed to send webhook: #{e.message}"
    exit 0
  end
end

Run all tests:

ruby /home/jared/systems-admin/scripts/notify_backup_test.rb

Expected: all tests pass (roughly 16+ assertions, 0 failures)

Commit:

cd /home/jared/systems-admin
git add scripts/
git commit -m "feat(backup-monitor): implement BackupNotifier and CLI entry point"

Task 6: Deploy to `~/.local/bin/` and verify locally

Files:

Deploy: ~/.local/bin/notify_backup.rb
Copy and make executable:

cp /home/jared/systems-admin/scripts/notify_backup.rb ~/.local/bin/notify_backup.rb
chmod +x ~/.local/bin/notify_backup.rb

Verify --help output shows usage cleanly:

notify_backup.rb --help

Expected: usage banner with Required/Optional sections visible

Verify missing required arg exits with a clear message:

notify_backup.rb --machine desktop --destination synology

Expected: Error: --status is required. Run with --help for usage.

Verify missing ENV var exits 0 with warning (ENV vars not yet set in this test):

env -u N8N_SWANSONCLOUD_URL notify_backup.rb --machine desktop --destination synology --status success

Expected: notify_backup: failed to send webhook: key not found: "N8N_SWANSONCLOUD_URL" and exit code 0:

echo $?   # must print 0

Task 7: Create `backup_runs` datatable in n8n

This is a UI task in n8n. Open https://n8n.swansoncloud.com.

In the left sidebar, click "Variables" (or "Data" depending on your n8n version). Look for a "Tables" section.
Create a new table named exactly: backup_runs
Add the following columns (use string type for all unless noted):

Column name	Type
`timestamp`	String
`machine`	String
`destination`	String
`status`	String
`bytes_transferred`	String
`snapshot_id`	String
`log_path`	String
`notes`	String

Note: n8n auto-assigns an id column. If the UI only lets you create columns at workflow time (not pre-create), skip this step — Workflow 1 will create them automatically on first insert.

Task 8: n8n Workflow 1 — Receiver

Build this workflow in the n8n UI. Create a new workflow named "Backup Monitor — Receiver".

Store the auth token as an n8n credential first:
- Go to Credentials → New
- Type: Header Auth
- Name: backup-monitor-token
- Name field: Authorization
- Value field: Bearer <paste value of N8N_SWANSONCLOUD_BACKUP_AUTH_TOKEN from your terminal>
- Save
Add node 1 — Webhook:
- Node type: Webhook
- HTTP Method: POST
- Path: backup-notify
- Authentication: Header Auth → select backup-monitor-token
- Response Mode: Using 'Respond to Webhook' Node
- Save
Add node 2 — Insert to datatable:
- Node type: n8n (the built-in n8n node)
- Resource: Database
- Operation: Insert
- Table: backup_runs
- Map each field from the incoming JSON body:
  - timestamp → {{ $json.body.timestamp }}
  - machine → {{ $json.body.machine }}
  - destination → {{ $json.body.destination }}
  - status → {{ $json.body.status }}
  - bytes_transferred → {{ $json.body.bytes_transferred }}
  - snapshot_id → {{ $json.body.snapshot_id }}
  - log_path → {{ $json.body.log_path }}
  - notes → {{ $json.body.notes }}
- Connect from Webhook node
Add node 3 — Respond to Webhook:
- Node type: Respond to Webhook
- Response Code: 200
- Response Body: {"ok":true}
- Connect from the Insert node
Activate the workflow (toggle in top-right)
Test with a real webhook call (ENV vars must be sourced):

source ~/.credentials
notify_backup.rb \
  --machine desktop \
  --destination test \
  --status success \
  --notes "smoke test $(date)"

Expected: no error output from notify_backup.rb

Verify the row appeared in the datatable: In n8n, open the backup_runs table and confirm a row exists with machine=desktop, destination=test, status=success.
Delete the test row from the datatable before continuing.

Task 9: n8n Workflow 2 — Reporter

Build this workflow in the n8n UI. Create a new workflow named "Backup Monitor — Weekly Reporter".

Store MailPace token as a credential:
- Go to Credentials → New
- Type: Header Auth
- Name: mailpace-token
- Name field: MailPace-Server-Token
- Value field: your MailPace server API token (find it at app.mailpace.com → Server Settings)
- Save
Add node 1 — Schedule Trigger:
- Node type: Schedule Trigger
- Trigger interval: Weeks
- Day of week: Friday
- Hour: 8
- Minute: 0

Add node 2 — Set: Expected Sources:

Node type: Set
Mode: Manual

Add one field:

Name: sources
Type: Array
Value:

[
  {"machine": "desktop", "destination": "synology"},
  {"machine": "desktop", "destination": "backblaze-b2"},
  {"machine": "vps-ovh-prod-01", "destination": "local"}
]

Connect from Schedule Trigger

Add node 3 — Read Datatable:
- Node type: n8n
- Resource: Database
- Operation: Get Many Rows
- Table: backup_runs
- Filter: timestamp is greater than {{ new Date(Date.now() - 7 * 24 * 60 * 60 * 1000).toISOString() }}
- Connect from Set node
Add node 4 — Evaluate Sources (Code node):
- Node type: Code
- Language: JavaScript
- Connect from Read Datatable node
- Paste this code exactly:

const sources = $('Set: Expected Sources').first().json.sources;
const runs = $('Read Datatable').all().map(item => item.json);
const weekAgo = new Date(Date.now() - 7 * 24 * 60 * 60 * 1000);

const results = sources.map(source => {
  const sourceRuns = runs.filter(r =>
    r.machine === source.machine &&
    r.destination === source.destination &&
    new Date(r.timestamp) >= weekAgo
  );

  const hasSuccess = sourceRuns.some(r => r.status === 'success');
  const hasAnyRun  = sourceRuns.length > 0;

  const status = hasSuccess ? 'green' : hasAnyRun ? 'yellow' : 'red';

  const lastSuccess = sourceRuns
    .filter(r => r.status === 'success')
    .sort((a, b) => new Date(b.timestamp) - new Date(a.timestamp))[0] || null;

  return { ...source, status, runs: sourceRuns, lastSuccess };
});

const overallStatus = results.some(r => r.status === 'red')    ? 'red'
                    : results.some(r => r.status === 'yellow')  ? 'yellow'
                    : 'green';

return [{ json: { results, overallStatus } }];

Add node 5 — Build Email (Code node):
- Node type: Code
- Language: JavaScript
- Connect from Evaluate Sources node
- Paste this code exactly:

const { results, overallStatus } = $input.first().json;

const icon  = { green: '✓', yellow: '⚠', red: '✗' };
const label = { green: 'Backups healthy', yellow: 'Backup warning', red: 'Backup failure' };

const weekOf = new Date().toLocaleDateString('en-US', { month: 'short', day: 'numeric' });
const subject = `${icon[overallStatus]} ${label[overallStatus]} — week of ${weekOf}`;

const fmt = ts => ts ? new Date(ts).toLocaleString('en-US', { month: 'short', day: 'numeric', hour: '2-digit', minute: '2-digit' }) : 'unknown';

let body = '';

if (overallStatus === 'green') {
  body += `All backup sources ran successfully this week.\n\n`;
  body += results.map(r =>
    `✓  ${r.machine} / ${r.destination}  (last success: ${fmt(r.lastSuccess?.timestamp)})`
  ).join('\n');

} else {
  const problems = results.filter(r => r.status !== 'green');
  const healthy  = results.filter(r => r.status === 'green');

  body += `${problems.length} source(s) need attention:\n\n`;

  for (const r of problems) {
    body += `${icon[r.status]}  ${r.machine} / ${r.destination}\n`;

    if (r.status === 'yellow') {
      body += `   Ran this week but never succeeded.\n`;
      const shown = r.runs.slice(0, 3);
      for (const run of shown) {
        body += `   ${fmt(run.timestamp)}  [${run.status}]`;
        if (run.log_path) body += `  log: ${run.log_path}`;
        body += '\n';
        if (run.notes) body += `   notes: ${run.notes.slice(0, 300).replace(/\n/g, ' ')}\n`;
      }
      body += `\n   Next steps:\n`;
      body += `   1. Check the log at the path shown above\n`;
      body += `   2. For Synology failures: verify LAN reachability (ssh synology) and give disks 30s to spin up\n`;
      body += `   3. Re-run manually:  bash ~/.local/bin/backup.sh\n`;
      body += `   4. One-off skip (power outage, travel) is expected — yellow is a heads-up, not an alarm\n\n`;
    }

    if (r.status === 'red') {
      body += `   No runs reported all week — silent failure.\n`;
      if (r.lastSuccess) {
        body += `   Last known good run: ${fmt(r.lastSuccess.timestamp)}\n`;
        if (r.lastSuccess.log_path) body += `   Last log: ${r.lastSuccess.log_path}\n`;
      } else {
        body += `   No history found in the datatable.\n`;
      }
      body += `\n   Next steps:\n`;
      body += `   1. Check systemd timer:  systemctl --user status restic-backup.timer\n`;
      body += `   2. Verify backup.sh calls notify_backup.rb at the end\n`;
      body += `   3. Check n8n Workflow 1 execution log for webhook errors\n`;
      body += `   4. Run manually:  bash ~/.local/bin/backup.sh\n\n`;
    }
  }

  if (healthy.length > 0) {
    body += `Healthy sources:\n`;
    body += healthy.map(r => `✓  ${r.machine} / ${r.destination}`).join('\n');
  }
}

return [{ json: { subject, body } }];

Add node 6 — Send via MailPace (HTTP Request node):
- Node type: HTTP Request
- Method: POST
- URL: https://app.mailpace.com/api/v1/send
- Authentication: Predefined Credential Type → Header Auth → mailpace-token
- Body Content Type: JSON
- Body parameters:
  - from: backups@swansoncloud.com (or whichever sending address your MailPace server is configured for)
  - to: jaredmswanson@gmail.com
  - subject: {{ $json.subject }}
  - text_body: {{ $json.body }}
- Connect from Build Email node
Test the reporter by running it manually:
- In n8n, open the workflow and click "Test workflow" (or "Execute Workflow")
- Check that an email arrives at jaredmswanson@gmail.com
- Expect RED or YELLOW since no real backup data is in the table yet — that's correct
Activate the workflow (toggle in top-right)

Task 10: Integrate `notify_backup.rb` into desktop `~/.local/bin/backup.sh`

The existing backup.sh uses set -euo pipefail. We add a helper function that temporarily disables set -e to capture the exit code of run_backup, then always re-enables it, then sends the notification.

File: ~/.local/bin/backup.sh

Current structure for reference (key lines):

Line 3: set -euo pipefail
Line 16: run_backup() function definition
Line 35: Synology conditional block
Line 43–45: B2 unlock + run_backup
Add the run_and_notify helper function to ~/.local/bin/backup.sh immediately after the existing run_backup function (after line ~32):

run_and_notify() {
  local repo="$1"
  local label="$2"
  local machine="$3"
  local destination="$4"

  set +e
  run_backup "$repo" "$label"
  local exit_code=$?
  set -e

  local status="failure"
  [[ $exit_code -eq 0 ]] && status="success"

  local snap
  snap=$(grep -oE 'snapshot [a-f0-9]+' "$LOG" 2>/dev/null | tail -1 | awk '{print $2}')

  notify_backup.rb \
    --machine "$machine" \
    --destination "$destination" \
    --status "$status" \
    ${snap:+--snapshot "$snap"} \
    --log-path "$LOG" \
    --notes "$(tail -10 "$LOG")" || true

  return $exit_code
}

Replace the Synology block (lines ~35–40) with:

# Synology: attempt only if reachable (SSH key auth, 30s timeout for HDD spin-up)
if ssh -o ConnectTimeout=30 -o BatchMode=yes synology true 2>/dev/null; then
  run_and_notify "sftp:${SYNOLOGY_HOST}:${SYNOLOGY_PATH}" "Synology" "desktop" "synology"
else
  echo "Synology unreachable — skipping SFTP backup" | tee -a "$LOG"
  notify_backup.rb \
    --machine desktop \
    --destination synology \
    --status skipped \
    --log-path "$LOG" \
    --notes "Synology unreachable at $(date)" || true
fi

Replace the B2 lines (~43–45) with:

# B2: clear any stale lock before attempting (safe no-op if no lock exists)
restic -r "b2:${B2_BUCKET}:" unlock --remove-all 2>&1 | tee -a "$LOG"
run_and_notify "b2:${B2_BUCKET}:" "Backblaze B2" "desktop" "backblaze-b2"

Verify the final file looks correct (no duplicate lines, indentation intact):

cat -n ~/.local/bin/backup.sh

Run a dry-run to confirm the script is syntactically valid:

bash -n ~/.local/bin/backup.sh

Expected: no output (no syntax errors)

Copy the updated backup.sh back to the source in this repo so it stays tracked:

cp ~/.local/bin/backup.sh /home/jared/systems-admin/scripts/backup.sh

Commit:

cd /home/jared/systems-admin
git add scripts/backup.sh
git commit -m "feat(backup-monitor): integrate notify_backup.rb into desktop backup.sh"

Task 11: Deploy and integrate on OVH VPS

The VPS backup script is at ~/services/backup.sh on 15.204.247.153. The notifier script and ENV vars must be deployed there too.

Check if Ruby is available on the VPS:

ssh jared@15.204.247.153 'ruby --version'

If Ruby is not installed:

ssh jared@15.204.247.153 'sudo apt-get install -y ruby'

Copy notify_backup.rb to the VPS:

scp ~/.local/bin/notify_backup.rb jared@15.204.247.153:~/.local/bin/notify_backup.rb
ssh jared@15.204.247.153 'chmod +x ~/.local/bin/notify_backup.rb'

Add the ENV vars to the VPS credentials file. SSH to the server and edit ~/.credentials:

ssh jared@15.204.247.153

Then in the SSH session, append to ~/.credentials:

export N8N_SWANSONCLOUD_URL="https://n8n.swansoncloud.com"
export N8N_SWANSONCLOUD_BACKUP_AUTH_TOKEN="<same token value as on desktop>"

Then source it:

source ~/.credentials

Verify the notifier works from the VPS (still in the SSH session):

notify_backup.rb --machine vps-ovh-prod-01 --destination local --status success --notes "manual test $(date)"

Expected: no error output

Verify the test row appeared in the n8n backup_runs datatable, then delete it.
Add the notify call to ~/services/backup.sh on the VPS. Open the file:

nano ~/services/backup.sh

Find the final success line (likely an echo at the end). Before it, add:

source "$HOME/.credentials"
SNAP=$(grep -oE 'snapshot [a-f0-9]+' "${LOG_FILE}" 2>/dev/null | tail -1 | awk '{print $2}')
notify_backup.rb \
  --machine vps-ovh-prod-01 \
  --destination local \
  --status success \
  --log-path "${LOG_FILE}" \
  ${SNAP:+--snapshot "$SNAP"} \
  --notes "$(tail -10 "${LOG_FILE}")" || true

Note: The VPS backup.sh uses LOG_FILE (not LOG) — confirm the variable name before saving.

Verify syntax:

bash -n ~/services/backup.sh

Expected: no output

Exit the SSH session. Back on desktop, commit the documentation of this change:

cd /home/jared/systems-admin
git commit --allow-empty -m "feat(backup-monitor): deploy notify_backup.rb to OVH VPS and integrate into backup.sh"

Task 12: End-to-end smoke test

Trigger the desktop backup manually and watch for webhook activity:

source ~/.credentials && bash ~/.local/bin/backup.sh

Expected: backup runs as normal, no new errors in output

Check the n8n backup_runs datatable — two new rows should appear: one for desktop/synology (success or skipped) and one for desktop/backblaze-b2 (success)
Trigger the weekly reporter manually in n8n (open Workflow 2 → Execute Workflow). With at least some runs in the table, you should receive an email at jaredmswanson@gmail.com.
Verify the email format:
- If all sources have at least one success: subject line starts with ✓
- If any source has no success: subject starts with ⚠ or ✗ with detail in the body
Remove any test rows from backup_runs (rows with destination=test from Task 8)
Final commit:

cd /home/jared/systems-admin
git add -A
git commit -m "docs(backup-monitor): implementation complete — smoke test passed"

Self-Review Notes

Spec coverage: Ruby script (Tasks 1–6) ✓ · Datatable (Task 7) ✓ · Workflow 1 (Task 8) ✓ · Workflow 2 (Task 9) ✓ · Desktop integration (Task 10) ✓ · VPS integration (Task 11) ✓ · Email format green/yellow/red (Task 9) ✓ · Auth token from ENV (Tasks 8, 10) ✓ · log_path field (Tasks 2–3, 10) ✓ · skipped status (Task 10) ✓
backup.sh variable name: The VPS script uses LOG_FILE not LOG — Task 11 calls this out explicitly.
MailPace from address: Needs to match a verified sending domain in your MailPace account. Confirm before activating Workflow 2.
n8n node naming: The Code node in Task 9 references $('Set: Expected Sources') by name — the Set node in step 2 must be named exactly Set: Expected Sources or update the reference to match.

33 KiB Raw Blame History Unescape Escape

n8n Backup Monitor Implementation Plan

File Map

Task 1: Script skeleton and test infrastructure

Task 2: ByteFormatter

Task 3: PayloadBuilder

Task 4: WebhookClient

Task 5: BackupNotifier and CLI entry point

Task 6: Deploy to ~/.local/bin/ and verify locally

Task 7: Create backup_runs datatable in n8n

Task 8: n8n Workflow 1 — Receiver

Task 9: n8n Workflow 2 — Reporter

Task 10: Integrate notify_backup.rb into desktop ~/.local/bin/backup.sh

Task 11: Deploy and integrate on OVH VPS

Task 12: End-to-end smoke test

Self-Review Notes

33 KiB

Raw Blame History

Task 6: Deploy to `~/.local/bin/` and verify locally

Task 7: Create `backup_runs` datatable in n8n

Task 10: Integrate `notify_backup.rb` into desktop `~/.local/bin/backup.sh`