From 5fcc7fb5156714b1d909c46f637cb680e8035396 Mon Sep 17 00:00:00 2001 From: jared Date: Thu, 7 May 2026 10:59:20 -0400 Subject: [PATCH] docs(backup-monitor): replace UI-click workflow tasks with JSON file approach MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add JS code files + Ruby build script + Python validator to file map - Replace Tasks 7-9 (UI clicks) with Tasks 7-10: datatable bootstrap, validator script, build/validate workflow JSON, API import with credentials - Fix MailPace credential header: X-Server-Token (not MailPace-Server-Token) - Renumber old Tasks 10-12 → 11-13 - Update self-review notes with data table uncertainty and node naming Co-Authored-By: Claude Sonnet 4.6 --- .../plans/2026-05-07-n8n-backup-monitor.md | 828 +++++++++++++----- 1 file changed, 631 insertions(+), 197 deletions(-) diff --git a/docs/superpowers/plans/2026-05-07-n8n-backup-monitor.md b/docs/superpowers/plans/2026-05-07-n8n-backup-monitor.md index ec3e9e8..a07ec5e 100644 --- a/docs/superpowers/plans/2026-05-07-n8n-backup-monitor.md +++ b/docs/superpowers/plans/2026-05-07-n8n-backup-monitor.md @@ -16,12 +16,16 @@ |------|--------|----------------| | `scripts/notify_backup.rb` | Create | Source-of-record for the notifier script | | `scripts/notify_backup_test.rb` | Create | Minitest suite for all four classes | +| `scripts/n8n-evaluate-sources.js` | Create | JS code for the Evaluate Sources Code node | +| `scripts/n8n-build-email.js` | Create | JS code for the Build Email Code node | +| `scripts/build-n8n-workflows.rb` | Create | Assembles workflow JSON using `JSON.generate` (handles all JS escaping) | +| `scripts/n8n-workflow-receiver.json` | Generated | Importable Workflow 1 JSON | +| `scripts/n8n-workflow-reporter.json` | Generated | Importable Workflow 2 JSON | +| `scripts/validate-n8n-workflow.py` | Create | Structural validator for n8n workflow JSON files | | `~/.local/bin/notify_backup.rb` | Deploy | Runtime location on desktop | | `~/.local/bin/backup.sh` | Modify | Add notify calls after each restic run | | `~/services/backup.sh` on OVH | Modify via SSH | Add notify calls after VPS backup run | -n8n workflows are configured in the n8n UI — no local files, but each task below gives exact node settings. - --- ## Task 1: Script skeleton and test infrastructure @@ -588,144 +592,217 @@ echo $? # must print 0 --- -## Task 7: Create `backup_runs` datatable in n8n +## Task 7: Bootstrap — create datatable and verify Data Table node JSON shape -This is a UI task in n8n. Open https://n8n.swansoncloud.com. +Both research agents flagged the Data Table node (`n8n-nodes-base.dataTable`) as a recent addition with uncertain parameter structure. This task pins the exact shape before generating workflow JSON. -- [ ] In the left sidebar, click **"Variables"** (or **"Data"** depending on your n8n version). Look for a **"Tables"** section. +- [ ] Open https://n8n.swansoncloud.com. In the left sidebar find **"Tables"** (under Variables or Data). Create a new table named exactly `backup_runs`. Add string columns: `timestamp`, `machine`, `destination`, `status`, `bytes_transferred`, `snapshot_id`, `log_path`, `notes`. n8n auto-adds an `id` column. -- [ ] Create a new table named exactly: `backup_runs` +- [ ] Create a scratch workflow named `_datatable-shape-probe`: + - Add a **Manual Trigger** node + - Add a **Data Table** node, configure it: operation = **Insert**, table = `backup_runs`, map one dummy field + - Connect Manual Trigger → Data Table + - Add a second **Data Table** node: operation = **Get Many Rows**, table = `backup_runs` + - Connect the Insert node → Get Many Rows + - Save (do not activate) -- [ ] Add the following columns (use string type for all unless noted): +- [ ] Export the scratch workflow: click the `⋮` menu → **Download**. Save as `/tmp/datatable-probe.json`. -| Column name | Type | -|---|---| -| `timestamp` | String | -| `machine` | String | -| `destination` | String | -| `status` | String | -| `bytes_transferred` | String | -| `snapshot_id` | String | -| `log_path` | String | -| `notes` | String | - -Note: n8n auto-assigns an `id` column. If the UI only lets you create columns at workflow time (not pre-create), skip this step — Workflow 1 will create them automatically on first insert. - ---- - -## Task 8: n8n Workflow 1 — Receiver - -Build this workflow in the n8n UI. Create a new workflow named **"Backup Monitor — Receiver"**. - -- [ ] **Store the auth token as an n8n credential first:** - - Go to **Credentials** → **New** - - Type: **Header Auth** - - Name: `backup-monitor-token` - - Name field: `Authorization` - - Value field: `Bearer ` - - Save - -- [ ] **Add node 1 — Webhook:** - - Node type: **Webhook** - - HTTP Method: `POST` - - Path: `backup-notify` - - Authentication: **Header Auth** → select `backup-monitor-token` - - Response Mode: **Using 'Respond to Webhook' Node** - - Save - -- [ ] **Add node 2 — Insert to datatable:** - - Node type: **n8n** (the built-in n8n node) - - Resource: **Database** - - Operation: **Insert** - - Table: `backup_runs` - - Map each field from the incoming JSON body: - - `timestamp` → `{{ $json.body.timestamp }}` - - `machine` → `{{ $json.body.machine }}` - - `destination` → `{{ $json.body.destination }}` - - `status` → `{{ $json.body.status }}` - - `bytes_transferred` → `{{ $json.body.bytes_transferred }}` - - `snapshot_id` → `{{ $json.body.snapshot_id }}` - - `log_path` → `{{ $json.body.log_path }}` - - `notes` → `{{ $json.body.notes }}` - - Connect from Webhook node - -- [ ] **Add node 3 — Respond to Webhook:** - - Node type: **Respond to Webhook** - - Response Code: `200` - - Response Body: `{"ok":true}` - - Connect from the Insert node - -- [ ] **Activate the workflow** (toggle in top-right) - -- [ ] **Test with a real webhook call** (ENV vars must be sourced): +- [ ] Inspect the exported JSON to extract the exact node structure: ```bash -source ~/.credentials -notify_backup.rb \ - --machine desktop \ - --destination test \ - --status success \ - --notes "smoke test $(date)" +python3 -c " +import json, sys +wf = json.load(open('/tmp/datatable-probe.json')) +for n in wf['nodes']: + if 'dataTable' in n.get('type','') or 'dataStore' in n.get('type',''): + print(json.dumps(n, indent=2)) +" ``` -Expected: no error output from `notify_backup.rb` +Note the `type` string (likely `n8n-nodes-base.dataTable`), `typeVersion`, and the exact keys inside `parameters` (especially the property wrapping the table reference and the filter shape). -- [ ] **Verify the row appeared in the datatable:** In n8n, open the `backup_runs` table and confirm a row exists with `machine=desktop`, `destination=test`, `status=success`. +- [ ] Also note the datatable ID from the exported JSON (the `value` field inside the table picker). You will need this to replace `REPLACE_WITH_DATATABLE_ID` in Task 9. -- [ ] Delete the test row from the datatable before continuing. +- [ ] If the `type` string or `typeVersion` differ from `n8n-nodes-base.dataTable` / `1`, update the two constants at the top of `scripts/build-n8n-workflows.rb` (created in Task 9) before running it. + +- [ ] Delete the scratch workflow from n8n. --- -## Task 9: n8n Workflow 2 — Reporter +## Task 8: Write JSON validation script -Build this workflow in the n8n UI. Create a new workflow named **"Backup Monitor — Weekly Reporter"**. +**Files:** +- Create: `scripts/validate-n8n-workflow.py` -- [ ] **Store MailPace token as a credential:** - - Go to **Credentials** → **New** - - Type: **Header Auth** - - Name: `mailpace-token` - - Name field: `MailPace-Server-Token` - - Value field: your MailPace server API token (find it at app.mailpace.com → Server Settings) - - Save +- [ ] Create `scripts/validate-n8n-workflow.py` with this content (Python stdlib, no pip required): -- [ ] **Add node 1 — Schedule Trigger:** - - Node type: **Schedule Trigger** - - Trigger interval: **Weeks** - - Day of week: **Friday** - - Hour: `8` - - Minute: `0` +```python +#!/usr/bin/env python3 +"""Validate an n8n workflow JSON file for structural correctness. -- [ ] **Add node 2 — Set: Expected Sources:** - - Node type: **Set** - - Mode: **Manual** - - Add one field: - - Name: `sources` - - Type: **Array** - - Value: - ```json - [ - {"machine": "desktop", "destination": "synology"}, - {"machine": "desktop", "destination": "backblaze-b2"}, - {"machine": "vps-ovh-prod-01", "destination": "local"} - ] - ``` - - Connect from Schedule Trigger +Usage: validate-n8n-workflow.py path/to/workflow.json +Exit codes: 0 = valid, 1 = invalid, 2 = usage error. +""" +import json +import sys +from pathlib import Path -- [ ] **Add node 3 — Read Datatable:** - - Node type: **n8n** - - Resource: **Database** - - Operation: **Get Many Rows** - - Table: `backup_runs` - - Filter: `timestamp` is greater than `{{ new Date(Date.now() - 7 * 24 * 60 * 60 * 1000).toISOString() }}` - - Connect from Set node +KNOWN_MIN_TYPE_VERSIONS = { + "n8n-nodes-base.code": 2, + "n8n-nodes-base.httpRequest": 4.2, + "n8n-nodes-base.webhook": 2, + "n8n-nodes-base.set": 3.4, + "n8n-nodes-base.scheduleTrigger": 1.2, +} -- [ ] **Add node 4 — Evaluate Sources (Code node):** - - Node type: **Code** - - Language: **JavaScript** - - Connect from Read Datatable node - - Paste this code exactly: +REQUIRED_TOP = ["name", "nodes", "connections"] +REQUIRED_NODE = ["id", "name", "type", "typeVersion", "position", "parameters"] + + +def validate(workflow): + errors, warnings = [], [] + if not isinstance(workflow, dict): + return ["Root must be a JSON object."], [] + for key in REQUIRED_TOP: + if key not in workflow: + errors.append(f"Top-level: missing required field '{key}'.") + if errors: + return errors, warnings + if not isinstance(workflow["nodes"], list): + errors.append("Top-level 'nodes' must be an array.") + if not isinstance(workflow["connections"], dict): + errors.append("Top-level 'connections' must be an object.") + if errors: + return errors, warnings + + seen_names, seen_ids = set(), set() + for i, node in enumerate(workflow["nodes"]): + loc = f"nodes[{i}]" + if not isinstance(node, dict): + errors.append(f"{loc}: must be an object.") + continue + for key in REQUIRED_NODE: + if key not in node: + errors.append(f"{loc}: missing required field '{key}'.") + name = node.get("name") + if name: + loc = f"nodes[{i}] '{name}'" + if name in seen_names: + errors.append(f"{loc}: duplicate node name.") + seen_names.add(name) + nid = node.get("id") + if nid: + if nid in seen_ids: + errors.append(f"{loc}: duplicate node id '{nid}'.") + seen_ids.add(nid) + pos = node.get("position") + if pos is not None and not ( + isinstance(pos, list) and len(pos) == 2 + and all(isinstance(v, (int, float)) for v in pos) + ): + errors.append(f"{loc}: 'position' must be [number, number].") + if node.get("parameters") is not None and not isinstance(node["parameters"], dict): + errors.append(f"{loc}: 'parameters' must be an object.") + creds = node.get("credentials") + if isinstance(creds, dict): + for ctype, cred in creds.items(): + if not isinstance(cred, dict) or "name" not in cred: + errors.append(f"{loc}: credentials['{ctype}'] must have a 'name' field.") + ntype, tv = node.get("type"), node.get("typeVersion") + if isinstance(tv, (int, float)) and ntype in KNOWN_MIN_TYPE_VERSIONS: + min_tv = KNOWN_MIN_TYPE_VERSIONS[ntype] + if tv < min_tv: + warnings.append(f"{loc}: typeVersion {tv} for '{ntype}' is below recommended {min_tv}.") + + for src, outputs in workflow["connections"].items(): + if src not in seen_names: + errors.append(f"connections: source '{src}' does not match any node name.") + if not isinstance(outputs, dict): + errors.append(f"connections['{src}']: must be an object.") + continue + for out_type, out_list in outputs.items(): + if not isinstance(out_list, list): + continue + for oi, branch in enumerate(out_list): + if branch is None: + continue + if not isinstance(branch, list): + errors.append(f"connections['{src}']['{out_type}'][{oi}]: must be an array.") + continue + for ti, target in enumerate(branch): + if not isinstance(target, dict): + errors.append(f"connections['{src}']['{out_type}'][{oi}][{ti}]: must be an object.") + continue + for k in ("node", "type", "index"): + if k not in target: + errors.append(f"connections['{src}']['{out_type}'][{oi}][{ti}]: missing '{k}'.") + tnode = target.get("node") + if tnode and tnode not in seen_names: + errors.append(f"connections['{src}']['{out_type}'][{oi}][{ti}]: target '{tnode}' not found.") + return errors, warnings + + +def main(): + if len(sys.argv) != 2: + print("Usage: validate-n8n-workflow.py ", file=sys.stderr) + sys.exit(2) + path = Path(sys.argv[1]) + try: + workflow = json.loads(path.read_text()) + except FileNotFoundError: + print(f"File not found: {path}", file=sys.stderr) + sys.exit(2) + except json.JSONDecodeError as e: + print(f"Invalid JSON: {e}", file=sys.stderr) + sys.exit(1) + errors, warnings = validate(workflow) + for w in warnings: + print(f"WARN: {w}") + for e in errors: + print(f"ERROR: {e}", file=sys.stderr) + if errors: + print(f"\n{len(errors)} error(s), {len(warnings)} warning(s).", file=sys.stderr) + sys.exit(1) + print(f"OK — {len(workflow['nodes'])} nodes, {len(workflow['connections'])} connections, {len(warnings)} warning(s).") + + +if __name__ == "__main__": + main() +``` + +- [ ] Make it executable and test it on a known-bad payload: +```bash +chmod +x /home/jared/systems-admin/scripts/validate-n8n-workflow.py +echo '{"name":"test","nodes":[],"connections":{"missing-node":{"main":[[]]}}}' \ + | python3 /home/jared/systems-admin/scripts/validate-n8n-workflow.py /dev/stdin +``` +Expected: `ERROR: connections: source 'missing-node' does not match any node name.` + +- [ ] Commit: +```bash +cd /home/jared/systems-admin +git add scripts/validate-n8n-workflow.py +git commit -m "feat(backup-monitor): add n8n workflow JSON validator" +``` + +--- + +## Task 9: Write JS code files, build script, generate and validate workflow JSONs + +**Files:** +- Create: `scripts/n8n-evaluate-sources.js` +- Create: `scripts/n8n-build-email.js` +- Create: `scripts/build-n8n-workflows.rb` +- Generated: `scripts/n8n-workflow-receiver.json` +- Generated: `scripts/n8n-workflow-reporter.json` + +The build script uses Ruby's `JSON.generate` to handle all escaping automatically — no manual string escaping needed for the Code node JS. + +- [ ] Create `scripts/n8n-evaluate-sources.js`: ```javascript -const sources = $('Set: Expected Sources').first().json.sources; -const runs = $('Read Datatable').all().map(item => item.json); +// Runs once for all items. References upstream nodes by exact name. +const sources = $('Set sources').first().json.sources; +const runs = $('Get backup_runs (last 7d)').all().map(item => item.json); const weekAgo = new Date(Date.now() - 7 * 24 * 60 * 60 * 1000); const results = sources.map(source => { @@ -734,124 +811,479 @@ const results = sources.map(source => { r.destination === source.destination && new Date(r.timestamp) >= weekAgo ); - const hasSuccess = sourceRuns.some(r => r.status === 'success'); - const hasAnyRun = sourceRuns.length > 0; - + const hasAnyRun = sourceRuns.length > 0; const status = hasSuccess ? 'green' : hasAnyRun ? 'yellow' : 'red'; - const lastSuccess = sourceRuns .filter(r => r.status === 'success') .sort((a, b) => new Date(b.timestamp) - new Date(a.timestamp))[0] || null; - return { ...source, status, runs: sourceRuns, lastSuccess }; }); -const overallStatus = results.some(r => r.status === 'red') ? 'red' - : results.some(r => r.status === 'yellow') ? 'yellow' - : 'green'; +const overallStatus = results.some(r => r.status === 'red') ? 'red' + : results.some(r => r.status === 'yellow') ? 'yellow' : 'green'; return [{ json: { results, overallStatus } }]; ``` -- [ ] **Add node 5 — Build Email (Code node):** - - Node type: **Code** - - Language: **JavaScript** - - Connect from Evaluate Sources node - - Paste this code exactly: +- [ ] Create `scripts/n8n-build-email.js`: ```javascript +// Builds subject + body from Evaluate Sources output. +// Uses an array of lines joined at the end to avoid \n escape issues in JSON storage. const { results, overallStatus } = $input.first().json; - const icon = { green: '✓', yellow: '⚠', red: '✗' }; const label = { green: 'Backups healthy', yellow: 'Backup warning', red: 'Backup failure' }; - const weekOf = new Date().toLocaleDateString('en-US', { month: 'short', day: 'numeric' }); -const subject = `${icon[overallStatus]} ${label[overallStatus]} — week of ${weekOf}`; +const subject = icon[overallStatus] + ' ' + label[overallStatus] + ' — week of ' + weekOf; -const fmt = ts => ts ? new Date(ts).toLocaleString('en-US', { month: 'short', day: 'numeric', hour: '2-digit', minute: '2-digit' }) : 'unknown'; +const fmt = ts => ts + ? new Date(ts).toLocaleString('en-US', { month: 'short', day: 'numeric', hour: '2-digit', minute: '2-digit' }) + : 'unknown'; -let body = ''; +const L = []; if (overallStatus === 'green') { - body += `All backup sources ran successfully this week.\n\n`; - body += results.map(r => - `✓ ${r.machine} / ${r.destination} (last success: ${fmt(r.lastSuccess?.timestamp)})` - ).join('\n'); - + L.push('All backup sources ran successfully this week.'); + L.push(''); + results.forEach(r => + L.push('✓ ' + r.machine + ' / ' + r.destination + + ' (last success: ' + fmt(r.lastSuccess && r.lastSuccess.timestamp) + ')') + ); } else { const problems = results.filter(r => r.status !== 'green'); const healthy = results.filter(r => r.status === 'green'); - body += `${problems.length} source(s) need attention:\n\n`; + L.push(problems.length + ' source(s) need attention:'); + L.push(''); - for (const r of problems) { - body += `${icon[r.status]} ${r.machine} / ${r.destination}\n`; + problems.forEach(r => { + L.push(icon[r.status] + ' ' + r.machine + ' / ' + r.destination); if (r.status === 'yellow') { - body += ` Ran this week but never succeeded.\n`; - const shown = r.runs.slice(0, 3); - for (const run of shown) { - body += ` ${fmt(run.timestamp)} [${run.status}]`; - if (run.log_path) body += ` log: ${run.log_path}`; - body += '\n'; - if (run.notes) body += ` notes: ${run.notes.slice(0, 300).replace(/\n/g, ' ')}\n`; - } - body += `\n Next steps:\n`; - body += ` 1. Check the log at the path shown above\n`; - body += ` 2. For Synology failures: verify LAN reachability (ssh synology) and give disks 30s to spin up\n`; - body += ` 3. Re-run manually: bash ~/.local/bin/backup.sh\n`; - body += ` 4. One-off skip (power outage, travel) is expected — yellow is a heads-up, not an alarm\n\n`; + L.push(' Ran this week but never succeeded.'); + r.runs.slice(0, 3).forEach(run => { + var line = ' ' + fmt(run.timestamp) + ' [' + run.status + ']'; + if (run.log_path) line += ' log: ' + run.log_path; + L.push(line); + if (run.notes) L.push(' notes: ' + (run.notes || '').slice(0, 300)); + }); + L.push(''); + L.push(' Next steps:'); + L.push(' 1. Check the log at the path shown above'); + L.push(' 2. For Synology failures: verify LAN reachability (ssh synology) — disks need ~30s to spin up'); + L.push(' 3. Re-run manually: bash ~/.local/bin/backup.sh'); + L.push(' 4. One-off skip (power outage, travel) is expected — yellow is a heads-up, not an alarm'); + L.push(''); } if (r.status === 'red') { - body += ` No runs reported all week — silent failure.\n`; + L.push(' No runs reported all week — silent failure.'); if (r.lastSuccess) { - body += ` Last known good run: ${fmt(r.lastSuccess.timestamp)}\n`; - if (r.lastSuccess.log_path) body += ` Last log: ${r.lastSuccess.log_path}\n`; + L.push(' Last known good run: ' + fmt(r.lastSuccess.timestamp)); + if (r.lastSuccess.log_path) L.push(' Last log: ' + r.lastSuccess.log_path); } else { - body += ` No history found in the datatable.\n`; + L.push(' No history found in the datatable.'); } - body += `\n Next steps:\n`; - body += ` 1. Check systemd timer: systemctl --user status restic-backup.timer\n`; - body += ` 2. Verify backup.sh calls notify_backup.rb at the end\n`; - body += ` 3. Check n8n Workflow 1 execution log for webhook errors\n`; - body += ` 4. Run manually: bash ~/.local/bin/backup.sh\n\n`; + L.push(''); + L.push(' Next steps:'); + L.push(' 1. Check systemd timer: systemctl --user status restic-backup.timer'); + L.push(' 2. Verify backup.sh calls notify_backup.rb at the end'); + L.push(' 3. Check n8n Workflow 1 execution log for webhook errors'); + L.push(' 4. Run manually: bash ~/.local/bin/backup.sh'); + L.push(''); } - } + }); if (healthy.length > 0) { - body += `Healthy sources:\n`; - body += healthy.map(r => `✓ ${r.machine} / ${r.destination}`).join('\n'); + L.push('Healthy sources:'); + healthy.forEach(r => L.push('✓ ' + r.machine + ' / ' + r.destination)); } } -return [{ json: { subject, body } }]; +return [{ json: { subject, body: L.join('\n') } }]; ``` -- [ ] **Add node 6 — Send via MailPace (HTTP Request node):** - - Node type: **HTTP Request** - - Method: `POST` - - URL: `https://app.mailpace.com/api/v1/send` - - Authentication: **Predefined Credential Type** → **Header Auth** → `mailpace-token` - - Body Content Type: **JSON** - - Body parameters: - - `from`: `backups@swansoncloud.com` (or whichever sending address your MailPace server is configured for) - - `to`: `jaredmswanson@gmail.com` - - `subject`: `{{ $json.subject }}` - - `text_body`: `{{ $json.body }}` - - Connect from Build Email node +- [ ] Create `scripts/build-n8n-workflows.rb`: -- [ ] **Test the reporter by running it manually:** - - In n8n, open the workflow and click **"Test workflow"** (or **"Execute Workflow"**) - - Check that an email arrives at jaredmswanson@gmail.com - - Expect RED or YELLOW since no real backup data is in the table yet — that's correct +```ruby +#!/usr/bin/env ruby +# frozen_string_literal: true +# +# Generates importable n8n workflow JSON from component pieces. +# Run: ruby scripts/build-n8n-workflows.rb +# Output: scripts/n8n-workflow-receiver.json +# scripts/n8n-workflow-reporter.json +# +# Before running: +# 1. Complete Task 7 and verify DATATABLE_TYPE / DATATABLE_TYPE_VERSION below. +# 2. After Task 10 (credentials set up), replace REPLACE_WITH_*_CREDENTIAL_ID +# in the generated JSON and re-run to regenerate clean files before import. -- [ ] **Activate the workflow** (toggle in top-right) +require 'json' +require 'securerandom' + +SCRIPTS_DIR = File.dirname(__FILE__) + +# Verify against the exported scratch-workflow JSON from Task 7. +DATATABLE_TYPE = 'n8n-nodes-base.dataTable' +DATATABLE_TYPE_VERSION = 1 + +evaluate_sources_js = File.read(File.join(SCRIPTS_DIR, 'n8n-evaluate-sources.js')) +build_email_js = File.read(File.join(SCRIPTS_DIR, 'n8n-build-email.js')) + +# ── Workflow 1: Receiver ────────────────────────────────────────────────────── + +receiver = { + name: 'Backup Monitor — Receiver', + nodes: [ + { + parameters: { + httpMethod: 'POST', + path: 'backup-notify', + authentication: 'headerAuth', + responseMode: 'responseNode', + options: {} + }, + id: SecureRandom.uuid, + name: 'Webhook', + type: 'n8n-nodes-base.webhook', + typeVersion: 2, + position: [240, 300], + webhookId: SecureRandom.uuid, + credentials: { + httpHeaderAuth: { + id: 'REPLACE_WITH_BACKUP_CREDENTIAL_ID', + name: 'backup-monitor-token' + } + } + }, + { + parameters: { + resource: 'row', + operation: 'insert', + dataTableId: { + '__rl' => true, + mode: 'list', + value: 'REPLACE_WITH_DATATABLE_ID', + cachedResultName: 'backup_runs' + }, + columns: { + mappingMode: 'defineBelow', + value: { + timestamp: '={{ $json.body.timestamp }}', + machine: '={{ $json.body.machine }}', + destination: '={{ $json.body.destination }}', + status: '={{ $json.body.status }}', + bytes_transferred: '={{ $json.body.bytes_transferred }}', + snapshot_id: '={{ $json.body.snapshot_id }}', + log_path: '={{ $json.body.log_path }}', + notes: '={{ $json.body.notes }}' + } + } + }, + id: SecureRandom.uuid, + name: 'Insert to backup_runs', + type: DATATABLE_TYPE, + typeVersion: DATATABLE_TYPE_VERSION, + position: [520, 300] + }, + { + parameters: { + respondWith: 'json', + responseBody: '={{ JSON.stringify({ ok: true }) }}', + options: { responseCode: 200 } + }, + id: SecureRandom.uuid, + name: 'Respond 200', + type: 'n8n-nodes-base.respondToWebhook', + typeVersion: 1.1, + position: [800, 300] + } + ], + connections: { + 'Webhook' => { main: [[{ node: 'Insert to backup_runs', type: 'main', index: 0 }]] }, + 'Insert to backup_runs' => { main: [[{ node: 'Respond 200', type: 'main', index: 0 }]] } + }, + active: false, + settings: { executionOrder: 'v1' }, + pinData: {}, + meta: { templateCredsSetupCompleted: true }, + tags: [] +} + +# ── Workflow 2: Reporter ────────────────────────────────────────────────────── + +reporter = { + name: 'Backup Monitor — Reporter', + nodes: [ + { + parameters: { + rule: { + interval: [ + { + field: 'weeks', + weeksInterval: 1, + triggerAtDay: [5], + triggerAtHour: 8, + triggerAtMinute: 0 + } + ] + } + }, + id: SecureRandom.uuid, + name: 'Every Friday 8 AM', + type: 'n8n-nodes-base.scheduleTrigger', + typeVersion: 1.2, + position: [240, 300] + }, + { + parameters: { + mode: 'manual', + includeOtherFields: false, + assignments: { + assignments: [ + { + id: SecureRandom.uuid, + name: 'sources', + value: '={{ [{"machine":"desktop","destination":"synology"},{"machine":"desktop","destination":"backblaze-b2"},{"machine":"vps-ovh-prod-01","destination":"local"}] }}', + type: 'arrayValue' + } + ] + }, + options: {} + }, + id: SecureRandom.uuid, + name: 'Set sources', + type: 'n8n-nodes-base.set', + typeVersion: 3.4, + position: [460, 300] + }, + { + parameters: { + resource: 'row', + operation: 'getMany', + dataTableId: { + '__rl' => true, + mode: 'list', + value: 'REPLACE_WITH_DATATABLE_ID', + cachedResultName: 'backup_runs' + }, + filters: { + conditions: [ + { + keyName: 'timestamp', + condition: 'gt', + keyValue: '={{ $now.minus({days: 7}).toISO() }}' + } + ] + }, + options: {} + }, + id: SecureRandom.uuid, + name: 'Get backup_runs (last 7d)', + type: DATATABLE_TYPE, + typeVersion: DATATABLE_TYPE_VERSION, + position: [680, 300] + }, + { + parameters: { + mode: 'runOnceForAllItems', + language: 'javaScript', + jsCode: evaluate_sources_js + }, + id: SecureRandom.uuid, + name: 'Evaluate Sources', + type: 'n8n-nodes-base.code', + typeVersion: 2, + position: [900, 300] + }, + { + parameters: { + mode: 'runOnceForAllItems', + language: 'javaScript', + jsCode: build_email_js + }, + id: SecureRandom.uuid, + name: 'Build Email', + type: 'n8n-nodes-base.code', + typeVersion: 2, + position: [1120, 300] + }, + { + parameters: { + method: 'POST', + url: 'https://app.mailpace.com/api/v1/send', + authentication: 'genericCredentialType', + genericAuthType: 'httpHeaderAuth', + sendBody: true, + specifyBody: 'json', + # Expression returning a JS object; n8n serializes it to JSON automatically. + # Replace REPLACE_WITH_MAILPACE_FROM_ADDRESS with your verified MailPace sending address. + jsonBody: '={{ { from: "REPLACE_WITH_MAILPACE_FROM_ADDRESS", to: "jaredmswanson@gmail.com", subject: $json.subject, text_body: $json.body } }}', + options: {} + }, + id: SecureRandom.uuid, + name: 'Send via MailPace', + type: 'n8n-nodes-base.httpRequest', + typeVersion: 4.2, + position: [1340, 300], + credentials: { + httpHeaderAuth: { + id: 'REPLACE_WITH_MAILPACE_CREDENTIAL_ID', + name: 'mailpace-token' + } + } + } + ], + connections: { + 'Every Friday 8 AM' => { main: [[{ node: 'Set sources', type: 'main', index: 0 }]] }, + 'Set sources' => { main: [[{ node: 'Get backup_runs (last 7d)', type: 'main', index: 0 }]] }, + 'Get backup_runs (last 7d)' => { main: [[{ node: 'Evaluate Sources', type: 'main', index: 0 }]] }, + 'Evaluate Sources' => { main: [[{ node: 'Build Email', type: 'main', index: 0 }]] }, + 'Build Email' => { main: [[{ node: 'Send via MailPace', type: 'main', index: 0 }]] } + }, + active: false, + settings: { executionOrder: 'v1' }, + pinData: {}, + meta: { templateCredsSetupCompleted: true }, + tags: [] +} + +File.write(File.join(SCRIPTS_DIR, 'n8n-workflow-receiver.json'), JSON.pretty_generate(receiver)) +File.write(File.join(SCRIPTS_DIR, 'n8n-workflow-reporter.json'), JSON.pretty_generate(reporter)) + +puts 'Generated:' +puts " #{SCRIPTS_DIR}/n8n-workflow-receiver.json" +puts " #{SCRIPTS_DIR}/n8n-workflow-reporter.json" +``` + +- [ ] Run the build script: +```bash +cd /home/jared/systems-admin +ruby scripts/build-n8n-workflows.rb +``` +Expected: two files created with no error output + +- [ ] Validate both generated files: +```bash +python3 scripts/validate-n8n-workflow.py scripts/n8n-workflow-receiver.json +python3 scripts/validate-n8n-workflow.py scripts/n8n-workflow-reporter.json +``` +Expected: both print `OK — N nodes, N connections, 0 warning(s).` + +If warnings appear about `typeVersion`, update the constants in `build-n8n-workflows.rb` to match what Task 7 found, re-run, and re-validate. + +- [ ] Commit: +```bash +git add scripts/n8n-evaluate-sources.js scripts/n8n-build-email.js \ + scripts/build-n8n-workflows.rb \ + scripts/n8n-workflow-receiver.json scripts/n8n-workflow-reporter.json +git commit -m "feat(backup-monitor): add workflow JS files, build script, and generated workflow JSON" +``` --- -## Task 10: Integrate `notify_backup.rb` into desktop `~/.local/bin/backup.sh` +## Task 10: Set up credentials and import workflows via n8n API + +- [ ] **Create the backup-monitor credential in n8n UI:** + - Credentials → New → **Header Auth** + - Name: `backup-monitor-token` + - Header Name field: `Authorization` + - Header Value field: `Bearer ` + - Save + +- [ ] **Create the MailPace credential in n8n UI:** + - Credentials → New → **Header Auth** + - Name: `mailpace-token` + - Header Name field: `X-Server-Token` + - Header Value field: your MailPace server API token (find at app.mailpace.com → Server Settings) + - Save + +- [ ] **Get credential IDs from the n8n API:** +```bash +source ~/.credentials +curl -s "https://n8n.swansoncloud.com/api/v1/credentials" \ + -H "X-N8N-API-KEY: $N8N_SWANSONCLOUD_API_KEY" | python3 -m json.tool | grep -E '"id"|"name"' +``` +Note the numeric `id` for `backup-monitor-token` and `mailpace-token`. + +- [ ] **Fill in the credential IDs and datatable ID.** Edit `scripts/build-n8n-workflows.rb`: + - Replace `REPLACE_WITH_BACKUP_CREDENTIAL_ID` with the backup-monitor-token ID + - Replace `REPLACE_WITH_MAILPACE_CREDENTIAL_ID` with the mailpace-token ID + - Replace both `REPLACE_WITH_DATATABLE_ID` occurrences with the datatable ID from Task 7 + - Replace `REPLACE_WITH_MAILPACE_FROM_ADDRESS` with your verified MailPace sending address + +- [ ] **Regenerate the workflow JSON files:** +```bash +ruby scripts/build-n8n-workflows.rb +``` + +- [ ] **Validate again after substitution:** +```bash +python3 scripts/validate-n8n-workflow.py scripts/n8n-workflow-receiver.json +python3 scripts/validate-n8n-workflow.py scripts/n8n-workflow-reporter.json +``` +Expected: both `OK` + +- [ ] **Import Workflow 1 via the n8n API:** +```bash +source ~/.credentials +curl -s -X POST "https://n8n.swansoncloud.com/api/v1/workflows" \ + -H "X-N8N-API-KEY: $N8N_SWANSONCLOUD_API_KEY" \ + -H "Content-Type: application/json" \ + -d @scripts/n8n-workflow-receiver.json | python3 -m json.tool | grep '"id"' | head -1 +``` +Note the workflow `id` returned (e.g. `42`). + +- [ ] **Activate Workflow 1:** +```bash +curl -s -X PATCH "https://n8n.swansoncloud.com/api/v1/workflows/WORKFLOW_1_ID" \ + -H "X-N8N-API-KEY: $N8N_SWANSONCLOUD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"active": true}' | python3 -m json.tool | grep '"active"' +``` +Expected: `"active": true` + +- [ ] **Import Workflow 2:** +```bash +curl -s -X POST "https://n8n.swansoncloud.com/api/v1/workflows" \ + -H "X-N8N-API-KEY: $N8N_SWANSONCLOUD_API_KEY" \ + -H "Content-Type: application/json" \ + -d @scripts/n8n-workflow-reporter.json | python3 -m json.tool | grep '"id"' | head -1 +``` + +- [ ] **Activate Workflow 2:** +```bash +curl -s -X PATCH "https://n8n.swansoncloud.com/api/v1/workflows/WORKFLOW_2_ID" \ + -H "X-N8N-API-KEY: $N8N_SWANSONCLOUD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"active": true}' | python3 -m json.tool | grep '"active"' +``` + +- [ ] **Verify both workflows are visible and active** in the n8n UI at https://n8n.swansoncloud.com. + +- [ ] **Test Workflow 1 with a real webhook call:** +```bash +source ~/.credentials +notify_backup.rb \ + --machine desktop \ + --destination test \ + --status success \ + --notes "smoke test $(date)" +``` +Expected: no error output. Verify a row appears in the `backup_runs` table in n8n, then delete it. + +- [ ] Commit the final build script with real IDs: +```bash +cd /home/jared/systems-admin +git add scripts/build-n8n-workflows.rb scripts/n8n-workflow-receiver.json scripts/n8n-workflow-reporter.json +git commit -m "feat(backup-monitor): fill credential/datatable IDs and reimport workflows" +``` + +--- + +## Task 11: Integrate `notify_backup.rb` into desktop `~/.local/bin/backup.sh` The existing `backup.sh` uses `set -euo pipefail`. We add a helper function that temporarily disables `set -e` to capture the exit code of `run_backup`, then always re-enables it, then sends the notification. @@ -945,7 +1377,7 @@ git commit -m "feat(backup-monitor): integrate notify_backup.rb into desktop bac --- -## Task 11: Deploy and integrate on OVH VPS +## Task 12: Deploy and integrate on OVH VPS The VPS backup script is at `~/services/backup.sh` on `15.204.247.153`. The notifier script and ENV vars must be deployed there too. @@ -1018,7 +1450,7 @@ git commit --allow-empty -m "feat(backup-monitor): deploy notify_backup.rb to OV --- -## Task 12: End-to-end smoke test +## Task 13: End-to-end smoke test - [ ] **Trigger the desktop backup manually** and watch for webhook activity: ```bash @@ -1034,7 +1466,7 @@ Expected: backup runs as normal, no new errors in output - If all sources have at least one success: subject line starts with `✓` - If any source has no success: subject starts with `⚠` or `✗` with detail in the body -- [ ] **Remove any test rows** from `backup_runs` (rows with `destination=test` from Task 8) +- [ ] **Remove any test rows** from `backup_runs` (rows with `destination=test` from Task 10) - [ ] Final commit: ```bash @@ -1047,7 +1479,9 @@ git commit -m "docs(backup-monitor): implementation complete — smoke test pass ## Self-Review Notes -- **Spec coverage:** Ruby script (Tasks 1–6) ✓ · Datatable (Task 7) ✓ · Workflow 1 (Task 8) ✓ · Workflow 2 (Task 9) ✓ · Desktop integration (Task 10) ✓ · VPS integration (Task 11) ✓ · Email format green/yellow/red (Task 9) ✓ · Auth token from ENV (Tasks 8, 10) ✓ · `log_path` field (Tasks 2–3, 10) ✓ · `skipped` status (Task 10) ✓ -- **`backup.sh` variable name:** The VPS script uses `LOG_FILE` not `LOG` — Task 11 calls this out explicitly. -- **MailPace `from` address:** Needs to match a verified sending domain in your MailPace account. Confirm before activating Workflow 2. -- **n8n node naming:** The Code node in Task 9 references `$('Set: Expected Sources')` by name — the Set node in step 2 must be named exactly `Set: Expected Sources` or update the reference to match. +- **Spec coverage:** Ruby script (Tasks 1–6) ✓ · Datatable bootstrap (Task 7) ✓ · Validator (Task 8) ✓ · Workflow JSON generation (Task 9) ✓ · Credential setup + import (Task 10) ✓ · Desktop integration (Task 11) ✓ · VPS integration (Task 12) ✓ · Email format green/yellow/red (Task 9) ✓ · Auth token from ENV (Tasks 9–10) ✓ · `log_path` field (Tasks 2–3, 11) ✓ · `skipped` status (Task 11) ✓ +- **Data Table node uncertainty:** Both research agents flagged the Data Table node type as uncertain. Task 7 resolves this by exporting a probe workflow. If the type or parameter shape differs from `n8n-nodes-base.dataTable`, update the constants in `build-n8n-workflows.rb` and regenerate before import. +- **MailPace credential header:** Must be `X-Server-Token` (not `Authorization` or `MailPace-Server-Token`). This is set in Task 10 when creating the `mailpace-token` Header Auth credential. +- **MailPace `from` address:** Must match a verified sending domain in your MailPace server. Confirm at app.mailpace.com before filling in `build-n8n-workflows.rb`. +- **`backup.sh` variable name:** The VPS script uses `LOG_FILE` not `LOG` — Task 12 calls this out explicitly. +- **Node name consistency:** `n8n-evaluate-sources.js` references `$('Set sources')` and `$('Get backup_runs (last 7d)')` — these must match the `name` fields in `build-n8n-workflows.rb` exactly (they do as written).