jared/server-ovh-prod-01-main
Archived
Watch
1
0
Fork
You've already forked server-ovh-prod-01-main
 0
Code Issues 5 Pull requests Projects Releases Packages Wiki Activity

Roll out Watchtower opt-in auto-update to remaining safe services #2

Open
opened 2026-08-12 15:29:24 +00:00 by jared · 3 comments
jared commented 2026-08-12 15:29:24 +00:00
Owner
Copy link

Context

Watchtower (nickfedor fork) runs daily at 03:00 with WATCHTOWER_LABEL_ENABLE=true, but no service carried the opt-in label, so nothing auto-updated for months (surfaced 2026-08-12 via the Vaultwarden 1.36.0 skeleton-hang incident). Pilot completed successfully on apprise: label added, on-demand --run-once detected and applied a real update, app verified reachable (200) after.

Steps (one service at a time, in this order)

For each service in: vaultwarden, nocodb, n8n (prod), n8n-test, n8n-swanson, firecrawl-api, firecrawl-playwright-service:

  1. Baseline: record image digest/created date, container health, and an app-reachability check (Traefik host label via curl -H 'Host: <domain>' https://localhost/<path> from the server, or tailnet/localhost port if not Traefik-routed). Record baseline HTTP status.
  2. Add label com.centurylinklabs.watchtower.enable: "true" to the service in ~/services/<name>/docker-compose.yml on jared@15.204.247.153; docker compose up -d.
  3. Trigger docker run --rm -v /var/run/docker.sock:/var/run/docker.sock nickfedor/watchtower --run-once --label-enable <container>.
  4. Verify: container running/healthy, last 20 log lines clean, reachability curl matches baseline status.
  5. For n8n instances: verify UI loads and a workflow executes before moving on. For vaultwarden: verify /alive 200 and /api/config.
  6. Record before/after digests in the ticket comments.

In scope

The seven services listed. Wiring Watchtower notifications (WATCHTOWER_NOTIFICATION_URL) to the local apprise instance so update runs stop being silent.

Out of scope

Databases (postgres, mariadb, redis, rabbitmq), traefik, forgejo, wordpress, local-built images (remetrics-*, strapi-remetrics) — never label these. Version-controlling ~/services (separate ticket). Forgejo/WordPress upgrades (separate tickets).

Acceptance criteria

  • All seven services carry the enable label and survived a --run-once pass with baseline-matching reachability.
  • Watchtower scheduled run reports scanned>0 in its logs on the next 03:00 pass.
  • Watchtower sends a notification via apprise on update (test with one forced run).
  • Per-service before/after digests recorded in ticket comments.
## Context Watchtower (nickfedor fork) runs daily at 03:00 with WATCHTOWER_LABEL_ENABLE=true, but no service carried the opt-in label, so nothing auto-updated for months (surfaced 2026-08-12 via the Vaultwarden 1.36.0 skeleton-hang incident). Pilot completed successfully on apprise: label added, on-demand `--run-once` detected and applied a real update, app verified reachable (200) after. ## Steps (one service at a time, in this order) For each service in: vaultwarden, nocodb, n8n (prod), n8n-test, n8n-swanson, firecrawl-api, firecrawl-playwright-service: 1. Baseline: record image digest/created date, container health, and an app-reachability check (Traefik host label via `curl -H 'Host: <domain>' https://localhost/<path>` from the server, or tailnet/localhost port if not Traefik-routed). Record baseline HTTP status. 2. Add label `com.centurylinklabs.watchtower.enable: "true"` to the service in `~/services/<name>/docker-compose.yml` on jared@15.204.247.153; `docker compose up -d`. 3. Trigger `docker run --rm -v /var/run/docker.sock:/var/run/docker.sock nickfedor/watchtower --run-once --label-enable <container>`. 4. Verify: container running/healthy, last 20 log lines clean, reachability curl matches baseline status. 5. For n8n instances: verify UI loads and a workflow executes before moving on. For vaultwarden: verify /alive 200 and /api/config. 6. Record before/after digests in the ticket comments. ## In scope The seven services listed. Wiring Watchtower notifications (WATCHTOWER_NOTIFICATION_URL) to the local apprise instance so update runs stop being silent. ## Out of scope Databases (postgres, mariadb, redis, rabbitmq), traefik, forgejo, wordpress, local-built images (remetrics-*, strapi-remetrics) — never label these. Version-controlling ~/services (separate ticket). Forgejo/WordPress upgrades (separate tickets). ## Acceptance criteria - [ ] All seven services carry the enable label and survived a --run-once pass with baseline-matching reachability. - [ ] Watchtower scheduled run reports scanned>0 in its logs on the next 03:00 pass. - [ ] Watchtower sends a notification via apprise on update (test with one forced run). - [ ] Per-service before/after digests recorded in ticket comments.
jared commented 2026-08-12 15:57:12 +00:00
Author
Owner
Copy link

Starting rollout: vaultwarden, nocodb, n8n (prod/test/swanson), firecrawl-api, firecrawl-playwright-service, one at a time, then wiring apprise notifications. Fresh full backup from 11:42 today covers all services.

Starting rollout: vaultwarden, nocodb, n8n (prod/test/swanson), firecrawl-api, firecrawl-playwright-service, one at a time, then wiring apprise notifications. Fresh full backup from 11:42 today covers all services.
jared commented 2026-08-12 16:08:50 +00:00
Author
Owner
Copy link

Rollout complete. Summary:

Service Baseline Update applied Post-change Commit
vaultwarden 200 no (already 1.37.1) 200 ac82bb1
nocodb 200 yes (fdd5b8a5->75c189a2) 200 e2640b4
n8n (prod) 200 yes (2.27.4->2.34.5) 200 e535de3
n8n-test 200 no (already latest) 200 0451242
n8n-swanson 200 yes (2.27.4->2.34.5) 200 6479797
firecrawl-api 200 yes (211b9a1d->926d5812) 200 n/a (dir git-excluded, vendored upstream checkout)
firecrawl-playwright-service n/a (no exposed port; verified via container status + api dependency) yes (3920120c->403caf7a) Up healthy n/a (same as above)

All n8n migrations logged clean, workflows reactivated, no errors. Notifications wired: watchtower's WATCHTOWER_NOTIFICATION_URL now points at generic://:8901/notify/ with disabletls=yes (commit f29f48d). Test run confirmed delivery end-to-end -- apprise logs show 'Sent Pushover notification to ALL_DEVICES' and 200 response.

Follow-ups: WATCHTOWER_NOTIFICATION_REPORT not enabled (deferred per ticket scope) -- session-summary notifications will only fire when an update actually occurs, not on every 3am scan with zero updates. Confirm scanned>0 in watchtower logs after the next 03:00 scheduled pass.

Rollout complete. Summary: | Service | Baseline | Update applied | Post-change | Commit | |---|---|---|---|---| | vaultwarden | 200 | no (already 1.37.1) | 200 | ac82bb1 | | nocodb | 200 | yes (fdd5b8a5->75c189a2) | 200 | e2640b4 | | n8n (prod) | 200 | yes (2.27.4->2.34.5) | 200 | e535de3 | | n8n-test | 200 | no (already latest) | 200 | 0451242 | | n8n-swanson | 200 | yes (2.27.4->2.34.5) | 200 | 6479797 | | firecrawl-api | 200 | yes (211b9a1d->926d5812) | 200 | n/a (dir git-excluded, vendored upstream checkout) | | firecrawl-playwright-service | n/a (no exposed port; verified via container status + api dependency) | yes (3920120c->403caf7a) | Up healthy | n/a (same as above) | All n8n migrations logged clean, workflows reactivated, no errors. Notifications wired: watchtower's WATCHTOWER_NOTIFICATION_URL now points at generic://<tailnet-ip>:8901/notify/<apprise-key> with disabletls=yes (commit f29f48d). Test run confirmed delivery end-to-end -- apprise logs show 'Sent Pushover notification to ALL_DEVICES' and 200 response. Follow-ups: WATCHTOWER_NOTIFICATION_REPORT not enabled (deferred per ticket scope) -- session-summary notifications will only fire when an update actually occurs, not on every 3am scan with zero updates. Confirm scanned>0 in watchtower logs after the next 03:00 scheduled pass.
jared commented 2026-08-12 20:16:48 +00:00
Author
Owner
Copy link

Migrated to jared/servers#9

Migrated to jared/servers#9
Commenting is not possible because the repository is archived.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
P0

   
P1

   
P2

   
P3

   
next

   
ready-for-agent

   
ready-for-human

   
recurring

   
review

   
waiting
No labels
P0
P1
P2
P3
next
ready-for-agent
ready-for-human
recurring
review
waiting
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
jared/server-ovh-prod-01-main#2
No description provided.