jared/server-ovh-prod-01-main
Archived
Watch
1
0
Fork
You've already forked server-ovh-prod-01-main
 0
Code Issues 5 Pull requests Projects Releases Packages Wiki Activity

Establish monthly fleet image update review (recurring) #5

Open
opened 2026-08-12 15:30:18 +00:00 by jared · 3 comments
jared commented 2026-08-12 15:30:18 +00:00
Owner
Copy link

Context

The 2026-08-12 audit found Watchtower inert for months and several services excluded from auto-update by design (databases, traefik, forgejo, wordpress, local builds). Those excluded services need a low-noise manual cadence: one monthly digest + guided updates, instead of daily update alerts. Intended to become a recurring issue (human applies the recurring label / occurrence mechanics — AI must not create or close recurring issues).

Steps (each monthly occurrence)

  1. Audit: on jared@15.204.247.153 list every running container with image tag + image created date; compare pinned/excluded services against upstream latest (registry tags / GitHub releases).
  2. Produce one digest table: service | running | latest | risk class (patch/minor/major) | recommendation.
  3. Apply safe patch/minor updates for excluded-but-manual services (databases via minor-tag bumps only, traefik minor) using the standard playbook: stop → tar data dir to ~/backups → pull → up -d → verify reachability matches baseline.
  4. Flag majors (postgres/mariadb majors, forgejo, wordpress core) as separate tickets — never apply in this pass.
  5. Also check Watchtower's own logs: confirm scheduled runs still report scanned>0 and notifications fire.
  6. Record the digest + actions as an occurrence comment.

In scope

Excluded/manual services. Watchtower health check. Digest reporting.

Out of scope

Major version upgrades (spawn dedicated tickets). Services already on Watchtower auto-update (only verify Watchtower covers them).

Acceptance criteria (per occurrence)

  • Digest table posted as an occurrence comment.
  • Safe minor/patch updates applied with pre-update backups and post-update verification noted.
  • Any major-version needs filed as separate tickets and linked.
  • Watchtower scheduled-run health confirmed.
## Context The 2026-08-12 audit found Watchtower inert for months and several services excluded from auto-update by design (databases, traefik, forgejo, wordpress, local builds). Those excluded services need a low-noise manual cadence: one monthly digest + guided updates, instead of daily update alerts. Intended to become a recurring issue (human applies the recurring label / occurrence mechanics — AI must not create or close recurring issues). ## Steps (each monthly occurrence) 1. Audit: on jared@15.204.247.153 list every running container with image tag + image created date; compare pinned/excluded services against upstream latest (registry tags / GitHub releases). 2. Produce one digest table: service | running | latest | risk class (patch/minor/major) | recommendation. 3. Apply safe patch/minor updates for excluded-but-manual services (databases via minor-tag bumps only, traefik minor) using the standard playbook: stop → tar data dir to ~/backups → pull → up -d → verify reachability matches baseline. 4. Flag majors (postgres/mariadb majors, forgejo, wordpress core) as separate tickets — never apply in this pass. 5. Also check Watchtower's own logs: confirm scheduled runs still report scanned>0 and notifications fire. 6. Record the digest + actions as an occurrence comment. ## In scope Excluded/manual services. Watchtower health check. Digest reporting. ## Out of scope Major version upgrades (spawn dedicated tickets). Services already on Watchtower auto-update (only verify Watchtower covers them). ## Acceptance criteria (per occurrence) - [ ] Digest table posted as an occurrence comment. - [ ] Safe minor/patch updates applied with pre-update backups and post-update verification noted. - [ ] Any major-version needs filed as separate tickets and linked. - [ ] Watchtower scheduled-run health confirmed.
jared commented 2026-08-12 16:43:06 +00:00
Author
Owner
Copy link

First occurrence of the monthly fleet image update review starting now (2026-08-12). Building digest table for excluded/manual services + Watchtower health check.

First occurrence of the monthly fleet image update review starting now (2026-08-12). Building digest table for excluded/manual services + Watchtower health check.
jared commented 2026-08-12 16:46:19 +00:00
Author
Owner
Copy link

Occurrence 1 — 2026-08-12

Digest (excluded/manual services; Watchtower-managed services verified covered, not re-listed):

Service Image Watchtower Latest available Class Recommendation
traefik traefik:v3.6 → recreated No v3.6.25 (within v3.6 floating tag) patch Applied — pulled + recreated
forgejo codeberg.org/forgejo/forgejo:16.0.2 No (pinned) 16.0.2 up-to-date none (pinned earlier today)
bestsolartech wordpress wordpress:7.0.3-php8.3-apache No (pinned) tracked in #7 excluded-by-design see #7
bestsolartech mariadb mariadb:10.11 No tracked in #7 excluded-by-design see #7
invoiceninja app invoiceninja/invoiceninja:5 No 5 (floating, no v6) up-to-date none
invoiceninja mariadb mariadb:10.11 No 11.4/11.8/12.x majors exist major filed #9
invoiceninja nginx nginx:latest No floating tag up-to-date none
landhometeam nginx nginx:alpine No floating tag up-to-date none
n8n postgres postgres:16 No 18 (major) major filed #8
n8n-test postgres postgres:16 No 18 (major) major filed #8
remetrics postgres postgres:15 No 18 (major) major filed #8
remetrics/strapi-remetrics local staging-baked builds No N/A excluded-by-design local builds, out of scope
remetrics redis, firecrawl redis redis:alpine No floating tag up-to-date none
firecrawl rabbitmq rabbitmq:3-management No floating tag up-to-date none

Watchtower-managed (n8n, n8n-swanson, n8n-test app, vaultwarden, apprise, nocodb, firecrawl api/playwright) — verified com.centurylinklabs.watchtower.enable=true label present in each compose file; not re-audited individually this pass.

Actions taken:

  • Backed up traefik config/state to ~/backups/traefik-pre-update-20260812.tar.gz (sudo tar, includes acme.json).
  • Baseline-verified 5 routed domains (n8n.swansoncloud.com, bitwarden.swansoncloud.com, forgejo.swansoncloud.com, bestsolartech.com, remetrics.io) all 200 before change.
  • docker compose pull && up -d on traefik — image tag v3.6 unchanged (floating), resolved to v3.6.25 (from an older v3.6 build pulled 2026-06-22). No compose file diff, so no git commit needed.
  • Post-update: same 5 domains re-verified 200; traefik version confirms 3.6.25 running.

Majors flagged (new tickets, not applied):

  • #8 — Postgres major-version upgrade evaluation (n8n, n8n-test, remetrics: postgres 15/16 → 18)
  • #9 — MariaDB major-version upgrade evaluation for invoiceninja (10.11 → 11.x/12.x); distinct from #7's bestsolartech mariadb

Watchtower health: Running nickfedor/watchtower:latest v1.18.1, schedule 0 0 3 * * *, notifications via shoutrrr generic webhook configured, label-gated (WATCHTOWER_LABEL_ENABLE=true). Log shows Next scheduled run: 2026-08-13 03:00:00 UTC. Container was recently restarted (uptime ~34 min at audit time) as part of earlier updates today, so no historical "scanned>0" log entry available yet from this run cycle — recommend confirming a scanned>0 line after tonight's 03:00 UTC run.

Deferred to next occurrence: DB image refreshes for postgres/mariadb/redis containers on floating tags — no pinned-stale version found requiring immediate action; majors tracked in #8/#9 instead.

This occurrence is complete. Ticket #5 remains open (recurring — not closed by AI).

## Occurrence 1 — 2026-08-12 **Digest** (excluded/manual services; Watchtower-managed services verified covered, not re-listed): | Service | Image | Watchtower | Latest available | Class | Recommendation | |---|---|---|---|---|---| | traefik | traefik:v3.6 → recreated | No | v3.6.25 (within v3.6 floating tag) | patch | **Applied** — pulled + recreated | | forgejo | codeberg.org/forgejo/forgejo:16.0.2 | No (pinned) | 16.0.2 | up-to-date | none (pinned earlier today) | | bestsolartech wordpress | wordpress:7.0.3-php8.3-apache | No (pinned) | tracked in #7 | excluded-by-design | see #7 | | bestsolartech mariadb | mariadb:10.11 | No | tracked in #7 | excluded-by-design | see #7 | | invoiceninja app | invoiceninja/invoiceninja:5 | No | 5 (floating, no v6) | up-to-date | none | | invoiceninja mariadb | mariadb:10.11 | No | 11.4/11.8/12.x majors exist | major | filed #9 | | invoiceninja nginx | nginx:latest | No | floating tag | up-to-date | none | | landhometeam nginx | nginx:alpine | No | floating tag | up-to-date | none | | n8n postgres | postgres:16 | No | 18 (major) | major | filed #8 | | n8n-test postgres | postgres:16 | No | 18 (major) | major | filed #8 | | remetrics postgres | postgres:15 | No | 18 (major) | major | filed #8 | | remetrics/strapi-remetrics | local staging-baked builds | No | N/A | excluded-by-design | local builds, out of scope | | remetrics redis, firecrawl redis | redis:alpine | No | floating tag | up-to-date | none | | firecrawl rabbitmq | rabbitmq:3-management | No | floating tag | up-to-date | none | Watchtower-managed (n8n, n8n-swanson, n8n-test app, vaultwarden, apprise, nocodb, firecrawl api/playwright) — verified `com.centurylinklabs.watchtower.enable=true` label present in each compose file; not re-audited individually this pass. **Actions taken:** - Backed up traefik config/state to `~/backups/traefik-pre-update-20260812.tar.gz` (sudo tar, includes acme.json). - Baseline-verified 5 routed domains (n8n.swansoncloud.com, bitwarden.swansoncloud.com, forgejo.swansoncloud.com, bestsolartech.com, remetrics.io) all 200 before change. - `docker compose pull && up -d` on traefik — image tag `v3.6` unchanged (floating), resolved to v3.6.25 (from an older v3.6 build pulled 2026-06-22). No compose file diff, so no git commit needed. - Post-update: same 5 domains re-verified 200; `traefik version` confirms 3.6.25 running. **Majors flagged (new tickets, not applied):** - #8 — Postgres major-version upgrade evaluation (n8n, n8n-test, remetrics: postgres 15/16 → 18) - #9 — MariaDB major-version upgrade evaluation for invoiceninja (10.11 → 11.x/12.x); distinct from #7's bestsolartech mariadb **Watchtower health:** Running `nickfedor/watchtower:latest` v1.18.1, schedule `0 0 3 * * *`, notifications via shoutrrr generic webhook configured, label-gated (`WATCHTOWER_LABEL_ENABLE=true`). Log shows `Next scheduled run: 2026-08-13 03:00:00 UTC`. Container was recently restarted (uptime ~34 min at audit time) as part of earlier updates today, so no historical "scanned>0" log entry available yet from this run cycle — recommend confirming a scanned>0 line after tonight's 03:00 UTC run. **Deferred to next occurrence:** DB image refreshes for postgres/mariadb/redis containers on floating tags — no pinned-stale version found requiring immediate action; majors tracked in #8/#9 instead. This occurrence is complete. Ticket #5 remains open (recurring — not closed by AI).
jared commented 2026-08-12 20:16:49 +00:00
Author
Owner
Copy link

Migrated to jared/servers#10

Migrated to jared/servers#10
Commenting is not possible because the repository is archived.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
P0

   
P1

   
P2

   
P3

   
next

   
ready-for-agent

   
ready-for-human

   
recurring

   
review

   
waiting
No labels
P0
P1
P2
P3
next
ready-for-agent
ready-for-human
recurring
review
waiting
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
jared/server-ovh-prod-01-main#5
No description provided.