jared
/
server-ovh-prod-01-main
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
server-ovh-prod-01-main/docs/health-monitoring.md

7.7 KiB
Raw Blame History

Health Monitoring

This document describes the automated health and maintenance scripts running on systems-prod-01, their actual scope and behavior (verified 2026-06-25), and known gaps.

See also:

syshealth Command

The syshealth alias is defined in ~/.bashrc and points to:

/home/jared/services/monitoring/scripts/health-check.sh

This script was deployed and made executable on 2026-06-25. Running syshealth from any SSH session now works correctly.

What health-check.sh checks

  • System: disk usage (warns >80%), memory, load average
  • All containers: flags not-running, RestartCount > 3, or unhealthy status
  • Per-service HTTP probes: 13 Traefik hosts — non-2xx/3xx treated as failure (Traefik dashboard 401 = auth, treated as up)
  • TLS cert expiry: warns if any cert expires in < 14 days
  • Backup freshness: warns if the latest archive is > 26 hours old

Exits non-zero on any CRITICAL failure.

First run (2026-06-25) — all green

  • Disk: 23% used
  • All 21 containers healthy
  • All 13 hosts returned 2xx/3xx (or 401 for Traefik dashboard)
  • Shortest cert: bestsolartech.com, 35 days remaining (expires 2026-07-31)
  • Backup: ~4 hours old

Cron Schedule (Current)

*/10 * * * *   /home/jared/services/container-health-monitor.sh
30   4 * * *   /home/jared/services/daily-maintenance.sh
0    7 * * *   /home/jared/services/backup.sh >> /home/jared/backups/cron.log 2>&1
0    1 * * *   /home/jared/services/remetrics/sitemap_refresh.sh >> /home/jared/services/remetrics/sitemap.log 2>&1

container-health-monitor.sh

Location: ~/services/container-health-monitor.sh Schedule: Every 10 minutes (all day) Log: ~/services/monitor.log

What it actually does

Despite its generic name, this script monitors BestSolarTech only. It:

  1. Checks if bestsolartech_wp and bestsolartech_db containers are running
  2. Checks if either container's memory usage exceeds 40% of host memory
  3. Probes https://bestsolartech.com and checks for HTTP 200/301/302
  4. If any check fails, runs docker compose restart on the BestSolarTech stack
  5. On every run, executes: docker system prune -f (no --volumes flag)

What it does NOT monitor

  • n8n (prod, test, swanson)
  • remetrics, strapi-remetrics
  • Vaultwarden
  • Forgejo
  • NocoDB
  • Invoice Ninja
  • LandHomeTeam
  • Traefik itself

These services have no automated health monitoring beyond Watchtower's image-update checks.

daily-maintenance.sh

Location: ~/services/daily-maintenance.sh Schedule: Daily at 04:30 server time (de-collided from Watchtower's 03:00 window; moved 2026-06-25) Backup: ~/services/daily-maintenance.sh.bak-2026-06-25 Log: ~/services/maintenance.log

What it actually does

Despite the generic name, this script is BestSolarTech-specific:

  1. Clears /tmp in the bestsolartech_wp container
  2. Optimizes hardcoded MariaDB tables (wp_vuhx_posts, wp_vuhx_postmeta, wp_vuhx_options)
  3. Restarts mariadb service in the BestSolarTech stack, waits 30 seconds, restarts wordpress
  4. Runs docker system prune -f (no --volumes flag — FIXED 2026-06-25: --volumes was removed to prevent accidental removal of volumes from temporarily stopped services)
  5. Logs container stats for bestsolartech_wp and bestsolartech_db

This script does not perform any maintenance on n8n, remetrics, Invoice Ninja, or any other service.

backup.sh

Location: ~/services/backup.sh Schedule: Nightly at 07:00 UTC (02:00 EST) Log: ~/backups/cron.log

What it covers

Auto-discovers all docker-compose.yml files under ~/services/ (maxdepth 2) and:

  • Detects database type from container image names (MariaDB, PostgreSQL, MongoDB, PocketBase)
  • Dumps databases using the appropriate method:
    • MariaDB: mysqldump --all-databases
    • PostgreSQL: pg_dumpall
  • Tars service directories

After a successful run, calls ~/.local/bin/notify_backup.rb to send a success notification.

Retention: 2 days on-server. The Synology NAS pulls ~/backups/current/latest.tar.gz nightly at 03:00 EST. The NAS also syncs to B2 offsite via Synology Hyper Backup.

Note on backup freshness: notify_backup.rb is called only on success; a cron failure produces no notification. The syshealth script now covers backup freshness (warns if latest archive > 26 hours old).

Watchtower

Container: watchtower-watchtower-1 (nickfedor/watchtower:latest) Schedule: Daily at 03:00 server time (cron6 format: 0 0 3 * * *) Scope: Opt-in only — FIXED 2026-06-25: WATCHTOWER_LABEL_ENABLE=true added to Watchtower compose Backup: ~/services/watchtower/docker-compose.yml.bak-2026-06-25

How it works

Watchtower polls Docker Hub (and any configured registries) for updated images. If a newer image is found for a running container's tag, Watchtower pulls the new image, stops the container, and starts a new one.

WATCHTOWER_CLEANUP=true means old images are removed after an update.

Opt-in label (current default: no containers opted in)

WATCHTOWER_LABEL_ENABLE=true is now set, so Watchtower only updates containers carrying:

com.centurylinklabs.watchtower.enable=true

As of 2026-06-25, zero containers are opted in — auto-update is deliberately disabled across all services until each is explicitly opted in. To enable auto-update for a service, add to its compose:

labels:
  - "com.centurylinklabs.watchtower.enable=true"

Containers with local-only image tags that have no registry counterpart (remetrics-web:staging-baked, remetrics-worker:staging-baked, strapi-remetrics:staging) remain safe regardless — Watchtower cannot find a remote image to pull, so it skips them.

Monitoring Gaps (Known, 2026-06-25)

These gaps were identified during a server architecture review. The following remain unaddressed:

  1. No alerting / cron wiring for health-check.sh: syshealth runs on-demand only. No cron job runs it automatically, and there is no notification path (email, webhook) if checks fail. Silent failures will not surface unless someone manually runs syshealth.

  2. Per-service monitoring beyond health-check.sh: The BestSolarTech-only container-health-monitor.sh still auto-restarts that service on failure. No equivalent auto-remediation exists for n8n, remetrics, Invoice Ninja, Vaultwarden, Forgejo, or NocoDB.

  3. BestSolarTech-only legacy scripts: container-health-monitor.sh and daily-maintenance.sh have misleading generic names but remain BestSolarTech-specific. These could be renamed or generalized in the future.

Manual Health Commands

# Run full health check (all green as of 2026-06-25)
ssh jared@15.204.247.153 "syshealth"

# All containers running?
ssh jared@15.204.247.153 "docker ps"

# Resource usage snapshot
ssh jared@15.204.247.153 "docker stats --no-stream"

# Disk usage
ssh jared@15.204.247.153 "df -h"

# Memory
ssh jared@15.204.247.153 "free -h"

# Traefik logs (last hour, errors only)
ssh jared@15.204.247.153 "docker logs traefik --since 1h 2>&1 | grep -i error"

# Recent backup log
ssh jared@15.204.247.153 "tail -30 ~/backups/cron.log"

# Monitor log (BestSolarTech health checks)
ssh jared@15.204.247.153 "tail -50 ~/services/monitor.log"

# Maintenance log
ssh jared@15.204.247.153 "tail -50 ~/services/maintenance.log"

See Also