# Health Monitoring This document describes the automated health and maintenance scripts running on systems-prod-01, their actual scope and behavior (verified 2026-06-25), and known gaps. See also: - [Monitoring & Automation](./monitoring-automation.md) — original overview (predates this doc; some details are superseded here) - [Cert Management](./cert-management.md) — certificate expiry monitoring status --- ## syshealth Command The `syshealth` alias is defined in `~/.bashrc` and points to: ``` /home/jared/services/monitoring/scripts/health-check.sh ``` This script was deployed and made executable on 2026-06-25. Running `syshealth` from any SSH session now works correctly. ### What health-check.sh checks - **System**: disk usage (warns >80%), memory, load average - **All containers**: flags not-running, RestartCount > 3, or unhealthy status - **Per-service HTTP probes**: 13 Traefik hosts — non-2xx/3xx treated as failure (Traefik dashboard 401 = auth, treated as up) - **TLS cert expiry**: warns if any cert expires in < 14 days - **Backup freshness**: warns if the latest archive is > 26 hours old Exits non-zero on any CRITICAL failure. ### First run (2026-06-25) — all green - Disk: 23% used - All 21 containers healthy - All 13 hosts returned 2xx/3xx (or 401 for Traefik dashboard) - Shortest cert: bestsolartech.com, 35 days remaining (expires 2026-07-31) - Backup: ~4 hours old --- ## Cron Schedule (Current) ``` */10 * * * * /home/jared/services/container-health-monitor.sh 30 4 * * * /home/jared/services/daily-maintenance.sh 0 7 * * * /home/jared/services/backup.sh >> /home/jared/backups/cron.log 2>&1 0 1 * * * /home/jared/services/remetrics/sitemap_refresh.sh >> /home/jared/services/remetrics/sitemap.log 2>&1 ``` --- ## container-health-monitor.sh **Location**: `~/services/container-health-monitor.sh` **Schedule**: Every 10 minutes (all day) **Log**: `~/services/monitor.log` ### What it actually does Despite its generic name, this script monitors **BestSolarTech only**. It: 1. Checks if `bestsolartech_wp` and `bestsolartech_db` containers are running 2. Checks if either container's memory usage exceeds 40% of host memory 3. Probes `https://bestsolartech.com` and checks for HTTP 200/301/302 4. If any check fails, runs `docker compose restart` on the BestSolarTech stack 5. **On every run**, executes: `docker system prune -f` (no `--volumes` flag) ### What it does NOT monitor - n8n (prod, test, swanson) - remetrics, strapi-remetrics - Vaultwarden - Forgejo - NocoDB - Invoice Ninja - LandHomeTeam - Traefik itself These services have **no automated health monitoring** beyond Watchtower's image-update checks. --- ## daily-maintenance.sh **Location**: `~/services/daily-maintenance.sh` **Schedule**: Daily at 04:30 server time (de-collided from Watchtower's 03:00 window; moved 2026-06-25) **Backup**: `~/services/daily-maintenance.sh.bak-2026-06-25` **Log**: `~/services/maintenance.log` ### What it actually does Despite the generic name, this script is **BestSolarTech-specific**: 1. Clears `/tmp` in the `bestsolartech_wp` container 2. Optimizes hardcoded MariaDB tables (`wp_vuhx_posts`, `wp_vuhx_postmeta`, `wp_vuhx_options`) 3. Restarts `mariadb` service in the BestSolarTech stack, waits 30 seconds, restarts `wordpress` 4. Runs `docker system prune -f` (no `--volumes` flag — **FIXED 2026-06-25**: `--volumes` was removed to prevent accidental removal of volumes from temporarily stopped services) 5. Logs container stats for `bestsolartech_wp` and `bestsolartech_db` This script does not perform any maintenance on n8n, remetrics, Invoice Ninja, or any other service. --- ## backup.sh **Location**: `~/services/backup.sh` **Schedule**: Nightly at 07:00 UTC (02:00 EST) **Log**: `~/backups/cron.log` ### What it covers Auto-discovers all `docker-compose.yml` files under `~/services/` (maxdepth 2) and: - Detects database type from container image names (MariaDB, PostgreSQL, MongoDB, PocketBase) - Dumps databases using the appropriate method: - MariaDB: `mysqldump --all-databases` - PostgreSQL: `pg_dumpall` - Tars service directories After a successful run, calls `~/.local/bin/notify_backup.rb` to send a success notification. **Retention**: 2 days on-server. The Synology NAS pulls `~/backups/current/latest.tar.gz` nightly at 03:00 EST. The NAS also syncs to B2 offsite via Synology Hyper Backup. **Note on backup freshness**: `notify_backup.rb` is called only on success; a cron failure produces no notification. The `syshealth` script now covers backup freshness (warns if latest archive > 26 hours old). --- ## Watchtower **Container**: `watchtower-watchtower-1` (`nickfedor/watchtower:latest`) **Schedule**: Daily at 03:00 server time (cron6 format: `0 0 3 * * *`) **Scope**: Opt-in only — **FIXED 2026-06-25**: `WATCHTOWER_LABEL_ENABLE=true` added to Watchtower compose **Backup**: `~/services/watchtower/docker-compose.yml.bak-2026-06-25` ### How it works Watchtower polls Docker Hub (and any configured registries) for updated images. If a newer image is found for a running container's tag, Watchtower pulls the new image, stops the container, and starts a new one. `WATCHTOWER_CLEANUP=true` means old images are removed after an update. ### Opt-in label (current default: no containers opted in) `WATCHTOWER_LABEL_ENABLE=true` is now set, so Watchtower only updates containers carrying: ``` com.centurylinklabs.watchtower.enable=true ``` As of 2026-06-25, **zero containers are opted in** — auto-update is deliberately disabled across all services until each is explicitly opted in. To enable auto-update for a service, add to its compose: ```yaml labels: - "com.centurylinklabs.watchtower.enable=true" ``` Containers with local-only image tags that have no registry counterpart (`remetrics-web:staging-baked`, `remetrics-worker:staging-baked`, `strapi-remetrics:staging`) remain safe regardless — Watchtower cannot find a remote image to pull, so it skips them. --- ## Monitoring Gaps (Known, 2026-06-25) These gaps were identified during a server architecture review. The following remain unaddressed: 1. **No alerting / cron wiring for health-check.sh**: `syshealth` runs on-demand only. No cron job runs it automatically, and there is no notification path (email, webhook) if checks fail. Silent failures will not surface unless someone manually runs `syshealth`. 2. **Per-service monitoring beyond health-check.sh**: The BestSolarTech-only `container-health-monitor.sh` still auto-restarts that service on failure. No equivalent auto-remediation exists for n8n, remetrics, Invoice Ninja, Vaultwarden, Forgejo, or NocoDB. 3. **BestSolarTech-only legacy scripts**: `container-health-monitor.sh` and `daily-maintenance.sh` have misleading generic names but remain BestSolarTech-specific. These could be renamed or generalized in the future. --- ## Manual Health Commands ```bash # Run full health check (all green as of 2026-06-25) ssh jared@15.204.247.153 "syshealth" # All containers running? ssh jared@15.204.247.153 "docker ps" # Resource usage snapshot ssh jared@15.204.247.153 "docker stats --no-stream" # Disk usage ssh jared@15.204.247.153 "df -h" # Memory ssh jared@15.204.247.153 "free -h" # Traefik logs (last hour, errors only) ssh jared@15.204.247.153 "docker logs traefik --since 1h 2>&1 | grep -i error" # Recent backup log ssh jared@15.204.247.153 "tail -30 ~/backups/cron.log" # Monitor log (BestSolarTech health checks) ssh jared@15.204.247.153 "tail -50 ~/services/monitor.log" # Maintenance log ssh jared@15.204.247.153 "tail -50 ~/services/maintenance.log" ``` --- ## See Also - [Monitoring & Automation](./monitoring-automation.md) — legacy overview doc - [Cert Management](./cert-management.md) — cert expiry checking - [Disaster Recovery](./disaster-recovery.md) — backup restore procedures