207 lines
7.7 KiB
Markdown
207 lines
7.7 KiB
Markdown
# Health Monitoring
|
|
|
|
This document describes the automated health and maintenance scripts running on systems-prod-01, their actual scope and behavior (verified 2026-06-25), and known gaps.
|
|
|
|
See also:
|
|
- [Monitoring & Automation](./monitoring-automation.md) — original overview (predates this doc; some details are superseded here)
|
|
- [Cert Management](./cert-management.md) — certificate expiry monitoring status
|
|
|
|
---
|
|
|
|
## syshealth Command
|
|
|
|
The `syshealth` alias is defined in `~/.bashrc` and points to:
|
|
```
|
|
/home/jared/services/monitoring/scripts/health-check.sh
|
|
```
|
|
|
|
This script was deployed and made executable on 2026-06-25. Running `syshealth` from any SSH session now works correctly.
|
|
|
|
### What health-check.sh checks
|
|
|
|
- **System**: disk usage (warns >80%), memory, load average
|
|
- **All containers**: flags not-running, RestartCount > 3, or unhealthy status
|
|
- **Per-service HTTP probes**: 13 Traefik hosts — non-2xx/3xx treated as failure (Traefik dashboard 401 = auth, treated as up)
|
|
- **TLS cert expiry**: warns if any cert expires in < 14 days
|
|
- **Backup freshness**: warns if the latest archive is > 26 hours old
|
|
|
|
Exits non-zero on any CRITICAL failure.
|
|
|
|
### First run (2026-06-25) — all green
|
|
|
|
- Disk: 23% used
|
|
- All 21 containers healthy
|
|
- All 13 hosts returned 2xx/3xx (or 401 for Traefik dashboard)
|
|
- Shortest cert: bestsolartech.com, 35 days remaining (expires 2026-07-31)
|
|
- Backup: ~4 hours old
|
|
|
|
---
|
|
|
|
## Cron Schedule (Current)
|
|
|
|
```
|
|
*/10 * * * * /home/jared/services/container-health-monitor.sh
|
|
30 4 * * * /home/jared/services/daily-maintenance.sh
|
|
0 7 * * * /home/jared/services/backup.sh >> /home/jared/backups/cron.log 2>&1
|
|
0 1 * * * /home/jared/services/remetrics/sitemap_refresh.sh >> /home/jared/services/remetrics/sitemap.log 2>&1
|
|
```
|
|
|
|
---
|
|
|
|
## container-health-monitor.sh
|
|
|
|
**Location**: `~/services/container-health-monitor.sh`
|
|
**Schedule**: Every 10 minutes (all day)
|
|
**Log**: `~/services/monitor.log`
|
|
|
|
### What it actually does
|
|
|
|
Despite its generic name, this script monitors **BestSolarTech only**. It:
|
|
|
|
1. Checks if `bestsolartech_wp` and `bestsolartech_db` containers are running
|
|
2. Checks if either container's memory usage exceeds 40% of host memory
|
|
3. Probes `https://bestsolartech.com` and checks for HTTP 200/301/302
|
|
4. If any check fails, runs `docker compose restart` on the BestSolarTech stack
|
|
5. **On every run**, executes: `docker system prune -f` (no `--volumes` flag)
|
|
|
|
### What it does NOT monitor
|
|
|
|
- n8n (prod, test, swanson)
|
|
- remetrics, strapi-remetrics
|
|
- Vaultwarden
|
|
- Forgejo
|
|
- NocoDB
|
|
- Invoice Ninja
|
|
- LandHomeTeam
|
|
- Traefik itself
|
|
|
|
These services have **no automated health monitoring** beyond Watchtower's image-update checks.
|
|
|
|
---
|
|
|
|
## daily-maintenance.sh
|
|
|
|
**Location**: `~/services/daily-maintenance.sh`
|
|
**Schedule**: Daily at 04:30 server time (de-collided from Watchtower's 03:00 window; moved 2026-06-25)
|
|
**Backup**: `~/services/daily-maintenance.sh.bak-2026-06-25`
|
|
**Log**: `~/services/maintenance.log`
|
|
|
|
### What it actually does
|
|
|
|
Despite the generic name, this script is **BestSolarTech-specific**:
|
|
|
|
1. Clears `/tmp` in the `bestsolartech_wp` container
|
|
2. Optimizes hardcoded MariaDB tables (`wp_vuhx_posts`, `wp_vuhx_postmeta`, `wp_vuhx_options`)
|
|
3. Restarts `mariadb` service in the BestSolarTech stack, waits 30 seconds, restarts `wordpress`
|
|
4. Runs `docker system prune -f` (no `--volumes` flag — **FIXED 2026-06-25**: `--volumes` was removed to prevent accidental removal of volumes from temporarily stopped services)
|
|
5. Logs container stats for `bestsolartech_wp` and `bestsolartech_db`
|
|
|
|
This script does not perform any maintenance on n8n, remetrics, Invoice Ninja, or any other service.
|
|
|
|
---
|
|
|
|
## backup.sh
|
|
|
|
**Location**: `~/services/backup.sh`
|
|
**Schedule**: Nightly at 07:00 UTC (02:00 EST)
|
|
**Log**: `~/backups/cron.log`
|
|
|
|
### What it covers
|
|
|
|
Auto-discovers all `docker-compose.yml` files under `~/services/` (maxdepth 2) and:
|
|
- Detects database type from container image names (MariaDB, PostgreSQL, MongoDB, PocketBase)
|
|
- Dumps databases using the appropriate method:
|
|
- MariaDB: `mysqldump --all-databases`
|
|
- PostgreSQL: `pg_dumpall`
|
|
- Tars service directories
|
|
|
|
After a successful run, calls `~/.local/bin/notify_backup.rb` to send a success notification.
|
|
|
|
**Retention**: 2 days on-server. The Synology NAS pulls `~/backups/current/latest.tar.gz` nightly at 03:00 EST. The NAS also syncs to B2 offsite via Synology Hyper Backup.
|
|
|
|
**Note on backup freshness**: `notify_backup.rb` is called only on success; a cron failure produces no notification. The `syshealth` script now covers backup freshness (warns if latest archive > 26 hours old).
|
|
|
|
---
|
|
|
|
## Watchtower
|
|
|
|
**Container**: `watchtower-watchtower-1` (`nickfedor/watchtower:latest`)
|
|
**Schedule**: Daily at 03:00 server time (cron6 format: `0 0 3 * * *`)
|
|
**Scope**: Opt-in only — **FIXED 2026-06-25**: `WATCHTOWER_LABEL_ENABLE=true` added to Watchtower compose
|
|
**Backup**: `~/services/watchtower/docker-compose.yml.bak-2026-06-25`
|
|
|
|
### How it works
|
|
|
|
Watchtower polls Docker Hub (and any configured registries) for updated images. If a newer image is found for a running container's tag, Watchtower pulls the new image, stops the container, and starts a new one.
|
|
|
|
`WATCHTOWER_CLEANUP=true` means old images are removed after an update.
|
|
|
|
### Opt-in label (current default: no containers opted in)
|
|
|
|
`WATCHTOWER_LABEL_ENABLE=true` is now set, so Watchtower only updates containers carrying:
|
|
```
|
|
com.centurylinklabs.watchtower.enable=true
|
|
```
|
|
|
|
As of 2026-06-25, **zero containers are opted in** — auto-update is deliberately disabled across all services until each is explicitly opted in. To enable auto-update for a service, add to its compose:
|
|
|
|
```yaml
|
|
labels:
|
|
- "com.centurylinklabs.watchtower.enable=true"
|
|
```
|
|
|
|
Containers with local-only image tags that have no registry counterpart (`remetrics-web:staging-baked`, `remetrics-worker:staging-baked`, `strapi-remetrics:staging`) remain safe regardless — Watchtower cannot find a remote image to pull, so it skips them.
|
|
|
|
---
|
|
|
|
## Monitoring Gaps (Known, 2026-06-25)
|
|
|
|
These gaps were identified during a server architecture review. The following remain unaddressed:
|
|
|
|
1. **No alerting / cron wiring for health-check.sh**: `syshealth` runs on-demand only. No cron job runs it automatically, and there is no notification path (email, webhook) if checks fail. Silent failures will not surface unless someone manually runs `syshealth`.
|
|
|
|
2. **Per-service monitoring beyond health-check.sh**: The BestSolarTech-only `container-health-monitor.sh` still auto-restarts that service on failure. No equivalent auto-remediation exists for n8n, remetrics, Invoice Ninja, Vaultwarden, Forgejo, or NocoDB.
|
|
|
|
3. **BestSolarTech-only legacy scripts**: `container-health-monitor.sh` and `daily-maintenance.sh` have misleading generic names but remain BestSolarTech-specific. These could be renamed or generalized in the future.
|
|
|
|
---
|
|
|
|
## Manual Health Commands
|
|
|
|
```bash
|
|
# Run full health check (all green as of 2026-06-25)
|
|
ssh jared@15.204.247.153 "syshealth"
|
|
|
|
# All containers running?
|
|
ssh jared@15.204.247.153 "docker ps"
|
|
|
|
# Resource usage snapshot
|
|
ssh jared@15.204.247.153 "docker stats --no-stream"
|
|
|
|
# Disk usage
|
|
ssh jared@15.204.247.153 "df -h"
|
|
|
|
# Memory
|
|
ssh jared@15.204.247.153 "free -h"
|
|
|
|
# Traefik logs (last hour, errors only)
|
|
ssh jared@15.204.247.153 "docker logs traefik --since 1h 2>&1 | grep -i error"
|
|
|
|
# Recent backup log
|
|
ssh jared@15.204.247.153 "tail -30 ~/backups/cron.log"
|
|
|
|
# Monitor log (BestSolarTech health checks)
|
|
ssh jared@15.204.247.153 "tail -50 ~/services/monitor.log"
|
|
|
|
# Maintenance log
|
|
ssh jared@15.204.247.153 "tail -50 ~/services/maintenance.log"
|
|
```
|
|
|
|
---
|
|
|
|
## See Also
|
|
|
|
- [Monitoring & Automation](./monitoring-automation.md) — legacy overview doc
|
|
- [Cert Management](./cert-management.md) — cert expiry checking
|
|
- [Disaster Recovery](./disaster-recovery.md) — backup restore procedures
|