server-ovh-prod-01-main/docs/health-monitoring.md

# Health Monitoring

This document describes the automated health and maintenance scripts running on systems-prod-01, their actual scope and behavior (verified 2026-06-25), and known gaps.

See also:
- [Monitoring & Automation](./monitoring-automation.md) — original overview (predates this doc; some details are superseded here)
- [Cert Management](./cert-management.md) — certificate expiry monitoring status

---

## syshealth Command

The `syshealth` alias is defined in `~/.bashrc` and points to:
```
/home/jared/services/monitoring/scripts/health-check.sh
```

This script was deployed and made executable on 2026-06-25. Running `syshealth` from any SSH session now works correctly.

### What health-check.sh checks

- **System**: disk usage (warns >80%), memory, load average
- **All containers**: flags not-running, RestartCount > 3, or unhealthy status
- **Per-service HTTP probes**: 13 Traefik hosts — non-2xx/3xx treated as failure (Traefik dashboard 401 = auth, treated as up)
- **TLS cert expiry**: warns if any cert expires in < 14 days
- **Backup freshness**: warns if the latest archive is > 26 hours old

Exits non-zero on any CRITICAL failure.

### First run (2026-06-25) — all green

- Disk: 23% used
- All 21 containers healthy
- All 13 hosts returned 2xx/3xx (or 401 for Traefik dashboard)
- Shortest cert: bestsolartech.com, 35 days remaining (expires 2026-07-31)
- Backup: ~4 hours old

---

## Cron Schedule (Current)

```
*/10 * * * *   /home/jared/services/container-health-monitor.sh
30   4 * * *   /home/jared/services/daily-maintenance.sh
0    7 * * *   /home/jared/services/backup.sh >> /home/jared/backups/cron.log 2>&1
0    1 * * *   /home/jared/services/remetrics/sitemap_refresh.sh >> /home/jared/services/remetrics/sitemap.log 2>&1
```

---

## container-health-monitor.sh

**Location**: `~/services/container-health-monitor.sh`
**Schedule**: Every 10 minutes (all day)
**Log**: `~/services/monitor.log`

### What it actually does

Despite its generic name, this script monitors **BestSolarTech only**. It:

1. Checks if `bestsolartech_wp` and `bestsolartech_db` containers are running
2. Checks if either container's memory usage exceeds 40% of host memory
3. Probes `https://bestsolartech.com` and checks for HTTP 200/301/302
4. If any check fails, runs `docker compose restart` on the BestSolarTech stack
5. **On every run**, executes: `docker system prune -f` (no `--volumes` flag)

### What it does NOT monitor

- n8n (prod, test, swanson)
- remetrics, strapi-remetrics
- Vaultwarden
- Forgejo
- NocoDB
- Invoice Ninja
- LandHomeTeam
- Traefik itself

These services have **no automated health monitoring** beyond Watchtower's image-update checks.

---

## daily-maintenance.sh

**Location**: `~/services/daily-maintenance.sh`
**Schedule**: Daily at 04:30 server time (de-collided from Watchtower's 03:00 window; moved 2026-06-25)
**Backup**: `~/services/daily-maintenance.sh.bak-2026-06-25`
**Log**: `~/services/maintenance.log`

### What it actually does

Despite the generic name, this script is **BestSolarTech-specific**:

1. Clears `/tmp` in the `bestsolartech_wp` container
2. Optimizes hardcoded MariaDB tables (`wp_vuhx_posts`, `wp_vuhx_postmeta`, `wp_vuhx_options`)
3. Restarts `mariadb` service in the BestSolarTech stack, waits 30 seconds, restarts `wordpress`
4. Runs `docker system prune -f` (no `--volumes` flag — **FIXED 2026-06-25**: `--volumes` was removed to prevent accidental removal of volumes from temporarily stopped services)
5. Logs container stats for `bestsolartech_wp` and `bestsolartech_db`

This script does not perform any maintenance on n8n, remetrics, Invoice Ninja, or any other service.

---

## backup.sh

**Location**: `~/services/backup.sh`
**Schedule**: Nightly at 07:00 UTC (02:00 EST)
**Log**: `~/backups/cron.log`

### What it covers

Auto-discovers all `docker-compose.yml` files under `~/services/` (maxdepth 2) and:
- Detects database type from container image names (MariaDB, PostgreSQL, MongoDB, PocketBase)
- Dumps databases using the appropriate method:
  - MariaDB: `mysqldump --all-databases`
  - PostgreSQL: `pg_dumpall`
- Tars service directories

After a successful run, calls `~/.local/bin/notify_backup.rb` to send a success notification.

**Retention**: 2 days on-server. The Synology NAS pulls `~/backups/current/latest.tar.gz` nightly at 03:00 EST. The NAS also syncs to B2 offsite via Synology Hyper Backup.

**Note on backup freshness**: `notify_backup.rb` is called only on success; a cron failure produces no notification. The `syshealth` script now covers backup freshness (warns if latest archive > 26 hours old).

---

## Watchtower

**Container**: `watchtower-watchtower-1` (`nickfedor/watchtower:latest`)
**Schedule**: Daily at 03:00 server time (cron6 format: `0 0 3 * * *`)
**Scope**: Opt-in only — **FIXED 2026-06-25**: `WATCHTOWER_LABEL_ENABLE=true` added to Watchtower compose
**Backup**: `~/services/watchtower/docker-compose.yml.bak-2026-06-25`

### How it works

Watchtower polls Docker Hub (and any configured registries) for updated images. If a newer image is found for a running container's tag, Watchtower pulls the new image, stops the container, and starts a new one.

`WATCHTOWER_CLEANUP=true` means old images are removed after an update.

### Opt-in label (current default: no containers opted in)

`WATCHTOWER_LABEL_ENABLE=true` is now set, so Watchtower only updates containers carrying:
```
com.centurylinklabs.watchtower.enable=true
```

As of 2026-06-25, **zero containers are opted in** — auto-update is deliberately disabled across all services until each is explicitly opted in. To enable auto-update for a service, add to its compose:

```yaml
labels:
  - "com.centurylinklabs.watchtower.enable=true"
```

Containers with local-only image tags that have no registry counterpart (`remetrics-web:staging-baked`, `remetrics-worker:staging-baked`, `strapi-remetrics:staging`) remain safe regardless — Watchtower cannot find a remote image to pull, so it skips them.

---

## Monitoring Gaps (Known, 2026-06-25)

These gaps were identified during a server architecture review. The following remain unaddressed:

1. **No alerting / cron wiring for health-check.sh**: `syshealth` runs on-demand only. No cron job runs it automatically, and there is no notification path (email, webhook) if checks fail. Silent failures will not surface unless someone manually runs `syshealth`.

2. **Per-service monitoring beyond health-check.sh**: The BestSolarTech-only `container-health-monitor.sh` still auto-restarts that service on failure. No equivalent auto-remediation exists for n8n, remetrics, Invoice Ninja, Vaultwarden, Forgejo, or NocoDB.

3. **BestSolarTech-only legacy scripts**: `container-health-monitor.sh` and `daily-maintenance.sh` have misleading generic names but remain BestSolarTech-specific. These could be renamed or generalized in the future.

---

## Manual Health Commands

```bash
# Run full health check (all green as of 2026-06-25)
ssh jared@15.204.247.153 "syshealth"

# All containers running?
ssh jared@15.204.247.153 "docker ps"

# Resource usage snapshot
ssh jared@15.204.247.153 "docker stats --no-stream"

# Disk usage
ssh jared@15.204.247.153 "df -h"

# Memory
ssh jared@15.204.247.153 "free -h"

# Traefik logs (last hour, errors only)
ssh jared@15.204.247.153 "docker logs traefik --since 1h 2>&1 | grep -i error"

# Recent backup log
ssh jared@15.204.247.153 "tail -30 ~/backups/cron.log"

# Monitor log (BestSolarTech health checks)
ssh jared@15.204.247.153 "tail -50 ~/services/monitor.log"

# Maintenance log
ssh jared@15.204.247.153 "tail -50 ~/services/maintenance.log"
```

---

## See Also

- [Monitoring & Automation](./monitoring-automation.md) — legacy overview doc
- [Cert Management](./cert-management.md) — cert expiry checking
- [Disaster Recovery](./disaster-recovery.md) — backup restore procedures