jared
/
server-ovh-prod-01-main
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

docs: document server architecture, add health tooling, harden ops hazards 

- server architecture + progressive-disclosure docs; new health-check.sh / syshealth health monitoring doc; remetrics operations + update-workflow runbooks; cert-management doc (two ACME resolvers); recorded ops-hazard fixes (prune cron de-fanged + rescheduled 04:30, Watchtower switched to opt-in); recorded GHCR image DR backup + Bitbucket deploy-key connection; corrected stale profile.yaml/migration-plan facts.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Jared Swanson 2026-06-25 12:20:08 -04:00
parent 25726b2289
commit bcb361127e
8 changed files with 1039 additions and 59 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

13
CLAUDE.md
View File

@ -6,13 +6,15 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
This is a dedicated workspace for managing the systems-prod-01 server (OVHcloud VPS, IP: 15.204.247.153) with a focus on Docker containerization and Traefik reverse proxy. This is a dedicated workspace for managing the systems-prod-01 server (OVHcloud VPS, IP: 15.204.247.153) with a focus on Docker containerization and Traefik reverse proxy.
**Migration status:** The original purpose was migrating services off a DigitalOcean VPS (`hyperthrive-docker-services`, 161.35.12.75). That migration is **complete** and the DigitalOcean VPS was **deleted on 2026-06-24** — this OVH box is now the sole production host with no DigitalOcean fallback. The project's ongoing role is operating and maintaining this server. **Migration status:** The original purpose was migrating services off a DigitalOcean VPS (`hyperthrive-docker-services`, 161.35.12.75). That migration is **complete** and the DigitalOcean VPS was **deleted on 2026-06-24** — for those services this OVH box is now the sole production host with no DigitalOcean fallback. The project's ongoing role is operating and maintaining this server.
Additionally, on **2026-06-24** `remetrics.io` (Rails app + Strapi CMS) was migrated to this OVH box from a **separate** DigitalOcean web droplet (147.182.131.217, distinct from `hyperthrive-docker-services`). That remetrics droplet is **frozen and kept as a short-term rollback (not yet deleted)** — the one remaining DigitalOcean fallback. See [migrations/migration-plan.md](migrations/migration-plan.md) (remetrics addendum).
## Server Architecture ## Server Architecture
The production server uses: The production server uses:
- **Containerization**: All services run in Docker containers with Docker Compose - **Containerization**: All services run in Docker containers with Docker Compose
- **Reverse Proxy**: Traefik v3.0 handles SSL termination and routing - **Reverse Proxy**: Traefik v3.6 handles SSL termination and routing
- **Service Structure**: Services live in `/home/jared/services/` on the remote server with shallow directory structure (e.g., `~/services/bestsolartech`, `~/services/traefik`) - **Service Structure**: Services live in `/home/jared/services/` on the remote server with shallow directory structure (e.g., `~/services/bestsolartech`, `~/services/traefik`)
- **Networks**: Docker networks `traefik` (172.20.0.0/16) for external-facing services and `internal` (172.21.0.0/16) for backend communication - **Networks**: Docker networks `traefik` (172.20.0.0/16) for external-facing services and `internal` (172.21.0.0/16) for backend communication
@ -26,7 +28,7 @@ ssh jared@15.204.247.153
# Check service status # Check service status
ssh jared@15.204.247.153 "docker ps" ssh jared@15.204.247.153 "docker ps"
# Check system health # Check system health (runs health-check.sh — fixed 2026-06-25, see docs/health-monitoring.md)
ssh jared@15.204.247.153 "syshealth" ssh jared@15.204.247.153 "syshealth"
``` ```
@ -66,9 +68,13 @@ For detailed operational guides, consult these specialized documents:
- **[Services Inventory](docs/services-inventory.md)** - Current active services, domains, and container details (🔄 **updated frequently**) - **[Services Inventory](docs/services-inventory.md)** - Current active services, domains, and container details (🔄 **updated frequently**)
- **[WordPress Operations](docs/wordpress-operations.md)** - WordPress-specific deployment, configuration, and troubleshooting - **[WordPress Operations](docs/wordpress-operations.md)** - WordPress-specific deployment, configuration, and troubleshooting
- **[Monitoring & Automation](docs/monitoring-automation.md)** - Automated health checks, maintenance scripts, and log analysis - **[Monitoring & Automation](docs/monitoring-automation.md)** - Automated health checks, maintenance scripts, and log analysis
- **[Health Monitoring](docs/health-monitoring.md)** - Script coverage, Watchtower behavior, syshealth status, and monitoring gaps
- **[Configuration Reference](docs/configuration-reference.md)** - Docker Compose templates and Traefik configuration patterns - **[Configuration Reference](docs/configuration-reference.md)** - Docker Compose templates and Traefik configuration patterns
- **[Cert Management](docs/cert-management.md)** - Two ACME resolvers (letsencrypt HTTP-01 and cloudflare DNS-01), cert storage, and renewal
- **[Documentation Maintenance](docs/documentation-maintenance.md)** - When and how to update documentation - **[Documentation Maintenance](docs/documentation-maintenance.md)** - When and how to update documentation
- **[Disaster Recovery](docs/disaster-recovery.md)** - Backup system overview and restore procedures - **[Disaster Recovery](docs/disaster-recovery.md)** - Backup system overview and restore procedures
- **[remetrics Operations](docs/remetrics-operations.md)** - remetrics.io runbook: stack layout, Sidekiq, CMS coupling, DigitalOcean rollback
- **[remetrics Update Workflow](docs/remetrics-update-workflow.md)** - How to deploy code changes to the remetrics Rails app and Strapi CMS
## Migration Tracking ## Migration Tracking
@ -95,7 +101,6 @@ complexity threshold.
- Synthesize subagent summaries into responses - Synthesize subagent summaries into responses
- Does NOT call Read, Write, Bash, Grep, or Glob — those are direct operations, not delegation - Does NOT call Read, Write, Bash, Grep, or Glob — those are direct operations, not delegation
- Does NOT read files to prepare delegation specs — write specs from the user's request - Does NOT read files to prepare delegation specs — write specs from the user's request
- NLM skill triggers (feature complete, debug, security, doc sync) are dispatched via Agent tool
**Subagents return:** brief summary + paths to artifacts. Not full file contents. **Subagents return:** brief summary + paths to artifacts. Not full file contents.

172
docs/cert-management.md Normal file
View File

@ -0,0 +1,172 @@
# Certificate Management
TLS certificates on systems-prod-01 are managed automatically by Traefik via ACME. There are two resolvers in use — one for most services and a separate one for remetrics.
See also:
- [remetrics Operations](./remetrics-operations.md) — why remetrics uses a different resolver
- [Configuration Reference](./configuration-reference.md) — Traefik label syntax
- [Monitoring & Automation](./monitoring-automation.md) — health check status
---
## Two ACME Resolvers
| Resolver name | Challenge type | Used by | Config key |
|---|---|---|---|
| `letsencrypt` | HTTP-01 | All services except remetrics | `certresolver=letsencrypt` |
| `cloudflare` | DNS-01 | remetrics.io, www.remetrics.io, cms.remetrics.io | `certresolver=cloudflare` |
### letsencrypt (HTTP-01)
Used by: n8n, n8n-test, Invoice Ninja, Vaultwarden, NocoDB, Forgejo, BestSolarTech, LandHomeTeam, and all other services not listed under cloudflare.
HTTP-01 requires that the domain resolves to this server and that port 80 is reachable. Traefik answers the ACME challenge automatically via the HTTP entrypoint.
### cloudflare (DNS-01)
Used by: remetrics.io, www.remetrics.io, cms.remetrics.io.
DNS-01 was added during the 2026-06-24 remetrics migration to allow certificates to be pre-issued before the DNS cutover, closing the cert-availability gap. It uses Cloudflare API credentials (stored in Traefik's environment, not documented here — see `credentials.md`).
DNS-01 works even when the domain does not yet resolve to this server, making it the right choice for any future migration that needs pre-issued certs.
---
## Where Certificates Are Stored
Traefik stores ACME data in two JSON files on the server:
| File | Resolver | Location |
|---|---|---|
| `acme.json` | `letsencrypt` (HTTP-01) | `~/services/traefik/letsencrypt/acme.json` |
| `acme-cloudflare.json` | `cloudflare` (DNS-01) | `~/services/traefik/letsencrypt/acme-cloudflare.json` |
Both files are bind-mounted into the Traefik container. They persist across Traefik restarts.
**Do not delete these files** unless you want Traefik to re-issue all certificates from scratch on next start. Let's Encrypt has rate limits (5 failures per domain per hour, 50 new certs per domain per week).
---
## Viewing Certificate Status
### Check all Traefik-managed certs (from ACME JSON)
```bash
ssh jared@15.204.247.153 "cat ~/services/traefik/letsencrypt/acme.json | python3 -c \"
import json,sys,base64,datetime
data=json.load(sys.stdin)
for r,v in data.items():
for d in v.get('Certificates',[]):
cert=base64.b64decode(d['certificate'])
# pipe to openssl for full details
print(d['domain']['main'])
\""
```
For a more readable output, pipe the cert bytes to `openssl x509 -noout -dates`:
```bash
ssh jared@15.204.247.153 "python3 - <<'EOF'
import json, base64, subprocess
for fname in ['acme.json', 'acme-cloudflare.json']:
try:
with open(f'/home/jared/services/traefik/letsencrypt/{fname}') as f:
data = json.load(f)
for resolver, v in data.items():
for entry in v.get('Certificates', []):
domain = entry['domain']['main']
cert_pem = base64.b64decode(entry['certificate'])
result = subprocess.run(
['openssl', 'x509', '-noout', '-enddate'],
input=cert_pem, capture_output=True
)
print(f'{domain}: {result.stdout.decode().strip()}')
except Exception as e:
print(f'{fname}: {e}')
EOF"
```
### Check expiry via TLS handshake (for live domains)
```bash
# Check a single domain
ssh jared@15.204.247.153 "echo | openssl s_client -connect remetrics.io:443 -servername remetrics.io 2>/dev/null | openssl x509 -noout -enddate"
# Quick loop over all critical domains
ssh jared@15.204.247.153 "for domain in remetrics.io cms.remetrics.io n8n.hyperthrive.io bestsolartech.com; do
expiry=\$(echo | openssl s_client -connect \${domain}:443 -servername \${domain} 2>/dev/null | openssl x509 -noout -enddate 2>/dev/null)
echo \"\${domain}: \${expiry}\"
done"
```
---
## How Renewal Works
Traefik renews certificates automatically when they have ~30 days remaining. No manual action is needed under normal operation.
For the `letsencrypt` resolver: Traefik handles the HTTP-01 challenge internally. Ensure port 80 remains open (UFW rule `80/tcp ALLOW IN Anywhere` is active).
For the `cloudflare` resolver: Traefik uses the Cloudflare API to create and remove DNS TXT records. No port needs to be open. The Cloudflare API token must remain valid — check `credentials.md` for the token and its expiry.
---
## Renewal Troubleshooting
If a certificate fails to renew:
```bash
# Check Traefik logs for ACME errors
ssh jared@15.204.247.153 "docker logs traefik --since 24h 2>&1 | grep -i 'acme\|certificate\|error'"
```
Common causes:
- **HTTP-01 failure**: port 80 is blocked, or the domain no longer resolves to this server
- **DNS-01 failure**: Cloudflare API token expired or revoked; check `credentials.md`
- **Rate limit hit**: too many failed attempts; wait and retry; check logs for "rate limit" messages
- **acme.json permissions**: the file must be readable by the Traefik container; `chmod 600` is the correct permission (set in Traefik's static config)
To force a renewal (use sparingly — rate limits apply):
```bash
# Backup the acme.json, delete the cert entry for the domain, restart Traefik
ssh jared@15.204.247.153 "cp ~/services/traefik/letsencrypt/acme.json ~/backups/acme-backup-$(date +%Y%m%d).json"
# Then edit acme.json to remove the certificate entry for the affected domain
ssh jared@15.204.247.153 "docker restart traefik"
```
---
## Adding a New Service with a Certificate
For new services using HTTP-01 (standard):
```yaml
labels:
- "traefik.http.routers.MYSERVICE.rule=Host(`mydomain.com`)"
- "traefik.http.routers.MYSERVICE.entrypoints=websecure"
- "traefik.http.routers.MYSERVICE.tls=true"
- "traefik.http.routers.MYSERVICE.tls.certresolver=letsencrypt"
```
For a service that needs DNS-01 (e.g., pre-issuing a cert before DNS cutover):
```yaml
labels:
- "traefik.http.routers.MYSERVICE.tls.certresolver=cloudflare"
```
See [Configuration Reference](./configuration-reference.md) for full label templates.
---
## Certificate Expiry Monitoring — Current Status
> **TODO**: There is currently no automated cert expiry alerting. Traefik auto-renews, but if renewal silently fails, no alert is sent.
>
> A cert-check script was proposed during the 2026-06-25 architecture review. It would check expiry days for all domains and alert if under 14 days. This has not yet been implemented. See [Monitoring & Automation](./monitoring-automation.md) for the gap list.
---
## See Also
- [remetrics Operations](./remetrics-operations.md) — DNS-01 background
- [Configuration Reference](./configuration-reference.md) — Traefik label templates
- [Monitoring & Automation](./monitoring-automation.md) — monitoring gaps

206
docs/health-monitoring.md Normal file
View File

@ -0,0 +1,206 @@
# Health Monitoring
This document describes the automated health and maintenance scripts running on systems-prod-01, their actual scope and behavior (verified 2026-06-25), and known gaps.
See also:
- [Monitoring & Automation](./monitoring-automation.md) — original overview (predates this doc; some details are superseded here)
- [Cert Management](./cert-management.md) — certificate expiry monitoring status
---
## syshealth Command
The `syshealth` alias is defined in `~/.bashrc` and points to:
```
/home/jared/services/monitoring/scripts/health-check.sh
```
This script was deployed and made executable on 2026-06-25. Running `syshealth` from any SSH session now works correctly.
### What health-check.sh checks
- **System**: disk usage (warns >80%), memory, load average
- **All containers**: flags not-running, RestartCount > 3, or unhealthy status
- **Per-service HTTP probes**: 13 Traefik hosts — non-2xx/3xx treated as failure (Traefik dashboard 401 = auth, treated as up)
- **TLS cert expiry**: warns if any cert expires in < 14 days
- **Backup freshness**: warns if the latest archive is > 26 hours old
Exits non-zero on any CRITICAL failure.
### First run (2026-06-25) — all green
- Disk: 23% used
- All 21 containers healthy
- All 13 hosts returned 2xx/3xx (or 401 for Traefik dashboard)
- Shortest cert: bestsolartech.com, 35 days remaining (expires 2026-07-31)
- Backup: ~4 hours old
---
## Cron Schedule (Current)
```
*/10 * * * * /home/jared/services/container-health-monitor.sh
30 4 * * * /home/jared/services/daily-maintenance.sh
0 7 * * * /home/jared/services/backup.sh >> /home/jared/backups/cron.log 2>&1
0 1 * * * /home/jared/services/remetrics/sitemap_refresh.sh >> /home/jared/services/remetrics/sitemap.log 2>&1
```
---
## container-health-monitor.sh
**Location**: `~/services/container-health-monitor.sh`
**Schedule**: Every 10 minutes (all day)
**Log**: `~/services/monitor.log`
### What it actually does
Despite its generic name, this script monitors **BestSolarTech only**. It:
1. Checks if `bestsolartech_wp` and `bestsolartech_db` containers are running
2. Checks if either container's memory usage exceeds 40% of host memory
3. Probes `https://bestsolartech.com` and checks for HTTP 200/301/302
4. If any check fails, runs `docker compose restart` on the BestSolarTech stack
5. **On every run**, executes: `docker system prune -f` (no `--volumes` flag)
### What it does NOT monitor
- n8n (prod, test, swanson)
- remetrics, strapi-remetrics
- Vaultwarden
- Forgejo
- NocoDB
- Invoice Ninja
- LandHomeTeam
- Traefik itself
These services have **no automated health monitoring** beyond Watchtower's image-update checks.
---
## daily-maintenance.sh
**Location**: `~/services/daily-maintenance.sh`
**Schedule**: Daily at 04:30 server time (de-collided from Watchtower's 03:00 window; moved 2026-06-25)
**Backup**: `~/services/daily-maintenance.sh.bak-2026-06-25`
**Log**: `~/services/maintenance.log`
### What it actually does
Despite the generic name, this script is **BestSolarTech-specific**:
1. Clears `/tmp` in the `bestsolartech_wp` container
2. Optimizes hardcoded MariaDB tables (`wp_vuhx_posts`, `wp_vuhx_postmeta`, `wp_vuhx_options`)
3. Restarts `mariadb` service in the BestSolarTech stack, waits 30 seconds, restarts `wordpress`
4. Runs `docker system prune -f` (no `--volumes` flag — **FIXED 2026-06-25**: `--volumes` was removed to prevent accidental removal of volumes from temporarily stopped services)
5. Logs container stats for `bestsolartech_wp` and `bestsolartech_db`
This script does not perform any maintenance on n8n, remetrics, Invoice Ninja, or any other service.
---
## backup.sh
**Location**: `~/services/backup.sh`
**Schedule**: Nightly at 07:00 UTC (02:00 EST)
**Log**: `~/backups/cron.log`
### What it covers
Auto-discovers all `docker-compose.yml` files under `~/services/` (maxdepth 2) and:
- Detects database type from container image names (MariaDB, PostgreSQL, MongoDB, PocketBase)
- Dumps databases using the appropriate method:
- MariaDB: `mysqldump --all-databases`
- PostgreSQL: `pg_dumpall`
- Tars service directories
After a successful run, calls `~/.local/bin/notify_backup.rb` to send a success notification.
**Retention**: 2 days on-server. The Synology NAS pulls `~/backups/current/latest.tar.gz` nightly at 03:00 EST. The NAS also syncs to B2 offsite via Synology Hyper Backup.
**Note on backup freshness**: `notify_backup.rb` is called only on success; a cron failure produces no notification. The `syshealth` script now covers backup freshness (warns if latest archive > 26 hours old).
---
## Watchtower
**Container**: `watchtower-watchtower-1` (`nickfedor/watchtower:latest`)
**Schedule**: Daily at 03:00 server time (cron6 format: `0 0 3 * * *`)
**Scope**: Opt-in only — **FIXED 2026-06-25**: `WATCHTOWER_LABEL_ENABLE=true` added to Watchtower compose
**Backup**: `~/services/watchtower/docker-compose.yml.bak-2026-06-25`
### How it works
Watchtower polls Docker Hub (and any configured registries) for updated images. If a newer image is found for a running container's tag, Watchtower pulls the new image, stops the container, and starts a new one.
`WATCHTOWER_CLEANUP=true` means old images are removed after an update.
### Opt-in label (current default: no containers opted in)
`WATCHTOWER_LABEL_ENABLE=true` is now set, so Watchtower only updates containers carrying:
```
com.centurylinklabs.watchtower.enable=true
```
As of 2026-06-25, **zero containers are opted in** — auto-update is deliberately disabled across all services until each is explicitly opted in. To enable auto-update for a service, add to its compose:
```yaml
labels:
- "com.centurylinklabs.watchtower.enable=true"
```
Containers with local-only image tags that have no registry counterpart (`remetrics-web:staging-baked`, `remetrics-worker:staging-baked`, `strapi-remetrics:staging`) remain safe regardless — Watchtower cannot find a remote image to pull, so it skips them.
---
## Monitoring Gaps (Known, 2026-06-25)
These gaps were identified during a server architecture review. The following remain unaddressed:
1. **No alerting / cron wiring for health-check.sh**: `syshealth` runs on-demand only. No cron job runs it automatically, and there is no notification path (email, webhook) if checks fail. Silent failures will not surface unless someone manually runs `syshealth`.
2. **Per-service monitoring beyond health-check.sh**: The BestSolarTech-only `container-health-monitor.sh` still auto-restarts that service on failure. No equivalent auto-remediation exists for n8n, remetrics, Invoice Ninja, Vaultwarden, Forgejo, or NocoDB.
3. **BestSolarTech-only legacy scripts**: `container-health-monitor.sh` and `daily-maintenance.sh` have misleading generic names but remain BestSolarTech-specific. These could be renamed or generalized in the future.
---
## Manual Health Commands
```bash
# Run full health check (all green as of 2026-06-25)
ssh jared@15.204.247.153 "syshealth"
# All containers running?
ssh jared@15.204.247.153 "docker ps"
# Resource usage snapshot
ssh jared@15.204.247.153 "docker stats --no-stream"
# Disk usage
ssh jared@15.204.247.153 "df -h"
# Memory
ssh jared@15.204.247.153 "free -h"
# Traefik logs (last hour, errors only)
ssh jared@15.204.247.153 "docker logs traefik --since 1h 2>&1 | grep -i error"
# Recent backup log
ssh jared@15.204.247.153 "tail -30 ~/backups/cron.log"
# Monitor log (BestSolarTech health checks)
ssh jared@15.204.247.153 "tail -50 ~/services/monitor.log"
# Maintenance log
ssh jared@15.204.247.153 "tail -50 ~/services/maintenance.log"
```
---
## See Also
- [Monitoring & Automation](./monitoring-automation.md) — legacy overview doc
- [Cert Management](./cert-management.md) — cert expiry checking
- [Disaster Recovery](./disaster-recovery.md) — backup restore procedures

244
docs/remetrics-operations.md Normal file
View File

@ -0,0 +1,244 @@
# remetrics Operations
Runbook for the remetrics.io production stack running on systems-prod-01 (OVH, 15.204.247.153).
See also:
- [remetrics Update Workflow](./remetrics-update-workflow.md) — how to deploy code changes
- [Cert Management](./cert-management.md) — Cloudflare DNS-01 resolver used by this stack
- [Services Inventory](./services-inventory.md) — container listing
- [Disaster Recovery](./disaster-recovery.md) — backup restore procedures
---
## Stack Overview
remetrics is two separate Compose projects:
| Project | Directory | Containers | Purpose |
|---|---|---|---|
| `remetrics` | `~/services/remetrics/` | web, worker, postgres, redis | Rails app + Sidekiq |
| `strapi-remetrics` | `~/services/strapi-remetrics/` | strapi | Headless CMS |
### Container summary
| Container | Image | Networks |
|---|---|---|
| `remetrics-remetrics-1` | `remetrics-web:staging-baked` | `traefik`, `internal` |
| `remetrics-worker-1` | `remetrics-worker:staging-baked` | `internal` only |
| `remetrics-db-1` | `postgres:15` | `internal` only |
| `remetrics-redis-1` | `redis:alpine` | `internal` only |
| `strapi-remetrics` | `strapi-remetrics:staging` | `traefik`, `strapi-remetrics_default` |
The Rails web and worker images (`staging-baked`) were built during the 2026-06-24 migration via a `docker commit`-style bake. The primary copy is on this server; a DR backup was pushed to GHCR on 2026-06-25:
- `ghcr.io/jaredswanson/remetrics-web:staging-baked` / `:dr-2026-06-25`
- `ghcr.io/jaredswanson/remetrics-worker:staging-baked` / `:dr-2026-06-25` (same image as web — one build, two commands)
GHCR is private; pulling requires `docker login ghcr.io` with a token that has `read:packages`. See [remetrics Update Workflow](./remetrics-update-workflow.md) for how to rebuild them.
The Strapi image (`strapi-remetrics:staging`) originated from a private GCP registry but is now a local tag, also backed up to GHCR (`ghcr.io/jaredswanson/strapi-remetrics:staging` / `:dr-2026-06-25`). The compose file carries a `# Do NOT pull / rebuild` comment because that GCP registry is gone. See the Strapi section of the update workflow before attempting any Strapi image changes.
---
## Domains and TLS
| Domain | Service | Cert Resolver |
|---|---|---|
| `remetrics.io` | Rails web | `cloudflare` (DNS-01) |
| `www.remetrics.io` | Rails web | `cloudflare` (DNS-01) |
| `cms.remetrics.io` | Strapi CMS | `cloudflare` (DNS-01) |
All three domains use the `cloudflare` ACME resolver (DNS-01), not the default `letsencrypt` HTTP-01 resolver used by other services. DNS-01 was required to pre-issue certificates before the DNS cutover window. See [Cert Management](./cert-management.md) for where certs are stored and how to check expiry.
---
## CMS URL — Hardcoded Coupling
**Important**: The Rails app fetches CMS content from a **hardcoded** URL:
```
https://cms.remetrics.io
```
This is not env-configurable. It is set directly in the app source at:
- `app/models/strapi/article.rb`
- `app/models/strapi/help_page.rb`
**Consequence**: If `strapi-remetrics` is down or `cms.remetrics.io` is unreachable, the Rails web process will fail to render any CMS-backed pages. Strapi and Rails must be kept running together. This also means that renaming the CMS domain or moving Strapi to a new host requires a code change and image rebuild — not just a config change.
---
## Image Provenance
Source repository for the Rails app: `git@bitbucket.org:jared_/remetrics_app.git`
The on-disk source is at `~/services/remetrics/app/` (bind-mounted into both web and worker containers at `/app`). This is a live git repo tracking the Bitbucket remote. The Bitbucket read-only deploy key is configured and verified; on-box source is in sync with `origin/master` (commit 7aa117df as of 2026-06-25).
The Strapi source (package.json, src/, config/) lives at `~/services/strapi-remetrics/strapi/`. There is no separate Bitbucket repo for Strapi; the source files are the canonical copy on this server.
---
## Data Layout
### Rails database
- Container: `remetrics-db-1` (postgres:15)
- Database name: `remetrics_production`
- User: `estatioro`
- Storage: Docker named volume `remetrics_remetrics_pgdata`
- 480 users migrated from DigitalOcean (zero data loss, 2026-06-24)
The named volume is backed up nightly by `~/services/backup.sh` via `pg_dumpall`. Confirm with:
```bash
ssh jared@15.204.247.153 "ls -lh ~/backups/current/"
```
### Strapi database
- Engine: SQLite
- File location: `~/services/strapi-remetrics/strapi/database/` (bind-mounted)
- Backed up nightly as part of the strapi-remetrics service-files tar
### Strapi media uploads
- Storage: GCS bucket `remetrics-strapi`
- GCS service account key: `~/services/strapi-remetrics/strapi/gcs/` (bind-mounted)
- Not backed up by backup.sh (lives in GCS, not on-server)
### Redis
Redis (`redis:alpine`) has **no named volume** — data is ephemeral and will be lost on container removal. This is acceptable because Redis is used only for the Sidekiq job queue; job state is persisted in Postgres. Active in-flight jobs may be lost if Redis is removed, but queued jobs that have already been enqueued will requeue on next start.
### Migration artifact
`~/services/remetrics/remetrics_production.dump` is the Postgres dump from the 2026-06-24 migration. It is safe to move to backup storage and remove from the services directory once you have confirmed nightly backups are running cleanly.
---
## Sidekiq Health
Sidekiq runs as the `worker` service (`remetrics-worker-1`). There is no healthcheck configured on this container.
To check Sidekiq status:
```bash
# Check worker container is running
ssh jared@15.204.247.153 "docker ps --filter name=remetrics-worker"
# View recent worker logs
ssh jared@15.204.247.153 "docker logs remetrics-worker-1 --tail 50"
# Check Sidekiq queue depth (requires rails console or Sidekiq web UI)
ssh jared@15.204.247.153 "docker exec -it remetrics-remetrics-1 bundle exec rails runner 'puts Sidekiq::Queue.all.map{|q| \"#{q.name}: #{q.size}\"}'"
```
There is no Sidekiq Web UI exposed through Traefik. Access requires a rails console or direct container exec.
---
## Cron Jobs
```
# Sitemap refresh (added at OVH cutover 2026-06-24)
0 1 * * * /home/jared/services/remetrics/sitemap_refresh.sh >> /home/jared/services/remetrics/sitemap.log 2>&1
```
This replaces a broken cron that ran on the old DigitalOcean box and referenced a non-existent container.
---
## EOL Version Notes
The remetrics stack runs several components past or approaching end-of-life. These are known gaps pending a rebuild cycle:
| Component | Deployed | EOL status |
|---|---|---|
| Ruby | ~3.1 | EOL March 2025 — no upstream security patches |
| Rails | 7.0.4.2 | Past security maintenance window |
| Node.js (Dockerfile.production ARG) | 16.14.2 | EOL September 2023 |
| Strapi | 4.9.2 | ~16 minor releases behind latest 4.x; Strapi 5 is a breaking major |
| PostgreSQL | 15 | Supported until November 2027 — no urgency |
See [remetrics Update Workflow](./remetrics-update-workflow.md) for the recommended upgrade sequence when doing a rebuild.
---
## Divergences from House Style
These are known differences between remetrics and how other services are configured. They are documented here so future changes don't inadvertently "fix" them without understanding the intent:
1. **Cert resolver**: remetrics uses `cloudflare` (DNS-01) while all other services use `letsencrypt` (HTTP-01). This is intentional — DNS-01 was required for zero-downtime cutover cert pre-issue.
2. **No HTTP→HTTPS redirect router**: remetrics has no explicit port-80 router with redirect middleware. Other services (e.g., BestSolarTech) define explicit HTTP routers. Whether Traefik's global `entrypoints.web.http.redirections` covers port-80 for remetrics.io is unverified — **open question**.
3. **Restart policy**: `unless-stopped` (not `always`). Manual `docker stop` will not auto-restart the containers after a daemon restart until you run `docker compose up -d` again.
4. **Secrets hardcoded in compose**: `POSTGRES_PASSWORD`, `SECRET_KEY_BASE`, and `OPENAI_API_KEY` are cleartext inline values in `~/services/remetrics/docker-compose.yml`. Moving these to a `.env` file is a prerequisite before the compose file can be checked into version control. See [remetrics Update Workflow](./remetrics-update-workflow.md).
5. **No healthchecks**: None of the remetrics containers define Docker healthchecks. `depends_on` runs without condition checks — the db may not be ready when Rails starts. In practice puma retries on boot, but this can cause slower start times.
6. **Shared `internal` network**: The `internal` network (172.21.0.0/16) is shared server-wide. Any future service added to `internal` can reach remetrics Postgres (5432) and Redis (6379) by container DNS name. Redis has no password.
---
## Rollback to DigitalOcean
The DigitalOcean remetrics web droplet at **147.182.131.217** is currently **frozen** (apex returns 503, Sidekiq stopped) and kept as a short-term rollback. It has NOT been deleted.
To roll back to DigitalOcean:
1. **Stop new writes on OVH** — put the Rails app in maintenance mode or stop the web container:
```bash
ssh jared@15.204.247.153 "cd ~/services/remetrics && docker compose stop remetrics"
```
2. **Dump current OVH database** for any data written since cutover:
```bash
ssh jared@15.204.247.153 "docker exec remetrics-db-1 pg_dumpall -U estatioro > /tmp/remetrics-rollback.sql"
```
3. **Repoint DNS** — change the three Cloudflare A-records (`remetrics.io`, `www.remetrics.io`, `cms.remetrics.io`) from `15.204.247.153` back to `147.182.131.217`.
4. **Unfreeze the DO droplet** — restore the Sidekiq worker and any nginx config you stopped during the cutover freeze.
5. **Restore OVH data to DO** — apply the step-2 dump to the DO Postgres if any new data needs to be preserved.
The DigitalOcean droplet still holds its original data as of the 2026-06-24 cutover. After rollback, reassess whether to re-migrate or decommission the OVH containers.
**Decommission decision**: the DO droplet should be deleted once remetrics has been stable on OVH for a sufficient burn-in period (suggest 2–4 weeks). Update this doc and `migrations/migration-plan.md` when decommission is complete.
---
## Common Operations
### Restart a single remetrics container
```bash
ssh jared@15.204.247.153 "cd ~/services/remetrics && docker compose restart remetrics"
ssh jared@15.204.247.153 "cd ~/services/remetrics && docker compose restart worker"
```
### Restart Strapi
```bash
ssh jared@15.204.247.153 "cd ~/services/strapi-remetrics && docker compose restart strapi"
```
### View logs
```bash
# Rails web (last 100 lines)
ssh jared@15.204.247.153 "docker logs remetrics-remetrics-1 --tail 100"
# Sidekiq worker
ssh jared@15.204.247.153 "docker logs remetrics-worker-1 --tail 100"
# Strapi
ssh jared@15.204.247.153 "docker logs strapi-remetrics --tail 100"
```
### Run a Rails console
```bash
ssh jared@15.204.247.153 "docker exec -it remetrics-remetrics-1 bundle exec rails console"
```
### Check resource usage
```bash
ssh jared@15.204.247.153 "docker stats --no-stream remetrics-remetrics-1 remetrics-worker-1 remetrics-db-1 remetrics-redis-1 strapi-remetrics"
```
Current baseline (2026-06-25): web ~517 MiB, worker ~212 MiB, Strapi ~161 MiB. Total stack ~947 MiB of 22.9 GiB available.
---
## See Also
- [remetrics Update Workflow](./remetrics-update-workflow.md)
- [Cert Management](./cert-management.md)
- [Disaster Recovery](./disaster-recovery.md)
- [Services Inventory](./services-inventory.md)

263
docs/remetrics-update-workflow.md Normal file
View File

@ -0,0 +1,263 @@
# remetrics Update Workflow
Step-by-step runbook for deploying code changes to the remetrics.io stack on systems-prod-01.
See also:
- [remetrics Operations](./remetrics-operations.md) — stack overview, data layout, rollback
- [Services Inventory](./services-inventory.md) — container details
---
## Prerequisites — Pending Owner Action
The following one-time setup steps must be complete before the full update workflow is available. Steps marked **DONE** are complete.
### ~~PREREQ 1: Verify Bitbucket SSH access from OVH box~~ — DONE (2026-06-25)
The read-only Bitbucket deploy key is authorized and verified. The on-box source at `~/services/remetrics/app/` is in sync with `origin/master` (commit 7aa117df). `git pull` from the server works without any further setup.
### PREREQ 2: Add a `build:` stanza to the remetrics compose file
The current `~/services/remetrics/docker-compose.yml` has no `build:` section — it only references the existing local image tags (`remetrics-web:staging-baked`). Running `docker compose build` today does nothing.
Before the first rebuild, add a `build:` block to both the `remetrics` and `worker` services in the compose file:
```yaml
remetrics:
build:
context: ./app
dockerfile: Dockerfile.production
image: remetrics-web:production-YYYYMMDD
```
The `IMAGE_TAG` variable approach (using a `.env` in the compose directory) lets you build a dated tag without editing the compose file each time:
```yaml
image: remetrics-web:${IMAGE_TAG:-staging-baked}
```
Then set `IMAGE_TAG` in the environment before running `docker compose build`.
This setup should be tested in a dry run (build only, no `up`) before the next time a real update is needed.
### PREREQ 3: Move secrets out of compose (before any version control use)
`POSTGRES_PASSWORD`, `SECRET_KEY_BASE`, and `OPENAI_API_KEY` are currently hardcoded as cleartext in `~/services/remetrics/docker-compose.yml`. If the compose file is ever committed to a git repo or synced to a remote, these secrets are exposed.
Move them to `~/services/remetrics/.env` and reference via `env_file:` or `${VAR}` substitution in the compose file. This step is required before pushing compose files to any registry or version control.
### ~~PREREQ 4: Push current images to a registry (image safety net)~~ — DONE (2026-06-25)
All three images have been pushed to GHCR (private, owner: jaredswanson) as a DR snapshot:
| Tag | Also tagged | Digest |
|---|---|---|
| `ghcr.io/jaredswanson/remetrics-web:staging-baked` | `:dr-2026-06-25` | `sha256:d614b4cf…` |
| `ghcr.io/jaredswanson/remetrics-worker:staging-baked` | `:dr-2026-06-25` | same as web |
| `ghcr.io/jaredswanson/strapi-remetrics:staging` | `:dr-2026-06-25` | `sha256:4eea9dc5…` |
**Note**: `remetrics-web` and `remetrics-worker` are the **same image** (one Dockerfile build, two different container commands). They share a digest.
To pull in a DR scenario, first authenticate:
```bash
docker login ghcr.io # requires a GitHub token with read:packages scope
docker pull ghcr.io/jaredswanson/remetrics-web:dr-2026-06-25
```
Once images are in a registry, Watchtower could potentially auto-update them. Decide whether that is desired before setting this up (see [Services Inventory](./services-inventory.md) — Watchtower section).
---
## Rails App Update Workflow
**Trigger**: new commits on the Bitbucket `master` branch of `jared_/remetrics_app`.
### Step 1 — Pull latest source
```bash
ssh jared@15.204.247.153
cd ~/services/remetrics/app
git pull origin master
```
Review the log for database migrations or Gemfile.lock changes:
```bash
git log --oneline origin/master..HEAD # what just pulled
git diff HEAD~1 HEAD -- db/migrate/ # any new migrations?
git diff HEAD~1 HEAD -- Gemfile.lock # gem changes?
```
- **Gem changes** (Gemfile.lock changed): a full image rebuild is required (Step 3).
- **No gem changes, no assets changed**: you may be able to skip the rebuild and just restart (the volume mount overlays source on top of the baked image). Test this carefully — if in doubt, rebuild.
- **Asset changes** (app/assets/, app/javascript/): rebuild required; assets are precompiled at image build time.
### Step 2 — Tag the current image as rollback
Before touching anything, preserve the current image:
```bash
DATE=$(date +%Y%m%d)
docker tag remetrics-web:staging-baked remetrics-web:rollback-$DATE
docker tag remetrics-worker:staging-baked remetrics-worker:rollback-$DATE
```
Keep at most two rollback tags; remove older ones with `docker rmi`.
### Step 3 — Build new images
> **Requires PREREQ 2 (build: stanza) to be complete first.**
```bash
cd ~/services/remetrics
DATE=$(date +%Y%m%d)
docker build -f app/Dockerfile.production -t remetrics-web:production-$DATE app/
docker build -f app/Dockerfile.production -t remetrics-worker:production-$DATE app/
```
Expected build time: 3–8 minutes if Gemfile unchanged (bundle cache hit); longer if gems changed.
Update the compose file (or `.env` IMAGE_TAG variable) to reference the new dated tag before proceeding.
### Step 4 — Run database migrations
Run migrations against the live database using a one-off container before cutting over traffic. This allows aborting if migrations fail without taking the site down:
```bash
docker run --rm \
--network internal \
-e RAILS_ENV=production \
-e DATABASE_URL=postgres://estatioro:PASSWORD@remetrics-db-1/remetrics_production \
remetrics-web:production-$DATE \
bundle exec rails db:migrate
```
> **Note**: Replace `PASSWORD` with the actual value from `~/services/remetrics/docker-compose.yml`. Once PREREQ 3 is complete, this can use `--env-file ~/services/remetrics/.env` instead.
If migrations fail, abort — existing containers are still running the old code on the old schema. Investigate and fix before retrying.
### Step 5 — Restart services with new image
```bash
cd ~/services/remetrics
docker compose up -d --no-deps remetrics worker
```
`--no-deps` prevents Postgres and Redis from being touched. Compose starts new containers and stops old ones.
### Step 6 — Verify
```bash
# Check containers came up
docker compose ps
# Tail logs for errors
docker logs remetrics-remetrics-1 --tail 50
# HTTP probe
curl -sf https://remetrics.io/up && echo "OK" || echo "FAILED"
```
### Step 7 — Rollback if needed
If the new deployment is broken:
```bash
cd ~/services/remetrics
# Edit compose IMAGE_TAG back to the rollback tag, then:
docker compose up -d --no-deps remetrics worker
```
Rolling back database migrations is generally not safe. If a migration ran and the new code is being reverted, prefer a forward fix over schema rollback.
---
## Code-Only Restart (No Gem/Asset Changes)
When a git pull brings only Ruby code changes (no Gemfile.lock, no asset changes), you may be able to skip the rebuild because the bind mount overlays the source on top of the baked image:
```bash
cd ~/services/remetrics/app && git pull origin master
# Verify: no Gemfile.lock or db/migrate/ changes
cd ~/services/remetrics && docker compose restart remetrics worker
```
This is faster but less reliable — if any initialization depends on the build-time environment (precompiled assets, baked gems), the restart alone may not pick up all changes. When in doubt, do a full rebuild.
---
## Strapi Update Workflow
Strapi updates are riskier than Rails updates because of unclear image provenance.
**Day-to-day content changes** (new articles, config edits): No image update needed. All Strapi content and config live in the bind-mounted `~/services/strapi-remetrics/strapi/` directories. Changes are live immediately.
**Strapi version upgrade** (security patch, feature update):
> The current image (`strapi-remetrics:staging`) originated from a private GCP registry and is 3 years old. The compose `# Do NOT pull / rebuild` note reflects that the GCP registry is gone. To do a version upgrade you must write a Dockerfile.
1. **Back up the SQLite database first**:
```bash
ssh jared@15.204.247.153 "cp -r ~/services/strapi-remetrics/strapi/database/ ~/backups/strapi-sqlite-$(date +%Y%m%d)/"
```
2. **Write a Dockerfile** in `~/services/strapi-remetrics/` using the on-disk source:
```dockerfile
FROM node:18-alpine
WORKDIR /app
COPY strapi/package.json strapi/yarn.lock ./
RUN yarn install --frozen-lockfile
RUN yarn add @strapi-community/strapi-provider-upload-google-cloud-storage
COPY strapi/src/ ./src/
COPY strapi/config/ ./config/
RUN yarn build
CMD ["yarn", "start"]
```
3. **Add a `build:` stanza** to `~/services/strapi-remetrics/docker-compose.yml` and remove the `Do NOT pull / rebuild` comment once the Dockerfile exists.
4. **Build and tag**:
```bash
cd ~/services/strapi-remetrics
docker build -t strapi-remetrics:production-$(date +%Y%m%d) .
```
5. **Update compose and restart**:
```bash
docker compose up -d --no-deps strapi
```
6. Strapi auto-runs schema migrations on startup — no manual step needed for SQLite.
---
## Asset Precompile Note
`Dockerfile.production` runs `bin/rails assets:precompile` at build time. However, the compose file mounts `./app:/app` into the container, which overlays the bind-mount directory over `/app`. If `~/services/remetrics/app/public/assets/` exists on disk (i.e., is not gitignored and empty), the compiled assets from the image layer are shadowed by the on-disk directory.
**Recommended**: ensure `app/public/assets/` is in `.gitignore` and not present on the host (or is empty) so that the image's precompiled assets are used. Alternatively, remove the `./app:/app` bind mount entirely and COPY source into the image at build time — this makes images fully self-contained and removes the shadowing concern.
---
## Build Location
Build on the OVH server directly (no external CI needed). The server has the source, the Dockerfile, and sufficient resources (~22 GiB RAM, 8 vCores). The house style for all other services uses pre-built upstream images with no CI pipeline; introducing Bitbucket Pipelines or GitHub Actions adds registry costs and complexity not warranted for a single-server setup.
---
## Version Upgrade Priority Order
When a rebuild is scheduled, upgrade these in order (each requires a Dockerfile change):
1. **Node 16 → 20** in `Dockerfile.production` (Node 16 is EOL since Sept 2023)
2. **Strapi 4.9.2 → latest 4.x** (security patches, same major — test on staging)
3. **Rails 7.0 → 7.1 or 7.2** (security maintenance window passed — moderate effort)
4. **Ruby 3.1 → 3.3** (EOL March 2025 — requires base image change + gem compat check)
Do not jump to Strapi 5 — it is a breaking major. Get to latest 4.x first.
---
## See Also
- [remetrics Operations](./remetrics-operations.md) — day-to-day ops, rollback to DigitalOcean
- [Disaster Recovery](./disaster-recovery.md) — full restore from backup

72
docs/services-inventory.md
View File

@ -3,9 +3,14 @@
Production server: 15.204.247.153 Production server: 15.204.247.153
> Migration complete. The source DigitalOcean VPS (`hyperthrive-docker-services`, > Migration complete. The source DigitalOcean VPS (`hyperthrive-docker-services`,
> 161.35.12.75) was deleted on 2026-06-24. Every service below now runs solely > 161.35.12.75) was deleted on 2026-06-24. Services originating from that VPS now
> on this OVH host — there is no longer any DigitalOcean fallback. ("Deployed: > run solely on this OVH host with no DigitalOcean fallback. ("Deployed:
> Previous" entries were migrated from the old VPS in earlier sessions.) > Previous" entries were migrated from the old VPS in earlier sessions.)
>
> **Exception — remetrics:** `remetrics` and `strapi-remetrics` were migrated on
> 2026-06-24 from a *separate* DigitalOcean web droplet (147.182.131.217). That
> droplet is currently **frozen and kept as a short-term rollback (NOT deleted)**
> pending decommission — it is the only remaining DigitalOcean fallback.
## Active Services ## Active Services
@ -98,19 +103,70 @@ Production server: 15.204.247.153
- **Purpose**: Automated container updates - **Purpose**: Automated container updates
- **Deployed**: Previous - **Deployed**: Previous
### remetrics (Rails Application)
- **Status**: Running
- **Domain**: remetrics.io, www.remetrics.io
- **Port**: 3000 (internal), exposed via Traefik HTTPS
- **Docker Network**: traefik, internal
- **Service Directory**: ~/services/remetrics/
- **Containers**:
- `remetrics-remetrics-1` — Rails web, image `remetrics-web:staging-baked`
- `remetrics-worker-1` — Sidekiq worker, image `remetrics-worker:staging-baked`
- `remetrics-db-1` — PostgreSQL (db `remetrics_production`, user `estatioro`)
- redis (job queue / cache)
- **Database**: PostgreSQL — `remetrics_production` (480 users migrated, zero data loss)
- **Traefik Config**:
- Routers for remetrics.io / www.remetrics.io use `certresolver=cloudflare` (ACME DNS-01)
- **Notes**:
- Web and worker images are **baked** to be GitHub-independent at boot (no `bundle install` / no git fetch at startup).
- The Rails app fetches CMS content from a **HARDCODED** URL `https://cms.remetrics.io`
(in app source `strapi/article.rb` and `help_page.rb`) — NOT env-configurable.
- Sitemap refresh cron on OVH: `~/services/remetrics/sitemap_refresh.sh` at 01:00
(replaces a broken DO cron that referenced a non-existent container).
- **Migration**: From DigitalOcean web droplet 147.182.131.217 (frozen rollback, not deleted)
- **Deployed**: 2026-06-24 (migrated from DigitalOcean)
### strapi-remetrics (Strapi CMS)
- **Status**: Running
- **Domain**: cms.remetrics.io
- **Port**: 1337 (internal), exposed via Traefik HTTPS
- **Docker Network**: traefik
- **Service Directory**: ~/services/strapi-remetrics/
- **Container**: `strapi-remetrics`
- **Database**: SQLite (local)
- **Storage**: Media stored on GCS bucket `remetrics-strapi`
- **Traefik Config**:
- Router for cms.remetrics.io uses `certresolver=cloudflare` (ACME DNS-01)
- **Notes**:
- This is the CMS that the Rails `remetrics` app reads from via the hardcoded `https://cms.remetrics.io` URL.
- `strapi.remetrics.io` (a legacy hostname) remains on the old DO box and is dead/unused — NOT served from OVH.
- **Migration**: From DigitalOcean web droplet 147.182.131.217 (frozen rollback, not deleted)
- **Deployed**: 2026-06-24 (migrated from DigitalOcean)
## Automation & Utilities ## Automation & Utilities
### backup.sh ### backup.sh
- Location: ~/services/backup.sh - Location: ~/services/backup.sh
- Purpose: Automated backup of all service data - Purpose: Automated backup of all service data (auto-discovers all services)
- Frequency: Scheduled via cron - Schedule: Nightly at 02:00 EST (07:00 UTC)
- See [Disaster Recovery](./disaster-recovery.md) for restore procedures
### container-health-monitor.sh ### container-health-monitor.sh
- Location: ~/services/container-health-monitor.sh - Location: ~/services/container-health-monitor.sh
- Purpose: Monitor container health and generate alerts - Purpose: Health monitor for BestSolarTech containers only (misleading name — not server-wide)
- Frequency: Scheduled via cron - Schedule: Every 10 minutes
- See [Health Monitoring](./health-monitoring.md) for coverage details and gaps
### daily-maintenance.sh ### daily-maintenance.sh
- Location: ~/services/daily-maintenance.sh - Location: ~/services/daily-maintenance.sh
- Purpose: Daily maintenance tasks - Purpose: Daily maintenance for BestSolarTech only (misleading name — not server-wide)
- Frequency: Scheduled via cron - Schedule: Daily at 04:30 EST (moved from 03:00 on 2026-06-25 to de-collide from Watchtower)
---
## Known Gaps
### mailpace.env — UNDOCUMENTED
- Location: `~/services/mailpace.env` (in the `~/services/` root)
- An environment file for MailPace transactional email exists but no container is documented as using it.
- **TODO**: Identify which service(s) consume this env file and add them to the inventory. MailPace may be referenced by an existing service's compose file via a path like `env_file: ../mailpace.env`.

44
migrations/migration-plan.md
View File

@ -194,6 +194,12 @@ This document outlines the complete migration plan for consolidating DigitalOcea
--- ---
## Historical / Superseded — Phases 4–7
> **Note (2026-06-25)**: The migration is complete. Phases 4–7 below were the original incremental plan template. They were never executed as discrete phases — services were migrated individually as needed, culminating in the remetrics addendum below. These sections are preserved as a historical record but do not reflect actual execution steps.
---
## Phase 4: Low-Risk Service Migration ## Phase 4: Low-Risk Service Migration
**Goal**: Migrate static websites and development instances **Goal**: Migrate static websites and development instances
@ -427,3 +433,41 @@ This document outlines the complete migration plan for consolidating DigitalOcea
*Phase 1 Status: ✅ COMPLETE - All tasks successfully finished* *Phase 1 Status: ✅ COMPLETE - All tasks successfully finished*
*Phase 2 Status: ✅ COMPLETE - Docker environment ready* *Phase 2 Status: ✅ COMPLETE - Docker environment ready*
*Ready for Phase 3: Service Migration Planning* *Ready for Phase 3: Service Migration Planning*
---
## Addendum: remetrics.io Migration ✅ COMPLETE (2026-06-24)
> **Note**: This was a *separate* migration from a **different** DigitalOcean box than
> the Phases above. Phases 1-7 cover the `hyperthrive-docker-services` VPS (161.35.12.75,
> deleted 2026-06-24). remetrics lived on a standalone DigitalOcean **web droplet
> (147.182.131.217)**, unrelated to the hyperthrive box. It is recorded here as an
> addendum rather than retrofitted into a phase.
**Outcome**: `remetrics.io` (Rails app + Strapi CMS) was migrated from the DigitalOcean
web droplet (147.182.131.217) to the OVH server (15.204.247.153) on 2026-06-24.
- **Services on OVH**:
- `~/services/remetrics/` — Rails web (`remetrics-remetrics-1`), Sidekiq worker
(`remetrics-worker-1`), Postgres (`remetrics-db-1`, db `remetrics_production`), redis.
Web/worker images baked GitHub-independent (no boot-time bundle/git).
- `~/services/strapi-remetrics/` — Strapi (`strapi-remetrics`), SQLite, media on GCS
bucket `remetrics-strapi`.
- **DNS** (Cloudflare zone `remetrics.io`, grey-cloud A-records → 15.204.247.153):
`remetrics.io`, `www.remetrics.io`, `cms.remetrics.io`. `temp.remetrics.io` and the
dead/unused `strapi.remetrics.io` intentionally remain on the old DO box.
- **Cutover**: ~4m40s frozen window, **zero data loss** (480 users migrated exactly; a
503 "freeze" on the old box + final `pg_dump` captured a signup that landed seconds
before cutover).
- **TLS**: Added a second Traefik cert-resolver `cloudflare` (ACME DNS-01, storage
`/letsencrypt/acme-cloudflare.json`) alongside the existing `letsencrypt` (HTTP-01).
remetrics/cms/www routers use `certresolver=cloudflare`. Certs were **pre-issued via
DNS-01 before the DNS flip** for a zero-cert-window cutover.
- **Backups**: `~/services/backup.sh` auto-discovers both new services (remetrics Postgres
via `pg_dumpall`, Strapi SQLite via service-files tar); confirmed in nightly backup.
- **CMS coupling**: The Rails app fetches CMS from a HARDCODED `https://cms.remetrics.io`
(in `strapi/article.rb` and `help_page.rb`) — not env-configurable.
**Rollback status**: The DigitalOcean remetrics web droplet (147.182.131.217) is currently
**FROZEN** (apex returns 503, sidekiq worker stopped) and kept as a short-term rollback.
It is **NOT yet deleted** — decommission pending.

82
profile.yaml
View File

@ -1,5 +1,5 @@
# systems-prod-01 Server Profile # systems-prod-01 Server Profile
# Last Updated: 2025-09-18 # Last Updated: 2026-06-25
# Updated By: jared # Updated By: jared
server: server:
@ -50,18 +50,17 @@ security:
software: software:
docker: docker:
version: 28.4.0 version: 29.2.1
compose_version: 2.39.3 compose_version: 2.39.3
user_in_group: jared user_in_group: jared
installed_from: official_convenience_script installed_from: official_convenience_script
daemon_config: production_ready daemon_config: production_ready
networks: networks:
- web (172.20.0.0/16) - traefik (172.20.0.0/16) # external-facing; Traefik routes here
- internal (172.21.0.0/16) - internal (172.21.0.0/16) # backend communication
runtime: runtime:
- traefik:v3.0 (reverse proxy) - traefik:v3.6 (reverse proxy)
- ctop (container monitoring)
- apache2-utils (for htpasswd) - apache2-utils (for htpasswd)
system_packages: system_packages:
@ -72,36 +71,40 @@ software:
services: services:
- name: traefik - name: traefik
type: reverse_proxy type: reverse_proxy
version: v3.0 version: v3.6
status: running status: running
ports: [80, 443, 8080] ports: [80, 443]
ssl: letsencrypt ssl: letsencrypt (HTTP-01) + cloudflare (DNS-01 for remetrics)
dashboard: https://traefik.systems-prod-01.vps.ovh.us dashboard: protected (see credentials.md)
directory: /home/jared/services/traefik directory: /home/jared/services/traefik
- name: test-whoami
type: test_service
status: running
purpose: ssl_verification
monitoring: monitoring:
status: basic_configured status: partial_coverage
tools: tools:
- ctop (container monitoring) - htop (system monitoring, manual)
- htop (system monitoring) - container-health-monitor.sh (BestSolarTech-only, runs every 10 min)
- health-check.sh (system health) - daily-maintenance.sh (BestSolarTech-only, runs 4:30 AM EST — FIXED 2026-06-25)
- log-summary.sh (log analysis) - health-check.sh (all services, on-demand via syshealth — FIXED 2026-06-25)
- backup.sh (all services, runs 2 AM EST)
- watchtower (opt-in only via WATCHTOWER_LABEL_ENABLE=true — FIXED 2026-06-25, runs 3 AM)
notes:
- syshealth now works (points to ~/services/monitoring/scripts/health-check.sh, deployed 2026-06-25)
- Per-service auto-remediation only for BestSolarTech; health-check.sh covers detection for all
- See docs/health-monitoring.md for full gap list
logs: local_with_rotation logs: local_with_rotation
directory: /home/jared/services/monitoring
alerts: none alerts: none
backups: backups:
status: not_configured status: configured_and_active
strategy: TBD strategy: pg_dumpall (PostgreSQL), mysqldump (MariaDB), service-dir tar
frequency: TBD frequency: nightly at 02:00 EST (07:00 UTC)
retention: TBD retention:
location: TBD on_server: 2 days (~/backups/current/)
tested: N/A synology_nas: pulled nightly at 03:00 EST
offsite_b2: via Synology Hyper Backup
script: ~/services/backup.sh (auto-discovers all services in ~/services/)
notification: ~/.local/bin/notify_backup.rb (on success)
tested: yes (remetrics 480-user migration confirmed, 2026-06-24)
networking: networking:
vpc: N/A vpc: N/A
@ -135,23 +138,10 @@ cost:
credits_used: N/A credits_used: N/A
notes: notes:
- Server provisioned for cost optimization migration from DigitalOcean - MIGRATION COMPLETE (2026-06-24): All services migrated from DigitalOcean to OVH
- Significantly larger specs (8 vCores, 24GB RAM) vs current 1 vCore, 2GB setup - Source DigitalOcean VPS (hyperthrive-docker-services, 161.35.12.75) DELETED 2026-06-24
- Will consolidate multiple DigitalOcean services onto single server - remetrics.io migrated 2026-06-24 from separate DO droplet (147.182.131.217); that droplet FROZEN as rollback (not yet deleted)
- Next payment date: 2025-10-18 - 21 containers running as of 2026-06-25 (see docs/services-inventory.md for full list)
- Automatic renewal enabled
- Phase 1.1 completed: Custom user setup, SSH keys, ubuntu user removed
- Phase 1.2 completed: Security hardening (SSH, firewall, auto-updates)
- Phase 1.3 completed: System preparation (packages, timezone, swap, tools)
- Phase 1.4 completed: Final validation of all Phase 1 tasks
- PHASE 1 COMPLETE: Ready for Phase 2 Docker environment setup
- Phase 2.1 completed: Docker v28.4.0 and Compose v2.39.3 installed
- Phase 2.2 completed: Service directory structure at /home/jared/services/
- Phase 2.3 completed: Traefik v3.0 reverse proxy with SSL configured
- Phase 2.4 completed: Basic monitoring tools and scripts installed
- PHASE 2 COMPLETE: Docker environment ready for service deployment
- Hostname mismatch: shows "vps-5d79b132" instead of "systems-prod-01"
- Passwordless sudo configured for automation - Passwordless sudo configured for automation
- 4GB swap file active with optimal swappiness setting - 4GB swap file active
- Traefik dashboard: admin / dH1CtZuVLBUMig7P - Historical phase notes (Phases 1-7 setup) moved to migrations/migration-plan.md
- Directory structure: /home/jared/services/{traefik,web,databases,automation,monitoring,shared,templates}