server-ovh-prod-01-main/docs/remetrics-operations.md

# remetrics Operations

Runbook for the remetrics.io production stack running on systems-prod-01 (OVH, 15.204.247.153).

See also:
- [remetrics Update Workflow](./remetrics-update-workflow.md) — how to deploy code changes
- [Cert Management](./cert-management.md) — Cloudflare DNS-01 resolver used by this stack
- [Services Inventory](./services-inventory.md) — container listing
- [Disaster Recovery](./disaster-recovery.md) — backup restore procedures

---

## Stack Overview

remetrics is two separate Compose projects:

| Project | Directory | Containers | Purpose |
|---|---|---|---|
| `remetrics` | `~/services/remetrics/` | web, worker, postgres, redis | Rails app + Sidekiq |
| `strapi-remetrics` | `~/services/strapi-remetrics/` | strapi | Headless CMS |

### Container summary

| Container | Image | Networks |
|---|---|---|
| `remetrics-remetrics-1` | `remetrics-web:staging-baked` | `traefik`, `internal` |
| `remetrics-worker-1` | `remetrics-worker:staging-baked` | `internal` only |
| `remetrics-db-1` | `postgres:15` | `internal` only |
| `remetrics-redis-1` | `redis:alpine` | `internal` only |
| `strapi-remetrics` | `strapi-remetrics:staging` | `traefik`, `strapi-remetrics_default` |

The Rails web and worker images (`staging-baked`) were built during the 2026-06-24 migration via a `docker commit`-style bake. The primary copy is on this server; a DR backup was pushed to GHCR on 2026-06-25:
- `ghcr.io/jaredswanson/remetrics-web:staging-baked` / `:dr-2026-06-25`
- `ghcr.io/jaredswanson/remetrics-worker:staging-baked` / `:dr-2026-06-25` (same image as web — one build, two commands)

GHCR is private; pulling requires `docker login ghcr.io` with a token that has `read:packages`. See [remetrics Update Workflow](./remetrics-update-workflow.md) for how to rebuild them.

The Strapi image (`strapi-remetrics:staging`) originated from a private GCP registry but is now a local tag, also backed up to GHCR (`ghcr.io/jaredswanson/strapi-remetrics:staging` / `:dr-2026-06-25`). The compose file carries a `# Do NOT pull / rebuild` comment because that GCP registry is gone. See the Strapi section of the update workflow before attempting any Strapi image changes.

---

## Domains and TLS

| Domain | Service | Cert Resolver |
|---|---|---|
| `remetrics.io` | Rails web | `cloudflare` (DNS-01) |
| `www.remetrics.io` | Rails web | `cloudflare` (DNS-01) |
| `cms.remetrics.io` | Strapi CMS | `cloudflare` (DNS-01) |

All three domains use the `cloudflare` ACME resolver (DNS-01), not the default `letsencrypt` HTTP-01 resolver used by other services. DNS-01 was required to pre-issue certificates before the DNS cutover window. See [Cert Management](./cert-management.md) for where certs are stored and how to check expiry.

---

## CMS URL — Hardcoded Coupling

**Important**: The Rails app fetches CMS content from a **hardcoded** URL:

```
https://cms.remetrics.io
```

This is not env-configurable. It is set directly in the app source at:
- `app/models/strapi/article.rb`
- `app/models/strapi/help_page.rb`

**Consequence**: If `strapi-remetrics` is down or `cms.remetrics.io` is unreachable, the Rails web process will fail to render any CMS-backed pages. Strapi and Rails must be kept running together. This also means that renaming the CMS domain or moving Strapi to a new host requires a code change and image rebuild — not just a config change.

---

## Image Provenance

Source repository for the Rails app: `git@bitbucket.org:jared_/remetrics_app.git`

The on-disk source is at `~/services/remetrics/app/` (bind-mounted into both web and worker containers at `/app`). This is a live git repo tracking the Bitbucket remote. The Bitbucket read-only deploy key is configured and verified; on-box source is in sync with `origin/master` (commit 7aa117df as of 2026-06-25).

The Strapi source (package.json, src/, config/) lives at `~/services/strapi-remetrics/strapi/`. There is no separate Bitbucket repo for Strapi; the source files are the canonical copy on this server.

---

## Data Layout

### Rails database
- Container: `remetrics-db-1` (postgres:15)
- Database name: `remetrics_production`
- User: `estatioro`
- Storage: Docker named volume `remetrics_remetrics_pgdata`
- 480 users migrated from DigitalOcean (zero data loss, 2026-06-24)

The named volume is backed up nightly by `~/services/backup.sh` via `pg_dumpall`. Confirm with:
```bash
ssh jared@15.204.247.153 "ls -lh ~/backups/current/"
```

### Strapi database
- Engine: SQLite
- File location: `~/services/strapi-remetrics/strapi/database/` (bind-mounted)
- Backed up nightly as part of the strapi-remetrics service-files tar

### Strapi media uploads
- Storage: GCS bucket `remetrics-strapi`
- GCS service account key: `~/services/strapi-remetrics/strapi/gcs/` (bind-mounted)
- Not backed up by backup.sh (lives in GCS, not on-server)

### Redis
Redis (`redis:alpine`) has **no named volume** — data is ephemeral and will be lost on container removal. This is acceptable because Redis is used only for the Sidekiq job queue; job state is persisted in Postgres. Active in-flight jobs may be lost if Redis is removed, but queued jobs that have already been enqueued will requeue on next start.

### Migration artifact
`~/services/remetrics/remetrics_production.dump` is the Postgres dump from the 2026-06-24 migration. It is safe to move to backup storage and remove from the services directory once you have confirmed nightly backups are running cleanly.

---

## Sidekiq Health

Sidekiq runs as the `worker` service (`remetrics-worker-1`). There is no healthcheck configured on this container.

To check Sidekiq status:
```bash
# Check worker container is running
ssh jared@15.204.247.153 "docker ps --filter name=remetrics-worker"

# View recent worker logs
ssh jared@15.204.247.153 "docker logs remetrics-worker-1 --tail 50"

# Check Sidekiq queue depth (requires rails console or Sidekiq web UI)
ssh jared@15.204.247.153 "docker exec -it remetrics-remetrics-1 bundle exec rails runner 'puts Sidekiq::Queue.all.map{|q| \"#{q.name}: #{q.size}\"}'"
```

There is no Sidekiq Web UI exposed through Traefik. Access requires a rails console or direct container exec.

---

## Cron Jobs

```
# Sitemap refresh (added at OVH cutover 2026-06-24)
0 1 * * * /home/jared/services/remetrics/sitemap_refresh.sh >> /home/jared/services/remetrics/sitemap.log 2>&1
```

This replaces a broken cron that ran on the old DigitalOcean box and referenced a non-existent container.

---

## EOL Version Notes

The remetrics stack runs several components past or approaching end-of-life. These are known gaps pending a rebuild cycle:

| Component | Deployed | EOL status |
|---|---|---|
| Ruby | ~3.1 | EOL March 2025 — no upstream security patches |
| Rails | 7.0.4.2 | Past security maintenance window |
| Node.js (Dockerfile.production ARG) | 16.14.2 | EOL September 2023 |
| Strapi | 4.9.2 | ~16 minor releases behind latest 4.x; Strapi 5 is a breaking major |
| PostgreSQL | 15 | Supported until November 2027 — no urgency |

See [remetrics Update Workflow](./remetrics-update-workflow.md) for the recommended upgrade sequence when doing a rebuild.

---

## Divergences from House Style

These are known differences between remetrics and how other services are configured. They are documented here so future changes don't inadvertently "fix" them without understanding the intent:

1. **Cert resolver**: remetrics uses `cloudflare` (DNS-01) while all other services use `letsencrypt` (HTTP-01). This is intentional — DNS-01 was required for zero-downtime cutover cert pre-issue.
2. **No HTTP→HTTPS redirect router**: remetrics has no explicit port-80 router with redirect middleware. Other services (e.g., BestSolarTech) define explicit HTTP routers. Whether Traefik's global `entrypoints.web.http.redirections` covers port-80 for remetrics.io is unverified — **open question**.
3. **Restart policy**: `unless-stopped` (not `always`). Manual `docker stop` will not auto-restart the containers after a daemon restart until you run `docker compose up -d` again.
4. **Secrets hardcoded in compose**: `POSTGRES_PASSWORD`, `SECRET_KEY_BASE`, and `OPENAI_API_KEY` are cleartext inline values in `~/services/remetrics/docker-compose.yml`. Moving these to a `.env` file is a prerequisite before the compose file can be checked into version control. See [remetrics Update Workflow](./remetrics-update-workflow.md).
5. **No healthchecks**: None of the remetrics containers define Docker healthchecks. `depends_on` runs without condition checks — the db may not be ready when Rails starts. In practice puma retries on boot, but this can cause slower start times.
6. **Shared `internal` network**: The `internal` network (172.21.0.0/16) is shared server-wide. Any future service added to `internal` can reach remetrics Postgres (5432) and Redis (6379) by container DNS name. Redis has no password.

---

## Rollback to DigitalOcean

The DigitalOcean remetrics web droplet at **147.182.131.217** is currently **frozen** (apex returns 503, Sidekiq stopped) and kept as a short-term rollback. It has NOT been deleted.

To roll back to DigitalOcean:
1. **Stop new writes on OVH** — put the Rails app in maintenance mode or stop the web container:
   ```bash
   ssh jared@15.204.247.153 "cd ~/services/remetrics && docker compose stop remetrics"
   ```
2. **Dump current OVH database** for any data written since cutover:
   ```bash
   ssh jared@15.204.247.153 "docker exec remetrics-db-1 pg_dumpall -U estatioro > /tmp/remetrics-rollback.sql"
   ```
3. **Repoint DNS** — change the three Cloudflare A-records (`remetrics.io`, `www.remetrics.io`, `cms.remetrics.io`) from `15.204.247.153` back to `147.182.131.217`.
4. **Unfreeze the DO droplet** — restore the Sidekiq worker and any nginx config you stopped during the cutover freeze.
5. **Restore OVH data to DO** — apply the step-2 dump to the DO Postgres if any new data needs to be preserved.

The DigitalOcean droplet still holds its original data as of the 2026-06-24 cutover. After rollback, reassess whether to re-migrate or decommission the OVH containers.

**Decommission decision**: the DO droplet should be deleted once remetrics has been stable on OVH for a sufficient burn-in period (suggest 2–4 weeks). Update this doc and `migrations/migration-plan.md` when decommission is complete.

---

## Common Operations

### Restart a single remetrics container

```bash
ssh jared@15.204.247.153 "cd ~/services/remetrics && docker compose restart remetrics"
ssh jared@15.204.247.153 "cd ~/services/remetrics && docker compose restart worker"
```

### Restart Strapi

```bash
ssh jared@15.204.247.153 "cd ~/services/strapi-remetrics && docker compose restart strapi"
```

### View logs

```bash
# Rails web (last 100 lines)
ssh jared@15.204.247.153 "docker logs remetrics-remetrics-1 --tail 100"

# Sidekiq worker
ssh jared@15.204.247.153 "docker logs remetrics-worker-1 --tail 100"

# Strapi
ssh jared@15.204.247.153 "docker logs strapi-remetrics --tail 100"
```

### Run a Rails console

```bash
ssh jared@15.204.247.153 "docker exec -it remetrics-remetrics-1 bundle exec rails console"
```

### Check resource usage

```bash
ssh jared@15.204.247.153 "docker stats --no-stream remetrics-remetrics-1 remetrics-worker-1 remetrics-db-1 remetrics-redis-1 strapi-remetrics"
```

Current baseline (2026-06-25): web ~517 MiB, worker ~212 MiB, Strapi ~161 MiB. Total stack ~947 MiB of 22.9 GiB available.

---

## See Also

- [remetrics Update Workflow](./remetrics-update-workflow.md)
- [Cert Management](./cert-management.md)
- [Disaster Recovery](./disaster-recovery.md)
- [Services Inventory](./services-inventory.md)