jared
/
server-ovh-prod-01-main
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
server-ovh-prod-01-main/docs/remetrics-operations.md

11 KiB
Raw Blame History

remetrics Operations

Runbook for the remetrics.io production stack running on systems-prod-01 (OVH, 15.204.247.153).

See also:

Stack Overview

remetrics is two separate Compose projects:

Project Directory Containers Purpose
remetrics ~/services/remetrics/ web, worker, postgres, redis Rails app + Sidekiq
strapi-remetrics ~/services/strapi-remetrics/ strapi Headless CMS

Container summary

Container Image Networks
remetrics-remetrics-1 remetrics-web:staging-baked traefik, internal
remetrics-worker-1 remetrics-worker:staging-baked internal only
remetrics-db-1 postgres:15 internal only
remetrics-redis-1 redis:alpine internal only
strapi-remetrics strapi-remetrics:staging traefik, strapi-remetrics_default

The Rails web and worker images (staging-baked) were built during the 2026-06-24 migration via a docker commit-style bake. The primary copy is on this server; a DR backup was pushed to GHCR on 2026-06-25:

  • ghcr.io/jaredswanson/remetrics-web:staging-baked / :dr-2026-06-25
  • ghcr.io/jaredswanson/remetrics-worker:staging-baked / :dr-2026-06-25 (same image as web — one build, two commands)

GHCR is private; pulling requires docker login ghcr.io with a token that has read:packages. See remetrics Update Workflow for how to rebuild them.

The Strapi image (strapi-remetrics:staging) originated from a private GCP registry but is now a local tag, also backed up to GHCR (ghcr.io/jaredswanson/strapi-remetrics:staging / :dr-2026-06-25). The compose file carries a # Do NOT pull / rebuild comment because that GCP registry is gone. See the Strapi section of the update workflow before attempting any Strapi image changes.

Domains and TLS

Domain Service Cert Resolver
remetrics.io Rails web cloudflare (DNS-01)
www.remetrics.io Rails web cloudflare (DNS-01)
cms.remetrics.io Strapi CMS cloudflare (DNS-01)

All three domains use the cloudflare ACME resolver (DNS-01), not the default letsencrypt HTTP-01 resolver used by other services. DNS-01 was required to pre-issue certificates before the DNS cutover window. See Cert Management for where certs are stored and how to check expiry.

CMS URL — Hardcoded Coupling

Important: The Rails app fetches CMS content from a hardcoded URL:

https://cms.remetrics.io

This is not env-configurable. It is set directly in the app source at:

  • app/models/strapi/article.rb
  • app/models/strapi/help_page.rb

Consequence: If strapi-remetrics is down or cms.remetrics.io is unreachable, the Rails web process will fail to render any CMS-backed pages. Strapi and Rails must be kept running together. This also means that renaming the CMS domain or moving Strapi to a new host requires a code change and image rebuild — not just a config change.

Image Provenance

Source repository for the Rails app: git@bitbucket.org:jared_/remetrics_app.git

The on-disk source is at ~/services/remetrics/app/ (bind-mounted into both web and worker containers at /app). This is a live git repo tracking the Bitbucket remote. The Bitbucket read-only deploy key is configured and verified; on-box source is in sync with origin/master (commit 7aa117df as of 2026-06-25).

The Strapi source (package.json, src/, config/) lives at ~/services/strapi-remetrics/strapi/. There is no separate Bitbucket repo for Strapi; the source files are the canonical copy on this server.

Data Layout

Rails database

  • Container: remetrics-db-1 (postgres:15)
  • Database name: remetrics_production
  • User: estatioro
  • Storage: Docker named volume remetrics_remetrics_pgdata
  • 480 users migrated from DigitalOcean (zero data loss, 2026-06-24)

The named volume is backed up nightly by ~/services/backup.sh via pg_dumpall. Confirm with:

ssh jared@15.204.247.153 "ls -lh ~/backups/current/"

Strapi database

  • Engine: SQLite
  • File location: ~/services/strapi-remetrics/strapi/database/ (bind-mounted)
  • Backed up nightly as part of the strapi-remetrics service-files tar

Strapi media uploads

  • Storage: GCS bucket remetrics-strapi
  • GCS service account key: ~/services/strapi-remetrics/strapi/gcs/ (bind-mounted)
  • Not backed up by backup.sh (lives in GCS, not on-server)

Redis

Redis (redis:alpine) has no named volume — data is ephemeral and will be lost on container removal. This is acceptable because Redis is used only for the Sidekiq job queue; job state is persisted in Postgres. Active in-flight jobs may be lost if Redis is removed, but queued jobs that have already been enqueued will requeue on next start.

Migration artifact

~/services/remetrics/remetrics_production.dump is the Postgres dump from the 2026-06-24 migration. It is safe to move to backup storage and remove from the services directory once you have confirmed nightly backups are running cleanly.

Sidekiq Health

Sidekiq runs as the worker service (remetrics-worker-1). There is no healthcheck configured on this container.

To check Sidekiq status:

# Check worker container is running
ssh jared@15.204.247.153 "docker ps --filter name=remetrics-worker"

# View recent worker logs
ssh jared@15.204.247.153 "docker logs remetrics-worker-1 --tail 50"

# Check Sidekiq queue depth (requires rails console or Sidekiq web UI)
ssh jared@15.204.247.153 "docker exec -it remetrics-remetrics-1 bundle exec rails runner 'puts Sidekiq::Queue.all.map{|q| \"#{q.name}: #{q.size}\"}'"

There is no Sidekiq Web UI exposed through Traefik. Access requires a rails console or direct container exec.

Cron Jobs

# Sitemap refresh (added at OVH cutover 2026-06-24)
0 1 * * * /home/jared/services/remetrics/sitemap_refresh.sh >> /home/jared/services/remetrics/sitemap.log 2>&1

This replaces a broken cron that ran on the old DigitalOcean box and referenced a non-existent container.

EOL Version Notes

The remetrics stack runs several components past or approaching end-of-life. These are known gaps pending a rebuild cycle:

Component Deployed EOL status
Ruby ~3.1 EOL March 2025 — no upstream security patches
Rails 7.0.4.2 Past security maintenance window
Node.js (Dockerfile.production ARG) 16.14.2 EOL September 2023
Strapi 4.9.2 ~16 minor releases behind latest 4.x; Strapi 5 is a breaking major
PostgreSQL 15 Supported until November 2027 — no urgency

See remetrics Update Workflow for the recommended upgrade sequence when doing a rebuild.

Divergences from House Style

These are known differences between remetrics and how other services are configured. They are documented here so future changes don't inadvertently "fix" them without understanding the intent:

  1. Cert resolver: remetrics uses cloudflare (DNS-01) while all other services use letsencrypt (HTTP-01). This is intentional — DNS-01 was required for zero-downtime cutover cert pre-issue.
  2. No HTTP→HTTPS redirect router: remetrics has no explicit port-80 router with redirect middleware. Other services (e.g., BestSolarTech) define explicit HTTP routers. Whether Traefik's global entrypoints.web.http.redirections covers port-80 for remetrics.io is unverified — open question.
  3. Restart policy: unless-stopped (not always). Manual docker stop will not auto-restart the containers after a daemon restart until you run docker compose up -d again.
  4. Secrets hardcoded in compose: POSTGRES_PASSWORD, SECRET_KEY_BASE, and OPENAI_API_KEY are cleartext inline values in ~/services/remetrics/docker-compose.yml. Moving these to a .env file is a prerequisite before the compose file can be checked into version control. See remetrics Update Workflow.
  5. No healthchecks: None of the remetrics containers define Docker healthchecks. depends_on runs without condition checks — the db may not be ready when Rails starts. In practice puma retries on boot, but this can cause slower start times.
  6. Shared internal network: The internal network (172.21.0.0/16) is shared server-wide. Any future service added to internal can reach remetrics Postgres (5432) and Redis (6379) by container DNS name. Redis has no password.

Rollback to DigitalOcean

The DigitalOcean remetrics web droplet at 147.182.131.217 is currently frozen (apex returns 503, Sidekiq stopped) and kept as a short-term rollback. It has NOT been deleted.

To roll back to DigitalOcean:

  1. Stop new writes on OVH — put the Rails app in maintenance mode or stop the web container: 
    ssh jared@15.204.247.153 "cd ~/services/remetrics && docker compose stop remetrics"
  2. Dump current OVH database for any data written since cutover: 
    ssh jared@15.204.247.153 "docker exec remetrics-db-1 pg_dumpall -U estatioro > /tmp/remetrics-rollback.sql"
  3. Repoint DNS — change the three Cloudflare A-records (remetrics.io, www.remetrics.io, cms.remetrics.io) from 15.204.247.153 back to 147.182.131.217.
  4. Unfreeze the DO droplet — restore the Sidekiq worker and any nginx config you stopped during the cutover freeze.
  5. Restore OVH data to DO — apply the step-2 dump to the DO Postgres if any new data needs to be preserved.

The DigitalOcean droplet still holds its original data as of the 2026-06-24 cutover. After rollback, reassess whether to re-migrate or decommission the OVH containers.

Decommission decision: the DO droplet should be deleted once remetrics has been stable on OVH for a sufficient burn-in period (suggest 2–4 weeks). Update this doc and migrations/migration-plan.md when decommission is complete.

Common Operations

Restart a single remetrics container

ssh jared@15.204.247.153 "cd ~/services/remetrics && docker compose restart remetrics"
ssh jared@15.204.247.153 "cd ~/services/remetrics && docker compose restart worker"

Restart Strapi

ssh jared@15.204.247.153 "cd ~/services/strapi-remetrics && docker compose restart strapi"

View logs

# Rails web (last 100 lines)
ssh jared@15.204.247.153 "docker logs remetrics-remetrics-1 --tail 100"

# Sidekiq worker
ssh jared@15.204.247.153 "docker logs remetrics-worker-1 --tail 100"

# Strapi
ssh jared@15.204.247.153 "docker logs strapi-remetrics --tail 100"

Run a Rails console

ssh jared@15.204.247.153 "docker exec -it remetrics-remetrics-1 bundle exec rails console"

Check resource usage

ssh jared@15.204.247.153 "docker stats --no-stream remetrics-remetrics-1 remetrics-worker-1 remetrics-db-1 remetrics-redis-1 strapi-remetrics"

Current baseline (2026-06-25): web ~517 MiB, worker ~212 MiB, Strapi ~161 MiB. Total stack ~947 MiB of 22.9 GiB available.

See Also