11 KiB

Raw Blame History

remetrics Update Workflow

Step-by-step runbook for deploying code changes to the remetrics.io stack on systems-prod-01.

Prerequisites — Pending Owner Action

The following one-time setup steps must be complete before the full update workflow is available. Steps marked DONE are complete.

PREREQ 1: Verify Bitbucket SSH access from OVH box — DONE (2026-06-25)

The read-only Bitbucket deploy key is authorized and verified. The on-box source at ~/services/remetrics/app/ is in sync with origin/master (commit 7aa117df). git pull from the server works without any further setup.

PREREQ 2: Add a `build:` stanza to the remetrics compose file

The current ~/services/remetrics/docker-compose.yml has no build: section — it only references the existing local image tags (remetrics-web:staging-baked). Running docker compose build today does nothing.

Before the first rebuild, add a build: block to both the remetrics and worker services in the compose file:

remetrics:
  build:
    context: ./app
    dockerfile: Dockerfile.production
  image: remetrics-web:production-YYYYMMDD

The IMAGE_TAG variable approach (using a .env in the compose directory) lets you build a dated tag without editing the compose file each time:

  image: remetrics-web:${IMAGE_TAG:-staging-baked}

Then set IMAGE_TAG in the environment before running docker compose build.

This setup should be tested in a dry run (build only, no up) before the next time a real update is needed.

PREREQ 3: Move secrets out of compose — DONE (2026-06-25)

POSTGRES_PASSWORD, SECRET_KEY_BASE, and OPENAI_API_KEY have been extracted from inline cleartext in ~/services/remetrics/docker-compose.yml to a gitignored ~/services/remetrics/.env (mode 600), referenced via ${VAR} interpolation. Verified byte-identical resolved config; zero container disruption. Backup: docker-compose.yml.bak-2026-06-25.

Note: strapi-remetrics already used env_file: ./strapi/.env — its pattern was already correct; no change was needed there. The ~/services/remetrics/.env file now EXISTS on the server.

PREREQ 4: Push current images to a registry (image safety net) — DONE (2026-06-25)

All three images have been pushed to GHCR (private, owner: jaredswanson) as a DR snapshot:

Tag	Also tagged	Digest
`ghcr.io/jaredswanson/remetrics-web:staging-baked`	`:dr-2026-06-25`	`sha256:d614b4cf…`
`ghcr.io/jaredswanson/remetrics-worker:staging-baked`	`:dr-2026-06-25`	same as web
`ghcr.io/jaredswanson/strapi-remetrics:staging`	`:dr-2026-06-25`	`sha256:4eea9dc5…`

Note: remetrics-web and remetrics-worker are the same image (one Dockerfile build, two different container commands). They share a digest.

To pull in a DR scenario, first authenticate:

docker login ghcr.io  # requires a GitHub token with read:packages scope
docker pull ghcr.io/jaredswanson/remetrics-web:dr-2026-06-25

Once images are in a registry, Watchtower could potentially auto-update them. Decide whether that is desired before setting this up (see Services Inventory — Watchtower section).

Rails App Update Workflow

Trigger: new commits on the Bitbucket master branch of jared_/remetrics_app.

Step 1 — Pull latest source

ssh jared@15.204.247.153
cd ~/services/remetrics/app
git pull origin master

Review the log for database migrations or Gemfile.lock changes:

git log --oneline origin/master..HEAD  # what just pulled
git diff HEAD~1 HEAD -- db/migrate/    # any new migrations?
git diff HEAD~1 HEAD -- Gemfile.lock   # gem changes?

Gem changes (Gemfile.lock changed): a full image rebuild is required (Step 3).
No gem changes, no assets changed: you may be able to skip the rebuild and just restart (the volume mount overlays source on top of the baked image). Test this carefully — if in doubt, rebuild.
Asset changes (app/assets/, app/javascript/): rebuild required; assets are precompiled at image build time.

Step 2 — Tag the current image as rollback

Before touching anything, preserve the current image:

DATE=$(date +%Y%m%d)
docker tag remetrics-web:staging-baked remetrics-web:rollback-$DATE
docker tag remetrics-worker:staging-baked remetrics-worker:rollback-$DATE

Keep at most two rollback tags; remove older ones with docker rmi.

Step 3 — Build new images

Requires PREREQ 2 (build: stanza) to be complete first.

cd ~/services/remetrics
DATE=$(date +%Y%m%d)
docker build -f app/Dockerfile.production -t remetrics-web:production-$DATE app/
docker build -f app/Dockerfile.production -t remetrics-worker:production-$DATE app/

Expected build time: 3–8 minutes if Gemfile unchanged (bundle cache hit); longer if gems changed.

Update the compose file (or .env IMAGE_TAG variable) to reference the new dated tag before proceeding.

Step 4 — Run database migrations

Run migrations against the live database using a one-off container before cutting over traffic. This allows aborting if migrations fail without taking the site down:

docker run --rm \
  --network internal \
  --env-file ~/services/remetrics/.env \
  -e RAILS_ENV=production \
  remetrics-web:production-$DATE \
  bundle exec rails db:migrate

Note: ~/services/remetrics/.env contains POSTGRES_PASSWORD and the other secrets — use --env-file to pass them without hardcoding values in the command. The DATABASE_URL is constructed from these vars inside the Rails app.

If migrations fail, abort — existing containers are still running the old code on the old schema. Investigate and fix before retrying.

Step 5 — Restart services with new image

cd ~/services/remetrics
docker compose up -d --no-deps remetrics worker

--no-deps prevents Postgres and Redis from being touched. Compose starts new containers and stops old ones.

Step 6 — Verify

# Check containers came up
docker compose ps

# Tail logs for errors
docker logs remetrics-remetrics-1 --tail 50

# HTTP probe
curl -sf https://remetrics.io/up && echo "OK" || echo "FAILED"

Step 7 — Rollback if needed

If the new deployment is broken:

cd ~/services/remetrics
# Edit compose IMAGE_TAG back to the rollback tag, then:
docker compose up -d --no-deps remetrics worker

Rolling back database migrations is generally not safe. If a migration ran and the new code is being reverted, prefer a forward fix over schema rollback.

Code-Only Restart (No Gem/Asset Changes)

When a git pull brings only Ruby code changes (no Gemfile.lock, no asset changes), you may be able to skip the rebuild because the bind mount overlays the source on top of the baked image:

cd ~/services/remetrics/app && git pull origin master
# Verify: no Gemfile.lock or db/migrate/ changes
cd ~/services/remetrics && docker compose restart remetrics worker

This is faster but less reliable — if any initialization depends on the build-time environment (precompiled assets, baked gems), the restart alone may not pick up all changes. When in doubt, do a full rebuild.

Strapi Update Workflow

Strapi updates are riskier than Rails updates because of unclear image provenance.

Day-to-day content changes (new articles, config edits): No image update needed. All Strapi content and config live in the bind-mounted ~/services/strapi-remetrics/strapi/ directories. Changes are live immediately.

Strapi version upgrade (security patch, feature update):

The current image (strapi-remetrics:staging) originated from a private GCP registry and is 3 years old. The compose # Do NOT pull / rebuild note reflects that the GCP registry is gone. To do a version upgrade you must write a Dockerfile.

Back up the SQLite database first:

ssh jared@15.204.247.153 "cp -r ~/services/strapi-remetrics/strapi/database/ ~/backups/strapi-sqlite-$(date +%Y%m%d)/"

Write a Dockerfile in ~/services/strapi-remetrics/ using the on-disk source:

FROM node:18-alpine
WORKDIR /app
COPY strapi/package.json strapi/yarn.lock ./
RUN yarn install --frozen-lockfile
RUN yarn add @strapi-community/strapi-provider-upload-google-cloud-storage
COPY strapi/src/ ./src/
COPY strapi/config/ ./config/
RUN yarn build
CMD ["yarn", "start"]

Add a build: stanza to ~/services/strapi-remetrics/docker-compose.yml and remove the Do NOT pull / rebuild comment once the Dockerfile exists.

Build and tag:

cd ~/services/strapi-remetrics
docker build -t strapi-remetrics:production-$(date +%Y%m%d) .

Update compose and restart:
```
docker compose up -d --no-deps strapi
```
Strapi auto-runs schema migrations on startup — no manual step needed for SQLite.

Asset Precompile Note

Dockerfile.production runs bin/rails assets:precompile at build time. However, the compose file mounts ./app:/app into the container, which overlays the bind-mount directory over /app. If ~/services/remetrics/app/public/assets/ exists on disk (i.e., is not gitignored and empty), the compiled assets from the image layer are shadowed by the on-disk directory.

Recommended: ensure app/public/assets/ is in .gitignore and not present on the host (or is empty) so that the image's precompiled assets are used. Alternatively, remove the ./app:/app bind mount entirely and COPY source into the image at build time — this makes images fully self-contained and removes the shadowing concern.

Build Location

Build on the OVH server directly (no external CI needed). The server has the source, the Dockerfile, and sufficient resources (~22 GiB RAM, 8 vCores). The house style for all other services uses pre-built upstream images with no CI pipeline; introducing Bitbucket Pipelines or GitHub Actions adds registry costs and complexity not warranted for a single-server setup.

Version Upgrade Priority Order

When a rebuild is scheduled, upgrade these in order (each requires a Dockerfile change):

Node 16 → 20 in Dockerfile.production (Node 16 is EOL since Sept 2023)
Strapi 4.9.2 → latest 4.x (security patches, same major — test on staging)
Rails 7.0 → 7.1 or 7.2 (security maintenance window passed — moderate effort)
Ruby 3.1 → 3.3 (EOL March 2025 — requires base image change + gem compat check)

Do not jump to Strapi 5 — it is a breaking major. Get to latest 4.x first.

11 KiB Raw Blame History Unescape Escape