server-ovh-prod-01-main/docs/remetrics-update-workflow.md

# remetrics Update Workflow

Step-by-step runbook for deploying code changes to the remetrics.io stack on systems-prod-01.

See also:
- [remetrics Operations](./remetrics-operations.md) — stack overview, data layout, rollback
- [Services Inventory](./services-inventory.md) — container details

---

## Prerequisites — Pending Owner Action

The following one-time setup steps must be complete before the full update workflow is available. Steps marked **DONE** are complete.

### ~~PREREQ 1: Verify Bitbucket SSH access from OVH box~~ — DONE (2026-06-25)

The read-only Bitbucket deploy key is authorized and verified. The on-box source at `~/services/remetrics/app/` is in sync with `origin/master` (commit 7aa117df). `git pull` from the server works without any further setup.

### PREREQ 2: Add a `build:` stanza to the remetrics compose file

The current `~/services/remetrics/docker-compose.yml` has no `build:` section — it only references the existing local image tags (`remetrics-web:staging-baked`). Running `docker compose build` today does nothing.

Before the first rebuild, add a `build:` block to both the `remetrics` and `worker` services in the compose file:

```yaml
remetrics:
  build:
    context: ./app
    dockerfile: Dockerfile.production
  image: remetrics-web:production-YYYYMMDD
```

The `IMAGE_TAG` variable approach (using a `.env` in the compose directory) lets you build a dated tag without editing the compose file each time:

```yaml
  image: remetrics-web:${IMAGE_TAG:-staging-baked}
```

Then set `IMAGE_TAG` in the environment before running `docker compose build`.

This setup should be tested in a dry run (build only, no `up`) before the next time a real update is needed.

### PREREQ 3: Move secrets out of compose (before any version control use)

`POSTGRES_PASSWORD`, `SECRET_KEY_BASE`, and `OPENAI_API_KEY` are currently hardcoded as cleartext in `~/services/remetrics/docker-compose.yml`. If the compose file is ever committed to a git repo or synced to a remote, these secrets are exposed.

Move them to `~/services/remetrics/.env` and reference via `env_file:` or `${VAR}` substitution in the compose file. This step is required before pushing compose files to any registry or version control.

### ~~PREREQ 4: Push current images to a registry (image safety net)~~ — DONE (2026-06-25)

All three images have been pushed to GHCR (private, owner: jaredswanson) as a DR snapshot:

| Tag | Also tagged | Digest |
|---|---|---|
| `ghcr.io/jaredswanson/remetrics-web:staging-baked` | `:dr-2026-06-25` | `sha256:d614b4cf…` |
| `ghcr.io/jaredswanson/remetrics-worker:staging-baked` | `:dr-2026-06-25` | same as web |
| `ghcr.io/jaredswanson/strapi-remetrics:staging` | `:dr-2026-06-25` | `sha256:4eea9dc5…` |

**Note**: `remetrics-web` and `remetrics-worker` are the **same image** (one Dockerfile build, two different container commands). They share a digest.

To pull in a DR scenario, first authenticate:
```bash
docker login ghcr.io  # requires a GitHub token with read:packages scope
docker pull ghcr.io/jaredswanson/remetrics-web:dr-2026-06-25
```

Once images are in a registry, Watchtower could potentially auto-update them. Decide whether that is desired before setting this up (see [Services Inventory](./services-inventory.md) — Watchtower section).

---

## Rails App Update Workflow

**Trigger**: new commits on the Bitbucket `master` branch of `jared_/remetrics_app`.

### Step 1 — Pull latest source

```bash
ssh jared@15.204.247.153
cd ~/services/remetrics/app
git pull origin master
```

Review the log for database migrations or Gemfile.lock changes:
```bash
git log --oneline origin/master..HEAD  # what just pulled
git diff HEAD~1 HEAD -- db/migrate/    # any new migrations?
git diff HEAD~1 HEAD -- Gemfile.lock   # gem changes?
```

- **Gem changes** (Gemfile.lock changed): a full image rebuild is required (Step 3).
- **No gem changes, no assets changed**: you may be able to skip the rebuild and just restart (the volume mount overlays source on top of the baked image). Test this carefully — if in doubt, rebuild.
- **Asset changes** (app/assets/, app/javascript/): rebuild required; assets are precompiled at image build time.

### Step 2 — Tag the current image as rollback

Before touching anything, preserve the current image:

```bash
DATE=$(date +%Y%m%d)
docker tag remetrics-web:staging-baked remetrics-web:rollback-$DATE
docker tag remetrics-worker:staging-baked remetrics-worker:rollback-$DATE
```

Keep at most two rollback tags; remove older ones with `docker rmi`.

### Step 3 — Build new images

> **Requires PREREQ 2 (build: stanza) to be complete first.**

```bash
cd ~/services/remetrics
DATE=$(date +%Y%m%d)
docker build -f app/Dockerfile.production -t remetrics-web:production-$DATE app/
docker build -f app/Dockerfile.production -t remetrics-worker:production-$DATE app/
```

Expected build time: 3–8 minutes if Gemfile unchanged (bundle cache hit); longer if gems changed.

Update the compose file (or `.env` IMAGE_TAG variable) to reference the new dated tag before proceeding.

### Step 4 — Run database migrations

Run migrations against the live database using a one-off container before cutting over traffic. This allows aborting if migrations fail without taking the site down:

```bash
docker run --rm \
  --network internal \
  -e RAILS_ENV=production \
  -e DATABASE_URL=postgres://estatioro:PASSWORD@remetrics-db-1/remetrics_production \
  remetrics-web:production-$DATE \
  bundle exec rails db:migrate
```

> **Note**: Replace `PASSWORD` with the actual value from `~/services/remetrics/docker-compose.yml`. Once PREREQ 3 is complete, this can use `--env-file ~/services/remetrics/.env` instead.

If migrations fail, abort — existing containers are still running the old code on the old schema. Investigate and fix before retrying.

### Step 5 — Restart services with new image

```bash
cd ~/services/remetrics
docker compose up -d --no-deps remetrics worker
```

`--no-deps` prevents Postgres and Redis from being touched. Compose starts new containers and stops old ones.

### Step 6 — Verify

```bash
# Check containers came up
docker compose ps

# Tail logs for errors
docker logs remetrics-remetrics-1 --tail 50

# HTTP probe
curl -sf https://remetrics.io/up && echo "OK" || echo "FAILED"
```

### Step 7 — Rollback if needed

If the new deployment is broken:

```bash
cd ~/services/remetrics
# Edit compose IMAGE_TAG back to the rollback tag, then:
docker compose up -d --no-deps remetrics worker
```

Rolling back database migrations is generally not safe. If a migration ran and the new code is being reverted, prefer a forward fix over schema rollback.

---

## Code-Only Restart (No Gem/Asset Changes)

When a git pull brings only Ruby code changes (no Gemfile.lock, no asset changes), you may be able to skip the rebuild because the bind mount overlays the source on top of the baked image:

```bash
cd ~/services/remetrics/app && git pull origin master
# Verify: no Gemfile.lock or db/migrate/ changes
cd ~/services/remetrics && docker compose restart remetrics worker
```

This is faster but less reliable — if any initialization depends on the build-time environment (precompiled assets, baked gems), the restart alone may not pick up all changes. When in doubt, do a full rebuild.

---

## Strapi Update Workflow

Strapi updates are riskier than Rails updates because of unclear image provenance.

**Day-to-day content changes** (new articles, config edits): No image update needed. All Strapi content and config live in the bind-mounted `~/services/strapi-remetrics/strapi/` directories. Changes are live immediately.

**Strapi version upgrade** (security patch, feature update):

> The current image (`strapi-remetrics:staging`) originated from a private GCP registry and is 3 years old. The compose `# Do NOT pull / rebuild` note reflects that the GCP registry is gone. To do a version upgrade you must write a Dockerfile.

1. **Back up the SQLite database first**:
   ```bash
   ssh jared@15.204.247.153 "cp -r ~/services/strapi-remetrics/strapi/database/ ~/backups/strapi-sqlite-$(date +%Y%m%d)/"
   ```

2. **Write a Dockerfile** in `~/services/strapi-remetrics/` using the on-disk source:
   ```dockerfile
   FROM node:18-alpine
   WORKDIR /app
   COPY strapi/package.json strapi/yarn.lock ./
   RUN yarn install --frozen-lockfile
   RUN yarn add @strapi-community/strapi-provider-upload-google-cloud-storage
   COPY strapi/src/ ./src/
   COPY strapi/config/ ./config/
   RUN yarn build
   CMD ["yarn", "start"]
   ```

3. **Add a `build:` stanza** to `~/services/strapi-remetrics/docker-compose.yml` and remove the `Do NOT pull / rebuild` comment once the Dockerfile exists.

4. **Build and tag**:
   ```bash
   cd ~/services/strapi-remetrics
   docker build -t strapi-remetrics:production-$(date +%Y%m%d) .
   ```

5. **Update compose and restart**:
   ```bash
   docker compose up -d --no-deps strapi
   ```

6. Strapi auto-runs schema migrations on startup — no manual step needed for SQLite.

---

## Asset Precompile Note

`Dockerfile.production` runs `bin/rails assets:precompile` at build time. However, the compose file mounts `./app:/app` into the container, which overlays the bind-mount directory over `/app`. If `~/services/remetrics/app/public/assets/` exists on disk (i.e., is not gitignored and empty), the compiled assets from the image layer are shadowed by the on-disk directory.

**Recommended**: ensure `app/public/assets/` is in `.gitignore` and not present on the host (or is empty) so that the image's precompiled assets are used. Alternatively, remove the `./app:/app` bind mount entirely and COPY source into the image at build time — this makes images fully self-contained and removes the shadowing concern.

---

## Build Location

Build on the OVH server directly (no external CI needed). The server has the source, the Dockerfile, and sufficient resources (~22 GiB RAM, 8 vCores). The house style for all other services uses pre-built upstream images with no CI pipeline; introducing Bitbucket Pipelines or GitHub Actions adds registry costs and complexity not warranted for a single-server setup.

---

## Version Upgrade Priority Order

When a rebuild is scheduled, upgrade these in order (each requires a Dockerfile change):

1. **Node 16 → 20** in `Dockerfile.production` (Node 16 is EOL since Sept 2023)
2. **Strapi 4.9.2 → latest 4.x** (security patches, same major — test on staging)
3. **Rails 7.0 → 7.1 or 7.2** (security maintenance window passed — moderate effort)
4. **Ruby 3.1 → 3.3** (EOL March 2025 — requires base image change + gem compat check)

Do not jump to Strapi 5 — it is a breaking major. Get to latest 4.x first.

---

## See Also

- [remetrics Operations](./remetrics-operations.md) — day-to-day ops, rollback to DigitalOcean
- [Disaster Recovery](./disaster-recovery.md) — full restore from backup