server-ovh-prod-01-main/docs/cert-management.md

# Certificate Management

TLS certificates on systems-prod-01 are managed automatically by Traefik via ACME. There are two resolvers in use — one for most services and a separate one for remetrics.

See also:
- [remetrics Operations](./remetrics-operations.md) — why remetrics uses a different resolver
- [Configuration Reference](./configuration-reference.md) — Traefik label syntax
- [Monitoring & Automation](./monitoring-automation.md) — health check status

---

## Two ACME Resolvers

| Resolver name | Challenge type | Used by | Config key |
|---|---|---|---|
| `letsencrypt` | HTTP-01 | All services except remetrics | `certresolver=letsencrypt` |
| `cloudflare` | DNS-01 | remetrics.io, www.remetrics.io, cms.remetrics.io | `certresolver=cloudflare` |

### letsencrypt (HTTP-01)

Used by: n8n, n8n-test, Invoice Ninja, Vaultwarden, NocoDB, Forgejo, BestSolarTech, LandHomeTeam, and all other services not listed under cloudflare.

HTTP-01 requires that the domain resolves to this server and that port 80 is reachable. Traefik answers the ACME challenge automatically via the HTTP entrypoint.

### cloudflare (DNS-01)

Used by: remetrics.io, www.remetrics.io, cms.remetrics.io.

DNS-01 was added during the 2026-06-24 remetrics migration to allow certificates to be pre-issued before the DNS cutover, closing the cert-availability gap. It uses Cloudflare API credentials (stored in Traefik's environment, not documented here — see `credentials.md`).

DNS-01 works even when the domain does not yet resolve to this server, making it the right choice for any future migration that needs pre-issued certs.

---

## Where Certificates Are Stored

Traefik stores ACME data in two JSON files on the server:

| File | Resolver | Location |
|---|---|---|
| `acme.json` | `letsencrypt` (HTTP-01) | `~/services/traefik/letsencrypt/acme.json` |
| `acme-cloudflare.json` | `cloudflare` (DNS-01) | `~/services/traefik/letsencrypt/acme-cloudflare.json` |

Both files are bind-mounted into the Traefik container. They persist across Traefik restarts.

**Do not delete these files** unless you want Traefik to re-issue all certificates from scratch on next start. Let's Encrypt has rate limits (5 failures per domain per hour, 50 new certs per domain per week).

---

## Viewing Certificate Status

### Check all Traefik-managed certs (from ACME JSON)

```bash
ssh jared@15.204.247.153 "cat ~/services/traefik/letsencrypt/acme.json | python3 -c \"
import json,sys,base64,datetime
data=json.load(sys.stdin)
for r,v in data.items():
  for d in v.get('Certificates',[]):
    cert=base64.b64decode(d['certificate'])
    # pipe to openssl for full details
    print(d['domain']['main'])
\""
```

For a more readable output, pipe the cert bytes to `openssl x509 -noout -dates`:

```bash
ssh jared@15.204.247.153 "python3 - <<'EOF'
import json, base64, subprocess
for fname in ['acme.json', 'acme-cloudflare.json']:
    try:
        with open(f'/home/jared/services/traefik/letsencrypt/{fname}') as f:
            data = json.load(f)
        for resolver, v in data.items():
            for entry in v.get('Certificates', []):
                domain = entry['domain']['main']
                cert_pem = base64.b64decode(entry['certificate'])
                result = subprocess.run(
                    ['openssl', 'x509', '-noout', '-enddate'],
                    input=cert_pem, capture_output=True
                )
                print(f'{domain}: {result.stdout.decode().strip()}')
    except Exception as e:
        print(f'{fname}: {e}')
EOF"
```

### Check expiry via TLS handshake (for live domains)

```bash
# Check a single domain
ssh jared@15.204.247.153 "echo | openssl s_client -connect remetrics.io:443 -servername remetrics.io 2>/dev/null | openssl x509 -noout -enddate"

# Quick loop over all critical domains
ssh jared@15.204.247.153 "for domain in remetrics.io cms.remetrics.io n8n.hyperthrive.io bestsolartech.com; do
  expiry=\$(echo | openssl s_client -connect \${domain}:443 -servername \${domain} 2>/dev/null | openssl x509 -noout -enddate 2>/dev/null)
  echo \"\${domain}: \${expiry}\"
done"
```

---

## How Renewal Works

Traefik renews certificates automatically when they have ~30 days remaining. No manual action is needed under normal operation.

For the `letsencrypt` resolver: Traefik handles the HTTP-01 challenge internally. Ensure port 80 remains open (UFW rule `80/tcp ALLOW IN Anywhere` is active).

For the `cloudflare` resolver: Traefik uses the Cloudflare API to create and remove DNS TXT records. No port needs to be open. The Cloudflare API token must remain valid — check `credentials.md` for the token and its expiry.

---

## Renewal Troubleshooting

If a certificate fails to renew:

```bash
# Check Traefik logs for ACME errors
ssh jared@15.204.247.153 "docker logs traefik --since 24h 2>&1 | grep -i 'acme\|certificate\|error'"
```

Common causes:
- **HTTP-01 failure**: port 80 is blocked, or the domain no longer resolves to this server
- **DNS-01 failure**: Cloudflare API token expired or revoked; check `credentials.md`
- **Rate limit hit**: too many failed attempts; wait and retry; check logs for "rate limit" messages
- **acme.json permissions**: the file must be readable by the Traefik container; `chmod 600` is the correct permission (set in Traefik's static config)

To force a renewal (use sparingly — rate limits apply):
```bash
# Backup the acme.json, delete the cert entry for the domain, restart Traefik
ssh jared@15.204.247.153 "cp ~/services/traefik/letsencrypt/acme.json ~/backups/acme-backup-$(date +%Y%m%d).json"
# Then edit acme.json to remove the certificate entry for the affected domain
ssh jared@15.204.247.153 "docker restart traefik"
```

---

## Adding a New Service with a Certificate

For new services using HTTP-01 (standard):
```yaml
labels:
  - "traefik.http.routers.MYSERVICE.rule=Host(`mydomain.com`)"
  - "traefik.http.routers.MYSERVICE.entrypoints=websecure"
  - "traefik.http.routers.MYSERVICE.tls=true"
  - "traefik.http.routers.MYSERVICE.tls.certresolver=letsencrypt"
```

For a service that needs DNS-01 (e.g., pre-issuing a cert before DNS cutover):
```yaml
labels:
  - "traefik.http.routers.MYSERVICE.tls.certresolver=cloudflare"
```

See [Configuration Reference](./configuration-reference.md) for full label templates.

---

## Certificate Expiry Monitoring — Current Status

> **TODO**: There is currently no automated cert expiry alerting. Traefik auto-renews, but if renewal silently fails, no alert is sent.
>
> A cert-check script was proposed during the 2026-06-25 architecture review. It would check expiry days for all domains and alert if under 14 days. This has not yet been implemented. See [Monitoring & Automation](./monitoring-automation.md) for the gap list.

---

## See Also

- [remetrics Operations](./remetrics-operations.md) — DNS-01 background
- [Configuration Reference](./configuration-reference.md) — Traefik label templates
- [Monitoring & Automation](./monitoring-automation.md) — monitoring gaps