173 lines
6.9 KiB
Markdown
173 lines
6.9 KiB
Markdown
# Certificate Management
|
|
|
|
TLS certificates on systems-prod-01 are managed automatically by Traefik via ACME. There are two resolvers in use — one for most services and a separate one for remetrics.
|
|
|
|
See also:
|
|
- [remetrics Operations](./remetrics-operations.md) — why remetrics uses a different resolver
|
|
- [Configuration Reference](./configuration-reference.md) — Traefik label syntax
|
|
- [Monitoring & Automation](./monitoring-automation.md) — health check status
|
|
|
|
---
|
|
|
|
## Two ACME Resolvers
|
|
|
|
| Resolver name | Challenge type | Used by | Config key |
|
|
|---|---|---|---|
|
|
| `letsencrypt` | HTTP-01 | All services except remetrics | `certresolver=letsencrypt` |
|
|
| `cloudflare` | DNS-01 | remetrics.io, www.remetrics.io, cms.remetrics.io | `certresolver=cloudflare` |
|
|
|
|
### letsencrypt (HTTP-01)
|
|
|
|
Used by: n8n, n8n-test, Invoice Ninja, Vaultwarden, NocoDB, Forgejo, BestSolarTech, LandHomeTeam, and all other services not listed under cloudflare.
|
|
|
|
HTTP-01 requires that the domain resolves to this server and that port 80 is reachable. Traefik answers the ACME challenge automatically via the HTTP entrypoint.
|
|
|
|
### cloudflare (DNS-01)
|
|
|
|
Used by: remetrics.io, www.remetrics.io, cms.remetrics.io.
|
|
|
|
DNS-01 was added during the 2026-06-24 remetrics migration to allow certificates to be pre-issued before the DNS cutover, closing the cert-availability gap. It uses Cloudflare API credentials (stored in Traefik's environment, not documented here — see `credentials.md`).
|
|
|
|
DNS-01 works even when the domain does not yet resolve to this server, making it the right choice for any future migration that needs pre-issued certs.
|
|
|
|
---
|
|
|
|
## Where Certificates Are Stored
|
|
|
|
Traefik stores ACME data in two JSON files on the server:
|
|
|
|
| File | Resolver | Location |
|
|
|---|---|---|
|
|
| `acme.json` | `letsencrypt` (HTTP-01) | `~/services/traefik/letsencrypt/acme.json` |
|
|
| `acme-cloudflare.json` | `cloudflare` (DNS-01) | `~/services/traefik/letsencrypt/acme-cloudflare.json` |
|
|
|
|
Both files are bind-mounted into the Traefik container. They persist across Traefik restarts.
|
|
|
|
**Do not delete these files** unless you want Traefik to re-issue all certificates from scratch on next start. Let's Encrypt has rate limits (5 failures per domain per hour, 50 new certs per domain per week).
|
|
|
|
---
|
|
|
|
## Viewing Certificate Status
|
|
|
|
### Check all Traefik-managed certs (from ACME JSON)
|
|
|
|
```bash
|
|
ssh jared@15.204.247.153 "cat ~/services/traefik/letsencrypt/acme.json | python3 -c \"
|
|
import json,sys,base64,datetime
|
|
data=json.load(sys.stdin)
|
|
for r,v in data.items():
|
|
for d in v.get('Certificates',[]):
|
|
cert=base64.b64decode(d['certificate'])
|
|
# pipe to openssl for full details
|
|
print(d['domain']['main'])
|
|
\""
|
|
```
|
|
|
|
For a more readable output, pipe the cert bytes to `openssl x509 -noout -dates`:
|
|
|
|
```bash
|
|
ssh jared@15.204.247.153 "python3 - <<'EOF'
|
|
import json, base64, subprocess
|
|
for fname in ['acme.json', 'acme-cloudflare.json']:
|
|
try:
|
|
with open(f'/home/jared/services/traefik/letsencrypt/{fname}') as f:
|
|
data = json.load(f)
|
|
for resolver, v in data.items():
|
|
for entry in v.get('Certificates', []):
|
|
domain = entry['domain']['main']
|
|
cert_pem = base64.b64decode(entry['certificate'])
|
|
result = subprocess.run(
|
|
['openssl', 'x509', '-noout', '-enddate'],
|
|
input=cert_pem, capture_output=True
|
|
)
|
|
print(f'{domain}: {result.stdout.decode().strip()}')
|
|
except Exception as e:
|
|
print(f'{fname}: {e}')
|
|
EOF"
|
|
```
|
|
|
|
### Check expiry via TLS handshake (for live domains)
|
|
|
|
```bash
|
|
# Check a single domain
|
|
ssh jared@15.204.247.153 "echo | openssl s_client -connect remetrics.io:443 -servername remetrics.io 2>/dev/null | openssl x509 -noout -enddate"
|
|
|
|
# Quick loop over all critical domains
|
|
ssh jared@15.204.247.153 "for domain in remetrics.io cms.remetrics.io n8n.hyperthrive.io bestsolartech.com; do
|
|
expiry=\$(echo | openssl s_client -connect \${domain}:443 -servername \${domain} 2>/dev/null | openssl x509 -noout -enddate 2>/dev/null)
|
|
echo \"\${domain}: \${expiry}\"
|
|
done"
|
|
```
|
|
|
|
---
|
|
|
|
## How Renewal Works
|
|
|
|
Traefik renews certificates automatically when they have ~30 days remaining. No manual action is needed under normal operation.
|
|
|
|
For the `letsencrypt` resolver: Traefik handles the HTTP-01 challenge internally. Ensure port 80 remains open (UFW rule `80/tcp ALLOW IN Anywhere` is active).
|
|
|
|
For the `cloudflare` resolver: Traefik uses the Cloudflare API to create and remove DNS TXT records. No port needs to be open. The Cloudflare API token must remain valid — check `credentials.md` for the token and its expiry.
|
|
|
|
---
|
|
|
|
## Renewal Troubleshooting
|
|
|
|
If a certificate fails to renew:
|
|
|
|
```bash
|
|
# Check Traefik logs for ACME errors
|
|
ssh jared@15.204.247.153 "docker logs traefik --since 24h 2>&1 | grep -i 'acme\|certificate\|error'"
|
|
```
|
|
|
|
Common causes:
|
|
- **HTTP-01 failure**: port 80 is blocked, or the domain no longer resolves to this server
|
|
- **DNS-01 failure**: Cloudflare API token expired or revoked; check `credentials.md`
|
|
- **Rate limit hit**: too many failed attempts; wait and retry; check logs for "rate limit" messages
|
|
- **acme.json permissions**: the file must be readable by the Traefik container; `chmod 600` is the correct permission (set in Traefik's static config)
|
|
|
|
To force a renewal (use sparingly — rate limits apply):
|
|
```bash
|
|
# Backup the acme.json, delete the cert entry for the domain, restart Traefik
|
|
ssh jared@15.204.247.153 "cp ~/services/traefik/letsencrypt/acme.json ~/backups/acme-backup-$(date +%Y%m%d).json"
|
|
# Then edit acme.json to remove the certificate entry for the affected domain
|
|
ssh jared@15.204.247.153 "docker restart traefik"
|
|
```
|
|
|
|
---
|
|
|
|
## Adding a New Service with a Certificate
|
|
|
|
For new services using HTTP-01 (standard):
|
|
```yaml
|
|
labels:
|
|
- "traefik.http.routers.MYSERVICE.rule=Host(`mydomain.com`)"
|
|
- "traefik.http.routers.MYSERVICE.entrypoints=websecure"
|
|
- "traefik.http.routers.MYSERVICE.tls=true"
|
|
- "traefik.http.routers.MYSERVICE.tls.certresolver=letsencrypt"
|
|
```
|
|
|
|
For a service that needs DNS-01 (e.g., pre-issuing a cert before DNS cutover):
|
|
```yaml
|
|
labels:
|
|
- "traefik.http.routers.MYSERVICE.tls.certresolver=cloudflare"
|
|
```
|
|
|
|
See [Configuration Reference](./configuration-reference.md) for full label templates.
|
|
|
|
---
|
|
|
|
## Certificate Expiry Monitoring — Current Status
|
|
|
|
> **TODO**: There is currently no automated cert expiry alerting. Traefik auto-renews, but if renewal silently fails, no alert is sent.
|
|
>
|
|
> A cert-check script was proposed during the 2026-06-25 architecture review. It would check expiry days for all domains and alert if under 14 days. This has not yet been implemented. See [Monitoring & Automation](./monitoring-automation.md) for the gap list.
|
|
|
|
---
|
|
|
|
## See Also
|
|
|
|
- [remetrics Operations](./remetrics-operations.md) — DNS-01 background
|
|
- [Configuration Reference](./configuration-reference.md) — Traefik label templates
|
|
- [Monitoring & Automation](./monitoring-automation.md) — monitoring gaps
|