6.9 KiB

Raw Blame History

Certificate Management

TLS certificates on systems-prod-01 are managed automatically by Traefik via ACME. There are two resolvers in use — one for most services and a separate one for remetrics.

Two ACME Resolvers

Resolver name	Challenge type	Used by	Config key
`letsencrypt`	HTTP-01	All services except remetrics	`certresolver=letsencrypt`
`cloudflare`	DNS-01	remetrics.io, www.remetrics.io, cms.remetrics.io	`certresolver=cloudflare`

letsencrypt (HTTP-01)

Used by: n8n, n8n-test, Invoice Ninja, Vaultwarden, NocoDB, Forgejo, BestSolarTech, LandHomeTeam, and all other services not listed under cloudflare.

HTTP-01 requires that the domain resolves to this server and that port 80 is reachable. Traefik answers the ACME challenge automatically via the HTTP entrypoint.

cloudflare (DNS-01)

Used by: remetrics.io, www.remetrics.io, cms.remetrics.io.

DNS-01 was added during the 2026-06-24 remetrics migration to allow certificates to be pre-issued before the DNS cutover, closing the cert-availability gap. It uses Cloudflare API credentials (stored in Traefik's environment, not documented here — see credentials.md).

DNS-01 works even when the domain does not yet resolve to this server, making it the right choice for any future migration that needs pre-issued certs.

Where Certificates Are Stored

Traefik stores ACME data in two JSON files on the server:

File	Resolver	Location
`acme.json`	`letsencrypt` (HTTP-01)	`~/services/traefik/letsencrypt/acme.json`
`acme-cloudflare.json`	`cloudflare` (DNS-01)	`~/services/traefik/letsencrypt/acme-cloudflare.json`

Both files are bind-mounted into the Traefik container. They persist across Traefik restarts.

Do not delete these files unless you want Traefik to re-issue all certificates from scratch on next start. Let's Encrypt has rate limits (5 failures per domain per hour, 50 new certs per domain per week).

Viewing Certificate Status

Check all Traefik-managed certs (from ACME JSON)

ssh jared@15.204.247.153 "cat ~/services/traefik/letsencrypt/acme.json | python3 -c \"
import json,sys,base64,datetime
data=json.load(sys.stdin)
for r,v in data.items():
  for d in v.get('Certificates',[]):
    cert=base64.b64decode(d['certificate'])
    # pipe to openssl for full details
    print(d['domain']['main'])
\""

For a more readable output, pipe the cert bytes to openssl x509 -noout -dates:

ssh jared@15.204.247.153 "python3 - <<'EOF'
import json, base64, subprocess
for fname in ['acme.json', 'acme-cloudflare.json']:
    try:
        with open(f'/home/jared/services/traefik/letsencrypt/{fname}') as f:
            data = json.load(f)
        for resolver, v in data.items():
            for entry in v.get('Certificates', []):
                domain = entry['domain']['main']
                cert_pem = base64.b64decode(entry['certificate'])
                result = subprocess.run(
                    ['openssl', 'x509', '-noout', '-enddate'],
                    input=cert_pem, capture_output=True
                )
                print(f'{domain}: {result.stdout.decode().strip()}')
    except Exception as e:
        print(f'{fname}: {e}')
EOF"

Check expiry via TLS handshake (for live domains)

# Check a single domain
ssh jared@15.204.247.153 "echo | openssl s_client -connect remetrics.io:443 -servername remetrics.io 2>/dev/null | openssl x509 -noout -enddate"

# Quick loop over all critical domains
ssh jared@15.204.247.153 "for domain in remetrics.io cms.remetrics.io n8n.hyperthrive.io bestsolartech.com; do
  expiry=\$(echo | openssl s_client -connect \${domain}:443 -servername \${domain} 2>/dev/null | openssl x509 -noout -enddate 2>/dev/null)
  echo \"\${domain}: \${expiry}\"
done"

How Renewal Works

Traefik renews certificates automatically when they have ~30 days remaining. No manual action is needed under normal operation.

For the letsencrypt resolver: Traefik handles the HTTP-01 challenge internally. Ensure port 80 remains open (UFW rule 80/tcp ALLOW IN Anywhere is active).

For the cloudflare resolver: Traefik uses the Cloudflare API to create and remove DNS TXT records. No port needs to be open. The Cloudflare API token must remain valid — check credentials.md for the token and its expiry.

Renewal Troubleshooting

If a certificate fails to renew:

# Check Traefik logs for ACME errors
ssh jared@15.204.247.153 "docker logs traefik --since 24h 2>&1 | grep -i 'acme\|certificate\|error'"

Common causes:

HTTP-01 failure: port 80 is blocked, or the domain no longer resolves to this server
DNS-01 failure: Cloudflare API token expired or revoked; check credentials.md
Rate limit hit: too many failed attempts; wait and retry; check logs for "rate limit" messages
acme.json permissions: the file must be readable by the Traefik container; chmod 600 is the correct permission (set in Traefik's static config)

To force a renewal (use sparingly — rate limits apply):

# Backup the acme.json, delete the cert entry for the domain, restart Traefik
ssh jared@15.204.247.153 "cp ~/services/traefik/letsencrypt/acme.json ~/backups/acme-backup-$(date +%Y%m%d).json"
# Then edit acme.json to remove the certificate entry for the affected domain
ssh jared@15.204.247.153 "docker restart traefik"

Adding a New Service with a Certificate

For new services using HTTP-01 (standard):

labels:
  - "traefik.http.routers.MYSERVICE.rule=Host(`mydomain.com`)"
  - "traefik.http.routers.MYSERVICE.entrypoints=websecure"
  - "traefik.http.routers.MYSERVICE.tls=true"
  - "traefik.http.routers.MYSERVICE.tls.certresolver=letsencrypt"

For a service that needs DNS-01 (e.g., pre-issuing a cert before DNS cutover):

labels:
  - "traefik.http.routers.MYSERVICE.tls.certresolver=cloudflare"

See Configuration Reference for full label templates.

Certificate Expiry Monitoring — Current Status

TODO: There is currently no automated cert expiry alerting. Traefik auto-renews, but if renewal silently fails, no alert is sent.

A cert-check script was proposed during the 2026-06-25 architecture review. It would check expiry days for all domains and alert if under 14 days. This has not yet been implemented. See Monitoring & Automation for the gap list.

6.9 KiB Raw Blame History