server-ovh-prod-01-main/docs/disaster-recovery.md

Disaster Recovery Guide

This document covers how to restore services from backups in the event of server failure.

Understanding Hyper Backup's Storage Format

Hyper Backup does NOT create timestamped files like backup-2025-12-09.tar.gz. Instead, it uses a deduplicated block storage format (.hbk folder).

What you'll see in B2:

ovh-prod-01.hbk/
├── Config/      # Backup configuration
├── Control/     # Version metadata and timestamps
├── Guard/       # Integrity verification data
├── Pool/        # Deduplicated data blocks (shared across all versions)
└── *.db files   # Index databases

Why this design?

• A 1.6GB backup that changes 50MB daily only stores ~50MB new data per version
• Much more storage-efficient than full timestamped copies
• All version history is embedded in the metadata

To browse backup versions:

1. Open Hyper Backup on Synology
2. Select the backup task
3. Click "Version List" button
4. Timeline shows all versions with timestamps - select one to browse/restore

To restore without Synology access:

• Download the entire .hbk folder from B2
• Use Hyper Backup Explorer (free Synology desktop tool for Windows/Mac/Linux)
• Browse and extract any version from the backup set

Backup Locations

Location	Path	Contents
OVH Server	~/backups/current/latest.tar.gz	Most recent backup (2 days retention)
Synology NAS	/volume1/backups/ovh-prod-01/	Pulled backups with Smart Recycle retention
Backblaze B2	ovh-prod-01-backups bucket	Offsite mirror of Synology

Backup Contents

The latest.tar.gz contains:

backups/
├── SERVICE_NAME/
│   ├── docker-compose.yml      # Service configuration
│   ├── .env                    # Environment variables (if present)
│   └── db_backup.sql           # Database dump (MariaDB/PostgreSQL)
│       or db_backup/           # MongoDB dump directory
│       or pb_data.tar.gz       # PocketBase data

Recovery Procedures

Scenario 1: Single Service Recovery

If one service is corrupted but the server is intact:

# 1. SSH into server
ssh jared@15.204.247.153

# 2. Stop the affected service
cd ~/services/SERVICE_NAME
docker compose down

# 3. Extract backup
cd ~/backups/current
tar -xzf latest.tar.gz

# 4. Restore database (MariaDB example)
docker exec -i SERVICE_NAME-db mysql -u root -pPASSWORD DATABASE < backups/SERVICE_NAME/db_backup.sql

# 5. Restart service
cd ~/services/SERVICE_NAME
docker compose up -d

Scenario 2: Full Server Recovery

If the server is lost and you need to rebuild:

Step 1: Provision New Server

• OVHcloud VPS or equivalent
• Ubuntu 22.04 LTS recommended
• Install Docker and Docker Compose

Step 2: Restore Network Infrastructure

# Create Docker networks
docker network create --subnet=172.20.0.0/16 traefik
docker network create --subnet=172.21.0.0/16 internal

# Create services directory
mkdir -p ~/services

Step 3: Get Backup from Synology or B2

From Synology (if accessible):

scp jared@192.168.86.31:/volume1/backups/ovh-prod-01/latest.tar.gz ~/

From Backblaze B2:

1. Open Synology Hyper Backup
2. Click the restore icon
3. Browse to desired backup version
4. Download/restore latest.tar.gz

Or use B2 CLI:

b2 authorize-account APPLICATION_KEY_ID APPLICATION_KEY
b2 download-file-by-name ovh-prod-01-backups latest.tar.gz ./latest.tar.gz

Step 4: Restore Traefik First

# Extract backup
tar -xzf latest.tar.gz

# Copy Traefik config
cp -r backups/traefik ~/services/

# Start Traefik
cd ~/services/traefik
docker compose up -d

Step 5: Restore Each Service

For each service directory in the backup:

# Copy service files
cp -r backups/SERVICE_NAME ~/services/

# Start service (creates containers and volumes)
cd ~/services/SERVICE_NAME
docker compose up -d

# Restore database if applicable
# MariaDB:
docker exec -i CONTAINER_NAME mysql -u root -pPASSWORD DATABASE < db_backup.sql

# PostgreSQL:
docker exec -i CONTAINER_NAME psql -U USER DATABASE < db_backup.sql

# MongoDB:
docker exec -i CONTAINER_NAME mongorestore --archive < db_backup.archive

Step 6: Verify Services

# Check all containers are running
docker ps

# Check Traefik dashboard for routing
# Verify each service responds on its domain

Step 7: Update DNS

Point domains to new server IP if changed.

Credentials Required for Recovery

All credentials stored in Bitwarden:

• Server SSH: Search "OVH" or "15.204.247.153"
• Backblaze B2: Search "Backblaze | ovh-prod-01-backups key"
• Database passwords: In each service's .env file (included in backup)
• Traefik dashboard: In credentials.md (this repo)

Testing Recovery

Periodically test recovery by:

1. Spinning up a test VPS
2. Following the full server recovery procedure
3. Verifying services come up correctly
4. Destroying test VPS

Contact

Server admin: Jared Swanson