server-ovh-prod-01-main/docs/disaster-recovery.md

# Disaster Recovery Guide

This document covers how to restore services from backups in the event of server failure.

## Understanding Hyper Backup's Storage Format

Hyper Backup does NOT create timestamped files like `backup-2025-12-09.tar.gz`. Instead, it uses a **deduplicated block storage format** (`.hbk` folder).

**What you'll see in B2:**
```
ovh-prod-01.hbk/
├── Config/      # Backup configuration
├── Control/     # Version metadata and timestamps
├── Guard/       # Integrity verification data
├── Pool/        # Deduplicated data blocks (shared across all versions)
└── *.db files   # Index databases
```

**Why this design?**
- A 1.6GB backup that changes 50MB daily only stores ~50MB new data per version
- Much more storage-efficient than full timestamped copies
- All version history is embedded in the metadata

**To browse backup versions:**
1. Open Hyper Backup on Synology
2. Select the backup task
3. Click **"Version List"** button
4. Timeline shows all versions with timestamps - select one to browse/restore

**To restore without Synology access:**
- Download the entire `.hbk` folder from B2
- Use **Hyper Backup Explorer** (free Synology desktop tool for Windows/Mac/Linux)
- Browse and extract any version from the backup set

## Backup Locations

| Location | Path | Contents |
|----------|------|----------|
| OVH Server | `~/backups/current/latest.tar.gz` | Most recent backup (2 days retention) |
| Synology NAS | `/volume1/backups/ovh-prod-01/` | Pulled backups with Smart Recycle retention |
| Backblaze B2 | `ovh-prod-01-backups` bucket | Offsite mirror of Synology |

## Backup Contents

The `latest.tar.gz` contains:
```
backups/
├── SERVICE_NAME/
│   ├── docker-compose.yml      # Service configuration
│   ├── .env                    # Environment variables (if present)
│   └── db_backup.sql           # Database dump (MariaDB/PostgreSQL)
│       or db_backup/           # MongoDB dump directory
│       or pb_data.tar.gz       # PocketBase data
```

## Recovery Procedures

### Scenario 1: Single Service Recovery

If one service is corrupted but the server is intact:

```bash
# 1. SSH into server
ssh jared@15.204.247.153

# 2. Stop the affected service
cd ~/services/SERVICE_NAME
docker compose down

# 3. Extract backup
cd ~/backups/current
tar -xzf latest.tar.gz

# 4. Restore database (MariaDB example)
docker exec -i SERVICE_NAME-db mysql -u root -pPASSWORD DATABASE < backups/SERVICE_NAME/db_backup.sql

# 5. Restart service
cd ~/services/SERVICE_NAME
docker compose up -d
```

### Scenario 2: Full Server Recovery

If the server is lost and you need to rebuild:

#### Step 1: Provision New Server
- OVHcloud VPS or equivalent
- Ubuntu 22.04 LTS recommended
- Install Docker and Docker Compose

#### Step 2: Restore Network Infrastructure
```bash
# Create Docker networks
docker network create --subnet=172.20.0.0/16 traefik
docker network create --subnet=172.21.0.0/16 internal

# Create services directory
mkdir -p ~/services
```

#### Step 3: Get Backup from Synology or B2

**From Synology (if accessible):**
```bash
scp jared@192.168.86.31:/volume1/backups/ovh-prod-01/latest.tar.gz ~/
```

**From Backblaze B2:**
1. Open Synology Hyper Backup
2. Click the restore icon
3. Browse to desired backup version
4. Download/restore `latest.tar.gz`

Or use B2 CLI:
```bash
b2 authorize-account APPLICATION_KEY_ID APPLICATION_KEY
b2 download-file-by-name ovh-prod-01-backups latest.tar.gz ./latest.tar.gz
```

#### Step 4: Restore Traefik First
```bash
# Extract backup
tar -xzf latest.tar.gz

# Copy Traefik config
cp -r backups/traefik ~/services/

# Start Traefik
cd ~/services/traefik
docker compose up -d
```

#### Step 5: Restore Each Service
For each service directory in the backup:

```bash
# Copy service files
cp -r backups/SERVICE_NAME ~/services/

# Start service (creates containers and volumes)
cd ~/services/SERVICE_NAME
docker compose up -d

# Restore database if applicable
# MariaDB:
docker exec -i CONTAINER_NAME mysql -u root -pPASSWORD DATABASE < db_backup.sql

# PostgreSQL:
docker exec -i CONTAINER_NAME psql -U USER DATABASE < db_backup.sql

# MongoDB:
docker exec -i CONTAINER_NAME mongorestore --archive < db_backup.archive
```

#### Step 6: Verify Services
```bash
# Check all containers are running
docker ps

# Check Traefik dashboard for routing
# Verify each service responds on its domain
```

#### Step 7: Update DNS
Point domains to new server IP if changed.

## Credentials Required for Recovery

All credentials stored in Bitwarden:
- **Server SSH**: Search "OVH" or "15.204.247.153"
- **Backblaze B2**: Search "Backblaze | ovh-prod-01-backups key"
- **Database passwords**: In each service's `.env` file (included in backup)
- **Traefik dashboard**: In `credentials.md` (this repo)

## Testing Recovery

Periodically test recovery by:
1. Spinning up a test VPS
2. Following the full server recovery procedure
3. Verifying services come up correctly
4. Destroying test VPS

## Contact

Server admin: Jared Swanson