server-ovh-prod-01-main/migrations/migration-plan.md

# Systems Production Server Migration Plan

**STATUS**: ✅ **MIGRATION COMPLETE (2026-06-24)** — All services migrated from
DigitalOcean to OVH and verified live. The source DigitalOcean VPS
(`hyperthrive-docker-services`, 161.35.12.75) was **deleted on 2026-06-24**.
This document is retained as a historical record. The phase markers below
reflect the original incremental plan; all phases are now done.

## Overview
This document outlines the complete migration plan for consolidating DigitalOcean droplets to the systems production server (systems-prod-01.vps.ovh.us). Each phase is designed to be executed in separate sessions to maintain manageable context windows and reduce risk.

## Migration Strategy
- **Approach**: Incremental service migration with parallel infrastructure
- **Risk Mitigation**: Keep DigitalOcean services running until fully migrated and tested
- **Rollback**: Maintain ability to rollback at each phase
- **Validation**: Test each service thoroughly before DNS cutover

---

## Phase 1: Server Setup & Security Hardening
**Goal**: Establish secure, functional base server ready for service deployment

### Prerequisites
- [x] Server provisioned and documented
- [x] SSH credentials received from OVH via email
- [x] SSH public key ready for deployment
- [x] Domain names identified for services

### Tasks
#### 1.1 Initial Access & User Setup ✅ COMPLETED
- [x] Connect to server using initial credentials: `ssh ubuntu@15.204.247.153`
- [x] Change default ubuntu user password
- [x] Create 'jared' user with sudo privileges
- [x] Copy SSH public key to server
- [x] Test SSH key authentication
- [x] Remove ubuntu user (security enhancement)

#### 1.2 Security Hardening ✅ COMPLETED
- [x] Disable password authentication in SSH config
- [x] Disable root login via SSH
- [x] Configure UFW firewall with basic rules:
  - SSH (port 22)
  - HTTP (port 80)
  - HTTPS (port 443)
- [x] Enable UFW firewall
- [x] Configure automatic security updates

#### 1.3 System Preparation ✅ COMPLETED
- [x] Update all system packages (completed)
- [x] Install essential tools (all required tools installed)
- [x] Configure timezone and NTP (America/New_York, chrony active)
- [x] Set up log rotation (custom configuration created)
- [x] Configure swap file if needed (4GB swap active)

#### 1.4 Validation ✅ COMPLETED
- [x] SSH access working with keys only (verified)
- [x] Firewall rules verified (UFW active, SSH/HTTP/HTTPS)
- [x] System fully updated (all packages current)
- [x] User permissions working correctly (jared user with sudo)
- [x] All services running properly (SSH, chrony, UFW)

### Deliverables
- [x] Updated profile.yaml with actual server configuration
- [x] initial-setup.md runbook documenting exact commands used
- [x] Security checklist completed
- [x] phase1-2-security.sh automated security hardening script
- [x] phase1-2-completion.md documentation
- [x] phase1-3-system-prep.sh automated system preparation script
- [x] phase1-3-manual-commands.md manual command reference
- [x] enable-passwordless-sudo.sh for automation setup
- [x] phase1-3-completion.md status documentation
- [x] phase1-4-completion.md final validation documentation

### Exit Criteria ✅ ALL MET
- [x] Server accessible only via SSH keys
- [x] Firewall properly configured and active
- [x] All security hardening complete
- [x] System updated and ready for software installation

## 🎉 PHASE 1 COMPLETE

**Status**: All Phase 1 tasks successfully completed on 2025-09-18
**Ready for**: Phase 2 Docker Environment Setup

---

## Phase 2: Docker Environment Setup
**Goal**: Install and configure Docker environment for containerized services

### Prerequisites
- [x] Phase 1 completed successfully
- [x] Server accessible and secure

### Tasks
#### 2.1 Docker Installation ✅ COMPLETED
- [x] Install Docker using official script
- [x] Install Docker Compose
- [x] Add jared user to docker group
- [x] Configure Docker daemon settings
- [x] Test Docker installation

#### 2.2 Service Directory Structure ✅ COMPLETED
- [x] Create `/home/jared/services/` directory hierarchy
- [x] Set up proper permissions and ownership
- [x] Create shared Docker networks
- [x] Configure Docker logging

#### 2.3 Traefik Reverse Proxy ✅ COMPLETED
- [x] Create traefik service directory
- [x] Configure Traefik docker-compose.yml
- [x] Set up Let's Encrypt configuration
- [x] Configure Traefik dashboard
- [x] Test Traefik startup and SSL certificate generation

#### 2.4 Base Monitoring ✅ COMPLETED
- [x] Set up basic system monitoring
- [x] Configure log aggregation
- [x] Test monitoring setup

### Deliverables ✅ ALL COMPLETE
- [x] Docker environment fully configured
- [x] Traefik running and obtaining SSL certificates
- [x] Service deployment template created
- [x] Monitoring basics in place

### Exit Criteria ✅ ALL MET
- [x] Docker and Docker Compose working
- [x] Traefik successfully routing traffic and managing SSL
- [x] Basic monitoring operational
- [x] Ready for service deployment

## 🎉 PHASE 2 COMPLETE

**Status**: All Phase 2 tasks successfully completed on 2025-09-18
**Ready for**: Phase 3 Service Migration Planning

---

## Phase 3: Service Migration Planning
**Goal**: Document and prepare for individual service migrations

### Prerequisites
- [x] Phase 2 completed successfully
- [x] Inventory of all DigitalOcean services

### Tasks
#### 3.1 Service Inventory ✅ COMPLETED
- [x] Document all current DigitalOcean services
- [x] Map service dependencies
- [x] Identify databases and persistent data
- [x] Document current domain configurations
- [x] List environment variables and secrets

#### 3.2 Migration Order Planning ✅ COMPLETED
- [x] Categorize services by risk level
- [x] Define migration order (low to high risk)
- [x] Identify testing requirements for each service
- [x] Plan rollback procedures

#### 3.3 WordPress Sites Migration ✅ COMPLETED
##### BestSolarTech.com ✅ COMPLETED (2025-09-19)
- [x] Migrated WordPress files (1.5GB) and database (599MB) from backup
- [x] Configured Docker Compose with WordPress 6.7 and MariaDB 10.11
- [x] Disabled Real Simple SSL plugin (handled by Traefik now)
- [x] Added proxy SSL detection to wp-config.php
- [x] Tested on https://temp.bestsolartech.com with HTTPS working
- [x] DNS cutover completed - live on https://bestsolartech.com
- [x] Admin panel fully functional
- **Status**: ✅ LIVE IN PRODUCTION

#### 3.4 DNS Preparation
- [ ] Document current DNS configurations
- [ ] Plan DNS cutover strategy
- [ ] Set up DNS records for new server (with temporary domains)
- [ ] Reduce TTL values for production domains

#### 3.4 Data Migration Strategy
- [ ] Plan database migration approaches
- [ ] Identify file storage migration needs
- [ ] Plan backup and restoration procedures
- [ ] Test data migration with non-critical services

### Deliverables
- [ ] Complete service inventory with dependencies
- [ ] Migration order and timeline
- [ ] DNS migration plan
- [ ] Data migration procedures

### Exit Criteria
- All services documented and prioritized
- Migration procedures defined and tested
- DNS infrastructure ready for cutover
- Ready to begin service-by-service migration

---

## Historical / Superseded — Phases 4–7

> **Note (2026-06-25)**: The migration is complete. Phases 4–7 below were the original incremental plan template. They were never executed as discrete phases — services were migrated individually as needed, culminating in the remetrics addendum below. These sections are preserved as a historical record but do not reflect actual execution steps.

---

## Phase 4: Low-Risk Service Migration
**Goal**: Migrate static websites and development instances

### Prerequisites
- [x] Phase 3 completed successfully
- [ ] Test migration procedures validated

### Target Services (Examples)
- Static websites
- Development/staging instances
- Non-critical applications

### Tasks
#### 4.1 First Service Migration
- [ ] Choose lowest-risk service for initial migration
- [ ] Deploy service on systems production server
- [ ] Configure Traefik routing
- [ ] Test functionality on temporary domain
- [ ] Update DNS for service cutover
- [ ] Verify service working on production domain
- [ ] Monitor for issues

#### 4.2 Additional Low-Risk Services
- [ ] Repeat migration process for each low-risk service
- [ ] Document any issues and solutions
- [ ] Refine migration procedures

#### 4.3 Validation & Cleanup
- [ ] Verify all migrated services functioning
- [ ] Begin cleanup of DigitalOcean resources for migrated services
- [ ] Update documentation

### Deliverables
- [ ] Low-risk services successfully migrated
- [ ] Refined migration procedures
- [ ] Lessons learned documented

### Exit Criteria
- All low-risk services migrated and functional
- Migration procedures proven and documented
- Ready for medium-risk service migration

---

## Phase 5: Medium-Risk Service Migration
**Goal**: Migrate applications with minimal database dependencies

### Prerequisites
- [x] Phase 4 completed successfully
- [ ] Migration procedures proven with low-risk services

### Target Services (Examples)
- Simple web applications
- Services with small databases
- Non-critical automation tools

### Tasks
#### 5.1 Database Migration Preparation
- [ ] Set up database services on systems production server
- [ ] Test database backup and restore procedures
- [ ] Plan database migration windows

#### 5.2 Service Migration
- [ ] Migrate each medium-risk service individually
- [ ] Handle database migrations with minimal downtime
- [ ] Test all functionality thoroughly
- [ ] Update DNS and monitor

#### 5.3 Performance Optimization
- [ ] Monitor resource usage
- [ ] Optimize container configurations
- [ ] Tune database performance

### Deliverables
- [ ] Medium-risk services migrated
- [ ] Database migration procedures validated
- [ ] Performance optimizations applied

### Exit Criteria
- All medium-risk services migrated successfully
- Database migration procedures proven
- System performance optimized
- Ready for high-risk service migration

---

## Phase 6: High-Risk Service Migration
**Goal**: Migrate critical production services

### Prerequisites
- [x] Phase 5 completed successfully
- [ ] All procedures tested and validated
- [ ] Rollback plans confirmed

### Target Services (Examples)
- Production n8n automation
- Critical databases
- Production web applications

### Tasks
#### 6.1 Pre-Migration Preparation
- [ ] Schedule maintenance windows
- [ ] Prepare comprehensive rollback procedures
- [ ] Set up monitoring and alerting
- [ ] Create full system backups

#### 6.2 Critical Service Migration
- [ ] Migrate each critical service with maximum care
- [ ] Minimize downtime through careful planning
- [ ] Perform extensive testing
- [ ] Monitor closely post-migration

#### 6.3 Production Validation
- [ ] Verify all critical services functioning
- [ ] Confirm all integrations working
- [ ] Monitor performance and stability

### Deliverables
- [ ] All critical services migrated
- [ ] Full system validation complete
- [ ] Production stability confirmed

### Exit Criteria
- All services successfully migrated
- No critical issues identified
- System performance meeting requirements
- Ready for final cleanup

---

## Phase 7: Final Cleanup & Optimization
**Goal**: Complete migration and optimize new environment

### Prerequisites
- [x] Phase 6 completed successfully
- [ ] All services verified functional

### Tasks
#### 7.1 DigitalOcean Cleanup
- [ ] Take final backups of all DigitalOcean services
- [ ] Destroy DigitalOcean droplets
- [ ] Update billing and account settings
- [ ] Document cost savings achieved

#### 7.2 Final Optimization
- [ ] Optimize resource allocation
- [ ] Implement comprehensive monitoring
- [ ] Set up automated backups
- [ ] Fine-tune security settings

#### 7.3 Documentation Updates
- [ ] Update all server documentation
- [ ] Create operational runbooks
- [ ] Document lessons learned
- [ ] Create disaster recovery procedures

### Deliverables
- [ ] DigitalOcean resources decommissioned
- [ ] Systems production environment fully optimized
- [ ] Complete documentation set
- [ ] Cost savings validated

### Exit Criteria
- Migration completely finished
- All documentation current
- Monitoring and backups operational
- Cost optimization achieved

---

## Risk Management

### Rollback Procedures
1. **Per-Phase Rollback**: Maintain DigitalOcean services until phase completion
2. **Service-Level Rollback**: DNS cutback to DigitalOcean for individual services
3. **Full Rollback**: Complete revert to DigitalOcean if critical issues arise

### Monitoring & Validation
- Service health checks before and after each migration
- Performance monitoring during transition periods
- User impact assessment and communication

### Emergency Contacts
- OVH Support: [Contact information when available]
- DNS Provider Support: [Contact information when available]
- Backup contact methods for critical issues

---

## Success Metrics

### Cost Optimization
- Target: 30-50% reduction in monthly hosting costs
- Measure: Compare DigitalOcean bills before/after migration

### Performance
- Target: Maintain or improve service response times
- Measure: Response time monitoring and comparison

### Reliability
- Target: 99.9% uptime for all migrated services
- Measure: Uptime monitoring and incident tracking

### Security
- Target: No security incidents during or after migration
- Measure: Security audit and monitoring

---

## Timeline Estimation

| Phase | Estimated Duration | Dependencies | Status |
|-------|-------------------|--------------|---------|
| Phase 1 | 1-2 sessions (2-4 hours) | OVH credentials | ✅ COMPLETE (1 day) |
| Phase 2 | 1-2 sessions (2-4 hours) | Phase 1 complete | ✅ COMPLETE (1 day) |
| Phase 3 | 1 session (2-3 hours) | Service inventory | 🔄 NEXT |
| Phase 4 | 2-3 sessions (4-6 hours) | Low-risk services identified | ⏳ PENDING |
| Phase 5 | 3-4 sessions (6-8 hours) | Medium-risk services ready | ⏳ PENDING |
| Phase 6 | 4-5 sessions (8-10 hours) | High-risk service preparation | ⏳ PENDING |
| Phase 7 | 1-2 sessions (2-4 hours) | All services migrated | ⏳ PENDING |

**Total Estimated Time**: 12-19 sessions (26-39 hours)
**Completed**: 2 phases (2 days)
**Remaining**: 5 phases

---

*This plan is a living document and should be updated as the migration progresses.*

*Last updated: 2025-09-18*
*Phase 1 Status: ✅ COMPLETE - All tasks successfully finished*
*Phase 2 Status: ✅ COMPLETE - Docker environment ready*
*Ready for Phase 3: Service Migration Planning*

---

## Addendum: remetrics.io Migration ✅ COMPLETE (2026-06-24)

> **Note**: This was a *separate* migration from a **different** DigitalOcean box than
> the Phases above. Phases 1-7 cover the `hyperthrive-docker-services` VPS (161.35.12.75,
> deleted 2026-06-24). remetrics lived on a standalone DigitalOcean **web droplet
> (147.182.131.217)**, unrelated to the hyperthrive box. It is recorded here as an
> addendum rather than retrofitted into a phase.

**Outcome**: `remetrics.io` (Rails app + Strapi CMS) was migrated from the DigitalOcean
web droplet (147.182.131.217) to the OVH server (15.204.247.153) on 2026-06-24.

- **Services on OVH**:
  - `~/services/remetrics/` — Rails web (`remetrics-remetrics-1`), Sidekiq worker
    (`remetrics-worker-1`), Postgres (`remetrics-db-1`, db `remetrics_production`), redis.
    Web/worker images baked GitHub-independent (no boot-time bundle/git).
  - `~/services/strapi-remetrics/` — Strapi (`strapi-remetrics`), SQLite, media on GCS
    bucket `remetrics-strapi`.
- **DNS** (Cloudflare zone `remetrics.io`, grey-cloud A-records → 15.204.247.153):
  `remetrics.io`, `www.remetrics.io`, `cms.remetrics.io`. `temp.remetrics.io` and the
  dead/unused `strapi.remetrics.io` intentionally remain on the old DO box.
- **Cutover**: ~4m40s frozen window, **zero data loss** (480 users migrated exactly; a
  503 "freeze" on the old box + final `pg_dump` captured a signup that landed seconds
  before cutover).
- **TLS**: Added a second Traefik cert-resolver `cloudflare` (ACME DNS-01, storage
  `/letsencrypt/acme-cloudflare.json`) alongside the existing `letsencrypt` (HTTP-01).
  remetrics/cms/www routers use `certresolver=cloudflare`. Certs were **pre-issued via
  DNS-01 before the DNS flip** for a zero-cert-window cutover.
- **Backups**: `~/services/backup.sh` auto-discovers both new services (remetrics Postgres
  via `pg_dumpall`, Strapi SQLite via service-files tar); confirmed in nightly backup.
- **CMS coupling**: The Rails app fetches CMS from a HARDCODED `https://cms.remetrics.io`
  (in `strapi/article.rb` and `help_page.rb`) — not env-configurable.

**Rollback status**: The DigitalOcean remetrics web droplet (147.182.131.217) is currently
**FROZEN** (apex returns 503, sidekiq worker stopped) and kept as a short-term rollback.
It is **NOT yet deleted** — decommission pending.