# Monitoring & Automation

This document describes the automated monitoring and maintenance systems running on systems-prod-01.

## Overview

The server has two automated scripts running via cron to ensure service health and prevent resource exhaustion:

1. **Health Monitor** - Runs every 10 minutes, monitors and restarts unhealthy containers
2. **Daily Maintenance** - Runs at 3 AM EST, performs cleanup and optimization

## Health Monitor

**Script:** `~/services/container-health-monitor.sh`
**Schedule:** Every 10 minutes
**Log:** `~/services/monitor.log`

### What It Does

- Monitors container health status
- Checks memory usage for each container
- Restarts containers if memory exceeds 40%
- Checks site availability (HTTP status codes)
- Cleans up Docker resources (unused volumes, images)
- Logs all actions with timestamps

### Manual Commands

```bash
# View recent monitoring activity
ssh jared@15.204.247.153 "tail -50 ~/services/monitor.log"

# Run health check manually
ssh jared@15.204.247.153 "~/services/container-health-monitor.sh"

# View cron schedule
ssh jared@15.204.247.153 "crontab -l"

# Watch monitoring in real-time
ssh jared@15.204.247.153 "tail -f ~/services/monitor.log"
```

### Memory Threshold Configuration

The 40% memory restart threshold can be adjusted by editing the script:

```bash
ssh jared@15.204.247.153 "nano ~/services/container-health-monitor.sh"

# Look for the line:
# MEMORY_THRESHOLD=40
```

## Daily Maintenance

**Script:** `~/services/daily-maintenance.sh`
**Schedule:** Daily at 3:00 AM EST
**Log:** `~/services/maintenance.log`

### What It Does

- Clears temporary files and cache
- Optimizes database tables (WordPress, etc.)
- Performs controlled container restarts
- Prevents memory buildup
- Prunes unused Docker resources
- Logs maintenance summary

### Manual Commands

```bash
# View recent maintenance activity
ssh jared@15.204.247.153 "tail -50 ~/services/maintenance.log"

# Run maintenance manually (caution: will restart containers)
ssh jared@15.204.247.153 "~/services/daily-maintenance.sh"

# View maintenance schedule
ssh jared@15.204.247.153 "crontab -l | grep maintenance"
```

## Cron Configuration

To view or modify the cron schedule:

```bash
# View current cron jobs
ssh jared@15.204.247.153 "crontab -l"

# Edit cron schedule (if needed)
ssh jared@15.204.247.153 "crontab -e"
```

**Current Schedule:**
```cron
# Health monitor - every 10 minutes
*/10 * * * * /home/jared/services/container-health-monitor.sh

# Daily maintenance - 3 AM EST
0 3 * * * /home/jared/services/daily-maintenance.sh
```

## Log Management

### Log Locations

- Health monitor: `~/services/monitor.log`
- Daily maintenance: `~/services/maintenance.log`
- Docker logs: `docker logs CONTAINER_NAME`
- Traefik logs: `~/services/traefik/logs/`

### Log Analysis Commands

```bash
# Count container restarts today
ssh jared@15.204.247.153 "grep 'restarting' ~/services/monitor.log | grep $(date +%Y-%m-%d) | wc -l"

# Find memory-related restarts
ssh jared@15.204.247.153 "grep 'memory exceeds' ~/services/monitor.log | tail -20"

# View maintenance summaries
ssh jared@15.204.247.153 "grep 'Maintenance complete' ~/services/maintenance.log | tail -10"

# Check for errors in logs
ssh jared@15.204.247.153 "grep -i error ~/services/monitor.log | tail -20"
```

### Log Rotation

Logs are automatically rotated via system logrotate configuration to prevent disk space issues.

**Check logrotate config:**
```bash
ssh jared@15.204.247.153 "cat /etc/logrotate.d/container-monitoring"
```

## Alerting

### Current Status
**Status:** Basic monitoring only (no external alerting)

### Future Improvements
Consider adding:
- Email notifications for critical events
- Slack/Discord webhooks for alerts
- Prometheus + Grafana for metrics visualization
- Uptime monitoring (UptimeRobot, etc.)

## Troubleshooting

### Container Keeps Restarting
**Symptom:** Health monitor constantly restarts a container
**Possible Causes:**
- Memory leak in application
- Memory threshold too low
- Container crash loop

**Investigation:**
```bash
# Check container logs
ssh jared@15.204.247.153 "docker logs CONTAINER_NAME --tail 100"

# Check memory usage trends
ssh jared@15.204.247.153 "docker stats --no-stream"

# Review health monitor logs
ssh jared@15.204.247.153 "grep CONTAINER_NAME ~/services/monitor.log | tail -20"
```

### Monitoring Script Not Running
**Symptom:** No recent entries in monitor.log
**Possible Causes:**
- Cron service stopped
- Script permissions issue
- Script path incorrect

**Investigation:**
```bash
# Check cron service
ssh jared@15.204.247.153 "systemctl status cron"

# Check script permissions
ssh jared@15.204.247.153 "ls -la ~/services/container-health-monitor.sh"

# Test script manually
ssh jared@15.204.247.153 "~/services/container-health-monitor.sh"
```

### Maintenance Window Issues
**Symptom:** 3 AM maintenance causing problems
**Solution:** Adjust maintenance time

```bash
# Edit cron schedule
ssh jared@15.204.247.153 "crontab -e"

# Change from 3 AM to different time (example: 2 AM)
# From: 0 3 * * * /home/jared/services/daily-maintenance.sh
# To:   0 2 * * * /home/jared/services/daily-maintenance.sh
```

## Health Check Endpoints

Services should implement health check endpoints for better monitoring:

**Example Docker Compose health check:**
```yaml
services:
  myservice:
    image: myapp:latest
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
```

## Monitoring Best Practices

1. **Review logs weekly** - Check for unusual patterns
2. **Monitor disk space** - Ensure logs aren't filling disk
3. **Track restart frequency** - Frequent restarts indicate problems
4. **Update scripts** - Keep monitoring scripts current
5. **Test manually** - Periodically run scripts manually to verify

## Custom Monitoring Scripts

To add custom monitoring for specific services:

```bash
# Create custom script
ssh jared@15.204.247.153 "nano ~/services/custom-monitor.sh"

# Make executable
ssh jared@15.204.247.153 "chmod +x ~/services/custom-monitor.sh"

# Add to cron
ssh jared@15.204.247.153 "crontab -e"
# Add: */15 * * * * /home/jared/services/custom-monitor.sh
```

## System Health Commands

In addition to automated monitoring, these commands provide quick system health checks:

```bash
# Overall system health (custom alias)
ssh jared@15.204.247.153 "syshealth"

# Log summary (custom alias)
ssh jared@15.204.247.153 "logsummary"

# Docker resource usage
ssh jared@15.204.247.153 "docker system df"

# Disk usage
ssh jared@15.204.247.153 "df -h"

# Memory usage
ssh jared@15.204.247.153 "free -h"

# Load average
ssh jared@15.204.247.153 "uptime"
```

## See Also

- [Services Inventory](./services-inventory.md) - Current services being monitored
- [CLAUDE.md](../CLAUDE.md) - General server operations
- [Documentation Maintenance](./documentation-maintenance.md) - Keeping docs current