jared
/
server-ovh-prod-01-main
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
server-ovh-prod-01-main/docs/monitoring-automation.md

6.8 KiB
Raw Permalink Blame History

Monitoring & Automation

This document describes the automated monitoring and maintenance systems running on systems-prod-01.

Overview

The server has two automated scripts running via cron to ensure service health and prevent resource exhaustion:

  1. Health Monitor - Runs every 10 minutes, monitors and restarts unhealthy containers
  2. Daily Maintenance - Runs at 3 AM EST, performs cleanup and optimization

Health Monitor

Script: ~/services/container-health-monitor.sh Schedule: Every 10 minutes Log: ~/services/monitor.log

What It Does

  • Monitors container health status
  • Checks memory usage for each container
  • Restarts containers if memory exceeds 40%
  • Checks site availability (HTTP status codes)
  • Cleans up Docker resources (unused volumes, images)
  • Logs all actions with timestamps

Manual Commands

# View recent monitoring activity
ssh jared@15.204.247.153 "tail -50 ~/services/monitor.log"

# Run health check manually
ssh jared@15.204.247.153 "~/services/container-health-monitor.sh"

# View cron schedule
ssh jared@15.204.247.153 "crontab -l"

# Watch monitoring in real-time
ssh jared@15.204.247.153 "tail -f ~/services/monitor.log"

Memory Threshold Configuration

The 40% memory restart threshold can be adjusted by editing the script:

ssh jared@15.204.247.153 "nano ~/services/container-health-monitor.sh"

# Look for the line:
# MEMORY_THRESHOLD=40

Daily Maintenance

Script: ~/services/daily-maintenance.sh Schedule: Daily at 3:00 AM EST Log: ~/services/maintenance.log

What It Does

  • Clears temporary files and cache
  • Optimizes database tables (WordPress, etc.)
  • Performs controlled container restarts
  • Prevents memory buildup
  • Prunes unused Docker resources
  • Logs maintenance summary

Manual Commands

# View recent maintenance activity
ssh jared@15.204.247.153 "tail -50 ~/services/maintenance.log"

# Run maintenance manually (caution: will restart containers)
ssh jared@15.204.247.153 "~/services/daily-maintenance.sh"

# View maintenance schedule
ssh jared@15.204.247.153 "crontab -l | grep maintenance"

Cron Configuration

To view or modify the cron schedule:

# View current cron jobs
ssh jared@15.204.247.153 "crontab -l"

# Edit cron schedule (if needed)
ssh jared@15.204.247.153 "crontab -e"

Current Schedule:

# Health monitor - every 10 minutes
*/10 * * * * /home/jared/services/container-health-monitor.sh

# Daily maintenance - 3 AM EST
0 3 * * * /home/jared/services/daily-maintenance.sh

Log Management

Log Locations

  • Health monitor: ~/services/monitor.log
  • Daily maintenance: ~/services/maintenance.log
  • Docker logs: docker logs CONTAINER_NAME
  • Traefik logs: ~/services/traefik/logs/

Log Analysis Commands

# Count container restarts today
ssh jared@15.204.247.153 "grep 'restarting' ~/services/monitor.log | grep $(date +%Y-%m-%d) | wc -l"

# Find memory-related restarts
ssh jared@15.204.247.153 "grep 'memory exceeds' ~/services/monitor.log | tail -20"

# View maintenance summaries
ssh jared@15.204.247.153 "grep 'Maintenance complete' ~/services/maintenance.log | tail -10"

# Check for errors in logs
ssh jared@15.204.247.153 "grep -i error ~/services/monitor.log | tail -20"

Log Rotation

Logs are automatically rotated via system logrotate configuration to prevent disk space issues.

Check logrotate config:

ssh jared@15.204.247.153 "cat /etc/logrotate.d/container-monitoring"

Alerting

Current Status

Status: Basic monitoring only (no external alerting)

Future Improvements

Consider adding:

  • Email notifications for critical events
  • Slack/Discord webhooks for alerts
  • Prometheus + Grafana for metrics visualization
  • Uptime monitoring (UptimeRobot, etc.)

Troubleshooting

Container Keeps Restarting

Symptom: Health monitor constantly restarts a container Possible Causes:

  • Memory leak in application
  • Memory threshold too low
  • Container crash loop

Investigation:

# Check container logs
ssh jared@15.204.247.153 "docker logs CONTAINER_NAME --tail 100"

# Check memory usage trends
ssh jared@15.204.247.153 "docker stats --no-stream"

# Review health monitor logs
ssh jared@15.204.247.153 "grep CONTAINER_NAME ~/services/monitor.log | tail -20"

Monitoring Script Not Running

Symptom: No recent entries in monitor.log Possible Causes:

  • Cron service stopped
  • Script permissions issue
  • Script path incorrect

Investigation:

# Check cron service
ssh jared@15.204.247.153 "systemctl status cron"

# Check script permissions
ssh jared@15.204.247.153 "ls -la ~/services/container-health-monitor.sh"

# Test script manually
ssh jared@15.204.247.153 "~/services/container-health-monitor.sh"

Maintenance Window Issues

Symptom: 3 AM maintenance causing problems Solution: Adjust maintenance time

# Edit cron schedule
ssh jared@15.204.247.153 "crontab -e"

# Change from 3 AM to different time (example: 2 AM)
# From: 0 3 * * * /home/jared/services/daily-maintenance.sh
# To:   0 2 * * * /home/jared/services/daily-maintenance.sh

Health Check Endpoints

Services should implement health check endpoints for better monitoring:

Example Docker Compose health check:

services:
  myservice:
    image: myapp:latest
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s

Monitoring Best Practices

  1. Review logs weekly - Check for unusual patterns
  2. Monitor disk space - Ensure logs aren't filling disk
  3. Track restart frequency - Frequent restarts indicate problems
  4. Update scripts - Keep monitoring scripts current
  5. Test manually - Periodically run scripts manually to verify

Custom Monitoring Scripts

To add custom monitoring for specific services:

# Create custom script
ssh jared@15.204.247.153 "nano ~/services/custom-monitor.sh"

# Make executable
ssh jared@15.204.247.153 "chmod +x ~/services/custom-monitor.sh"

# Add to cron
ssh jared@15.204.247.153 "crontab -e"
# Add: */15 * * * * /home/jared/services/custom-monitor.sh

System Health Commands

In addition to automated monitoring, these commands provide quick system health checks:

# Overall system health (custom alias)
ssh jared@15.204.247.153 "syshealth"

# Log summary (custom alias)
ssh jared@15.204.247.153 "logsummary"

# Docker resource usage
ssh jared@15.204.247.153 "docker system df"

# Disk usage
ssh jared@15.204.247.153 "df -h"

# Memory usage
ssh jared@15.204.247.153 "free -h"

# Load average
ssh jared@15.204.247.153 "uptime"

See Also