# Invoice Ninja

**Type**: Invoicing & Billing Application
**Domain**: https://invoice.hyperthrive.io
**Version**: 5.12.28
**Architecture**: Official Docker stack from [invoiceninja/dockerfiles](https://github.com/invoiceninja/dockerfiles)
**Deployment Date**: 2025-10-14

## Architecture

Invoice Ninja runs using the official Docker Compose stack with three services:

1. **invoiceninja-server-1** - Nginx web server (port 80 internal)
2. **invoiceninja-app-1** - Laravel application with supervisord (PHP-FPM, queue workers, scheduler)
3. **invoiceninja-db-1** - MariaDB 10.11 database

### Traefik Integration

The nginx service connects to the external `traefik` network and uses labels for automatic SSL/routing:

- **Domain**: invoice.hyperthrive.io
- **Protocol**: HTTPS via Let's Encrypt
- **Entry Point**: websecure (443)
- **Internal Port**: 80 (nginx)

### Directory Structure (Server)

```
~/services/invoiceninja/
├── docker-compose.yml          # Main compose file
├── env                         # Environment configuration
├── config/
│   └── nginx/
│       └── in-vhost.conf       # Nginx vhost config
├── docker/
│   ├── app/
│   │   ├── public/            # Application public files (shared with nginx)
│   │   └── storage/           # Application storage (logs, cache, uploads)
│   └── mysql/
│       └── data/              # Database files
```

**Important**: The `docker/app/` directories must be owned by user `1500:1500` (the container's runtime user).

## Common Operations

### Check Status

```bash
ssh jared@15.204.247.153 "docker ps --filter name=invoiceninja"
```

### View Logs

```bash
# Application logs
ssh jared@15.204.247.153 "docker logs invoiceninja-app-1 --tail 50"

# Database logs
ssh jared@15.204.247.153 "docker logs invoiceninja-db-1 --tail 50"

# Nginx logs
ssh jared@15.204.247.153 "docker logs invoiceninja-server-1 --tail 50"
```

### Update Invoice Ninja

```bash
ssh jared@15.204.247.153 "cd ~/services/invoiceninja && \
  docker compose pull && \
  docker compose up -d && \
  docker exec invoiceninja-app-1 php artisan migrate --force && \
  docker exec invoiceninja-app-1 php artisan optimize"
```

### Backup Database

```bash
ssh jared@15.204.247.153 "cd ~/services/invoiceninja && \
  docker compose exec -T db sh -c 'exec mysqldump --all-databases -uroot -p\"\$MYSQL_ROOT_PASSWORD\"' \
  > ~/backups/invoiceninja-$(date +%Y%m%d-%H%M%S).sql"
```

### Restore Database

```bash
ssh jared@15.204.247.153 "cd ~/services/invoiceninja && \
  docker compose exec -T db sh -c 'exec mysql -uroot -p\"\$MYSQL_ROOT_PASSWORD\"' \
  < ~/backups/invoiceninja-TIMESTAMP.sql"
```

### Restart Services

```bash
ssh jared@15.204.247.153 "cd ~/services/invoiceninja && docker compose restart"
```

### Run Artisan Commands

```bash
# Generic pattern
ssh jared@15.204.247.153 "docker exec invoiceninja-app-1 php artisan <command>"

# Examples
ssh jared@15.204.247.153 "docker exec invoiceninja-app-1 php artisan queue:work"
ssh jared@15.204.247.153 "docker exec invoiceninja-app-1 php artisan cache:clear"
ssh jared@15.204.247.153 "docker exec invoiceninja-app-1 php artisan db:seed"
```

## Key Configuration

### Environment Variables (env file)

Critical settings in `~/services/invoiceninja/env`:

- `APP_URL=https://invoice.hyperthrive.io` - Public URL
- `APP_KEY=base64:...` - Encryption key (DO NOT change after initial setup)
- `DB_HOST=db` - Database hostname (container name)
- `MYSQL_ROOT_PASSWORD` - DB root password
- `MYSQL_DATABASE=ninja` - Database name
- `MYSQL_USER=ninja` - DB user
- `MYSQL_PASSWORD` - DB password
- `TRUSTED_PROXIES=*` - Trust Traefik proxy
- `MAIL_MAILER=log` - Mail driver (currently logging only)

### Traefik Labels (docker-compose.yml)

```yaml
labels:
  - "traefik.enable=true"
  - "traefik.http.routers.invoiceninja.rule=Host(`invoice.hyperthrive.io`)"
  - "traefik.http.routers.invoiceninja.entrypoints=websecure"
  - "traefik.http.routers.invoiceninja.tls.certresolver=letsencrypt"
  - "traefik.http.services.invoiceninja.loadbalancer.server.port=80"
  - "traefik.docker.network=traefik"
```

## Troubleshooting

### Permission Denied Errors

If the app container fails with "Permission denied" creating directories:

```bash
ssh jared@15.204.247.153 "cd ~/services/invoiceninja && \
  docker compose down && \
  sudo chown -R 1500:1500 docker/app && \
  docker compose up -d"
```

### Database Connection Issues

Check database is healthy:
```bash
ssh jared@15.204.247.153 "docker inspect invoiceninja-db-1 --format='{{.State.Health.Status}}'"
```

Should return `healthy`. If not, check logs.

### 502 Bad Gateway

Usually means the app container isn't running or PHP-FPM hasn't started:

1. Check app container status: `docker ps --filter name=invoiceninja-app-1`
2. Check app logs: `docker logs invoiceninja-app-1 --tail 100`
3. Look for PHP-FPM startup message: "ready to handle connections"

### SSL Certificate Issues

Traefik automatically handles SSL. If certificate issues occur:

1. Check Traefik logs: `docker logs traefik`
2. Verify DNS points to server: `dig invoice.hyperthrive.io`
3. Check Traefik dashboard for router/certificate status

### Queue Not Processing

The official stack includes queue workers via supervisord. Check they're running:

```bash
ssh jared@15.204.247.153 "docker exec invoiceninja-app-1 supervisorctl status"
```

Should show `queue-worker_00` and `queue-worker_01` as RUNNING.

### Client Portal 500 Errors on Invoice View (pdf_variables bug — IN 5.13.22)

**Symptom**: All clients get HTTP 500 when viewing invoices via the client portal.

**Laravel log signature** (check `docker/app/storage/logs/laravel.log` or `docker logs invoiceninja-app-1`):

```
Undefined property: stdClass::$product_columns in PdfSlot.php:149
```

**Root cause**: The `pdf_variables` field in company settings was never properly initialized (left as empty object `{}`). `PdfSlot.php` crashes when it tries to access named columns on the empty object. This is an unreported upstream bug in Invoice Ninja 5.13.22 with no upstream fix as of 2026-05-15.

**Fix** — run once to populate the defaults:

```bash
docker exec invoiceninja-app-1 php artisan tinker --execute="
\$company = App\Models\Company::first();
\$settings = \$company->settings;
\$defaults = App\DataMapper\CompanySettings::getEntityVariableDefaults();
\$settings->pdf_variables = (object) \$defaults;
\$company->settings = \$settings;
\$company->save();
echo 'Done: ' . json_encode(\$company->fresh()->settings->pdf_variables->product_columns);
"
```

A successful run prints the populated `product_columns` array. No restart required — the fix takes effect immediately.

### MariaDB Health Check Unhealthy

**Symptom**: `docker ps` shows `invoiceninja-db-1` with status `(unhealthy)`.

**Root cause**: The healthcheck user's password in the container's `.my-healthcheck.cnf` is out of sync with the actual MariaDB user password (can happen after env changes or manual password resets).

**Fix** — sync the password:

```bash
docker exec invoiceninja-db-1 mariadb -uroot -p<ROOT_PASS> -e \
  "ALTER USER 'healthcheck'@'localhost' IDENTIFIED BY '<password_from_.my-healthcheck.cnf>'"
```

Replace `<ROOT_PASS>` with the value of `MYSQL_ROOT_PASSWORD` from `~/services/invoiceninja/env`, and `<password_from_.my-healthcheck.cnf>` with the password found in that file inside the container:

```bash
docker exec invoiceninja-db-1 cat /root/.my-healthcheck.cnf
```

After running the ALTER USER, the container transitions to `healthy` within one health-check interval (typically 30 s).

### Duplicate Stripe Payment Gateway

**Symptom**: Two Stripe gateways appear in Settings > Payment Gateways, or payments route through an old/invalid gateway config.

**Fix**: Delete the older/unused gateway entry from the Invoice Ninja admin UI (Settings > Payment Gateways → delete). Keep only the active gateway.

**Also check**: After any gateway cleanup, verify `accepted_credit_cards` is non-zero. A value of `0` means no card types are accepted; set it to `7` (Visa + Mastercard + Amex) or the desired bitmask via Settings > Payment Gateways > Edit.

## Migration History

This deployment uses the **official Invoice Ninja Docker stack** (as of 2025-10-14). Previous custom stack was replaced to align with upstream recommendations and simplify maintenance.

**Pre-migration backup location**: `~/backups/invoiceninja-pre-official/` on server (retained until deployment proven stable)

## Incidents

### 2026-05-15 — Client Portal 500 / MariaDB Unhealthy / Duplicate Gateway

**Symptoms observed:**
- All clients received HTTP 500 on invoice view in the client portal
- `invoiceninja-db-1` had been showing `(unhealthy)` for 29 days
- Two Stripe payment gateways were present; `accepted_credit_cards` was `0`

**Root causes & fixes:**

| Issue | Root cause | Fix applied |
|-------|------------|-------------|
| Portal 500 errors | `pdf_variables` field never initialized (`{}`); `PdfSlot.php:149` crashes on `->product_columns` | Populated defaults via `tinker` (see troubleshooting section above) |
| MariaDB unhealthy | Healthcheck user password out of sync with `.my-healthcheck.cnf` | `ALTER USER 'healthcheck'@'localhost' IDENTIFIED BY '...'` |
| Duplicate Stripe gateway | Old gateway left over from previous config | Deleted stale gateway from admin UI |
| `accepted_credit_cards: 0` | Never set after initial setup | Set to `7` (Visa + Mastercard + Amex) in gateway settings |

**Notes:**
- The `pdf_variables` bug is unreported upstream as of 2026-05-15 (Invoice Ninja 5.13.22). Watch future releases for a permanent fix.
- The MariaDB health check had been broken for 29 days before discovery — consider adding an external health alert for this container.

## References

- Official Docker Repository: https://github.com/invoiceninja/dockerfiles
- Invoice Ninja Docs: https://invoiceninja.github.io
- Server Profile: `profile.yaml`
- Configuration Template: `configs/invoiceninja-official-template.yml`