# Relaystation Runbook

This runbook covers the most common on-call scenarios for Relaystation, the
webhook and event relay service described in `docs/architecture.md`.

## Symptom: rising `events.dropped` counter

Relaystation drops events when the in-memory dispatch queue is full or when
retries exhaust `MAX_ATTEMPTS` in `src/retry.js`. Check `queue depth` log
lines from `[queue]` first — a steadily climbing queue depth almost always
means a downstream worker or dispatcher has stopped making progress, not
that traffic has genuinely spiked.

Steps:

1. Grep the relay log for `[queue] queue depth` and confirm whether depth is
   climbing monotonically rather than oscillating with normal load.
2. Check worker heartbeat logs for a `heartbeat missed` or `marked dead`
   entry around the same time window. A dead worker reduces delivery
   capacity, which is a common root cause of queue growth even when
   producer traffic is flat.
3. If a worker is dead, restart it. The queue should drain once capacity is
   restored; there is no automatic recovery today.
4. If no worker issue is found, check downstream service latency — a slow
   downstream can back up the queue just as effectively as a dead worker.

## Symptom: burst of `signature token expired` warnings

This indicates a tenant's issued auth token expired mid-session. Relaystation
does not auto-renew tokens; the producer must reissue one.

Steps:

1. Confirm the tenant named in the `[auth]` warning lines.
2. Check whether `INFO [retry] scheduling retry (attempt N) after 401` lines
   are climbing for the same tenant — this is expected fallout, since a 401
   triggers the standard retry path rather than a fast-fail.
3. Ask the tenant's integration owner to reissue a signed token. Once valid
   requests resume, the retry backlog for that tenant drains on its own.
4. If the retry volume is large enough to threaten other tenants' delivery
   latency, consider temporarily disabling the offending route in
   `services/` rather than waiting out the backlog.

## Escalation

If RelayStation is dropping events for more than 15 minutes with no
identified root cause, page the platform on-call rotation rather than
continuing to investigate solo — queue backlogs compound quickly once
retries start stacking on top of fresh traffic.

## Useful commands

- `node -e "require('./src/metrics').snapshot()"` — not directly usable from
  a separate process, but illustrates which counters exist.
- Tail `logs/relay-YYYY-MM-DD.log` and `logs/worker-YYYY-MM-DD.log` together,
  sorted by timestamp, to correlate dispatcher and worker behavior.
