57 lines
2.6 KiB
Markdown
57 lines
2.6 KiB
Markdown
|
# Relaystation Runbook
|
||
|
|
|
||
|
|
This runbook covers the most common on-call scenarios for Relaystation, the
|
||
|
|
webhook and event relay service described in `docs/architecture.md`.
|
||
|
|
|
||
|
|
## Symptom: rising `events.dropped` counter
|
||
|
|
|
||
|
|
Relaystation drops events when the in-memory dispatch queue is full or when
|
||
|
|
retries exhaust `MAX_ATTEMPTS` in `src/retry.js`. Check `queue depth` log
|
||
|
|
lines from `[queue]` first — a steadily climbing queue depth almost always
|
||
|
|
means a downstream worker or dispatcher has stopped making progress, not
|
||
|
|
that traffic has genuinely spiked.
|
||
|
|
|
||
|
|
Steps:
|
||
|
|
|
||
|
|
1. Grep the relay log for `[queue] queue depth` and confirm whether depth is
|
||
|
|
climbing monotonically rather than oscillating with normal load.
|
||
|
|
2. Check worker heartbeat logs for a `heartbeat missed` or `marked dead`
|
||
|
|
entry around the same time window. A dead worker reduces delivery
|
||
|
|
capacity, which is a common root cause of queue growth even when
|
||
|
|
producer traffic is flat.
|
||
|
|
3. If a worker is dead, restart it. The queue should drain once capacity is
|
||
|
|
restored; there is no automatic recovery today.
|
||
|
|
4. If no worker issue is found, check downstream service latency — a slow
|
||
|
|
downstream can back up the queue just as effectively as a dead worker.
|
||
|
|
|
||
|
|
## Symptom: burst of `signature token expired` warnings
|
||
|
|
|
||
|
|
This indicates a tenant's issued auth token expired mid-session. Relaystation
|
||
|
|
does not auto-renew tokens; the producer must reissue one.
|
||
|
|
|
||
|
|
Steps:
|
||
|
|
|
||
|
|
1. Confirm the tenant named in the `[auth]` warning lines.
|
||
|
|
2. Check whether `INFO [retry] scheduling retry (attempt N) after 401` lines
|
||
|
|
are climbing for the same tenant — this is expected fallout, since a 401
|
||
|
|
triggers the standard retry path rather than a fast-fail.
|
||
|
|
3. Ask the tenant's integration owner to reissue a signed token. Once valid
|
||
|
|
requests resume, the retry backlog for that tenant drains on its own.
|
||
|
|
4. If the retry volume is large enough to threaten other tenants' delivery
|
||
|
|
latency, consider temporarily disabling the offending route in
|
||
|
|
`services/` rather than waiting out the backlog.
|
||
|
|
|
||
|
|
## Escalation
|
||
|
|
|
||
|
|
If RelayStation is dropping events for more than 15 minutes with no
|
||
|
|
identified root cause, page the platform on-call rotation rather than
|
||
|
|
continuing to investigate solo — queue backlogs compound quickly once
|
||
|
|
retries start stacking on top of fresh traffic.
|
||
|
|
|
||
|
|
## Useful commands
|
||
|
|
|
||
|
|
- `node -e "require('./src/metrics').snapshot()"` — not directly usable from
|
||
|
|
a separate process, but illustrates which counters exist.
|
||
|
|
- Tail `logs/relay-YYYY-MM-DD.log` and `logs/worker-YYYY-MM-DD.log` together,
|
||
|
|
sorted by timestamp, to correlate dispatcher and worker behavior.
|