cc-os/plugins/os-context/eval/fixture/project/docs/runbook.md

Relaystation Runbook

This runbook covers the most common on-call scenarios for Relaystation, the webhook and event relay service described in docs/architecture.md.

Symptom: rising events.dropped counter

Relaystation drops events when the in-memory dispatch queue is full or when retries exhaust MAX_ATTEMPTS in src/retry.js. Check queue depth log lines from [queue] first — a steadily climbing queue depth almost always means a downstream worker or dispatcher has stopped making progress, not that traffic has genuinely spiked.

Steps:

1. Grep the relay log for [queue] queue depth and confirm whether depth is climbing monotonically rather than oscillating with normal load.
2. Check worker heartbeat logs for a heartbeat missed or marked dead entry around the same time window. A dead worker reduces delivery capacity, which is a common root cause of queue growth even when producer traffic is flat.
3. If a worker is dead, restart it. The queue should drain once capacity is restored; there is no automatic recovery today.
4. If no worker issue is found, check downstream service latency — a slow downstream can back up the queue just as effectively as a dead worker.

Symptom: burst of signature token expired warnings

This indicates a tenant's issued auth token expired mid-session. Relaystation does not auto-renew tokens; the producer must reissue one.

Steps:

1. Confirm the tenant named in the [auth] warning lines.
2. Check whether INFO [retry] scheduling retry (attempt N) after 401 lines are climbing for the same tenant — this is expected fallout, since a 401 triggers the standard retry path rather than a fast-fail.
3. Ask the tenant's integration owner to reissue a signed token. Once valid requests resume, the retry backlog for that tenant drains on its own.
4. If the retry volume is large enough to threaten other tenants' delivery latency, consider temporarily disabling the offending route in services/ rather than waiting out the backlog.

Escalation

If RelayStation is dropping events for more than 15 minutes with no identified root cause, page the platform on-call rotation rather than continuing to investigate solo — queue backlogs compound quickly once retries start stacking on top of fresh traffic.

Useful commands

• node -e "require('./src/metrics').snapshot()" — not directly usable from a separate process, but illustrates which counters exist.
• Tail logs/relay-YYYY-MM-DD.log and logs/worker-YYYY-MM-DD.log together, sorted by timestamp, to correlate dispatcher and worker behavior.