3.6 KiB
Per-service readiness probes
Relaystation has no way to ask "is downstream service X currently healthy"
before dispatching to it — router.resolve matches purely on topic prefix,
with no health awareness. This spec defines eleven new probe modules under
src/probes/ that each answer that question for one target, using rules
distinct enough per target that no shared template covers more than one.
Conventions
'use strict';, plain object export viamodule.exports = { ... }, same style assrc/metrics.js.- No new npm dependencies; Node core modules only.
- ~20-40 lines per file including requires and exports.
- Tests are not required for this pass.
Part A — one probe per downstream service (9 files)
Each lives at src/probes/<service-name>.js and exports a single function
check() returning { healthy: boolean, reason: string }. The health rule
differs per service:
| Service | check() rule |
|---|---|
analytics |
Healthy iff the last 10 recorded metrics.get('events.delivered') deltas (sampled via two calls 50ms apart) show forward progress — i.e. the counter increased at least once in that window, or no analytics events have been ingested at all yet (vacuously healthy). |
audit |
Healthy iff metrics.get('events.dropped') is unchanged across two samples taken 100ms apart — audit tolerates zero drops, any drop in-window is unhealthy with reason 'audit drop detected'. |
billing |
Healthy iff metrics.get('retry.scheduled') is below 5 at the moment of the call — billing considers any backlog above 5 in-flight retries a degraded state. |
crm |
Always healthy unless metrics.get('events.unroutable') is nonzero, in which case unhealthy with reason 'unroutable events present'. |
email |
Healthy iff a lightweight TCP connect (via net.connect, 500ms timeout, immediately destroyed) to email.internal.relaystation.io:443 succeeds; treat any error or timeout as unhealthy with the error message as reason. |
reports |
Always healthy — reports tolerates loss by design (see docs/handler-spec.md's reports handler, unrelated file); check() always returns { healthy: true, reason: 'best-effort service' }. |
search |
Healthy iff metrics.get('events.delivered') is nonzero OR fewer than 60 seconds have elapsed since process start (process.uptime() < 60) — search is allowed a cold-start grace period. |
sms |
Healthy iff metrics.get('retry.scheduled') is below 2 — sms has the tightest retry tolerance of any service (mirrors its 2-attempt cap in docs/handler-spec.md, unrelated file). |
webhooks |
Healthy iff metrics.get('events.dropped') divided by (metrics.get('events.ingested') or 1) is below 0.05 (a 5% drop-rate threshold). |
Part B — two core probes (2 files)
src/probes/queue.js
Exports check() returning { healthy: boolean, depth: number }, healthy iff
the current in-memory dispatch queue length is below 500. Read the queue via
require('../relay').pendingQueue.length (already exported by src/relay.js).
src/probes/broker.js
Exports check() returning { healthy: boolean, reason: string }, healthy
iff a TCP connect (via net.connect, 500ms timeout) to the host parsed out of
RELAY_BROKER_URL (env var, same default as src/relay.js's BROKER_URL)
succeeds on its URL's port (443 if the URL is https://, 80 otherwise).
Expected files
src/probes/analytics.jssrc/probes/audit.jssrc/probes/billing.jssrc/probes/crm.jssrc/probes/email.jssrc/probes/reports.jssrc/probes/search.jssrc/probes/sms.jssrc/probes/webhooks.jssrc/probes/queue.jssrc/probes/broker.js