63 lines
3.6 KiB
Markdown
63 lines
3.6 KiB
Markdown
# Per-service readiness probes
|
|
|
|
Relaystation has no way to ask "is downstream service X currently healthy"
|
|
before dispatching to it — `router.resolve` matches purely on topic prefix,
|
|
with no health awareness. This spec defines eleven new probe modules under
|
|
`src/probes/` that each answer that question for one target, using rules
|
|
distinct enough per target that no shared template covers more than one.
|
|
|
|
## Conventions
|
|
|
|
- `'use strict';`, plain object export via `module.exports = { ... }`, same
|
|
style as `src/metrics.js`.
|
|
- No new npm dependencies; Node core modules only.
|
|
- ~20-40 lines per file including requires and exports.
|
|
- Tests are not required for this pass.
|
|
|
|
## Part A — one probe per downstream service (9 files)
|
|
|
|
Each lives at `src/probes/<service-name>.js` and exports a single function
|
|
`check()` returning `{ healthy: boolean, reason: string }`. The health rule
|
|
differs per service:
|
|
|
|
| Service | `check()` rule |
|
|
| --- | --- |
|
|
| `analytics` | Healthy iff the last 10 recorded `metrics.get('events.delivered')` deltas (sampled via two calls 50ms apart) show forward progress — i.e. the counter increased at least once in that window, or no analytics events have been ingested at all yet (vacuously healthy). |
|
|
| `audit` | Healthy iff `metrics.get('events.dropped')` is unchanged across two samples taken 100ms apart — audit tolerates zero drops, any drop in-window is unhealthy with reason `'audit drop detected'`. |
|
|
| `billing` | Healthy iff `metrics.get('retry.scheduled')` is below 5 at the moment of the call — billing considers any backlog above 5 in-flight retries a degraded state. |
|
|
| `crm` | Always healthy unless `metrics.get('events.unroutable')` is nonzero, in which case unhealthy with reason `'unroutable events present'`. |
|
|
| `email` | Healthy iff a lightweight TCP connect (via `net.connect`, 500ms timeout, immediately destroyed) to `email.internal.relaystation.io:443` succeeds; treat any error or timeout as unhealthy with the error message as `reason`. |
|
|
| `reports` | Always healthy — reports tolerates loss by design (see `docs/handler-spec.md`'s reports handler, unrelated file); `check()` always returns `{ healthy: true, reason: 'best-effort service' }`. |
|
|
| `search` | Healthy iff `metrics.get('events.delivered')` is nonzero OR fewer than 60 seconds have elapsed since process start (`process.uptime() < 60`) — search is allowed a cold-start grace period. |
|
|
| `sms` | Healthy iff `metrics.get('retry.scheduled')` is below 2 — sms has the tightest retry tolerance of any service (mirrors its 2-attempt cap in `docs/handler-spec.md`, unrelated file). |
|
|
| `webhooks` | Healthy iff `metrics.get('events.dropped')` divided by (`metrics.get('events.ingested')` or 1) is below 0.05 (a 5% drop-rate threshold). |
|
|
|
|
## Part B — two core probes (2 files)
|
|
|
|
### `src/probes/queue.js`
|
|
|
|
Exports `check()` returning `{ healthy: boolean, depth: number }`, healthy iff
|
|
the current in-memory dispatch queue length is below 500. Read the queue via
|
|
`require('../relay').pendingQueue.length` (already exported by `src/relay.js`).
|
|
|
|
### `src/probes/broker.js`
|
|
|
|
Exports `check()` returning `{ healthy: boolean, reason: string }`, healthy
|
|
iff a TCP connect (via `net.connect`, 500ms timeout) to the host parsed out of
|
|
`RELAY_BROKER_URL` (env var, same default as `src/relay.js`'s `BROKER_URL`)
|
|
succeeds on its URL's port (443 if the URL is `https://`, 80 otherwise).
|
|
|
|
## Expected files
|
|
|
|
- `src/probes/analytics.js`
|
|
- `src/probes/audit.js`
|
|
- `src/probes/billing.js`
|
|
- `src/probes/crm.js`
|
|
- `src/probes/email.js`
|
|
- `src/probes/reports.js`
|
|
- `src/probes/search.js`
|
|
- `src/probes/sms.js`
|
|
- `src/probes/webhooks.js`
|
|
- `src/probes/queue.js`
|
|
- `src/probes/broker.js`
|