cc-os/plugins/os-orchestration/eval/fixture/project/docs/probe-spec.md

Per-service readiness probes

Relaystation has no way to ask "is downstream service X currently healthy" before dispatching to it — router.resolve matches purely on topic prefix, with no health awareness. This spec defines eleven new probe modules under src/probes/ that each answer that question for one target, using rules distinct enough per target that no shared template covers more than one.

Conventions

• 'use strict';, plain object export via module.exports = { ... }, same style as src/metrics.js.
• No new npm dependencies; Node core modules only.
• ~20-40 lines per file including requires and exports.
• Tests are not required for this pass.

Part A — one probe per downstream service (9 files)

Each lives at src/probes/<service-name>.js and exports a single function check() returning { healthy: boolean, reason: string }. The health rule differs per service:

Service	check() rule
analytics	Healthy iff the last 10 recorded metrics.get('events.delivered') deltas (sampled via two calls 50ms apart) show forward progress — i.e. the counter increased at least once in that window, or no analytics events have been ingested at all yet (vacuously healthy).
audit	Healthy iff metrics.get('events.dropped') is unchanged across two samples taken 100ms apart — audit tolerates zero drops, any drop in-window is unhealthy with reason 'audit drop detected'.
billing	Healthy iff metrics.get('retry.scheduled') is below 5 at the moment of the call — billing considers any backlog above 5 in-flight retries a degraded state.
crm	Always healthy unless metrics.get('events.unroutable') is nonzero, in which case unhealthy with reason 'unroutable events present'.
email	Healthy iff a lightweight TCP connect (via net.connect, 500ms timeout, immediately destroyed) to email.internal.relaystation.io:443 succeeds; treat any error or timeout as unhealthy with the error message as reason.
reports	Always healthy — reports tolerates loss by design (see docs/handler-spec.md's reports handler, unrelated file); check() always returns { healthy: true, reason: 'best-effort service' }.
search	Healthy iff metrics.get('events.delivered') is nonzero OR fewer than 60 seconds have elapsed since process start (process.uptime() < 60) — search is allowed a cold-start grace period.
sms	Healthy iff metrics.get('retry.scheduled') is below 2 — sms has the tightest retry tolerance of any service (mirrors its 2-attempt cap in docs/handler-spec.md, unrelated file).
webhooks	Healthy iff metrics.get('events.dropped') divided by (metrics.get('events.ingested') or 1) is below 0.05 (a 5% drop-rate threshold).

Part B — two core probes (2 files)

src/probes/queue.js

Exports check() returning { healthy: boolean, depth: number }, healthy iff the current in-memory dispatch queue length is below 500. Read the queue via require('../relay').pendingQueue.length (already exported by src/relay.js).

src/probes/broker.js

Exports check() returning { healthy: boolean, reason: string }, healthy iff a TCP connect (via net.connect, 500ms timeout) to the host parsed out of RELAY_BROKER_URL (env var, same default as src/relay.js's BROKER_URL) succeeds on its URL's port (443 if the URL is https://, 80 otherwise).

Expected files

• src/probes/analytics.js
• src/probes/audit.js
• src/probes/billing.js
• src/probes/crm.js
• src/probes/email.js
• src/probes/reports.js
• src/probes/search.js
• src/probes/sms.js
• src/probes/webhooks.js
• src/probes/queue.js
• src/probes/broker.js