cc-os/plugins/os-orchestration/eval/fixture/project/docs/handler-spec.md

# Per-service delivery handlers

Relaystation currently delivers every event through the single generic path in
`src/relay.js` (`deliver` → `fetchLike`, with failures handed to `src/retry.js`).
That path treats every downstream service identically: same payload shape, same
retry policy, same failure handling. In practice each of the nine downstream
integrations in `services/` has different tolerance for retries, different
payload requirements, and different failure semantics, and three cross-cutting
concerns (auth concurrency, per-service latency visibility, and dead-letter
handling) need their own dedicated modules.

This spec defines twelve new handler modules under `src/handlers/`. Each module
is self-contained and does not require changes to `src/relay.js`, `src/router.js`,
`src/retry.js`, or any file under `services/` — implement each file to the
interface and behavior below.

## Conventions

- `'use strict';` at the top, matching the rest of `src/`.
- Each module exports a plain object via `module.exports = { ... }` (same style
  as `src/metrics.js` / `src/store.js`), not a class.
- Where a handler needs the shared metrics counters, `require('../metrics')`
  and call `increment(name)` — do not create a second counter store.
- Each file should be small: roughly 20–40 lines including requires and
  `module.exports`.
- No new npm dependencies. Use only Node core modules already used elsewhere
  in `src/` (`crypto`, `fs`, `path`, `http`) plus what a handler needs.
- Tests are not required for this pass — implementation only.

## Part A — one handler per downstream service (9 files)

Each service handler lives at `src/handlers/<service-name>.js` and exports two
functions:

```js
module.exports = {
  prepare(event),   // returns the outbound payload object for this service
  isRetryable(err), // returns true/false given an error object with a
                     // `statusCode` field (may be undefined for network errors)
};
```

`err.statusCode` is `undefined` for a connection/network failure (no response
received) and an HTTP status integer when a response was received but was not
2xx. Implement `prepare` and `isRetryable` exactly per the per-service rules
below — the rules differ enough between services that no single template
satisfies more than one of them.

| Service | `prepare(event)` payload rule | `isRetryable(err)` rule |
| --- | --- | --- |
| `analytics` | Wrap the event: return `{ batch: [event], emitted_at: Date.now() }`. | Retry when `statusCode` is undefined (network) or `>= 500`. Never retry on any 4xx. |
| `audit` | Return the event plus an integrity field: `{ ...event, hash: sha256-hex of JSON.stringify(event) }` (use `crypto.createHash('sha256')`). | Retry **only** on a network failure (`statusCode === undefined`). Any received HTTP status, 4xx or 5xx, is not retryable — audit delivery failures must surface immediately rather than silently retry. |
| `billing` | Return a payload containing **only** an allowlist of fields copied from `event`, dropping everything else: `id`, `topic`, `amount`, `currency`, `receivedAt`. Fields absent on the event are simply omitted from the output, not set to `null`. | Retry only on `statusCode === 429` or `statusCode === 503`. Nothing else (not network failures, not other 5xx) is retryable. |
| `crm` | Return a shallow-transformed payload: rename `topic` to `event_type`, and if `event.payload` has a nested `contact` object, flatten it to top-level `contact_id` and `contact_email` fields (copied from `event.payload.contact.id` / `.email` if present) instead of nesting it. | Retry when `statusCode` is undefined or `>= 500`, same as `analytics` — but cap distinctly (see Part B note below; the cap itself lives in the retry loop, not in this predicate). |
| `email` | If `event.payload` is missing a `to` field, do not build a payload at all — call `metrics.increment('handlers.email.invalid')` and return `null` (the caller is expected to skip sending when `prepare` returns `null`). Otherwise return the event unchanged. | Retry when `statusCode` is undefined, `429`, or `>= 500`. |
| `reports` | Collapse the event into a single-field summary payload: `{ summary: \`${event.topic} at ${event.receivedAt}\` }` (use the actual template literal). | Never retryable — always return `false`, regardless of `err`. Reports tolerates loss; a single attempt is sufficient. |
| `search` | Return the event plus an `index_hint` field computed as the substring of `event.topic` before its first `.` (or the whole topic if there is no `.`). | Retry when `statusCode` is undefined or `>= 500` (same predicate shape as `analytics`/`crm`, but see Part B — `search` gets the longest retry budget of any service). |
| `sms` | Return the event with `payload.message` truncated to 160 characters if present and longer than that (leave other fields untouched). | Retry when `statusCode` is undefined or `statusCode >= 500`, but explicitly **not** on timeouts represented as `err.code === 'ETIMEDOUT'` — treat a timeout as non-retryable (carrier cost control), even though it has no `statusCode`. |
| `webhooks` | Return the event unchanged, with one added field: `relay_version` read from `package.json`'s `version` field (`require('../../package.json').version`). | Retry on any non-2xx status, i.e. `statusCode === undefined || statusCode < 200 || statusCode >= 300`. |

Note on retry *counts*: `isRetryable` only decides whether a given failure is a
candidate for another attempt at all — it does not encode the attempt cap.
Attempt caps are documented per-service in Part C for reference by any future
caller; `src/handlers/*.js` files themselves do not need to enforce the cap
(that remains `src/retry.js`'s job when it is later wired up — not part of
this pass).

## Part B — three core handlers (3 files)

These are structurally different from the service handlers above — each
addresses a different cross-cutting concern, not a per-service payload/retry
rule.

### `src/handlers/auth.js` — per-tenant concurrency limiter

Tracks how many deliveries are currently in flight for each tenant and enforces
a maximum concurrency of 4 simultaneous in-flight deliveries per tenant. Export:

```js
module.exports = {
  acquire(tenant), // returns true if the tenant is under its concurrency cap
                    // (and increments its in-flight count), false if the
                    // tenant is already at the cap (does not increment)
  release(tenant),  // decrements the tenant's in-flight count, floored at 0
  inFlight(tenant),  // returns the current in-flight count for a tenant (0 if unseen)
};
```

Keep the per-tenant counts in a plain object keyed by tenant name, module-scoped
(same pattern as `tokenCache` in `src/auth.js` or `counters` in
`src/metrics.js`). The concurrency cap (4) should be a named constant at the
top of the file.

### `src/handlers/metrics.js` — per-service latency histogram

Records delivery latency (milliseconds) per service and can report p50/p95.
Export:

```js
module.exports = {
  record(service, latencyMs), // append a sample for that service
  p50(service),                 // median of recorded samples for that service, or null if none
  p95(service),                 // 95th percentile (nearest-rank) for that service, or null if none
  reset(),                       // clear all recorded samples
};
```

Store samples per service in an array (module-scoped object keyed by service
name). Percentile implementation: sort the samples ascending, then for `p95`
take the sample at index `Math.ceil(0.95 * n) - 1` (nearest-rank method,
clamped to a valid index); `p50` uses `0.5` the same way. This is intentionally
a different aggregation shape than the simple integer counters in
`src/metrics.js` — do not just wrap the existing counters module.

### `src/handlers/store.js` — dead-letter compaction

When an event has exhausted its retry attempts (mirrors `MAX_ATTEMPTS` in
`src/retry.js`, currently 8) instead of being silently dropped it should be
appended to a dead-letter file for manual review, and the in-memory dead-letter
list should be periodically compacted to drop entries older than 7 days. Export:

```js
module.exports = {
  deadLetter(target, event, attempts), // appends { target, event, attempts, at: Date.now() }
                                         // to data/deadletter.log (JSON line, same
                                         // append-on-write pattern as src/store.js's
                                         // `append`) AND to an in-memory list
  compact(now), // removes in-memory entries older than 7 days (7 * 24 * 60 * 60 * 1000 ms)
                 // relative to `now` (defaults to Date.now() if not passed);
                 // does not touch the on-disk log
  list(),        // returns the current in-memory dead-letter list
};
```

Use `path.join(__dirname, '..', '..', 'data', 'deadletter.log')` for the file
path (mirrors `DATA_DIR`/`LOG_FILE` in `src/store.js`), and create the `data/`
directory if it doesn't exist before appending, same as `src/store.js` does.

## Part C — reference: attempt caps (for future wiring, not required this pass)

| Service | Max attempts |
| --- | --- |
| analytics | 5 |
| audit | 1 (no retry) |
| billing | 4 |
| crm | 3 |
| email | 6 |
| reports | 1 (no retry) |
| search | 10 |
| sms | 2 |
| webhooks | 8 (matches `src/retry.js` `MAX_ATTEMPTS`) |

## Expected files

Implement all twelve files listed below. Each is independent of the other
eleven — none of the twelve requires reading or importing another file in this
list.

- `src/handlers/analytics.js`
- `src/handlers/audit.js`
- `src/handlers/billing.js`
- `src/handlers/crm.js`
- `src/handlers/email.js`
- `src/handlers/reports.js`
- `src/handlers/search.js`
- `src/handlers/sms.js`
- `src/handlers/webhooks.js`
- `src/handlers/auth.js`
- `src/handlers/metrics.js`
- `src/handlers/store.js`