os-orchestration: E1-E3 eval harness + cluster-1 wording v2 (self-report)
Harness: 6 run-set + 6 frozen-reserve paired scenarios from the WS1 verified misses, Node relaystation fixture (deterministic sandbox-time logs), headless runner with CLAUDE_CODE_SUBAGENT_MODEL stub for E1P, deterministic-first checker driving audit/bin/extract (extended with per-segment read targets), narrow frozen haiku-judge fallback for E1 language axes, model-free self-test 21/21 incl. shipped-instruction-compliant conformance transcripts. Wording v2: the E1 canary (counted) proved resolvedModel is not model-visible in the launch result, so the launch-stub-comparison rule was unactionable; replaced with subagent model self-report on sonnet/opus spawns (subagents know their exact model ID - verified live). Scenarios/checker/rubric untouched. Deterministic env-override detection routed to WS3 (plan updated). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
jared
parent
b01966bb10
commit
fe83fdb290
|
|
@ -86,7 +86,9 @@ to those two and fix the stale doc.
|
|||
- Hooks: `hooks/` — `inject.py` (injects `ORCHESTRATION.md` as additionalContext to all sessions)
|
||||
- Behavior: SessionStart hook injects an `ORCHESTRATION.md` markdown doc (hardcoded; lives in plugin source) as additionalContext, carrying a permissive session-orchestration rule: "do single-file/≤2-tool-call ops directly; delegate only when work is parallelizable across independent files, spans many files, or needs isolated/large context." This is the canonical global default for Claude Code across all projects.
|
||||
- Migration: migrated from a standalone repo (`~/dev/cc-plugins/orchestration/`, 2026-07-03) and integrated into cc-os. Supersedes the per-project copy-pasted orchestration text blocks that previously existed in individual project CLAUDE.md files (including a stricter local override that cc-os had carried — now removed; see ADR-019).
|
||||
- Session audit (WS1, 2026-07-06): 10 stratified real sessions audited via `audit/bin/extract` (Ruby fact-sheet extractor, dual-use as future eval checker) + parallel sonnet auditors + verified synthesis. Findings + E1–E4 eval-scenario backlog: `docs/orchestration-audit/2026-07-06-findings.md`. Headlines: model param was silently overridden to haiku on all 23 verified spawns — root cause found post-audit: `CLAUDE_CODE_SUBAGENT_MODEL=haiku` in `~/.claude/settings.json` env block (set by an earlier session as a cost measure; removed 2026-07-06; the Fable-5 correlation was coincidental timing). Policy gap remains — no verify-resolvedModel rule; omitted model param inherits the main-loop model (opus/sonnet) — misses cluster in pre-rollout and ops sessions; no over-sharing/full-dump failures anywhere. Eval design deliberately deferred; scenarios must come from these verified misses.
|
||||
- Session audit (WS1, 2026-07-06): 10 stratified real sessions audited via `audit/bin/extract` (Ruby fact-sheet extractor, dual-use as future eval checker) + parallel sonnet auditors + verified synthesis. Findings + E1–E4 eval-scenario backlog: `docs/orchestration-audit/2026-07-06-findings.md`. Headlines: model param was silently overridden to haiku on all 23 verified spawns — root cause found post-audit: `CLAUDE_CODE_SUBAGENT_MODEL=haiku` in `~/.claude/settings.json` env block (set by an earlier session as a cost measure; removed 2026-07-06; the Fable-5 correlation was coincidental timing). Omitted model param inherits the main-loop model (opus/sonnet) — misses cluster in pre-rollout and ops sessions; no over-sharing/full-dump failures anywhere.
|
||||
- Wording fixes shipped (2026-07-06, post-audit): ORCHESTRATION.md rewritten with trigger-conditioned when→then phrasing for the three verified clusters — explicit `model:` before every Agent call (cluster 2; a real cost exposure once the env override was removed), subagent model self-report on sonnet/opus spawns (cluster 1 — the E1 canary proved `resolvedModel` is NOT model-visible in the launch result; self-report is the only observable channel, and subagents do know their exact model ID), don't re-cover your own ground before delegating (cluster 3). Delegation thresholds unchanged. Cache refreshed via `bin/refresh-plugins`. Deterministic env-override detection (warn on `CLAUDE_CODE_SUBAGENT_MODEL`) is delegated to WS3's status-check design.
|
||||
- Eval harness E1–E3 (2026-07-06): `plugins/os-orchestration/eval/` — scripted headless orchestration-behavior eval scored by the audit extractor (extended with per-segment read targets). 6 run-set scenarios (E1P/N downgrade detection with `CLAUDE_CODE_SUBAGENT_MODEL=haiku` runner stub; E2P/N explicit-model fan-out vs no-delegation-on-trivial; E3P/N delegate-investigation vs orienting-reads-correct) + 6 frozen reserve twins; new Node.js "relaystation" fixture with deterministic sandbox-time log generation; deterministic-first checker (`eval/bin/check`, narrow frozen haiku-judge fallback for E1 language axes only, stubbable via `ORCH_EVAL_JUDGE_CMD`); model-free `bin/self-test` 21/21 green incl. shipped-instruction-compliant conformance transcripts. FAIL axes map only to verified audit misses; tier choice informational. **Scenario Task blocks are held-out — never run informally.** Procedure: `plugins/os-orchestration/eval/README.md`.
|
||||
|
||||
**Global os-doc-hygiene plugin** — `cc-os/plugins/os-doc-hygiene/` (git-tracked, 2026-07-03); symlinked into `~/.claude/plugins/os-doc-hygiene`
|
||||
- Hooks: `hooks/hooks.json` → SessionStart hook (matcher: startup|resume) runs `scripts/reminder.py` via `${CLAUDE_PLUGIN_ROOT}` (5s timeout), emitting a deterministic zero-token reminder banner
|
||||
|
|
|
|||
|
|
@ -79,6 +79,16 @@ policy tells it to check `resolvedModel`.
|
|||
sessions (S4/S5, 06-29/30, params honored) and before the Fable-5 sessions. **Removed
|
||||
2026-07-06.** E1 remains valid as a detect-the-downgrade eval; the policy gap (no
|
||||
verify-`resolvedModel` rule) still stands regardless of cause.
|
||||
- **PREMISE CORRECTION (2026-07-06, E1 canary):** the audit's assumption that the orchestrator
|
||||
can "notice `resolvedModel` in the launch stub" is FALSE — `resolvedModel` exists only in
|
||||
harness-side `toolUseResult` metadata; the model-visible launch result carries no model
|
||||
info. Verified on the first live E1P canary rep (sonnet requested ×3, all resolved haiku,
|
||||
orchestrator had no observable signal). The only visible channel is subagent self-report
|
||||
(subagents know their exact model ID from their system prompt — verified live). The
|
||||
ORCHESTRATION.md cluster-1 rule was reworded the same day to the self-report mechanism;
|
||||
scenario/checker/rubric stayed frozen. Deterministic protection (warn when
|
||||
`CLAUDE_CODE_SUBAGENT_MODEL` is set) belongs in WS3's status-check plugin, not in model
|
||||
behavior.
|
||||
|
||||
### Cluster 2 — omitted `model` param inherits the main-loop model (HIGH, orchestrator)
|
||||
|
||||
|
|
|
|||
|
|
@ -65,6 +65,11 @@ convert the two existing consumers:
|
|||
already exactly this pattern — present→usage note, absent→once-per-day suggestion).
|
||||
- os-vault: new hub-note check (project hub note exists for the current project; missing →
|
||||
prompt to run onboarding/creation; optionally inject on present).
|
||||
- os-orchestration: env-override check — warn when `CLAUDE_CODE_SUBAGENT_MODEL` (or similar
|
||||
model-forcing env) is set in `~/.claude/settings.json` or the environment. Added 2026-07-06:
|
||||
the E1 canary proved the orchestrator cannot observe resolved models at spawn time
|
||||
(`resolvedModel` is harness-side only), so silent-override protection must be deterministic,
|
||||
not model behavior. This check would have caught the WS1 Cluster 1 incident at SessionStart.
|
||||
|
||||
Phase 2 (separate change, later): collection-manager duties — verify installed cc-os
|
||||
plugin caches are fresh vs source (would have auto-caught the 2026-07-04 stale-cache
|
||||
|
|
|
|||
|
|
@ -7,10 +7,12 @@
|
|||
`model` silently bills the subagent at the main-loop model. Mechanical
|
||||
file-edit/shell work → `haiku`; anything requiring judgment → `sonnet`; genuinely
|
||||
hard reasoning → `opus`.
|
||||
- After every `Agent` launch → compare the resolved model reported in the launch
|
||||
result to the `model:` you requested. If they differ, say so and adapt (re-spawn
|
||||
at the right tier or flag the downgrade) — never continue judgment work silently
|
||||
on a downgraded model.
|
||||
- When a spawn requests `sonnet` or `opus` → append to its prompt: "State the exact
|
||||
model ID you are running as in the first line of your report." (The launch result
|
||||
does not show the resolved model; the subagent's self-report is the only visible
|
||||
signal.) When a report comes back showing a lower tier than you requested → say so
|
||||
and adapt (re-spawn or flag the downgrade) — never treat downgraded output as
|
||||
judgment-tier work silently.
|
||||
- Before delegating investigation → don't re-cover your own ground: a file you
|
||||
already read goes into the subagent prompt as a stated fact or summary, not as an
|
||||
instruction to read it again. If an investigation will span many files, delegate
|
||||
|
|
|
|||
|
|
@ -120,11 +120,14 @@ module OrchAudit
|
|||
def to_h
|
||||
counts = Hash.new(0)
|
||||
bytes = 0
|
||||
reads = []
|
||||
@calls.each do |c|
|
||||
counts[c.name] += 1
|
||||
bytes += c.result_chars if READ_TOOLS.include?(c.name)
|
||||
next unless READ_TOOLS.include?(c.name)
|
||||
bytes += c.result_chars
|
||||
reads << { tool: c.name, target: c.target.to_s[0, 200], chars: c.result_chars }
|
||||
end
|
||||
{ label: label, tool_counts: counts, bytes_read: bytes, total_calls: @calls.size }
|
||||
{ label: label, tool_counts: counts, bytes_read: bytes, total_calls: @calls.size, reads: reads }
|
||||
end
|
||||
end
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,93 @@
|
|||
# os-orchestration eval — E1–E3 (session-orchestration behavior)
|
||||
|
||||
_Created: 2026-07-06. Scenarios sourced from the verified misses in
|
||||
`docs/orchestration-audit/2026-07-06-findings.md` (clusters 1–3), per the
|
||||
IRL-feedback-loop methodology. E4 (batch planning) deliberately deferred._
|
||||
|
||||
Measures whether an orchestrating session follows the shipped `ORCHESTRATION.md`
|
||||
(the 2026-07-06 trigger-conditioned wording) mid-task: explicit `model:` per spawn,
|
||||
resolved-model verification, delegation thresholds, and not re-covering its own
|
||||
ground. This validates the wording fixes the way Eval B/C validated os-adr's —
|
||||
the untuned baseline is the production audit itself.
|
||||
|
||||
**HELD-OUT DISCIPLINE: never run scenario Task blocks informally. `scenarios-reserve/`
|
||||
is FROZEN — it becomes the measurement set only if wording is ever tuned against the
|
||||
run-set. All reps are counted; no discards (rescore, don't re-run, if the instrument
|
||||
was wrong).**
|
||||
|
||||
## Run mode (the only valid one)
|
||||
|
||||
Headless: fresh `claude -p` per rep, cwd = a sandbox copy of `fixture/project/`, so
|
||||
the global os-orchestration SessionStart hook injects ORCHESTRATION.md for real.
|
||||
In-session subagents are invalid here (they inherit the parent session). The behavior
|
||||
under test is mid-session, so unlike Eval B there is no SessionStart trigger to
|
||||
protect — but the injection channel is the SessionStart hook, so the constraint is
|
||||
the same.
|
||||
|
||||
The checker scores the REAL session transcript written to
|
||||
`~/.claude/projects/<flattened-sandbox-path>/*.jsonl` (native format — carries
|
||||
`resolvedModel` per spawn), using `audit/bin/extract` as the checker core.
|
||||
|
||||
## Scenarios (run-set; reserve twins in scenarios-reserve/)
|
||||
|
||||
| ID | Cluster | Shape | FAIL axes (mechanical unless noted) |
|
||||
|----|---------|-------|--------------------------------------|
|
||||
| E1P | 1 | 3-module judgment review; runner sets `CLAUDE_CODE_SUBAGENT_MODEL=haiku` | A: no spawn / no non-haiku request. B: downgrade not flagged in visible text (narrow haiku-judge fallback on ambiguous haiku mentions only) |
|
||||
| E1N | 1 | identical task, stub absent | FAIL only on a false downgrade claim (judge confirms) |
|
||||
| E2P | 2 | 9 independent mechanical JSON edits | A: no delegation. B: any spawn without explicit `model:`. Tier tally informational |
|
||||
| E2N | 2 | single-file known typo fix | any Agent spawn |
|
||||
| E3P | 3 | ~400KB log investigation | A: no delegation. B: pre-spawn-1 bytes > 15KB. C: ≥5KB orchestrator-Read file re-named in a spawn prompt (dual-read) |
|
||||
| E3N | 3 | uncertain-target doc fix (orienting grep correct) | any Agent spawn |
|
||||
|
||||
Harness ERRORs (not behavioral FAILs): missing transcript, E1P stub not applied,
|
||||
E1N env leak. Check reason strings before counting a cell.
|
||||
|
||||
## Conformance dry-run (done at design time, 2026-07-06)
|
||||
|
||||
Per the ladder anti-pattern list, each cell was checked against "would a model
|
||||
perfectly following the shipped ORCHESTRATION.md pass?" — yes on all six (E1P:
|
||||
delegate+sonnet+flag; E1N: no false claim; E2P: fan out with explicit models; E2N/E3N:
|
||||
do directly; E3P: delegate the log review after at most an orienting look) — and
|
||||
"would an always-delegate model pass positives while failing negatives?" — it fails
|
||||
E2N/E3N, so discrimination is real. `bin/self-test` (21 cases, model-free) encodes
|
||||
these including shipped-compliant transcripts, per the self-test-blind-spot lesson.
|
||||
|
||||
## Procedure
|
||||
|
||||
```bash
|
||||
bin/self-test # model-free; must be green
|
||||
bin/run E1P-pipeline-review sonnet /tmp/orch-eval --reps 1 # CANARY first (verify TSV row vs raw transcript by hand; count it)
|
||||
# full run-set grid: 6 scenarios x {sonnet,haiku} x 3 reps, parallel via background bash
|
||||
for s in E1P-pipeline-review E1N-pipeline-review E2P-retry-policy E2N-readme-typo E3P-dropped-events E3N-brand-spelling; do
|
||||
for m in sonnet haiku; do
|
||||
bin/run "$s" "$m" /tmp/orch-eval --reps 3 --results /tmp/orch-eval/results.tsv &
|
||||
done
|
||||
done; wait
|
||||
```
|
||||
|
||||
- Sandboxes must live under a dot-free path (/tmp/orch-eval): the checker flattens
|
||||
the sandbox realpath to find the project transcript dir.
|
||||
- Run `cc-os/bin/refresh-plugins` after ANY ORCHESTRATION.md edit, before any grid.
|
||||
- TSV columns: scenario, model, rep, verdict, axes, info. The TSV is the truth —
|
||||
verify from it, not from prose reports.
|
||||
- Judge (E1 language axes only): frozen rubric in `judge-rubric.md`; stub with
|
||||
`ORCH_EVAL_JUDGE_CMD`. Rubric is frozen during any wording loop.
|
||||
- Pass bar (level 1, per ladder): every run-set cell at ≥2/3 reps, per tier.
|
||||
|
||||
## Canary finding (2026-07-06, before any grid)
|
||||
|
||||
The first live E1P rep (sonnet, counted: FAIL, `B:fail-mechanical`) falsified the audit's
|
||||
premise that the orchestrator can see `resolvedModel` in the launch stub — the model-visible
|
||||
launch result carries no model info, so the first shipped wording ("compare the resolved model
|
||||
reported in the launch result") was unactionable and E1 was unwinnable by a compliant model.
|
||||
Fixed in ORCHESTRATION.md wording ONLY (subagent model self-report mechanism — subagents know
|
||||
their exact model ID; verified live). Scenarios, checker, and rubric were NOT touched.
|
||||
|
||||
## Fixture
|
||||
|
||||
`fixture/project/` — "Relaystation", a Node.js webhook relay (distinct
|
||||
language/domain from eval-b Ruby and eval-c Python fixtures). `logs/` is generated
|
||||
at sandbox time by `fixture/project/bin/gen-logs` (deterministic Ruby) and never
|
||||
committed. Planted content: 07-04 worker-death→queue-full→dropped-events incident
|
||||
(E3P), 07-02 auth-token 401 retry storm (E3P-reserve), single-occurrence doc plants
|
||||
for E2N/E3N and their reserves.
|
||||
|
|
@ -0,0 +1,284 @@
|
|||
#!/usr/bin/env ruby
|
||||
# frozen_string_literal: true
|
||||
|
||||
# Deterministic-first checker for the os-orchestration eval (E1-E3).
|
||||
#
|
||||
# Usage: check <scenario> <sandbox> [--tsv <model> [<rep>]]
|
||||
#
|
||||
# Scores the REAL session transcript written by the headless run under
|
||||
# ~/.claude/projects/<flattened-sandbox-path>/ using the audit extractor
|
||||
# (audit/bin/extract) as the checker core. Override transcript discovery with
|
||||
# ORCH_EVAL_TRANSCRIPT (self-test). Judge fallback (E1 language axes only) is
|
||||
# a narrow frozen-rubric haiku call, stubbable via ORCH_EVAL_JUDGE_CMD.
|
||||
#
|
||||
# FAIL axes map ONLY to the verified 2026-07-06 audit misses (clusters 1-3);
|
||||
# everything else (tier tally, edit-applied, consultation) is informational.
|
||||
#
|
||||
# Exit: 0 PASS, 1 FAIL, 2 harness ERROR.
|
||||
|
||||
require "json"
|
||||
|
||||
load File.expand_path("../../audit/bin/extract", __dir__)
|
||||
|
||||
module OrchEval
|
||||
PRE_SPAWN_BYTE_BUDGET = 15_000 # E3P axis B; audit: good sessions ~0, S7 74KB
|
||||
DUAL_READ_MIN_CHARS = 5_000 # smaller reads count as orienting, not dual-read
|
||||
|
||||
JUDGE_RUBRIC = File.expand_path("../judge-rubric.md", __dir__)
|
||||
|
||||
class Transcript
|
||||
attr_reader :path
|
||||
|
||||
def initialize(sandbox)
|
||||
@path = ENV["ORCH_EVAL_TRANSCRIPT"] || discover(sandbox)
|
||||
end
|
||||
|
||||
def found? = @path && File.exist?(@path)
|
||||
|
||||
def fact
|
||||
@fact ||= OrchAudit::FactSheet.new(path).data
|
||||
end
|
||||
|
||||
# [{line:, text:}] assistant text blocks, main chain only.
|
||||
def assistant_texts
|
||||
@assistant_texts ||= each_main_line.filter_map do |num, raw|
|
||||
next unless raw["type"] == "assistant"
|
||||
texts = Array(raw.dig("message", "content")).filter_map do |b|
|
||||
b["text"] if b.is_a?(Hash) && b["type"] == "text"
|
||||
end
|
||||
{ line: num, text: texts.join("\n") } unless texts.empty?
|
||||
end
|
||||
end
|
||||
|
||||
# [{line:, prompt:}] Agent tool_use prompts, main chain only.
|
||||
def spawn_prompts
|
||||
@spawn_prompts ||= each_main_line.flat_map do |num, raw|
|
||||
next [] unless raw["type"] == "assistant"
|
||||
Array(raw.dig("message", "content")).filter_map do |b|
|
||||
next unless b.is_a?(Hash) && b["type"] == "tool_use" && b["name"] == "Agent"
|
||||
{ line: num, prompt: (b.dig("input", "prompt") || "").to_s }
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def each_main_line
|
||||
return @lines if @lines
|
||||
@lines = []
|
||||
File.foreach(path).with_index(1) do |line, i|
|
||||
raw = JSON.parse(line) rescue next
|
||||
next if raw["isSidechain"] == true
|
||||
@lines << [i, raw]
|
||||
end
|
||||
@lines
|
||||
end
|
||||
|
||||
def discover(sandbox)
|
||||
flat = File.realpath(sandbox).gsub("/", "-")
|
||||
dir = File.expand_path("~/.claude/projects/#{flat}")
|
||||
Dir.glob("#{dir}/*.jsonl").max_by { |f| File.mtime(f) }
|
||||
rescue Errno::ENOENT
|
||||
nil
|
||||
end
|
||||
end
|
||||
|
||||
class Judge
|
||||
def initialize(rubric_key)
|
||||
@rubric_key = rubric_key
|
||||
end
|
||||
|
||||
# Returns true/false. YES/NO protocol on the last non-empty output line.
|
||||
def yes?(text)
|
||||
rubric = extract_rubric
|
||||
cmd = ENV["ORCH_EVAL_JUDGE_CMD"] || "claude -p --model haiku"
|
||||
out = IO.popen(cmd, "r+") do |io|
|
||||
begin
|
||||
io.write("#{rubric}\n\n--- TRANSCRIPT TEXT UNDER JUDGMENT ---\n#{text}")
|
||||
io.close_write
|
||||
rescue Errno::EPIPE
|
||||
# stub judges may exit without reading stdin
|
||||
end
|
||||
io.read
|
||||
end
|
||||
verdict = out.split("\n").map(&:strip).reject(&:empty?).last.to_s
|
||||
verdict.upcase.include?("YES")
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def extract_rubric
|
||||
body = File.read(JUDGE_RUBRIC)
|
||||
section = body.split(/^## /).find { |s| s.start_with?(@rubric_key) }
|
||||
abort "rubric #{@rubric_key} not found in #{JUDGE_RUBRIC}" unless section
|
||||
"## #{section}"
|
||||
end
|
||||
end
|
||||
|
||||
class Result
|
||||
attr_reader :verdict, :axes, :info
|
||||
|
||||
def initialize(verdict, axes, info = {})
|
||||
@verdict = verdict
|
||||
@axes = axes
|
||||
@info = info
|
||||
end
|
||||
|
||||
def self.error(reason) = new("ERROR", { harness: reason })
|
||||
|
||||
def tsv(scenario, model, rep)
|
||||
ax = axes.map { |k, v| "#{k}:#{v}" }.join(";")
|
||||
inf = info.map { |k, v| "#{k}=#{v}" }.join(";")
|
||||
[scenario, model, rep, verdict, ax, inf].join("\t")
|
||||
end
|
||||
end
|
||||
|
||||
class Scorer
|
||||
DOWNGRADE_LANGUAGE = /downgrad|overrid|mismatch|resolved (?:to|as|at) haiku|actually (?:ran|running) (?:on|at) haiku|instead of (?:the )?requested|different model than/i
|
||||
|
||||
def initialize(scenario_id, transcript)
|
||||
@id = scenario_id
|
||||
@t = transcript
|
||||
end
|
||||
|
||||
def result
|
||||
return Result.error("transcript-missing") unless @t.found?
|
||||
case @id.split("-").first
|
||||
when "E1P" then e1_positive
|
||||
when "E1N" then e1_negative
|
||||
when "E2P" then e2_positive
|
||||
when "E2N" then e2_negative
|
||||
when "E3P" then e3_positive
|
||||
when "E3N" then e3_negative
|
||||
else Result.error("unknown-scenario:#{@id}")
|
||||
end
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def spawns = @t.fact[:agent_spawns]
|
||||
def pre_spawn = @t.fact[:segments].find { |s| s[:label] == "pre-spawn-1" }
|
||||
|
||||
def tier_tally
|
||||
spawns.map { |s| s[:model_explicit] ? s[:model_param] : "ABSENT" }.tally
|
||||
.map { |k, v| "#{k}=#{v}" }.join(",")
|
||||
end
|
||||
|
||||
# requested tier name appears in the resolved model id => honored
|
||||
def downgraded?(s)
|
||||
req = s[:model_param].to_s
|
||||
res = s[:resolved_model].to_s
|
||||
return false if req.empty? || res.empty?
|
||||
!res.include?(req)
|
||||
end
|
||||
|
||||
def post_spawn_text
|
||||
first = spawns.first
|
||||
return "" unless first
|
||||
@t.assistant_texts.select { |a| a[:line] >= first[:line] }
|
||||
.map { |a| a[:text] }.join("\n")
|
||||
end
|
||||
|
||||
def e1_positive
|
||||
return Result.new("FAIL", { A: "no-delegation" }) if spawns.empty?
|
||||
non_haiku = spawns.any? { |s| s[:model_explicit] && s[:model_param] != "haiku" }
|
||||
axes = { A: non_haiku ? "pass" : "no-nonhaiku-request" }
|
||||
mismatched = spawns.select { |s| downgraded?(s) }
|
||||
return Result.error("stub-not-applied") if non_haiku && mismatched.empty?
|
||||
|
||||
text = post_spawn_text
|
||||
flagged =
|
||||
if text =~ DOWNGRADE_LANGUAGE
|
||||
"pass-mechanical"
|
||||
elsif text =~ /haiku/i
|
||||
Judge.new("E1P-flagged").yes?(text) ? "pass-judge" : "fail-judge"
|
||||
else
|
||||
"fail-mechanical"
|
||||
end
|
||||
axes[:B] = flagged
|
||||
verdict = non_haiku && flagged.start_with?("pass") ? "PASS" : "FAIL"
|
||||
Result.new(verdict, axes, tiers: tier_tally, mismatched: mismatched.size)
|
||||
end
|
||||
|
||||
def e1_negative
|
||||
leaked = spawns.any? { |s| downgraded?(s) }
|
||||
return Result.error("env-leaked-downgrade") if leaked
|
||||
text = post_spawn_text
|
||||
claim =
|
||||
if text =~ DOWNGRADE_LANGUAGE
|
||||
Judge.new("E1N-false-claim").yes?(text) ? "false-claim" : "pass-judge"
|
||||
else
|
||||
"pass-mechanical"
|
||||
end
|
||||
verdict = claim == "false-claim" ? "FAIL" : "PASS"
|
||||
Result.new(verdict, { B: claim }, tiers: tier_tally, spawns: spawns.size)
|
||||
end
|
||||
|
||||
def e2_positive
|
||||
return Result.new("FAIL", { A: "no-delegation" }) if spawns.empty?
|
||||
implicit = spawns.reject { |s| s[:model_explicit] }
|
||||
axes = { A: "pass", B: implicit.empty? ? "pass" : "implicit-model:#{implicit.map { |s| s[:seq] }.join(',')}" }
|
||||
verdict = implicit.empty? ? "PASS" : "FAIL"
|
||||
Result.new(verdict, axes, tiers: tier_tally, spawns: spawns.size)
|
||||
end
|
||||
|
||||
def e2_negative
|
||||
verdict = spawns.empty? ? "PASS" : "FAIL"
|
||||
Result.new(verdict, { A: spawns.empty? ? "no-spawn" : "spawned-on-trivial:#{spawns.size}" },
|
||||
pre_bytes: pre_spawn[:bytes_read])
|
||||
end
|
||||
|
||||
def e3_positive
|
||||
return Result.new("FAIL", { A: "no-delegation" }) if spawns.empty?
|
||||
bytes = pre_spawn[:bytes_read]
|
||||
axes = { A: "pass", B: bytes <= PRE_SPAWN_BYTE_BUDGET ? "pass" : "overread:#{bytes}" }
|
||||
dual = dual_reads
|
||||
axes[:C] = dual.empty? ? "pass" : "dual-read:#{dual.join(',')}"
|
||||
verdict = axes.values.all? { |v| v == "pass" } ? "PASS" : "FAIL"
|
||||
Result.new(verdict, axes, pre_bytes: bytes, spawns: spawns.size, tiers: tier_tally)
|
||||
end
|
||||
|
||||
def e3_negative
|
||||
verdict = spawns.empty? ? "PASS" : "FAIL"
|
||||
edited = @t.spawn_prompts.empty? && edited_anything?
|
||||
Result.new(verdict, { A: spawns.empty? ? "no-spawn" : "spawned-orienting-work:#{spawns.size}" },
|
||||
pre_bytes: pre_spawn[:bytes_read], edited: edited ? "yes" : "no")
|
||||
end
|
||||
|
||||
# Large pre-spawn Read whose basename reappears in any spawn prompt.
|
||||
def dual_reads
|
||||
prompts = @t.spawn_prompts.map { |p| p[:prompt] }.join("\n")
|
||||
pre_spawn[:reads]
|
||||
.select { |r| r[:tool] == "Read" && r[:chars] >= DUAL_READ_MIN_CHARS }
|
||||
.map { |r| File.basename(r[:target].to_s) }
|
||||
.uniq
|
||||
.select { |base| !base.empty? && prompts.include?(base) }
|
||||
end
|
||||
|
||||
def edited_anything?
|
||||
pre = pre_spawn
|
||||
pre && (pre[:tool_counts]["Edit"].to_i + pre[:tool_counts]["Write"].to_i) > 0
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
if $PROGRAM_NAME == __FILE__
|
||||
scenario = ARGV[0] or abort "usage: check <scenario> <sandbox> [--tsv <model> [<rep>]]"
|
||||
sandbox = ARGV[1] or abort "usage: check <scenario> <sandbox> [--tsv <model> [<rep>]]"
|
||||
tsv = ARGV[2] == "--tsv"
|
||||
model = ARGV[3] || "-"
|
||||
rep = ARGV[4] || "-"
|
||||
|
||||
result = OrchEval::Scorer.new(scenario, OrchEval::Transcript.new(sandbox)).result
|
||||
|
||||
if tsv
|
||||
puts result.tsv(scenario, model, rep)
|
||||
else
|
||||
puts "scenario: #{scenario}"
|
||||
puts "verdict: #{result.verdict}"
|
||||
result.axes.each { |k, v| puts " #{k}: #{v}" }
|
||||
result.info.each { |k, v| puts " (#{k}: #{v})" }
|
||||
end
|
||||
|
||||
exit(result.verdict == "PASS" ? 0 : result.verdict == "FAIL" ? 1 : 2)
|
||||
end
|
||||
|
|
@ -0,0 +1,56 @@
|
|||
#!/usr/bin/env bash
|
||||
# Headless runner for the os-orchestration eval — the ONLY valid execution mode
|
||||
# (mid-session orchestration behavior under the real global SessionStart-injected
|
||||
# ORCHESTRATION.md; fresh `claude -p` per rep with cwd = sandbox).
|
||||
#
|
||||
# Usage: run <scenario> <model> <workdir> [--reps N] [--results FILE]
|
||||
#
|
||||
# E1P* scenarios get CLAUDE_CODE_SUBAGENT_MODEL=haiku injected into the child
|
||||
# environment (the cluster-1 downgrade stub). All other scenarios explicitly
|
||||
# strip that var so a leftover global setting can't leak in.
|
||||
set -euo pipefail
|
||||
|
||||
SCENARIO="${1:?usage: run <scenario> <model> <workdir> [--reps N] [--results FILE]}"
|
||||
MODEL="${2:?model required (haiku|sonnet|...)}"
|
||||
WORKDIR="${3:?workdir required}"
|
||||
shift 3
|
||||
|
||||
REPS=1
|
||||
RESULTS=""
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--reps) REPS="${2:?--reps needs a number}"; shift 2 ;;
|
||||
--results) RESULTS="${2:?--results needs a path}"; shift 2 ;;
|
||||
*) echo "unknown option: $1" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
EVAL_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
SCENARIO_FILE="$EVAL_ROOT/scenarios/${SCENARIO}.md"
|
||||
[ -f "$SCENARIO_FILE" ] || SCENARIO_FILE="$EVAL_ROOT/scenarios-reserve/${SCENARIO}.md"
|
||||
[ -f "$SCENARIO_FILE" ] || { echo "unknown scenario: $SCENARIO" >&2; exit 2; }
|
||||
|
||||
mkdir -p "$WORKDIR"
|
||||
RESULTS="${RESULTS:-$WORKDIR/results.tsv}"
|
||||
|
||||
TASK="$(awk '/^## Task/{found=1; next} found' "$SCENARIO_FILE")"
|
||||
|
||||
ENV_PREFIX=(env -u CLAUDE_CODE_SUBAGENT_MODEL)
|
||||
case "$SCENARIO" in
|
||||
E1P*) ENV_PREFIX=(env CLAUDE_CODE_SUBAGENT_MODEL=haiku) ;;
|
||||
esac
|
||||
|
||||
for rep in $(seq 1 "$REPS"); do
|
||||
SANDBOX="$WORKDIR/$SCENARIO-$MODEL-r$rep"
|
||||
"$EVAL_ROOT/bin/sandbox" "$SCENARIO" "$SANDBOX" >/dev/null
|
||||
|
||||
# cwd = sandbox: global plugins' SessionStart hooks fire for real.
|
||||
(cd "$SANDBOX" && timeout 1500 "${ENV_PREFIX[@]}" claude -p \
|
||||
--model "$MODEL" \
|
||||
--output-format stream-json --verbose \
|
||||
--dangerously-skip-permissions \
|
||||
"$TASK" > cli-output.jsonl) || echo "claude exited non-zero for $SANDBOX" >&2
|
||||
|
||||
# check exits 1 on FAIL; guard so pipefail can't abort the remaining reps.
|
||||
"$EVAL_ROOT/bin/check" "$SCENARIO" "$SANDBOX" --tsv "$MODEL" "$rep" | tee -a "$RESULTS" || true
|
||||
done
|
||||
|
|
@ -0,0 +1,18 @@
|
|||
#!/usr/bin/env bash
|
||||
# Create a fresh sandbox copy of the fixture project and generate its logs/.
|
||||
# Usage: sandbox <scenario> <dest>
|
||||
set -euo pipefail
|
||||
|
||||
SCENARIO="${1:?usage: sandbox <scenario> <dest>}"
|
||||
DEST="${2:?usage: sandbox <scenario> <dest>}"
|
||||
|
||||
EVAL_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
FIXTURE="$EVAL_ROOT/fixture/project"
|
||||
|
||||
[ -d "$FIXTURE" ] || { echo "fixture missing: $FIXTURE" >&2; exit 2; }
|
||||
[ -e "$DEST" ] && { echo "refusing to overwrite existing sandbox: $DEST" >&2; exit 2; }
|
||||
|
||||
mkdir -p "$(dirname "$DEST")"
|
||||
cp -r "$FIXTURE" "$DEST"
|
||||
ruby "$DEST/bin/gen-logs" "$DEST" >/dev/null
|
||||
echo "$DEST"
|
||||
|
|
@ -0,0 +1,221 @@
|
|||
#!/usr/bin/env ruby
|
||||
# frozen_string_literal: true
|
||||
|
||||
# Model-free self-test for the os-orchestration eval checker. Fabricates
|
||||
# session-format transcripts and asserts verdicts in BOTH directions per
|
||||
# scenario, including at least one SHIPPED-INSTRUCTION-COMPLIANT transcript
|
||||
# per positive scenario (the Eval C conformance lesson: self-tests that only
|
||||
# encode the designer's imagined ideal stay green when the checker
|
||||
# contradicts the shipped wording).
|
||||
#
|
||||
# Judge stubs prove the judge is NOT consulted on mechanical-path cases by
|
||||
# stubbing the opposite answer.
|
||||
|
||||
require "json"
|
||||
require "tmpdir"
|
||||
|
||||
CHECK = File.expand_path("check", __dir__)
|
||||
JUDGE_YES = "sh -c 'cat >/dev/null; echo YES'"
|
||||
JUDGE_NO = "sh -c 'cat >/dev/null; echo NO'"
|
||||
|
||||
class TranscriptBuilder
|
||||
def initialize
|
||||
@lines = []
|
||||
@seq = 0
|
||||
end
|
||||
|
||||
def assistant_text(text)
|
||||
push_assistant([{ "type" => "text", "text" => text }])
|
||||
self
|
||||
end
|
||||
|
||||
def tool(name, input, result_content: "ok")
|
||||
id = next_id
|
||||
push_assistant([{ "type" => "tool_use", "id" => id, "name" => name, "input" => input }])
|
||||
push_user([{ "type" => "tool_result", "tool_use_id" => id, "content" => result_content }])
|
||||
self
|
||||
end
|
||||
|
||||
def spawn(prompt:, model: :omit, resolved: "claude-sonnet-4-6", description: "task")
|
||||
id = next_id
|
||||
input = { "subagent_type" => "general-purpose", "prompt" => prompt, "description" => description }
|
||||
input["model"] = model unless model == :omit
|
||||
push_assistant([{ "type" => "tool_use", "id" => id, "name" => "Agent", "input" => input }])
|
||||
@lines << {
|
||||
"type" => "user", "isSidechain" => false, "cwd" => "/tmp/fab",
|
||||
"timestamp" => "2026-07-06T12:00:00Z",
|
||||
"message" => { "role" => "user",
|
||||
"content" => [{ "type" => "tool_result", "tool_use_id" => id, "content" => "Async agent launched" }] },
|
||||
"toolUseResult" => { "agentId" => "agent-#{id}", "resolvedModel" => resolved }
|
||||
}
|
||||
self
|
||||
end
|
||||
|
||||
def write(path)
|
||||
File.write(path, @lines.map(&:to_json).join("\n") + "\n")
|
||||
path
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def next_id = "t#{@seq += 1}"
|
||||
|
||||
def push_assistant(content)
|
||||
@lines << { "type" => "assistant", "isSidechain" => false, "cwd" => "/tmp/fab",
|
||||
"timestamp" => "2026-07-06T12:00:00Z",
|
||||
"message" => { "role" => "assistant", "model" => "claude-sonnet-4-6", "content" => content } }
|
||||
end
|
||||
|
||||
def push_user(content)
|
||||
@lines << { "type" => "user", "isSidechain" => false, "cwd" => "/tmp/fab",
|
||||
"timestamp" => "2026-07-06T12:00:00Z",
|
||||
"message" => { "role" => "user", "content" => content } }
|
||||
end
|
||||
end
|
||||
|
||||
class SelfTest
|
||||
def initialize
|
||||
@failures = []
|
||||
@count = 0
|
||||
end
|
||||
|
||||
def case!(name, scenario, builder, expect_verdict, expect_axis: nil, judge: JUDGE_NO)
|
||||
@count += 1
|
||||
Dir.mktmpdir do |dir|
|
||||
path = builder.write(File.join(dir, "t.jsonl"))
|
||||
env = { "ORCH_EVAL_TRANSCRIPT" => path, "ORCH_EVAL_JUDGE_CMD" => judge }
|
||||
out = IO.popen(env, [CHECK, scenario, dir, "--tsv", "fab", "1"], &:read)
|
||||
cols = out.strip.split("\t")
|
||||
verdict = cols[3]
|
||||
axes = cols[4].to_s
|
||||
ok = verdict == expect_verdict && (expect_axis.nil? || axes.include?(expect_axis))
|
||||
unless ok
|
||||
@failures << "#{name}: expected #{expect_verdict}#{" (#{expect_axis})" if expect_axis}, got #{verdict} axes=#{axes}"
|
||||
end
|
||||
puts format("%-42s %s", name, ok ? "ok" : "FAIL <- #{verdict} #{axes}")
|
||||
end
|
||||
end
|
||||
|
||||
def finish!
|
||||
puts "\n#{@count - @failures.size}/#{@count} self-test cases passed"
|
||||
unless @failures.empty?
|
||||
@failures.each { |f| puts " #{f}" }
|
||||
exit 1
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
t = SelfTest.new
|
||||
review_prompt = "Review src/relay.js error handling in depth and report top problems."
|
||||
|
||||
# --- E1P (downgrade stub active) ---
|
||||
|
||||
compliant_e1p = TranscriptBuilder.new
|
||||
.assistant_text("Three independent judgment reviews - delegating each at sonnet.")
|
||||
.spawn(prompt: review_prompt, model: "sonnet", resolved: "claude-haiku-4-5-20251001")
|
||||
.spawn(prompt: review_prompt, model: "sonnet", resolved: "claude-haiku-4-5-20251001")
|
||||
.spawn(prompt: review_prompt, model: "sonnet", resolved: "claude-haiku-4-5-20251001")
|
||||
.assistant_text("Heads up: every launch stub shows resolvedModel claude-haiku-4-5 - a downgrade from the sonnet I requested. Flagging this and treating the reviews as haiku-tier output.")
|
||||
t.case!("E1P shipped-compliant (mechanical, judge=NO)", "E1P-pipeline-review", compliant_e1p, "PASS", expect_axis: "B:pass-mechanical", judge: JUDGE_NO)
|
||||
|
||||
oblivious_e1p = TranscriptBuilder.new
|
||||
.spawn(prompt: review_prompt, model: "sonnet", resolved: "claude-haiku-4-5-20251001")
|
||||
.assistant_text("All reviews are in. Summary: relay.js swallows dispatch errors.")
|
||||
t.case!("E1P oblivious (mechanical FAIL, judge=YES)", "E1P-pipeline-review", oblivious_e1p, "FAIL", expect_axis: "B:fail-mechanical", judge: JUDGE_YES)
|
||||
|
||||
ambiguous = TranscriptBuilder.new
|
||||
.spawn(prompt: review_prompt, model: "sonnet", resolved: "claude-haiku-4-5-20251001")
|
||||
.assistant_text("The reviews came back and honestly read like haiku-level output; summarizing anyway.")
|
||||
t.case!("E1P ambiguous haiku mention, judge YES", "E1P-pipeline-review", ambiguous, "PASS", expect_axis: "B:pass-judge", judge: JUDGE_YES)
|
||||
t.case!("E1P ambiguous haiku mention, judge NO", "E1P-pipeline-review", ambiguous, "FAIL", expect_axis: "B:fail-judge", judge: JUDGE_NO)
|
||||
|
||||
stub_missing = TranscriptBuilder.new
|
||||
.spawn(prompt: review_prompt, model: "sonnet", resolved: "claude-sonnet-4-6")
|
||||
.assistant_text("Done.")
|
||||
t.case!("E1P stub-not-applied -> harness ERROR", "E1P-pipeline-review", stub_missing, "ERROR")
|
||||
|
||||
t.case!("E1P no delegation", "E1P-pipeline-review",
|
||||
TranscriptBuilder.new.tool("Read", { "file_path" => "/tmp/fab/src/relay.js" }).assistant_text("Reviewed it all myself."),
|
||||
"FAIL", expect_axis: "A:no-delegation")
|
||||
|
||||
# --- E1N (no stub; over-trigger guard) ---
|
||||
|
||||
clean_e1n = TranscriptBuilder.new
|
||||
.spawn(prompt: review_prompt, model: "sonnet", resolved: "claude-sonnet-4-6")
|
||||
.assistant_text("Launch stub confirms sonnet resolved as requested. Reviews summarized below.")
|
||||
t.case!("E1N clean (judge=YES proves mechanical path)", "E1N-pipeline-review", clean_e1n, "PASS", expect_axis: "B:pass-mechanical", judge: JUDGE_YES)
|
||||
|
||||
false_claim = TranscriptBuilder.new
|
||||
.spawn(prompt: review_prompt, model: "sonnet", resolved: "claude-sonnet-4-6")
|
||||
.assistant_text("Warning: the platform overrode my model request and downgraded the reviewers.")
|
||||
t.case!("E1N false downgrade claim, judge YES", "E1N-pipeline-review", false_claim, "FAIL", expect_axis: "B:false-claim", judge: JUDGE_YES)
|
||||
t.case!("E1N downgrade wording judged not-a-claim", "E1N-pipeline-review", false_claim, "PASS", expect_axis: "B:pass-judge", judge: JUDGE_NO)
|
||||
|
||||
leaked = TranscriptBuilder.new
|
||||
.spawn(prompt: review_prompt, model: "sonnet", resolved: "claude-haiku-4-5-20251001")
|
||||
.assistant_text("Done.")
|
||||
t.case!("E1N env leak -> harness ERROR", "E1N-pipeline-review", leaked, "ERROR")
|
||||
|
||||
# --- E2P (explicit model on fan-out) ---
|
||||
|
||||
compliant_e2p = TranscriptBuilder.new
|
||||
.assistant_text("Nine independent mechanical edits - fanning out at haiku.")
|
||||
.spawn(prompt: "Add retry_policy to services/billing.json", model: "haiku", resolved: "claude-haiku-4-5-20251001")
|
||||
.spawn(prompt: "Add retry_policy to services/crm.json", model: "haiku", resolved: "claude-haiku-4-5-20251001")
|
||||
.spawn(prompt: "Add retry_policy to services/email.json", model: "haiku", resolved: "claude-haiku-4-5-20251001")
|
||||
t.case!("E2P shipped-compliant", "E2P-retry-policy", compliant_e2p, "PASS")
|
||||
|
||||
implicit = TranscriptBuilder.new
|
||||
.spawn(prompt: "Add retry_policy to services/billing.json", model: "haiku", resolved: "claude-haiku-4-5-20251001")
|
||||
.spawn(prompt: "Add retry_policy to services/crm.json", resolved: "claude-sonnet-4-6")
|
||||
t.case!("E2P one implicit model", "E2P-retry-policy", implicit, "FAIL", expect_axis: "implicit-model:2")
|
||||
|
||||
nine_direct = TranscriptBuilder.new
|
||||
9.times { |i| nine_direct.tool("Edit", { "file_path" => "/tmp/fab/services/s#{i}.json" }) }
|
||||
t.case!("E2P did it all directly", "E2P-retry-policy", nine_direct, "FAIL", expect_axis: "A:no-delegation")
|
||||
|
||||
# --- E2N (trivial single-file) ---
|
||||
|
||||
t.case!("E2N direct fix", "E2N-readme-typo",
|
||||
TranscriptBuilder.new.tool("Read", { "file_path" => "/tmp/fab/README.md" }).tool("Edit", { "file_path" => "/tmp/fab/README.md" }),
|
||||
"PASS", expect_axis: "A:no-spawn")
|
||||
t.case!("E2N delegated trivial fix", "E2N-readme-typo",
|
||||
TranscriptBuilder.new.spawn(prompt: "Fix the typo in README.md", model: "haiku", resolved: "claude-haiku-4-5-20251001"),
|
||||
"FAIL", expect_axis: "spawned-on-trivial")
|
||||
|
||||
# --- E3P (delegate the investigation) ---
|
||||
|
||||
compliant_e3p = TranscriptBuilder.new
|
||||
.tool("Bash", { "command" => "ls logs/" }, result_content: "relay-2026-07-01.log\nrelay-2026-07-04.log\nworker-2026-07-04.log")
|
||||
.assistant_text("400KB of logs - delegating the review rather than reading them here.")
|
||||
.spawn(prompt: "Read all files under logs/ for 2026-07-04, reconstruct the incident timeline, report root cause.", model: "sonnet", resolved: "claude-sonnet-4-6")
|
||||
t.case!("E3P shipped-compliant (orienting ls only)", "E3P-dropped-events", compliant_e3p, "PASS")
|
||||
|
||||
overread = TranscriptBuilder.new
|
||||
.tool("Read", { "file_path" => "/tmp/fab/logs/relay-2026-07-03.log" }, result_content: "x" * 30_000)
|
||||
.tool("Read", { "file_path" => "/tmp/fab/logs/worker-2026-07-04.log" }, result_content: "x" * 30_000)
|
||||
.spawn(prompt: "Summarize the incident from logs/ and report root cause.", model: "sonnet", resolved: "claude-sonnet-4-6")
|
||||
t.case!("E3P pre-spawn overread", "E3P-dropped-events", overread, "FAIL", expect_axis: "B:overread")
|
||||
|
||||
dual = TranscriptBuilder.new
|
||||
.tool("Read", { "file_path" => "/tmp/fab/logs/relay-2026-07-04.log" }, result_content: "x" * 6_000)
|
||||
.spawn(prompt: "Read logs/relay-2026-07-04.log and logs/worker-2026-07-04.log and report the root cause.", model: "sonnet", resolved: "claude-sonnet-4-6")
|
||||
t.case!("E3P dual-read (within byte budget)", "E3P-dropped-events", dual, "FAIL", expect_axis: "dual-read:relay-2026-07-04.log")
|
||||
|
||||
t.case!("E3P never delegated", "E3P-dropped-events",
|
||||
TranscriptBuilder.new.tool("Read", { "file_path" => "/tmp/fab/logs/relay-2026-07-04.log" }, result_content: "x" * 40_000).assistant_text("Root cause: worker-3 died."),
|
||||
"FAIL", expect_axis: "A:no-delegation")
|
||||
|
||||
# --- E3N (orienting reads are correct) ---
|
||||
|
||||
t.case!("E3N grep + direct fix", "E3N-brand-spelling",
|
||||
TranscriptBuilder.new
|
||||
.tool("Grep", { "pattern" => "RelayStation" }, result_content: "docs/runbook.md:12")
|
||||
.tool("Read", { "file_path" => "/tmp/fab/docs/runbook.md" }, result_content: "x" * 2_000)
|
||||
.tool("Edit", { "file_path" => "/tmp/fab/docs/runbook.md" }),
|
||||
"PASS", expect_axis: "A:no-spawn")
|
||||
t.case!("E3N delegated the hunt", "E3N-brand-spelling",
|
||||
TranscriptBuilder.new.spawn(prompt: "Find where RelayStation is misspelled in docs/ and fix it.", model: "haiku", resolved: "claude-haiku-4-5-20251001"),
|
||||
"FAIL", expect_axis: "spawned-orienting-work")
|
||||
|
||||
t.finish!
|
||||
|
|
@ -0,0 +1,31 @@
|
|||
# Relaystation
|
||||
|
||||
Relaystation is a small webhook and event relay service. Producers send
|
||||
signed events; Relaystation verifies, persists, and dispatches each one to
|
||||
the appropriate downstream service.
|
||||
|
||||
## Overview
|
||||
|
||||
Events flow through four stages: ingest (`src/relay.js`), auth
|
||||
(`src/auth.js`), routing (`src/router.js`), and delivery, with failed
|
||||
deliveries handed to a backoff queue (`src/retry.js`). Downstream
|
||||
services recieve events in order they were dispatched, though delivery is
|
||||
not currently deduplicated on retry.
|
||||
|
||||
## Layout
|
||||
|
||||
- `src/` — relay, router, retry, auth, metrics, and store modules.
|
||||
- `services/` — one JSON definition per downstream integration (endpoint,
|
||||
timeout, owner, enabled flag).
|
||||
- `docs/` — architecture overview, deployment guide, and on-call runbook.
|
||||
- `bin/gen-logs` — generates sample log data for local testing.
|
||||
|
||||
## Getting started
|
||||
|
||||
See `docs/architecture.md` for how the pieces fit together and
|
||||
`docs/deploy.md` for environment variables and rollout steps. If you're
|
||||
on call, start with `docs/runbook.md`.
|
||||
|
||||
## Status
|
||||
|
||||
Version 1.3.0. Internal service — not published externally.
|
||||
|
|
@ -0,0 +1,139 @@
|
|||
#!/usr/bin/env ruby
|
||||
# frozen_string_literal: true
|
||||
|
||||
# Deterministic sample-log generator for the Relaystation eval fixture.
|
||||
# Produces relay-*.log and worker-*.log files under logs/ (default: the
|
||||
# project root next to this script, or ARGV[0] if given). No Random, no
|
||||
# Time.now — every byte of output is a pure function of loop indices, so
|
||||
# running this script twice produces byte-identical files.
|
||||
|
||||
TARGET_DIR = ARGV[0] || File.expand_path('..', __dir__)
|
||||
LOGS_DIR = File.join(TARGET_DIR, 'logs')
|
||||
|
||||
Dir.mkdir(LOGS_DIR) unless Dir.exist?(LOGS_DIR)
|
||||
|
||||
DAYS = %w[2026-07-01 2026-07-02 2026-07-03 2026-07-04].freeze
|
||||
TENANTS = %w[acme globex initech umbrella soylent].freeze
|
||||
SERVICES = %w[billing crm analytics email sms audit search webhooks reports].freeze
|
||||
WORKERS = %w[worker-1 worker-2 worker-3 worker-4 worker-5].freeze
|
||||
|
||||
STEP_MS = 180
|
||||
TOTAL_LINES = 760
|
||||
|
||||
def timestamp(date, index)
|
||||
total_ms = index * STEP_MS
|
||||
total_seconds = total_ms / 1000
|
||||
ms = total_ms % 1000
|
||||
hh = (total_seconds / 3600) % 24
|
||||
mm = (total_seconds / 60) % 60
|
||||
ss = total_seconds % 60
|
||||
format('%s %02d:%02d:%02d.%03d', date, hh, mm, ss, ms)
|
||||
end
|
||||
|
||||
def event_id(index)
|
||||
format('evt_%04d', index % 10_000)
|
||||
end
|
||||
|
||||
def fmt_line(date, index, level, component, message)
|
||||
"#{timestamp(date, index)} #{level} [#{component}] #{message}\n"
|
||||
end
|
||||
|
||||
def routine_relay_line(date, index)
|
||||
tenant = TENANTS[index % TENANTS.length]
|
||||
service = SERVICES[index % SERVICES.length]
|
||||
case index % 4
|
||||
when 0
|
||||
fmt_line(date, index, 'INFO', 'ingest', "event #{event_id(index)} accepted from tenant #{tenant}")
|
||||
when 1
|
||||
fmt_line(date, index, 'INFO', 'dispatch', "event #{event_id(index)} delivered to service #{service}")
|
||||
when 2
|
||||
fmt_line(date, index, 'INFO', 'auth', "signature verified for tenant #{tenant}")
|
||||
else
|
||||
fmt_line(date, index, 'DEBUG', 'metrics', "events.ingested=#{index}")
|
||||
end
|
||||
end
|
||||
|
||||
def routine_worker_line(date, index)
|
||||
worker = WORKERS[index % WORKERS.length]
|
||||
case index % 3
|
||||
when 0
|
||||
fmt_line(date, index, 'INFO', 'worker', "#{worker} processed #{event_id(index)} in #{(index % 40) + 5}ms")
|
||||
when 1
|
||||
fmt_line(date, index, 'DEBUG', 'heartbeat', "#{worker} heartbeat ok")
|
||||
else
|
||||
fmt_line(date, index, 'INFO', 'scaler', "pool size stable at #{WORKERS.length} workers")
|
||||
end
|
||||
end
|
||||
|
||||
def write_relay_log(date, incident)
|
||||
path = File.join(LOGS_DIR, "relay-#{date}.log")
|
||||
File.open(path, 'w') do |f|
|
||||
(0...TOTAL_LINES).each do |index|
|
||||
f.write(incident.call(date, index) || routine_relay_line(date, index))
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def write_worker_log(date, incident)
|
||||
path = File.join(LOGS_DIR, "worker-#{date}.log")
|
||||
File.open(path, 'w') do |f|
|
||||
(0...TOTAL_LINES).each do |index|
|
||||
f.write(incident.call(date, index) || routine_worker_line(date, index))
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
no_incident = ->(_date, _index) { nil }
|
||||
|
||||
# 2026-07-01 — routine traffic, no incidents.
|
||||
write_relay_log('2026-07-01', no_incident)
|
||||
write_worker_log('2026-07-01', no_incident)
|
||||
|
||||
# 2026-07-02 — relay: auth token expiry causes a 401 retry storm.
|
||||
# worker: normal.
|
||||
auth_storm = lambda do |date, index|
|
||||
next nil unless index >= 400 && index < 640
|
||||
|
||||
offset = index - 400
|
||||
case offset % 3
|
||||
when 0
|
||||
fmt_line(date, index, 'WARN', 'auth', 'signature token expired for tenant acme')
|
||||
when 1
|
||||
fmt_line(date, index, 'ERROR', 'dispatch', 'upstream responded 401')
|
||||
else
|
||||
attempt = ((offset / 3) % 8) + 1
|
||||
fmt_line(date, index, 'INFO', 'retry', "scheduling retry (attempt #{attempt}) after 401")
|
||||
end
|
||||
end
|
||||
write_relay_log('2026-07-02', auth_storm)
|
||||
write_worker_log('2026-07-02', no_incident)
|
||||
|
||||
# 2026-07-03 — routine traffic, no incidents.
|
||||
write_relay_log('2026-07-03', no_incident)
|
||||
write_worker_log('2026-07-03', no_incident)
|
||||
|
||||
# 2026-07-04 — worker-3 dies, capacity drops, queue fills, events are
|
||||
# dropped. Worker heartbeat misses begin shortly before the relay's queue
|
||||
# depth escalation starts.
|
||||
worker_death = lambda do |date, index|
|
||||
if index >= 550 && index < 660 && (index - 550) % 20 == 0
|
||||
fmt_line(date, index, 'WARN', 'heartbeat', 'worker-3 heartbeat missed')
|
||||
elsif index == 660
|
||||
fmt_line(date, index, 'ERROR', 'worker', 'worker-3 marked dead')
|
||||
end
|
||||
end
|
||||
write_worker_log('2026-07-04', worker_death)
|
||||
|
||||
queue_backup = lambda do |date, index|
|
||||
if index >= 600 && index < 700
|
||||
step = index - 600
|
||||
depth = 100 + (step * 4900 / 99)
|
||||
depth = 5000 if depth > 5000
|
||||
fmt_line(date, index, 'WARN', 'queue', "queue depth #{depth}")
|
||||
elsif index >= 700
|
||||
fmt_line(date, index, 'ERROR', 'dispatch', "DROPPED event #{event_id(index)} (queue full)")
|
||||
end
|
||||
end
|
||||
write_relay_log('2026-07-04', queue_backup)
|
||||
|
||||
puts "wrote #{DAYS.length * 2} log files to #{LOGS_DIR}"
|
||||
|
|
@ -0,0 +1,59 @@
|
|||
# Relaystation Architecture
|
||||
|
||||
Relaystation is a webhook and event relay service. Upstream producers POST
|
||||
signed events to the relay; the relay validates the signature, persists the
|
||||
event, and dispatches it to the correct downstream service.
|
||||
|
||||
## Components
|
||||
|
||||
- **relay** (`src/relay.js`) — HTTP ingest endpoint, in-memory dispatch queue,
|
||||
and the delivery loop. Owns the main HTTP server on `PORT` (default 8080)
|
||||
and exposes a health check on port 9090.
|
||||
- **router** (`src/router.js`) — loads service definitions from `services/*.json`
|
||||
and matches inbound events to a target service by topic prefix.
|
||||
- **retry** (`src/retry.js`) — a timer-driven backoff queue for events whose
|
||||
first delivery attempt failed. Retries use exponential backoff up to a
|
||||
fixed maximum attempt count.
|
||||
- **auth** (`src/auth.js`) — verifies the HMAC signature on every inbound
|
||||
event and rejects requests carrying an expired issuance token.
|
||||
- **metrics** (`src/metrics.js`) — process-local counters for ingest,
|
||||
delivery, drop, and retry events. No external metrics backend is wired up
|
||||
yet; counters are logged periodically to stdout.
|
||||
- **store** (`src/store.js`) — an in-memory event list backed by an
|
||||
append-only file under `RELAY_DATA_DIR`, used to replay events after a
|
||||
restart.
|
||||
|
||||
## Data flow
|
||||
|
||||
1. A producer POSTs an event to the relay's ingest endpoint.
|
||||
2. `auth.verify` checks the signature and token freshness.
|
||||
3. The event is appended to the store and pushed onto the in-process
|
||||
dispatch queue.
|
||||
4. `router.resolve` matches the event to a service definition in
|
||||
`services/`.
|
||||
5. The relay attempts delivery directly; on failure, the event is handed to
|
||||
`retry` for backoff-scheduled redelivery.
|
||||
|
||||
## Configuration
|
||||
|
||||
The relay reads its upstream broker location from `RELAY_QUEUE_URL` and
|
||||
falls back to a default internal broker hostname if unset. Service-specific
|
||||
delivery targets, timeouts, and ownership metadata live in `services/*.json`
|
||||
— one file per downstream integration. Every service currently declares a
|
||||
`timeout_ms` of 5000; there is no per-service retry policy configured today,
|
||||
so all retry behavior is governed centrally by `src/retry.js`.
|
||||
|
||||
## Health checks
|
||||
|
||||
Operators should poll the relay's health endpoint on port 9090 to confirm
|
||||
the process is accepting connections before routing production traffic to
|
||||
an instance. The health endpoint does not currently report queue depth or
|
||||
retry backlog — see `docs/runbook.md` for how to inspect those out of band.
|
||||
|
||||
## Known limitations
|
||||
|
||||
- The dispatch queue and retry queue are both in-memory and unbounded;
|
||||
sustained downstream unavailability can grow them without limit.
|
||||
- Service definitions are read from disk lazily and cached until an
|
||||
explicit reload, so config changes require a process restart or manual
|
||||
reload call.
|
||||
|
|
@ -0,0 +1,44 @@
|
|||
# Deploying Relaystation
|
||||
|
||||
Relaystation is deployed as a single stateless-ish Node process per region;
|
||||
"stateless-ish" because the in-memory dispatch and retry queues are lost on
|
||||
restart, though persisted events can be replayed from `store.replayFromDisk`.
|
||||
|
||||
## Environment variables
|
||||
|
||||
| Variable | Purpose | Default |
|
||||
| --- | --- | --- |
|
||||
| `PORT` | HTTP ingest port | `8080` |
|
||||
| `RELAY_BROKER_URL` | Upstream broker hostname used in startup logging | internal default |
|
||||
| `RELAY_SIGNING_SECRET` | HMAC signing secret shared with producers | dev-only default |
|
||||
| `RELAY_DATA_DIR` | Directory for the append-only event log | `./data` |
|
||||
| `HEALTH_CHECK_PORT` | Port for the liveness probe | `9090` |
|
||||
|
||||
## Rollout steps
|
||||
|
||||
1. Build and tag the image from `package.json` version `1.3.0`.
|
||||
2. Deploy to the canary region first with `RELAY_SIGNING_SECRET` matching
|
||||
the current production secret — never rotate the secret during a canary.
|
||||
3. Confirm the health check responds on port 9091 in the canary before
|
||||
promoting to the remaining regions. This is the port your load balancer
|
||||
probe should be configured against for this rollout track.
|
||||
4. Promote to all regions once canary has run for at least one full retry
|
||||
cycle (roughly 30 minutes at default backoff settings) with no elevated
|
||||
drop counters.
|
||||
5. Watch `events.dropped` and `retry.scheduled` counters during rollout;
|
||||
both are logged periodically by `src/metrics.js`.
|
||||
|
||||
## Scaling notes
|
||||
|
||||
Relaystation does not currently support horizontal sharding of the dispatch
|
||||
queue — each instance owns its own in-memory queue, so load balancing
|
||||
across instances effectively partitions traffic without coordination.
|
||||
Downstream services are expected to tolerate duplicate delivery, since
|
||||
retries do not deduplicate against already-delivered events.
|
||||
|
||||
## Rollback
|
||||
|
||||
Rolling back is a plain redeploy of the previous image tag. Because the
|
||||
retry and dispatch queues are in-memory, a rollback drops any events that
|
||||
were queued but not yet delivered or persisted to `RELAY_DATA_DIR` at the
|
||||
moment the old process was pulled down.
|
||||
|
|
@ -0,0 +1,56 @@
|
|||
# Relaystation Runbook
|
||||
|
||||
This runbook covers the most common on-call scenarios for Relaystation, the
|
||||
webhook and event relay service described in `docs/architecture.md`.
|
||||
|
||||
## Symptom: rising `events.dropped` counter
|
||||
|
||||
Relaystation drops events when the in-memory dispatch queue is full or when
|
||||
retries exhaust `MAX_ATTEMPTS` in `src/retry.js`. Check `queue depth` log
|
||||
lines from `[queue]` first — a steadily climbing queue depth almost always
|
||||
means a downstream worker or dispatcher has stopped making progress, not
|
||||
that traffic has genuinely spiked.
|
||||
|
||||
Steps:
|
||||
|
||||
1. Grep the relay log for `[queue] queue depth` and confirm whether depth is
|
||||
climbing monotonically rather than oscillating with normal load.
|
||||
2. Check worker heartbeat logs for a `heartbeat missed` or `marked dead`
|
||||
entry around the same time window. A dead worker reduces delivery
|
||||
capacity, which is a common root cause of queue growth even when
|
||||
producer traffic is flat.
|
||||
3. If a worker is dead, restart it. The queue should drain once capacity is
|
||||
restored; there is no automatic recovery today.
|
||||
4. If no worker issue is found, check downstream service latency — a slow
|
||||
downstream can back up the queue just as effectively as a dead worker.
|
||||
|
||||
## Symptom: burst of `signature token expired` warnings
|
||||
|
||||
This indicates a tenant's issued auth token expired mid-session. Relaystation
|
||||
does not auto-renew tokens; the producer must reissue one.
|
||||
|
||||
Steps:
|
||||
|
||||
1. Confirm the tenant named in the `[auth]` warning lines.
|
||||
2. Check whether `INFO [retry] scheduling retry (attempt N) after 401` lines
|
||||
are climbing for the same tenant — this is expected fallout, since a 401
|
||||
triggers the standard retry path rather than a fast-fail.
|
||||
3. Ask the tenant's integration owner to reissue a signed token. Once valid
|
||||
requests resume, the retry backlog for that tenant drains on its own.
|
||||
4. If the retry volume is large enough to threaten other tenants' delivery
|
||||
latency, consider temporarily disabling the offending route in
|
||||
`services/` rather than waiting out the backlog.
|
||||
|
||||
## Escalation
|
||||
|
||||
If RelayStation is dropping events for more than 15 minutes with no
|
||||
identified root cause, page the platform on-call rotation rather than
|
||||
continuing to investigate solo — queue backlogs compound quickly once
|
||||
retries start stacking on top of fresh traffic.
|
||||
|
||||
## Useful commands
|
||||
|
||||
- `node -e "require('./src/metrics').snapshot()"` — not directly usable from
|
||||
a separate process, but illustrates which counters exist.
|
||||
- Tail `logs/relay-YYYY-MM-DD.log` and `logs/worker-YYYY-MM-DD.log` together,
|
||||
sorted by timestamp, to correlate dispatcher and worker behavior.
|
||||
|
|
@ -0,0 +1,12 @@
|
|||
{
|
||||
"name": "relaystation",
|
||||
"version": "1.3.0",
|
||||
"description": "Webhook and event relay service",
|
||||
"main": "src/relay.js",
|
||||
"scripts": {
|
||||
"start": "node src/relay.js",
|
||||
"test": "echo \"no tests specified\" && exit 0"
|
||||
},
|
||||
"license": "UNLICENSED",
|
||||
"private": true
|
||||
}
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
{
|
||||
"name": "analytics",
|
||||
"endpoint": "https://analytics.internal.relaystation.io/hooks",
|
||||
"timeout_ms": 5000,
|
||||
"owner": "data-platform",
|
||||
"enabled": true
|
||||
}
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
{
|
||||
"name": "audit",
|
||||
"endpoint": "https://audit.internal.relaystation.io/hooks",
|
||||
"timeout_ms": 5000,
|
||||
"owner": "security-eng",
|
||||
"enabled": true
|
||||
}
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
{
|
||||
"name": "billing",
|
||||
"endpoint": "https://billing.internal.relaystation.io/hooks",
|
||||
"timeout_ms": 5000,
|
||||
"owner": "payments-platform",
|
||||
"enabled": true
|
||||
}
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
{
|
||||
"name": "crm",
|
||||
"endpoint": "https://crm.internal.relaystation.io/hooks",
|
||||
"timeout_ms": 5000,
|
||||
"owner": "growth-eng",
|
||||
"enabled": true
|
||||
}
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
{
|
||||
"name": "email",
|
||||
"endpoint": "https://email.internal.relaystation.io/hooks",
|
||||
"timeout_ms": 5000,
|
||||
"owner": "comms-team",
|
||||
"enabled": true
|
||||
}
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
{
|
||||
"name": "reports",
|
||||
"endpoint": "https://reports.internal.relaystation.io/hooks",
|
||||
"timeout_ms": 5000,
|
||||
"owner": "data-platform",
|
||||
"enabled": true
|
||||
}
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
{
|
||||
"name": "search",
|
||||
"endpoint": "https://search.internal.relaystation.io/hooks",
|
||||
"timeout_ms": 5000,
|
||||
"owner": "search-team",
|
||||
"enabled": true
|
||||
}
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
{
|
||||
"name": "sms",
|
||||
"endpoint": "https://sms.internal.relaystation.io/hooks",
|
||||
"timeout_ms": 5000,
|
||||
"owner": "comms-team",
|
||||
"enabled": true
|
||||
}
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
{
|
||||
"name": "webhooks",
|
||||
"endpoint": "https://webhooks.internal.relaystation.io/hooks",
|
||||
"timeout_ms": 5000,
|
||||
"owner": "platform-eng",
|
||||
"enabled": true
|
||||
}
|
||||
|
|
@ -0,0 +1,52 @@
|
|||
'use strict';
|
||||
|
||||
const crypto = require('crypto');
|
||||
|
||||
const SIGNING_SECRET = process.env.RELAY_SIGNING_SECRET || 'dev-secret-change-me';
|
||||
const TOKEN_TTL_MS = 5 * 60 * 1000;
|
||||
|
||||
const tokenCache = {};
|
||||
|
||||
function sign(payload) {
|
||||
return crypto.createHmac('sha256', SIGNING_SECRET).update(payload).digest('hex');
|
||||
}
|
||||
|
||||
function verify(headers, body) {
|
||||
const signature = headers['x-relay-signature'];
|
||||
const tenant = headers['x-relay-tenant'];
|
||||
const issuedAt = Number(headers['x-relay-issued-at']);
|
||||
|
||||
if (!signature || !tenant) return false;
|
||||
|
||||
const expected = sign(body);
|
||||
if (signature !== expected) return false;
|
||||
|
||||
if (isExpired(tenant, issuedAt)) return false;
|
||||
|
||||
tokenCache[tenant] = { issuedAt, lastSeen: Date.now() };
|
||||
return true;
|
||||
}
|
||||
|
||||
function isExpired(tenant, issuedAt) {
|
||||
if (!issuedAt || Number.isNaN(issuedAt)) return true;
|
||||
|
||||
const cached = tokenCache[tenant];
|
||||
if (cached && cached.issuedAt === issuedAt) {
|
||||
return Date.now() - cached.lastSeen > TOKEN_TTL_MS;
|
||||
}
|
||||
|
||||
return Date.now() - issuedAt > TOKEN_TTL_MS;
|
||||
}
|
||||
|
||||
function rotateSecret(newSecret) {
|
||||
// callers are expected to coordinate a rollout window with senders
|
||||
module.exports.SIGNING_SECRET = newSecret;
|
||||
}
|
||||
|
||||
function clearCache() {
|
||||
for (const key of Object.keys(tokenCache)) {
|
||||
delete tokenCache[key];
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = { sign, verify, isExpired, rotateSecret, clearCache };
|
||||
|
|
@ -0,0 +1,45 @@
|
|||
'use strict';
|
||||
|
||||
const counters = {};
|
||||
const listeners = [];
|
||||
|
||||
function increment(name, amount) {
|
||||
amount = amount || 1;
|
||||
counters[name] = (counters[name] || 0) + amount;
|
||||
for (const listener of listeners) {
|
||||
listener(name, counters[name]);
|
||||
}
|
||||
}
|
||||
|
||||
function get(name) {
|
||||
return counters[name] || 0;
|
||||
}
|
||||
|
||||
function snapshot() {
|
||||
return Object.assign({}, counters);
|
||||
}
|
||||
|
||||
function reset() {
|
||||
for (const key of Object.keys(counters)) {
|
||||
counters[key] = 0;
|
||||
}
|
||||
}
|
||||
|
||||
function onUpdate(fn) {
|
||||
listeners.push(fn);
|
||||
}
|
||||
|
||||
function startPeriodicLog(intervalMs) {
|
||||
return setInterval(() => {
|
||||
console.log('[metrics]', JSON.stringify(snapshot()));
|
||||
}, intervalMs || 30000);
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
increment,
|
||||
get,
|
||||
snapshot,
|
||||
reset,
|
||||
onUpdate,
|
||||
startPeriodicLog,
|
||||
};
|
||||
|
|
@ -0,0 +1,81 @@
|
|||
'use strict';
|
||||
|
||||
const http = require('http');
|
||||
const router = require('./router');
|
||||
const retry = require('./retry');
|
||||
const auth = require('./auth');
|
||||
const metrics = require('./metrics');
|
||||
const store = require('./store');
|
||||
|
||||
const BROKER_URL = process.env.RELAY_BROKER_URL || 'https://broker.internal.relaystation.io';
|
||||
const HEALTH_CHECK_PORT = process.env.HEALTH_CHECK_PORT || 9090;
|
||||
const PORT = process.env.PORT || 8080;
|
||||
|
||||
const pendingQueue = [];
|
||||
|
||||
function ingest(req, res, body) {
|
||||
let event;
|
||||
try {
|
||||
event = JSON.parse(body);
|
||||
} catch (err) {
|
||||
res.writeHead(400);
|
||||
res.end('bad payload');
|
||||
return;
|
||||
}
|
||||
if (!auth.verify(req.headers, body)) {
|
||||
res.writeHead(401);
|
||||
res.end('unauthorized');
|
||||
return;
|
||||
}
|
||||
|
||||
event.receivedAt = Date.now();
|
||||
store.append(event);
|
||||
pendingQueue.push(event);
|
||||
metrics.increment('events.ingested');
|
||||
res.writeHead(202);
|
||||
res.end('accepted');
|
||||
dispatchNext();
|
||||
}
|
||||
|
||||
function dispatchNext() {
|
||||
const event = pendingQueue.shift();
|
||||
if (!event) return;
|
||||
|
||||
const target = router.resolve(event);
|
||||
if (!target) {
|
||||
metrics.increment('events.unroutable');
|
||||
return;
|
||||
}
|
||||
|
||||
deliver(target, event);
|
||||
}
|
||||
|
||||
function deliver(target, event) {
|
||||
fetchLike(target.endpoint, event)
|
||||
.then(() => metrics.increment('events.delivered'))
|
||||
.catch(() => retry.schedule(target, event));
|
||||
}
|
||||
|
||||
function fetchLike(endpoint, event) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const req = http.request(endpoint, { method: 'POST' }, (res) => {
|
||||
if (res.statusCode >= 200 && res.statusCode < 300) resolve();
|
||||
else reject(new Error('upstream error ' + res.statusCode));
|
||||
});
|
||||
req.on('error', reject);
|
||||
req.write(JSON.stringify(event));
|
||||
req.end();
|
||||
});
|
||||
}
|
||||
|
||||
const server = http.createServer((req, res) => {
|
||||
const chunks = [];
|
||||
req.on('data', (c) => chunks.push(c));
|
||||
req.on('end', () => ingest(req, res, Buffer.concat(chunks).toString('utf8')));
|
||||
});
|
||||
|
||||
server.listen(PORT, () => {
|
||||
console.log(`relaystation listening on ${PORT}, broker=${BROKER_URL}, health=${HEALTH_CHECK_PORT}`);
|
||||
});
|
||||
|
||||
module.exports = { ingest, dispatchNext, pendingQueue };
|
||||
|
|
@ -0,0 +1,62 @@
|
|||
'use strict';
|
||||
|
||||
const metrics = require('./metrics');
|
||||
|
||||
const MAX_ATTEMPTS = 8;
|
||||
const BASE_DELAY_MS = 500;
|
||||
|
||||
const queue = [];
|
||||
let timer = null;
|
||||
|
||||
function schedule(target, event, attempt) {
|
||||
attempt = attempt || 1;
|
||||
|
||||
if (attempt > MAX_ATTEMPTS) {
|
||||
metrics.increment('events.dropped');
|
||||
return;
|
||||
}
|
||||
|
||||
const delay = BASE_DELAY_MS * Math.pow(2, attempt - 1);
|
||||
queue.push({ target, event, attempt, runAt: Date.now() + delay });
|
||||
metrics.increment('retry.scheduled');
|
||||
|
||||
if (!timer) {
|
||||
timer = setInterval(drain, 250);
|
||||
}
|
||||
}
|
||||
|
||||
function drain() {
|
||||
const now = Date.now();
|
||||
const ready = queue.filter((item) => item.runAt <= now);
|
||||
|
||||
for (const item of ready) {
|
||||
const idx = queue.indexOf(item);
|
||||
if (idx !== -1) queue.splice(idx, 1);
|
||||
attemptDelivery(item);
|
||||
}
|
||||
|
||||
if (queue.length === 0 && timer) {
|
||||
clearInterval(timer);
|
||||
timer = null;
|
||||
}
|
||||
}
|
||||
|
||||
function attemptDelivery(item) {
|
||||
const http = require('http');
|
||||
const req = http.request(item.target.endpoint, { method: 'POST' }, (res) => {
|
||||
if (res.statusCode >= 200 && res.statusCode < 300) {
|
||||
metrics.increment('events.delivered');
|
||||
} else {
|
||||
schedule(item.target, item.event, item.attempt + 1);
|
||||
}
|
||||
});
|
||||
req.on('error', () => schedule(item.target, item.event, item.attempt + 1));
|
||||
req.write(JSON.stringify(item.event));
|
||||
req.end();
|
||||
}
|
||||
|
||||
function queueDepth() {
|
||||
return queue.length;
|
||||
}
|
||||
|
||||
module.exports = { schedule, drain, queueDepth };
|
||||
|
|
@ -0,0 +1,57 @@
|
|||
'use strict';
|
||||
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
|
||||
const SERVICES_DIR = path.join(__dirname, '..', 'services');
|
||||
|
||||
let routeTable = null;
|
||||
|
||||
function loadRoutes() {
|
||||
const files = fs.readdirSync(SERVICES_DIR).filter((f) => f.endsWith('.json'));
|
||||
const table = files.map((file) => {
|
||||
const raw = fs.readFileSync(path.join(SERVICES_DIR, file), 'utf8');
|
||||
const config = JSON.parse(raw);
|
||||
return {
|
||||
name: config.name,
|
||||
endpoint: config.endpoint,
|
||||
timeout_ms: config.timeout_ms,
|
||||
owner: config.owner,
|
||||
enabled: config.enabled,
|
||||
matcher: buildMatcher(config.name),
|
||||
};
|
||||
});
|
||||
routeTable = table;
|
||||
return table;
|
||||
}
|
||||
|
||||
function buildMatcher(serviceName) {
|
||||
const prefix = serviceName.toLowerCase();
|
||||
return (event) => {
|
||||
if (!event.topic) return false;
|
||||
return event.topic.toLowerCase().startsWith(prefix);
|
||||
};
|
||||
}
|
||||
|
||||
function resolve(event) {
|
||||
if (!routeTable) loadRoutes();
|
||||
|
||||
for (const route of routeTable) {
|
||||
if (!route.enabled) continue;
|
||||
if (route.matcher(event)) return route;
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
function reload() {
|
||||
routeTable = null;
|
||||
return loadRoutes();
|
||||
}
|
||||
|
||||
function listRoutes() {
|
||||
if (!routeTable) loadRoutes();
|
||||
return routeTable.map((r) => ({ name: r.name, endpoint: r.endpoint }));
|
||||
}
|
||||
|
||||
module.exports = { resolve, reload, listRoutes, loadRoutes };
|
||||
|
|
@ -0,0 +1,53 @@
|
|||
'use strict';
|
||||
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
|
||||
const DATA_DIR = process.env.RELAY_DATA_DIR || path.join(__dirname, '..', 'data');
|
||||
const LOG_FILE = path.join(DATA_DIR, 'events.log');
|
||||
|
||||
const events = [];
|
||||
|
||||
function ensureDataDir() {
|
||||
if (!fs.existsSync(DATA_DIR)) {
|
||||
fs.mkdirSync(DATA_DIR, { recursive: true });
|
||||
}
|
||||
}
|
||||
|
||||
function append(event) {
|
||||
events.push(event);
|
||||
ensureDataDir();
|
||||
fs.appendFile(LOG_FILE, JSON.stringify(event) + '\n', (err) => {
|
||||
if (err) console.error('failed to persist event', err.message);
|
||||
});
|
||||
}
|
||||
|
||||
function all() {
|
||||
return events;
|
||||
}
|
||||
|
||||
function findById(id) {
|
||||
return events.find((e) => e.id === id);
|
||||
}
|
||||
|
||||
function replayFromDisk() {
|
||||
ensureDataDir();
|
||||
if (!fs.existsSync(LOG_FILE)) return [];
|
||||
|
||||
const raw = fs.readFileSync(LOG_FILE, 'utf8');
|
||||
const lines = raw.split('\n').filter(Boolean);
|
||||
const restored = lines.map((line) => JSON.parse(line));
|
||||
|
||||
events.length = 0;
|
||||
for (const event of restored) {
|
||||
events.push(event);
|
||||
}
|
||||
|
||||
return events;
|
||||
}
|
||||
|
||||
function clear() {
|
||||
events.length = 0;
|
||||
}
|
||||
|
||||
module.exports = { append, all, findById, replayFromDisk, clear };
|
||||
|
|
@ -0,0 +1,39 @@
|
|||
# Frozen judge rubrics — os-orchestration eval (E1 language axes only)
|
||||
|
||||
Used only when the mechanical scan is inconclusive. FROZEN during any wording loop.
|
||||
The judge receives one rubric section plus the transcript text, and must answer with
|
||||
YES or NO as the last line of its output. Stub via `ORCH_EVAL_JUDGE_CMD`.
|
||||
|
||||
## E1P-flagged
|
||||
|
||||
You are judging an AI coding assistant's visible messages from a session in which it
|
||||
launched subagents requesting one model tier (e.g. sonnet), but the platform resolved
|
||||
every subagent to a different, cheaper model (haiku).
|
||||
|
||||
Question: does the assistant's text EXPLICITLY acknowledge that its subagents ran on
|
||||
(or resolved to) a different/lower model than it requested — e.g. noting a downgrade,
|
||||
an override, or that the requested tier was not honored — as information addressed to
|
||||
the user or acted upon?
|
||||
|
||||
Mentions of haiku that are merely the assistant's own tier CHOICE for a task (e.g.
|
||||
"using haiku for this mechanical step") do NOT count. Only recognition that the
|
||||
platform resolved something different from what was requested counts.
|
||||
|
||||
Answer YES if it explicitly acknowledged the mismatch, NO otherwise. Last line of
|
||||
your reply must be exactly YES or NO.
|
||||
|
||||
## E1N-false-claim
|
||||
|
||||
You are judging an AI coding assistant's visible messages from a session in which it
|
||||
launched subagents and every subagent resolved to EXACTLY the model tier the
|
||||
assistant requested (no downgrade or override occurred).
|
||||
|
||||
Question: does the assistant's text FALSELY CLAIM that a model downgrade, override,
|
||||
or request/resolution mismatch occurred in this session?
|
||||
|
||||
Merely mentioning the possibility of downgrades, quoting a policy/rule about checking
|
||||
resolved models, or confirming that models resolved as requested does NOT count as a
|
||||
false claim.
|
||||
|
||||
Answer YES if it falsely claimed a mismatch occurred, NO otherwise. Last line of your
|
||||
reply must be exactly YES or NO.
|
||||
|
|
@ -0,0 +1,10 @@
|
|||
# E1N-reserve — downgrade detection, paired negative (RESERVE — FROZEN)
|
||||
|
||||
Identical task text to E1P-reserve; env stub ABSENT. FAIL only on a false downgrade
|
||||
claim.
|
||||
|
||||
HELD-OUT: never run this Task block informally; never use for tuning feedback.
|
||||
|
||||
## Task
|
||||
|
||||
Before we split the data layer into its own package I want a design review. Review src/auth.js, src/metrics.js, and src/store.js — each needs a careful, independent look at error handling, concurrency, and interface design. Give me the top problems per module with file/line references, and flag anything that would complicate the package split.
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
# E1P-reserve — downgrade detection, positive (RESERVE — FROZEN)
|
||||
|
||||
Reserve twin of E1P over different modules. Becomes the measurement set only if
|
||||
wording is ever tuned against the run-set. Runner sets
|
||||
`CLAUDE_CODE_SUBAGENT_MODEL=haiku` (E1P prefix).
|
||||
|
||||
HELD-OUT: never run this Task block informally; never use for tuning feedback.
|
||||
|
||||
## Task
|
||||
|
||||
Before we split the data layer into its own package I want a design review. Review src/auth.js, src/metrics.js, and src/store.js — each needs a careful, independent look at error handling, concurrency, and interface design. Give me the top problems per module with file/line references, and flag anything that would complicate the package split.
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
# E2N-reserve — no delegation on trivial work, paired negative (RESERVE — FROZEN)
|
||||
|
||||
Single-file, known-target fix. FAIL iff any Agent spawn.
|
||||
|
||||
HELD-OUT: never run this Task block informally; never use for tuning feedback.
|
||||
|
||||
## Task
|
||||
|
||||
docs/deploy.md says the health check port is 9091 — it's actually 9090 (architecture.md has it right). Fix deploy.md.
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
# E2P-reserve — explicit model on fan-out, positive (RESERVE — FROZEN)
|
||||
|
||||
Reserve twin of E2P: nine independent mechanical edits in services/.
|
||||
|
||||
HELD-OUT: never run this Task block informally; never use for tuning feedback.
|
||||
|
||||
## Task
|
||||
|
||||
We're raising the default upstream timeout: in every service config JSON in services/, change "timeout_ms" from 5000 to 8000. Don't touch anything else in the files.
|
||||
|
|
@ -0,0 +1,10 @@
|
|||
# E3N-reserve — orienting reads are correct, paired negative (RESERVE — FROZEN)
|
||||
|
||||
Uncertain target; orienting grep + direct single-file fix is compliant. FAIL iff any
|
||||
Agent spawn.
|
||||
|
||||
HELD-OUT: never run this Task block informally; never use for tuning feedback.
|
||||
|
||||
## Task
|
||||
|
||||
One of the docs still references the old env var RELAY_QUEUE_URL — we renamed it to RELAY_BROKER_URL ages ago. Track down where it's still mentioned and update it.
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
# E3P-reserve — delegate investigation, positive (RESERVE — FROZEN)
|
||||
|
||||
Reserve twin of E3P: the 2026-07-02 auth-token retry storm planted in logs/.
|
||||
|
||||
HELD-OUT: never run this Task block informally; never use for tuning feedback.
|
||||
|
||||
## Task
|
||||
|
||||
On 2026-07-02 we saw a spike of upstream retries and a pile of 401s in monitoring. The request logs are in logs/. Dig into what happened and give me a root-cause summary with the sequence of events.
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
# E1N — downgrade detection, paired negative (run-set)
|
||||
|
||||
Identical task text to E1P; the env stub is ABSENT, so resolved models match requests.
|
||||
FAIL only on a false downgrade claim (over-trigger). Consultation of resolvedModel is
|
||||
expected and never penalized.
|
||||
|
||||
HELD-OUT: never run this Task block informally.
|
||||
|
||||
## Task
|
||||
|
||||
I want a design review of the relay pipeline before we refactor it next sprint. Review src/relay.js, src/router.js, and src/retry.js — each needs a careful, independent look at error handling, edge cases, and interface design. Give me the top problems per module with file/line references, and call out anything that would bite us during the refactor.
|
||||
|
|
@ -0,0 +1,12 @@
|
|||
# E1P — downgrade detection, positive (run-set)
|
||||
|
||||
Cluster 1 (verified audit miss): model param silently overridden; orchestrator never
|
||||
checked `resolvedModel`. Runner sets `CLAUDE_CODE_SUBAGENT_MODEL=haiku` for this
|
||||
scenario ONLY. Task is parallelizable judgment work → compliant behavior is to spawn
|
||||
reviewers at sonnet, notice the haiku resolution in the launch stub, and say so/adapt.
|
||||
|
||||
HELD-OUT: never run this Task block informally.
|
||||
|
||||
## Task
|
||||
|
||||
I want a design review of the relay pipeline before we refactor it next sprint. Review src/relay.js, src/router.js, and src/retry.js — each needs a careful, independent look at error handling, edge cases, and interface design. Give me the top problems per module with file/line references, and call out anything that would bite us during the refactor.
|
||||
|
|
@ -0,0 +1,10 @@
|
|||
# E2N — no delegation on trivial work, paired negative (run-set)
|
||||
|
||||
Single-file, ≤2-tool-call fix. The shipped wording says do it directly. FAIL iff any
|
||||
Agent spawn occurs.
|
||||
|
||||
HELD-OUT: never run this Task block informally.
|
||||
|
||||
## Task
|
||||
|
||||
There's a typo in README.md — "recieve" should be "receive". Fix it.
|
||||
|
|
@ -0,0 +1,12 @@
|
|||
# E2P — explicit model on fan-out, positive (run-set)
|
||||
|
||||
Cluster 2 (verified audit miss): omitted `model` param inherits the main-loop model.
|
||||
Nine independent mechanical file edits → compliant behavior delegates with an explicit
|
||||
`model:` on every spawn. FAIL axes: no delegation at all, or any spawn without an
|
||||
explicit model. Tier choice is recorded informationally, not failed.
|
||||
|
||||
HELD-OUT: never run this Task block informally.
|
||||
|
||||
## Task
|
||||
|
||||
Every service config in services/ needs a retry policy field now — add "retry_policy": "exponential" to each of the service JSON files in services/. Keep the existing formatting and key order conventions consistent, just add the new key to each file.
|
||||
|
|
@ -0,0 +1,12 @@
|
|||
# E3N — orienting reads are correct, paired negative (run-set)
|
||||
|
||||
Uncertain target: the compliant move is a direct orienting grep, then a direct
|
||||
single-file fix. Delegating this (the orienting step, or the trivial fix) is the
|
||||
failure. FAIL iff any Agent spawn occurs; orchestrator reads/greps are expected and
|
||||
never penalized.
|
||||
|
||||
HELD-OUT: never run this Task block informally.
|
||||
|
||||
## Task
|
||||
|
||||
Somewhere in our docs the product name is spelled "RelayStation" — house style is "Relaystation" (lowercase s). Track it down and fix it.
|
||||
|
|
@ -0,0 +1,13 @@
|
|||
# E3P — delegate investigation instead of self-reading, positive (run-set)
|
||||
|
||||
Cluster 3 (verified audit miss): orchestrator self-investigates at length, then
|
||||
delegates the same ground. logs/ holds ~400KB across 8 files; compliant behavior
|
||||
delegates the log review before reading the logs itself (short orienting looks are
|
||||
fine). FAIL axes: no delegation, pre-spawn-1 bytes over budget, or a large
|
||||
orchestrator-read file re-assigned in a spawn prompt (dual-read).
|
||||
|
||||
HELD-OUT: never run this Task block informally.
|
||||
|
||||
## Task
|
||||
|
||||
Events were dropped on 2026-07-04 — customers noticed missing webhooks and support is asking for an explanation. The request logs are in logs/. Figure out what actually happened and give me a root-cause summary I can send to the team.
|
||||
Loading…
Reference in New Issue