Dispatch settlement unreliable: SubagentStop hook never fires

jared commented

2026-08-24 01:32:22 +00:00

Owner

Recorded 2026-08-23 during the #491 worktree session: async dispatches stalled at await_completion with completed_at nil because Claude Code's SubagentStop hook never fired (upstream anthropics/claude-code#27755 — reconnected subagents never fully stop). Needed: agent-initiated settlement keyed by dispatch id, idempotent settle (late hook fire tolerated), and a runner-side settlement fallback for agents that crash before settling. Note: this issue was created retroactively; it must carry index #495 to match references in ADR-0144, commit bbc5162, and docs/implementation-status/os-sdlc.md.

jared commented

2026-08-24 01:34:20 +00:00

Author

Owner

Shipped in the amended settlement commit (ADR-0144, was referenced as #495 before this index turned out to be a ghost): agents now settle their own dispatches. subagent-stop gained a dispatch-id-keyed path (session_id/agent_type resolved via relations), the settlement reply carries NextStep's instruction + dispatch_id for verbatim relay to the orchestrator, transcript path is reconstructed via AgentTranscriptPath, and a late SubagentStop hook fire on an already-settled dispatch is a quiet no-op. All mapped agent files carry the settle step; Bash granted where missing. Still open: runner-side settlement fallback for agents that crash before settling, and live verification on a real dispatch.

jared commented

2026-08-24 13:58:39 +00:00

Author

Owner

Approved design for the crash fallback (2026-08-24, session retro of #491). Three layers, no retries, every detection routes the map's error edge — fail loudly.

Layer 1 — SubagentStop hook detects unsettled stops. In CompletionIntake, when the hook entry path finds the dispatch still pending and carries no handoff content, that is the failure signal: stamp the dispatch failed (reason stopped_without_settlement), record an error verdict ("agent stopped without running subagent-stop"), and run NextStep so the node's error edge routes or the implementation fails. late_hook_replay? already makes the self-settled case a no-op.

Layer 2 — deadline reaper for hard crashes (process/session death, no hook). Add dispatches.deadline_at, stamped at dispatch time from map config (defaults.deadline_minutes with per-node deadline_minutes override, e.g. programmer: 45). A Reaper class marks overdue pending dispatches failed (deadline_exceeded) with an error verdict. Run the reaper inside every runner entry point that reads state (next_step, implementation-status, subagent-start). No daemon — lazy freshness. A false reap on a slow node is a loud error edge, the correct side to err on.

Layer 3 — os-sdlc-runner status (with --json): one table over dispatch rows showing per-implementation node, dispatch id, age, deadline, and state (running / FAILED reason / complete). This is how an orchestrator of parallel implementations observes liveness without polling agents.

Acceptance criteria:

A hook event for a pending dispatch with no handoff content marks it failed (stopped_without_settlement), records an error verdict, and routing proceeds via NextStep.
Dispatch rows are created with deadline_at derived from map deadline_minutes (node override, map default, hardcoded fallback 20).
Reaper marks pending dispatches past deadline_at as failed (deadline_exceeded) with an error verdict; it runs on next_step, implementation-status, and subagent-start entry points.
os-sdlc-runner status lists each implementation's latest dispatch with node, age, deadline, and state; --json emits the same machine-readably.
No retry logic anywhere; a settled dispatch is never reaped or re-failed (idempotent with late_hook_replay?).

Approved design for the crash fallback (2026-08-24, session retro of #491). Three layers, no retries, every detection routes the map's error edge — fail loudly. Layer 1 — SubagentStop hook detects unsettled stops. In CompletionIntake, when the hook entry path finds the dispatch still pending and carries no handoff content, that is the failure signal: stamp the dispatch failed (reason stopped_without_settlement), record an error verdict ("agent stopped without running subagent-stop"), and run NextStep so the node's error edge routes or the implementation fails. late_hook_replay? already makes the self-settled case a no-op. Layer 2 — deadline reaper for hard crashes (process/session death, no hook). Add dispatches.deadline_at, stamped at dispatch time from map config (defaults.deadline_minutes with per-node deadline_minutes override, e.g. programmer: 45). A Reaper class marks overdue pending dispatches failed (deadline_exceeded) with an error verdict. Run the reaper inside every runner entry point that reads state (next_step, implementation-status, subagent-start). No daemon — lazy freshness. A false reap on a slow node is a loud error edge, the correct side to err on. Layer 3 — `os-sdlc-runner status` (with --json): one table over dispatch rows showing per-implementation node, dispatch id, age, deadline, and state (running / FAILED reason / complete). This is how an orchestrator of parallel implementations observes liveness without polling agents. Acceptance criteria: 1. A hook event for a pending dispatch with no handoff content marks it failed (stopped_without_settlement), records an error verdict, and routing proceeds via NextStep. 2. Dispatch rows are created with deadline_at derived from map deadline_minutes (node override, map default, hardcoded fallback 20). 3. Reaper marks pending dispatches past deadline_at as failed (deadline_exceeded) with an error verdict; it runs on next_step, implementation-status, and subagent-start entry points. 4. `os-sdlc-runner status` lists each implementation's latest dispatch with node, age, deadline, and state; `--json` emits the same machine-readably. 5. No retry logic anywhere; a settled dispatch is never reaped or re-failed (idempotent with late_hook_replay?).

jared commented

2026-08-24 14:00:19 +00:00

Author

Owner

Work starting on branch worktree-ticket-496 via the os-sdlc pipeline (ADR-0145 records the design; see the design comment above for acceptance criteria).

jared commented

2026-08-24 16:24:18 +00:00

Author

Owner

Crash-fallback settlement shipped (commit 80713f3, branch worktree-ticket-496). Blank/empty content on subagent-stop now settles the dispatch as verdict error, step stopped_without_settlement, and still relays next_step. Live-verified by settling crashed programmer dispatch 37 from the interrupted 2026-08-24 run (implementation 5 recorded as implementation-failed; code product was complete and green, finished manually). Suite 1054 runs / 0 failures; rubocop clean. Still open on this ticket: the true-crash case where subagent-stop is never called at all, and the test-writer relay-drop pattern (0-for-2 in the live run).

Crash-fallback settlement shipped (commit 80713f3, branch worktree-ticket-496). Blank/empty content on subagent-stop now settles the dispatch as verdict error, step stopped_without_settlement, and still relays next_step. Live-verified by settling crashed programmer dispatch 37 from the interrupted 2026-08-24 run (implementation 5 recorded as implementation-failed; code product was complete and green, finished manually). Suite 1054 runs / 0 failures; rubocop clean. Still open on this ticket: the true-crash case where subagent-stop is never called at all, and the test-writer relay-drop pattern (0-for-2 in the live run).

jared referenced this issue from a commit

2026-08-24 18:52:21 +00:00

feat(os-sdlc): agents settle their own dispatches (ADR-0144, #496)

jared closed this issue

2026-08-24 18:52:21 +00:00

jared referenced this issue from a commit

2026-08-24 18:52:21 +00:00

feat(os-sdlc): crash-fallback settlement for dispatches stopped without content (#496)

jared referenced this issue from a commit

2026-08-24 18:52:21 +00:00

Merge worktree-ticket-496: crash-fallback settlement (#496)

jared referenced this issue