Harden stay_serial! executor shutdown/start cycle against orphaned workers and leftover sentinels #503

New issue

Open

opened 2026-08-25 01:01:48 +00:00 by jared · 0 comments

jared commented

2026-08-25 01:01:48 +00:00

Owner

Context

plugins/os-sdlc/tests/test_helper.rb run_test_exclusively shuts down and restarts Minitest's parallel executor per serial test. In minitest-6.0.6 parallel.rb, Executor#start replaces @pool unconditionally: if shutdown raises or is interrupted, orphaned threads keep popping the shared @queue, and surplus nil sentinels pushed per shutdown cycle can kill the next pool's threads on their first pop. Observed during #502 verification: 1 fatal "No live threads left. Deadlock?" hang in 22 suite runs, main thread at parallel.rb:54 Thread#join under run_test_exclusively; no thread held the SerializedChdir Monitor, ruling out the new chdir lock.

Ticket-skeptic verdict: CREATE (root location is run_test_exclusively itself; provenance is an observed hang plus minitest source reading).

Tasks

Harden the cycle: restart only after successful shutdown, drain leftover sentinels, or replace shutdown/start with a read-write lock (parallel tests read, serial tests write).
Reassess whether HandRolledFetchTest still needs stay_serial! now that test_helper serializes every Dir.chdir behind a Monitor.

Acceptance criteria

Repeated full parallel suite runs show no executor deadlock/hang and stay at the known baseline failures.

Origin

Trigger: 10-consecutive-run acceptance check for #502 (ruby tests/all.rb loops)
Improvised this session: none (hang did not recur; no workaround applied)
Chain: fatal deadlock hang ← executor shutdown/start hole in run_test_exclusively (test_helper.rb) ← UPSTREAM-TOOL (minitest 6.0.6 parallel.rb Executor#start/#shutdown semantics)
Root candidate: this ticket

## Context plugins/os-sdlc/tests/test_helper.rb run_test_exclusively shuts down and restarts Minitest's parallel executor per serial test. In minitest-6.0.6 parallel.rb, Executor#start replaces @pool unconditionally: if shutdown raises or is interrupted, orphaned threads keep popping the shared @queue, and surplus nil sentinels pushed per shutdown cycle can kill the next pool's threads on their first pop. Observed during #502 verification: 1 fatal "No live threads left. Deadlock?" hang in 22 suite runs, main thread at parallel.rb:54 Thread#join under run_test_exclusively; no thread held the SerializedChdir Monitor, ruling out the new chdir lock. Ticket-skeptic verdict: CREATE (root location is run_test_exclusively itself; provenance is an observed hang plus minitest source reading). ## Tasks - [ ] Harden the cycle: restart only after successful shutdown, drain leftover sentinels, or replace shutdown/start with a read-write lock (parallel tests read, serial tests write). - [ ] Reassess whether HandRolledFetchTest still needs stay_serial! now that test_helper serializes every Dir.chdir behind a Monitor. ## Acceptance criteria - [ ] Repeated full parallel suite runs show no executor deadlock/hang and stay at the known baseline failures. ## Origin - Trigger: 10-consecutive-run acceptance check for #502 (ruby tests/all.rb loops) - Improvised this session: none (hang did not recur; no workaround applied) - Chain: fatal deadlock hang ← executor shutdown/start hole in run_test_exclusively (test_helper.rb) ← UPSTREAM-TOOL (minitest 6.0.6 parallel.rb Executor#start/#shutdown semantics) - Root candidate: this ticket

jared added the

plugin/os-sdlc

label

2026-08-25 01:01:48 +00:00

jared referenced this issue

2026-08-25 01:02:00 +00:00

Absolutize ResolverScenario root, drop its Dir.chdir, retire stay_serial! on resolver_test #502

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

jared/cc-os#503

No description provided.

Rows
Columns