jared/cc-os

Watch

Fork 0

lsp-probe daemon: concurrent first connects leave an orphan daemon until idle timeout #611

New issue

Closed

opened 2026-09-17 21:31:17 +00:00 by jared · 3 comments

jared commented

2026-09-17 21:31:17 +00:00

Owner

lsp-probe daemon: concurrent first connects leave an orphan daemon until idle timeout [capture]

Context

After the #606 merge (ADR-0201) the lsp-probe daemon is keyed on the project root and freshness is a socket connect. Two clients that connect at the same moment both see no socket and both spawn a daemon.

Observed

Two daemons for one root, started in the same second, both alive 17 minutes later. The hash directory holds one socket.

ps -o pid,ppid,lstart,args -p 3884471,3884483

3884471    1410 Thu Sep 17 17:13:47 2026 ruby plugins/os-sdlc/bin/lsp-probe serve --root /home/jared/dev/cc-os/plugins/os-sdlc
3884483    1410 Thu Sep 17 17:13:47 2026 ruby plugins/os-sdlc/bin/lsp-probe serve --root /home/jared/dev/cc-os/plugins/os-sdlc

ls -la plugins/os-sdlc/.cc-os/lsp/3e78cdd19064/
srwxr-xr-x. 1 jared jared 0 Sep 17 17:13 daemon.sock

Mechanism. Daemon#fresh? fails for both clients before either socket exists, so both call spawn!. Server.listen runs FileUtils.rm_f(sock_path) then UNIXServer.new(sock_path), so the second bind deletes the first daemon's socket. The first daemon keeps running with an unreachable listener because Server#abandoned? only checks File.exist?(sock_path), and the second daemon's socket satisfies that check. The orphan lives until IDLE_TIMEOUT (1800 s) and holds one ruby-lsp process the whole time.

Reproduce

cd plugins/os-sdlc && rm -rf .cc-os/lsp && (bin/evidence --root . Handoff >/dev/null &) && bin/evidence --root . Handoff >/dev/null; sleep 5; pgrep -af 'lsp-probe serve'

Two serve processes for the same root.

Expected

An evidence repro run: two parallel bin/evidence --root . Handoff invocations after rm -rf .cc-os/lsp leave exactly one lsp-probe serve process for the root.

Slice

Daemon#connect takes an exclusive File#flock on a spawn.lock file inside the per-root hash directory (Paths#dir) around spawn! unless fresh?. Paths#prepare! runs first so the directory exists before the lock file is opened. Inside the lock, after spawn!, connect polls fresh? until the socket accepts (short sleep, bounded, ~5 s cap) before releasing the lock, so a second waiting client sees a live socket rather than racing a fresh spawn. The lock releases when the block ends, and the kernel releases it if the process exits first. Client.new happens after the lock is released. Server.listen keeps FileUtils.rm_f for stale socket files from a crashed daemon; no other change to Server/abandoned? in this ticket. Amend ADR-0201 with one sentence noting the spawn lock.

Acceptance criteria

The evidence repro (cd plugins/os-sdlc && rm -rf .cc-os/lsp && (bin/evidence --root . Handoff >/dev/null &) && bin/evidence --root . Handoff >/dev/null; sleep 5; pgrep -af 'lsp-probe serve') leaves exactly one lsp-probe serve process for the root.
ADR-0201 carries one added sentence describing the spawn lock; no new ADR.

Non-goals

No change to Server#abandoned? or the socket inode check; the flock removes the race that made it matter.
No change to Server.listen's rm_f behavior beyond leaving it as-is for stale sockets from a crashed daemon.

Approach

# plugins/os-sdlc/lib/os_sdlc/lsp_probe/daemon.rb
def connect
  @paths.prepare!
  with_spawn_lock { spawn_and_wait unless fresh? }
  Client.new(@paths.sock_path)
end

def with_spawn_lock
  File.open(@paths.dir / "spawn.lock", File::RDWR | File::CREAT) do |lock|
    lock.flock(File::LOCK_EX)
    yield
  end
end

def spawn_and_wait
  spawn!
  wait_until_fresh
end

def wait_until_fresh(deadline: 5.0, step: 0.05)
  # poll fresh? until true; raise a named error with the sock path when the deadline passes
end

Race, before the fix:

Step	Client A	Client B	Sockets on disk
1	sees `fresh?` false	sees `fresh?` false	none
2	spawns a daemon	spawns a daemon	none yet
3	binds, socket appears	binds, `rm_f` deletes A's socket, then binds	B's socket only
4	connects to a dead listener	connects	one live socket, one orphan daemon for 30 minutes

Race, after the fix:

Step	Client A	Client B	Sockets on disk
1	takes the spawn lock	blocks on the spawn lock	none
2	spawns a daemon, waits for the socket	still blocked	none, then one
3	releases the lock, connects	acquires the lock, sees `fresh?` true, does not spawn	one
4	connected	releases the lock, connects	one daemon total

Rejected alternatives:

Inode check in Server#abandoned? (daemon exits when the socket path's inode is not its own): defends against a cause the lock removes; keep for a later ticket if orphans reappear.
Exclusive bind without rm_f in Server.listen: a stale socket file from a crashed daemon would then block every future spawn.
Pidfile: a stale pidfile after a crash needs its own liveness check; flock is released by the kernel on exit for free.

Conditions this approach depends on:

spawn_and_wait must not release the lock until fresh? is true, or a second client can spawn in the gap.
The lock file lives in Paths#dir (per-root hash directory), so two roots never contend.
Warm-root cost is one uncontended flock plus one connect; no polling.
ADR-0201 gains one sentence recording the spawn lock; no new ADR.

Owner sketch

class Daemon
  def connect
  end

  private

  def with_spawn_lock
  end

  def wait_until_fresh
  end
end

Cases

Given no socket for a root, when two Daemon.connect(root:) calls race, then exactly one lsp-probe serve process spawns and both clients connect.
Given a warm root with a live socket, when Daemon#connect runs, then it does not take the spawn! path and returns promptly.
Given a cold connect has finished and released the spawn lock, when a later cold connect happens, then it can acquire the lock and spawn again.
Given spawn! has run but the socket never appears, when Daemon#connect polls past the bound, then it raises a clear error instead of hanging.

Origin

Trigger: pgrep -af 'lsp-probe serve' after the #610 worktree finish, 2026-09-17; the two daemons came from a baseline suite run in the main checkout and a diagnosis agent running the evidence bin tests at the same time
Improvised this session: none; the two daemons were left to time out
Chain: orphan daemon ← second bind deletes the first socket, first server cannot tell ← abandoned? checks path existence only, no spawn lock ← DESIGN (ADR-0201, Daemon::Server.listen, Daemon#connect)
Root candidate: this ticket
Where: plugins/os-sdlc/lib/os_sdlc/lsp_probe/daemon.rb (connect, spawn!)
Session: 67967f79-5ade-4957-bd03-0a46b0f88a03
Transcript: /home/jared/dev/cc-os/.claude/projects/-home-jared-dev-cc-os/67967f79-5ade-4957-bd03-0a46b0f88a03.jsonl

Skeptic verdict

VERDICT: CREATE — observed resource leak traced to a design gap in ADR-0201 Server.listen/abandoned?; origin chain ends in DESIGN; root candidate is this ticket.

# lsp-probe daemon: concurrent first connects leave an orphan daemon until idle timeout [capture] ## Context After the #606 merge (ADR-0201) the lsp-probe daemon is keyed on the project root and freshness is a socket connect. Two clients that connect at the same moment both see no socket and both spawn a daemon. ## Observed Two daemons for one root, started in the same second, both alive 17 minutes later. The hash directory holds one socket. ``` ps -o pid,ppid,lstart,args -p 3884471,3884483 ``` ``` 3884471 1410 Thu Sep 17 17:13:47 2026 ruby plugins/os-sdlc/bin/lsp-probe serve --root /home/jared/dev/cc-os/plugins/os-sdlc 3884483 1410 Thu Sep 17 17:13:47 2026 ruby plugins/os-sdlc/bin/lsp-probe serve --root /home/jared/dev/cc-os/plugins/os-sdlc ls -la plugins/os-sdlc/.cc-os/lsp/3e78cdd19064/ srwxr-xr-x. 1 jared jared 0 Sep 17 17:13 daemon.sock ``` Mechanism. Daemon#fresh? fails for both clients before either socket exists, so both call spawn!. Server.listen runs FileUtils.rm_f(sock_path) then UNIXServer.new(sock_path), so the second bind deletes the first daemon's socket. The first daemon keeps running with an unreachable listener because Server#abandoned? only checks File.exist?(sock_path), and the second daemon's socket satisfies that check. The orphan lives until IDLE_TIMEOUT (1800 s) and holds one ruby-lsp process the whole time. ## Reproduce ``` cd plugins/os-sdlc && rm -rf .cc-os/lsp && (bin/evidence --root . Handoff >/dev/null &) && bin/evidence --root . Handoff >/dev/null; sleep 5; pgrep -af 'lsp-probe serve' ``` Two serve processes for the same root. ## Expected An evidence repro run: two parallel `bin/evidence --root . Handoff` invocations after `rm -rf .cc-os/lsp` leave exactly one `lsp-probe serve` process for the root. ## Slice Daemon#connect takes an exclusive `File#flock` on a `spawn.lock` file inside the per-root hash directory (`Paths#dir`) around `spawn! unless fresh?`. `Paths#prepare!` runs first so the directory exists before the lock file is opened. Inside the lock, after `spawn!`, connect polls `fresh?` until the socket accepts (short sleep, bounded, ~5 s cap) before releasing the lock, so a second waiting client sees a live socket rather than racing a fresh spawn. The lock releases when the block ends, and the kernel releases it if the process exits first. `Client.new` happens after the lock is released. `Server.listen` keeps `FileUtils.rm_f` for stale socket files from a crashed daemon; no other change to `Server`/`abandoned?` in this ticket. Amend ADR-0201 with one sentence noting the spawn lock. ## Acceptance criteria - The evidence repro (`cd plugins/os-sdlc && rm -rf .cc-os/lsp && (bin/evidence --root . Handoff >/dev/null &) && bin/evidence --root . Handoff >/dev/null; sleep 5; pgrep -af 'lsp-probe serve'`) leaves exactly one `lsp-probe serve` process for the root. - ADR-0201 carries one added sentence describing the spawn lock; no new ADR. ## Non-goals - No change to `Server#abandoned?` or the socket inode check; the flock removes the race that made it matter. - No change to `Server.listen`'s `rm_f` behavior beyond leaving it as-is for stale sockets from a crashed daemon. ## Approach ```ruby # plugins/os-sdlc/lib/os_sdlc/lsp_probe/daemon.rb def connect @paths.prepare! with_spawn_lock { spawn_and_wait unless fresh? } Client.new(@paths.sock_path) end def with_spawn_lock File.open(@paths.dir / "spawn.lock", File::RDWR | File::CREAT) do |lock| lock.flock(File::LOCK_EX) yield end end def spawn_and_wait spawn! wait_until_fresh end def wait_until_fresh(deadline: 5.0, step: 0.05) # poll fresh? until true; raise a named error with the sock path when the deadline passes end ``` Race, before the fix: | Step | Client A | Client B | Sockets on disk | |---|---|---|---| | 1 | sees `fresh?` false | sees `fresh?` false | none | | 2 | spawns a daemon | spawns a daemon | none yet | | 3 | binds, socket appears | binds, `rm_f` deletes A's socket, then binds | B's socket only | | 4 | connects to a dead listener | connects | one live socket, one orphan daemon for 30 minutes | Race, after the fix: | Step | Client A | Client B | Sockets on disk | |---|---|---|---| | 1 | takes the spawn lock | blocks on the spawn lock | none | | 2 | spawns a daemon, waits for the socket | still blocked | none, then one | | 3 | releases the lock, connects | acquires the lock, sees `fresh?` true, does not spawn | one | | 4 | connected | releases the lock, connects | one daemon total | Rejected alternatives: - Inode check in `Server#abandoned?` (daemon exits when the socket path's inode is not its own): defends against a cause the lock removes; keep for a later ticket if orphans reappear. - Exclusive bind without `rm_f` in `Server.listen`: a stale socket file from a crashed daemon would then block every future spawn. - Pidfile: a stale pidfile after a crash needs its own liveness check; flock is released by the kernel on exit for free. Conditions this approach depends on: - `spawn_and_wait` must not release the lock until `fresh?` is true, or a second client can spawn in the gap. - The lock file lives in `Paths#dir` (per-root hash directory), so two roots never contend. - Warm-root cost is one uncontended flock plus one connect; no polling. - ADR-0201 gains one sentence recording the spawn lock; no new ADR. ## Owner sketch ```ruby class Daemon def connect end private def with_spawn_lock end def wait_until_fresh end end ``` ## Cases 1. Given no socket for a root, when two `Daemon.connect(root:)` calls race, then exactly one `lsp-probe serve` process spawns and both clients connect. 2. Given a warm root with a live socket, when `Daemon#connect` runs, then it does not take the `spawn!` path and returns promptly. 3. Given a cold connect has finished and released the spawn lock, when a later cold connect happens, then it can acquire the lock and spawn again. 4. Given `spawn!` has run but the socket never appears, when `Daemon#connect` polls past the bound, then it raises a clear error instead of hanging. ## Origin - Trigger: pgrep -af 'lsp-probe serve' after the #610 worktree finish, 2026-09-17; the two daemons came from a baseline suite run in the main checkout and a diagnosis agent running the evidence bin tests at the same time - Improvised this session: none; the two daemons were left to time out - Chain: orphan daemon ← second bind deletes the first socket, first server cannot tell ← abandoned? checks path existence only, no spawn lock ← DESIGN (ADR-0201, Daemon::Server.listen, Daemon#connect) - Root candidate: this ticket - Where: plugins/os-sdlc/lib/os_sdlc/lsp_probe/daemon.rb (connect, spawn!) - Session: 67967f79-5ade-4957-bd03-0a46b0f88a03 - Transcript: /home/jared/dev/cc-os/.claude/projects/-home-jared-dev-cc-os/67967f79-5ade-4957-bd03-0a46b0f88a03.jsonl ## Skeptic verdict VERDICT: CREATE — observed resource leak traced to a design gap in ADR-0201 Server.listen/abandoned?; origin chain ends in DESIGN; root candidate is this ticket.

jared added the

plugin/os-sdlc

filed-by/agent

labels

2026-09-17 21:31:17 +00:00

jared referenced this issue

2026-09-18 13:40:13 +00:00

os-backlog validator: optional sections list for ## Retired tests and ## Approach #612

jared referenced this issue

2026-09-18 14:27:11 +00:00

Git::Repo#changed_paths mixes repo-relative and cwd-relative paths; nested targets get doubled paths in lint and diff-test #615

jared added the

ready-for-agent

label

2026-09-18 14:27:29 +00:00

jared commented

2026-09-18 14:27:29 +00:00

Author

Owner

Work started on branch ticket-611 via the os-sdlc pipeline (worktree .claude/worktrees/ticket-611). Approach section added before the run; #615 lint-gate path fix merged first.

jared commented

2026-09-18 14:36:28 +00:00

Author

Owner

Implementation 1 escalated at test-writer with unexpected_pass:test-writer/diff-test: diff-test exit 0, 0 ms, three times. Cause: #616 on this plugin. plugins/os-sdlc/.sdlc/project.yaml had no test_path, default test/ matched nothing under tests/, so the gate selected no files after #615 made paths cwd-relative. The written tests are red for the right reason (2 failures in tests/lsp_probe/daemon_test.rb). Fix: test_path: tests added to the config. Re-opening as implementation 2 with the red-round files on disk.

jared referenced this issue

2026-09-18 14:57:55 +00:00

subagent-start brief composition raises on an unmapped code file; ADR-0202 widened only the gate side #614

jared commented

2026-09-18 17:25:43 +00:00

Author

Owner

Resolution

Done: Daemon#connect holds an exclusive flock on spawn.lock in the per-root hash directory around fresh-check plus spawn and waits until the socket answers (SpawnTimeout after 5 s). Paths#lock_path owns the path. ADR-0201 amended with one sentence. Test fakes now listen (tests/support/listening_spawner.rb).

Evidence: Branch ticket-611 commit 94957824. os-sdlc suite 1349 runs green three times in a row; rubocop clean on 7 changed files. Live repro: daemons stopped, rm -rf .cc-os/lsp, two parallel bin/evidence runs, exactly one lsp-probe serve process for the root. Pipeline impl 2 reached implementation_complete after 19 dispatches; impl 1 escalated (unexpected_pass, #616 on this plugin, test_path added); programmer dispatch 10 stalled because the test-writer's concurrency test deadlocked under the fix (it waited for two spawns) and was hand-corrected; two suite runs went red intermittently during verification and did not reproduce in five later runs.

Follow-ups: #616 remains open (test_path default detection); this commit adds test_path: tests to plugins/os-sdlc/.sdlc/project.yaml as its case 3. The two non-reproducing suite reds are unticketed; no test names were captured.

## Resolution **Done:** Daemon#connect holds an exclusive flock on spawn.lock in the per-root hash directory around fresh-check plus spawn and waits until the socket answers (SpawnTimeout after 5 s). Paths#lock_path owns the path. ADR-0201 amended with one sentence. Test fakes now listen (tests/support/listening_spawner.rb). **Evidence:** Branch ticket-611 commit 94957824. os-sdlc suite 1349 runs green three times in a row; rubocop clean on 7 changed files. Live repro: daemons stopped, rm -rf .cc-os/lsp, two parallel bin/evidence runs, exactly one lsp-probe serve process for the root. Pipeline impl 2 reached implementation_complete after 19 dispatches; impl 1 escalated (unexpected_pass, #616 on this plugin, test_path added); programmer dispatch 10 stalled because the test-writer's concurrency test deadlocked under the fix (it waited for two spawns) and was hand-corrected; two suite runs went red intermittently during verification and did not reproduce in five later runs. **Follow-ups:** #616 remains open (test_path default detection); this commit adds test_path: tests to plugins/os-sdlc/.sdlc/project.yaml as its case 3. The two non-reproducing suite reds are unticketed; no test names were captured.

jared closed this issue

2026-09-18 17:25:43 +00:00

jared referenced this issue from a commit

2026-09-18 18:49:54 +00:00

feat(os-sdlc): lsp-probe Daemon#connect holds a per-root spawn lock and waits for the socket, so concurrent cold connects spawn one daemon (#611, amends ADR-0201)

jared referenced this issue from a commit

2026-09-18 18:49:54 +00:00

docs(status): os-sdlc leaf History rows for #615 and #611

~~jared referenced this issue 2026-09-19 21:08:42 +00:00~~

os-sdlc lsp-probe daemon socket path exceeds the 108-byte Unix socket limit in git worktrees #619