jared/cc-os

Watch

Fork 0

os-sdlc: evidence collect with general names runs 6 minutes and the probe daemon fails, which ends the implementation at change-auditor #1678

New issue

Open

opened 2026-11-10 19:47:03 +00:00 by jared · 3 comments

jared commented

2026-11-10 19:47:03 +00:00

Owner

Problem

The change-auditor agent runs bin/evidence collect --root <target root> --expect-new ... <tokens>. When the agent passes general names as tokens, the command does far more work than the audit needs, and the ruby-lsp probe daemon does not answer in time. The agent's role then requires the verdict error, and the Runner ends the implementation with implementation_failed (reason: verdict:error). The run cannot continue. The orchestrator must find the cause and open a new implementation.

Observed

On 2026-11-10 in a new worktree (ticket-1676), implementations 272 and 273 for ticket #1676 both failed at the first node. Dispatches 2668 and 2669 ran evidence collect with 21 tokens: CLI Emitter Result Run Boot Inputs CrcArgv ArgvOption UsageText TABLE ProcessIO Project Outcome SUBCOMMANDS connect_project! crc_run_dir_input crc_jobs crc_inward_delta crc_ledger_commits crc_shares shares_request. Both got RuntimeError: lsp-probe: the ruby-lsp daemon socket never came up at /run/user/1000/procman/835165fdb843/lsp-probe/daemon.sock from lib/os_sdlc/daemon/client.rb:59, at a different token each time (SenderRow#escalation_fields, then SenderRow#confirmed). The orchestrator then ran the same command by hand with a warm daemon: exit 0 after 6 min 1 s wall time (553 s user CPU), with 2,654,075 bytes on stdout. Implementation 274 passed the node after the orchestrator told the agent to pass only names specific to the ticket (Emitter, CrcArgv, shares_request) and to read the general names from the files.

Reproduce

In a fresh worktree of cc-os, from plugins/os-sdlc: bin/evidence collect --root "$PWD" --expect-new Crc::Jobs::CLI CLI Emitter Result Run Boot Inputs Project ProcessIO Outcome TABLE. Expected: an answer in seconds, or a refusal that names the tokens that are too general. Actual: minutes of work and megabytes of output; on a cold daemon, the socket error above.

Origin

Trigger: implementations 272 and 273 failed at change-auditor with verdict error (dispatches 2668, 2669)
Improvised this session: the orchestrator ran the evidence command by hand to find the cause, then gave the third auditor a token list
Chain: implementation ends at the first node ← role maps an evidence-tool failure to verdict error ← daemon socket does not come up under load ← evidence collect accepts any token and probes every sender of a general name
Session: a48658d5-b566-40ce-8257-ebeabf56f7e1
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os-plugins-os-sdlc/a48658d5-b566-40ce-8257-ebeabf56f7e1.jsonl
Where: plugins/os-sdlc/lib/os_sdlc/evidence/collection.rb (OsSdlc::Evidence::Collection), plugins/os-sdlc/lib/os_sdlc/daemon/client.rb:59 (OsSdlc::Daemon::Client#connect), plugins/os-sdlc/agents/change-auditor.md
Root candidate: none found; searched evidence collect, daemon socket, general names, too general, change-auditor, lsp-probe, evidence/collection

VERDICT: CREATE
REASON: This was a real, complete run in a fresh worktree, and it failed twice at the first node of #1676. The chain ends in the evidence tool (evidence collect accepting very general tokens), not in an agent improvisation. Open #967 covers a different fault, and open #1288 covers a PHANTOM verdict on non-Ruby files.

## Problem The `change-auditor` agent runs `bin/evidence collect --root <target root> --expect-new ... <tokens>`. When the agent passes general names as tokens, the command does far more work than the audit needs, and the `ruby-lsp` probe daemon does not answer in time. The agent's role then requires the verdict `error`, and the Runner ends the implementation with `implementation_failed` (`reason: verdict:error`). The run cannot continue. The orchestrator must find the cause and open a new implementation. ## Observed On 2026-11-10 in a new worktree (`ticket-1676`), implementations 272 and 273 for ticket #1676 both failed at the first node. Dispatches 2668 and 2669 ran `evidence collect` with 21 tokens: `CLI Emitter Result Run Boot Inputs CrcArgv ArgvOption UsageText TABLE ProcessIO Project Outcome SUBCOMMANDS connect_project! crc_run_dir_input crc_jobs crc_inward_delta crc_ledger_commits crc_shares shares_request`. Both got `RuntimeError: lsp-probe: the ruby-lsp daemon socket never came up at /run/user/1000/procman/835165fdb843/lsp-probe/daemon.sock` from `lib/os_sdlc/daemon/client.rb:59`, at a different token each time (`SenderRow#escalation_fields`, then `SenderRow#confirmed`). The orchestrator then ran the same command by hand with a warm daemon: exit 0 after 6 min 1 s wall time (553 s user CPU), with 2,654,075 bytes on stdout. Implementation 274 passed the node after the orchestrator told the agent to pass only names specific to the ticket (`Emitter`, `CrcArgv`, `shares_request`) and to read the general names from the files. ## Reproduce In a fresh worktree of cc-os, from `plugins/os-sdlc`: `bin/evidence collect --root "$PWD" --expect-new Crc::Jobs::CLI CLI Emitter Result Run Boot Inputs Project ProcessIO Outcome TABLE`. Expected: an answer in seconds, or a refusal that names the tokens that are too general. Actual: minutes of work and megabytes of output; on a cold daemon, the socket error above. ## Origin Trigger: implementations 272 and 273 failed at change-auditor with verdict error (dispatches 2668, 2669) Improvised this session: the orchestrator ran the evidence command by hand to find the cause, then gave the third auditor a token list Chain: implementation ends at the first node ← role maps an evidence-tool failure to verdict `error` ← daemon socket does not come up under load ← evidence collect accepts any token and probes every sender of a general name Session: a48658d5-b566-40ce-8257-ebeabf56f7e1 Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os-plugins-os-sdlc/a48658d5-b566-40ce-8257-ebeabf56f7e1.jsonl Where: plugins/os-sdlc/lib/os_sdlc/evidence/collection.rb (OsSdlc::Evidence::Collection), plugins/os-sdlc/lib/os_sdlc/daemon/client.rb:59 (OsSdlc::Daemon::Client#connect), plugins/os-sdlc/agents/change-auditor.md Root candidate: none found; searched evidence collect, daemon socket, general names, too general, change-auditor, lsp-probe, evidence/collection VERDICT: CREATE REASON: This was a real, complete run in a fresh worktree, and it failed twice at the first node of #1676. The chain ends in the evidence tool (`evidence collect` accepting very general tokens), not in an agent improvisation. Open #967 covers a different fault, and open #1288 covers a PHANTOM verdict on non-Ruby files.

jared added the

plugin/os-sdlc

filed-by/agent

category/code-change

labels

2026-11-10 19:47:03 +00:00

jared commented

2026-11-10 20:46:31 +00:00

Author

Owner

Occurrence: change-auditor slow with a narrow token list
Seen: implementation 275 (ticket #1677), dispatch 2685, node change-auditor
Run: session a48658d5-b566-40ce-8257-ebeabf56f7e1
Adds: the dispatch took about 40 minutes of wall time and returned verdict pass. The prompt gave a narrow token list and the ruby-lsp daemon was warm (lsp-probe symbols ran first from the worktree). A narrow token list therefore does not remove the delay. The cause of this delay is not measured yet.

Occurrence: change-auditor slow with a narrow token list Seen: implementation 275 (ticket #1677), dispatch 2685, node change-auditor Run: session a48658d5-b566-40ce-8257-ebeabf56f7e1 Adds: the dispatch took about 40 minutes of wall time and returned verdict pass. The prompt gave a narrow token list and the ruby-lsp daemon was warm (lsp-probe symbols ran first from the worktree). A narrow token list therefore does not remove the delay. The cause of this delay is not measured yet.

jared referenced this issue

2026-11-10 21:02:12 +00:00

os-sdlc: crc-census, crc-records, crc-tickets, crc-score, crc-placement, crc-file enter through Thor #1677

jared commented

2026-11-11 07:04:59 +00:00

Author

Owner

Observed

elapsed_s of each change-auditor dispatch, read from the Runner report in the transcript (session a48658d5, 2026-11-10):

Implementation	`elapsed_s`	Verdict	Note
270	606	pass
271	646	pass
272	116	error	`lsp-probe: the ruby-lsp daemon socket never came up at /run/user/1000/procman/835165fdb843/lsp-probe/daemon.sock`
273	186	error	same error
274	255	pass	the orchestrator told the agent to pass only ticket-specific names
275 (ticket #1677)	2369	pass	the session reported "about 40 min"
277 (ticket #1682)	206	pass	the prompt forbade the evidence tool and `lsp-probe`

For implementations 272 and 273 the agent ran bin/evidence collect with 21 tokens, among them general names (CLI, Result, Run, Project, ProcessIO).
The error came from RuntimeError at plugins/os-sdlc/lib/os_sdlc/daemon/client.rb:59 (connect raises after READY_TIMEOUT = 15 seconds, line 9).
lsp-probe warm --root <worktree> and a second run failed the same way (dispatch 2668).
The orchestrator ran the command by hand with a warm daemon: exit 0 after 6 min 1 s wall time, 553 s CPU, 2,654,075 bytes on stdout (#1678).
The user wrote "11 minutes on the change-auditor" while ticket #1682 ran.
The Runner recorded 206 s for that dispatch (implementation 277). The whole run took 38 min 5 s over 11 dispatches; the agent time was 1,313 s (22 min), and 16 min passed between dispatches.
The session did not find where the 11 minutes came from; it planned to add the difference to #1678.
The takedown design phase also runs bin/evidence collect once at final-check: four runs took 75 s to 327 s, and a 600 s timeout fails the run (plugins/os-sdlc/eval/ticket-takedown/lib/takedown/command/evidence_check.rb:5,44).
After the failures the user ordered: "After this ticket, Implement the rest by hand."
The session then dropped the pipeline for the remaining pieces.

Reproduce

cd plugins/os-sdlc && bin/evidence collect --root "$PWD" --expect-new Crc::Jobs::CLI CLI Emitter Result Run Boot Inputs Project ProcessIO Outcome TABLE

This is the command from #1678. Expected: an answer in seconds. Actual: minutes of work and megabytes of output; on a cold daemon, the socket error above.

Approach

Merge into #1678.
Add the gap that #1678 does not name: a change-auditor step that spends more than 600 s with no daemon failure (implementations 270, 271, 275), and 16 min between dispatches in implementation 277 that the Runner report does not explain.
Decide in the ticket decomposition whether evidence collect refuses or bounds a token list that names a general class, and whether the daemon starts before the first query (lsp-probe warm did not help).

Occurrence: Fix slow or failing evidence collect and lsp daemon in change-auditor Seen: plugins/os-sdlc/lib/os_sdlc/daemon/client.rb:9,56-59; plugins/os-sdlc/eval/ticket-takedown/lib/takedown/command/evidence_check.rb:5,44 Run: Session a48658d5-b566-40ce-8257-ebeabf56f7e1 (implementations of #1675 to #1682) Adds: per-implementation timings and the remark of the owner on 2026-11-10 ("11 minutes on the change-auditor is crazy. Something is broken - lsp probably."). Detail follows. ## Observed `elapsed_s` of each `change-auditor` dispatch, read from the Runner report in the transcript (session `a48658d5`, 2026-11-10): | Implementation | `elapsed_s` | Verdict | Note | |---|---|---|---| | 270 | 606 | pass | | | 271 | 646 | pass | | | 272 | 116 | error | `lsp-probe: the ruby-lsp daemon socket never came up at /run/user/1000/procman/835165fdb843/lsp-probe/daemon.sock` | | 273 | 186 | error | same error | | 274 | 255 | pass | the orchestrator told the agent to pass only ticket-specific names | | 275 (ticket #1677) | 2369 | pass | the session reported "about 40 min" | | 277 (ticket #1682) | 206 | pass | the prompt forbade the evidence tool and `lsp-probe` | For implementations 272 and 273 the agent ran `bin/evidence collect` with 21 tokens, among them general names (`CLI`, `Result`, `Run`, `Project`, `ProcessIO`). The error came from `RuntimeError` at `plugins/os-sdlc/lib/os_sdlc/daemon/client.rb:59` (`connect` raises after `READY_TIMEOUT = 15` seconds, line 9). `lsp-probe warm --root <worktree>` and a second run failed the same way (dispatch 2668). The orchestrator ran the command by hand with a warm daemon: exit 0 after 6 min 1 s wall time, 553 s CPU, 2,654,075 bytes on stdout (#1678). The user wrote "11 minutes on the change-auditor" while ticket #1682 ran. The Runner recorded 206 s for that dispatch (implementation 277). The whole run took 38 min 5 s over 11 dispatches; the agent time was 1,313 s (22 min), and 16 min passed between dispatches. The session did not find where the 11 minutes came from; it planned to add the difference to #1678. The takedown design phase also runs `bin/evidence collect` once at `final-check`: four runs took 75 s to 327 s, and a 600 s timeout fails the run (`plugins/os-sdlc/eval/ticket-takedown/lib/takedown/command/evidence_check.rb:5,44`). After the failures the user ordered: "After this ticket, Implement the rest by hand." The session then dropped the pipeline for the remaining pieces. ## Reproduce ``` cd plugins/os-sdlc && bin/evidence collect --root "$PWD" --expect-new Crc::Jobs::CLI CLI Emitter Result Run Boot Inputs Project ProcessIO Outcome TABLE ``` This is the command from #1678. Expected: an answer in seconds. Actual: minutes of work and megabytes of output; on a cold daemon, the socket error above. ## Approach Merge into #1678. Add the gap that #1678 does not name: a change-auditor step that spends more than 600 s with no daemon failure (implementations 270, 271, 275), and 16 min between dispatches in implementation 277 that the Runner report does not explain. Decide in the ticket decomposition whether `evidence collect` refuses or bounds a token list that names a general class, and whether the daemon starts before the first query (`lsp-probe warm` did not help).

jared commented

2026-11-12 17:52:00 +00:00

Author

Owner

Correction 2026-11-12

The "Reproduce" tokens name classes that no longer exist: Boot, Inputs, CrcArgv, and TABLE. The command in that section no longer reproduces the run as written.
Commits 07e780513 and e357262f3 (#1693, 2026-11-11) changed the daemon handshake and the process start of ruby-lsp.
Nobody has seen the socket failure again since #1693.
This comment does not claim that the defect is fixed. A new observation with current tokens must confirm or refute it.

Correction 2026-11-12 The "Reproduce" tokens name classes that no longer exist: `Boot`, `Inputs`, `CrcArgv`, and `TABLE`. The command in that section no longer reproduces the run as written. Commits 07e780513 and e357262f3 (#1693, 2026-11-11) changed the daemon handshake and the process start of `ruby-lsp`. Nobody has seen the socket failure again since #1693. This comment does not claim that the defect is fixed. A new observation with current tokens must confirm or refute it.

jared referenced this issue

2026-11-12 18:28:29 +00:00

os-sdlc entry: decide and build Thor for argument parsing behind the fixed os-sdlc-* script names (ADR-0232) #1646

jared referenced this issue

2026-11-13 20:39:47 +00:00

bin/evidence: a new file resolves to an unrelated file with the same basename #1760