Add a cop that flags tests starting our own plugin bin or hook as a subprocess #1337

Closed
opened 2026-10-25 19:49:02 +00:00 by jared · 7 comments
Owner

Problem

A test that starts this repository's own script as a process pays one Ruby boot per test, and two failures go silent: data the test seeds in its own process does not reach the script, and a crash exits 1 like an expected failure. Jared ruled on 2026-10-25 that a test calls the entry in-process and that one smoke test per script starts it as a process. On 2026-10-26 he ruled that the cop must be general (smell analysis section 8, option B): it must not name an entry class for each plugin, and it must not hard-code folder or file-name lists. Nothing stops a new test from starting its own script today. The Sdlc/Structural/ExecutableLoadedInTest message still says scripts are tested "only as subprocesses", which contradicts the ruling and that cop's own lesson.

One test outside a smoke file starts its script today: plugins/os-sdlc/tests/detector/holders_bin_test.rb:56. It stays a process because Detector::Holders::SiteOffenses builds Detector::CopSites with no ruby_version:, so CopSource.target_version asks RuboCop, which finds .ruby-version through the process working folder and memoizes it once per process.

Approach

Add the cop Sdlc/Structural/OwnEntryPointSpawnedInTest. It reports a call in a test file when the call starts a process (Open3.*, system, spawn, IO.popen, backticks) and one argument resolves to an entry point of this repository. An entry point is a file whose first line starts with #! or a file in a folder named bin: the same definition as Sdlc/Structural/ExitOutsideEntryPoint. For a spawned path, the shebang check reads the first line of the resolved file on disk. A test file is a file whose name ends in _test.rb or starts with test_. Extract that definition into one object that both cops use. The argument resolves when it is a File.expand_path(..., __dir__) call or a constant assigned one in the same file; extend ExecutableLoadedInTest::ResolvedPath or move it so both cops share it. The cop config key SmokeTestPatterns (default ["*_bin_smoke_test.rb", "test_*_bin_smoke.rb"]) names the smoke files, where the cop reports nothing. No Exclude list.

The message prescribes the shape, not a class: "Call the script's entry in the test process: arguments and streams in, status out. Only the smoke test starts the script."

Make holders_bin_test.rb call the entry in-process: SiteOffenses passes the Ruby version read from the target root's .ruby-version to CopSites. Delete the two-line comment above the spawn with the spawn.

Reword the ExecutableLoadedInTest message to match its lesson's correction: a test calls the library entry; one smoke test runs the script as a process.

Add the lint lesson and its smell analysis. The smell analysis is the 2026-10-26 one (scratch copy in the session; write it to docs/lint-lessons/analyses/sdlc-structural-own-entry-point-spawned-in-test.md).

Required work:

  1. Add the cop and enable it in plugins/os-sdlc/.rubocop.yml with no Exclude list. check: present OwnEntryPointSpawnedInTest in plugins/os-sdlc/.rubocop.yml
  2. Add SmokeTestPatterns to the cop config. check: present SmokeTestPatterns in plugins/os-sdlc/.rubocop.yml
  3. Reword the ExecutableLoadedInTest message. check: absent only as subprocesses in plugins/os-sdlc/lib/os_sdlc/cops/structural/executable_loaded_in_test.rb
  4. Make the holders test call its entry in-process. check: absent Open3.capture2(SCRIPT in plugins/os-sdlc/tests/detector/holders_bin_test.rb
  5. Add the lint lesson. check: present Sdlc/Structural/OwnEntryPointSpawnedInTest in plugins/os-sdlc/lint-lessons/sdlc-structural-own-entry-point-spawned-in-test.yml
  6. Add the smell analysis. check: present OwnEntryPointSpawnedInTest in docs/lint-lessons/analyses/sdlc-structural-own-entry-point-spawned-in-test.md

Acceptance criteria

  1. Given a test file that does not match SmokeTestPatterns and calls Open3.capture2e(BIN) where BIN = File.expand_path("../bin/tool", __dir__) in the same file, when rubocop runs, then the cop reports one offense with the message above.
  2. Given a test file that starts a file whose first line is #!/usr/bin/env ruby, outside a bin folder, with Open3.capture3("ruby", HOOK), when rubocop runs, then the cop reports one offense.
  3. Given a file that matches SmokeTestPatterns and starts its script, when rubocop runs, then the cop reports no offense.
  4. Given a test that runs git, RbConfig.ruby, "-e", code, or a path held in another file's constant, when rubocop runs, then the cop reports no offense.
  5. Given a non-test file that starts its own script, when rubocop runs, then the cop reports no offense.
  6. Given a target root whose .ruby-version differs from the process working folder's, when the holders detector runs in-process for that root, then it parses with the target's Ruby version.
  7. Given the repository after this ticket, when rubocop runs over plugins/, then the cop reports zero offenses.

Owner sketch

class RuboCop::Cop::Sdlc::Structural::OwnEntryPointSpawnedInTest < RuboCop::Cop::Base
  def on_send(node)
end
class RuboCop::Cop::Sdlc::Structural::EntryPoint   # shared with ExitOutsideEntryPoint
  def self.file?(path)
end

Non-goals

Record and replay of external tools (rubocop, the suite command) in tests. That is a separate time saving and is not measured.

Origin

  • Trigger: Jared ruled on 2026-10-25 that tests call the in-process entry; on 2026-10-26 he accepted the general cop (option B) in session 006abca1.
  • Improvised this session: none
  • Chain: new tests can start their own script ← no cop flags it ← the ExecutableLoadedInTest message says scripts are tested "only as subprocesses"
  • Root candidate: none found; searched Sdlc/Structural/ExecutableLoadedInTest, bin_smoke_test, new cop, lint lesson
  • Where: plugins/os-sdlc/lib/os_sdlc/cops/structural/executable_loaded_in_test.rb, plugins/os-sdlc/lib/os_sdlc/detector/holders/site_offenses.rb, plugins/os-sdlc/.rubocop.yml
  • Session: 01c29715-c945-43b4-86fc-6c5fc6c4dd3f
  • Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/01c29715-c945-43b4-86fc-6c5fc6c4dd3f.jsonl
## Problem A test that starts this repository's own script as a process pays one Ruby boot per test, and two failures go silent: data the test seeds in its own process does not reach the script, and a crash exits 1 like an expected failure. Jared ruled on 2026-10-25 that a test calls the entry in-process and that one smoke test per script starts it as a process. On 2026-10-26 he ruled that the cop must be general (smell analysis section 8, option B): it must not name an entry class for each plugin, and it must not hard-code folder or file-name lists. Nothing stops a new test from starting its own script today. The `Sdlc/Structural/ExecutableLoadedInTest` message still says scripts are tested "only as subprocesses", which contradicts the ruling and that cop's own lesson. One test outside a smoke file starts its script today: `plugins/os-sdlc/tests/detector/holders_bin_test.rb:56`. It stays a process because `Detector::Holders::SiteOffenses` builds `Detector::CopSites` with no `ruby_version:`, so `CopSource.target_version` asks RuboCop, which finds `.ruby-version` through the process working folder and memoizes it once per process. ## Approach Add the cop `Sdlc/Structural/OwnEntryPointSpawnedInTest`. It reports a call in a test file when the call starts a process (`Open3.*`, `system`, `spawn`, `IO.popen`, backticks) and one argument resolves to an entry point of this repository. An entry point is a file whose first line starts with `#!` or a file in a folder named `bin`: the same definition as `Sdlc/Structural/ExitOutsideEntryPoint`. For a spawned path, the shebang check reads the first line of the resolved file on disk. A test file is a file whose name ends in `_test.rb` or starts with `test_`. Extract that definition into one object that both cops use. The argument resolves when it is a `File.expand_path(..., __dir__)` call or a constant assigned one in the same file; extend `ExecutableLoadedInTest::ResolvedPath` or move it so both cops share it. The cop config key `SmokeTestPatterns` (default `["*_bin_smoke_test.rb", "test_*_bin_smoke.rb"]`) names the smoke files, where the cop reports nothing. No Exclude list. The message prescribes the shape, not a class: "Call the script's entry in the test process: arguments and streams in, status out. Only the smoke test starts the script." Make `holders_bin_test.rb` call the entry in-process: `SiteOffenses` passes the Ruby version read from the target root's `.ruby-version` to `CopSites`. Delete the two-line comment above the spawn with the spawn. Reword the `ExecutableLoadedInTest` message to match its lesson's correction: a test calls the library entry; one smoke test runs the script as a process. Add the lint lesson and its smell analysis. The smell analysis is the 2026-10-26 one (scratch copy in the session; write it to `docs/lint-lessons/analyses/sdlc-structural-own-entry-point-spawned-in-test.md`). Required work: 1. Add the cop and enable it in `plugins/os-sdlc/.rubocop.yml` with no Exclude list. check: present `OwnEntryPointSpawnedInTest` in `plugins/os-sdlc/.rubocop.yml` 2. Add `SmokeTestPatterns` to the cop config. check: present `SmokeTestPatterns` in `plugins/os-sdlc/.rubocop.yml` 3. Reword the `ExecutableLoadedInTest` message. check: absent `only as subprocesses` in `plugins/os-sdlc/lib/os_sdlc/cops/structural/executable_loaded_in_test.rb` 4. Make the holders test call its entry in-process. check: absent `Open3.capture2(SCRIPT` in `plugins/os-sdlc/tests/detector/holders_bin_test.rb` 5. Add the lint lesson. check: present `Sdlc/Structural/OwnEntryPointSpawnedInTest` in `plugins/os-sdlc/lint-lessons/sdlc-structural-own-entry-point-spawned-in-test.yml` 6. Add the smell analysis. check: present `OwnEntryPointSpawnedInTest` in `docs/lint-lessons/analyses/sdlc-structural-own-entry-point-spawned-in-test.md` ## Acceptance criteria 1. Given a test file that does not match `SmokeTestPatterns` and calls `Open3.capture2e(BIN)` where `BIN = File.expand_path("../bin/tool", __dir__)` in the same file, when rubocop runs, then the cop reports one offense with the message above. 2. Given a test file that starts a file whose first line is `#!/usr/bin/env ruby`, outside a `bin` folder, with `Open3.capture3("ruby", HOOK)`, when rubocop runs, then the cop reports one offense. 3. Given a file that matches `SmokeTestPatterns` and starts its script, when rubocop runs, then the cop reports no offense. 4. Given a test that runs `git`, `RbConfig.ruby, "-e", code`, or a path held in another file's constant, when rubocop runs, then the cop reports no offense. 5. Given a non-test file that starts its own script, when rubocop runs, then the cop reports no offense. 6. Given a target root whose `.ruby-version` differs from the process working folder's, when the holders detector runs in-process for that root, then it parses with the target's Ruby version. 7. Given the repository after this ticket, when rubocop runs over `plugins/`, then the cop reports zero offenses. ## Owner sketch ```ruby class RuboCop::Cop::Sdlc::Structural::OwnEntryPointSpawnedInTest < RuboCop::Cop::Base def on_send(node) end class RuboCop::Cop::Sdlc::Structural::EntryPoint # shared with ExitOutsideEntryPoint def self.file?(path) end ``` ## Non-goals Record and replay of external tools (`rubocop`, the suite command) in tests. That is a separate time saving and is not measured. ## Origin - Trigger: Jared ruled on 2026-10-25 that tests call the in-process entry; on 2026-10-26 he accepted the general cop (option B) in session 006abca1. - Improvised this session: none - Chain: new tests can start their own script ← no cop flags it ← the ExecutableLoadedInTest message says scripts are tested "only as subprocesses" - Root candidate: none found; searched Sdlc/Structural/ExecutableLoadedInTest, bin_smoke_test, new cop, lint lesson - Where: plugins/os-sdlc/lib/os_sdlc/cops/structural/executable_loaded_in_test.rb, plugins/os-sdlc/lib/os_sdlc/detector/holders/site_offenses.rb, plugins/os-sdlc/.rubocop.yml - Session: 01c29715-c945-43b4-86fc-6c5fc6c4dd3f - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/01c29715-c945-43b4-86fc-6c5fc6c4dd3f.jsonl
Author
Owner

Unblocked: #1329-#1336 are merged (main c9f5db8f) and closed. Known spawn sites the cop's zero-offense criterion (AC5) must meet:

  1. plugins/os-sdlc/tests/fast_gate/os_sdlc_lint_bin_test.rb runs Open3.capture2e(BIN) on bin/os-sdlc-lint. Main added it during the batch.
  2. cc-architect smoke files are named test_*_bin_smoke.rb (minitest file convention there), not *_bin_smoke_test.rb.
  3. Tests that start our library with ruby -I lib -e (db_connect_bang_models_test, event_test, worklist_rubocop_config_test, project_shell_test, step_run_project_shell_test) are outside the 2026-10-25 ruling.
  4. DetectorHoldersBinTest keeps one Open3 run until #1226 is fixed; the os-backlog Checkout USER test stays a subprocess until #1407 is fixed. Both carry constraint comments.
  5. The four os-sdlc hook smoke tests use the skip path (ruling R3).
    Source: docs/crc/os-sdlc/fast-batch-15/batch-state.md, Open threads.
Unblocked: #1329-#1336 are merged (main c9f5db8f) and closed. Known spawn sites the cop's zero-offense criterion (AC5) must meet: 1. plugins/os-sdlc/tests/fast_gate/os_sdlc_lint_bin_test.rb runs Open3.capture2e(BIN) on bin/os-sdlc-lint. Main added it during the batch. 2. cc-architect smoke files are named test_*_bin_smoke.rb (minitest file convention there), not *_bin_smoke_test.rb. 3. Tests that start our library with ruby -I lib -e (db_connect_bang_models_test, event_test, worklist_rubocop_config_test, project_shell_test, step_run_project_shell_test) are outside the 2026-10-25 ruling. 4. DetectorHoldersBinTest keeps one Open3 run until #1226 is fixed; the os-backlog Checkout USER test stays a subprocess until #1407 is fixed. Both carry constraint comments. 5. The four os-sdlc hook smoke tests use the skip path (ruling R3). Source: docs/crc/os-sdlc/fast-batch-15/batch-state.md, Open threads.
Author
Owner

Work started on branch ticket-1337 (os-sdlc implement pipeline). The last known spawn site (os_sdlc_lint_bin_test.rb) is converted on main f4cfb8fb.

Work started on branch ticket-1337 (os-sdlc implement pipeline). The last known spawn site (os_sdlc_lint_bin_test.rb) is converted on main f4cfb8fb.
Author
Owner

Blocked by #1226. Implementation 177 escalated (83, behavior-verifier bound). Hand-finished on branch ticket-1337 (714fe599, pushed): AC1-AC4 hold, os-sdlc suite 2750 runs green. AC5 fails on one offense: plugins/os-sdlc/tests/detector/holders_bin_test.rb:56, the Open3 run kept on purpose until #1226 (CopSites reads the target Ruby version from the process cwd; RuboCop memoizes it in-process). No Exclude or inline disable, per house rule. Resume: after #1226, convert holders_bin_test to InProcessBin, run the cop over the repo (0 offenses), one design audit of the branch, merge. The branch also replaces the Minitest/NoTestCases Exclude with Include: [/test//, /tests//] (cop files in lib/ named *_test.rb were false positives); confirm at merge. Body paths corrected to cops/structural/, plugins/os-sdlc/.rubocop.yml, and the sdlc-structural- lesson name.

Blocked by #1226. Implementation 177 escalated (83, behavior-verifier bound). Hand-finished on branch ticket-1337 (714fe599, pushed): AC1-AC4 hold, os-sdlc suite 2750 runs green. AC5 fails on one offense: plugins/os-sdlc/tests/detector/holders_bin_test.rb:56, the Open3 run kept on purpose until #1226 (CopSites reads the target Ruby version from the process cwd; RuboCop memoizes it in-process). No Exclude or inline disable, per house rule. Resume: after #1226, convert holders_bin_test to InProcessBin, run the cop over the repo (0 offenses), one design audit of the branch, merge. The branch also replaces the Minitest/NoTestCases Exclude with Include: [**/test/**/*, **/tests/**/*] (cop files in lib/ named *_test.rb were false positives); confirm at merge. Body paths corrected to cops/structural/, plugins/os-sdlc/.rubocop.yml, and the sdlc-structural- lesson name.
Author
Owner

Removed the stale waiting label and the "Blocked by: #1226" line. The holders change in this ticket (SiteOffenses passes the Ruby version from the target root's .ruby-version to CopSites) removes the dependency on #1226. Implementation starts now with option B (general cop Sdlc/Structural/OwnEntryPointSpawnedInTest).

Removed the stale `waiting` label and the "Blocked by: #1226" line. The holders change in this ticket (`SiteOffenses` passes the Ruby version from the target root's `.ruby-version` to `CopSites`) removes the dependency on #1226. Implementation starts now with option B (general cop `Sdlc/Structural/OwnEntryPointSpawnedInTest`).
Author
Owner

Work started on branch ticket-1337 (fresh from main e12f9297). The old narrow-design branch is kept as ticket-1337-narrow for reference.

Work started on branch `ticket-1337` (fresh from main e12f9297). The old narrow-design branch is kept as `ticket-1337-narrow` for reference.
Author
Owner

Follow-up dropped by ticket-skeptic: SiteOffenses reads .ruby-version, while RuboCop gives a target .rubocop.yml TargetRubyVersion priority. No target has both with different values, so the need is speculative (design-audit minor item, A12).

Follow-up dropped by ticket-skeptic: SiteOffenses reads .ruby-version, while RuboCop gives a target .rubocop.yml TargetRubyVersion priority. No target has both with different values, so the need is speculative (design-audit minor item, A12).
Author
Owner

Resolution

Done: Added cop Sdlc/Structural/OwnEntryPointSpawnedInTest (option B): it reports a test file (name test.rb or test.rb) that starts a process (Open3.*, system, spawn, IO.popen, backticks) on an entry point of this repository (first line #!, or a file in a bin folder), resolved from File.expand_path(..., dir) or a same-file constant. Config key SmokeTestPatterns (default bin_smoke_test.rb, test_bin_smoke.rb); no Exclude list. EntryPoint shared with ExitOutsideEntryPoint; ResolvedPath moved to Structural. ExecutableLoadedInTest message reworded. Holders::SiteOffenses passes the target root's .ruby-version to CopSites; holders_bin_test calls the entry in-process. Two more true hits made in-process: os-backlog checkout_test (env: passed to the CLI) and os-me reply_shape (run.rb is one line calling ReplyShape::EvalHarness; one smoke test). Lesson and smell analysis added. Also Minitest/NoTestCases Include for #1445.

Evidence: main 23b064dc (pushed). os-sdlc suite 2836 runs green in 35 s; os-me 443 and os-backlog 344 runs green. rubocop --only Sdlc/Structural/OwnEntryPointSpawnedInTest plugins/: 2750 files, no offenses. Lint worklist counts for os-sdlc, os-me, os-backlog are at or below main's. Hand-finished after the session-ID collision (#1447); implementation 201 abandoned.

Follow-ups: none (SiteOffenses .ruby-version precedence follow-up was dropped by ticket-skeptic)

## Resolution **Done:** Added cop Sdlc/Structural/OwnEntryPointSpawnedInTest (option B): it reports a test file (name *_test.rb or test_*.rb) that starts a process (Open3.*, system, spawn, IO.popen, backticks) on an entry point of this repository (first line #!, or a file in a bin folder), resolved from File.expand_path(..., __dir__) or a same-file constant. Config key SmokeTestPatterns (default *_bin_smoke_test.rb, test_*_bin_smoke.rb); no Exclude list. EntryPoint shared with ExitOutsideEntryPoint; ResolvedPath moved to Structural. ExecutableLoadedInTest message reworded. Holders::SiteOffenses passes the target root's .ruby-version to CopSites; holders_bin_test calls the entry in-process. Two more true hits made in-process: os-backlog checkout_test (env: passed to the CLI) and os-me reply_shape (run.rb is one line calling ReplyShape::EvalHarness; one smoke test). Lesson and smell analysis added. Also Minitest/NoTestCases Include for #1445. **Evidence:** main 23b064dc (pushed). os-sdlc suite 2836 runs green in 35 s; os-me 443 and os-backlog 344 runs green. rubocop --only Sdlc/Structural/OwnEntryPointSpawnedInTest plugins/: 2750 files, no offenses. Lint worklist counts for os-sdlc, os-me, os-backlog are at or below main's. Hand-finished after the session-ID collision (#1447); implementation 201 abandoned. **Follow-ups:** none (SiteOffenses .ruby-version precedence follow-up was dropped by ticket-skeptic)
jared closed this issue 2026-10-27 22:05:09 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
jared/cc-os#1337
No description provided.