jared/cc-os

Watch

Fork 0

Sdlc/Minitest/SeedSweepAssertion: stay wide, but skip assertions inside a define_method loop #1371

New issue

Closed

opened 2026-10-25 21:23:23 +00:00 by jared · 2 comments

jared commented

2026-10-25 21:23:23 +00:00

Owner

Problem

Rewritten 2026-10-27 (fast-batch-cop-matchers): added the message change of analysis option A (required work 3, acceptance criterion 5).

Sdlc/Minitest/SeedSweepAssertion flags an assertion inside a times or each block. Jared ruled on 2026-10-27 that the cop stays wide: an assertion inside a loop over fixed data is the smell too, because one test runs many cases and a failure does not name the case. The cop has one false positive: a loop that calls define_method defines one test for each element. Each test has its own name, so a failure names the case. The cop flags the assertion inside the defined method because the each block encloses it.

Observed

# plugins/os-me/test/extract_moment_kinds_test.rb:8
OsMe::MomentKind::KINDS.each do |kind|
  define_method("test_a_#{kind}_pair_is_stored_as_one_#{kind}_moment_with_no_label_fields") do
    ...
    assert_equal([...], rows.map { ... })   # flagged; each kind is its own test
  end
end

The original ticket asked to flag only loops that use randomness. That scope is overruled: the 17 hits on fixed data stay true positives.

Reproduce

class KindsTest < Minitest::Test
  %w[a b].each do |name|
    define_method("test_#{name}") { assert_equal name, name.downcase }
  end
end

Save as kinds_test.rb and run rubocop --only Sdlc/Minitest/SeedSweepAssertion: it reports an offense at the assertion.

Approach

Fix by shape in the matcher, never in a config Exclude list or a test-path exclusion. An assertion whose nearest enclosing block is a define_method (or define_singleton_method) call is not inside a sweep: stop the walk up the ancestors at that block.

Required work:

Stop the enclosing-loop search at a define_method block — check: present define_method in plugins/os-sdlc/lib/os_sdlc/cops/minitest/seed_sweep_assertion/assertion_candidate.rb
Say in the lesson that a define_method loop is not the smell — check: present define_method in plugins/os-sdlc/lint-lessons/*seed-sweep*.yml
Change the message so it names the wide smell (many cases in one test body; a failure does not name the case), not only randomness (analysis option A, docs/lint-lessons/analyses/sdlc-minitest-seed-sweep-assertion.md section 8) — check: absent RNG/iteration internals in plugins/os-sdlc/lib/os_sdlc/cops/minitest/seed_sweep_assertion.rb

Rejected: flag only loops that use randomness — Jared ruled 2026-10-27 that the cop stays wide; loops over fixed data are true positives.
Rejected: exclude the test file — a file list is a baseline.

Acceptance criteria

Given %w[a b].each { |n| define_method("test_#{n}") { assert_equal n, n.downcase } } in a _test.rb file, when the cop runs, then it reports no offense.
Given %w[a b].each { |n| assert_equal n, n.downcase } inside a test method, when the cop runs, then it reports an offense.
Given 5.times { assert_operator rand(3), :<, 3 } inside a test method, when the cop runs, then it reports an offense.
Given a define_method block whose body has its own each loop with an assertion inside it, when the cop runs, then it reports an offense for that inner loop.
Given %w[a b].each { |n| assert_equal n, n.downcase } inside a test method, when the cop runs, then the message says that many cases run in one test and names the corrections (one assertion per case, one assertion over the set, or a fixed seed).

Owner sketch

class RuboCop::Cop::Sdlc::Minitest::SeedSweepAssertion
  def on_send(node)
end

Origin

Trigger: #1325 zero-false-positive audit; rescoped 2026-10-27 by the lint-cops track (ruling: #1371 stays wide, rewrite AC) after fix-lints on os-me hit the define_method shape.
Improvised this session: none
Chain: false positive on a define_method loop ← enclosing-loop search walks past the define_method block ← DESIGN (plugins/os-sdlc/lib/os_sdlc/cops/minitest/seed_sweep_assertion/assertion_candidate.rb)
Root candidate: none found; searched SeedSweep, seed sweep, SeedSweepAssertion, define_method
Where: plugins/os-sdlc/lib/os_sdlc/cops/minitest/seed_sweep_assertion/assertion_candidate.rb
Session: 0b4d7950-df1c-4eba-a561-ccc19b386a9a
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-ticket-1424/0b4d7950-df1c-4eba-a561-ccc19b386a9a.jsonl

VERDICT: CREATE

## Problem Rewritten 2026-10-27 (fast-batch-cop-matchers): added the message change of analysis option A (required work 3, acceptance criterion 5). `Sdlc/Minitest/SeedSweepAssertion` flags an assertion inside a `times` or `each` block. Jared ruled on 2026-10-27 that the cop stays wide: an assertion inside a loop over fixed data is the smell too, because one test runs many cases and a failure does not name the case. The cop has one false positive: a loop that calls `define_method` defines one test for each element. Each test has its own name, so a failure names the case. The cop flags the assertion inside the defined method because the `each` block encloses it. ## Observed ```ruby # plugins/os-me/test/extract_moment_kinds_test.rb:8 OsMe::MomentKind::KINDS.each do |kind| define_method("test_a_#{kind}_pair_is_stored_as_one_#{kind}_moment_with_no_label_fields") do ... assert_equal([...], rows.map { ... }) # flagged; each kind is its own test end end ``` The original ticket asked to flag only loops that use randomness. That scope is overruled: the 17 hits on fixed data stay true positives. ## Reproduce ```ruby class KindsTest < Minitest::Test %w[a b].each do |name| define_method("test_#{name}") { assert_equal name, name.downcase } end end ``` Save as `kinds_test.rb` and run `rubocop --only Sdlc/Minitest/SeedSweepAssertion`: it reports an offense at the assertion. ## Approach Fix by shape in the matcher, never in a config Exclude list or a test-path exclusion. An assertion whose nearest enclosing block is a `define_method` (or `define_singleton_method`) call is not inside a sweep: stop the walk up the ancestors at that block. Required work: 1. Stop the enclosing-loop search at a `define_method` block — check: present `define_method` in `plugins/os-sdlc/lib/os_sdlc/cops/minitest/seed_sweep_assertion/assertion_candidate.rb` 2. Say in the lesson that a `define_method` loop is not the smell — check: present `define_method` in `plugins/os-sdlc/lint-lessons/*seed-sweep*.yml` 3. Change the message so it names the wide smell (many cases in one test body; a failure does not name the case), not only randomness (analysis option A, `docs/lint-lessons/analyses/sdlc-minitest-seed-sweep-assertion.md` section 8) — check: absent `RNG/iteration internals` in `plugins/os-sdlc/lib/os_sdlc/cops/minitest/seed_sweep_assertion.rb` Rejected: flag only loops that use randomness — Jared ruled 2026-10-27 that the cop stays wide; loops over fixed data are true positives. Rejected: exclude the test file — a file list is a baseline. ## Acceptance criteria 1. Given `%w[a b].each { |n| define_method("test_#{n}") { assert_equal n, n.downcase } }` in a `_test.rb` file, when the cop runs, then it reports no offense. 2. Given `%w[a b].each { |n| assert_equal n, n.downcase }` inside a test method, when the cop runs, then it reports an offense. 3. Given `5.times { assert_operator rand(3), :<, 3 }` inside a test method, when the cop runs, then it reports an offense. 4. Given a `define_method` block whose body has its own `each` loop with an assertion inside it, when the cop runs, then it reports an offense for that inner loop. 5. Given `%w[a b].each { |n| assert_equal n, n.downcase }` inside a test method, when the cop runs, then the message says that many cases run in one test and names the corrections (one assertion per case, one assertion over the set, or a fixed seed). ## Owner sketch ```ruby class RuboCop::Cop::Sdlc::Minitest::SeedSweepAssertion def on_send(node) end ``` ## Origin - Trigger: #1325 zero-false-positive audit; rescoped 2026-10-27 by the lint-cops track (ruling: #1371 stays wide, rewrite AC) after fix-lints on os-me hit the `define_method` shape. - Improvised this session: none - Chain: false positive on a define_method loop ← enclosing-loop search walks past the define_method block ← DESIGN (plugins/os-sdlc/lib/os_sdlc/cops/minitest/seed_sweep_assertion/assertion_candidate.rb) - Root candidate: none found; searched SeedSweep, seed sweep, SeedSweepAssertion, define_method - Where: plugins/os-sdlc/lib/os_sdlc/cops/minitest/seed_sweep_assertion/assertion_candidate.rb - Session: 0b4d7950-df1c-4eba-a561-ccc19b386a9a - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-ticket-1424/0b4d7950-df1c-4eba-a561-ccc19b386a9a.jsonl VERDICT: CREATE

jared added the

plugin/os-sdlc

filed-by/agent

category/code-change

labels

2026-10-25 21:23:23 +00:00

jared changed title from ~~Sdlc/Minitest/SeedSweepAssertion: flag only loops that involve randomness~~ to Sdlc/Minitest/SeedSweepAssertion: stay wide, but skip assertions inside a define_method loop

2026-10-27 17:25:18 +00:00

jared commented

2026-10-27 21:31:26 +00:00

Author

Owner

2026-10-27: work started in fast-batch-cop-matchers (AFK). Ticket confirmed valid on main a02b9458.

2026-10-27: work started in fast-batch-cop-matchers (AFK). Ticket confirmed valid on main a02b9458.

jared referenced this issue from a commit

2026-10-27 21:52:59 +00:00

fix(os-sdlc): SeedSweepAssertion stops at a define_method block and names the many-cases smell (#1371)

jared referenced this issue from a commit

2026-10-27 21:52:59 +00:00

docs(os-sdlc): fast-batch-cop-matchers state and History row (#1346 #1358 #1371)

jared commented

2026-10-27 21:53:11 +00:00

Author

Owner

Resolution

Done: SeedSweepAssertion loop search stops at the first define_method/define_singleton_method block; message names many cases in one test; lesson decision names define_method. Commit 1e9c1b2d (merged f5310f2b). Ticket rewritten to add the option A message change (AC 5); predicted row J1 (86%).

Evidence: AC1-AC5 fixtures in plugins/os-sdlc/tests/cops/seed_sweep_assertion_test.rb; plugins/os-me/test/extract_moment_kinds_test.rb now 0 offenses; suite 2825 runs 0 failures.

Follow-ups: Cop key SeedSweepAssertion no longer describes the wide rule; a rename is a separate change for Jared (cop key is parsed by configs and lessons).

## Resolution **Done:** SeedSweepAssertion loop search stops at the first define_method/define_singleton_method block; message names many cases in one test; lesson decision names define_method. Commit 1e9c1b2d (merged f5310f2b). Ticket rewritten to add the option A message change (AC 5); predicted row J1 (86%). **Evidence:** AC1-AC5 fixtures in plugins/os-sdlc/tests/cops/seed_sweep_assertion_test.rb; plugins/os-me/test/extract_moment_kinds_test.rb now 0 offenses; suite 2825 runs 0 failures. **Follow-ups:** Cop key SeedSweepAssertion no longer describes the wide rule; a rename is a separate change for Jared (cop key is parsed by configs and lessons).

jared closed this issue

2026-10-27 21:53:11 +00:00