jared/cc-os

Watch

Fork 0

Integration tests stop at the seams: rewrite the widest and slowest os-sdlc tests #1665

New issue

Closed

opened 2026-11-09 22:13:24 +00:00 by jared · 2 comments

jared commented

2026-11-09 22:13:24 +00:00

Owner

Problem

In matrix mode, Mutineer runs every covering test for each mutant. Pipeline tests reach most of lib/, so one mutant in a shared file runs many wide tests. Four tests that scan the real lib/ also run about 10 s or more each. The 10 s limit then stops the mutant before its other tests run. The second pass shows the result: 220 of the 235 timeouts, and 1,449 incomplete matrix rows. The incomplete matrix makes every redundant-test candidate unreliable.

Observed

The four slow tests, each run alone:

InlineDatasetQueryTest#test_passes_the_lib_sources: 9.8 s.
RegistryOutsideEntryPointTest#test_flags_a_model_lookup_outside_initialize_and_passes_the_look_alikes: 12.0 s.
CollaboratorPassAcceptanceTest#test_the_clump_check_has_clumps_to_inspect: 11.9 s.
test/collaborator_pass/command/measure_test.rb: 11.8 s for 6 tests.

Slow covering sets, which together run 5 s to 28 s unmutated: system_model/extractor/* (SystemModelClassNewDeclarationTest has one 3.6 s test), cops/structural/precedence_in_edge/* (8 files, 145 tests, 12.1 s), cops/structural/edge_scope.rb, project/config/document.rb (138 files, 26.6 s), and boot.rb (107 files, 27.9 s).

docs/research/mutation-testing/2026-11-09-second-pass/coverage_width.tsv ranks the 619 test files by the number of lib/ files each covers. The widest test, test/system_model_opportunities_cli_test.rb, covers 181 files. The ranking uses the pass 1 coverage map, because a reproduction run overwrote the pass 2 map.

Reproduce

n/a: the data comes from a whole pass. See docs/research/mutation-testing/2026-11-09-second-pass/README.md, sections "Timeouts" and "Coverage width".

Approach

Jared ruled on 2026-11-09: an integration test checks only the wiring. It checks that the entry builds the right objects, and that each message crosses each seam with the right arguments. Below the first layer it uses doubles or minimal data. Branch logic moves to paired unit tests.

Required work:

Rank the tests by coverage width from coverage_width.tsv, and rewrite the four slow tests first, then the widest tests, as wiring-only tests. Move each branch they covered to a paired unit test at test/os_sdlc/<lib path>_test.rb (ruling 100).
Run /os-sdlc:smell-analysis on the shape "wide integration test", and keep its decision memo.
Add an integration-test rule to plugins/os-sdlc/skills/lens-testing/SKILL.md — check: present integration test in plugins/os-sdlc/skills/lens-testing/SKILL.md
Add a Prism or RuboCop check for the shape only if a run over the ranked list gives zero false positives. Otherwise add none and record the run result in the memo.

Verify each rewrite with a targeted --no-matrix run of the files it covers. Run no whole pass.

Acceptance criteria

Given each of the four slow tests, when it runs alone, then it takes less than 1 s.
Given the top 10 tests by coverage width, when each runs, then it covers only its entry and its first-layer seams, or a ledger row explains why it does not.
Given a branch that a rewritten test used to cover, when a mutant changes that branch, then a paired unit test kills it.

Owner sketch

n/a: test and skill changes only.

Non-goals

The redundant-test review is ticket B. This ticket runs no whole pass.

Origin

Trigger: second whole-codebase mutation pass, 2026-11-09 (docs/research/mutation-testing/2026-11-09-second-pass/)
Improvised this session: none
Chain: timeouts and incomplete matrix rows ← slow and wide covering tests ← integration tests that run real lib/ code below the first seam ← DESIGN (no integration-test rule in lens-testing)
Root candidate: none found; searched integration test, slow test, wide test, coverage_width, lens-testing, Mutineer, mutation pass, timeouts (nearest: #1341, #1644, different faults)
Where: plugins/os-sdlc/test/, plugins/os-sdlc/skills/lens-testing/SKILL.md
Session: bc5ad309-36b7-4fe5-a67f-1d90187242eb
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os-plugins-os-sdlc/bc5ad309-36b7-4fe5-a67f-1d90187242eb.jsonl

VERDICT: CREATE (ticket-skeptic, 2026-11-09): no open ticket has the same mechanism; #1341 and #1644 are different faults.

## Problem In matrix mode, Mutineer runs every covering test for each mutant. Pipeline tests reach most of `lib/`, so one mutant in a shared file runs many wide tests. Four tests that scan the real `lib/` also run about 10 s or more each. The 10 s limit then stops the mutant before its other tests run. The second pass shows the result: 220 of the 235 timeouts, and 1,449 incomplete matrix rows. The incomplete matrix makes every redundant-test candidate unreliable. ## Observed The four slow tests, each run alone: 1. `InlineDatasetQueryTest#test_passes_the_lib_sources`: 9.8 s. 2. `RegistryOutsideEntryPointTest#test_flags_a_model_lookup_outside_initialize_and_passes_the_look_alikes`: 12.0 s. 3. `CollaboratorPassAcceptanceTest#test_the_clump_check_has_clumps_to_inspect`: 11.9 s. 4. `test/collaborator_pass/command/measure_test.rb`: 11.8 s for 6 tests. Slow covering sets, which together run 5 s to 28 s unmutated: `system_model/extractor/*` (`SystemModelClassNewDeclarationTest` has one 3.6 s test), `cops/structural/precedence_in_edge/*` (8 files, 145 tests, 12.1 s), `cops/structural/edge_scope.rb`, `project/config/document.rb` (138 files, 26.6 s), and `boot.rb` (107 files, 27.9 s). `docs/research/mutation-testing/2026-11-09-second-pass/coverage_width.tsv` ranks the 619 test files by the number of `lib/` files each covers. The widest test, `test/system_model_opportunities_cli_test.rb`, covers 181 files. The ranking uses the pass 1 coverage map, because a reproduction run overwrote the pass 2 map. ## Reproduce n/a: the data comes from a whole pass. See `docs/research/mutation-testing/2026-11-09-second-pass/README.md`, sections "Timeouts" and "Coverage width". ## Approach Jared ruled on 2026-11-09: an integration test checks only the wiring. It checks that the entry builds the right objects, and that each message crosses each seam with the right arguments. Below the first layer it uses doubles or minimal data. Branch logic moves to paired unit tests. Required work: 1. Rank the tests by coverage width from `coverage_width.tsv`, and rewrite the four slow tests first, then the widest tests, as wiring-only tests. Move each branch they covered to a paired unit test at `test/os_sdlc/<lib path>_test.rb` (ruling 100). 2. Run `/os-sdlc:smell-analysis` on the shape "wide integration test", and keep its decision memo. 3. Add an integration-test rule to `plugins/os-sdlc/skills/lens-testing/SKILL.md` — check: present `integration test` in `plugins/os-sdlc/skills/lens-testing/SKILL.md` 4. Add a Prism or RuboCop check for the shape only if a run over the ranked list gives zero false positives. Otherwise add none and record the run result in the memo. Verify each rewrite with a targeted `--no-matrix` run of the files it covers. Run no whole pass. ## Acceptance criteria 1. Given each of the four slow tests, when it runs alone, then it takes less than 1 s. 2. Given the top 10 tests by coverage width, when each runs, then it covers only its entry and its first-layer seams, or a ledger row explains why it does not. 3. Given a branch that a rewritten test used to cover, when a mutant changes that branch, then a paired unit test kills it. ## Owner sketch n/a: test and skill changes only. ## Non-goals The redundant-test review is ticket B. This ticket runs no whole pass. ## Origin - Trigger: second whole-codebase mutation pass, 2026-11-09 (`docs/research/mutation-testing/2026-11-09-second-pass/`) - Improvised this session: none - Chain: timeouts and incomplete matrix rows ← slow and wide covering tests ← integration tests that run real `lib/` code below the first seam ← DESIGN (no integration-test rule in lens-testing) - Root candidate: none found; searched integration test, slow test, wide test, coverage_width, lens-testing, Mutineer, mutation pass, timeouts (nearest: #1341, #1644, different faults) - Where: `plugins/os-sdlc/test/`, `plugins/os-sdlc/skills/lens-testing/SKILL.md` - Session: bc5ad309-36b7-4fe5-a67f-1d90187242eb - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os-plugins-os-sdlc/bc5ad309-36b7-4fe5-a67f-1d90187242eb.jsonl VERDICT: CREATE (ticket-skeptic, 2026-11-09): no open ticket has the same mechanism; #1341 and #1644 are different faults.

jared added the

plugin/os-sdlc

filed-by/user

category/code-change

labels

2026-11-09 22:13:24 +00:00

jared referenced this issue

2026-11-09 22:13:28 +00:00

Redundant-test review from a complete matrix #1666

jared commented

2026-11-10 14:54:01 +00:00

Author

Owner

Work started in session 720a7477, branch ticket-1665. Order: the 4 slow tests first (delete the two cop lib scans; fixture codebase for measure_test and CollaboratorPassAcceptanceTest), then the wide tests.

jared referenced this issue

2026-11-10 14:58:13 +00:00

CopSource resolves the target Ruby version once for a run, not once for each file #1670

jared referenced this issue

2026-11-10 15:19:22 +00:00

Redundant-test review from a complete matrix #1666

jared referenced this issue

2026-11-10 15:19:49 +00:00

Make the os-sdlc suite pass without bundler and stop ProcessStartGuard hiding assertion failures #1669

jared commented

2026-11-10 15:19:50 +00:00

Author

Owner

Resolution

Done: The four slow tests and one more (name_collision_findings_test) no longer read real lib/: the two cop scans are deleted and three collaborator-pass tests run over fixture codebases. The widest tests keep one journey for each entry (cli_next_instruction 25 to 1, cli_subagent_stop 22 to 1, crc_boot_entry 29 to 9, five pipeline journey files deleted, system-model CLI tests stop at Run); their branch checks are paired unit tests. lens-testing has T10 (integration test checks the wiring once for each entry) and T11 (real code is not test input). Smell analysis and decision memo are in the record. No cop: the probe over the tree gave false positives, recorded in the memo. No production code changed.

Evidence: Branch ticket-1665, commits 040e3a975 fca661195 7ee0bbaac 3ca1be08b dada64d09 6ae29f123 39f16222f d0b358eb1 f8ee98236 5be10430f; not merged yet. Record: docs/research/mutation-testing/2026-11-10-integration-tests/README.md. Suite 3,909 runs, 0 failures, 6.5 s. AC1: slow test files run in 0.11 to 0.46 s test time (about 1.0 to 1.4 s wall with start-up). AC2: each top-10 file keeps one journey for each entry or none. AC3: targeted --no-matrix runs for each new unit test, e.g. failure_directive 7/7, terminal_payload 9/9, escalation_reason 12/12, gate/run 31/31, hook_reading 7/7, project_bind 7/7, next_step 36/39. Targeted run on the 79 files with pass 2 timeouts: 226 timeouts become 42; 38 of the 42 come from worker load (0 of 9 sampled at --jobs 3).

Follow-ups: #1670 filed (CopSource makes a ConfigStore for each file). gdiff under the process guard: occurrence posted on #1669. Worker-load timeouts and the 4 covering-set timeouts (boot.rb:18, project config): evidence posted on #1666; decision memo option A recommends no more narrowing. Mutineer makes no mutants for a lambda held in a constant: reported to Jared, not filed (Mutineer is another project). Unit-test layout (test/implementation vs test/os_sdlc): owned by #1644, none new.

## Resolution **Done:** The four slow tests and one more (name_collision_findings_test) no longer read real lib/: the two cop scans are deleted and three collaborator-pass tests run over fixture codebases. The widest tests keep one journey for each entry (cli_next_instruction 25 to 1, cli_subagent_stop 22 to 1, crc_boot_entry 29 to 9, five pipeline journey files deleted, system-model CLI tests stop at Run); their branch checks are paired unit tests. lens-testing has T10 (integration test checks the wiring once for each entry) and T11 (real code is not test input). Smell analysis and decision memo are in the record. No cop: the probe over the tree gave false positives, recorded in the memo. No production code changed. **Evidence:** Branch ticket-1665, commits 040e3a975 fca661195 7ee0bbaac 3ca1be08b dada64d09 6ae29f123 39f16222f d0b358eb1 f8ee98236 5be10430f; not merged yet. Record: docs/research/mutation-testing/2026-11-10-integration-tests/README.md. Suite 3,909 runs, 0 failures, 6.5 s. AC1: slow test files run in 0.11 to 0.46 s test time (about 1.0 to 1.4 s wall with start-up). AC2: each top-10 file keeps one journey for each entry or none. AC3: targeted --no-matrix runs for each new unit test, e.g. failure_directive 7/7, terminal_payload 9/9, escalation_reason 12/12, gate/run 31/31, hook_reading 7/7, project_bind 7/7, next_step 36/39. Targeted run on the 79 files with pass 2 timeouts: 226 timeouts become 42; 38 of the 42 come from worker load (0 of 9 sampled at --jobs 3). **Follow-ups:** #1670 filed (CopSource makes a ConfigStore for each file). gdiff under the process guard: occurrence posted on #1669. Worker-load timeouts and the 4 covering-set timeouts (boot.rb:18, project config): evidence posted on #1666; decision memo option A recommends no more narrowing. Mutineer makes no mutants for a lambda held in a constant: reported to Jared, not filed (Mutineer is another project). Unit-test layout (test/implementation vs test/os_sdlc): owned by #1644, none new.

jared closed this issue

2026-11-10 15:19:50 +00:00

jared referenced this issue from a commit

2026-11-10 15:39:50 +00:00

test(os-sdlc): the two Sequel cop tests no longer scan lib; the RuboCop gate runs both cops over lib (#1665)

jared referenced this issue from a commit

2026-11-10 15:39:50 +00:00

test(os-sdlc): the collaborator-pass measure and acceptance tests run over a fixture codebase of five classes, not the real repository (#1665)

jared referenced this issue from a commit

2026-11-10 15:39:50 +00:00

test(os-sdlc): the name-collision findings test runs over a fixture codebase with one collision and one near miss (#1665)

jared referenced this issue from a commit