jared/cc-os

Watch

Fork 0

Redundant-test review from a complete matrix #1666

New issue

Closed

opened 2026-11-09 22:13:28 +00:00 by jared · 2 comments

jared commented

2026-11-09 22:13:28 +00:00

Owner

Problem

The second mutation pass lists 2,735 redundant-test candidates in overlap.tsv and report.tsv. They are not reliable. The matrix is incomplete (1,705 rows stopped early). A test that looks redundant can be the only killer of a mutant whose row stopped early. This ticket waits for #1665.

Observed

overlap.tsv holds 5,025 shared-kill pairs, 5,294 sole-foreign-killer rows, and 567 many-killer rows. The files with most candidates are cops/repeated_collaborator_construction_test.rb (50), cops/domain_effect_in_edge_test.rb (43), cops/precedence_in_edge_test.rb (41), crc/tickets_bin_test.rb (35), and system_model_catalog_test.rb (34).

Reproduce

n/a: the data comes from a whole pass.

Approach

After #1665, make one matrix run scoped to the directories that the candidates affect. Run no whole pass. Rebuild the candidate list from that run. For each candidate group, delete or merge the redundant tests, and record a ruling per group.

Required work: none beyond the acceptance criteria.

Acceptance criteria

Given the candidate list from the scoped matrix run, when this ticket closes, then each candidate group has a ruling or a deletion commit.
Given a deleted test, when the scoped matrix run repeats, then no mutant that the test killed becomes a survivor.

Blocked by

#1665 (integration tests stop at the seams).

Origin

Trigger: second whole-codebase mutation pass, 2026-11-09 (docs/research/mutation-testing/2026-11-09-second-pass/)
Improvised this session: none
Chain: unreliable redundancy candidates ← incomplete matrix ← slow covering tests ← DESIGN (see the blocking ticket)
Root candidate: none found; searched redundant, overlap.tsv, shared-kill, matrix, mutation (nearest: #1644, a different fault)
Where: plugins/os-sdlc/test/, docs/research/mutation-testing/2026-11-09-second-pass/overlap.tsv
Session: bc5ad309-36b7-4fe5-a67f-1d90187242eb
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os-plugins-os-sdlc/bc5ad309-36b7-4fe5-a67f-1d90187242eb.jsonl

VERDICT: CREATE (ticket-skeptic, 2026-11-09): no open ticket covers redundant-test deletion from a complete matrix.

## Problem The second mutation pass lists 2,735 redundant-test candidates in `overlap.tsv` and `report.tsv`. They are not reliable. The matrix is incomplete (1,705 rows stopped early). A test that looks redundant can be the only killer of a mutant whose row stopped early. This ticket waits for #1665. ## Observed `overlap.tsv` holds 5,025 shared-kill pairs, 5,294 sole-foreign-killer rows, and 567 many-killer rows. The files with most candidates are `cops/repeated_collaborator_construction_test.rb` (50), `cops/domain_effect_in_edge_test.rb` (43), `cops/precedence_in_edge_test.rb` (41), `crc/tickets_bin_test.rb` (35), and `system_model_catalog_test.rb` (34). ## Reproduce n/a: the data comes from a whole pass. ## Approach After #1665, make one matrix run scoped to the directories that the candidates affect. Run no whole pass. Rebuild the candidate list from that run. For each candidate group, delete or merge the redundant tests, and record a ruling per group. Required work: none beyond the acceptance criteria. ## Acceptance criteria 1. Given the candidate list from the scoped matrix run, when this ticket closes, then each candidate group has a ruling or a deletion commit. 2. Given a deleted test, when the scoped matrix run repeats, then no mutant that the test killed becomes a survivor. ## Blocked by #1665 (integration tests stop at the seams). ## Origin - Trigger: second whole-codebase mutation pass, 2026-11-09 (`docs/research/mutation-testing/2026-11-09-second-pass/`) - Improvised this session: none - Chain: unreliable redundancy candidates ← incomplete matrix ← slow covering tests ← DESIGN (see the blocking ticket) - Root candidate: none found; searched redundant, overlap.tsv, shared-kill, matrix, mutation (nearest: #1644, a different fault) - Where: `plugins/os-sdlc/test/`, `docs/research/mutation-testing/2026-11-09-second-pass/overlap.tsv` - Session: bc5ad309-36b7-4fe5-a67f-1d90187242eb - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os-plugins-os-sdlc/bc5ad309-36b7-4fe5-a67f-1d90187242eb.jsonl VERDICT: CREATE (ticket-skeptic, 2026-11-09): no open ticket covers redundant-test deletion from a complete matrix.

jared added the

plugin/os-sdlc

filed-by/user

category/code-change

labels

2026-11-09 22:13:28 +00:00

jared commented

2026-11-10 15:19:22 +00:00

Author

Owner

Evidence from #1665 (branch ticket-1665, record docs/research/mutation-testing/2026-11-10-integration-tests/README.md), for the scoped matrix run:

A targeted --no-matrix run on the 79 lib files with pass 2 timeouts gave 42 timeouts (pass 2: 226, hangs not counted), 1,373 mutants, 12 workers on 12 processors, load 9 to 12.
38 of the 42 come from worker load. A rerun of three of those files with --jobs 3 gave 71 mutants and 0 timeouts; the same files gave 9 timeouts at 12 workers.
4 come from a wide covering set: boot.rb:18 (98 test files call Boot.run), project/config/document.rb:14 and :78, project/target.rb:7 (143 to 147 test files load a project configuration).
The worker count for the matrix run is an open decision for Jared.

Evidence from #1665 (branch ticket-1665, record docs/research/mutation-testing/2026-11-10-integration-tests/README.md), for the scoped matrix run: 1. A targeted --no-matrix run on the 79 lib files with pass 2 timeouts gave 42 timeouts (pass 2: 226, hangs not counted), 1,373 mutants, 12 workers on 12 processors, load 9 to 12. 2. 38 of the 42 come from worker load. A rerun of three of those files with --jobs 3 gave 71 mutants and 0 timeouts; the same files gave 9 timeouts at 12 workers. 3. 4 come from a wide covering set: boot.rb:18 (98 test files call Boot.run), project/config/document.rb:14 and :78, project/target.rb:7 (143 to 147 test files load a project configuration). The worker count for the matrix run is an open decision for Jared.

jared referenced this issue

2026-11-10 15:19:50 +00:00

Integration tests stop at the seams: rewrite the widest and slowest os-sdlc tests #1665

jared commented

2026-11-10 16:49:20 +00:00

Author

Owner

Resolution

Done: Scoped matrix runs per candidate group (the candidates touch 930 of 1,001 lib files, so a directory scope is a whole pass). Every candidate group has a ruling: 117 keeps the cop, detector, and lint example tests; 118 deletes or merges a behavior test only when another test makes the same assertion at the same seam; 119 keeps the other candidates after a sample of 85 pairs gave 1 merge. One merge in test/crc/tickets_bin_test.rb.

Evidence: Commit 667c8de81 on ticket-1666. Record docs/research/mutation-testing/2026-11-10-redundant-test-review/README.md. AC 2: the removed test killed 135 mutants in 23 lib files; after the merge 0 survive (targeted --no-matrix run, --jobs 8). tickets_bin_test.rb: 45 runs, 0 failures.

Follow-ups: none: the 69 survivors in the g2 run sit in crc and boot lib files, which #1660 and #1664 own.

## Resolution **Done:** Scoped matrix runs per candidate group (the candidates touch 930 of 1,001 lib files, so a directory scope is a whole pass). Every candidate group has a ruling: 117 keeps the cop, detector, and lint example tests; 118 deletes or merges a behavior test only when another test makes the same assertion at the same seam; 119 keeps the other candidates after a sample of 85 pairs gave 1 merge. One merge in test/crc/tickets_bin_test.rb. **Evidence:** Commit 667c8de81 on ticket-1666. Record docs/research/mutation-testing/2026-11-10-redundant-test-review/README.md. AC 2: the removed test killed 135 mutants in 23 lib files; after the merge 0 survive (targeted --no-matrix run, --jobs 8). tickets_bin_test.rb: 45 runs, 0 failures. **Follow-ups:** none: the 69 survivors in the g2 run sit in crc and boot lib files, which #1660 and #1664 own.

jared closed this issue

2026-11-10 16:49:20 +00:00

jared referenced this issue from a commit

2026-11-10 16:50:28 +00:00

test(os-sdlc): #1666 redundant-test review: one merge in tickets_bin_test, rulings 117-119 keep the other candidates

jared referenced this issue

2026-11-10 16:51:37 +00:00

Mutation runs write CRC bin test fixtures into the real plugin tree #1671