jared/cc-os

Watch

Fork 0

Make the mutation gate refuse a mutated run that executed no tests #893

New issue

Closed

opened 2026-10-01 20:48:58 +00:00 by jared · 4 comments

jared commented

2026-10-01 20:48:58 +00:00

Owner

Problem

The mutation gate passes when the mutated test run exits nonzero for any reason. Mutation::Check#survived? (plugins/os-sdlc/lib/os_sdlc/adapter/mutation/check.rb) returns status.success? from the diff-test command. #875 says the check exits 0 only when the test run is red, "executed and failed". A patch that makes the production file fail to load (a syntax error, a missing constant) makes the run exit nonzero before any guard test executes. The gate then accepts a guard test that guards nothing.

Reproduce

n/a: found by reading the #875 diff (commit 461ff366); no run used it yet.

Approach

Required work: none beyond the acceptance criteria.

Do not add a new output parser or exit-code heuristic. The red-versus-broken rule already has one owner: Gate::StepOutcome#classify(expect_red: true) (plugins/os-sdlc/lib/os_sdlc/model/gate/step_outcome.rb), which uses TestReport#summary and Summary#executed_failure?. The test-writer red gate uses it. Mutation::Check captures the diff-test output and exit status, builds a Gate::StepOutcome for the diff-test step, and sends classify(expect_red: true):

:red passes (exit 0).
:unexpected_pass fails with the existing "mutation survived" line.
:broken fails with a line that says the mutated run executed no tests (load error or no test summary), naming the mutation path.

The ensure restore stays as it is. No map, ADR, or lens change: the guard map fail edge already routes to test-writer, which owns the patch.

Acceptance criteria

Given a test-writer handoff whose mutation patch makes the production file fail to load, when os-sdlc-mutation-check runs, then it exits nonzero, the output says the mutated run did not execute the tests, and the production file is restored.
Given a mutation patch that breaks the guarded behavior so that at least one test runs and fails, when the check runs, then it exits 0.

Owner sketch

class OsSdlc::Mutation::Check
  def report
  # private: outcome classifies the mutated run with Gate::StepOutcome#classify(expect_red: true)
end

Tests: one new case in plugins/os-sdlc/tests/controller/boot_mutation_check_test.rb with a load-breaking patch constant in plugins/os-sdlc/tests/support/mutation_check_fixture.rb. AC2 is the existing breaking-patch test.

Origin

Trigger: orchestrator diff review of #875 before merge, 2026-10-01
Improvised this session: none
Chain: a no-op guard test can pass the mutation gate ← survived? reads only the exit status ← DESIGN (plugins/os-sdlc/lib/os_sdlc/adapter/mutation/check.rb, commit 461ff366)
Root candidate: #859
Where: plugins/os-sdlc/lib/os_sdlc/adapter/mutation/check.rb
Session: 8cb89545-668f-4061-9d99-470dab1af38f
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/8cb89545-668f-4061-9d99-470dab1af38f.jsonl

Parent: #859

VERDICT: CREATE — survived? reads only the exit status; a load failure passes the gate (ticket-skeptic, claude-sonnet-5-5).

## Problem The mutation gate passes when the mutated test run exits nonzero for any reason. `Mutation::Check#survived?` (plugins/os-sdlc/lib/os_sdlc/adapter/mutation/check.rb) returns `status.success?` from the diff-test command. #875 says the check exits 0 only when the test run is red, "executed and failed". A patch that makes the production file fail to load (a syntax error, a missing constant) makes the run exit nonzero before any guard test executes. The gate then accepts a guard test that guards nothing. ## Reproduce n/a: found by reading the #875 diff (commit 461ff366); no run used it yet. ## Approach Required work: none beyond the acceptance criteria. Do not add a new output parser or exit-code heuristic. The red-versus-broken rule already has one owner: `Gate::StepOutcome#classify(expect_red: true)` (plugins/os-sdlc/lib/os_sdlc/model/gate/step_outcome.rb), which uses `TestReport#summary` and `Summary#executed_failure?`. The test-writer red gate uses it. `Mutation::Check` captures the diff-test output and exit status, builds a `Gate::StepOutcome` for the `diff-test` step, and sends `classify(expect_red: true)`: - `:red` passes (exit 0). - `:unexpected_pass` fails with the existing "mutation survived" line. - `:broken` fails with a line that says the mutated run executed no tests (load error or no test summary), naming the mutation path. The `ensure` restore stays as it is. No map, ADR, or lens change: the guard map `fail` edge already routes to test-writer, which owns the patch. ## Acceptance criteria 1. Given a test-writer handoff whose `mutation` patch makes the production file fail to load, when `os-sdlc-mutation-check` runs, then it exits nonzero, the output says the mutated run did not execute the tests, and the production file is restored. 2. Given a mutation patch that breaks the guarded behavior so that at least one test runs and fails, when the check runs, then it exits 0. ## Owner sketch ```ruby class OsSdlc::Mutation::Check def report # private: outcome classifies the mutated run with Gate::StepOutcome#classify(expect_red: true) end ``` Tests: one new case in plugins/os-sdlc/tests/controller/boot_mutation_check_test.rb with a load-breaking patch constant in plugins/os-sdlc/tests/support/mutation_check_fixture.rb. AC2 is the existing breaking-patch test. ## Origin - Trigger: orchestrator diff review of #875 before merge, 2026-10-01 - Improvised this session: none - Chain: a no-op guard test can pass the mutation gate ← `survived?` reads only the exit status ← DESIGN (plugins/os-sdlc/lib/os_sdlc/adapter/mutation/check.rb, commit 461ff366) - Root candidate: #859 - Where: plugins/os-sdlc/lib/os_sdlc/adapter/mutation/check.rb - Session: 8cb89545-668f-4061-9d99-470dab1af38f - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/8cb89545-668f-4061-9d99-470dab1af38f.jsonl Parent: #859 VERDICT: CREATE — `survived?` reads only the exit status; a load failure passes the gate (ticket-skeptic, claude-sonnet-5-5).

jared added the

plugin/os-sdlc

filed-by/agent

labels

2026-10-01 20:48:58 +00:00

jared referenced this issue

2026-10-01 20:49:04 +00:00

Add a mutation gate that proves a green guard test goes red on a broken patch #875

jared referenced this issue

2026-10-01 21:23:06 +00:00

no pipeline path for guard/characterization tests (unexpected_pass escalation) #859

jared commented

2026-10-02 19:43:50 +00:00

Author

Owner

PREDICTED RULING: REVISE

a. No. The body said "Required work: none" and named no mechanism. An implementer could add a new regex or exit-code test for "no tests ran".

b. Reasons:

feedback-remove-habits-dont-backfill-heuristics: a second red-versus-broken rule beside the one that exists is the same kind of bloat.
feedback-lens-request-is-a-design-symptom: challenge the code first. The challenge finds an existing owner (Gate::StepOutcome#classify(expect_red: true), the test-writer red gate). Mutation::Check skipped that owner.
feedback-fail-loudly-no-skip-paths: a broken run must fail, not count as red.
ADR-0258: the guard map and mutation gate are only map configuration, so this change needs no ADR.

c. The user would call it lazy to leave the mechanism to the programmer, or to parse the output again. Research done: check.rb, StepOutcome, TestReport/Summary, GreenAssert, guard map edges, boot_mutation_check_test.rb, the fixture, and ADR-0258.

d. Mutation::Check builds a Gate::StepOutcome for the diff-test run and classifies it with expect_red: true. :red passes. :unexpected_pass gives "mutation survived". :broken gives "mutated run executed no tests". Restore and map stay unchanged. The approach is now in the body.

e. Not read: the ticket comments (issue-show has none), the lens-* skills in full, and Gate::Commands for runners other than Minitest. A runner that does not print the Minitest summary would always classify as :broken.

PREDICTED RULING: REVISE a. No. The body said "Required work: none" and named no mechanism. An implementer could add a new regex or exit-code test for "no tests ran". b. Reasons: - feedback-remove-habits-dont-backfill-heuristics: a second red-versus-broken rule beside the one that exists is the same kind of bloat. - feedback-lens-request-is-a-design-symptom: challenge the code first. The challenge finds an existing owner (Gate::StepOutcome#classify(expect_red: true), the test-writer red gate). Mutation::Check skipped that owner. - feedback-fail-loudly-no-skip-paths: a broken run must fail, not count as red. - ADR-0258: the guard map and mutation gate are only map configuration, so this change needs no ADR. c. The user would call it lazy to leave the mechanism to the programmer, or to parse the output again. Research done: check.rb, StepOutcome, TestReport/Summary, GreenAssert, guard map edges, boot_mutation_check_test.rb, the fixture, and ADR-0258. d. Mutation::Check builds a Gate::StepOutcome for the diff-test run and classifies it with expect_red: true. :red passes. :unexpected_pass gives "mutation survived". :broken gives "mutated run executed no tests". Restore and map stay unchanged. The approach is now in the body. e. Not read: the ticket comments (issue-show has none), the lens-* skills in full, and Gate::Commands for runners other than Minitest. A runner that does not print the Minitest summary would always classify as :broken.

jared commented

2026-10-02 19:44:14 +00:00

Author

Owner

Work started on branch ticket-893 through /os-sdlc:implement (poodr-ticket-implementation map).

jared commented

2026-10-02 19:50:06 +00:00

Author

Owner

Resolution

Done: Mutation::Check now classifies the mutated diff-test run with Gate::StepOutcome#classify(expect_red: true): red passes, unexpected_pass fails as survived, broken (no tests executed) fails with a new message instead of counting as killed.

Evidence: Implementation 20 complete, 8 dispatches all pass, suite gate exit 0. Commits 8009c244 (code + tests) and a485ad62 (History row) on main.

Follow-ups: none

## Resolution **Done:** Mutation::Check now classifies the mutated diff-test run with Gate::StepOutcome#classify(expect_red: true): red passes, unexpected_pass fails as survived, broken (no tests executed) fails with a new message instead of counting as killed. **Evidence:** Implementation 20 complete, 8 dispatches all pass, suite gate exit 0. Commits 8009c244 (code + tests) and a485ad62 (History row) on main. **Follow-ups:** none

jared closed this issue

2026-10-02 19:50:06 +00:00

jared commented

2026-10-03 16:27:58 +00:00

Author

Owner

Finding after close, recorded 2026-10-03 (session a0dbd5c4). How much does this matter: a lot, if confirmed on a real target.

The shipped fix (8009c244) sends the mutated run to Gate::StepOutcome#classify(expect_red: true). That rule uses Summary#executed_failure?, which is runs.positive? && (failures + errors).positive?. On a target that loads code through Zeitwerk, a patch that breaks a file or names a missing constant shows as a Minitest Error inside a test (1 runs, 0 assertions, 0 failures, 1 errors). The gate classifies that as :red and accepts the mutation. These are the two defects that this ticket names. The shipped test passes only because its fixture loads the file with require_relative (tests/support/mutation_check_fixture.rb:10).

Evidence level: reproduced in a scratch Zeitwerk project only. No breaking mutation has run against a real target, and 8 pipeline databases hold 0 mutation handoffs and 0 guard-map runs.

Candidate design, not yet ruled on by Jared: only a failed assertion counts as a kill (Summary#failed_assertion?), and the guard map test-writer node gets gate_results so that a refused rerun sees the reason. A ticket-skeptic verdict of DROP stopped a new ticket until Jared rules. Draft and design records: docs/research/challenge-moves/2026-10-03-scoring/ and docs/research/challenge-moves/prediction-scores.md.

Finding after close, recorded 2026-10-03 (session a0dbd5c4). How much does this matter: a lot, if confirmed on a real target. The shipped fix (8009c244) sends the mutated run to `Gate::StepOutcome#classify(expect_red: true)`. That rule uses `Summary#executed_failure?`, which is `runs.positive? && (failures + errors).positive?`. On a target that loads code through Zeitwerk, a patch that breaks a file or names a missing constant shows as a Minitest Error inside a test (`1 runs, 0 assertions, 0 failures, 1 errors`). The gate classifies that as `:red` and accepts the mutation. These are the two defects that this ticket names. The shipped test passes only because its fixture loads the file with `require_relative` (`tests/support/mutation_check_fixture.rb:10`). Evidence level: reproduced in a scratch Zeitwerk project only. No breaking mutation has run against a real target, and 8 pipeline databases hold 0 mutation handoffs and 0 guard-map runs. Candidate design, not yet ruled on by Jared: only a failed assertion counts as a kill (`Summary#failed_assertion?`), and the guard map test-writer node gets `gate_results` so that a refused rerun sees the reason. A ticket-skeptic verdict of DROP stopped a new ticket until Jared rules. Draft and design records: `docs/research/challenge-moves/2026-10-03-scoring/` and `docs/research/challenge-moves/prediction-scores.md`.

jared referenced this issue from a commit

2026-10-03 17:55:50 +00:00

feat(os-sdlc): Audited POODR boundaries for #893; Mutation::Check sends Gate::StepOutcome#classify(expect_red: true); contracts protected. (#893)

jared referenced this issue from a commit

2026-10-03 17:55:50 +00:00

docs(os-sdlc): #893 History row

jared referenced this issue from a commit

2026-10-05 13:50:50 +00:00

feat(os-sdlc): takedown eval reference solutions, judge pair tooling, harness-first handoff

jared referenced this issue

2026-10-07 14:41:03 +00:00

Keep a guessed fix out of the ticket before the design phase #960