diff-test gate accepts a round as red when any test is red but test-reviewer requires all new tests red #953

New issue

Open

opened 2026-10-05 13:05:48 +00:00 by jared · 2 comments

jared commented

2026-10-05 13:05:48 +00:00

Owner

VERDICT: CREATE

Problem

The diff-test gate accepts a test-writer round as red when any test is red. The test-reviewer requires all new tests red. The two checks disagree, so a round passes the gate and then fails review. Dispatches 273, 275, 277, 279 passed the gate (exit 1 accepted). Reviewers 274, 276, 278, 280 failed the same rounds for green tests.

Reproduce

n/a: needs a live round with a mix of red and green new tests; implementation 30 shows it.

Approach

Required work:

Make the red classification require every new test red, or make the test-reviewer accept mixed rounds, so the two agree — check: present executed_failure? in plugins/os-sdlc/lib/os_sdlc/model/gate/step_outcome.rb

Acceptance criteria

Given a round with one red and one green new test, when the diff-test gate runs, then the gate and the test-reviewer give the same verdict.
Given a round with all new tests red, when the gate runs, then it reports red.

Owner sketch

class Gate::StepOutcome
  def classify_red
end

Origin

Trigger: os-sdlc implementation 30 of ticket #948.
Improvised this session: none
Chain: reviewer fails a gate-accepted round ← gate classifies red on any failing test, reviewer demands all red ← DESIGN (plugins/os-sdlc/lib/os_sdlc/model/gate/step_outcome.rb line 45, agents/test-reviewer.md line 20)
Where: plugins/os-sdlc/lib/os_sdlc/model/gate/step_outcome.rb
Session: ab599e53-dbf5-4953-b8bf-c598a9c5c16e
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-crc-os-adr-2026-10-04/ab599e53-dbf5-4953-b8bf-c598a9c5c16e.jsonl

VERDICT: CREATE ## Problem The diff-test gate accepts a test-writer round as red when any test is red. The test-reviewer requires all new tests red. The two checks disagree, so a round passes the gate and then fails review. Dispatches 273, 275, 277, 279 passed the gate (exit 1 accepted). Reviewers 274, 276, 278, 280 failed the same rounds for green tests. ## Reproduce n/a: needs a live round with a mix of red and green new tests; implementation 30 shows it. ## Approach Required work: 1. Make the red classification require every new test red, or make the test-reviewer accept mixed rounds, so the two agree — check: present `executed_failure?` in `plugins/os-sdlc/lib/os_sdlc/model/gate/step_outcome.rb` ## Acceptance criteria 1. Given a round with one red and one green new test, when the diff-test gate runs, then the gate and the test-reviewer give the same verdict. 2. Given a round with all new tests red, when the gate runs, then it reports red. ## Owner sketch ```ruby class Gate::StepOutcome def classify_red end ``` ## Origin - Trigger: os-sdlc implementation 30 of ticket #948. - Improvised this session: none - Chain: reviewer fails a gate-accepted round ← gate classifies red on any failing test, reviewer demands all red ← DESIGN (plugins/os-sdlc/lib/os_sdlc/model/gate/step_outcome.rb line 45, agents/test-reviewer.md line 20) - Where: plugins/os-sdlc/lib/os_sdlc/model/gate/step_outcome.rb - Session: ab599e53-dbf5-4953-b8bf-c598a9c5c16e - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-crc-os-adr-2026-10-04/ab599e53-dbf5-4953-b8bf-c598a9c5c16e.jsonl

jared added the

plugin/os-sdlc

filed-by/agent

labels

2026-10-05 13:05:48 +00:00

jared referenced this issue

2026-10-05 13:09:51 +00:00

os-adr: three copies of path and file-name knowledge (A12) #948

jared added the

category/code-change

label

2026-10-10 13:31:45 +00:00

jared commented

2026-10-12 14:31:05 +00:00

Author

Owner

The takedown's final-red gate had the same per-file weakness and #995 fixed it with a per-method red check (commit 7907c2b6: every new test method must be red at base). The place-tests diff-test gate still classifies red when any test fails (plugins/os-sdlc/lib/os_sdlc/model/gate/step_outcome.rb:45-49; map plugins/os-sdlc/maps/takedown-implement.yaml:20). The #995 approach is a candidate fix here.

The takedown's final-red gate had the same per-file weakness and #995 fixed it with a per-method red check (commit 7907c2b6: every new test method must be red at base). The place-tests diff-test gate still classifies red when any test fails (plugins/os-sdlc/lib/os_sdlc/model/gate/step_outcome.rb:45-49; map plugins/os-sdlc/maps/takedown-implement.yaml:20). The #995 approach is a candidate fix here.

jared commented

2026-10-26 16:02:28 +00:00

Author

Owner

Occurrence: Red-first round has no exemption for preservation tests; test-reviewer ends the run with verdict:error
Seen: plugins/os-sdlc/lib/os_sdlc/dispatch/brief/facts.rb:82 (round_tests: all new tests must be red); maps/poodr-ticket-implementation.yaml:92 (error edge)
Run: implementation 172 (#1370), session c6315119-bc8d-4ac6-98ec-96edc8052428, dispatches 1501 (reviewer fail, objection 161), 1502 (writer could not fix), 1503 (reviewer verdict: error)
Adds: a ticket-required preservation test (behavior already present) is green by nature, so a mixed round cannot pass and the reviewer settles error. Secondary: the error edge goes straight to implementation-failed and drops the reviewer's GAP text; an escalation would let the orchestrator recover. Workaround used on #1360: fold must-still-fire cases into one contrast test and state that existing tests prove preserved behavior.

Occurrence: Red-first round has no exemption for preservation tests; test-reviewer ends the run with verdict:error Seen: plugins/os-sdlc/lib/os_sdlc/dispatch/brief/facts.rb:82 (round_tests: all new tests must be red); maps/poodr-ticket-implementation.yaml:92 (error edge) Run: implementation 172 (#1370), session c6315119-bc8d-4ac6-98ec-96edc8052428, dispatches 1501 (reviewer fail, objection 161), 1502 (writer could not fix), 1503 (reviewer verdict: error) Adds: a ticket-required preservation test (behavior already present) is green by nature, so a mixed round cannot pass and the reviewer settles error. Secondary: the error edge goes straight to implementation-failed and drops the reviewer's GAP text; an escalation would let the orchestrator recover. Workaround used on #1360: fold must-still-fire cases into one contrast test and state that existing tests prove preserved behavior.

jared referenced this issue

2026-10-26 16:02:32 +00:00

Sdlc/Structural/ForwardingClass: skip role players and third-party adapters #1360