jared/cc-os

Watch

Fork 0

Run pytest targets in the diff-test gate #1625

New issue

Open

opened 2026-11-06 20:09:51 +00:00 by jared · 2 comments

jared commented

2026-11-06 20:09:51 +00:00

Owner

Problem

The diff-test gate cannot run a pytest target. Suite::DiffPlan#test_paths keeps only .rb paths, so a Python target gets a nil command. The gate then exits 0 and classifies unexpected_pass, even when every new test fails. No ticket on os-vault, os-context, or os-doc-hygiene can pass the red round. #1578, #1571, #1568, and #1570 are os-vault tickets that wait next in order.

Three causes hold the defect together, and the fix must remove all three:

Suite::DiffPlan builds a command for .rb tests only. For a .py code change with no test change, the widen-to-suite path also does not fire, because Changes#unmapped_code_paths keeps only Ruby files.
SuiteOutput#summary reads only the Minitest summary line. Pytest prints 1 failed, 2 passed in 0.1s, so the summary is nil. StepOutcome#classify (step_outcome.rb:27-48) then returns :broken for a correct red run and :failed for a nonzero green run. Unchanged::Measurement and Lint::GreenAssert also read SuiteOutput.
Gate::StepRun (step_run.rb:18) returns exit 0 for a nil command, so a missing test command counts as a successful run. A gate that runs no test must not pass.

Takedown run 1625-20261106-152708 fixed cause 1 only, and its grade was cause_remains.

Observed

Run 245 on #1576 escalated with unexpected_pass:test-writer/diff-test. The step output was empty.

pytest -q plugins/os-vault/tests/test_session_start_llamacpp_rebuild.py

7 failed in 1.14s

Reproduce

Open any implementation on --target os-vault, add one failing pytest file, and run os-sdlc-gate for the test-writer dispatch.

Origin

Trigger: implementation 245 for #1576
Improvised this session: hand-finish through the escalation procedure
Chain: unexpected_pass ← nil diff-test command ← path.end_with?(".rb") in diff_plan.rb:45 ← DESIGN (DiffPlan builds a ruby -I command only)
Root candidate: none found; searched pytest, diff-test, DiffPlan, diff_plan, unexpected_pass, python target. Related but not a match: #1554.
Where: plugins/os-sdlc/lib/os_sdlc/suite/diff_plan.rb
Session: 6a326402-c92a-438a-a4b8-ee2f3f512f78
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/6a326402-c92a-438a-a4b8-ee2f3f512f78.jsonl

VERDICT: CREATE

## Problem The diff-test gate cannot run a pytest target. `Suite::DiffPlan#test_paths` keeps only `.rb` paths, so a Python target gets a nil command. The gate then exits 0 and classifies `unexpected_pass`, even when every new test fails. No ticket on os-vault, os-context, or os-doc-hygiene can pass the red round. #1578, #1571, #1568, and #1570 are os-vault tickets that wait next in order. Three causes hold the defect together, and the fix must remove all three: 1. `Suite::DiffPlan` builds a command for `.rb` tests only. For a `.py` code change with no test change, the widen-to-suite path also does not fire, because `Changes#unmapped_code_paths` keeps only Ruby files. 2. `SuiteOutput#summary` reads only the Minitest summary line. Pytest prints `1 failed, 2 passed in 0.1s`, so the summary is nil. `StepOutcome#classify` (step_outcome.rb:27-48) then returns `:broken` for a correct red run and `:failed` for a nonzero green run. `Unchanged::Measurement` and `Lint::GreenAssert` also read `SuiteOutput`. 3. `Gate::StepRun` (step_run.rb:18) returns exit 0 for a nil command, so a missing test command counts as a successful run. A gate that runs no test must not pass. Takedown run 1625-20261106-152708 fixed cause 1 only, and its grade was `cause_remains`. ## Observed Run 245 on #1576 escalated with `unexpected_pass:test-writer/diff-test`. The step output was empty. ``` pytest -q plugins/os-vault/tests/test_session_start_llamacpp_rebuild.py ``` ``` 7 failed in 1.14s ``` ## Reproduce Open any implementation on `--target os-vault`, add one failing pytest file, and run `os-sdlc-gate` for the test-writer dispatch. ## Origin - Trigger: implementation 245 for #1576 - Improvised this session: hand-finish through the escalation procedure - Chain: unexpected_pass ← nil diff-test command ← `path.end_with?(".rb")` in diff_plan.rb:45 ← DESIGN (DiffPlan builds a `ruby -I` command only) - Root candidate: none found; searched pytest, diff-test, DiffPlan, diff_plan, unexpected_pass, python target. Related but not a match: #1554. - Where: plugins/os-sdlc/lib/os_sdlc/suite/diff_plan.rb - Session: 6a326402-c92a-438a-a4b8-ee2f3f512f78 - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/6a326402-c92a-438a-a4b8-ee2f3f512f78.jsonl VERDICT: CREATE

jared added the

plugin/os-sdlc

filed-by/agent

category/code-change

labels

2026-11-06 20:09:51 +00:00

jared referenced this issue

2026-11-06 20:25:06 +00:00

Switch the os-vault rebuild hook to Unsloth 9B V4 on llama.cpp #1576

jared commented

2026-11-06 20:27:05 +00:00

Author

Owner

/os-backlog:ticket-decompose routed this to /os-sdlc:ticket-takedown.

a. One change: SuiteOutput reads the target's test framework and returns the same Summary for pytest output. Suite::DiffPlan then builds pytest -q <paths> for .py tests.

b. Owner: SuiteOutput owns the summary, and StepOutcome#classify asks it (step_outcome.rb:27-48). New object: a pytest summary parser. Its secret is the pytest summary line (1 failed, 2 passed in 0.1s). DiffPlan owns the command.

Evidence from the probe:

SUPPORTED_TEST_FRAMEWORKS (lint/base_assert.rb:6) is not on the os-sdlc-gate path. It affects fix-lints only.
A pytest red run gives :broken, because SuiteOutput#summary is nil. A nonzero green run gives :failed.
Changes#unmapped_code_paths keeps only Ruby files, so widen-to-suite never fires for a .py code change. In the green round diff-test passes with no test run.
lint and test-lint have nil commands on a Python-only diff, so they pass with nothing checked.
reference/adding-a-summary-parser.md is stale: it names a minitest_summary.rb that does not exist and omits StepOutcome.
Also Ruby-only: Facts#round_tests and test_changes (dispatch/brief/facts.rb:80-100), GoalpostCheck::FILE_TOKEN (goalpost_check.rb:6), and agents/test-writer.md.

This ticket covers the test-gate part of #1554. #1578, #1571, #1568, #1570, and #1577 wait on it.

/os-backlog:ticket-decompose routed this to /os-sdlc:ticket-takedown. a. One change: `SuiteOutput` reads the target's test framework and returns the same `Summary` for pytest output. `Suite::DiffPlan` then builds `pytest -q <paths>` for `.py` tests. b. Owner: `SuiteOutput` owns the summary, and `StepOutcome#classify` asks it (step_outcome.rb:27-48). New object: a pytest summary parser. Its secret is the pytest summary line (`1 failed, 2 passed in 0.1s`). `DiffPlan` owns the command. Evidence from the probe: - `SUPPORTED_TEST_FRAMEWORKS` (lint/base_assert.rb:6) is not on the os-sdlc-gate path. It affects fix-lints only. - A pytest red run gives :broken, because `SuiteOutput#summary` is nil. A nonzero green run gives :failed. - `Changes#unmapped_code_paths` keeps only Ruby files, so widen-to-suite never fires for a `.py` code change. In the green round diff-test passes with no test run. - `lint` and `test-lint` have nil commands on a Python-only diff, so they pass with nothing checked. - reference/adding-a-summary-parser.md is stale: it names a minitest_summary.rb that does not exist and omits `StepOutcome`. - Also Ruby-only: `Facts#round_tests` and `test_changes` (dispatch/brief/facts.rb:80-100), `GoalpostCheck::FILE_TOKEN` (goalpost_check.rb:6), and `agents/test-writer.md`. This ticket covers the test-gate part of #1554. #1578, #1571, #1568, #1570, and #1577 wait on it.

jared commented

2026-11-06 21:02:04 +00:00

Author

Owner

Parked by the orchestrator on 2026-11-06. The os-vault Python code is due for the Ruby port (#1527), so pytest support in the gates would lose its value when the port ships. Takedown designs are not implemented. Python os-vault tickets are finished by hand with tests and one audit until then.

jared referenced this issue

2026-11-06 21:05:28 +00:00

Stop counting graphify path refusals as no path in the retrieval harness #1578

jared referenced this issue from a commit

2026-11-06 21:05:31 +00:00

docs(decisions): record the order of #1625 and the vault tickets

jared referenced this issue from a commit

2026-11-06 21:05:31 +00:00

docs(decisions): widen #1625 to the three causes

jared referenced this issue from a commit

2026-11-06 21:05:31 +00:00

docs(decisions): park #1625 and return to the vault order