os-sdlc: test taxonomy classifier (mutation testing increment 2) #1634

New issue

Closed

opened 2026-11-08 15:24:32 +00:00 by jared · 4 comments

jared commented 2026-11-08 15:24:32 +00:00

Owner

Copy link

Problem

Mutation-based removal applies only to BEHAVIOR tests. Other test kinds follow their own keep or delete rules. The pipeline cannot sort a test into its kind without an agent. The plan rules deterministic signals for PROMPT-TEXT, LINT-AS-TEST, and STRUCTURE, an owned allowlist for CONTRACT, BEHAVIOR by elimination, and a MIXED bucket for judgment.

Reproduce

n/a: feature ticket.

Approach

Source: docs/research/mutation-testing/plan-v1-2026-11-08.md, table "Increments, in order", row 2. The section "Rulings" is binding; do not re-decide it. Row 2 says "MISS bucket"; the Rulings say MIXED; use MIXED. Parent: #1638.

The classifier gives each test file one kind. It reads the file with Prism, so text inside a heredoc never counts as a signal (cop tests hold Ruby source in heredocs, for example tests/cops/effect_in_model_test.rb:27). The classifier is a Ruby object for #1636 and #1637. It has no CLI and no gate or map wiring.

taxonomy = OsSdlc::MutationTest::Taxonomy.for(target_dir, test_path: target.test.path)
taxonomy.kind("tests/tracker/null_contract_test.rb")
taxonomy.behavior?("tests/order_test.rb")

test_file is relative to the target directory, the same form as MutationTest::Scope#tests and the Mutineer report. target_dir is the directory that Implementation#mutation_test already builds (lib/os_sdlc/implementation.rb:85).

Kinds: BEHAVIOR, CONTRACT, STRUCTURE, PROMPT-TEXT, LINT-AS-TEST, MIXED.

What a test method reaches: a test method is a def test_* anywhere in the file. It reaches its own body, the setup body, the bodies of same-file methods it calls by bare name (followed transitively), and the values of same-file constants it reads by name.

Signals, over the nodes one test method reaches:

Kind	Signal
PROMPT-TEXT	A call to File.read, File.readlines, File.foreach, File.expand_path, File.join, Dir.glob or Dir.[] whose string-literal arguments, joined with /, match %r{(\A|/)(agents|skills)/(.*/)?[^/]*\.md\z}.
LINT-AS-TEST	A Dir.glob, Dir.[] or Find.find call whose arguments hold __dir__, __FILE__ or a constant read, in a method that also has an assert_empty, an assert_operator, or an assert_equal whose first argument is an Integer or Array literal.
STRUCTURE	An assert* or refute* call whose argument subtree holds exist? on File, Dir or FileTest, const_defined?, or respond_to?; or any call named eager_load or eager_load_all.

Method kind: no signal is BEHAVIOR (by elimination); one signal kind is that kind; two or more signal kinds is MIXED.

File kind, first rule that applies:

• The file is in the allowlist: CONTRACT.
• The file's top-level statements are all require or require_relative calls (a pure require pointer): STRUCTURE.
• The file defines no test_* method: MIXED (unclassified).
• Every method has the same kind: that kind. Otherwise: MIXED.

CONTRACT allowlist: <target dir>/<test_path>/contract_seams.yml; for os-sdlc, plugins/os-sdlc/tests/contract_seams.yml. A missing file is an empty allowlist. The format maps a seam to an owning side, and the side to one test file, so the format enforces "at most one test per seam per owning side". A side whose value is not one String makes the build raise YamlFileError.

tracker:
  OsSdlc::Tracker::Null: tests/tracker/null_contract_test.rb
  OsSdlc::Tracker::Forgejo: tests/tracker/forgejo_contract_test.rb
  OsSdlc::Tracker::Backlog: tests/tracker/backlog_contract_test.rb
procman-server-type:
  OsSdlc::Daemon::ProcmanSpec: tests/daemon/procman_spec_contract_test.rb

Seed seams: tracker — Tracker.for picks Forgejo or Backlog at run time (lib/os_sdlc/tracker.rb:5-9); three sides (lib/os_sdlc/tracker/null.rb:5, lib/os_sdlc/tracker/backlog.rb:31, lib/os_sdlc/tracker/forgejo.rb:22) share tests/support/tracker_contract_tests.rb:3 (lens T6). procman-server-type — Daemon.connect hands ProcmanSpec to the separate procman gem (lib/os_sdlc/daemon.rb:37, lib/os_sdlc/daemon/procman_spec.rb:8, Gemfile:20).

Rejected: file-level signals only (a file with one lint test and many behavior tests, tests/cops/inline_dataset_query_test.rb:65, would hide its behavior tests); grep over file text (heredoc fixtures give false STRUCTURE signals); the allowlist under .sdlc/ (.sdlc/.gitignore ignores all but project.yaml, and a target in a subdirectory has no .sdlc/).

Required work:

1. Add OsSdlc::MutationTest::Taxonomy with Taxonomy.for(root, test_path:), #kind(test_file) (the four file rules above) and #behavior?(test_file) — check: present def behavior? in plugins/os-sdlc/lib/os_sdlc/mutation_test/taxonomy.rb
2. Add MutationTest::Taxonomy::TestFile, built from source text and parsed with Prism; it answers require_pointer? and gives the nodes each test_* method reaches — check: present Prism.parse in plugins/os-sdlc/lib/os_sdlc/mutation_test/taxonomy/test_file.rb
3. Add MutationTest::Taxonomy::Signals, the owner of the three signal rules, returning the set of kinds found in a set of nodes — check: present LINT-AS-TEST in plugins/os-sdlc/lib/os_sdlc/mutation_test/taxonomy/signals.rb
4. Add MutationTest::Taxonomy::Seams, a value built from the parsed YAML mapping that answers include?(test_file) and raises YamlFileError naming the seam and side when a side's value is not one String — check: present YamlFileError in plugins/os-sdlc/lib/os_sdlc/mutation_test/taxonomy/seams.rb
5. Add the seed allowlist with the two seams above — check: present tests/tracker/null_contract_test.rb in plugins/os-sdlc/tests/contract_seams.yml
6. Make MutationTest::Scope.from keep only test paths that end in _test.rb. Today Changes#test_paths holds every changed path under the test directory, and MutineerRunner::Command (lib/os_sdlc/mutation_test/mutineer_runner/command.rb:23) passes each one to Mutineer as --test; this ticket's own contract_seams.yml, and any changed file under tests/fixtures/, would reach Mutineer as a test file — check: present _test.rb in plugins/os-sdlc/lib/os_sdlc/mutation_test/scope.rb

Acceptance criteria

1. Given a target with no allowlist file and a test file with no signal, when it is classified, then its kind is BEHAVIOR.
2. Given a test file whose helper reads an agents/*.md path held in a constant, when it is classified, then its kind is PROMPT-TEXT.
3. Given a test file whose setup reads a skills/**/SKILL.md path, when it is classified, then its kind is PROMPT-TEXT.
4. Given a test that asserts an empty list over a Dir.glob built from a constant path, when it is classified, then its kind is LINT-AS-TEST.
5. Given a test that asserts Object.const_defined?, when it is classified, then its kind is STRUCTURE.
6. Given a test that calls Zeitwerk::Loader.eager_load_all, when it is classified, then its kind is STRUCTURE.
7. Given a test that calls File.exist? only as a guard outside an assertion, when it is classified, then its kind is BEHAVIOR.
8. Given a test file whose heredoc holds the text assert File.exist?(...), when it is classified, then its kind is BEHAVIOR.
9. Given a test file with one BEHAVIOR test and one STRUCTURE test, when it is classified, then its kind is MIXED.
10. Given a test file that only requires another test file, when it is classified, then its kind is STRUCTURE.
11. Given a test file that defines no test method and is not in the allowlist, when it is classified, then its kind is MIXED.
12. Given an allowlisted test file that asserts respond_to?, when it is classified, then its kind is CONTRACT.
13. Given an allowlist side that lists two test files, when the taxonomy is built, then it raises YamlFileError naming the seam and the side.
14. Given one BEHAVIOR file and one STRUCTURE file, when behavior? is asked for each, then it is true for the first and false for the second.
15. Given changes whose test paths hold one _test.rb file and one .yml file, when a mutation scope is built from them, then its tests hold only the _test.rb file.

Owner sketch

Object	Owns	Message
MutationTest::Taxonomy	The file-kind rule (allowlist, then pointer, then no test method, then the fold of method kinds); reads the test file and the allowlist file	Taxonomy.for(root, test_path:), #kind(test_file), #behavior?(test_file)
MutationTest::Taxonomy::TestFile	The Prism parse of one source text: which methods are tests, and which nodes each test reaches	#require_pointer?, #test_methods
MutationTest::Taxonomy::Signals	The three signal rules (PROMPT-TEXT, LINT-AS-TEST, STRUCTURE)	#kinds(nodes)
MutationTest::Taxonomy::Seams	The CONTRACT allowlist and its one-file-per-side rule	Seams.new(mapping), #include?(test_file)
MutationTest::Scope (existing)	Which changed test paths are test files that Mutineer runs	Scope.from(changes, since, within:), #tests

Failing test at plugins/os-sdlc/tests/mutation_test/taxonomy_test.rb; every test is red today with NameError: uninitialized constant OsSdlc::MutationTest::Taxonomy.

# frozen_string_literal: true

require_relative "../test_helper"

module OsSdlc
  class MutationTestTaxonomyTest < TmpdirTestCase
    ORDER_TEST = <<~'RUBY'
      class OrderTest < Minitest::Test
        def test_total_sums_the_lines
          assert_equal 3, Order.new([1, 2]).total
        end
      end
    RUBY

    ORDER_DEFINED_TEST = <<~'RUBY'
      class OrderDefinedTest < Minitest::Test
        def test_order_is_defined
          assert Object.const_defined?(:Order)
        end
      end
    RUBY

    def test_a_file_with_no_signal_is_behavior
      write("tests/order_test.rb", ORDER_TEST)

      assert_equal "BEHAVIOR", kind("tests/order_test.rb")
    end

    def test_a_helper_that_reads_agent_markdown_through_a_constant_is_prompt_text
      write("tests/reviewer_agent_test.rb", <<~'RUBY')
        class ReviewerAgentTest < Minitest::Test
          PROMPT = File.expand_path("../agents/reviewer.md", __dir__)

          def test_names_the_verdict
            assert_match(/VERDICT/, prompt)
          end

          def prompt = File.read(PROMPT)
        end
      RUBY

      assert_equal "PROMPT-TEXT", kind("tests/reviewer_agent_test.rb")
    end

    def test_a_setup_that_reads_skill_markdown_is_prompt_text
      write("tests/review_skill_test.rb", <<~'RUBY')
        class ReviewSkillTest < Minitest::Test
          def setup
            @text = File.read(File.join("skills", "review", "SKILL.md"))
          end

          def test_names_the_lens
            assert_includes @text, "lens"
          end
        end
      RUBY

      assert_equal "PROMPT-TEXT", kind("tests/review_skill_test.rb")
    end

    def test_an_empty_list_asserted_over_a_codebase_glob_is_lint_as_test
      write("tests/no_env_test.rb", <<~'RUBY')
        class NoEnvTest < Minitest::Test
          LIB = File.expand_path("../lib", __dir__)

          def test_no_library_file_reads_env
            readers = Dir.glob(File.join(LIB, "**/*.rb")).select { |path| File.read(path).include?("ENV") }

            assert_empty readers
          end
        end
      RUBY

      assert_equal "LINT-AS-TEST", kind("tests/no_env_test.rb")
    end

    def test_an_asserted_const_defined_is_structure
      write("tests/order_defined_test.rb", ORDER_DEFINED_TEST)

      assert_equal "STRUCTURE", kind("tests/order_defined_test.rb")
    end

    def test_an_eager_load_is_structure
      write("tests/eager_load_test.rb", <<~'RUBY')
        class EagerLoadTest < Minitest::Test
          def test_the_library_eager_loads
            Zeitwerk::Loader.eager_load_all
          end
        end
      RUBY

      assert_equal "STRUCTURE", kind("tests/eager_load_test.rb")
    end

    def test_a_file_exist_guard_outside_an_assertion_is_behavior
      write("tests/cache_test.rb", <<~'RUBY')
        class CacheTest < Minitest::Test
          def test_writes_the_cache_once
            path = File.join(Dir.mktmpdir, "cache")
            File.write(path, "x") unless File.exist?(path)

            assert_equal "x", File.read(path)
          end
        end
      RUBY

      assert_equal "BEHAVIOR", kind("tests/cache_test.rb")
    end

    def test_signal_text_inside_a_heredoc_is_behavior
      write("tests/cop_test.rb", <<~'RUBY')
        class CopTest < Minitest::Test
          SOURCE = <<~SRC
            assert File.exist?("lib/order.rb")
          SRC

          def test_flags_the_source
            assert_equal 1, Cop.new.offenses(SOURCE).size
          end
        end
      RUBY

      assert_equal "BEHAVIOR", kind("tests/cop_test.rb")
    end

    def test_a_behavior_test_and_a_structure_test_in_one_file_is_mixed
      write("tests/order_mixed_test.rb", <<~'RUBY')
        class OrderMixedTest < Minitest::Test
          def test_total_sums_the_lines
            assert_equal 3, Order.new([1, 2]).total
          end

          def test_order_is_defined
            assert Object.const_defined?(:Order)
          end
        end
      RUBY

      assert_equal "MIXED", kind("tests/order_mixed_test.rb")
    end

    def test_a_pure_require_pointer_is_structure
      write("tests/old_order_test.rb", "require_relative \"order_test\"\n")

      assert_equal "STRUCTURE", kind("tests/old_order_test.rb")
    end

    def test_a_file_with_no_test_method_outside_the_allowlist_is_mixed
      write("tests/shared_tracker_test.rb", <<~'RUBY')
        class SharedTrackerTest < Minitest::Test
          include TrackerContractTests
        end
      RUBY

      assert_equal "MIXED", kind("tests/shared_tracker_test.rb")
    end

    def test_an_allowlisted_file_is_contract_even_with_a_structure_signal
      write("tests/contract_seams.yml", "tracker:\n  Tracker::Null: tests/null_contract_test.rb\n")
      write("tests/null_contract_test.rb", <<~'RUBY')
        class NullContractTest < Minitest::Test
          def test_answers_the_waiting_label_message
            assert Tracker::Null.new.respond_to?(:add_waiting_label)
          end
        end
      RUBY

      assert_equal "CONTRACT", kind("tests/null_contract_test.rb")
    end

    def test_an_allowlist_side_with_two_test_files_is_refused
      write("tests/contract_seams.yml", <<~YAML)
        tracker:
          Tracker::Null:
            - tests/a_contract_test.rb
            - tests/b_contract_test.rb
      YAML

      error = assert_raises(YamlFileError) { taxonomy }

      assert_includes error.message, "tracker"
      assert_includes error.message, "Tracker::Null"
    end

    def test_behavior_is_true_only_for_a_behavior_file
      write("tests/order_test.rb", ORDER_TEST)
      write("tests/order_defined_test.rb", ORDER_DEFINED_TEST)

      assert taxonomy.behavior?("tests/order_test.rb")
      refute taxonomy.behavior?("tests/order_defined_test.rb")
    end

    private

    def write(relative, text)
      path = File.join(@dir, relative)
      FileUtils.mkdir_p(File.dirname(path))
      File.write(path, text)
    end

    def taxonomy = MutationTest::Taxonomy.for(@dir, test_path: "tests")

    def kind(relative) = taxonomy.kind(relative)
  end
end

Failing test at plugins/os-sdlc/tests/mutation_test/scope_test.rb; red today because scope.tests also holds tests/contract_seams.yml.

# frozen_string_literal: true

require_relative "../test_helper"

module OsSdlc
  class MutationTestScopeTest < Minitest::Test
    Changes = Data.define(:code_paths, :test_paths, :unmapped_code_paths)
    Within = Data.define(:root) do
      def relative(path) = path.delete_prefix("#{root}/")
    end

    def test_from_keeps_only_ruby_test_files
      changes = Changes.new(code_paths: ["/t/lib/a.rb"], test_paths: ["/t/tests/a_test.rb", "/t/tests/contract_seams.yml"],
                            unmapped_code_paths: [])

      scope = MutationTest::Scope.from(changes, "abc", within: Within.new(root: "/t"))

      assert_equal ["tests/a_test.rb"], scope.tests
    end
  end
end

Out of scope

• Using the kinds in the blind filters (#1636) and the overlap report (#1637).
• Writing MIXED judgments or the signals that come from them (#1635 ledger).
• A CLI or bin script, and a row in Boot::Commands::TABLE.
• Listing every test file of a target; consumers pass the test files that Mutineer reports.
• A CHARACTERIZATION signal; the plan rules none, so such tests fall to BEHAVIOR by elimination.
• Per-test-method kinds in the output.
• Splitting or deleting MIXED files.
• Caching, flags, caps or timeouts.
• tests/system_model_viewer_procman_spec_contract_test.rb in the seed allowlist; its tests at lines 12 to 30 check behavior, so CONTRACT would hide them.

Origin

• Trigger: plan docs/research/mutation-testing/plan-v1-2026-11-08.md (commit b90d8253), ruled by Jared 2026-11-07 and 2026-11-08
• Improvised this session: none
• Chain: suite doubled in six weeks (757 test files, 3,572 test methods) ← suspected duplicated behavior and tests with no demonstrated value ← no mutation evidence in the gate. End: DESIGN (docs/research/mutation-testing/plan-v1-2026-11-08.md)
• Root candidate: none found; searched mutation, mutineer, mutant, survivor, taxonomy, redundant test, blind test, duplicate, overlap, ledger
• Where: plugins/os-sdlc (new classifier; placement by the architecture lens)
• Session: ff703e34-8046-4b24-af68-6ef76d8aa338
• Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os-plugins-os-sdlc/ff703e34-8046-4b24-af68-6ef76d8aa338.jsonl
• Decomposed: session 87bc217f-61e2-4d4d-8470-9f3c8fd1ef43 (takedown relevance refused it, see #1642)

VERDICT: CREATE

## Problem Mutation-based removal applies only to BEHAVIOR tests. Other test kinds follow their own keep or delete rules. The pipeline cannot sort a test into its kind without an agent. The plan rules deterministic signals for PROMPT-TEXT, LINT-AS-TEST, and STRUCTURE, an owned allowlist for CONTRACT, BEHAVIOR by elimination, and a MIXED bucket for judgment. ## Reproduce n/a: feature ticket. ## Approach Source: `docs/research/mutation-testing/plan-v1-2026-11-08.md`, table "Increments, in order", row 2. The section "Rulings" is binding; do not re-decide it. Row 2 says "MISS bucket"; the Rulings say MIXED; use MIXED. Parent: #1638. The classifier gives each test file one kind. It reads the file with Prism, so text inside a heredoc never counts as a signal (cop tests hold Ruby source in heredocs, for example `tests/cops/effect_in_model_test.rb:27`). The classifier is a Ruby object for #1636 and #1637. It has no CLI and no gate or map wiring. ```ruby taxonomy = OsSdlc::MutationTest::Taxonomy.for(target_dir, test_path: target.test.path) taxonomy.kind("tests/tracker/null_contract_test.rb") taxonomy.behavior?("tests/order_test.rb") ``` `test_file` is relative to the target directory, the same form as `MutationTest::Scope#tests` and the Mutineer report. `target_dir` is the directory that `Implementation#mutation_test` already builds (`lib/os_sdlc/implementation.rb:85`). Kinds: `BEHAVIOR`, `CONTRACT`, `STRUCTURE`, `PROMPT-TEXT`, `LINT-AS-TEST`, `MIXED`. What a test method reaches: a test method is a `def test_*` anywhere in the file. It reaches its own body, the `setup` body, the bodies of same-file methods it calls by bare name (followed transitively), and the values of same-file constants it reads by name. Signals, over the nodes one test method reaches: | Kind | Signal | |---|---| | PROMPT-TEXT | A call to `File.read`, `File.readlines`, `File.foreach`, `File.expand_path`, `File.join`, `Dir.glob` or `Dir.[]` whose string-literal arguments, joined with `/`, match `%r{(\A\|/)(agents\|skills)/(.*/)?[^/]*\.md\z}`. | | LINT-AS-TEST | A `Dir.glob`, `Dir.[]` or `Find.find` call whose arguments hold `__dir__`, `__FILE__` or a constant read, in a method that also has an `assert_empty`, an `assert_operator`, or an `assert_equal` whose first argument is an Integer or Array literal. | | STRUCTURE | An `assert*` or `refute*` call whose argument subtree holds `exist?` on `File`, `Dir` or `FileTest`, `const_defined?`, or `respond_to?`; or any call named `eager_load` or `eager_load_all`. | Method kind: no signal is BEHAVIOR (by elimination); one signal kind is that kind; two or more signal kinds is MIXED. File kind, first rule that applies: - The file is in the allowlist: CONTRACT. - The file's top-level statements are all `require` or `require_relative` calls (a pure require pointer): STRUCTURE. - The file defines no `test_*` method: MIXED (unclassified). - Every method has the same kind: that kind. Otherwise: MIXED. CONTRACT allowlist: `<target dir>/<test_path>/contract_seams.yml`; for os-sdlc, `plugins/os-sdlc/tests/contract_seams.yml`. A missing file is an empty allowlist. The format maps a seam to an owning side, and the side to one test file, so the format enforces "at most one test per seam per owning side". A side whose value is not one String makes the build raise `YamlFileError`. ```yaml tracker: OsSdlc::Tracker::Null: tests/tracker/null_contract_test.rb OsSdlc::Tracker::Forgejo: tests/tracker/forgejo_contract_test.rb OsSdlc::Tracker::Backlog: tests/tracker/backlog_contract_test.rb procman-server-type: OsSdlc::Daemon::ProcmanSpec: tests/daemon/procman_spec_contract_test.rb ``` Seed seams: `tracker` — `Tracker.for` picks Forgejo or Backlog at run time (`lib/os_sdlc/tracker.rb:5-9`); three sides (`lib/os_sdlc/tracker/null.rb:5`, `lib/os_sdlc/tracker/backlog.rb:31`, `lib/os_sdlc/tracker/forgejo.rb:22`) share `tests/support/tracker_contract_tests.rb:3` (lens T6). `procman-server-type` — `Daemon.connect` hands `ProcmanSpec` to the separate procman gem (`lib/os_sdlc/daemon.rb:37`, `lib/os_sdlc/daemon/procman_spec.rb:8`, `Gemfile:20`). Rejected: file-level signals only (a file with one lint test and many behavior tests, `tests/cops/inline_dataset_query_test.rb:65`, would hide its behavior tests); grep over file text (heredoc fixtures give false STRUCTURE signals); the allowlist under `.sdlc/` (`.sdlc/.gitignore` ignores all but `project.yaml`, and a target in a subdirectory has no `.sdlc/`). Required work: 1. Add `OsSdlc::MutationTest::Taxonomy` with `Taxonomy.for(root, test_path:)`, `#kind(test_file)` (the four file rules above) and `#behavior?(test_file)` — check: present `def behavior?` in `plugins/os-sdlc/lib/os_sdlc/mutation_test/taxonomy.rb` 2. Add `MutationTest::Taxonomy::TestFile`, built from source text and parsed with Prism; it answers `require_pointer?` and gives the nodes each `test_*` method reaches — check: present `Prism.parse` in `plugins/os-sdlc/lib/os_sdlc/mutation_test/taxonomy/test_file.rb` 3. Add `MutationTest::Taxonomy::Signals`, the owner of the three signal rules, returning the set of kinds found in a set of nodes — check: present `LINT-AS-TEST` in `plugins/os-sdlc/lib/os_sdlc/mutation_test/taxonomy/signals.rb` 4. Add `MutationTest::Taxonomy::Seams`, a value built from the parsed YAML mapping that answers `include?(test_file)` and raises `YamlFileError` naming the seam and side when a side's value is not one String — check: present `YamlFileError` in `plugins/os-sdlc/lib/os_sdlc/mutation_test/taxonomy/seams.rb` 5. Add the seed allowlist with the two seams above — check: present `tests/tracker/null_contract_test.rb` in `plugins/os-sdlc/tests/contract_seams.yml` 6. Make `MutationTest::Scope.from` keep only test paths that end in `_test.rb`. Today `Changes#test_paths` holds every changed path under the test directory, and `MutineerRunner::Command` (`lib/os_sdlc/mutation_test/mutineer_runner/command.rb:23`) passes each one to Mutineer as `--test`; this ticket's own `contract_seams.yml`, and any changed file under `tests/fixtures/`, would reach Mutineer as a test file — check: present `_test.rb` in `plugins/os-sdlc/lib/os_sdlc/mutation_test/scope.rb` ## Acceptance criteria 1. Given a target with no allowlist file and a test file with no signal, when it is classified, then its kind is BEHAVIOR. 2. Given a test file whose helper reads an `agents/*.md` path held in a constant, when it is classified, then its kind is PROMPT-TEXT. 3. Given a test file whose `setup` reads a `skills/**/SKILL.md` path, when it is classified, then its kind is PROMPT-TEXT. 4. Given a test that asserts an empty list over a `Dir.glob` built from a constant path, when it is classified, then its kind is LINT-AS-TEST. 5. Given a test that asserts `Object.const_defined?`, when it is classified, then its kind is STRUCTURE. 6. Given a test that calls `Zeitwerk::Loader.eager_load_all`, when it is classified, then its kind is STRUCTURE. 7. Given a test that calls `File.exist?` only as a guard outside an assertion, when it is classified, then its kind is BEHAVIOR. 8. Given a test file whose heredoc holds the text `assert File.exist?(...)`, when it is classified, then its kind is BEHAVIOR. 9. Given a test file with one BEHAVIOR test and one STRUCTURE test, when it is classified, then its kind is MIXED. 10. Given a test file that only requires another test file, when it is classified, then its kind is STRUCTURE. 11. Given a test file that defines no test method and is not in the allowlist, when it is classified, then its kind is MIXED. 12. Given an allowlisted test file that asserts `respond_to?`, when it is classified, then its kind is CONTRACT. 13. Given an allowlist side that lists two test files, when the taxonomy is built, then it raises `YamlFileError` naming the seam and the side. 14. Given one BEHAVIOR file and one STRUCTURE file, when `behavior?` is asked for each, then it is true for the first and false for the second. 15. Given changes whose test paths hold one `_test.rb` file and one `.yml` file, when a mutation scope is built from them, then its tests hold only the `_test.rb` file. ## Owner sketch | Object | Owns | Message | |---|---|---| | `MutationTest::Taxonomy` | The file-kind rule (allowlist, then pointer, then no test method, then the fold of method kinds); reads the test file and the allowlist file | `Taxonomy.for(root, test_path:)`, `#kind(test_file)`, `#behavior?(test_file)` | | `MutationTest::Taxonomy::TestFile` | The Prism parse of one source text: which methods are tests, and which nodes each test reaches | `#require_pointer?`, `#test_methods` | | `MutationTest::Taxonomy::Signals` | The three signal rules (PROMPT-TEXT, LINT-AS-TEST, STRUCTURE) | `#kinds(nodes)` | | `MutationTest::Taxonomy::Seams` | The CONTRACT allowlist and its one-file-per-side rule | `Seams.new(mapping)`, `#include?(test_file)` | | `MutationTest::Scope` (existing) | Which changed test paths are test files that Mutineer runs | `Scope.from(changes, since, within:)`, `#tests` | Failing test at `plugins/os-sdlc/tests/mutation_test/taxonomy_test.rb`; every test is red today with `NameError: uninitialized constant OsSdlc::MutationTest::Taxonomy`. ```ruby # frozen_string_literal: true require_relative "../test_helper" module OsSdlc class MutationTestTaxonomyTest < TmpdirTestCase ORDER_TEST = <<~'RUBY' class OrderTest < Minitest::Test def test_total_sums_the_lines assert_equal 3, Order.new([1, 2]).total end end RUBY ORDER_DEFINED_TEST = <<~'RUBY' class OrderDefinedTest < Minitest::Test def test_order_is_defined assert Object.const_defined?(:Order) end end RUBY def test_a_file_with_no_signal_is_behavior write("tests/order_test.rb", ORDER_TEST) assert_equal "BEHAVIOR", kind("tests/order_test.rb") end def test_a_helper_that_reads_agent_markdown_through_a_constant_is_prompt_text write("tests/reviewer_agent_test.rb", <<~'RUBY') class ReviewerAgentTest < Minitest::Test PROMPT = File.expand_path("../agents/reviewer.md", __dir__) def test_names_the_verdict assert_match(/VERDICT/, prompt) end def prompt = File.read(PROMPT) end RUBY assert_equal "PROMPT-TEXT", kind("tests/reviewer_agent_test.rb") end def test_a_setup_that_reads_skill_markdown_is_prompt_text write("tests/review_skill_test.rb", <<~'RUBY') class ReviewSkillTest < Minitest::Test def setup @text = File.read(File.join("skills", "review", "SKILL.md")) end def test_names_the_lens assert_includes @text, "lens" end end RUBY assert_equal "PROMPT-TEXT", kind("tests/review_skill_test.rb") end def test_an_empty_list_asserted_over_a_codebase_glob_is_lint_as_test write("tests/no_env_test.rb", <<~'RUBY') class NoEnvTest < Minitest::Test LIB = File.expand_path("../lib", __dir__) def test_no_library_file_reads_env readers = Dir.glob(File.join(LIB, "**/*.rb")).select { |path| File.read(path).include?("ENV") } assert_empty readers end end RUBY assert_equal "LINT-AS-TEST", kind("tests/no_env_test.rb") end def test_an_asserted_const_defined_is_structure write("tests/order_defined_test.rb", ORDER_DEFINED_TEST) assert_equal "STRUCTURE", kind("tests/order_defined_test.rb") end def test_an_eager_load_is_structure write("tests/eager_load_test.rb", <<~'RUBY') class EagerLoadTest < Minitest::Test def test_the_library_eager_loads Zeitwerk::Loader.eager_load_all end end RUBY assert_equal "STRUCTURE", kind("tests/eager_load_test.rb") end def test_a_file_exist_guard_outside_an_assertion_is_behavior write("tests/cache_test.rb", <<~'RUBY') class CacheTest < Minitest::Test def test_writes_the_cache_once path = File.join(Dir.mktmpdir, "cache") File.write(path, "x") unless File.exist?(path) assert_equal "x", File.read(path) end end RUBY assert_equal "BEHAVIOR", kind("tests/cache_test.rb") end def test_signal_text_inside_a_heredoc_is_behavior write("tests/cop_test.rb", <<~'RUBY') class CopTest < Minitest::Test SOURCE = <<~SRC assert File.exist?("lib/order.rb") SRC def test_flags_the_source assert_equal 1, Cop.new.offenses(SOURCE).size end end RUBY assert_equal "BEHAVIOR", kind("tests/cop_test.rb") end def test_a_behavior_test_and_a_structure_test_in_one_file_is_mixed write("tests/order_mixed_test.rb", <<~'RUBY') class OrderMixedTest < Minitest::Test def test_total_sums_the_lines assert_equal 3, Order.new([1, 2]).total end def test_order_is_defined assert Object.const_defined?(:Order) end end RUBY assert_equal "MIXED", kind("tests/order_mixed_test.rb") end def test_a_pure_require_pointer_is_structure write("tests/old_order_test.rb", "require_relative \"order_test\"\n") assert_equal "STRUCTURE", kind("tests/old_order_test.rb") end def test_a_file_with_no_test_method_outside_the_allowlist_is_mixed write("tests/shared_tracker_test.rb", <<~'RUBY') class SharedTrackerTest < Minitest::Test include TrackerContractTests end RUBY assert_equal "MIXED", kind("tests/shared_tracker_test.rb") end def test_an_allowlisted_file_is_contract_even_with_a_structure_signal write("tests/contract_seams.yml", "tracker:\n Tracker::Null: tests/null_contract_test.rb\n") write("tests/null_contract_test.rb", <<~'RUBY') class NullContractTest < Minitest::Test def test_answers_the_waiting_label_message assert Tracker::Null.new.respond_to?(:add_waiting_label) end end RUBY assert_equal "CONTRACT", kind("tests/null_contract_test.rb") end def test_an_allowlist_side_with_two_test_files_is_refused write("tests/contract_seams.yml", <<~YAML) tracker: Tracker::Null: - tests/a_contract_test.rb - tests/b_contract_test.rb YAML error = assert_raises(YamlFileError) { taxonomy } assert_includes error.message, "tracker" assert_includes error.message, "Tracker::Null" end def test_behavior_is_true_only_for_a_behavior_file write("tests/order_test.rb", ORDER_TEST) write("tests/order_defined_test.rb", ORDER_DEFINED_TEST) assert taxonomy.behavior?("tests/order_test.rb") refute taxonomy.behavior?("tests/order_defined_test.rb") end private def write(relative, text) path = File.join(@dir, relative) FileUtils.mkdir_p(File.dirname(path)) File.write(path, text) end def taxonomy = MutationTest::Taxonomy.for(@dir, test_path: "tests") def kind(relative) = taxonomy.kind(relative) end end ``` Failing test at `plugins/os-sdlc/tests/mutation_test/scope_test.rb`; red today because `scope.tests` also holds `tests/contract_seams.yml`. ```ruby # frozen_string_literal: true require_relative "../test_helper" module OsSdlc class MutationTestScopeTest < Minitest::Test Changes = Data.define(:code_paths, :test_paths, :unmapped_code_paths) Within = Data.define(:root) do def relative(path) = path.delete_prefix("#{root}/") end def test_from_keeps_only_ruby_test_files changes = Changes.new(code_paths: ["/t/lib/a.rb"], test_paths: ["/t/tests/a_test.rb", "/t/tests/contract_seams.yml"], unmapped_code_paths: []) scope = MutationTest::Scope.from(changes, "abc", within: Within.new(root: "/t")) assert_equal ["tests/a_test.rb"], scope.tests end end end ``` ## Out of scope - Using the kinds in the blind filters (#1636) and the overlap report (#1637). - Writing MIXED judgments or the signals that come from them (#1635 ledger). - A CLI or bin script, and a row in `Boot::Commands::TABLE`. - Listing every test file of a target; consumers pass the test files that Mutineer reports. - A CHARACTERIZATION signal; the plan rules none, so such tests fall to BEHAVIOR by elimination. - Per-test-method kinds in the output. - Splitting or deleting MIXED files. - Caching, flags, caps or timeouts. - `tests/system_model_viewer_procman_spec_contract_test.rb` in the seed allowlist; its tests at lines 12 to 30 check behavior, so CONTRACT would hide them. ## Origin - Trigger: plan `docs/research/mutation-testing/plan-v1-2026-11-08.md` (commit b90d8253), ruled by Jared 2026-11-07 and 2026-11-08 - Improvised this session: none - Chain: suite doubled in six weeks (757 test files, 3,572 test methods) ← suspected duplicated behavior and tests with no demonstrated value ← no mutation evidence in the gate. End: DESIGN (`docs/research/mutation-testing/plan-v1-2026-11-08.md`) - Root candidate: none found; searched mutation, mutineer, mutant, survivor, taxonomy, redundant test, blind test, duplicate, overlap, ledger - Where: plugins/os-sdlc (new classifier; placement by the architecture lens) - Session: ff703e34-8046-4b24-af68-6ef76d8aa338 - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os-plugins-os-sdlc/ff703e34-8046-4b24-af68-6ef76d8aa338.jsonl - Decomposed: session 87bc217f-61e2-4d4d-8470-9f3c8fd1ef43 (takedown relevance refused it, see #1642) VERDICT: CREATE

jared added the

plugin/os-sdlc

filed-by/user

category/code-change

labels 2026-11-08 15:24:32 +00:00

jared changed title from os-sdlc: per-ticket mutation survivor gate (mutation testing increment 1) to os-sdlc: test taxonomy classifier (mutation testing increment 2) 2026-11-08 15:24:53 +00:00

jared referenced this issue 2026-11-08 15:25:02 +00:00

os-sdlc: whole-codebase mutation pass (mutation testing increment 4) #1636

jared commented 2026-11-08 17:53:48 +00:00

Author

Owner

Copy link

Decompose routing (ticket-decompose, Phase Decide):

a. One change: a test-kind classifier that assigns each test file one kind (PROMPT-TEXT, LINT-AS-TEST, STRUCTURE, CONTRACT, BEHAVIOR, MIXED) from deterministic signals (plan Rulings line 36).
b. Owner: no existing object owns test kinds (no TestKind/taxonomy code in lib, bin, maps, hooks, tests, eval). New object; secret: the signals that decide a test file's kind.
c. Routed to /os-sdlc:ticket-takedown: step b names a new object. Takedown writes the Owner sketch.

Facts for takedown:

• Plan row 2 says "MISS bucket"; the binding Rulings say "MIXED". Use MIXED.
• Consumers: #1636 filters on BEHAVIOR; #1637 drops non-BEHAVIOR killers keyed by test_file; #1635 receives MIXED judgments.
• The plan does not rule the output format, the CLI name, or file versus method granularity. The lens (skills/lens-taxonomy-tests/SKILL.md:11) sorts per file and sends a two-kind file to a split, which supports a per-file key.
• No object lists every test file of a target; Suite::Candidates globs per code file.

Decompose routing (ticket-decompose, Phase Decide): a. One change: a test-kind classifier that assigns each test file one kind (PROMPT-TEXT, LINT-AS-TEST, STRUCTURE, CONTRACT, BEHAVIOR, MIXED) from deterministic signals (plan Rulings line 36). b. Owner: no existing object owns test kinds (no TestKind/taxonomy code in lib, bin, maps, hooks, tests, eval). New object; secret: the signals that decide a test file's kind. c. Routed to /os-sdlc:ticket-takedown: step b names a new object. Takedown writes the Owner sketch. Facts for takedown: - Plan row 2 says "MISS bucket"; the binding Rulings say "MIXED". Use MIXED. - Consumers: #1636 filters on BEHAVIOR; #1637 drops non-BEHAVIOR killers keyed by test_file; #1635 receives MIXED judgments. - The plan does not rule the output format, the CLI name, or file versus method granularity. The lens (skills/lens-taxonomy-tests/SKILL.md:11) sorts per file and sends a two-kind file to a split, which supports a per-file key. - No object lists every test file of a target; Suite::Candidates globs per code file.

jared commented 2026-11-08 17:55:14 +00:00

Author

Owner

Copy link

Takedown relevance: not relevant

Harm: A test-removal pass that runs without a test-kind classifier could delete or keep tests by mutation survival alone, including PROMPT-TEXT, LINT-AS-TEST, and STRUCTURE tests that the plan says follow their own keep rules. No such pass runs today, so no one is harmed now.
Protects: The plan's ruled rule that mutation-based removal applies only to BEHAVIOR tests, so the suite is not thinned by a gate that cannot tell test kinds apart.
Harm path: A future whole-codebase redundant-test pass (plan increment 4) would call the classifier before removing a test. Today the only mutation path is the per-ticket survivor gate in mutation_test/, which reports survivors and removes nothing, so no current command, hook, or skill reaches the missing classifier. (live: false)

Evidence:

• docs/research/mutation-testing/plan-v1-2026-11-08.md:3: Plan status reads 'ruled by Jared, not yet built'; the classifier is a planned increment, not shipped code.
• docs/research/mutation-testing/plan-v1-2026-11-08.md:12: Increment 2 (taxonomy classifier) is listed as a future output 'Kind per test'; increment 4 is the only consumer named.
• plugins/os-sdlc/lib/os_sdlc/mutation_test/mutineer_runner/report.rb:14: The existing report filter drops rows only by statement_removal containment; it has no test-kind input and no removal of tests.
• grep plugins/os-sdlc/lib for PROMPT-TEXT|LINT-AS-TEST|STRUCTURE|test_kind: No classifier or kind vocabulary exists in lib; the only 'classify' code is StepOutcome, Lint exit classification, and CRC file classification, none about test kinds.
• 1596f6ba: Most recent mutation commit in history fixes target root paths in the mutation test; it adds no classifier and deletes no tests.

Close candidate for Jared's batch ruling. The ticket stays open.

Takedown relevance: not relevant Harm: A test-removal pass that runs without a test-kind classifier could delete or keep tests by mutation survival alone, including PROMPT-TEXT, LINT-AS-TEST, and STRUCTURE tests that the plan says follow their own keep rules. No such pass runs today, so no one is harmed now. Protects: The plan's ruled rule that mutation-based removal applies only to BEHAVIOR tests, so the suite is not thinned by a gate that cannot tell test kinds apart. Harm path: A future whole-codebase redundant-test pass (plan increment 4) would call the classifier before removing a test. Today the only mutation path is the per-ticket survivor gate in mutation_test/, which reports survivors and removes nothing, so no current command, hook, or skill reaches the missing classifier. (live: false) Evidence: - docs/research/mutation-testing/plan-v1-2026-11-08.md:3: Plan status reads 'ruled by Jared, not yet built'; the classifier is a planned increment, not shipped code. - docs/research/mutation-testing/plan-v1-2026-11-08.md:12: Increment 2 (taxonomy classifier) is listed as a future output 'Kind per test'; increment 4 is the only consumer named. - plugins/os-sdlc/lib/os_sdlc/mutation_test/mutineer_runner/report.rb:14: The existing report filter drops rows only by statement_removal containment; it has no test-kind input and no removal of tests. - grep plugins/os-sdlc/lib for PROMPT-TEXT|LINT-AS-TEST|STRUCTURE|test_kind: No classifier or kind vocabulary exists in lib; the only 'classify' code is StepOutcome, Lint exit classification, and CRC file classification, none about test kinds. - 1596f6ba: Most recent mutation commit in history fixes target root paths in the mutation test; it adds no classifier and deletes no tests. Close candidate for Jared's batch ruling. The ticket stays open.

jared referenced this issue 2026-11-08 17:56:42 +00:00

Takedown relevance judge rejects ruled feature tickets as not relevant #1642

jared added the

ready-for-agent

label 2026-11-08 18:07:31 +00:00

jared commented 2026-11-08 18:08:01 +00:00

Author

Owner

Copy link

Work started: /os-sdlc:implement on branch ticket-1634 (worktree .claude/worktrees/ticket-1634), map poodr-ticket-implementation, target os-sdlc. First live run of the survivor-check and survivor-tests nodes.

Work started: /os-sdlc:implement on branch ticket-1634 (worktree .claude/worktrees/ticket-1634), map poodr-ticket-implementation, target os-sdlc. First live run of the survivor-check and survivor-tests nodes.

jared referenced this issue 2026-11-08 18:28:34 +00:00

programmer-repair gate omits the required-work step that sent it the failure #1643

jared commented 2026-11-08 18:28:45 +00:00

Author

Owner

Copy link

Resolution

Done: MutationTest::Taxonomy gives each target-relative test file one kind (BEHAVIOR, CONTRACT, STRUCTURE, PROMPT-TEXT, LINT-AS-TEST, MIXED) from Prism signals per test method, with the CONTRACT allowlist tests/contract_seams.yml; Scope.from passes only _test.rb paths to Mutineer. All 15 acceptance criteria verified by behavior-verifier.

Evidence: Implementation 266 on branch ticket-1634: commit 875b5785 (Runner commit, 11 files), history row 9deaf01a. Suite pass 54.0 s; survivor-check found survivors, survivor-tests killed them, survivor-check pass (3.6 s); contract-auditor pass. First live run of the survivor nodes.

Follow-ups: #1642 takedown relevance judge rejects ruled feature tickets (captured, skeptic CREATE). #1643 programmer-repair gate omits required-work (captured, skeptic CREATE).

## Resolution **Done:** MutationTest::Taxonomy gives each target-relative test file one kind (BEHAVIOR, CONTRACT, STRUCTURE, PROMPT-TEXT, LINT-AS-TEST, MIXED) from Prism signals per test method, with the CONTRACT allowlist tests/contract_seams.yml; Scope.from passes only _test.rb paths to Mutineer. All 15 acceptance criteria verified by behavior-verifier. **Evidence:** Implementation 266 on branch ticket-1634: commit 875b5785 (Runner commit, 11 files), history row 9deaf01a. Suite pass 54.0 s; survivor-check found survivors, survivor-tests killed them, survivor-check pass (3.6 s); contract-auditor pass. First live run of the survivor nodes. **Follow-ups:** #1642 takedown relevance judge rejects ruled feature tickets (captured, skeptic CREATE). #1643 programmer-repair gate omits required-work (captured, skeptic CREATE). <!-- os-backlog kind: resolution -->

jared closed this issue 2026-11-08 18:28:46 +00:00

jared referenced this issue from a commit 2026-11-08 18:29:55 +00:00

feat(os-sdlc): POODR audit passed: Taxonomy owns the file-kind rule, Signals the three signal rules, TestFile the Prism reach, Seams the allowlist; Scope.from filters _test.rb; R1-R6 present. (#1634)

jared referenced this issue from a commit 2026-11-08 18:29:55 +00:00

docs(os-sdlc): history row for #1634 taxonomy classifier

jared referenced this issue 2026-11-08 19:15:13 +00:00

os-backlog CLI has no label-add or label-remove command #867

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

ticket-1337

No results found.

Labels

Clear labels   

P0

Priority 0 — drop everything

  

P1

Priority 1 — next up

  

P2

Priority 2 — normal

  

P3

Priority 3 — someday

  

bug

Something is broken

  

category/code-change

Ticket category

  

category/cross-repo

Ticket category

  

category/eval-run

Ticket category

  

category/investigation

Ticket category

  

category/no-test-target

Ticket category

  

category/owner-ruling

Ticket category

  

category/prose-artifact

Ticket category

  

category/recurring-chore

Ticket category

  

category/structure

  

category/umbrella

Ticket category

  

crc-refactor

CRC refactor run work record; exempt from the ticket-skeptic

  

create

Ticket creates something new

  

delete

Ticket removes something

  

enhancement

New feature or improvement

  

filed-by/agent

  

filed-by/user

  

frozen

Pre-reset backlog freeze 2026-08-16; do not work unless rediscovered by a live run

  

lint-rule

Ticket proposes/modifies a lint rule or cop

  

needs-info

Waiting on reporter for more information

  

needs-triage

Maintainer needs to evaluate this issue

  

next

Human-curated next up — AI never applies or removes

  

plugin/cc-architect

Scope: the cc-architect plugin

  

plugin/os

Scope: the os plugin

  

plugin/os-adr

Scope: the os-adr plugin

  

plugin/os-aidd-lint

Scope: the os-aidd-lint plugin

  

plugin/os-backlog

Scope: the os-backlog plugin

  

plugin/os-context

Scope: the os-context plugin

  

plugin/os-doc-hygiene

Scope: the os-doc-hygiene plugin

  

plugin/os-sdlc

Scope: the os-sdlc plugin

  

plugin/os-vault

Scope: the os-vault plugin

  

project/cc-os

Parent-scope: cc-os repo-wide, not specific to one plugin

  

ready-for-agent

Triage complete; an agent can pick this up

  

ready-for-human

Human-owned; never picked up autonomously

  

recurring

Recurring flavor: occurrence outcomes commented, never closed by AI

  

review

Shipped semi work awaiting human sign-off

  

rule-proposal

A rule change that survived the gate; Jared reads these when he chooses

  

update

Ticket updates something existing

  

waiting

Blocked; blocker named in a comment

  

wayfinder:grilling

  

wayfinder:map

  

wayfinder:map

wayfinder map ticket

  

wayfinder:research

  

wayfinder:task

  

wayfinder:task

wayfinder map child: task

  

wontfix

Will not be actioned

  

worklist/deviations

Deviation incidents from skill/pipeline runs — evidence pile for correcting the owning artifact

  

worklist/lint-rule

Proposed os-sdlc or os-aidd-lint rule

  

worklist/new-implement-build

Updated implementation harness (ast-grep + stepper)

No labels

P0

P1

P2

P3

bug

category/code-change

category/cross-repo

category/eval-run

category/investigation

category/no-test-target

category/owner-ruling

category/prose-artifact

category/recurring-chore

category/structure

category/umbrella

crc-refactor

create

delete

enhancement

filed-by/agent

filed-by/user

frozen

lint-rule

needs-info

needs-triage

next

plugin/cc-architect

plugin/os

plugin/os-adr

plugin/os-aidd-lint

plugin/os-backlog

plugin/os-context

plugin/os-doc-hygiene

plugin/os-sdlc

plugin/os-vault

project/cc-os

ready-for-agent

ready-for-human

recurring

review

rule-proposal

update

waiting

wayfinder:grilling

wayfinder:map

wayfinder:map

wayfinder:research

wayfinder:task

wayfinder:task

wontfix

worklist/deviations

worklist/lint-rule

worklist/new-implement-build

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

jared/cc-os#1634

No description provided.