Cop: flag tests that read skills/ or agents/ *.md and string-match their prose (PROMPT-TEXT) #1444

New issue

Open

opened 2026-10-27 20:40:30 +00:00 by jared · 0 comments

jared commented

2026-10-27 20:40:30 +00:00

Owner

Problem

Tests that read a skills/ or agents/ *.md file and string-match its prose pin prompt wording.
The lens lens-taxonomy-tests sorts them as PROMPT-TEXT with lifetime "never". Evals judge prompts, not string matches.
Sdlc/Structural/RawFileContentAssertion catches them only by accident, and its fix text (load the file through the owning object's public reader) is the wrong fix. The right fix is to delete the test.
The redesign in #1008 drops this accidental catch, so it needs its own cop.

Observed

Deleted example, plugins/os-backlog/tests/category_documentation_test.rb (removed in fb040a58; quoted from git show fb040a58~1:plugins/os-backlog/tests/category_documentation_test.rb):

def test_capture_skill_names_crc_refactor_as_the_skeptic_exemption_for_run_work_records
  text = File.read(File.join(ROOT, "skills/capture/SKILL.md"))

  assert_match(/`crc-refactor`.*run work record/m, text)
  assert_match(/split ticket.*past the cap.*ticket-skeptic/m, text)
end

A second shape the cop does not flag today (same file, lines 12-22):

text = File.read(File.join(ROOT, "skills/route/SKILL.md"))

assert_empty(labels.reject { |label| text.include?(label) })

The new cop must flag both shapes.

Reproduce

n/a: design ticket for a new cop; the deleted tests in fb040a58~1 are the positive cases.

Approach

Add a cop under Sdlc/Structural that flags a test which reads a path under skills/ or agents/ ending in .md and asserts on its text with assert_match, assert_includes, include?, or =~. Its message says to delete the test, not to use a reader. Scope: plugin/os-sdlc.

Required work: none beyond the acceptance criteria.

Acceptance criteria

Given a test that does File.read on skills/*/SKILL.md and calls assert_match on the text, when the cop runs, then it flags the call and the message says to delete the test.
Given a test that reads skills/*/SKILL.md and calls labels.reject { text.include?(_1) }, when the cop runs, then it flags the call.
Given a test that reads an agents/*.md file and string-matches it, when the cop runs, then it flags the call.
Given a test that reads a file outside skills/ and agents/, when the cop runs, then it does not flag the call.

Origin

Trigger: Jared's ruling to delete the PROMPT-TEXT tests that RawFileContentAssertion flagged in os-backlog.
Improvised this session: none
Chain: RawFileContentAssertion flags prompt-text tests by accident ← the cop matches any File.read plus string match ← DESIGN (lens-taxonomy-tests names PROMPT-TEXT; no cop owns it)
Root candidate: none found; searched PROMPT-TEXT, RawFileContentAssertion, lens-taxonomy-tests, string-match, SKILL.md, skills/, agents/, prose (only #1008 matched; it is the parent)
Where: plugins/os-sdlc (Sdlc/Structural cops)
Session: 2d5f10d7-c85d-4b4a-b51d-5fc46b749416
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/2d5f10d7-c85d-4b4a-b51d-5fc46b749416.jsonl

VERDICT: CREATE

## Problem Tests that read a `skills/` or `agents/` `*.md` file and string-match its prose pin prompt wording. The lens `lens-taxonomy-tests` sorts them as PROMPT-TEXT with lifetime "never". Evals judge prompts, not string matches. `Sdlc/Structural/RawFileContentAssertion` catches them only by accident, and its fix text (load the file through the owning object's public reader) is the wrong fix. The right fix is to delete the test. The redesign in #1008 drops this accidental catch, so it needs its own cop. ## Observed Deleted example, `plugins/os-backlog/tests/category_documentation_test.rb` (removed in fb040a58; quoted from `git show fb040a58~1:plugins/os-backlog/tests/category_documentation_test.rb`): ```ruby def test_capture_skill_names_crc_refactor_as_the_skeptic_exemption_for_run_work_records text = File.read(File.join(ROOT, "skills/capture/SKILL.md")) assert_match(/`crc-refactor`.*run work record/m, text) assert_match(/split ticket.*past the cap.*ticket-skeptic/m, text) end ``` A second shape the cop does not flag today (same file, lines 12-22): ```ruby text = File.read(File.join(ROOT, "skills/route/SKILL.md")) assert_empty(labels.reject { |label| text.include?(label) }) ``` The new cop must flag both shapes. ## Reproduce n/a: design ticket for a new cop; the deleted tests in fb040a58~1 are the positive cases. ## Approach Add a cop under `Sdlc/Structural` that flags a test which reads a path under `skills/` or `agents/` ending in `.md` and asserts on its text with `assert_match`, `assert_includes`, `include?`, or `=~`. Its message says to delete the test, not to use a reader. Scope: plugin/os-sdlc. Required work: none beyond the acceptance criteria. ## Acceptance criteria 1. Given a test that does `File.read` on `skills/*/SKILL.md` and calls `assert_match` on the text, when the cop runs, then it flags the call and the message says to delete the test. 2. Given a test that reads `skills/*/SKILL.md` and calls `labels.reject { text.include?(_1) }`, when the cop runs, then it flags the call. 3. Given a test that reads an `agents/*.md` file and string-matches it, when the cop runs, then it flags the call. 4. Given a test that reads a file outside `skills/` and `agents/`, when the cop runs, then it does not flag the call. ## Origin - Trigger: Jared's ruling to delete the PROMPT-TEXT tests that RawFileContentAssertion flagged in os-backlog. - Improvised this session: none - Chain: RawFileContentAssertion flags prompt-text tests by accident ← the cop matches any File.read plus string match ← DESIGN (lens-taxonomy-tests names PROMPT-TEXT; no cop owns it) - Root candidate: none found; searched PROMPT-TEXT, RawFileContentAssertion, lens-taxonomy-tests, string-match, SKILL.md, skills/, agents/, prose (only #1008 matched; it is the parent) - Where: plugins/os-sdlc (Sdlc/Structural cops) - Session: 2d5f10d7-c85d-4b4a-b51d-5fc46b749416 - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/2d5f10d7-c85d-4b4a-b51d-5fc46b749416.jsonl VERDICT: CREATE

jared added the

plugin/os-sdlc

filed-by/user

category/code-change

labels

2026-10-27 20:40:30 +00:00

jared referenced this issue

2026-10-27 20:40:34 +00:00

Redesign RawFileContentAssertion into OwnedFileReadInTest (owner-index cop) #1008