jared/cc-os
Watch
1
0
Fork
You've already forked cc-os
 0
Code Issues 378 Pull requests Projects Releases Packages Wiki Activity

Stop takedown verifying defects already fixed on main #998

New issue
Open
opened 2026-10-12 13:18:45 +00:00 by jared · 6 comments
jared commented 2026-10-12 13:18:45 +00:00
Owner
Copy link

Problem

A takedown can verify a defect that is already fixed on main.
Two cards fail together: the research step accepts a reproduction that stubs the behavior in question, and the relevance judge misreads a fix as the defect.

Design

/home/jared/dev/cc-os/.claude/worktrees/prism-architect/docs/takedown/998-20261016-094801/final-design.json

Approach

Make Takedown::Research#verify! and Takedown::Relevance#verify! refuse on required structured reply fields (reproduction.stubs with a questioned flag; fix_checks with current_code_is_fix) instead of trusting reply shape, with real red tests this time.

Required work:

  1. Write plugins/os-sdlc/eval/ticket-takedown/lib/takedown/model/research.rb
  2. Write plugins/os-sdlc/eval/ticket-takedown/harness/schemas/research.json
  3. Write plugins/os-sdlc/eval/ticket-takedown/harness/schemas/relevance.json
  4. Write plugins/os-sdlc/eval/ticket-takedown/harness/cards/researcher.md
  5. Write plugins/os-sdlc/eval/ticket-takedown/harness/cards/relevance-judge.md
  6. Write plugins/os-sdlc/eval/ticket-takedown/test/run_support.rb
  7. Write plugins/os-sdlc/eval/ticket-takedown/test/model/research_test.rb
  8. Write plugins/os-sdlc/eval/ticket-takedown/test/model/harness_files_test.rb
  9. Write plugins/os-sdlc/eval/ticket-takedown/lib/takedown/model/relevance.rb
  10. Write test plugins/os-sdlc/eval/ticket-takedown/test/model/research_stub_test.rb
  11. Write test plugins/os-sdlc/eval/ticket-takedown/test/model/relevance_fix_check_test.rb

Acceptance criteria

  • ran true, stubs []: accepted
  • ran true, stub with questioned true: refused: research: reproduction stubs the questioned behavior
  • ran true, stub with questioned false: accepted
  • stubs missing or not an array: refused, same message
  • ran false with location and reason, stubs []: accepted
  • harm_path_live true, a fix check true: refused: relevance: current code is the fix
  • fix check true, harm_path_live false, verdict not_relevant: accepted
  • fix_checks key missing: KeyError, loud failure

Owner sketch

Responsibility Owner
Report each fake the reproduction used and whether it replaces the questioned behavior researcher reply reproduction.stubs (schema research.json, card researcher.md)
Refuse a reproduction that stubs the questioned behavior or omits stubs Takedown::Research#verify! (check unstubbed?)
Report, per history commit touching the harm path, whether the current code is that fix relevance reply fix_checks (schema relevance.json, card relevance-judge.md)
Refuse harm_path_live true when a fix check says the current code is the fix; fail loudly when fix_checks is missing Takedown::Relevance#verify!

plugins/os-sdlc/eval/ticket-takedown/test/model/research_stub_test.rb

# frozen_string_literal: true

require_relative "../run_support"

class ResearchStubTest < TakedownTest
  include RunSupport

  REFUSAL = "research: reproduction stubs the questioned behavior"

  def refusal(reproduction)
    data = RunSupport::RESEARCH.merge("reproduction" => reproduction)
    assert_raises(Takedown::Failure) { Takedown::Research.new(data).verify! }.message
  end

  def base_reproduction = RunSupport::RESEARCH.fetch("reproduction")

  def test_a_questioned_stub_is_refused
    stubbed = base_reproduction.merge("stubs" => [{ "target" => "repo.labels.list", "questioned" => true }])
    assert_equal REFUSAL, refusal(stubbed)
  end

  def test_a_reproduction_without_stubs_is_refused
    assert_equal REFUSAL, refusal(base_reproduction.reject { |key, _| key == "stubs" })
  end

  def test_the_research_schema_requires_stubs_and_the_card_defines_them
    profile = Takedown::StepProfile.all.fetch("research")
    schema = JSON.parse(profile.schema_text)
    assert_includes schema.dig("properties", "reproduction", "required"), "stubs"
    assert_includes profile.card_text, "`stubs`"
  end
end

plugins/os-sdlc/eval/ticket-takedown/test/model/relevance_fix_check_test.rb

# frozen_string_literal: true

require_relative "../run_support"

class RelevanceFixCheckTest < TakedownTest
  include RunSupport

  FIX = [{ "commit" => "0053cf03", "file" => "a.rb", "current_code_is_fix" => true,
           "finding" => "the limit call is the fix" }].freeze

  def test_a_live_verdict_is_refused_when_the_current_code_is_the_fix
    reply = RunSupport::RELEVANCE.merge("fix_checks" => FIX)
    error = assert_raises(Takedown::Failure) { Takedown::Relevance.new(reply).verify! }
    assert_equal "relevance: current code is the fix", error.message
  end

  def test_a_reply_without_fix_checks_fails_loudly
    reply = RunSupport::RELEVANCE.reject { |key, _| key == "fix_checks" }
    assert_raises(KeyError) { Takedown::Relevance.new(reply).verify! }
  end

  def test_a_plugin_absent_reply_carries_an_empty_fix_check_list
    reply = Takedown::Relevance.plugin_absent("os-gone")
    assert_empty reply.to_h.fetch("fix_checks")
    assert_equal reply, reply.verify!
  end
end
## Problem A takedown can verify a defect that is already fixed on main. Two cards fail together: the research step accepts a reproduction that stubs the behavior in question, and the relevance judge misreads a fix as the defect. ## Design /home/jared/dev/cc-os/.claude/worktrees/prism-architect/docs/takedown/998-20261016-094801/final-design.json ## Approach Make Takedown::Research#verify! and Takedown::Relevance#verify! refuse on required structured reply fields (reproduction.stubs with a questioned flag; fix_checks with current_code_is_fix) instead of trusting reply shape, with real red tests this time. Required work: 1. Write `plugins/os-sdlc/eval/ticket-takedown/lib/takedown/model/research.rb` 2. Write `plugins/os-sdlc/eval/ticket-takedown/harness/schemas/research.json` 3. Write `plugins/os-sdlc/eval/ticket-takedown/harness/schemas/relevance.json` 4. Write `plugins/os-sdlc/eval/ticket-takedown/harness/cards/researcher.md` 5. Write `plugins/os-sdlc/eval/ticket-takedown/harness/cards/relevance-judge.md` 6. Write `plugins/os-sdlc/eval/ticket-takedown/test/run_support.rb` 7. Write `plugins/os-sdlc/eval/ticket-takedown/test/model/research_test.rb` 8. Write `plugins/os-sdlc/eval/ticket-takedown/test/model/harness_files_test.rb` 9. Write `plugins/os-sdlc/eval/ticket-takedown/lib/takedown/model/relevance.rb` 10. Write test `plugins/os-sdlc/eval/ticket-takedown/test/model/research_stub_test.rb` 11. Write test `plugins/os-sdlc/eval/ticket-takedown/test/model/relevance_fix_check_test.rb` ## Acceptance criteria - ran true, stubs []: accepted - ran true, stub with questioned true: refused: research: reproduction stubs the questioned behavior - ran true, stub with questioned false: accepted - stubs missing or not an array: refused, same message - ran false with location and reason, stubs []: accepted - harm_path_live true, a fix check true: refused: relevance: current code is the fix - fix check true, harm_path_live false, verdict not_relevant: accepted - fix_checks key missing: KeyError, loud failure ## Owner sketch | Responsibility | Owner | |---|---| | Report each fake the reproduction used and whether it replaces the questioned behavior | researcher reply `reproduction.stubs` (schema research.json, card researcher.md) | | Refuse a reproduction that stubs the questioned behavior or omits `stubs` | `Takedown::Research#verify!` (check `unstubbed?`) | | Report, per history commit touching the harm path, whether the current code is that fix | relevance reply `fix_checks` (schema relevance.json, card relevance-judge.md) | | Refuse `harm_path_live` true when a fix check says the current code is the fix; fail loudly when `fix_checks` is missing | `Takedown::Relevance#verify!` | `plugins/os-sdlc/eval/ticket-takedown/test/model/research_stub_test.rb` ```ruby # frozen_string_literal: true require_relative "../run_support" class ResearchStubTest < TakedownTest include RunSupport REFUSAL = "research: reproduction stubs the questioned behavior" def refusal(reproduction) data = RunSupport::RESEARCH.merge("reproduction" => reproduction) assert_raises(Takedown::Failure) { Takedown::Research.new(data).verify! }.message end def base_reproduction = RunSupport::RESEARCH.fetch("reproduction") def test_a_questioned_stub_is_refused stubbed = base_reproduction.merge("stubs" => [{ "target" => "repo.labels.list", "questioned" => true }]) assert_equal REFUSAL, refusal(stubbed) end def test_a_reproduction_without_stubs_is_refused assert_equal REFUSAL, refusal(base_reproduction.reject { |key, _| key == "stubs" }) end def test_the_research_schema_requires_stubs_and_the_card_defines_them profile = Takedown::StepProfile.all.fetch("research") schema = JSON.parse(profile.schema_text) assert_includes schema.dig("properties", "reproduction", "required"), "stubs" assert_includes profile.card_text, "`stubs`" end end ``` `plugins/os-sdlc/eval/ticket-takedown/test/model/relevance_fix_check_test.rb` ```ruby # frozen_string_literal: true require_relative "../run_support" class RelevanceFixCheckTest < TakedownTest include RunSupport FIX = [{ "commit" => "0053cf03", "file" => "a.rb", "current_code_is_fix" => true, "finding" => "the limit call is the fix" }].freeze def test_a_live_verdict_is_refused_when_the_current_code_is_the_fix reply = RunSupport::RELEVANCE.merge("fix_checks" => FIX) error = assert_raises(Takedown::Failure) { Takedown::Relevance.new(reply).verify! } assert_equal "relevance: current code is the fix", error.message end def test_a_reply_without_fix_checks_fails_loudly reply = RunSupport::RELEVANCE.reject { |key, _| key == "fix_checks" } assert_raises(KeyError) { Takedown::Relevance.new(reply).verify! } end def test_a_plugin_absent_reply_carries_an_empty_fix_check_list reply = Takedown::Relevance.plugin_absent("os-gone") assert_empty reply.to_h.fetch("fix_checks") assert_equal reply, reply.verify! end end ```
jared commented 2026-10-12 14:30:48 +00:00
Author
Owner
Copy link

Takedown relevance: needs a ruling

Harm: A takedown research-only run can report verified: true for a defect that a live command shows is already fixed on main, so a person spends effort on a ticket that no longer needs work.
Protects: The takedown gate's purpose of stopping obsolete tickets before research and design money is spent, and of keeping its verdicts trustworthy for Jared's close and ruling decisions.
Harm path: bin/takedown-run <ticket> --research-only runs setup and then the research step, which calls Research#verify! and accepts any reproduction with ran: true and a non-blank command and output, so a stub that encodes the ticket's claim passes. (live: true)

Evidence:

  • plugins/os-sdlc/eval/ticket-takedown/bin/takedown-run: The executable exists in the plugin, so the research-only entry point is present.
  • plugins/os-sdlc/eval/ticket-takedown/lib/takedown/controller/run_cli.rb:63: The --research-only flag is parsed and sets research_only in the run request.
  • plugins/os-sdlc/eval/ticket-takedown/lib/takedown/command/research_step.rb:21: ResearchStep calls Research.new(run.replies.fetch(name)).verify! on the researcher reply.
  • plugins/os-sdlc/eval/ticket-takedown/lib/takedown/model/research.rb:14-36: verify! checks occurs?, shown?, and names_file?; reproduced? only tests ran == true with non-blank command and output_excerpt, so it never checks that the reproduction exercises the real collaborator.
  • plugins/os-sdlc/eval/ticket-takedown/lib/takedown/command/step_list.rb:38: The step list maps the research step name to ResearchStep, so the check runs in both live and research-only flows.
  • plugins/os-backlog/lib/backlog/forgejo_backend.rb:21: labels_list calls labels.list(limit: PAGE_LIMIT), the fix from 0053cf03, which shows the defect is fixed on main.
  • .takedown/history.txt:175-177: Commit 0053cf03 (2026-10-11) changed forgejo_backend.rb, which confirms the fix is on main at this checkout.

Conflicting ADRs:

  • ADR-0288: ADR-0288 states that verify! checks the shape of the reproduction and not its truth; criterion 2 requires verify! to refuse a reproduction whose key collaborator is a stub, which is a truth check and reverses that stated consequence.

The ticket is waiting for Jared's ruling on the ADRs above.

Takedown relevance: needs a ruling Harm: A takedown research-only run can report `verified: true` for a defect that a live command shows is already fixed on main, so a person spends effort on a ticket that no longer needs work. Protects: The takedown gate's purpose of stopping obsolete tickets before research and design money is spent, and of keeping its verdicts trustworthy for Jared's close and ruling decisions. Harm path: `bin/takedown-run <ticket> --research-only` runs setup and then the research step, which calls `Research#verify!` and accepts any reproduction with `ran: true` and a non-blank command and output, so a stub that encodes the ticket's claim passes. (live: true) Evidence: - plugins/os-sdlc/eval/ticket-takedown/bin/takedown-run: The executable exists in the plugin, so the research-only entry point is present. - plugins/os-sdlc/eval/ticket-takedown/lib/takedown/controller/run_cli.rb:63: The `--research-only` flag is parsed and sets research_only in the run request. - plugins/os-sdlc/eval/ticket-takedown/lib/takedown/command/research_step.rb:21: ResearchStep calls `Research.new(run.replies.fetch(name)).verify!` on the researcher reply. - plugins/os-sdlc/eval/ticket-takedown/lib/takedown/model/research.rb:14-36: verify! checks occurs?, shown?, and names_file?; reproduced? only tests ran == true with non-blank command and output_excerpt, so it never checks that the reproduction exercises the real collaborator. - plugins/os-sdlc/eval/ticket-takedown/lib/takedown/command/step_list.rb:38: The step list maps the research step name to ResearchStep, so the check runs in both live and research-only flows. - plugins/os-backlog/lib/backlog/forgejo_backend.rb:21: labels_list calls labels.list(limit: PAGE_LIMIT), the fix from 0053cf03, which shows the defect is fixed on main. - .takedown/history.txt:175-177: Commit 0053cf03 (2026-10-11) changed forgejo_backend.rb, which confirms the fix is on main at this checkout. Conflicting ADRs: - ADR-0288: ADR-0288 states that verify! checks the shape of the reproduction and not its truth; criterion 2 requires verify! to refuse a reproduction whose key collaborator is a stub, which is a truth check and reverses that stated consequence. The ticket is waiting for Jared's ruling on the ADRs above.
jared commented 2026-10-16 12:17:46 +00:00
Author
Owner
Copy link

ADR-0288 was removed in commit c5240610 (2026-10-15, "docs(adr): challenge every ADR with the reversal judge (ADR-0296)"). Under Jared's rule of 2026-10-16, the ruling question under ADR-0288 is void. Kept open: the defect is independent of the ADR. Research#verify! (plugins/os-sdlc/eval/ticket-takedown/lib/takedown/model/research.rb) has no check that rejects a reproduction whose key collaborator is a stub, and Relevance#verify! only checks that evidence exists. Work remains for acceptance criteria 1-3. The waiting label no longer has a reason; no tool here removes labels, so Jared or main should remove it.

ADR-0288 was removed in commit c5240610 (2026-10-15, "docs(adr): challenge every ADR with the reversal judge (ADR-0296)"). Under Jared's rule of 2026-10-16, the ruling question under ADR-0288 is void. Kept open: the defect is independent of the ADR. Research#verify! (plugins/os-sdlc/eval/ticket-takedown/lib/takedown/model/research.rb) has no check that rejects a reproduction whose key collaborator is a stub, and Relevance#verify! only checks that evidence exists. Work remains for acceptance criteria 1-3. The waiting label no longer has a reason; no tool here removes labels, so Jared or main should remove it.
jared commented 2026-10-16 13:17:41 +00:00
Author
Owner
Copy link

Work started 2026-10-16: live takedown run (session 7, branch prism-architect). The waiting label is stale (ADR-0288 removed in c5240610).

Work started 2026-10-16: live takedown run (session 7, branch prism-architect). The waiting label is stale (ADR-0288 removed in c5240610).
jared commented 2026-10-16 13:42:37 +00:00
Author
Owner
Copy link

Takedown run 998-20261016-091741 failed at green-suite: polish rounds traded old lint offenses in command/relevance_step.rb (present at HEAD) for new ones until the stall rule fired. Cleared that debt in ab429cf5. A repo-wide lint-sweep ticket was proposed and dropped by ticket-skeptic: no recorded requirement; fix lint in the files a trial ticket touches instead. Rerun follows.

Takedown run 998-20261016-091741 failed at green-suite: polish rounds traded old lint offenses in command/relevance_step.rb (present at HEAD) for new ones until the stall rule fired. Cleared that debt in ab429cf5. A repo-wide lint-sweep ticket was proposed and dropped by ticket-skeptic: no recorded requirement; fix lint in the files a trial ticket touches instead. Rerun follows.
jared commented 2026-10-16 14:05:09 +00:00
Author
Owner
Copy link

Implementation started: /os-sdlc:implement 998 takedown-implement on branch prism-architect (design docs/takedown/998-20261016-094801; grade removes_defect false, reported not gated per the 2026-10-16 ruling).

Implementation started: /os-sdlc:implement 998 takedown-implement on branch prism-architect (design docs/takedown/998-20261016-094801; grade removes_defect false, reported not gated per the 2026-10-16 ruling).
jared commented 2026-10-16 14:09:31 +00:00
Author
Owner
Copy link

Resolution

Done: Research#verify! requires reproduction.stubs and refuses a reproduction that stubs the questioned behavior; Relevance#verify! requires fix_checks and refuses a live verdict when a check says the current code is the fix; schemas and cards updated. Built from takedown run 998-20261016-094801 (2 rounds, $1.23) by implementation 49, then hand-finished: the behavior-verifier found that a not-live reply without fix_checks passed, fixed test-first. Implementation 49 abandoned (its reason text says 'in 3c'; read 'in 642a5b11').

Evidence: Commit 642a5b11. Takedown eval 607 runs, 0 failures; os-sdlc 1571 runs, 0 failures; rubocop clean on touched files. New red-then-green test test_a_not_live_reply_without_fix_checks_fails_loudly.

Follow-ups: none; the remaining agent-reported-flag gap is the review question above

Needs review: Grade for the final design: removes_defect false, cause_remains true. The stubs[].questioned and fix_checks[].current_code_is_fix flags are agent-reported; a mislabelled stub or misread fix still passes. Original criterion 1 (a live command shows the defect is already fixed, so verified must not be true) has no live re-run check: the scratch sandbox has no network. Decide whether that remaining gap needs its own ticket.

## Resolution **Done:** Research#verify! requires reproduction.stubs and refuses a reproduction that stubs the questioned behavior; Relevance#verify! requires fix_checks and refuses a live verdict when a check says the current code is the fix; schemas and cards updated. Built from takedown run 998-20261016-094801 (2 rounds, $1.23) by implementation 49, then hand-finished: the behavior-verifier found that a not-live reply without fix_checks passed, fixed test-first. Implementation 49 abandoned (its reason text says 'in 3c'; read 'in 642a5b11'). **Evidence:** Commit 642a5b11. Takedown eval 607 runs, 0 failures; os-sdlc 1571 runs, 0 failures; rubocop clean on touched files. New red-then-green test test_a_not_live_reply_without_fix_checks_fails_loudly. **Follow-ups:** none; the remaining agent-reported-flag gap is the review question above Needs review: Grade for the final design: removes_defect false, cause_remains true. The stubs[].questioned and fix_checks[].current_code_is_fix flags are agent-reported; a mislabelled stub or misread fix still passes. Original criterion 1 (a live command shows the defect is already fixed, so verified must not be true) has no live re-run check: the scratch sandbox has no network. Decide whether that remaining gap needs its own ticket.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
ticket-1337
No results found.
Labels
Clear labels   
P0
Priority 0 — drop everything

  
P1
Priority 1 — next up

  
P2
Priority 2 — normal

  
P3
Priority 3 — someday

  
bug
Something is broken

  
category/code-change
Ticket category

  
category/cross-repo
Ticket category

  
category/eval-run
Ticket category

  
category/investigation
Ticket category

  
category/no-test-target
Ticket category

  
category/owner-ruling
Ticket category

  
category/prose-artifact
Ticket category

  
category/recurring-chore
Ticket category

  
category/structure

   
category/umbrella
Ticket category

  
crc-refactor
CRC refactor run work record; exempt from the ticket-skeptic

  
create
Ticket creates something new

  
delete
Ticket removes something

  
enhancement
New feature or improvement

  
filed-by/agent

   
filed-by/user

   
frozen
Pre-reset backlog freeze 2026-08-16; do not work unless rediscovered by a live run

  
lint-rule
Ticket proposes/modifies a lint rule or cop

  
needs-info
Waiting on reporter for more information

  
needs-triage
Maintainer needs to evaluate this issue

  
next
Human-curated next up — AI never applies or removes

  
plugin/cc-architect
Scope: the cc-architect plugin

  
plugin/os
Scope: the os plugin

  
plugin/os-adr
Scope: the os-adr plugin

  
plugin/os-aidd-lint
Scope: the os-aidd-lint plugin

  
plugin/os-backlog
Scope: the os-backlog plugin

  
plugin/os-context
Scope: the os-context plugin

  
plugin/os-doc-hygiene
Scope: the os-doc-hygiene plugin

  
plugin/os-sdlc
Scope: the os-sdlc plugin

  
plugin/os-vault
Scope: the os-vault plugin

  
project/cc-os
Parent-scope: cc-os repo-wide, not specific to one plugin

  
ready-for-agent
Triage complete; an agent can pick this up

  
ready-for-human
Human-owned; never picked up autonomously

  
recurring
Recurring flavor: occurrence outcomes commented, never closed by AI

  
review
Shipped semi work awaiting human sign-off

  
rule-proposal
A rule change that survived the gate; Jared reads these when he chooses

  
update
Ticket updates something existing

  
waiting
Blocked; blocker named in a comment

  
wayfinder:grilling

   
wayfinder:map

   
wayfinder:map
wayfinder map ticket

  
wayfinder:research

   
wayfinder:task

   
wayfinder:task
wayfinder map child: task

  
wontfix
Will not be actioned

  
worklist/deviations
Deviation incidents from skill/pipeline runs — evidence pile for correcting the owning artifact

  
worklist/lint-rule
Proposed os-sdlc or os-aidd-lint rule

  
worklist/new-implement-build
Updated implementation harness (ast-grep + stepper)

No labels
P0
P1
P2
P3
bug
category/code-change
category/cross-repo
category/eval-run
category/investigation
category/no-test-target
category/owner-ruling
category/prose-artifact
category/recurring-chore
category/structure
category/umbrella
crc-refactor
create
delete
enhancement
filed-by/agent
filed-by/user
frozen
lint-rule
needs-info
needs-triage
next
plugin/cc-architect
plugin/os
plugin/os-adr
plugin/os-aidd-lint
plugin/os-backlog
plugin/os-context
plugin/os-doc-hygiene
plugin/os-sdlc
plugin/os-vault
project/cc-os
ready-for-agent
ready-for-human
recurring
review
rule-proposal
update
waiting
wayfinder:grilling
wayfinder:map
wayfinder:map
wayfinder:research
wayfinder:task
wayfinder:task
wontfix
worklist/deviations
worklist/lint-rule
worklist/new-implement-build
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
jared/cc-os#998
No description provided.