Run any plugin's tests in the takedown harness without a per-plugin table #963

New issue

Closed

opened 2026-10-07 15:38:01 +00:00 by jared · 1 comment

jared commented

2026-10-07 15:38:01 +00:00

Owner

Problem

The takedown harness knows a test command for one plugin only. A case on any other plugin fails before it can run its tests. So the eval can grade only os-sdlc tickets, and each new plugin needs a new table entry.

Observed

plugins/os-sdlc/eval/ticket-takedown/lib/takedown/adapter/plugin_tests.rb:5:

COMMANDS = { "os-sdlc" => %w[bundle exec ruby -Itests] }.freeze

Case #847 (plugin os-backlog) failed in the live-scope arm of the prism-architect eval with:

no test command known for plugin os-backlog

Reproduce

Run bin/takedown-run on case 847.

Acceptance criteria

Given a case on any plugin in this repository that has tests, when the harness runs its tests, then it runs them with no plugin-specific entry in the harness.
Given a plugin with no tests, when the harness runs, then it fails loudly and names the plugin.

Origin

Trigger: prism-architect takedown eval, live-scope arm, case 847 (session e753de0b-6998-472e-b596-c04b9d4454ac)
Improvised this session: none
Chain: #847 cannot run ← no command for os-backlog ← DESIGN: a fixed per-plugin table (plugin_tests.rb:5)
Root candidate: this ticket
Where: plugins/os-sdlc/eval/ticket-takedown/lib/takedown/adapter/plugin_tests.rb
Session: 1e202fec-cce9-40fd-aeb9-24a9faa69d96
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-prism-architect/1e202fec-cce9-40fd-aeb9-24a9faa69d96.jsonl

VERDICT: CREATE — the failure was seen in a full eval run (#847); the chain ends in a DESIGN choice; no other ticket covers plugin_tests.rb.

## Problem The takedown harness knows a test command for one plugin only. A case on any other plugin fails before it can run its tests. So the eval can grade only os-sdlc tickets, and each new plugin needs a new table entry. ## Observed `plugins/os-sdlc/eval/ticket-takedown/lib/takedown/adapter/plugin_tests.rb:5`: ```ruby COMMANDS = { "os-sdlc" => %w[bundle exec ruby -Itests] }.freeze ``` Case #847 (plugin os-backlog) failed in the live-scope arm of the prism-architect eval with: ``` no test command known for plugin os-backlog ``` ## Reproduce Run `bin/takedown-run` on case 847. ## Acceptance criteria 1. Given a case on any plugin in this repository that has tests, when the harness runs its tests, then it runs them with no plugin-specific entry in the harness. 2. Given a plugin with no tests, when the harness runs, then it fails loudly and names the plugin. ## Origin - Trigger: prism-architect takedown eval, live-scope arm, case 847 (session e753de0b-6998-472e-b596-c04b9d4454ac) - Improvised this session: none - Chain: #847 cannot run ← no command for os-backlog ← `DESIGN`: a fixed per-plugin table (`plugin_tests.rb:5`) - Root candidate: this ticket - Where: plugins/os-sdlc/eval/ticket-takedown/lib/takedown/adapter/plugin_tests.rb - Session: 1e202fec-cce9-40fd-aeb9-24a9faa69d96 - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-prism-architect/1e202fec-cce9-40fd-aeb9-24a9faa69d96.jsonl VERDICT: CREATE — the failure was seen in a full eval run (#847); the chain ends in a DESIGN choice; no other ticket covers plugin_tests.rb.

jared added the

plugin/os-sdlc

filed-by/agent

labels

2026-10-07 15:38:01 +00:00

jared referenced this issue

2026-10-07 18:23:08 +00:00

Give each takedown run its own directory when two start in the same second #964

jared referenced this issue

2026-10-07 18:49:57 +00:00

Make evidence collect skip git-ignored files #967

jared added the

category/no-test-target

label

2026-10-10 13:31:42 +00:00

jared commented

2026-10-18 18:59:02 +00:00

Author

Owner

Resolution

Done: Obsolete: takedown/adapter/plugin_tests.rb is gone; project_suites.rb replaced it.

Evidence: find plugins -name plugin_tests.rb: none; plugins/os-sdlc/eval/ticket-takedown/lib/takedown/adapter/project_suites.rb exists.

Follow-ups: none

## Resolution **Done:** Obsolete: takedown/adapter/plugin_tests.rb is gone; project_suites.rb replaced it. **Evidence:** find plugins -name plugin_tests.rb: none; plugins/os-sdlc/eval/ticket-takedown/lib/takedown/adapter/project_suites.rb exists. **Follow-ups:** none

jared closed this issue

2026-10-18 18:59:02 +00:00