Make the takedown eval and the production skill run one pipeline from one set of instructions #962

New issue

Open

opened 2026-10-07 15:37:19 +00:00 by jared · 1 comment

jared commented

2026-10-07 15:37:19 +00:00

Owner

Problem

The takedown eval and the production ticket-takedown skill run two different pipelines with different instruction text. A change that the eval proves does not reach a real /os-sdlc:ticket-takedown run. The skill says that the eval tests it, but the eval tests another design.

Observed

	Production skill	Eval harness
Roles	designer, challenger, judge	researcher, designer A, blind designer B, critic, grader
Instruction files	`plugins/os-sdlc/skills/ticket-takedown/references/*.md`	`plugins/os-sdlc/eval/ticket-takedown/harness/cards/*.md`
Same text	No. `references/designer.md` and `harness/cards/designer.md` differ from line 1.

The harness loads cards only through lib/takedown/model/step_profile.rb:77. The skill never reads harness/cards/, and SKILL.md:32 forbids reading plugins/os-sdlc/eval/.

The strongest eval finding depends on a role that production does not have: the blind designer wrote 21 of 23 designer-written final designs that matched the owner's ruling (see #960).

This ticket gives no approach on purpose (see #960).

Reproduce

diff plugins/os-sdlc/skills/ticket-takedown/references/designer.md plugins/os-sdlc/eval/ticket-takedown/harness/cards/designer.md

Acceptance criteria

Given a change to one role's instruction text, when a production takedown run and an eval run start, then both read the changed text.
Given the roles that a production run uses, when the eval runs a case, then it runs the same roles in the same order.

Origin

Trigger: review of the prism-architect takedown eval (session e753de0b-6998-472e-b596-c04b9d4454ac)
Improvised this session: none
Chain: eval gains do not reach production ← two instruction sets ← DESIGN: the harness replaced challenger with blind designer plus critic (plugins/os-sdlc/eval/ticket-takedown/README.md:5) and kept its own cards
Root candidate: this ticket
Where: plugins/os-sdlc/skills/ticket-takedown/SKILL.md; plugins/os-sdlc/eval/ticket-takedown/lib/takedown/model/step_profile.rb
Session: 1e202fec-cce9-40fd-aeb9-24a9faa69d96
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-prism-architect/1e202fec-cce9-40fd-aeb9-24a9faa69d96.jsonl

## Problem The takedown eval and the production ticket-takedown skill run two different pipelines with different instruction text. A change that the eval proves does not reach a real `/os-sdlc:ticket-takedown` run. The skill says that the eval tests it, but the eval tests another design. ## Observed | | Production skill | Eval harness | | --- | --- | --- | | Roles | designer, challenger, judge | researcher, designer A, blind designer B, critic, grader | | Instruction files | `plugins/os-sdlc/skills/ticket-takedown/references/*.md` | `plugins/os-sdlc/eval/ticket-takedown/harness/cards/*.md` | | Same text | No. `references/designer.md` and `harness/cards/designer.md` differ from line 1. | | The harness loads cards only through `lib/takedown/model/step_profile.rb:77`. The skill never reads `harness/cards/`, and `SKILL.md:32` forbids reading `plugins/os-sdlc/eval/`. The strongest eval finding depends on a role that production does not have: the blind designer wrote 21 of 23 designer-written final designs that matched the owner's ruling (see #960). This ticket gives no approach on purpose (see #960). ## Reproduce ``` diff plugins/os-sdlc/skills/ticket-takedown/references/designer.md plugins/os-sdlc/eval/ticket-takedown/harness/cards/designer.md ``` ## Acceptance criteria 1. Given a change to one role's instruction text, when a production takedown run and an eval run start, then both read the changed text. 2. Given the roles that a production run uses, when the eval runs a case, then it runs the same roles in the same order. ## Origin - Trigger: review of the prism-architect takedown eval (session e753de0b-6998-472e-b596-c04b9d4454ac) - Improvised this session: none - Chain: eval gains do not reach production ← two instruction sets ← `DESIGN`: the harness replaced challenger with blind designer plus critic (`plugins/os-sdlc/eval/ticket-takedown/README.md:5`) and kept its own cards - Root candidate: this ticket - Where: plugins/os-sdlc/skills/ticket-takedown/SKILL.md; plugins/os-sdlc/eval/ticket-takedown/lib/takedown/model/step_profile.rb - Session: 1e202fec-cce9-40fd-aeb9-24a9faa69d96 - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-prism-architect/1e202fec-cce9-40fd-aeb9-24a9faa69d96.jsonl

jared added the

plugin/os-sdlc

filed-by/user

labels

2026-10-07 15:37:19 +00:00

jared referenced this issue

2026-10-07 17:51:34 +00:00

Redesign the ticket-takedown process: one design phase takes up to an hour and millions of tokens #966

jared commented

2026-10-09 17:31:03 +00:00

Author

Owner

Related: #972 moves takedown designs to complete code and deletes the harness --implement mode (owner rulings 1-3 in docs/handoffs/2026-10-09-takedown-implement-path.md); it touches the same eval/production pipeline split.

jared referenced this issue from a commit

2026-10-09 18:43:29 +00:00

docs(adr): ADR-0275 ticket-takedown runs one pipeline, harness Flow is production (#962)

jared referenced this issue from a commit

2026-10-09 18:43:29 +00:00

feat(takedown): one pipeline, --live mode on bin/takedown-run (#962)

jared referenced this issue from a commit

2026-10-09 18:43:29 +00:00

docs(takedown): skill is a thin wrapper over takedown-run --live; README and History row (#962)

jared referenced this issue from a commit

2026-10-09 18:43:29 +00:00

Merge #962 one pipeline (--live) into prism-architect