Keep a guessed fix out of the ticket before the design phase #960

New issue

Open

opened 2026-10-07 14:41:03 +00:00 by jared · 1 comment

jared commented

2026-10-07 14:41:03 +00:00

Owner

Problem

A ticket can carry a guessed fix before any design work. The capture template allows an ## Approach at capture time ("optional at capture"). The ticket-takedown designer that reads the full ticket treats that guess as the truth and builds on it. The owner then rejects the design because it uses the guessed fix. The ticket and the ruling give the designer opposite signals.

Observed

In the prism-architect eval (session e753de0b, 2026-10-06), two designers ran on each case:

Designer	Reads the ticket's fix	Final designs that matched the owner's ruling
A, ticket designer	yes	2
B, blind designer	no	21
Critic-revised	n/a	4

Examples:

#860, designer A: "The ticket's approach is correct." It built the two-line swap that the owner rejected.
#906: the ticket's own fix (widen the tag regex) is the rejected design widen-item-tag.
#907: the ticket listed the ruled fix under Rejected: "update-index ... That needs a user ruling, and an option that keeps the ADR is available."

Limit of this evidence: designer A and designer B also differ in other inputs. Designer A also reads the title, the owner sketch, and the comments. No run removed only the fix from designer A's input.

Questions this ticket must answer before it is ready:

Does designer A reach the ruling more often when the eval removes only the ## Approach and ## Owner sketch sections from its ticket? Test with fixed research replies (--research-reply) on #860, #890, #893, #906, #907.
If yes: does capture stop writing an approach, or does it write the guess under a label that marks it as unverified?
Which os-backlog checks and os-sdlc gates read ## Approach before the design phase and would need to change?
The production ticket-takedown skill does not use the harness cards. Where does the change apply in production?

This ticket gives no approach on purpose.

Reproduce

n/a: design ticket. The results are under plugins/os-sdlc/eval/ticket-takedown/results/prism-architect/ (labels in each run's grade.json, final designer in final-design.json).

Origin

Trigger: prism-architect takedown eval, session e753de0b-6998-472e-b596-c04b9d4454ac, and its review in this session
Improvised this session: none
Chain: designs match the rejected fix ← designer A builds on the ticket's guessed fix (quote in Observed 1; source lib/takedown/presenter/run_prompts.rb:19 gives designer A the full ticket) ← tickets carry a guessed fix before design ← DESIGN: plugins/os-backlog/references/templates/capture.md makes the Approach section "optional at capture"
Root candidate: this ticket
Where: plugins/os-backlog/references/templates/capture.md; plugins/os-sdlc/eval/ticket-takedown/lib/takedown/presenter/run_prompts.rb
Session: 1e202fec-cce9-40fd-aeb9-24a9faa69d96
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-prism-architect/1e202fec-cce9-40fd-aeb9-24a9faa69d96.jsonl

## Problem A ticket can carry a guessed fix before any design work. The capture template allows an `## Approach` at capture time ("optional at capture"). The ticket-takedown designer that reads the full ticket treats that guess as the truth and builds on it. The owner then rejects the design because it uses the guessed fix. The ticket and the ruling give the designer opposite signals. ## Observed In the prism-architect eval (session e753de0b, 2026-10-06), two designers ran on each case: | Designer | Reads the ticket's fix | Final designs that matched the owner's ruling | | --- | --- | --- | | A, ticket designer | yes | 2 | | B, blind designer | no | 21 | | Critic-revised | n/a | 4 | Examples: 1. #860, designer A: "The ticket's approach is correct." It built the two-line swap that the owner rejected. 2. #906: the ticket's own fix (widen the tag regex) is the rejected design `widen-item-tag`. 3. #907: the ticket listed the ruled fix under Rejected: "`update-index` ... That needs a user ruling, and an option that keeps the ADR is available." Limit of this evidence: designer A and designer B also differ in other inputs. Designer A also reads the title, the owner sketch, and the comments. No run removed only the fix from designer A's input. Questions this ticket must answer before it is ready: 1. Does designer A reach the ruling more often when the eval removes only the `## Approach` and `## Owner sketch` sections from its ticket? Test with fixed research replies (`--research-reply`) on #860, #890, #893, #906, #907. 2. If yes: does capture stop writing an approach, or does it write the guess under a label that marks it as unverified? 3. Which os-backlog checks and os-sdlc gates read `## Approach` before the design phase and would need to change? 4. The production ticket-takedown skill does not use the harness cards. Where does the change apply in production? This ticket gives no approach on purpose. ## Reproduce n/a: design ticket. The results are under `plugins/os-sdlc/eval/ticket-takedown/results/prism-architect/` (labels in each run's `grade.json`, final designer in `final-design.json`). ## Origin - Trigger: prism-architect takedown eval, session e753de0b-6998-472e-b596-c04b9d4454ac, and its review in this session - Improvised this session: none - Chain: designs match the rejected fix ← designer A builds on the ticket's guessed fix (quote in Observed 1; source `lib/takedown/presenter/run_prompts.rb:19` gives designer A the full ticket) ← tickets carry a guessed fix before design ← `DESIGN`: `plugins/os-backlog/references/templates/capture.md` makes the Approach section "optional at capture" - Root candidate: this ticket - Where: plugins/os-backlog/references/templates/capture.md; plugins/os-sdlc/eval/ticket-takedown/lib/takedown/presenter/run_prompts.rb - Session: 1e202fec-cce9-40fd-aeb9-24a9faa69d96 - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-prism-architect/1e202fec-cce9-40fd-aeb9-24a9faa69d96.jsonl

jared added the

plugin/os-backlog

plugin/os-sdlc

filed-by/user

labels

2026-10-07 14:41:03 +00:00

jared referenced this issue

2026-10-07 15:37:19 +00:00

Make the takedown eval and the production skill run one pipeline from one set of instructions #962

jared referenced this issue

2026-10-07 15:38:30 +00:00

issue-create ends the Origin section at an inline heading mark and refuses a valid body #965

jared referenced this issue

2026-10-07 17:51:34 +00:00

Redesign the ticket-takedown process: one design phase takes up to an hour and millions of tokens #966

jared referenced this issue

2026-10-07 21:20:14 +00:00

Extract ticket and backlog behavior into an os-backlog gem #969

jared commented

2026-10-07 21:20:22 +00:00

Author

Owner

Owner ruling 2026-10-07 (session 87a712ed): ticket and backlog behavior moves into an os-backlog gem (#969, ADR-0279). The one TicketTemplate owner this ticket wants belongs in that gem.

jared added the

category/eval-run

label

2026-10-10 13:31:43 +00:00