Trial takedown on 10 code-change tickets, then run the implement-side eval #975

New issue

Open

opened 2026-10-09 17:31:02 +00:00 by jared · 0 comments

jared commented

2026-10-09 17:31:02 +00:00

Owner

Problem

No evidence shows that the takedown design-first path completes a ticket green at a lower cost than poodr-ticket-implementation. The owner ruled on 2026-10-09 (handoff docs/handoffs/2026-10-09-takedown-implement-path.md) that a trial on 10 code-change tickets comes first, then an implement-side eval.

Reproduce

n/a: trial and eval ticket.

Approach

Handoff build-order steps 8 and 9. Trial: 10 code-change tickets across at least 3 targets, estimated 40 to 85 USD. Record per ticket: design cost and time, rounds, live grade, implement report (os-sdlc-implementation-report <id>: dispatches, elapsed_s, retries, tokens), terminal node. Implement-side eval only if at least 3 of 10 reach parity or better and complete green. Cases are frozen designs at their base commits. Metric: USD and wall time. Hard gate: implementation_complete, suite green, design tests unmodified. Levers: per-node model, place-tests and place-code cards, keep or drop behavior-verifier and contract-auditor. Read ~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md and docs/eval-discipline.md before the eval.

Blocked by: #972 (takedown-implement map), #974 (live grading), #973 (ticket category).

Required work:

Record the trial table for the 10 tickets in a handoff under docs/handoffs/.
If the gate of 3 of 10 passes, add frozen implement-side eval cases and run the eval.

Acceptance criteria

Given the trial, when it ends, then each of the 10 tickets has a row with all recorded fields.
Given fewer than 3 of 10 at parity or better and green, when the trial ends, then the handoff reports it and no implement-side eval runs.

Owner sketch

class Takedown::Presenter::TrialRow
  def render(report)
end

Origin

Trigger: Owner rulings of 2026-10-09 recorded in docs/handoffs/2026-10-09-takedown-implement-path.md.
Improvised this session: none
Chain: takedown design ends at a sketch ← ADR-0276 §5 and §9 (DESIGN, docs/adr/0276-ticket-takedown-runs-one-pipeline-the-eval-harness-flow-is-the-production-pipeline-live-mode.md)
Root candidate: none
Where: plugins/os-sdlc/eval/ticket-takedown/, plugins/os-sdlc/maps/, plugins/os-sdlc/skills/implement/SKILL.md
Session: 58499175-8e67-4b3a-8cfe-ed601216e8da
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-prism-architect/58499175-8e67-4b3a-8cfe-ed601216e8da.jsonl

VERDICT: CREATE

## Problem No evidence shows that the takedown design-first path completes a ticket green at a lower cost than `poodr-ticket-implementation`. The owner ruled on 2026-10-09 (handoff `docs/handoffs/2026-10-09-takedown-implement-path.md`) that a trial on 10 `code-change` tickets comes first, then an implement-side eval. ## Reproduce n/a: trial and eval ticket. ## Approach Handoff build-order steps 8 and 9. Trial: 10 `code-change` tickets across at least 3 targets, estimated 40 to 85 USD. Record per ticket: design cost and time, rounds, live grade, implement report (`os-sdlc-implementation-report <id>`: dispatches, elapsed_s, retries, tokens), terminal node. Implement-side eval only if at least 3 of 10 reach parity or better and complete green. Cases are frozen designs at their base commits. Metric: USD and wall time. Hard gate: `implementation_complete`, suite green, design tests unmodified. Levers: per-node model, place-tests and place-code cards, keep or drop behavior-verifier and contract-auditor. Read `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md` and `docs/eval-discipline.md` before the eval. Blocked by: #972 (takedown-implement map), #974 (live grading), #973 (ticket category). Required work: 1. Record the trial table for the 10 tickets in a handoff under `docs/handoffs/`. 2. If the gate of 3 of 10 passes, add frozen implement-side eval cases and run the eval. ## Acceptance criteria 1. Given the trial, when it ends, then each of the 10 tickets has a row with all recorded fields. 2. Given fewer than 3 of 10 at parity or better and green, when the trial ends, then the handoff reports it and no implement-side eval runs. ## Owner sketch ```ruby class Takedown::Presenter::TrialRow def render(report) end ``` ## Origin - Trigger: Owner rulings of 2026-10-09 recorded in `docs/handoffs/2026-10-09-takedown-implement-path.md`. - Improvised this session: none - Chain: takedown design ends at a sketch ← ADR-0276 §5 and §9 (DESIGN, `docs/adr/0276-ticket-takedown-runs-one-pipeline-the-eval-harness-flow-is-the-production-pipeline-live-mode.md`) - Root candidate: none - Where: `plugins/os-sdlc/eval/ticket-takedown/`, `plugins/os-sdlc/maps/`, `plugins/os-sdlc/skills/implement/SKILL.md` - Session: 58499175-8e67-4b3a-8cfe-ed601216e8da - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-prism-architect/58499175-8e67-4b3a-8cfe-ed601216e8da.jsonl VERDICT: CREATE

jared added the

plugin/os-sdlc

filed-by/user

category/eval-run

labels

2026-10-09 17:31:02 +00:00

jared referenced this issue

2026-10-14 17:14:24 +00:00

Takedown design phase must lint the design's tests and code before the design is final #1005