jared/cc-os

Watch

Fork 0

Accept extra empty critique labels in the takedown critique step #1426

New issue

Closed

opened 2026-10-27 17:01:49 +00:00 by jared · 2 comments

jared commented

2026-10-27 17:01:49 +00:00

Owner

Problem

A ticket-takedown run fails at the critique step when the critic returns more design labels than the run expects. The critic invented label B. The prompt showed only Design A. The check treats any label mismatch as a failed step.

Observed

Run docs/takedown/1416-20261027-125313 stopped at step 07 after grade-design-1: cause_remains with this error.

FAILED critique-1: critique covers labels A, B, expected A

The check that raises it is plugins/os-sdlc/eval/ticket-takedown/lib/takedown/command/critique_step.rb:26. It compares Critique#labels (plugins/os-sdlc/eval/ticket-takedown/lib/takedown/model/critique.rb:23) with run.labels.keys.sort by strict equality (critique_step.rb:21). labels.json of the run holds only {"1": {"A": "design-1"}}.
The prompt did not invite label B. 07-critique-1.prompt.md has one ## Design A heading (line 206), one ## Failing test check for Design A heading (line 604), and no other design label. The critic invented B. 07-critique-1.reply.json lists {"label": "B", "objections": []} and its reason says "There is no Design B."
The previous run of the same ticket, docs/takedown/1416-20261027-123627, passed this step with labels ['A']. The failure is intermittent.

Reproduce

n/a: the critic reply is nondeterministic. Evidence is in docs/takedown/1416-20261027-125313/07-critique-1.reply.json.

Approach

Either constrain the critic to the expected labels, or ignore labels that are not expected. An empty extra design (no objections) carries no information.

Required work:

Make CritiqueStep#check accept a critique that holds extra labels with no objections — check: absent critique covers labels in plugins/os-sdlc/eval/ticket-takedown/lib/takedown/command/critique_step.rb

Acceptance criteria

Given a critique that covers labels A and B and a run that expects only A, when the critique step runs, then the step does not fail.
Given a critique that omits an expected label, when the critique step runs, then the step still fails.

Owner sketch

class Takedown::CritiqueStep
  def check(found, expected)
  end
end

Origin

Trigger: docs/takedown/1416-20261027-125313, step 07 critique-1
Improvised this session: none
Chain: FAILED critique-1 <- strict label equality in CritiqueStep#check (critique_step.rb:21,26) <- critic added unprompted Design B (reply.json) - DESIGN
Root candidate: none found; searched CritiqueStep, critique_step, critique covers labels, critique-1, extra label, takedown critique
Skeptic: VERDICT: CREATE
Where: plugins/os-sdlc/eval/ticket-takedown/lib/takedown/command/critique_step.rb
Session: c843bd27-b3bf-4e36-9243-df1697457146
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/c843bd27-b3bf-4e36-9243-df1697457146.jsonl

## Problem A ticket-takedown run fails at the critique step when the critic returns more design labels than the run expects. The critic invented label B. The prompt showed only Design A. The check treats any label mismatch as a failed step. ## Observed Run `docs/takedown/1416-20261027-125313` stopped at step 07 after `grade-design-1: cause_remains` with this error. ``` FAILED critique-1: critique covers labels A, B, expected A ``` The check that raises it is `plugins/os-sdlc/eval/ticket-takedown/lib/takedown/command/critique_step.rb:26`. It compares `Critique#labels` (`plugins/os-sdlc/eval/ticket-takedown/lib/takedown/model/critique.rb:23`) with `run.labels.keys.sort` by strict equality (`critique_step.rb:21`). `labels.json` of the run holds only `{"1": {"A": "design-1"}}`. The prompt did not invite label B. `07-critique-1.prompt.md` has one `## Design A` heading (line 206), one `## Failing test check for Design A` heading (line 604), and no other design label. The critic invented B. `07-critique-1.reply.json` lists `{"label": "B", "objections": []}` and its `reason` says "There is no Design B." The previous run of the same ticket, `docs/takedown/1416-20261027-123627`, passed this step with labels `['A']`. The failure is intermittent. ## Reproduce n/a: the critic reply is nondeterministic. Evidence is in `docs/takedown/1416-20261027-125313/07-critique-1.reply.json`. ## Approach Either constrain the critic to the expected labels, or ignore labels that are not expected. An empty extra design (no objections) carries no information. Required work: 1. Make `CritiqueStep#check` accept a critique that holds extra labels with no objections — check: absent `critique covers labels` in `plugins/os-sdlc/eval/ticket-takedown/lib/takedown/command/critique_step.rb` ## Acceptance criteria 1. Given a critique that covers labels A and B and a run that expects only A, when the critique step runs, then the step does not fail. 2. Given a critique that omits an expected label, when the critique step runs, then the step still fails. ## Owner sketch ```ruby class Takedown::CritiqueStep def check(found, expected) end end ``` ## Origin - Trigger: `docs/takedown/1416-20261027-125313`, step 07 `critique-1` - Improvised this session: none - Chain: FAILED critique-1 <- strict label equality in `CritiqueStep#check` (critique_step.rb:21,26) <- critic added unprompted Design B (reply.json) - DESIGN - Root candidate: none found; searched CritiqueStep, critique_step, critique covers labels, critique-1, extra label, takedown critique - Skeptic: VERDICT: CREATE - Where: plugins/os-sdlc/eval/ticket-takedown/lib/takedown/command/critique_step.rb - Session: c843bd27-b3bf-4e36-9243-df1697457146 - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/c843bd27-b3bf-4e36-9243-df1697457146.jsonl

jared added the

plugin/os-sdlc

filed-by/agent

category/code-change

labels

2026-10-27 17:01:49 +00:00

jared referenced this issue

2026-10-27 17:10:55 +00:00

Guard-map test-reviewer always settles error for missing red_tests #1416

jared commented

2026-10-27 17:46:02 +00:00

Author

Owner

Work started in worktree ticket-1426 (branch ticket-1426).

jared referenced this issue from a commit

2026-10-27 17:49:08 +00:00

fix(os-sdlc): accept extra empty critique labels in the takedown critique step (#1426)

jared referenced this issue

2026-10-27 17:50:45 +00:00

Takedown critic card invites a label B the run did not give #1433

jared commented

2026-10-27 17:50:53 +00:00

Author

Owner

Resolution

Done: Critique.new(data, expected:) drops extra designs that carry no objections; CritiqueStep, Flow and RunPrompts pass the expected labels. A missing expected label or an extra label with objections still fails.

Evidence: main merge of commit 4b75e564 (in d6190950 history). test/command/critique_step_test.rb: 4 tests. Takedown suite 624 runs, 0 failures; rubocop clean. Built by hand in a worktree, test first, not via /os-sdlc:implement.

Follow-ups: #1433 critic card label B wording (skeptic VERDICT: CREATE, no root candidate); setup_step.rb:5 ParameterListSignalsMissingObject offense predates this change: dropped, owned by the lint-cops track.

## Resolution **Done:** Critique.new(data, expected:) drops extra designs that carry no objections; CritiqueStep, Flow and RunPrompts pass the expected labels. A missing expected label or an extra label with objections still fails. **Evidence:** main merge of commit 4b75e564 (in d6190950 history). test/command/critique_step_test.rb: 4 tests. Takedown suite 624 runs, 0 failures; rubocop clean. Built by hand in a worktree, test first, not via /os-sdlc:implement. **Follow-ups:** #1433 critic card label B wording (skeptic VERDICT: CREATE, no root candidate); setup_step.rb:5 ParameterListSignalsMissingObject offense predates this change: dropped, owned by the lint-cops track.

jared closed this issue

2026-10-27 17:50:54 +00:00

jared referenced this issue from a commit

2026-10-27 17:50:57 +00:00

docs(os-sdlc): History rows for #1425 and #1426