Grade takedown designs on a scale that can rank a design above the ruling #961

New issue

Open

opened 2026-10-07 15:17:43 +00:00 by jared · 1 comment

jared commented

2026-10-07 15:17:43 +00:00

Owner

Problem

The takedown grader measures only how close a design is to the owner's ruling. A design that is better than the ruling gets a score of 2 at most. So the eval cannot find a design that raises the bar, and it can only teach the designers to copy the owner. The owner's goal is to trust the pipeline to find designs at or above the ruling, and to spot-check it only on occasion.

The owner wants the fix that removes the cause, not the smallest fix. A small fix that stops the symptom leaves the cause in place, and the next form of the same defect becomes a new ticket. The guiding question: which design makes the most future fixes unnecessary?

Observed

Today's score, from lib/takedown/model/grade_score.rb:10-21:

Score	Rule
0	Does not remove the defect
1	Removes the defect, different approach, matches a rejected design
2	Removes the defect, different approach, matches no rejected design
3	Same approach as the ruling, some key findings missed
4	Same approach as the ruling, all key findings stated

The scale the owner wants. The ruling is the bar:

Level	What the grader must show
Inferior	Does not remove the defect, or uses a design that the owner rejected
Workaround	Stops the symptom and leaves the cause. The same defect can come back in a new form.
Parity	Removes the same cause as the ruling
Superior	Removes the cause, and also removes a related defect that exists today
Far superior	Makes the whole class of defect impossible

The size of the change, the number of files, and the blast radius are not costs. A large change can be wrong, and the tests catch that. In the prism-architect eval the critic rejected the #860 ruling because it was "a large rewrite with several unverified claims". A judge with no calibration will probably have the same bias.

Examples from the prism-architect eval:

Case	Workaround	Ruling	Class of defect made impossible
#860	Swap two lines so `--help` is read first	Thor parses all flags; delete the hand parsers	A hand parser that reads flags in the wrong order
#906	Widen the tag regex	Items become YAML with a `path:` key	Code that misreads agent prose with a regex
#907	Skip paths whose deletion is staged	`git update-index`, which handles every index state	A git verb that fails on one index state

Reproduce

n/a: design ticket.

Acceptance criteria

Given a final design and the ruling, when the grader runs, then it reports one of the five levels and a one-sentence reason that names the cause and the class of defect.
Given the ruling and a design the owner rejected, when the judge compares them, then it reports the rejected design as Inferior or Workaround.
Given the ruling and the same ruling in different words, when the judge compares them, then it reports Parity.
Given any pair, when the judge runs, then it does not know which design is the ruling; it sees two designs in random order.
Given the calibration pairs from the five graded cases, when the judge runs on all of them, then the eval reports its agreement rate, and the judge is not used to grade runs until the owner accepts that rate.

Origin

Trigger: review of the prism-architect takedown eval (session e753de0b-6998-472e-b596-c04b9d4454ac); owner request in this session
Improvised this session: none
Chain: eval cannot find a design better than the ruling ← score caps a different approach at 2 (lib/takedown/model/grade_score.rb:10-21) ← DESIGN: the grader card defines only same_approach and matches_rejected_design (harness/cards/grader.md:7-12)
Root candidate: this ticket
Where: plugins/os-sdlc/eval/ticket-takedown/lib/takedown/model/grade_score.rb; plugins/os-sdlc/eval/ticket-takedown/harness/cards/grader.md
Session: 1e202fec-cce9-40fd-aeb9-24a9faa69d96
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-prism-architect/1e202fec-cce9-40fd-aeb9-24a9faa69d96.jsonl

## Problem The takedown grader measures only how close a design is to the owner's ruling. A design that is better than the ruling gets a score of 2 at most. So the eval cannot find a design that raises the bar, and it can only teach the designers to copy the owner. The owner's goal is to trust the pipeline to find designs at or above the ruling, and to spot-check it only on occasion. The owner wants the fix that removes the cause, not the smallest fix. A small fix that stops the symptom leaves the cause in place, and the next form of the same defect becomes a new ticket. The guiding question: which design makes the most future fixes unnecessary? ## Observed Today's score, from `lib/takedown/model/grade_score.rb:10-21`: | Score | Rule | | --- | --- | | 0 | Does not remove the defect | | 1 | Removes the defect, different approach, matches a rejected design | | 2 | Removes the defect, different approach, matches no rejected design | | 3 | Same approach as the ruling, some key findings missed | | 4 | Same approach as the ruling, all key findings stated | The scale the owner wants. The ruling is the bar: | Level | What the grader must show | | --- | --- | | Inferior | Does not remove the defect, or uses a design that the owner rejected | | Workaround | Stops the symptom and leaves the cause. The same defect can come back in a new form. | | Parity | Removes the same cause as the ruling | | Superior | Removes the cause, and also removes a related defect that exists today | | Far superior | Makes the whole class of defect impossible | The size of the change, the number of files, and the blast radius are not costs. A large change can be wrong, and the tests catch that. In the prism-architect eval the critic rejected the #860 ruling because it was "a large rewrite with several unverified claims". A judge with no calibration will probably have the same bias. Examples from the prism-architect eval: | Case | Workaround | Ruling | Class of defect made impossible | | --- | --- | --- | --- | | #860 | Swap two lines so `--help` is read first | Thor parses all flags; delete the hand parsers | A hand parser that reads flags in the wrong order | | #906 | Widen the tag regex | Items become YAML with a `path:` key | Code that misreads agent prose with a regex | | #907 | Skip paths whose deletion is staged | `git update-index`, which handles every index state | A git verb that fails on one index state | ## Reproduce n/a: design ticket. ## Acceptance criteria 1. Given a final design and the ruling, when the grader runs, then it reports one of the five levels and a one-sentence reason that names the cause and the class of defect. 2. Given the ruling and a design the owner rejected, when the judge compares them, then it reports the rejected design as Inferior or Workaround. 3. Given the ruling and the same ruling in different words, when the judge compares them, then it reports Parity. 4. Given any pair, when the judge runs, then it does not know which design is the ruling; it sees two designs in random order. 5. Given the calibration pairs from the five graded cases, when the judge runs on all of them, then the eval reports its agreement rate, and the judge is not used to grade runs until the owner accepts that rate. ## Origin - Trigger: review of the prism-architect takedown eval (session e753de0b-6998-472e-b596-c04b9d4454ac); owner request in this session - Improvised this session: none - Chain: eval cannot find a design better than the ruling ← score caps a different approach at 2 (`lib/takedown/model/grade_score.rb:10-21`) ← `DESIGN`: the grader card defines only `same_approach` and `matches_rejected_design` (`harness/cards/grader.md:7-12`) - Root candidate: this ticket - Where: plugins/os-sdlc/eval/ticket-takedown/lib/takedown/model/grade_score.rb; plugins/os-sdlc/eval/ticket-takedown/harness/cards/grader.md - Session: 1e202fec-cce9-40fd-aeb9-24a9faa69d96 - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-prism-architect/1e202fec-cce9-40fd-aeb9-24a9faa69d96.jsonl

jared added the

plugin/os-sdlc

filed-by/user

labels

2026-10-07 15:17:43 +00:00

jared commented

2026-10-07 15:35:11 +00:00

Author

Owner

Owner direction, 2026-10-07: the five levels replace the 0-4 score. They are not a second rubric on top of it.

Each level must point to something visible in the design, not a forecast:

Workaround: the mechanism that produced the defect still exists after the change.
Parity: that mechanism is gone, as in the ruling.
Far superior: the judge names the class of defect and shows that the mechanism that produces it no longer exists.

Example, #906: the widened regex still reads agent prose, so a new tag form breaks it again (Workaround). With YAML items no regex reads prose, so no tag form can break it (Far superior).

Confidence comes from calibration on known pairs (acceptance criteria 2-5), not from more rubric text.

Owner direction, 2026-10-07: the five levels replace the 0-4 score. They are not a second rubric on top of it. Each level must point to something visible in the design, not a forecast: - Workaround: the mechanism that produced the defect still exists after the change. - Parity: that mechanism is gone, as in the ruling. - Far superior: the judge names the class of defect and shows that the mechanism that produces it no longer exists. Example, #906: the widened regex still reads agent prose, so a new tag form breaks it again (Workaround). With YAML items no regex reads prose, so no tag form can break it (Far superior). Confidence comes from calibration on known pairs (acceptance criteria 2-5), not from more rubric text.

jared referenced this issue

2026-10-07 17:51:34 +00:00

Redesign the ticket-takedown process: one design phase takes up to an hour and millions of tokens #966

~~jared referenced this issue 2026-10-09 17:30:57 +00:00~~

Grade every takedown design against the stated cause, and compare each revision with the previous design #974

jared referenced this issue from a commit

2026-10-09 18:43:29 +00:00

docs(takedown-eval): owner rulings on 847 and 861 graders; agent-drafted restated solutions (#961)

jared referenced this issue from a commit

2026-10-09 18:43:29 +00:00

feat(takedown-eval): five-level grade derived in code from grader facts (#961)

jared referenced this issue from a commit