jared/cc-os

Watch

Fork 0

Grade every takedown design against the stated cause, and compare each revision with the previous design #974

New issue

Closed

opened 2026-10-09 17:30:57 +00:00 by jared · 3 comments

jared commented

2026-10-09 17:30:57 +00:00

Owner

Problem

Live takedown runs skip the grader: LiveContext builds RunPlan.new(grade: false). The owner ruled on 2026-10-09 (handoff docs/handoffs/2026-10-09-takedown-implement-path.md, rulings 5 and 6) that the AI grades live designs and that the owner does not rule on designs.

Reproduce

n/a: feature ticket.

Approach

Jared ruled option C on 2026-10-10. ADR-0285 records the ruling. It amends ADR-0278 and ADR-0281.

Every design in a run (each blind design and each revision) gets a grade, in live mode and in case mode. The grade is the grader facts against the stated cause. The grader never uses the ticket ## Approach or any optional ticket section as a baseline.
Each grader call grades exactly one design. The prompt carries the stated cause and the related-defect list of the run so far (ids X1, X2, and so on). The grader reuses these ids and appends a new id only when needed.
Ruby derives an absolute level from the facts of the design: 0 defect_remains, 1 cause_remains, 2 cause_removed, 3 class_removed. Ruby reports removed related defects but does not rank them.
Design N (N >= 2) also gets a comparison level against design N-1, from the stored facts of both designs, with the GradeSide#level logic (inferior to far_superior). Design 1 has no comparison.
Case mode keeps grade.json as the final design versus the ruling. The ruling is graded alone once and compared in Ruby. The eval metric and the rank readers do not change.
The grader card changes, so its calibration digest changes. Calibration must pass the 0.8 bar again before live grades are trusted.

Not in this ticket: a stop rule at diminishing returns (separate ticket). The ADR-source half of #970 stays on #970.

Required work:

Build RunPlan for live runs with grade: true — check: absent grade: false in plugins/os-sdlc/eval/ticket-takedown/lib/takedown/**/*.rb
Change harness/cards/grader.md and its schema so one call grades one design and takes the related-defect list — check: grep -n "X1" plugins/os-sdlc/eval/ticket-takedown/harness/cards/grader.md finds the id rule
Grade each blind design and each revision in the run, and append new related-defect ids to the run list — check: a run test shows the same id across two grader calls
Derive the absolute level 0 to 3 in Ruby from the facts of one design — check: unit test for Takedown::Model::Grade for each level
Derive the comparison level of design N against design N-1 from stored facts with GradeSide#level — check: unit test; design 1 has no comparison
Remove every use of the ticket ## Approach as a grade baseline — check: absent Approach in the grade code path under lib/takedown/
Keep case-mode grade.json unchanged — check: existing case-mode and rank-reader tests pass without edits
Report the levels in the run result
Rerun bin/grade-calibrate until agreement is 0.8 or better

Acceptance criteria

Given a live run, when the design phase ends, then the run result holds an absolute level from 0 to 3 for each design.
Given a live run with N >= 2 designs, when the run ends, then design N holds a comparison level against design N-1, and design 1 holds none.
Given two grader calls in one run, when the second call finds a known related defect, then it reuses the id from the shared list, and the prompt of every call carries that list.
Given a ticket with a ## Approach section, when the grader runs, then no grader prompt or level uses that section.
Given a case run, when it ends, then grade.json holds the final design versus the ruling in the same shape as before.
Given the new grader card, when calibration runs, then agreement is 0.8 or better.

Owner sketch

class Takedown::Model::Grade
  def absolute_level
    # 0 defect_remains, 1 cause_remains, 2 cause_removed, 3 class_removed
  end

  def comparison_level(previous)
    # GradeSide#level over the stored facts of both designs
  end
end

Origin

Trigger: Owner rulings of 2026-10-09 recorded in docs/handoffs/2026-10-09-takedown-implement-path.md, and option C of 2026-10-10 (ADR-0285).
Improvised this session: none
Chain: takedown design ends at a sketch ← ADR-0276 §5 and §9 (DESIGN, docs/adr/0276-ticket-takedown-runs-one-pipeline-the-eval-harness-flow-is-the-production-pipeline-live-mode.md)
Root candidate: none
Where: plugins/os-sdlc/eval/ticket-takedown/, plugins/os-sdlc/maps/, plugins/os-sdlc/skills/implement/SKILL.md
Session: 58499175-8e67-4b3a-8cfe-ed601216e8da
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-prism-architect/58499175-8e67-4b3a-8cfe-ed601216e8da.jsonl

## Problem Live takedown runs skip the grader: `LiveContext` builds `RunPlan.new(grade: false)`. The owner ruled on 2026-10-09 (handoff `docs/handoffs/2026-10-09-takedown-implement-path.md`, rulings 5 and 6) that the AI grades live designs and that the owner does not rule on designs. ## Reproduce n/a: feature ticket. ## Approach Jared ruled option C on 2026-10-10. ADR-0285 records the ruling. It amends ADR-0278 and ADR-0281. 1. Every design in a run (each blind design and each revision) gets a grade, in live mode and in case mode. The grade is the grader facts against the stated cause. The grader never uses the ticket `## Approach` or any optional ticket section as a baseline. 2. Each grader call grades exactly one design. The prompt carries the stated cause and the related-defect list of the run so far (ids X1, X2, and so on). The grader reuses these ids and appends a new id only when needed. 3. Ruby derives an absolute level from the facts of the design: 0 defect_remains, 1 cause_remains, 2 cause_removed, 3 class_removed. Ruby reports removed related defects but does not rank them. 4. Design N (N >= 2) also gets a comparison level against design N-1, from the stored facts of both designs, with the `GradeSide#level` logic (inferior to far_superior). Design 1 has no comparison. 5. Case mode keeps `grade.json` as the final design versus the ruling. The ruling is graded alone once and compared in Ruby. The eval metric and the rank readers do not change. 6. The grader card changes, so its calibration digest changes. Calibration must pass the 0.8 bar again before live grades are trusted. Not in this ticket: a stop rule at diminishing returns (separate ticket). The ADR-source half of #970 stays on #970. Required work: 1. Build `RunPlan` for live runs with `grade: true` — check: absent `grade: false` in `plugins/os-sdlc/eval/ticket-takedown/lib/takedown/**/*.rb` 2. Change `harness/cards/grader.md` and its schema so one call grades one design and takes the related-defect list — check: `grep -n "X1" plugins/os-sdlc/eval/ticket-takedown/harness/cards/grader.md` finds the id rule 3. Grade each blind design and each revision in the run, and append new related-defect ids to the run list — check: a run test shows the same id across two grader calls 4. Derive the absolute level 0 to 3 in Ruby from the facts of one design — check: unit test for `Takedown::Model::Grade` for each level 5. Derive the comparison level of design N against design N-1 from stored facts with `GradeSide#level` — check: unit test; design 1 has no comparison 6. Remove every use of the ticket `## Approach` as a grade baseline — check: absent `Approach` in the grade code path under `lib/takedown/` 7. Keep case-mode `grade.json` unchanged — check: existing case-mode and rank-reader tests pass without edits 8. Report the levels in the run result 9. Rerun `bin/grade-calibrate` until agreement is 0.8 or better ## Acceptance criteria 1. Given a live run, when the design phase ends, then the run result holds an absolute level from 0 to 3 for each design. 2. Given a live run with N >= 2 designs, when the run ends, then design N holds a comparison level against design N-1, and design 1 holds none. 3. Given two grader calls in one run, when the second call finds a known related defect, then it reuses the id from the shared list, and the prompt of every call carries that list. 4. Given a ticket with a `## Approach` section, when the grader runs, then no grader prompt or level uses that section. 5. Given a case run, when it ends, then `grade.json` holds the final design versus the ruling in the same shape as before. 6. Given the new grader card, when calibration runs, then agreement is 0.8 or better. ## Owner sketch ```ruby class Takedown::Model::Grade def absolute_level # 0 defect_remains, 1 cause_remains, 2 cause_removed, 3 class_removed end def comparison_level(previous) # GradeSide#level over the stored facts of both designs end end ``` ## Origin - Trigger: Owner rulings of 2026-10-09 recorded in `docs/handoffs/2026-10-09-takedown-implement-path.md`, and option C of 2026-10-10 (ADR-0285). - Improvised this session: none - Chain: takedown design ends at a sketch ← ADR-0276 §5 and §9 (DESIGN, `docs/adr/0276-ticket-takedown-runs-one-pipeline-the-eval-harness-flow-is-the-production-pipeline-live-mode.md`) - Root candidate: none - Where: `plugins/os-sdlc/eval/ticket-takedown/`, `plugins/os-sdlc/maps/`, `plugins/os-sdlc/skills/implement/SKILL.md` - Session: 58499175-8e67-4b3a-8cfe-ed601216e8da - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-prism-architect/58499175-8e67-4b3a-8cfe-ed601216e8da.jsonl

jared added the

plugin/os-sdlc

filed-by/user

labels

2026-10-09 17:30:57 +00:00

jared referenced this issue

2026-10-09 17:31:02 +00:00

Trial takedown on 10 code-change tickets, then run the implement-side eval #975

jared referenced this issue

2026-10-09 17:31:02 +00:00

Grade every takedown design and decide which ADRs the critic reads #970

jared changed title from ~~Run the grader in live takedown against the researcher cause and the ticket approach~~ to Grade every takedown design against the stated cause, and compare each revision with the previous design

2026-10-10 11:27:55 +00:00

jared referenced this issue

2026-10-10 11:27:56 +00:00

Grade every takedown design and decide which ADRs the critic reads #970

jared referenced this issue

2026-10-10 11:28:16 +00:00

Stop takedown revisions when a new design does not improve on the previous design #978

jared commented

2026-10-10 11:40:23 +00:00

Author

Owner

Built and merged to main at 086636ff (ADR-0285): every design is graded alone against the stated cause (absolute level 0-3); a revision is also compared with the design it revised; a blind design with the previous blind design. The researcher now states the cause. Case grade.json unchanged. Suites: takedown 475/0, os-sdlc 1513/0. Open before close: bin/grade-calibrate must pass 0.8 on the new card (case runs refuse until then), then one live run to check AC1.

Built and merged to main at 086636ff (ADR-0285): every design is graded alone against the stated cause (absolute level 0-3); a revision is also compared with the design it revised; a blind design with the previous blind design. The researcher now states the cause. Case grade.json unchanged. Suites: takedown 475/0, os-sdlc 1513/0. Open before close: bin/grade-calibrate must pass 0.8 on the new card (case runs refuse until then), then one live run to check AC1.

jared referenced this issue from a commit

2026-10-10 11:43:56 +00:00

docs(adr): ADR-0285 takedown grades every design against the stated cause, amends ADR-0278 and ADR-0281 (#974)

jared referenced this issue from a commit

2026-10-10 11:43:56 +00:00

feat(takedown): GradeSide derives an absolute level from its own facts (#974)

jared referenced this issue from a commit

2026-10-10 11:43:56 +00:00

feat(takedown): RunGrades keeps one related-defect list and derives absolute and comparison levels per design (#974)

jared referenced this issue from a commit

2026-10-10 11:43:56 +00:00

feat(takedown): grade every design alone in live and case mode; the ruling is graded alone and compared in Ruby (#974)

jared referenced this issue from a commit

2026-10-10 11:43:56 +00:00

fix(takedown): a revision grade compares with the design it revised (#974)

jared commented

2026-10-10 12:17:42 +00:00

Author

Owner

Step 6 built on main (3bc6f306). Comparison rule (ADR-0286, amends ADR-0285 decision 4): a design after the first is a challenger, not a revision. Ruby compares it with the best previously graded design in the run (highest absolute rank; tie: the design that removes more related defects; further tie: the most recent). grade-.json comparison.against names that design. Reason: comparing with one predecessor restricts creativity; comparing with all adds cost and no information. Calibration passed: 0.94 agreement over 35 rows (threshold 0.8, bba7b76e). Suites: takedown 477/0, os-sdlc 1513/0. Only the live-run check for AC1 remains (planned as the #977 tracer).

Step 6 built on main (3bc6f306). Comparison rule (ADR-0286, amends ADR-0285 decision 4): a design after the first is a challenger, not a revision. Ruby compares it with the best previously graded design in the run (highest absolute rank; tie: the design that removes more related defects; further tie: the most recent). grade-<step>.json comparison.against names that design. Reason: comparing with one predecessor restricts creativity; comparing with all adds cost and no information. Calibration passed: 0.94 agreement over 35 rows (threshold 0.8, bba7b76e). Suites: takedown 477/0, os-sdlc 1513/0. Only the live-run check for AC1 remains (planned as the #977 tracer).

jared referenced this issue

2026-10-10 12:44:06 +00:00

Check the researcher's stated cause before any design runs #980

jared referenced this issue

2026-10-10 12:44:06 +00:00

Measure how often the researcher's cause is shallower than the ruled cause, and flag runs where the critic names a deeper cause #981

jared added the

category/no-test-target

label

2026-10-10 13:31:40 +00:00

jared commented

2026-10-10 13:41:51 +00:00

Author

Owner

Resolution

Done: Every design is graded alone against the stated cause (ADR-0285); a later design is compared with the best previous design (ADR-0286); the reviser prompt and card now make each revision a challenger of the best design so far (e94a252a).

Evidence: Live run docs/takedown/194-20261010-093151 (.33, 15 steps, 0 failed): grade-design-1 class_removed rank 3; grade-revise-1 cause_removed rank 2, parity vs design-1; grade-revise-2 class_removed rank 3, parity vs design-1. Calibration 0.94 (bba7b76e). rake test 480 runs 0 failures; os-sdlc suite 1513 runs 0 failures.

Follow-ups: #978 stop rule at diminishing returns (uses the absolute levels); #980 and #981 cover the unchecked researcher cause.

## Resolution **Done:** Every design is graded alone against the stated cause (ADR-0285); a later design is compared with the best previous design (ADR-0286); the reviser prompt and card now make each revision a challenger of the best design so far (e94a252a). **Evidence:** Live run docs/takedown/194-20261010-093151 (.33, 15 steps, 0 failed): grade-design-1 class_removed rank 3; grade-revise-1 cause_removed rank 2, parity vs design-1; grade-revise-2 class_removed rank 3, parity vs design-1. Calibration 0.94 (bba7b76e). rake test 480 runs 0 failures; os-sdlc suite 1513 runs 0 failures. **Follow-ups:** #978 stop rule at diminishing returns (uses the absolute levels); #980 and #981 cover the unchecked researcher cause.

jared closed this issue

2026-10-10 13:41:51 +00:00

jared referenced this issue from a commit

2026-10-12 14:02:16 +00:00

chore(takedown): recalibrate the single-design grader, 0.94 agreement (#974)

jared referenced this issue from a commit

2026-10-12 14:02:16 +00:00

feat(takedown): a later design is compared with the best previous design in the run (#974)

jared referenced this issue from a commit

2026-10-12 14:02:16 +00:00

docs: ADR-0286 challenger comparison, step 6 records (#974)

jared referenced this issue from a commit