Measure how often the researcher's cause is shallower than the ruled cause, and flag runs where the critic names a deeper cause #981

New issue

Open

opened 2026-10-10 12:44:06 +00:00 by jared · 1 comment

jared commented

2026-10-10 12:44:06 +00:00

Owner

Problem

ADR-0285 made the researcher state cause {mechanism, defect_class}.
In live mode the grader and the designers use this cause unchecked (see the sibling ticket "Check the researcher's stated cause before any design runs").
We do not know how often the researcher's cause is shallower than, or wrong against, the ruled cause.
Without this number, we cannot decide whether a blind cause check is needed now.
Source: devil's-advocate review.
Related: the sibling ticket above (its part (b) waits on this result). Parent work: #974.
Constraint: do not change harness/cards/grader.md or its schema.

Reproduce

n/a: measurement ticket, nothing to reproduce.

Approach

(a) In Ruby, set a cause_challenged flag in the run summary when any critique has non-empty adr_causes (harness/cards/critic.md:26-28,67). It reads a structured field. It does no prose parsing.
(b) For the training cases, compare the researcher's cause with the ruled cause in the case's reference/key-findings.yaml (lib/takedown/query/grade_inputs.rb:21-24). Classify each case as same, shallower, or wrong.

Required work:

Set the flag in the run summary — check: present cause_challenged in plugins/os-sdlc/eval/ticket-takedown/lib/**/*.rb

Orchestrator work: run the training-case comparison (b) and report the counts on this ticket.

Acceptance criteria

Given a run where any critique has non-empty adr_causes, when the run summary is written, then cause_challenged is true.
Given a run where every critique has empty adr_causes, when the run summary is written, then cause_challenged is false.
Given the training cases, when the comparison runs, then each case is classified as same, shallower, or wrong.

Origin

Trigger: devil's-advocate review of ADR-0285 cause flow, requested by Jared 2026-10-10
Improvised this session: none
Chain: cause quality unmeasured ← ADR-0285 added cause with no metric (ADR-0285) ← DESIGN (ADR-0285)
Root candidate: none
Where: plugins/os-sdlc/eval/ticket-takedown/lib/takedown/query/grade_inputs.rb
Session: ace499f5-e9c6-470b-9571-70c97b32ef50
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-prism-architect/ace499f5-e9c6-470b-9571-70c97b32ef50.jsonl
Skeptic: VERDICT: CREATE

## Problem ADR-0285 made the researcher state `cause {mechanism, defect_class}`. In live mode the grader and the designers use this cause unchecked (see the sibling ticket "Check the researcher's stated cause before any design runs"). We do not know how often the researcher's cause is shallower than, or wrong against, the ruled cause. Without this number, we cannot decide whether a blind cause check is needed now. Source: devil's-advocate review. Related: the sibling ticket above (its part (b) waits on this result). Parent work: #974. Constraint: do not change `harness/cards/grader.md` or its schema. ## Reproduce n/a: measurement ticket, nothing to reproduce. ## Approach (a) In Ruby, set a `cause_challenged` flag in the run summary when any critique has non-empty `adr_causes` (`harness/cards/critic.md:26-28,67`). It reads a structured field. It does no prose parsing. (b) For the training cases, compare the researcher's `cause` with the ruled cause in the case's `reference/key-findings.yaml` (`lib/takedown/query/grade_inputs.rb:21-24`). Classify each case as same, shallower, or wrong. Required work: 1. Set the flag in the run summary — check: present `cause_challenged` in `plugins/os-sdlc/eval/ticket-takedown/lib/**/*.rb` Orchestrator work: run the training-case comparison (b) and report the counts on this ticket. ## Acceptance criteria 1. Given a run where any critique has non-empty `adr_causes`, when the run summary is written, then `cause_challenged` is true. 2. Given a run where every critique has empty `adr_causes`, when the run summary is written, then `cause_challenged` is false. 3. Given the training cases, when the comparison runs, then each case is classified as same, shallower, or wrong. ## Origin - Trigger: devil's-advocate review of ADR-0285 cause flow, requested by Jared 2026-10-10 - Improvised this session: none - Chain: cause quality unmeasured ← ADR-0285 added `cause` with no metric (ADR-0285) ← DESIGN (ADR-0285) - Root candidate: none - Where: plugins/os-sdlc/eval/ticket-takedown/lib/takedown/query/grade_inputs.rb - Session: ace499f5-e9c6-470b-9571-70c97b32ef50 - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-prism-architect/ace499f5-e9c6-470b-9571-70c97b32ef50.jsonl - Skeptic: VERDICT: CREATE

jared added the

plugin/os-sdlc

filed-by/user

labels

2026-10-10 12:44:06 +00:00

jared referenced this issue

2026-10-10 13:41:51 +00:00

Grade every takedown design against the stated cause, and compare each revision with the previous design #974

jared added the

category/umbrella

label

2026-10-10 13:50:38 +00:00

jared referenced this issue

2026-10-12 14:33:28 +00:00

Set cause_challenged in the takedown run's final record #999

jared referenced this issue

2026-10-12 14:33:29 +00:00

Classify researcher cause vs ruled cause over training cases #1000

jared commented

2026-10-12 14:33:58 +00:00

Author

Owner

Decomposed 2026-10-12. Children: #999 (cause_challenged flag, AC 1-2), #1000 (same/shallower/wrong classification over training cases, AC 3). Umbrella stays open until both close.