Classify researcher cause vs ruled cause over training cases #1000

New issue

Open

opened 2026-10-12 14:33:29 +00:00 by jared · 0 comments

jared commented

2026-10-12 14:33:29 +00:00

Owner

Problem

We do not know how often the researcher's cause is shallower than, or wrong against, the ruled cause. Without this count, #980 part (b) (blind cause check) cannot be decided. Child of #981 (AC 3); parent work #974.

Approach

Orchestrator work: for each training case only, compare the researcher's cause from its run with the ruled cause in reference/key-findings.yaml (lib/takedown/query/grade_inputs.rb:21-24). Classify each case as same, shallower, or wrong. Never open held-out or reserve cases (docs/eval-discipline.md). Do not change harness/cards/grader.md or its schema. Post the per-case table and counts as a comment on this ticket and on #981.

Required work: none beyond the acceptance criteria.

Acceptance criteria

Given the training cases, when the comparison runs, then each case is classified as same, shallower, or wrong against its ruled cause.
Given the classification, when it is complete, then the counts are posted on this ticket and on #981.
Given the run, when it finishes, then no held-out or reserve case was opened.

Owner sketch

Orchestrator work, no code object. The orchestrator is the owner.

Origin

Trigger: decomposition of #981 on 2026-10-12
Improvised this session: none
Chain: cause quality unmeasured ← ADR-0285 added cause with no metric (ADR-0285) ← DESIGN (ADR-0285)
Root candidate: none
Where: plugins/os-sdlc/eval/ticket-takedown/cases (training cases)
Skeptic: VERDICT: CREATE

## Problem We do not know how often the researcher's `cause` is shallower than, or wrong against, the ruled cause. Without this count, #980 part (b) (blind cause check) cannot be decided. Child of #981 (AC 3); parent work #974. ## Approach Orchestrator work: for each training case only, compare the researcher's `cause` from its run with the ruled cause in `reference/key-findings.yaml` (`lib/takedown/query/grade_inputs.rb:21-24`). Classify each case as same, shallower, or wrong. Never open held-out or reserve cases (docs/eval-discipline.md). Do not change `harness/cards/grader.md` or its schema. Post the per-case table and counts as a comment on this ticket and on #981. Required work: none beyond the acceptance criteria. ## Acceptance criteria 1. Given the training cases, when the comparison runs, then each case is classified as same, shallower, or wrong against its ruled cause. 2. Given the classification, when it is complete, then the counts are posted on this ticket and on #981. 3. Given the run, when it finishes, then no held-out or reserve case was opened. ## Owner sketch Orchestrator work, no code object. The orchestrator is the owner. ## Origin - Trigger: decomposition of #981 on 2026-10-12 - Improvised this session: none - Chain: cause quality unmeasured ← ADR-0285 added `cause` with no metric (ADR-0285) ← DESIGN (ADR-0285) - Root candidate: none - Where: plugins/os-sdlc/eval/ticket-takedown/cases (training cases) - Skeptic: VERDICT: CREATE

jared added the

plugin/os-sdlc

filed-by/agent

category/eval-run

labels

2026-10-12 14:33:29 +00:00

jared referenced this issue

2026-10-12 14:33:58 +00:00

Measure how often the researcher's cause is shallower than the ruled cause, and flag runs where the critic names a deeper cause #981

jared referenced this issue

2026-10-12 14:33:58 +00:00

Check the researcher's stated cause before any design runs #980