Grade every takedown design and decide which ADRs the critic reads #970

New issue

Open

opened 2026-10-09 16:58:42 +00:00 by jared · 3 comments

jared commented

2026-10-09 16:58:42 +00:00

Owner

Problem

The takedown pipeline grades only the final design. So the grade drop between a blind design and an ADR-constrained design is not a number. The critic also reads the ADRs of the case base-commit snapshot. The snapshot for case 650 (base 1280b01d) has no ADR-0279. So the critic's adr_causes was empty on a design that crosses a plugin boundary.

Observed

Run 650-20261008-112340 scored far_superior (4). Its design shares one plugin's class across the boundary, which ADR-0279 decision 4 forbids. No ADR text in the snapshot could flag it.

Reproduce

n/a: design ticket. See docs/handoffs/2026-10-08-takedown-autoresearch.md, section "ADR-blind designer built and rerun".

Approach

Grade every design the pipeline produces (ADR-0281 decision 4), not only the final one. Decide whether the critic reads ADRs from the current tree instead of the case snapshot. The decision needs an ADR if it reverses ADR-0281.

Origin

Trigger: the 2026-10-08 ticket-takedown autoresearch session; freeze-and-merge bookkeeping on 2026-10-09.
Improvised this session: none.
Chain: final-only grade <- ADR-0281 decision 4 not built; critic ADR source <- ScratchRepo archives base_commit docs/adr (DESIGN, ADR-0281)
Root candidate: none.
Where: plugins/os-sdlc/eval/ticket-takedown (grader, ScratchRepo)
Session: 58499175-8e67-4b3a-8cfe-ed601216e8da
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-prism-architect/58499175-8e67-4b3a-8cfe-ed601216e8da.jsonl

VERDICT: CREATE

## Problem The takedown pipeline grades only the final design. So the grade drop between a blind design and an ADR-constrained design is not a number. The critic also reads the ADRs of the case base-commit snapshot. The snapshot for case 650 (base 1280b01d) has no ADR-0279. So the critic's `adr_causes` was empty on a design that crosses a plugin boundary. ## Observed Run 650-20261008-112340 scored far_superior (4). Its design shares one plugin's class across the boundary, which ADR-0279 decision 4 forbids. No ADR text in the snapshot could flag it. ## Reproduce n/a: design ticket. See docs/handoffs/2026-10-08-takedown-autoresearch.md, section "ADR-blind designer built and rerun". ## Approach Grade every design the pipeline produces (ADR-0281 decision 4), not only the final one. Decide whether the critic reads ADRs from the current tree instead of the case snapshot. The decision needs an ADR if it reverses ADR-0281. ## Origin - Trigger: the 2026-10-08 ticket-takedown autoresearch session; freeze-and-merge bookkeeping on 2026-10-09. - Improvised this session: none. - Chain: final-only grade <- ADR-0281 decision 4 not built; critic ADR source <- ScratchRepo archives base_commit docs/adr (DESIGN, ADR-0281) - Root candidate: none. - Where: plugins/os-sdlc/eval/ticket-takedown (grader, ScratchRepo) - Session: 58499175-8e67-4b3a-8cfe-ed601216e8da - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-prism-architect/58499175-8e67-4b3a-8cfe-ed601216e8da.jsonl VERDICT: CREATE

jared added the

plugin/os-sdlc

filed-by/agent

labels

2026-10-09 16:58:42 +00:00

jared commented

2026-10-09 16:58:53 +00:00

Author

Owner

Skeptic DROP verdicts for related items, folded here for the owner. (1) ADR-blindness gaps: DenySettings writes Grep/Glob deny rules the claude CLI ignores (the Read rule in 3d78ceae already hides all three, so only cleanup); ticket text can name ADRs and reaches blind roles; map-researcher path still sees ADRs. (2) Numeric pass bar for ticket-takedown before it leaves EXPERIMENTAL: depends on the grading and ADR-visibility decisions here. (3) Owner rulings pending (kept in docs/handoffs/2026-10-08-takedown-autoresearch.md): #823 held-out reference File.exist? guard; 7 training cases still confirmed: false; may tickets carry design constraints (ADR-0209/0190/0255); 803 candidate lens tactic 'a dead optional argument stays after its last caller goes'; may a lens case cite a held-out or training eval ticket (Case 11 now uses invented names, fbbb708a). (4) Reword of cases/650 shipped.md files was done inline, not ticketed.

Skeptic DROP verdicts for related items, folded here for the owner. (1) ADR-blindness gaps: DenySettings writes Grep/Glob deny rules the claude CLI ignores (the Read rule in 3d78ceae already hides all three, so only cleanup); ticket text can name ADRs and reaches blind roles; map-researcher path still sees ADRs. (2) Numeric pass bar for ticket-takedown before it leaves EXPERIMENTAL: depends on the grading and ADR-visibility decisions here. (3) Owner rulings pending (kept in docs/handoffs/2026-10-08-takedown-autoresearch.md): #823 held-out reference File.exist? guard; 7 training cases still confirmed: false; may tickets carry design constraints (ADR-0209/0190/0255); 803 candidate lens tactic 'a dead optional argument stays after its last caller goes'; may a lens case cite a held-out or training eval ticket (Case 11 now uses invented names, fbbb708a). (4) Reword of cases/650 shipped.md files was done inline, not ticketed.

jared referenced this issue

2026-10-09 17:30:57 +00:00

Grade every takedown design against the stated cause, and compare each revision with the previous design #974

jared commented

2026-10-09 17:31:02 +00:00

Author

Owner

Related: #974 turns on the grader in live takedown runs (cause = researcher cause, baseline = ticket approach), per owner rulings 5 and 6 in docs/handoffs/2026-10-09-takedown-implement-path.md. This ticket stays scoped to per-design grading in the eval and the critic ADR source.