takedown metrics.tsv records $0 cost for rejected polish rounds #1017

New issue

Open

opened 2026-10-16 11:39:11 +00:00 by jared · 0 comments

jared commented

2026-10-16 11:39:11 +00:00

Owner

Problem

The takedown run record under-reports cost. A rejected polish round loses the model role and cost of its model call.

Observed

In run docs/takedown/1010-20261015-175013, polish-4 was rejected.
metrics.tsv logs the polish-4 row with role driver and cost 0.
24-polish-4.json in the same run directory shows a real model call that cost about $0.0698.
The run total cost is lower than the real cost.

Reproduce

n/a: see run docs/takedown/1010-20261015-175013, row polish-4 in metrics.tsv.

Approach

In plugins/os-sdlc/eval/ticket-takedown/lib/takedown/command/polish_step.rb, StepOutcome.failed("driver", ...) replaces the model call's outcome. The model role and cost are lost.
A rejected round must keep the model role and its real cost, and must still record the rejection.

Required work:

Keep the model outcome's role and cost on a rejected polish round — check: absent StepOutcome.failed("driver", "polish round in plugins/os-sdlc/eval/ticket-takedown/lib/takedown/command/polish_step.rb

Acceptance criteria

Given a polish round that the pipeline rejects, when the step writes its metrics.tsv row, then the row has the model role and the real cost of the model call.
Given a rejected polish round, when the run ends, then the rejection is still recorded.

Origin

Trigger: tracer run 1010-20261015-175013, polish-4 rejected
Improvised this session: none
Chain: metrics.tsv cost 0 for polish-4 ← StepOutcome.failed("driver", ...) replaces the model outcome (plugins/os-sdlc/eval/ticket-takedown/lib/takedown/command/polish_step.rb:20) ← DESIGN (polish_step.rb)
Root candidate: none
Where: plugins/os-sdlc/eval/ticket-takedown/lib/takedown/command/polish_step.rb
Session: 45b8f46a-1b27-4c0c-a606-a1d5073eef03
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-prism-architect/45b8f46a-1b27-4c0c-a606-a1d5073eef03.jsonl

## Problem The takedown run record under-reports cost. A rejected polish round loses the model role and cost of its model call. ## Observed In run `docs/takedown/1010-20261015-175013`, polish-4 was rejected. `metrics.tsv` logs the polish-4 row with role `driver` and cost 0. `24-polish-4.json` in the same run directory shows a real model call that cost about $0.0698. The run total cost is lower than the real cost. ## Reproduce n/a: see run `docs/takedown/1010-20261015-175013`, row polish-4 in `metrics.tsv`. ## Approach In `plugins/os-sdlc/eval/ticket-takedown/lib/takedown/command/polish_step.rb`, `StepOutcome.failed("driver", ...)` replaces the model call's outcome. The model role and cost are lost. A rejected round must keep the model role and its real cost, and must still record the rejection. Required work: 1. Keep the model outcome's role and cost on a rejected polish round — check: absent `StepOutcome.failed("driver", "polish round` in `plugins/os-sdlc/eval/ticket-takedown/lib/takedown/command/polish_step.rb` ## Acceptance criteria 1. Given a polish round that the pipeline rejects, when the step writes its `metrics.tsv` row, then the row has the model role and the real cost of the model call. 2. Given a rejected polish round, when the run ends, then the rejection is still recorded. ## Origin - Trigger: tracer run `1010-20261015-175013`, polish-4 rejected - Improvised this session: none - Chain: metrics.tsv cost 0 for polish-4 ← `StepOutcome.failed("driver", ...)` replaces the model outcome (plugins/os-sdlc/eval/ticket-takedown/lib/takedown/command/polish_step.rb:20) ← DESIGN (polish_step.rb) - Root candidate: none - Where: plugins/os-sdlc/eval/ticket-takedown/lib/takedown/command/polish_step.rb - Session: 45b8f46a-1b27-4c0c-a606-a1d5073eef03 - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-prism-architect/45b8f46a-1b27-4c0c-a606-a1d5073eef03.jsonl

jared added the

plugin/os-sdlc

filed-by/user

category/code-change

labels

2026-10-16 11:39:11 +00:00

jared referenced this issue

2026-10-16 12:05:01 +00:00

Fix MutableCollectionLeakViaAttrReader: skip private readers, report every attr, decide by reader return #1010