Fold os-me eval/reply_shape/run.rb onto the os-me harness classes #1422

New issue

Closed

opened 2026-10-27 16:37:33 +00:00 by jared · 1 comment

jared commented

2026-10-27 16:37:33 +00:00

Owner

Problem

plugins/os-me/eval/reply_shape/run.rb repeats logic that os-me harness classes already own.

Flag parsing: lib/os_me/row_labels/eval_harness.rb:8-61 vs eval/reply_shape/run.rb:11-41.
Reps x jobs pool: lever_run.rb:13-36 vs reply_shape/run.rb:64-88.
Agreement and kappa: agreement_score.rb:17-61 vs reply_shape/run.rb:90-124.

Reproduce

n/a: duplication; no behavior to reproduce.

Approach

Make reply_shape/run.rb use LeverRun, AgreementScore, and one flag-parsing owner.
Capture the current summary lines and TSV output first, then compare after the change (characterization comparison).

Required work:

Remove the duplicated kappa code from the script — check: absent kappa in plugins/os-me/eval/reply_shape/run.rb

Acceptance criteria

Given the change, when reply_shape/run.rb runs, then it uses LeverRun, AgreementScore, and one flag-parsing owner.
Given a characterization comparison against the pre-change output, when the script runs, then the summary lines and TSV columns are unchanged.

Owner sketch

class OsMe::RowLabels::EvalHarness
  def parse_flags(argv)
end

Origin

Trigger: proposed ticket 1 in #1413.
Improvised this session: none
Chain: duplicate code ← script written beside the harness ← DESIGN (plugins/os-me/eval/reply_shape/run.rb)
Root candidate: none found; searched reply_shape, LeverRun, AgreementScore, kappa
Where: plugins/os-me/eval/reply_shape/run.rb
Session: c843bd27-b3bf-4e36-9243-df1697457146
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/c843bd27-b3bf-4e36-9243-df1697457146.jsonl

References #1413.
VERDICT: CREATE

## Problem `plugins/os-me/eval/reply_shape/run.rb` repeats logic that os-me harness classes already own. - Flag parsing: `lib/os_me/row_labels/eval_harness.rb:8-61` vs `eval/reply_shape/run.rb:11-41`. - Reps x jobs pool: `lever_run.rb:13-36` vs `reply_shape/run.rb:64-88`. - Agreement and kappa: `agreement_score.rb:17-61` vs `reply_shape/run.rb:90-124`. ## Reproduce n/a: duplication; no behavior to reproduce. ## Approach Make `reply_shape/run.rb` use `LeverRun`, `AgreementScore`, and one flag-parsing owner. Capture the current summary lines and TSV output first, then compare after the change (characterization comparison). Required work: 1. Remove the duplicated kappa code from the script — check: absent `kappa` in `plugins/os-me/eval/reply_shape/run.rb` ## Acceptance criteria 1. Given the change, when `reply_shape/run.rb` runs, then it uses `LeverRun`, `AgreementScore`, and one flag-parsing owner. 2. Given a characterization comparison against the pre-change output, when the script runs, then the summary lines and TSV columns are unchanged. ## Owner sketch ```ruby class OsMe::RowLabels::EvalHarness def parse_flags(argv) end ``` ## Origin - Trigger: proposed ticket 1 in #1413. - Improvised this session: none - Chain: duplicate code ← script written beside the harness ← DESIGN (plugins/os-me/eval/reply_shape/run.rb) - Root candidate: none found; searched reply_shape, LeverRun, AgreementScore, kappa - Where: plugins/os-me/eval/reply_shape/run.rb - Session: c843bd27-b3bf-4e36-9243-df1697457146 - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/c843bd27-b3bf-4e36-9243-df1697457146.jsonl References #1413. VERDICT: CREATE

jared added the

project/cc-os

filed-by/user

category/code-change

labels

2026-10-27 16:37:33 +00:00

jared referenced this issue

2026-10-27 16:37:51 +00:00

Make os-me eval harnesses reuse shared eval harness code instead of duplicating it #1413

jared commented

2026-10-27 16:57:01 +00:00

Author

Owner

Resolution

Done: reply_shape/run.rb now uses LeverRun, AgreementScore and the new OsMe::EvalOptions (sole flag parser, shared with RowLabels::EvalHarness). New ReplyShapeCheck, ReplyShapeLabel, ReplyShapeReport; kappa absent from run.rb. Declared changes: a tied majority vote counts as a miss (old rule: rep1 wins); unknown/unusable adapter exits 2 with EvalAdapter's message. Hand-finished outside the pipeline: a behavior-preserving ticket fails at guard-map test-reviewer (#1416).

Evidence: main 1d3ad4e9 (cee58623, 2ec320aa). Characterization test passes against the pre-change run.rb and after; os-me suite 406 runs 0 failures; rubocop clean. Fresh opus review verdict: merge; its four follow-ups applied in 2ec320aa.

Follow-ups: none

## Resolution **Done:** reply_shape/run.rb now uses LeverRun, AgreementScore and the new OsMe::EvalOptions (sole flag parser, shared with RowLabels::EvalHarness). New ReplyShapeCheck, ReplyShapeLabel, ReplyShapeReport; kappa absent from run.rb. Declared changes: a tied majority vote counts as a miss (old rule: rep1 wins); unknown/unusable adapter exits 2 with EvalAdapter's message. Hand-finished outside the pipeline: a behavior-preserving ticket fails at guard-map test-reviewer (#1416). **Evidence:** main 1d3ad4e9 (cee58623, 2ec320aa). Characterization test passes against the pre-change run.rb and after; os-me suite 406 runs 0 failures; rubocop clean. Fresh opus review verdict: merge; its four follow-ups applied in 2ec320aa. **Follow-ups:** none

jared closed this issue

2026-10-27 16:57:01 +00:00

jared referenced this issue from a commit

2026-10-27 17:10:02 +00:00

refactor(os-me): fold reply_shape eval onto LeverRun, AgreementScore and shared flag parsing (#1422)

jared referenced this issue from a commit

2026-10-27 17:10:02 +00:00

refactor(os-me): move reply_shape line formatting out of AgreementScore; pin tie and adapter changes (#1422)