decision-system S8 gate: compare HeadlessClaude and the Ollama adapter on the ReplyShape sample #1274

New issue

Closed

opened 2026-10-23 18:40:16 +00:00 by jared · 2 comments

jared commented

2026-10-23 18:40:16 +00:00

Owner

Problem

S8 of docs/decision-system/build-plan.md has a gate: the chosen adapter's agreement is not worse than HeadlessClaude on the sample, and its cost per run is lower. #1250 and #1266 built the gem move but did not run the comparison. The S8 row stays "gate open" until this run.

Approach

Run plugins/os-me/eval/reply_shape/run.rb once per adapter: OsMe::HeadlessClaude (Sonnet) and the gem Ollama adapter. Record agreement, kappa, unreadable count, and cost per run for each. The frozen 50 is the final check only. Use the tune and holdout splits to choose, then read the frozen 50 once for the chosen adapter.

Required work: none beyond the acceptance criteria.

Orchestrator work: update the S8 row in docs/decision-system/build-plan.md and add one History row to docs/implementation-status/os-me.md.

Acceptance criteria

Given both adapters, when the eval runs on the holdout split, then a table shows agreement, kappa, unreadable count, and cost per run for each adapter.
Given the table, when one adapter is chosen, then the frozen 50 is read once for it and the S8 gate verdict (pass or fail) is written in the build plan.

Origin

Trigger: #1266 close; S8 gate left open
Improvised this session: none
Chain: S8 gate open ← comparison not in #1250 or #1266 scope ← stage split. Stop at DESIGN (docs/decision-system/build-plan.md S8).
Root candidate: none found; searched S8 gate, HeadlessClaude, Ollama adapter, ReplyShape, eval/reply_shape, adapter comparison
Where: plugins/os-me/eval/reply_shape/run.rb
Session: 4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-ticket-1250/4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5.jsonl

VERDICT: CREATE (user-approved; backlog search only)

## Problem S8 of `docs/decision-system/build-plan.md` has a gate: the chosen adapter's agreement is not worse than `HeadlessClaude` on the sample, and its cost per run is lower. #1250 and #1266 built the gem move but did not run the comparison. The S8 row stays "gate open" until this run. ## Approach Run `plugins/os-me/eval/reply_shape/run.rb` once per adapter: `OsMe::HeadlessClaude` (Sonnet) and the gem Ollama adapter. Record agreement, kappa, unreadable count, and cost per run for each. The frozen 50 is the final check only. Use the tune and holdout splits to choose, then read the frozen 50 once for the chosen adapter. Required work: none beyond the acceptance criteria. Orchestrator work: update the S8 row in `docs/decision-system/build-plan.md` and add one History row to `docs/implementation-status/os-me.md`. ## Acceptance criteria 1. Given both adapters, when the eval runs on the holdout split, then a table shows agreement, kappa, unreadable count, and cost per run for each adapter. 2. Given the table, when one adapter is chosen, then the frozen 50 is read once for it and the S8 gate verdict (pass or fail) is written in the build plan. ## Origin - Trigger: #1266 close; S8 gate left open - Improvised this session: none - Chain: S8 gate open ← comparison not in #1250 or #1266 scope ← stage split. Stop at `DESIGN` (`docs/decision-system/build-plan.md` S8). - Root candidate: none found; searched S8 gate, HeadlessClaude, Ollama adapter, ReplyShape, eval/reply_shape, adapter comparison - Where: plugins/os-me/eval/reply_shape/run.rb - Session: 4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5 - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-ticket-1250/4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5.jsonl VERDICT: CREATE (user-approved; backlog search only)

jared added the

project/cc-os

filed-by/user

category/eval-run

labels

2026-10-23 18:40:16 +00:00

jared referenced this issue

2026-10-23 18:40:43 +00:00

decision-system S8b: move os-me callers to Decision::Model and delete the port #1266

jared commented

2026-10-23 19:39:13 +00:00

Author

Owner

Work started on branch ticket-1274 (worktree).

jared referenced this issue

2026-10-23 20:37:33 +00:00

Switch os-me decision-model callers to Jev as the default adapter #1286