Switch os-me decision-model callers to Jev as the default adapter #1286

New issue

Open

opened 2026-10-23 20:37:33 +00:00 by jared · 0 comments

jared commented

2026-10-23 20:37:33 +00:00

Owner

Problem

Jared ruled on 2026-10-23 that Jev is the default decision model everywhere, until the local models are tuned. os-me still asks its decision-model questions through OsMe::HeadlessClaude (Claude Sonnet). Each Sonnet call costs more and takes about 11 times longer than a Jev call for the same agreement.

Observed

#1274 measured ReplyShape with the tuned criteria, 3 reps:

Adapter	Holdout (25)	Frozen 50	Holdout wall time	Holdout cost
Jev (`CloudflareJev`)	88.0% (kappa 0.825)	80.0% (kappa 0.684; sure-only 93.8%)	4.6 s	$0.0042
Sonnet (`HeadlessClaude`)	85.3% (kappa 0.789)	83.3%	53.0 s	not recorded

The ReplyShape gate is at least 80% on the frozen 50. Jev passes it with zero margin.

Reproduce

n/a: design change, nothing to reproduce.

Approach

Build every os-me decision-model client on Decision::Model::Adapters::CloudflareJev. The account comes from CF_ACCOUNT_ID and CF_API_TOKEN (credvault). A missing variable must stop the run at once (ENV.fetch with no default). Callers in scope: ReplyShape, AnswerSource, WrongCost::Assessor, RowLabels, MomentExtractor, MomentKind. Decompose before ready-for-agent: list each construction site and its test seam.

Orchestrator work: re-check the S7 moment-kind gate (80% agreement on a checked sample) with Jev, because S7 was measured with Sonnet.

Non-goals

Tuning the local Ollama models.
Deleting OsMe::HeadlessClaude. It stays as an adapter for comparison runs.

Origin

Trigger: Jared's ruling in session c651c4e1 after the #1274 holdout and frozen-50 runs.
Improvised this session: none
Chain: os-me labels with Sonnet ← S8 built the gem adapters but kept HeadlessClaude as the default ← no default ruling before #1274. Stop at DESIGN (docs/decision-system/build-plan.md S8).
Root candidate: none found; searched Jev, CloudflareJev, HeadlessClaude, decision-model, adapter, ReplyShape, MomentExtractor, LABELER_MODEL
Where: plugins/os-me/lib/os_me/moment_extractor.rb (LABELER_MODEL); plugins/os-me/lib/os_me/headless_claude.rb
Session: c651c4e1-e3c2-4331-b329-4e40198a4391
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/c651c4e1-e3c2-4331-b329-4e40198a4391.jsonl

VERDICT: CREATE (user-approved; backlog search only)

## Problem Jared ruled on 2026-10-23 that Jev is the default decision model everywhere, until the local models are tuned. os-me still asks its decision-model questions through `OsMe::HeadlessClaude` (Claude Sonnet). Each Sonnet call costs more and takes about 11 times longer than a Jev call for the same agreement. ## Observed #1274 measured ReplyShape with the tuned criteria, 3 reps: | Adapter | Holdout (25) | Frozen 50 | Holdout wall time | Holdout cost | |---|---|---|---|---| | Jev (`CloudflareJev`) | 88.0% (kappa 0.825) | 80.0% (kappa 0.684; sure-only 93.8%) | 4.6 s | $0.0042 | | Sonnet (`HeadlessClaude`) | 85.3% (kappa 0.789) | 83.3% | 53.0 s | not recorded | The ReplyShape gate is at least 80% on the frozen 50. Jev passes it with zero margin. ## Reproduce n/a: design change, nothing to reproduce. ## Approach Build every os-me decision-model client on `Decision::Model::Adapters::CloudflareJev`. The account comes from `CF_ACCOUNT_ID` and `CF_API_TOKEN` (credvault). A missing variable must stop the run at once (`ENV.fetch` with no default). Callers in scope: `ReplyShape`, `AnswerSource`, `WrongCost::Assessor`, `RowLabels`, `MomentExtractor`, `MomentKind`. Decompose before ready-for-agent: list each construction site and its test seam. Orchestrator work: re-check the S7 moment-kind gate (80% agreement on a checked sample) with Jev, because S7 was measured with Sonnet. ## Non-goals - Tuning the local Ollama models. - Deleting `OsMe::HeadlessClaude`. It stays as an adapter for comparison runs. ## Origin - Trigger: Jared's ruling in session c651c4e1 after the #1274 holdout and frozen-50 runs. - Improvised this session: none - Chain: os-me labels with Sonnet ← S8 built the gem adapters but kept `HeadlessClaude` as the default ← no default ruling before #1274. Stop at `DESIGN` (`docs/decision-system/build-plan.md` S8). - Root candidate: none found; searched Jev, CloudflareJev, HeadlessClaude, decision-model, adapter, ReplyShape, MomentExtractor, LABELER_MODEL - Where: plugins/os-me/lib/os_me/moment_extractor.rb (`LABELER_MODEL`); plugins/os-me/lib/os_me/headless_claude.rb - Session: c651c4e1-e3c2-4331-b329-4e40198a4391 - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/c651c4e1-e3c2-4331-b329-4e40198a4391.jsonl VERDICT: CREATE (user-approved; backlog search only)

jared added the

project/cc-os

filed-by/user

category/code-change

labels

2026-10-23 20:37:33 +00:00

jared referenced this issue from a commit

2026-10-23 21:14:35 +00:00

feat(os-me): Add the decision-model gem skill; record Jev frozen-50 result (#1274)

jared referenced this issue

2026-10-23 21:14:43 +00:00

decision-system S8 gate: compare HeadlessClaude and the Ollama adapter on the ReplyShape sample #1274