jared/cc-os

Watch

Fork 0

os-me: eval adapter choice moves to one shared object; row_labels gains --adapter #1291

New issue

Closed

opened 2026-10-23 21:44:55 +00:00 by jared · 2 comments

jared commented

2026-10-23 21:44:55 +00:00

Owner

Problem

Jev beat Sonnet on the ReplyShape holdout split: 88.0% against 85.3%, 4.6 s against 53 s, $0.0042 per run. The S10 lever evals must now run on Jev. They cannot do that today.

The adapter choice lives only inside plugins/os-me/eval/reply_shape/run.rb: ADAPTERS = %w[headless ollama jev], a jev_adapter method that reads CF_ACCOUNT_ID and CF_API_TOKEN with ENV.fetch, and a build_adapter case. plugins/os-me/eval/row_labels/run.rb (S10 T2, #1285) has no --adapter flag. Its build_client always builds OsMe::HeadlessClaude, and its --model default is OsMe::MomentExtractor::LABELER_MODEL ('sonnet').

Two more facts:

In reply_shape, --model goes to HeadlessClaude (with LABELER_MODEL as the fallback) and to SystemOne.ollama (gem default tev1:4b). The script ignores --model for jev, so the jev adapter always uses the gem default typesafe/jev.
In reply_shape, a missing CF variable raises a KeyError from run_all, outside the begin/rescue. The script then stops with a backtrace.

Approach

Add one lib object, OsMe::EvalAdapter, that maps an adapter name to a built Decision::Model adapter. Both eval scripts use it. The jev branch reads the two CF variables through an injectable env: (default ENV). If a variable is missing, build raises OsMe::Error with the variable name and never the value. An unknown name raises OsMe::Error with the list of names.

reply_shape keeps its flags, its validation message, its TSV, and its summary and cost lines. It only takes the name list and the adapter from the new object.

row_labels gets --adapter VALUE (default headless, so a run without the flag works as before). Its --model default becomes nil, and EvalAdapter applies LABELER_MODEL only on the headless branch. If the default stayed 'sonnet', ollama would get the wrong model. row_labels prints no cost line today, and LeverResult does not carry usage, so this ticket adds no cost line (see Scope).

The tests load EvalAdapter from lib/ and do not touch eval/. The worktree guard blocks shell paths that contain "eval". The two script-level criteria (7 and 8) extend the existing test/row_labels_run_test.rb, which already gets the script path from Ruby (File.expand_path('../eval/row_labels/run.rb', __dir__)) and starts it with Open3, not with a shell command line.

Required work:

Create OsMe::EvalAdapter with NAMES and .build(name, model: nil, env: ENV) as in the Owner sketch. check: present NAMES = %w[headless ollama jev].freeze in plugins/os-me/lib/os_me/eval_adapter.rb
In .build, raise OsMe::Error for an unknown name and list the names. check: present unknown adapter in plugins/os-me/lib/os_me/eval_adapter.rb
In reply_shape, remove ADAPTERS, jev_adapter, and build_adapter. check: absent def jev_adapter in plugins/os-me/eval/reply_shape/run.rb
In reply_shape, validate_choices uses OsMe::EvalAdapter::NAMES for --adapter. check: present adapter: OsMe::EvalAdapter::NAMES in plugins/os-me/eval/reply_shape/run.rb
In reply_shape, run_all gets its adapter from OsMe::EvalAdapter.build. check: present OsMe::EvalAdapter.build(options[:adapter], model: options[:model]) in plugins/os-me/eval/reply_shape/run.rb
In row_labels, add adapter: nil to FLAGS, set the defaults to adapter: 'headless', model: nil, and build the client from EvalAdapter. check: present OsMe::EvalAdapter.build(options[:adapter], model: options[:model]) in plugins/os-me/eval/row_labels/run.rb
Remove the row_labels --model default LABELER_MODEL. check: absent model: OsMe::MomentExtractor::LABELER_MODEL in plugins/os-me/eval/row_labels/run.rb
Add the EvalAdapter tests (criteria 1-6) and extend the row_labels script test (criteria 7-8). check: present class EvalAdapterTest in plugins/os-me/test/eval_adapter_test.rb

Acceptance criteria

Given no model, when OsMe::EvalAdapter.build('headless') runs, then the result is an OsMe::HeadlessClaude whose name is headless:sonnet.
Given model: 'haiku', when OsMe::EvalAdapter.build('headless', model: 'haiku') runs, then the result's name is headless:haiku.
Given model: 'tev1:4b', when OsMe::EvalAdapter.build('ollama', model: 'tev1:4b') runs, then the result is a Decision::Model::Adapters::SystemOne whose name is ollama:tev1:4b, and the test sends no request.
Given env: { 'CF_ACCOUNT_ID' => 'fake-account', 'CF_API_TOKEN' => 'fake-token' } and model: 'sonnet', when OsMe::EvalAdapter.build('jev', model: 'sonnet', env:) runs, then the result is a Decision::Model::Adapters::CloudflareJev whose name is cloudflare:typesafe/jev. The jev adapter ignores the model, as reply_shape does today.
Given env: { 'CF_ACCOUNT_ID' => 'fake-account' } (no token), when build('jev', env:) runs, then it raises OsMe::Error. The message contains CF_API_TOKEN and does not contain fake-account.
Given the name bogus, when OsMe::EvalAdapter.build('bogus') runs, then it raises OsMe::Error. The message contains bogus and headless, ollama, jev.
Given a valid tune labels file, when row_labels runs with --adapter bogus, then it exits 2, and stderr starts with error: and contains headless, ollama, jev.
Given a valid tune labels file and a child environment with CF_ACCOUNT_ID and CF_API_TOKEN unset (an Open3.capture3 env hash with nil values), when row_labels runs with --adapter jev, then it exits 2, stderr contains CF_ACCOUNT_ID, and the script makes no model call.

Owner sketch

# plugins/os-me/lib/os_me/eval_adapter.rb
module OsMe
  module EvalAdapter
    NAMES = %w[headless ollama jev].freeze
    CF_VARS = %w[CF_ACCOUNT_ID CF_API_TOKEN].freeze

    def self.build(name, model: nil, env: ENV)
      case name
      when 'headless' then HeadlessClaude.new(runner: HeadlessRunner.new, model: model || MomentExtractor::LABELER_MODEL)
      when 'ollama' then Decision::Model::Adapters::SystemOne.ollama(**{ model: model }.compact)
      when 'jev' then Decision::Model::Adapters::CloudflareJev.build(account: account(env))
      else raise Error, "unknown adapter #{name}; choose #{NAMES.join(', ')}"
      end
    end

    def self.account(env)
      account_id, api_token = CF_VARS.map { |key| env.fetch(key) { raise Error, "#{key} is not set; the jev adapter needs it" } }
      Decision::Model::Adapters::CloudflareAccount.new(account_id: account_id, api_token: api_token)
    end
    private_class_method :account
  end
end

# plugins/os-me/eval/row_labels/run.rb
#   FLAGS gains adapter: nil; parse_options defaults { jobs: 8, reps: 3, adapter: 'headless', model: nil }
#   check = OsMe::LeverCheck.for(options[:lever],
#             OsMe::LabelerClient.for(OsMe::EvalAdapter.build(options[:adapter], model: options[:model])))
#   The existing rescue of OsMe::Error gives "error: ..." and exit 2.

# plugins/os-me/eval/reply_shape/run.rb
#   validate_choices: { split: SPLITS, adapter: OsMe::EvalAdapter::NAMES }
#   run_all: adapter = OsMe::EvalAdapter.build(options[:adapter], model: options[:model])
#   The jev_cost_usd line stays as it is.

Scope and blocking

In scope: plugins/os-me/lib/os_me/eval_adapter.rb (new), plugins/os-me/eval/reply_shape/run.rb, plugins/os-me/eval/row_labels/run.rb, plugins/os-me/test/eval_adapter_test.rb (new), plugins/os-me/test/row_labels_run_test.rb.
Out of scope: a cost line for row_labels. LeverRun/LeverResult do not carry usage, so a jev_cost_usd line needs usage plumbing through LeverCheck first. Also out of scope: moving reply_shape's adapter build into its begin/rescue, a --model pass-through for jev, and RowLabels.build, which stays on headless.
Tests: no network and no model call. Adapter construction only builds connection objects. Tests pass fake env values through env: and do not change the real ENV.
Blocking: none. #1285 (row_labels) and #1274 (reply_shape --adapter) are merged.

Origin

Trigger: Jared asked on 2026-10-23 whether S10 should use Jev after its ReplyShape holdout beat Sonnet at a fraction of the cost
Improvised this session: none
Chain: S10 lever baselines ran on Sonnet ← row_labels has no adapter switch ← adapter choice lives inside reply_shape/run.rb only. Stop at CODE (plugins/os-me/eval/reply_shape/run.rb).
Root candidate: none found; searched ADAPTERS, jev_adapter, CloudflareJev
Where: plugins/os-me/eval/reply_shape/run.rb; plugins/os-me/eval/row_labels/run.rb
Session: 4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5.jsonl

VERDICT: CREATE (user-approved; backlog search only)

## Problem Jev beat Sonnet on the ReplyShape holdout split: 88.0% against 85.3%, 4.6 s against 53 s, $0.0042 per run. The S10 lever evals must now run on Jev. They cannot do that today. The adapter choice lives only inside `plugins/os-me/eval/reply_shape/run.rb`: `ADAPTERS = %w[headless ollama jev]`, a `jev_adapter` method that reads `CF_ACCOUNT_ID` and `CF_API_TOKEN` with `ENV.fetch`, and a `build_adapter` case. `plugins/os-me/eval/row_labels/run.rb` (S10 T2, #1285) has no `--adapter` flag. Its `build_client` always builds `OsMe::HeadlessClaude`, and its `--model` default is `OsMe::MomentExtractor::LABELER_MODEL` (`'sonnet'`). Two more facts: - In reply_shape, `--model` goes to `HeadlessClaude` (with `LABELER_MODEL` as the fallback) and to `SystemOne.ollama` (gem default `tev1:4b`). The script ignores `--model` for jev, so the jev adapter always uses the gem default `typesafe/jev`. - In reply_shape, a missing CF variable raises a `KeyError` from `run_all`, outside the `begin/rescue`. The script then stops with a backtrace. ## Approach Add one lib object, `OsMe::EvalAdapter`, that maps an adapter name to a built `Decision::Model` adapter. Both eval scripts use it. The jev branch reads the two CF variables through an injectable `env:` (default `ENV`). If a variable is missing, `build` raises `OsMe::Error` with the variable name and never the value. An unknown name raises `OsMe::Error` with the list of names. reply_shape keeps its flags, its validation message, its TSV, and its summary and cost lines. It only takes the name list and the adapter from the new object. row_labels gets `--adapter VALUE` (default `headless`, so a run without the flag works as before). Its `--model` default becomes `nil`, and `EvalAdapter` applies `LABELER_MODEL` only on the headless branch. If the default stayed `'sonnet'`, ollama would get the wrong model. row_labels prints no cost line today, and `LeverResult` does not carry usage, so this ticket adds no cost line (see Scope). The tests load `EvalAdapter` from `lib/` and do not touch `eval/`. The worktree guard blocks shell paths that contain "eval". The two script-level criteria (7 and 8) extend the existing `test/row_labels_run_test.rb`, which already gets the script path from Ruby (`File.expand_path('../eval/row_labels/run.rb', __dir__)`) and starts it with `Open3`, not with a shell command line. Required work: 1. Create `OsMe::EvalAdapter` with `NAMES` and `.build(name, model: nil, env: ENV)` as in the Owner sketch. check: present `NAMES = %w[headless ollama jev].freeze` in `plugins/os-me/lib/os_me/eval_adapter.rb` 2. In `.build`, raise `OsMe::Error` for an unknown name and list the names. check: present `unknown adapter` in `plugins/os-me/lib/os_me/eval_adapter.rb` 3. In reply_shape, remove `ADAPTERS`, `jev_adapter`, and `build_adapter`. check: absent `def jev_adapter` in `plugins/os-me/eval/reply_shape/run.rb` 4. In reply_shape, `validate_choices` uses `OsMe::EvalAdapter::NAMES` for `--adapter`. check: present `adapter: OsMe::EvalAdapter::NAMES` in `plugins/os-me/eval/reply_shape/run.rb` 5. In reply_shape, `run_all` gets its adapter from `OsMe::EvalAdapter.build`. check: present `OsMe::EvalAdapter.build(options[:adapter], model: options[:model])` in `plugins/os-me/eval/reply_shape/run.rb` 6. In row_labels, add `adapter: nil` to `FLAGS`, set the defaults to `adapter: 'headless', model: nil`, and build the client from `EvalAdapter`. check: present `OsMe::EvalAdapter.build(options[:adapter], model: options[:model])` in `plugins/os-me/eval/row_labels/run.rb` 7. Remove the row_labels `--model` default `LABELER_MODEL`. check: absent `model: OsMe::MomentExtractor::LABELER_MODEL` in `plugins/os-me/eval/row_labels/run.rb` 8. Add the `EvalAdapter` tests (criteria 1-6) and extend the row_labels script test (criteria 7-8). check: present `class EvalAdapterTest` in `plugins/os-me/test/eval_adapter_test.rb` ## Acceptance criteria 1. Given no model, when `OsMe::EvalAdapter.build('headless')` runs, then the result is an `OsMe::HeadlessClaude` whose `name` is `headless:sonnet`. 2. Given `model: 'haiku'`, when `OsMe::EvalAdapter.build('headless', model: 'haiku')` runs, then the result's `name` is `headless:haiku`. 3. Given `model: 'tev1:4b'`, when `OsMe::EvalAdapter.build('ollama', model: 'tev1:4b')` runs, then the result is a `Decision::Model::Adapters::SystemOne` whose `name` is `ollama:tev1:4b`, and the test sends no request. 4. Given `env: { 'CF_ACCOUNT_ID' => 'fake-account', 'CF_API_TOKEN' => 'fake-token' }` and `model: 'sonnet'`, when `OsMe::EvalAdapter.build('jev', model: 'sonnet', env:)` runs, then the result is a `Decision::Model::Adapters::CloudflareJev` whose `name` is `cloudflare:typesafe/jev`. The jev adapter ignores the model, as reply_shape does today. 5. Given `env: { 'CF_ACCOUNT_ID' => 'fake-account' }` (no token), when `build('jev', env:)` runs, then it raises `OsMe::Error`. The message contains `CF_API_TOKEN` and does not contain `fake-account`. 6. Given the name `bogus`, when `OsMe::EvalAdapter.build('bogus')` runs, then it raises `OsMe::Error`. The message contains `bogus` and `headless, ollama, jev`. 7. Given a valid tune labels file, when row_labels runs with `--adapter bogus`, then it exits 2, and stderr starts with `error:` and contains `headless, ollama, jev`. 8. Given a valid tune labels file and a child environment with `CF_ACCOUNT_ID` and `CF_API_TOKEN` unset (an `Open3.capture3` env hash with nil values), when row_labels runs with `--adapter jev`, then it exits 2, stderr contains `CF_ACCOUNT_ID`, and the script makes no model call. ## Owner sketch ```ruby # plugins/os-me/lib/os_me/eval_adapter.rb module OsMe module EvalAdapter NAMES = %w[headless ollama jev].freeze CF_VARS = %w[CF_ACCOUNT_ID CF_API_TOKEN].freeze def self.build(name, model: nil, env: ENV) case name when 'headless' then HeadlessClaude.new(runner: HeadlessRunner.new, model: model || MomentExtractor::LABELER_MODEL) when 'ollama' then Decision::Model::Adapters::SystemOne.ollama(**{ model: model }.compact) when 'jev' then Decision::Model::Adapters::CloudflareJev.build(account: account(env)) else raise Error, "unknown adapter #{name}; choose #{NAMES.join(', ')}" end end def self.account(env) account_id, api_token = CF_VARS.map { |key| env.fetch(key) { raise Error, "#{key} is not set; the jev adapter needs it" } } Decision::Model::Adapters::CloudflareAccount.new(account_id: account_id, api_token: api_token) end private_class_method :account end end # plugins/os-me/eval/row_labels/run.rb # FLAGS gains adapter: nil; parse_options defaults { jobs: 8, reps: 3, adapter: 'headless', model: nil } # check = OsMe::LeverCheck.for(options[:lever], # OsMe::LabelerClient.for(OsMe::EvalAdapter.build(options[:adapter], model: options[:model]))) # The existing rescue of OsMe::Error gives "error: ..." and exit 2. # plugins/os-me/eval/reply_shape/run.rb # validate_choices: { split: SPLITS, adapter: OsMe::EvalAdapter::NAMES } # run_all: adapter = OsMe::EvalAdapter.build(options[:adapter], model: options[:model]) # The jev_cost_usd line stays as it is. ``` ## Scope and blocking - In scope: `plugins/os-me/lib/os_me/eval_adapter.rb` (new), `plugins/os-me/eval/reply_shape/run.rb`, `plugins/os-me/eval/row_labels/run.rb`, `plugins/os-me/test/eval_adapter_test.rb` (new), `plugins/os-me/test/row_labels_run_test.rb`. - Out of scope: a cost line for row_labels. `LeverRun`/`LeverResult` do not carry usage, so a `jev_cost_usd` line needs usage plumbing through `LeverCheck` first. Also out of scope: moving reply_shape's adapter build into its `begin/rescue`, a `--model` pass-through for jev, and `RowLabels.build`, which stays on headless. - Tests: no network and no model call. Adapter construction only builds connection objects. Tests pass fake env values through `env:` and do not change the real `ENV`. - Blocking: none. #1285 (row_labels) and #1274 (reply_shape `--adapter`) are merged. ## Origin - Trigger: Jared asked on 2026-10-23 whether S10 should use Jev after its ReplyShape holdout beat Sonnet at a fraction of the cost - Improvised this session: none - Chain: S10 lever baselines ran on Sonnet ← row_labels has no adapter switch ← adapter choice lives inside reply_shape/run.rb only. Stop at `CODE` (`plugins/os-me/eval/reply_shape/run.rb`). - Root candidate: none found; searched ADAPTERS, jev_adapter, CloudflareJev - Where: plugins/os-me/eval/reply_shape/run.rb; plugins/os-me/eval/row_labels/run.rb - Session: 4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5 - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5.jsonl VERDICT: CREATE (user-approved; backlog search only)

jared added the

project/cc-os

filed-by/user

category/code-change

labels

2026-10-23 21:44:55 +00:00

jared commented

2026-10-23 21:58:13 +00:00

Author

Owner

Work started on branch ticket-1291 (os-sdlc implement).

jared commented

2026-10-23 22:02:56 +00:00

Author

Owner

Resolution

Done: OsMe::EvalAdapter owns adapter selection by name; reply_shape and row_labels both use it; row_labels gains --adapter headless|ollama|jev

Evidence: Implementation 153 complete in 4 min 23 s, 8 dispatches, all pass; suite gate green

Follow-ups: row_labels cost line needs usage carried through LeverRun (deferred, not filed: no current need); one LENS line (A6, ENV default) goes to the lens challenge

## Resolution **Done:** OsMe::EvalAdapter owns adapter selection by name; reply_shape and row_labels both use it; row_labels gains --adapter headless|ollama|jev **Evidence:** Implementation 153 complete in 4 min 23 s, 8 dispatches, all pass; suite gate green **Follow-ups:** row_labels cost line needs usage carried through LeverRun (deferred, not filed: no current need); one LENS line (A6, ENV default) goes to the lens challenge

jared closed this issue

2026-10-23 22:02:56 +00:00

jared referenced this issue from a commit

2026-10-23 22:04:01 +00:00

feat(os-me): Audited POODR boundaries for #1291: OsMe::EvalAdapter owns adapter selection by name; both eval scripts ask it and keep their own flag and error handling; contracts held; R1-R8 present. (#1291)

jared referenced this issue from a commit

2026-10-23 22:04:01 +00:00

docs(os-me): Record #1291 eval adapter switch (#1291)

jared referenced this issue

2026-10-23 22:06:09 +00:00

os-me: EvalAdapter.build takes env: from the entry script, not an ENV default #1295