jared/cc-os

Watch

Fork 0

Make os-me eval harnesses reuse shared eval harness code instead of duplicating it #1413

New issue

Open

opened 2026-10-26 21:50:08 +00:00 by jared · 3 comments

jared commented

2026-10-26 21:50:08 +00:00

Owner

Problem

The eval harnesses repeat the same work. fast-batch-15 ruling R5 moved the bodies of plugins/os-me/eval/row_labels/run.rb and confidence_extractor/run.rb into OsMe::RowLabels::EvalHarness and OsMe::ConfidenceExtractor::EvalHarness (commit 58fea20a). Each class parses the same kind of flags (lever, labels, split, out, reps, jobs, model, adapter) and runs reps across jobs. os-sdlc has its own eval harness under plugins/os-sdlc/eval/ticket-takedown. Jared (2026-10-26): "The eval harness code should tie into the os-sdlc code wherever reasonable. No sense in duplicating functionality."

Reproduce

n/a: investigation. Compare plugins/os-me/lib/os_me/row_labels/eval_harness.rb, plugins/os-me/lib/os_me/confidence_extractor/eval_harness.rb, and the harness under plugins/os-sdlc/eval/ticket-takedown.

Approach

List what the harnesses duplicate: flag parsing, rep and job scheduling, model and adapter choice, score output.
Choose one owner for each duplicated job. The plugins install independently, so an os-me dependency on os-sdlc code is an architecture question. Run /os-adr:find before you choose.
File one code-change ticket for each move that the investigation proves.

Origin

Trigger: Jared review of fast-batch-15 ruling R5, 2026-10-26
Improvised this session: none
Chain: duplicated harness code ← each eval grew its own run.rb ← no shared eval owner (DESIGN)
Root candidate: none found; searched eval harness, EvalHarness, reps, jobs, row_labels, confidence_extractor, autoresearch, duplicate, ticket-takedown
Where: plugins/os-me/lib/os_me/*/eval_harness.rb; plugins/os-sdlc/eval/ticket-takedown
Session: 6c6b6eae-4ebd-4ac4-9b32-535875a6b39b
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-fast-batch-15/6c6b6eae-4ebd-4ac4-9b32-535875a6b39b.jsonl

VERDICT: CREATE

## Problem The eval harnesses repeat the same work. fast-batch-15 ruling R5 moved the bodies of `plugins/os-me/eval/row_labels/run.rb` and `confidence_extractor/run.rb` into `OsMe::RowLabels::EvalHarness` and `OsMe::ConfidenceExtractor::EvalHarness` (commit 58fea20a). Each class parses the same kind of flags (`lever`, `labels`, `split`, `out`, `reps`, `jobs`, `model`, `adapter`) and runs reps across jobs. os-sdlc has its own eval harness under `plugins/os-sdlc/eval/ticket-takedown`. Jared (2026-10-26): "The eval harness code should tie into the os-sdlc code wherever reasonable. No sense in duplicating functionality." ## Reproduce n/a: investigation. Compare `plugins/os-me/lib/os_me/row_labels/eval_harness.rb`, `plugins/os-me/lib/os_me/confidence_extractor/eval_harness.rb`, and the harness under `plugins/os-sdlc/eval/ticket-takedown`. ## Approach 1. List what the harnesses duplicate: flag parsing, rep and job scheduling, model and adapter choice, score output. 2. Choose one owner for each duplicated job. The plugins install independently, so an os-me dependency on os-sdlc code is an architecture question. Run `/os-adr:find` before you choose. 3. File one code-change ticket for each move that the investigation proves. ## Origin - Trigger: Jared review of fast-batch-15 ruling R5, 2026-10-26 - Improvised this session: none - Chain: duplicated harness code ← each eval grew its own run.rb ← no shared eval owner (DESIGN) - Root candidate: none found; searched eval harness, EvalHarness, reps, jobs, row_labels, confidence_extractor, autoresearch, duplicate, ticket-takedown - Where: plugins/os-me/lib/os_me/*/eval_harness.rb; plugins/os-sdlc/eval/ticket-takedown - Session: 6c6b6eae-4ebd-4ac4-9b32-535875a6b39b - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-fast-batch-15/6c6b6eae-4ebd-4ac4-9b32-535875a6b39b.jsonl VERDICT: CREATE

jared added the

project/cc-os

plugin/os-sdlc

filed-by/user

category/investigation

labels

2026-10-26 21:50:08 +00:00

jared commented

2026-10-27 16:16:45 +00:00

Author

Owner

Findings (2026-10-27, takedown track; read-only investigation, nothing built, no cross-plugin owner chosen).

Only one capability is duplicated across plugins: the headless claude -p call and its reply parsing, in three copies.

argv: os-me lib/os_me/headless_runner.rb:11-19; takedown lib/takedown/model/model_request.rb:8-13; os-adr lib/adr/judge/claude_command.rb:24-28. Same core flags (--output-format json, --no-session-persistence, --setting-sources '', --json-schema, --system-prompt).
process: os-me headless_runner.rb:31-36 (capture3, no timeout); takedown adapter/process_runner.rb:3-50 (stdin, timeout, pgroup kill).
reply parse: os-me headless_runner.rb:48-62; takedown parser/claude_reply.rb:30-70 (same keys: usage, total_cost_usd, structured_output, is_error).

The other harness duplication is inside os-me only, so there is no os-sdlc eval harness for os-me to reuse there: flag parsing (row_labels/eval_harness.rb:8-61 vs eval/reply_shape/run.rb:11-41), reps x jobs pool (lever_run.rb:13-36 vs reply_shape/run.rb:64-88), agreement/kappa (agreement_score.rb:17-61 vs reply_shape/run.rb:90-124).

ADRs found: ADR-0279 (shared behavior crosses the plugin boundary only through the domain owner's gem), ADR-0093 (os-sdlc may depend on os-backlog, not the reverse; nothing lets os-me depend on os-sdlc), ADR-0166 (one boot file per plugin sets BUNDLE_GEMFILE; requiring Takedown from os-me would load two bundles), ADR-0219 (precedent: procman gem for shared behavior).

Options for the claude call (needs Jared's ruling, likely a new ADR):
A. os-me requires os-sdlc code: require File.expand_path('../../../os-sdlc/eval/ticket-takedown/lib/takedown', __dir__) — conflicts with ADR-0279 and ADR-0166.
B. New gem on the procman pattern: gem 'claude-headless', path: '~/dev/ruby-gems/claude-headless'; ClaudeHeadless::Request.new(model:, schema:, system_prompt:, tools:).call(prompt, cwd:) -> ClaudeHeadless::Reply. Fits ADR-0279 and ADR-0219.
C. Keep the copies across plugins; fix only the os-me internal duplicates.

Proposed tickets, in order:

Fold eval/reply_shape/run.rb onto the os-me harness classes (LeverRun, AgreementScore, one flag-parsing owner); summary lines and TSV columns unchanged. No ruling needed.
One owner for the headless claude call (after the A/B/C ruling): one class builds argv, runs with stdin and timeout, parses usage/cost/structured output; HeadlessRunner, ModelRequest/ClaudeReply, and Adr::Judge::ClaudeCommand call it; an ADR records the dependency.

Findings (2026-10-27, takedown track; read-only investigation, nothing built, no cross-plugin owner chosen). Only one capability is duplicated across plugins: the headless `claude -p` call and its reply parsing, in three copies. - argv: os-me `lib/os_me/headless_runner.rb:11-19`; takedown `lib/takedown/model/model_request.rb:8-13`; os-adr `lib/adr/judge/claude_command.rb:24-28`. Same core flags (`--output-format json`, `--no-session-persistence`, `--setting-sources ''`, `--json-schema`, `--system-prompt`). - process: os-me `headless_runner.rb:31-36` (capture3, no timeout); takedown `adapter/process_runner.rb:3-50` (stdin, timeout, pgroup kill). - reply parse: os-me `headless_runner.rb:48-62`; takedown `parser/claude_reply.rb:30-70` (same keys: usage, total_cost_usd, structured_output, is_error). The other harness duplication is inside os-me only, so there is no os-sdlc eval harness for os-me to reuse there: flag parsing (`row_labels/eval_harness.rb:8-61` vs `eval/reply_shape/run.rb:11-41`), reps x jobs pool (`lever_run.rb:13-36` vs `reply_shape/run.rb:64-88`), agreement/kappa (`agreement_score.rb:17-61` vs `reply_shape/run.rb:90-124`). ADRs found: ADR-0279 (shared behavior crosses the plugin boundary only through the domain owner's gem), ADR-0093 (os-sdlc may depend on os-backlog, not the reverse; nothing lets os-me depend on os-sdlc), ADR-0166 (one boot file per plugin sets BUNDLE_GEMFILE; requiring Takedown from os-me would load two bundles), ADR-0219 (precedent: procman gem for shared behavior). Options for the claude call (needs Jared's ruling, likely a new ADR): A. os-me requires os-sdlc code: `require File.expand_path('../../../os-sdlc/eval/ticket-takedown/lib/takedown', __dir__)` — conflicts with ADR-0279 and ADR-0166. B. New gem on the procman pattern: `gem 'claude-headless', path: '~/dev/ruby-gems/claude-headless'`; `ClaudeHeadless::Request.new(model:, schema:, system_prompt:, tools:).call(prompt, cwd:)` -> `ClaudeHeadless::Reply`. Fits ADR-0279 and ADR-0219. C. Keep the copies across plugins; fix only the os-me internal duplicates. Proposed tickets, in order: 1. Fold `eval/reply_shape/run.rb` onto the os-me harness classes (LeverRun, AgreementScore, one flag-parsing owner); summary lines and TSV columns unchanged. No ruling needed. 2. One owner for the headless claude call (after the A/B/C ruling): one class builds argv, runs with stdin and timeout, parses usage/cost/structured output; HeadlessRunner, ModelRequest/ClaudeReply, and Adr::Judge::ClaudeCommand call it; an ADR records the dependency.

jared referenced this issue

2026-10-27 16:37:33 +00:00

Fold os-me eval/reply_shape/run.rb onto the os-me harness classes #1422

jared commented

2026-10-27 16:37:51 +00:00

Author

Owner

Filed from proposed ticket 1: #1422 (fold os-me eval/reply_shape/run.rb onto the os-me harness classes).

jared commented

2026-10-27 17:22:04 +00:00

Author

Owner

Ruling and design state for ticket 2 (one owner for the headless claude call), 2026-10-27, session c843bd27.

Jared ruled: option A, widened. Not a narrow claude-headless gem. A provider-neutral LLM gem on top of RubyLLM, so that adding a provider means adding one adapter.

Proposed layering (not yet an ADR):

ruby_llm 2.1.x (external): Anthropic API, OpenAI, Ollama, OpenAI-compatible endpoints such as Cloudflare Workers AI /ai/v1.
llm (new, ours): provider profiles, credentials, a claude_code provider (the claude -p CLI), a price table for model IDs that RubyLLM does not know, and the decision-endpoint wire format. Entry points: LLM.chat, LLM.embed, LLM.decide.
decision-model: keeps the meaning (Question, Answer, Reply, Confidence, CriteriaRules, Evaluation). It moves its transport (Http::Connection, CloudflareAccount, Wire::*, the SystemOne and CloudflareJev adapters, Price.jev) onto llm. Its public Client interface stays the same, because email-triage also calls it.
evals: NOT created yet. decision-model ADR 0001 makes the eval CLI, config format and report.json public contracts, and Evaluation already covers runner, splits, frozen log, kappa and reports. Add reps x jobs, the keep/discard loop and a run ledger there. Extract evals only when the first eval that is not a decision eval needs it (likely ticket-takedown), with its own superseding ADR.

Spike (scratchpad, throwaway), RubyLLM 2.1.1 on Ruby 4.0.2:

Custom claude_code provider: verified. About 44 lines; it overrides Provider#complete, the same way ruby_llm-codex does. It made two real calls (plain, and with_schema through --json-schema). Content must be a String, so structured output is JSON-encoded and read with .parsed.
New model IDs: claude-haiku-5-5 raises ModelNotFoundError unless assume_model_exists: true is set, and then it has no price. So llm needs its own price table.
Two OpenAI-compatible endpoints in one process: verified through RubyLLM.context.
The Anthropic API path is untested: no ANTHROPIC_API_KEY is set.

Cost note: claude -p with a Max login bills against the subscription. The API bills per token, and the account also has a Max-plan API credit balance. Callers choose the provider per call. Do not assume the API is cheaper.

Next: ADR through /os-adr:create (layering, ruby_llm dependency, gem names; Jared has not yet confirmed the names llm and evals). Then the llm tracer bullet: Anthropic profile plus claude_code provider; move os-adr Adr::Judge::ClaudeCommand onto it. Then decision-model on llm.

Ruling and design state for ticket 2 (one owner for the headless claude call), 2026-10-27, session c843bd27. Jared ruled: option A, widened. Not a narrow `claude-headless` gem. A provider-neutral LLM gem on top of RubyLLM, so that adding a provider means adding one adapter. Proposed layering (not yet an ADR): - `ruby_llm` 2.1.x (external): Anthropic API, OpenAI, Ollama, OpenAI-compatible endpoints such as Cloudflare Workers AI `/ai/v1`. - `llm` (new, ours): provider profiles, credentials, a `claude_code` provider (the `claude -p` CLI), a price table for model IDs that RubyLLM does not know, and the decision-endpoint wire format. Entry points: `LLM.chat`, `LLM.embed`, `LLM.decide`. - `decision-model`: keeps the meaning (Question, Answer, Reply, Confidence, CriteriaRules, Evaluation). It moves its transport (Http::Connection, CloudflareAccount, Wire::*, the SystemOne and CloudflareJev adapters, Price.jev) onto `llm`. Its public `Client` interface stays the same, because email-triage also calls it. - `evals`: NOT created yet. decision-model ADR 0001 makes the eval CLI, config format and report.json public contracts, and Evaluation already covers runner, splits, frozen log, kappa and reports. Add reps x jobs, the keep/discard loop and a run ledger there. Extract `evals` only when the first eval that is not a decision eval needs it (likely ticket-takedown), with its own superseding ADR. Spike (scratchpad, throwaway), RubyLLM 2.1.1 on Ruby 4.0.2: - Custom `claude_code` provider: verified. About 44 lines; it overrides `Provider#complete`, the same way ruby_llm-codex does. It made two real calls (plain, and `with_schema` through `--json-schema`). Content must be a String, so structured output is JSON-encoded and read with `.parsed`. - New model IDs: `claude-haiku-5-5` raises ModelNotFoundError unless `assume_model_exists: true` is set, and then it has no price. So `llm` needs its own price table. - Two OpenAI-compatible endpoints in one process: verified through `RubyLLM.context`. - The Anthropic API path is untested: no ANTHROPIC_API_KEY is set. Cost note: `claude -p` with a Max login bills against the subscription. The API bills per token, and the account also has a Max-plan API credit balance. Callers choose the provider per call. Do not assume the API is cheaper. Next: ADR through /os-adr:create (layering, ruby_llm dependency, gem names; Jared has not yet confirmed the names `llm` and `evals`). Then the `llm` tracer bullet: Anthropic profile plus claude_code provider; move os-adr `Adr::Judge::ClaudeCommand` onto it. Then decision-model on `llm`.

jared referenced this issue from a commit

2026-10-27 17:39:43 +00:00

docs: draft spec for the LLM stack gem (services, model types, harnesses)