jared/cc-os

Watch

Fork 0

decision-system S5: route judgment rows in os-sdlc flows #1247

New issue

Closed

opened 2026-10-23 15:17:26 +00:00 by jared · 2 comments

jared commented

2026-10-23 15:17:26 +00:00

Owner

Problem

os-sdlc flows (fast-batch, crc-refactor) put each row in a tier by the table in plugins/os-sdlc/references/decision-tiers.md. No object computes the cheapest route of a row from the levers, and no record shows where that route and the tier disagree. Stage S11 must compare the two on outcomes, so it needs one log line for each judgment row. The design is in docs/decision-system/levers.md (CostModel and RouteSelector) and docs/decision-system/operations.md (RowRouter, "What operations writes"). This ticket is the first of two S5 tickets. S5b adds ReviewOrder and RandomSample.

Approach

All new classes live in os-me (plugins/os-me/lib/os_me/). O-9 is ruled: the levers live in os-me, and os-sdlc calls the os-me CLIs as a subprocess. os-sdlc gets no new code. The flows are LLM orchestrators that run the CLI from their skill text.

CalibrationPrior gives p_wrong(route, source) from a constant table. It has the interface that RouteSelector will later get from calibration data (S10, S11). Values:

route	rule	precedent	facts	experiment	taste
`:alone`	0.02	0.10	0.10	0.30	0.50
`:predict`	0.02	0.05	0.05	0.20	0.30

RouteSelector#choose(wrong_cost, source) returns a RouteSelector::Route (name, expected_minutes). It compares three routes in this order and returns the first with the lowest expected_minutes:

decide_alone: p_wrong(:alone, source) × cost.wrong_minutes(wrong_cost)
predict: cost.predict_minutes + p_wrong(:predict, source) × cost.wrong_minutes(wrong_cost)
defer (or ask when ask_allowed: true): cost.ping_minutes

source is a Symbol (AnswerSource::Result#source). The design sketch takes a row argument too; choose does not, because no route uses it. RouteSelector runs in shadow mode only (O-4 AFK default). It never changes the tier.

RowRouter.build(store_path) uses CostModel.new(minutes_per_dollar: 1.0, predict_dollars: 0.5) (so predict_minutes is 0.5, and ping_minutes is 10 by O-6), CalibrationPrior.new, ask_allowed: false, and RouteLog.new(File.join(store_path, 'moments', 'route-log.jsonl')). RowRouter#route(labels, row_id: nil, tier: nil, flow: nil) returns labels.to_h plus the key shadow_route (a String) at the end. When row_id is given, it also appends one RouteLog line. RowLabels gets readers source and wrong_cost, so that RowRouter can give them to RouteSelector.

RouteLog#append(row_id:, flow:, tier:, shadow_route:, source:, level:) creates the parent directory if it is absent and appends one JSON line with the keys in this order: row_id, flow, tier, shadow_route, source, level, recorded_at (UTC ISO 8601, for example 2026-10-23T14:05:09Z). It appends each call; it does not remove duplicates.

os-me-route ROW_FILE gets three options: --row-id ID, --tier TIER (one of mechanical, low-stakes, judgment), and --flow NAME. With no options, it prints the S4 JSON plus shadow_route and writes no log line. The os-me predict skill calls it this way today, so this form must stay. With --row-id, the command also needs --tier and --flow. It checks this before it asks the model. Update the expected lines in plugins/os-me/test/route_cli_test.rb for the new key.

Orchestrator work (after this ticket merges, fast path, not in the pipeline):

In decision-tiers.md "Final judgment round", step 1: for each open row, write the row text to a file and run ruby ~/.claude/plugins/os-me/bin/os-me-route FILE --row-id <id> --tier judgment --flow <flow>. Put the JSON in the memo of the row.
Point fast-batch and crc-refactor SKILL.md at that step, with the text os-me-route.
In the os-me predict skill, step 1: use the labels in the memo when they are there; else run os-me-route as today.
Run bin/refresh-plugins.

Required work:

Add CalibrationPrior with the p_wrong table — check: present class CalibrationPrior in plugins/os-me/lib/os_me/calibration_prior.rb
Add RouteSelector and RouteSelector::Route — check: present def choose(wrong_cost, source) in plugins/os-me/lib/os_me/route_selector.rb
Add RouteLog — check: present class RouteLog in plugins/os-me/lib/os_me/route_log.rb
Add RowRouter with build and route — check: present class RowRouter in plugins/os-me/lib/os_me/row_router.rb
Add readers source and wrong_cost to RowLabels — check: present attr_reader :source, :wrong_cost in plugins/os-me/lib/os_me/row_labels.rb
Add the options --row-id, --tier, and --flow to the route command — check: present :row_id in plugins/os-me/lib/os_me/cli.rb

Acceptance criteria

Given CostModel.new(minutes_per_dollar: 1.0, predict_dollars: 0.5), CalibrationPrior.new, ask_allowed: false, WrongCost.new(hard_to_undo: false, compounds: true, stays_hidden: false), and source :facts, when RouteSelector#choose runs, then the route name is :predict and expected_minutes is within 0.001 of 2.0.
Given the same cost model and prior, WrongCost.new(hard_to_undo: true, compounds: true, stays_hidden: true), and source :taste, when RouteSelector#choose runs with ask_allowed: false, then the route is :defer with expected_minutes 10; with ask_allowed: true, the route is :ask with expected_minutes 10.
Given the same cost model and prior, WrongCost.new(hard_to_undo: false, compounds: false, stays_hidden: false), and source :rule, when RouteSelector#choose runs, then the route is :decide_alone and expected_minutes is within 0.001 of 0.1.
Given CalibrationPrior.new, when p_wrong(:alone, :taste) and p_wrong(:predict, :facts) run, then they return 0.5 and 0.05; p_wrong(:ask, :facts) raises KeyError.
Given a path DIR/moments/route-log.jsonl whose moments directory does not exist, when RouteLog#append(row_id: "b1/F3", flow: "fast-batch", tier: "judgment", shadow_route: "predict", source: "facts", level: "medium") runs, then the file has one line; its keys are, in order, row_id, flow, tier, shadow_route, source, level, recorded_at; the first six values are the given ones; recorded_at matches /\A\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ\z/.
Given RowLabels.new(source: AnswerSource::Result.new(source: :facts), wrong_cost: WrongCost.new(hard_to_undo: false, compounds: true, stays_hidden: false)) and RowRouter.build(DIR), when route(labels, row_id: "b1/F3", tier: "judgment", flow: "fast-batch") runs, then it returns { source: :facts, max_rung: 2, novel: false, wrong_cost: { hard_to_undo: false, compounds: true, stays_hidden: false }, level: :medium, required_band: 70, shadow_route: "predict" }, and DIR/moments/route-log.jsonl has one line with "shadow_route":"predict".
Given a row file with the text Move Gate::StepOutcome to OsSdlc::StepOutcome., --store DIR, and a fake claude that answers facts, no, yes, no, when os-me-route ROW_FILE --store DIR runs with no other option, then stdout is exactly {"source":"facts","max_rung":2,"novel":false,"wrong_cost":{"hard_to_undo":false,"compounds":true,"stays_hidden":false},"level":"medium","required_band":70,"shadow_route":"predict"} plus a newline, the exit status is 0, and DIR/moments/route-log.jsonl does not exist.
Given the row file and fake claude of criterion 7, when os-me-route ROW_FILE --store DIR --row-id b1/F3 --tier judgment --flow fast-batch runs, then stdout is the line of criterion 7, and DIR/moments/route-log.jsonl has one line whose first six values are "b1/F3", "fast-batch", "judgment", "predict", "facts", "medium".
Given the row file of criterion 7, when os-me-route ROW_FILE --store DIR --row-id b1/F3 runs with no --tier, then the exit status is not 0, stdout is empty, and the fake claude gets no call.
Given a fake claude whose first answer is whim, when os-me-route ROW_FILE --store DIR --row-id b1/F3 --tier judgment --flow fast-batch runs, then the exit status is not 0, stdout is empty, and DIR/moments/route-log.jsonl does not exist.

Owner sketch

class OsMe::CalibrationPrior
  P_WRONG = {
    alone: { rule: 0.02, precedent: 0.10, facts: 0.10, experiment: 0.30, taste: 0.50 },
    predict: { rule: 0.02, precedent: 0.05, facts: 0.05, experiment: 0.20, taste: 0.30 }
  }.freeze
  def p_wrong(route, source) = P_WRONG.fetch(route).fetch(source)
end

class OsMe::RouteSelector
  Route = Data.define(:name, :expected_minutes)
  def initialize(cost_model, calibration, ask_allowed: false); end
  def choose(wrong_cost, source); end
end

class OsMe::RouteLog
  def initialize(path); end
  def append(row_id:, flow:, tier:, shadow_route:, source:, level:); end
end

class OsMe::RowRouter
  def self.build(store_path); end
  def initialize(selector:, route_log:); end
  def route(labels, row_id: nil, tier: nil, flow: nil); end
end

Non-goals

Replacing the tier table with RouteSelector (O-4). The shadow route only goes to the log and the JSON.
The route comparison report (S11) and p_wrong from calibration data (S10, S11).
An --ask-allowed option. AFK flows always use defer.
Removing duplicate log lines when a row is routed twice.
ReviewOrder and RandomSample (S5b).
os-sdlc code. The skill text and decision-tiers.md change as orchestrator work.

Origin

Trigger: User asked for the decision-system build tickets this session. Rewritten 2026-10-23 with exact criteria after the #1245 lesson.
Improvised this session: none
Chain: off-design S3 build ← acceptance criteria too loose ← first ticket draft. Stop at DESIGN (docs/decision-system/levers.md, docs/decision-system/operations.md).
Root candidate: none found; searched RouteSelector, os-me-route, fast-batch, judgment
Where: docs/decision-system/build-plan.md S5; docs/decision-system/levers.md; docs/decision-system/operations.md; plugins/os-me/lib/os_me/; plugins/os-me/test/route_cli_test.rb
Session: 4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5.jsonl

VERDICT: CREATE (user-approved; backlog search only)

## Problem os-sdlc flows (`fast-batch`, `crc-refactor`) put each row in a tier by the table in `plugins/os-sdlc/references/decision-tiers.md`. No object computes the cheapest route of a row from the levers, and no record shows where that route and the tier disagree. Stage S11 must compare the two on outcomes, so it needs one log line for each judgment row. The design is in `docs/decision-system/levers.md` (`CostModel` and `RouteSelector`) and `docs/decision-system/operations.md` (`RowRouter`, "What operations writes"). This ticket is the first of two S5 tickets. S5b adds `ReviewOrder` and `RandomSample`. ## Approach All new classes live in os-me (`plugins/os-me/lib/os_me/`). O-9 is ruled: the levers live in os-me, and os-sdlc calls the os-me CLIs as a subprocess. os-sdlc gets no new code. The flows are LLM orchestrators that run the CLI from their skill text. `CalibrationPrior` gives `p_wrong(route, source)` from a constant table. It has the interface that `RouteSelector` will later get from calibration data (S10, S11). Values: | route | rule | precedent | facts | experiment | taste | |---|---|---|---|---|---| | `:alone` | 0.02 | 0.10 | 0.10 | 0.30 | 0.50 | | `:predict` | 0.02 | 0.05 | 0.05 | 0.20 | 0.30 | `RouteSelector#choose(wrong_cost, source)` returns a `RouteSelector::Route` (`name`, `expected_minutes`). It compares three routes in this order and returns the first with the lowest `expected_minutes`: - `decide_alone`: `p_wrong(:alone, source) × cost.wrong_minutes(wrong_cost)` - `predict`: `cost.predict_minutes + p_wrong(:predict, source) × cost.wrong_minutes(wrong_cost)` - `defer` (or `ask` when `ask_allowed: true`): `cost.ping_minutes` `source` is a `Symbol` (`AnswerSource::Result#source`). The design sketch takes a `row` argument too; `choose` does not, because no route uses it. `RouteSelector` runs in shadow mode only (O-4 AFK default). It never changes the tier. `RowRouter.build(store_path)` uses `CostModel.new(minutes_per_dollar: 1.0, predict_dollars: 0.5)` (so `predict_minutes` is 0.5, and `ping_minutes` is 10 by O-6), `CalibrationPrior.new`, `ask_allowed: false`, and `RouteLog.new(File.join(store_path, 'moments', 'route-log.jsonl'))`. `RowRouter#route(labels, row_id: nil, tier: nil, flow: nil)` returns `labels.to_h` plus the key `shadow_route` (a String) at the end. When `row_id` is given, it also appends one `RouteLog` line. `RowLabels` gets readers `source` and `wrong_cost`, so that `RowRouter` can give them to `RouteSelector`. `RouteLog#append(row_id:, flow:, tier:, shadow_route:, source:, level:)` creates the parent directory if it is absent and appends one JSON line with the keys in this order: `row_id`, `flow`, `tier`, `shadow_route`, `source`, `level`, `recorded_at` (UTC ISO 8601, for example `2026-10-23T14:05:09Z`). It appends each call; it does not remove duplicates. `os-me-route ROW_FILE` gets three options: `--row-id ID`, `--tier TIER` (one of `mechanical`, `low-stakes`, `judgment`), and `--flow NAME`. With no options, it prints the S4 JSON plus `shadow_route` and writes no log line. The `os-me` predict skill calls it this way today, so this form must stay. With `--row-id`, the command also needs `--tier` and `--flow`. It checks this before it asks the model. Update the expected lines in `plugins/os-me/test/route_cli_test.rb` for the new key. Orchestrator work (after this ticket merges, fast path, not in the pipeline): - In `decision-tiers.md` "Final judgment round", step 1: for each open row, write the row text to a file and run `ruby ~/.claude/plugins/os-me/bin/os-me-route FILE --row-id <id> --tier judgment --flow <flow>`. Put the JSON in the memo of the row. - Point `fast-batch` and `crc-refactor` `SKILL.md` at that step, with the text `os-me-route`. - In the os-me predict skill, step 1: use the labels in the memo when they are there; else run `os-me-route` as today. - Run `bin/refresh-plugins`. Required work: 1. Add `CalibrationPrior` with the `p_wrong` table — check: present `class CalibrationPrior` in `plugins/os-me/lib/os_me/calibration_prior.rb` 2. Add `RouteSelector` and `RouteSelector::Route` — check: present `def choose(wrong_cost, source)` in `plugins/os-me/lib/os_me/route_selector.rb` 3. Add `RouteLog` — check: present `class RouteLog` in `plugins/os-me/lib/os_me/route_log.rb` 4. Add `RowRouter` with `build` and `route` — check: present `class RowRouter` in `plugins/os-me/lib/os_me/row_router.rb` 5. Add readers `source` and `wrong_cost` to `RowLabels` — check: present `attr_reader :source, :wrong_cost` in `plugins/os-me/lib/os_me/row_labels.rb` 6. Add the options `--row-id`, `--tier`, and `--flow` to the `route` command — check: present `:row_id` in `plugins/os-me/lib/os_me/cli.rb` ## Acceptance criteria 1. Given `CostModel.new(minutes_per_dollar: 1.0, predict_dollars: 0.5)`, `CalibrationPrior.new`, `ask_allowed: false`, `WrongCost.new(hard_to_undo: false, compounds: true, stays_hidden: false)`, and source `:facts`, when `RouteSelector#choose` runs, then the route name is `:predict` and `expected_minutes` is within 0.001 of `2.0`. 2. Given the same cost model and prior, `WrongCost.new(hard_to_undo: true, compounds: true, stays_hidden: true)`, and source `:taste`, when `RouteSelector#choose` runs with `ask_allowed: false`, then the route is `:defer` with `expected_minutes` `10`; with `ask_allowed: true`, the route is `:ask` with `expected_minutes` `10`. 3. Given the same cost model and prior, `WrongCost.new(hard_to_undo: false, compounds: false, stays_hidden: false)`, and source `:rule`, when `RouteSelector#choose` runs, then the route is `:decide_alone` and `expected_minutes` is within 0.001 of `0.1`. 4. Given `CalibrationPrior.new`, when `p_wrong(:alone, :taste)` and `p_wrong(:predict, :facts)` run, then they return `0.5` and `0.05`; `p_wrong(:ask, :facts)` raises `KeyError`. 5. Given a path `DIR/moments/route-log.jsonl` whose `moments` directory does not exist, when `RouteLog#append(row_id: "b1/F3", flow: "fast-batch", tier: "judgment", shadow_route: "predict", source: "facts", level: "medium")` runs, then the file has one line; its keys are, in order, `row_id`, `flow`, `tier`, `shadow_route`, `source`, `level`, `recorded_at`; the first six values are the given ones; `recorded_at` matches `/\A\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ\z/`. 6. Given `RowLabels.new(source: AnswerSource::Result.new(source: :facts), wrong_cost: WrongCost.new(hard_to_undo: false, compounds: true, stays_hidden: false))` and `RowRouter.build(DIR)`, when `route(labels, row_id: "b1/F3", tier: "judgment", flow: "fast-batch")` runs, then it returns `{ source: :facts, max_rung: 2, novel: false, wrong_cost: { hard_to_undo: false, compounds: true, stays_hidden: false }, level: :medium, required_band: 70, shadow_route: "predict" }`, and `DIR/moments/route-log.jsonl` has one line with `"shadow_route":"predict"`. 7. Given a row file with the text `Move Gate::StepOutcome to OsSdlc::StepOutcome.`, `--store DIR`, and a fake `claude` that answers `facts`, `no`, `yes`, `no`, when `os-me-route ROW_FILE --store DIR` runs with no other option, then stdout is exactly `{"source":"facts","max_rung":2,"novel":false,"wrong_cost":{"hard_to_undo":false,"compounds":true,"stays_hidden":false},"level":"medium","required_band":70,"shadow_route":"predict"}` plus a newline, the exit status is 0, and `DIR/moments/route-log.jsonl` does not exist. 8. Given the row file and fake `claude` of criterion 7, when `os-me-route ROW_FILE --store DIR --row-id b1/F3 --tier judgment --flow fast-batch` runs, then stdout is the line of criterion 7, and `DIR/moments/route-log.jsonl` has one line whose first six values are `"b1/F3"`, `"fast-batch"`, `"judgment"`, `"predict"`, `"facts"`, `"medium"`. 9. Given the row file of criterion 7, when `os-me-route ROW_FILE --store DIR --row-id b1/F3` runs with no `--tier`, then the exit status is not 0, stdout is empty, and the fake `claude` gets no call. 10. Given a fake `claude` whose first answer is `whim`, when `os-me-route ROW_FILE --store DIR --row-id b1/F3 --tier judgment --flow fast-batch` runs, then the exit status is not 0, stdout is empty, and `DIR/moments/route-log.jsonl` does not exist. ## Owner sketch ```ruby class OsMe::CalibrationPrior P_WRONG = { alone: { rule: 0.02, precedent: 0.10, facts: 0.10, experiment: 0.30, taste: 0.50 }, predict: { rule: 0.02, precedent: 0.05, facts: 0.05, experiment: 0.20, taste: 0.30 } }.freeze def p_wrong(route, source) = P_WRONG.fetch(route).fetch(source) end class OsMe::RouteSelector Route = Data.define(:name, :expected_minutes) def initialize(cost_model, calibration, ask_allowed: false); end def choose(wrong_cost, source); end end class OsMe::RouteLog def initialize(path); end def append(row_id:, flow:, tier:, shadow_route:, source:, level:); end end class OsMe::RowRouter def self.build(store_path); end def initialize(selector:, route_log:); end def route(labels, row_id: nil, tier: nil, flow: nil); end end ``` ## Non-goals - Replacing the tier table with `RouteSelector` (O-4). The shadow route only goes to the log and the JSON. - The route comparison report (S11) and `p_wrong` from calibration data (S10, S11). - An `--ask-allowed` option. AFK flows always use `defer`. - Removing duplicate log lines when a row is routed twice. - `ReviewOrder` and `RandomSample` (S5b). - os-sdlc code. The skill text and `decision-tiers.md` change as orchestrator work. ## Origin - Trigger: User asked for the decision-system build tickets this session. Rewritten 2026-10-23 with exact criteria after the #1245 lesson. - Improvised this session: none - Chain: off-design S3 build ← acceptance criteria too loose ← first ticket draft. Stop at `DESIGN` (`docs/decision-system/levers.md`, `docs/decision-system/operations.md`). - Root candidate: none found; searched RouteSelector, os-me-route, fast-batch, judgment - Where: docs/decision-system/build-plan.md S5; docs/decision-system/levers.md; docs/decision-system/operations.md; plugins/os-me/lib/os_me/; plugins/os-me/test/route_cli_test.rb - Session: 4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5 - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5.jsonl VERDICT: CREATE (user-approved; backlog search only)

jared added the

labels

2026-10-23 15:17:26 +00:00

jared referenced this issue

2026-10-23 15:17:30 +00:00

decision-system S11: route comparison report #1253

jared referenced this issue

2026-10-23 17:39:45 +00:00

decision-system S4c: lever fields in PredictionParser and PredictionLog, and the os-me-route CLI #1263

jared referenced this issue

2026-10-23 17:46:12 +00:00

decision-system S5b: review order and 5% sample CLIs #1265

jared commented

2026-10-23 17:47:28 +00:00

Author

Owner

Work started on branch ticket-1247 (/os-sdlc:implement, S5).

jared commented

2026-10-23 18:01:00 +00:00

Author

Owner

Resolution

Done: S5 shadow route built: CalibrationPrior#p_wrong, RouteSelector (+Route), RouteLog, RowRouter, RowLabels readers, os-me-route --row-id/--tier/--flow writing route-log.jsonl; implementation 131 complete.

Evidence: Branch ticket-1247, Runner commit; one test-review round, one required-work repair, one refactor round (RowRouter takes labels, CLI owns options and file read); suite and lint green; 16 dispatches, 13 min.

Follow-ups: S5b #1265 (review order, 5% sample). Skill text for decision-tiers.md, fast-batch and crc-refactor to call os-me-route is orchestrator work after #1265.

## Resolution **Done:** S5 shadow route built: CalibrationPrior#p_wrong, RouteSelector (+Route), RouteLog, RowRouter, RowLabels readers, os-me-route --row-id/--tier/--flow writing route-log.jsonl; implementation 131 complete. **Evidence:** Branch ticket-1247, Runner commit; one test-review round, one required-work repair, one refactor round (RowRouter takes labels, CLI owns options and file read); suite and lint green; 16 dispatches, 13 min. **Follow-ups:** S5b #1265 (review order, 5% sample). Skill text for decision-tiers.md, fast-batch and crc-refactor to call os-me-route is orchestrator work after #1265.

jared closed this issue

2026-10-23 18:01:00 +00:00

jared referenced this issue from a commit

2026-10-23 18:01:56 +00:00

feat(os-me): Audited POODR boundaries for the S5 shadow route; design and contracts hold, all six WORK items present. (#1247)

jared referenced this issue from a commit

2026-10-23 18:14:13 +00:00

docs(os-sdlc): final judgment round routes rows through os-me-route (S5, #1247, #1265)