jared/cc-os

Watch

Fork 0

decision-system S11: route comparison report #1253

New issue

Closed

opened 2026-10-23 15:17:30 +00:00 by jared · 2 comments

jared commented

2026-10-23 15:17:30 +00:00

Owner

Problem

RouteSelector runs in shadow mode (S5). Each judgment row of a fast-batch or crc-refactor run writes one line to STORE/moments/route-log.jsonl (row_id flow tier shadow_route source level recorded_at). No command reads that file. Nobody can see where the decided tier rule (plugins/os-sdlc/references/decision-tiers.md) and RouteSelector disagree, what happened to those rows, or what each rule would cost in Jared-minutes. Item O-4 (docs/decision-system/decisions-register.md) needs this report before anyone proposes a change to the tier rule.

Approach

Add OsMe::RouteComparison and the command os-me-eval route-comparison [--outcomes FILE] (Thor, in OsMe::CLI, with the --store class option).

Tier route of a row. The tier rule maps mechanical and low-stakes to decide_alone and judgment to predict (decision-tiers.md: judgment rows are predicted at the final round; flows never ask). An unknown tier raises OsMe::Error.
Disagreement. The tier route differs from the logged shadow_route.
Jared-minutes. Expected minutes per route from the S5 CostModel and CalibrationPrior, the same formula RouteSelector uses (levers.md; learning-loops.md metric "Total expected Jared-minutes on replayed rows"). The tier rule total sums the tier route of every row; the RouteSelector total sums the logged shadow route of every row. Ping cost stays 10 minutes, marked as a guess (O-6 AFK default).
Outcomes. The route log has no link to the prediction log (prediction ids are session:a, route row ids are b1/F3, and the route log has no session). So outcomes come from an optional JSONL file, one {"row_id": ..., "outcome": ...} per line, with the glossary vocabulary accept, modify, override, none. A row with no line shows none. An unknown outcome raises OsMe::Error.
Report only (O-4 AFK default). The command reads and prints. It writes no file.

Output shape (exact):

Route comparison: 3 rows, disagreements: 1
| Row | Flow | Tier rule | RouteSelector | Outcome |
|---|---|---|---|---|
| b1/F7 | fast-batch | predict | defer | none |

Jared-minutes (expected): tier rule 74.6, RouteSelector 12.1
Ping cost of 10 Jared-minutes is a guess (O-6).
Report only. A change to the tier rule goes to Jared (O-4).

Minutes print with one decimal.

Required work:

Add OsMe::RouteComparison with render, built from the route log entries, the outcomes, and a RouteSelector — check: present class RouteComparison in plugins/os-me/lib/os_me/route_comparison.rb
Make RouteSelector#routes(wrong_cost, source) public, returning all three Route values; choose returns the cheapest of them — check: present def routes( in plugins/os-me/lib/os_me/route_selector.rb
Move the S5 selector construction into RouteSelector.standard; RowRouter.build uses it — check: absent CostModel.new in plugins/os-me/lib/os_me/row_router.rb
Give RouteLog the Store path (RouteLog.in_store(store_path)) and a reader entries that raises OsMe::Error naming the path when the file is absent; RowRouter.build uses in_store — check: absent route-log.jsonl in plugins/os-me/lib/os_me/row_router.rb
Add the route-comparison command to OsMe::CLI with an --outcomes option — check: present def route_comparison in plugins/os-me/lib/os_me/cli.rb

Acceptance criteria

Fixture for criteria 1 to 4 and 7: a Store whose moments/route-log.jsonl holds three rows:
b1/F3 (fast-batch, judgment, shadow predict, facts, medium), b1/F7 (fast-batch, judgment, shadow defer, taste, high), b1/F9 (fast-batch, low-stakes, shadow decide_alone, rule, low).

Given the three-row Store, when os-me-eval route-comparison --store DIR runs, then stdout starts with Route comparison: 3 rows, disagreements: 1, holds the line | b1/F7 | fast-batch | predict | defer | none |, holds no table line for b1/F3 or b1/F9, and the exit status is 0.
Given the three-row Store and an outcomes file with the line {"row_id":"b1/F7","outcome":"override"}, when the command runs with --outcomes FILE, then the b1/F7 line ends with | override |.
Given the three-row Store, when the command runs, then stdout holds Jared-minutes (expected): tier rule 74.6, RouteSelector 12.1.
Given the three-row Store, when the command runs, then the last two lines of stdout are Ping cost of 10 Jared-minutes is a guess (O-6). and Report only. A change to the tier rule goes to Jared (O-4).
Given a route log with one row whose tier is urgent, when the command runs, then the exit status is not 0, stdout is empty, and stderr holds urgent.
Given a Store with no moments/route-log.jsonl, when the command runs, then the exit status is not 0 and stderr holds the route log path.
Given the three-row Store and an outcomes file with the line {"row_id":"b1/F7","outcome":"maybe"}, when the command runs, then the exit status is not 0, stdout is empty, and stderr holds maybe.
Given the three-row Store, when the command runs, then every file under the Store has the same bytes and the same file list as before the run.
Given RouteSelector.standard and a medium wrong cost with source facts, when routes is called, then it returns decide_alone 3.0, predict 2.0, and defer 10 expected minutes, and choose returns the predict route.

All tests go under plugins/os-me/test (for example route_comparison_cli_test.rb and route_selector_test.rb). They call OsMe::CLI.start or RouteSelector directly. No model call, no network.

Owner sketch

module OsMe
  class RouteComparison
    TIER_ROUTES = { 'mechanical' => :decide_alone, 'low-stakes' => :decide_alone, 'judgment' => :predict }.freeze
    OUTCOMES = %w[accept modify override none].freeze
    Level = Data.define(:level)

    def self.from_store(store_path, outcomes_path: nil)
      new(entries: RouteLog.in_store(store_path).entries,
          outcomes: outcomes_path ? read_outcomes(outcomes_path) : {},
          selector: RouteSelector.standard)
    end

    def initialize(entries:, outcomes:, selector:)
      @entries = entries
      @outcomes = outcomes
      @selector = selector
    end

    def render; end

    private

    def tier_route(entry)
      TIER_ROUTES.fetch(entry.fetch('tier')) { |tier| raise Error, "unknown tier: #{tier}" }
    end

    def minutes(entry, name)
      @selector.routes(Level.new(entry.fetch('level').to_sym), entry.fetch('source').to_sym)
               .find { |route| route.name == name }.expected_minutes
    end
  end

  class RouteSelector
    def self.standard
      new(CostModel.new(minutes_per_dollar: 1.0, predict_dollars: 0.5), CalibrationPrior.new)
    end

    def choose(wrong_cost, source) = routes(wrong_cost, source).min_by(&:expected_minutes)

    def routes(wrong_cost, source); end
  end

  class RouteLog
    def self.in_store(store_path) = new(File.join(store_path, 'moments', 'route-log.jsonl'))

    def entries; end
  end
end

Level is a duck for WrongCost: CostModel#wrong_minutes reads only #level, and the route log keeps only the level.

Scope and blocking

Blocked by: #1247 (S5 shadow route, built 2026-10-23).
Non-goals: any change to the tier rule or decision-tiers.md (O-4); a writer for the outcomes file; a link from route log rows to prediction log records; realized (outcome-based) minutes; changes to CostModel values or CalibrationPrior priors.

AFK defaults used: O-4 (report only), O-6 (ping cost 10, a guess).

Origin

Trigger: User asked for the decision-system build tickets this session
Improvised this session: none
Chain: no ticket for this stage ← the build plan names the stage and the user asked for the tickets. Stop at DESIGN (docs/decision-system/build-plan.md).
Root candidate: none found; searched RouteComparison, route-log, RouteSelector
Where: docs/decision-system/build-plan.md S11; plugins/os-me/lib/os_me/
Session: 4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5.jsonl

VERDICT: CREATE (user-approved; backlog search only)

## Problem `RouteSelector` runs in shadow mode (S5). Each judgment row of a fast-batch or crc-refactor run writes one line to `STORE/moments/route-log.jsonl` (`row_id flow tier shadow_route source level recorded_at`). No command reads that file. Nobody can see where the decided tier rule (`plugins/os-sdlc/references/decision-tiers.md`) and `RouteSelector` disagree, what happened to those rows, or what each rule would cost in Jared-minutes. Item O-4 (`docs/decision-system/decisions-register.md`) needs this report before anyone proposes a change to the tier rule. ## Approach Add `OsMe::RouteComparison` and the command `os-me-eval route-comparison [--outcomes FILE]` (Thor, in `OsMe::CLI`, with the `--store` class option). - **Tier route of a row.** The tier rule maps `mechanical` and `low-stakes` to `decide_alone` and `judgment` to `predict` (decision-tiers.md: judgment rows are predicted at the final round; flows never ask). An unknown tier raises `OsMe::Error`. - **Disagreement.** The tier route differs from the logged `shadow_route`. - **Jared-minutes.** Expected minutes per route from the S5 `CostModel` and `CalibrationPrior`, the same formula `RouteSelector` uses (levers.md; learning-loops.md metric "Total expected Jared-minutes on replayed rows"). The tier rule total sums the tier route of every row; the `RouteSelector` total sums the logged shadow route of every row. Ping cost stays 10 minutes, marked as a guess (O-6 AFK default). - **Outcomes.** The route log has no link to the prediction log (prediction ids are `session:a`, route row ids are `b1/F3`, and the route log has no session). So outcomes come from an optional JSONL file, one `{"row_id": ..., "outcome": ...}` per line, with the glossary vocabulary `accept`, `modify`, `override`, `none`. A row with no line shows `none`. An unknown outcome raises `OsMe::Error`. - **Report only (O-4 AFK default).** The command reads and prints. It writes no file. Output shape (exact): ``` Route comparison: 3 rows, disagreements: 1 | Row | Flow | Tier rule | RouteSelector | Outcome | |---|---|---|---|---| | b1/F7 | fast-batch | predict | defer | none | Jared-minutes (expected): tier rule 74.6, RouteSelector 12.1 Ping cost of 10 Jared-minutes is a guess (O-6). Report only. A change to the tier rule goes to Jared (O-4). ``` Minutes print with one decimal. Required work: 1. Add `OsMe::RouteComparison` with `render`, built from the route log entries, the outcomes, and a `RouteSelector` — check: present `class RouteComparison` in `plugins/os-me/lib/os_me/route_comparison.rb` 2. Make `RouteSelector#routes(wrong_cost, source)` public, returning all three `Route` values; `choose` returns the cheapest of them — check: present `def routes(` in `plugins/os-me/lib/os_me/route_selector.rb` 3. Move the S5 selector construction into `RouteSelector.standard`; `RowRouter.build` uses it — check: absent `CostModel.new` in `plugins/os-me/lib/os_me/row_router.rb` 4. Give `RouteLog` the Store path (`RouteLog.in_store(store_path)`) and a reader `entries` that raises `OsMe::Error` naming the path when the file is absent; `RowRouter.build` uses `in_store` — check: absent `route-log.jsonl` in `plugins/os-me/lib/os_me/row_router.rb` 5. Add the `route-comparison` command to `OsMe::CLI` with an `--outcomes` option — check: present `def route_comparison` in `plugins/os-me/lib/os_me/cli.rb` ## Acceptance criteria Fixture for criteria 1 to 4 and 7: a Store whose `moments/route-log.jsonl` holds three rows: `b1/F3` (fast-batch, judgment, shadow `predict`, facts, medium), `b1/F7` (fast-batch, judgment, shadow `defer`, taste, high), `b1/F9` (fast-batch, low-stakes, shadow `decide_alone`, rule, low). 1. Given the three-row Store, when `os-me-eval route-comparison --store DIR` runs, then stdout starts with `Route comparison: 3 rows, disagreements: 1`, holds the line `| b1/F7 | fast-batch | predict | defer | none |`, holds no table line for `b1/F3` or `b1/F9`, and the exit status is 0. 2. Given the three-row Store and an outcomes file with the line `{"row_id":"b1/F7","outcome":"override"}`, when the command runs with `--outcomes FILE`, then the `b1/F7` line ends with `| override |`. 3. Given the three-row Store, when the command runs, then stdout holds `Jared-minutes (expected): tier rule 74.6, RouteSelector 12.1`. 4. Given the three-row Store, when the command runs, then the last two lines of stdout are `Ping cost of 10 Jared-minutes is a guess (O-6).` and `Report only. A change to the tier rule goes to Jared (O-4).` 5. Given a route log with one row whose tier is `urgent`, when the command runs, then the exit status is not 0, stdout is empty, and stderr holds `urgent`. 6. Given a Store with no `moments/route-log.jsonl`, when the command runs, then the exit status is not 0 and stderr holds the route log path. 7. Given the three-row Store and an outcomes file with the line `{"row_id":"b1/F7","outcome":"maybe"}`, when the command runs, then the exit status is not 0, stdout is empty, and stderr holds `maybe`. 8. Given the three-row Store, when the command runs, then every file under the Store has the same bytes and the same file list as before the run. 9. Given `RouteSelector.standard` and a medium wrong cost with source `facts`, when `routes` is called, then it returns `decide_alone` 3.0, `predict` 2.0, and `defer` 10 expected minutes, and `choose` returns the `predict` route. All tests go under `plugins/os-me/test` (for example `route_comparison_cli_test.rb` and `route_selector_test.rb`). They call `OsMe::CLI.start` or `RouteSelector` directly. No model call, no network. ## Owner sketch ```ruby module OsMe class RouteComparison TIER_ROUTES = { 'mechanical' => :decide_alone, 'low-stakes' => :decide_alone, 'judgment' => :predict }.freeze OUTCOMES = %w[accept modify override none].freeze Level = Data.define(:level) def self.from_store(store_path, outcomes_path: nil) new(entries: RouteLog.in_store(store_path).entries, outcomes: outcomes_path ? read_outcomes(outcomes_path) : {}, selector: RouteSelector.standard) end def initialize(entries:, outcomes:, selector:) @entries = entries @outcomes = outcomes @selector = selector end def render; end private def tier_route(entry) TIER_ROUTES.fetch(entry.fetch('tier')) { |tier| raise Error, "unknown tier: #{tier}" } end def minutes(entry, name) @selector.routes(Level.new(entry.fetch('level').to_sym), entry.fetch('source').to_sym) .find { |route| route.name == name }.expected_minutes end end class RouteSelector def self.standard new(CostModel.new(minutes_per_dollar: 1.0, predict_dollars: 0.5), CalibrationPrior.new) end def choose(wrong_cost, source) = routes(wrong_cost, source).min_by(&:expected_minutes) def routes(wrong_cost, source); end end class RouteLog def self.in_store(store_path) = new(File.join(store_path, 'moments', 'route-log.jsonl')) def entries; end end end ``` `Level` is a duck for `WrongCost`: `CostModel#wrong_minutes` reads only `#level`, and the route log keeps only the level. ## Scope and blocking Blocked by: #1247 (S5 shadow route, built 2026-10-23). Non-goals: any change to the tier rule or `decision-tiers.md` (O-4); a writer for the outcomes file; a link from route log rows to prediction log records; realized (outcome-based) minutes; changes to `CostModel` values or `CalibrationPrior` priors. AFK defaults used: O-4 (report only), O-6 (ping cost 10, a guess). ## Origin - Trigger: User asked for the decision-system build tickets this session - Improvised this session: none - Chain: no ticket for this stage ← the build plan names the stage and the user asked for the tickets. Stop at `DESIGN` (`docs/decision-system/build-plan.md`). - Root candidate: none found; searched RouteComparison, route-log, RouteSelector - Where: docs/decision-system/build-plan.md S11; plugins/os-me/lib/os_me/ - Session: 4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5 - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5.jsonl VERDICT: CREATE (user-approved; backlog search only)

jared added the

project/cc-os

filed-by/user

category/code-change

labels

2026-10-23 15:17:30 +00:00

jared commented

2026-10-23 19:35:12 +00:00

Author

Owner

Work started on branch ticket-1253 (os-sdlc implement, AFK decision-system build).

jared commented

2026-10-23 19:42:22 +00:00

Author

Owner

Resolution

Done: os-me-eval route-comparison reports where the tier rule and the RouteSelector shadow route disagree, the outcome of each such row, and the expected Jared-minutes of each rule. Report only (O-4).

Evidence: Implementation 139: 10 dispatches, 0 retries, 6 min 41 s, implementation_complete. Commits bd59bc70 and 1f24f5f3 on ticket-1253. Suite gate green.

Follow-ups: Dropped for now: a writer for the outcomes file and a link from route-log rows to prediction records. No route-log rows exist yet (S5 gate open), so neither has data to act on; revisit when the S5 gate closes.

## Resolution **Done:** os-me-eval route-comparison reports where the tier rule and the RouteSelector shadow route disagree, the outcome of each such row, and the expected Jared-minutes of each rule. Report only (O-4). **Evidence:** Implementation 139: 10 dispatches, 0 retries, 6 min 41 s, implementation_complete. Commits bd59bc70 and 1f24f5f3 on ticket-1253. Suite gate green. **Follow-ups:** Dropped for now: a writer for the outcomes file and a link from route-log rows to prediction records. No route-log rows exist yet (S5 gate open), so neither has data to act on; revisit when the S5 gate closes.

jared closed this issue

2026-10-23 19:42:22 +00:00

jared referenced this issue from a commit

2026-10-23 19:43:25 +00:00

feat(os-me): Audited POODR boundaries for RouteComparison, RouteSelector.standard and routes, and RouteLog.in_store and entries. Design and contracts hold, and all five WORK items are present. (#1253)

jared referenced this issue from a commit

2026-10-23 19:43:25 +00:00

docs(decision-system): S11 built, report only (#1253)