jared/cc-os

Watch

Fork 0

decision-system S5b: review order and 5% sample CLIs #1265

New issue

Closed

opened 2026-10-23 17:46:12 +00:00 by jared · 2 comments

jared commented

2026-10-23 17:46:12 +00:00

Owner

Problem

The reports of os-sdlc flows list predicted rulings for optional review in the order the run made them. The row with the largest expected loss can be last. Rows that a run decides alone and does not list (mechanical rows) never get a check, so no one sees an error at high confidence. The design is in docs/decision-system/review.md (ReviewOrder, RandomSample). This ticket is the second of two S5 tickets. It uses CalibrationPrior from S5.

Approach

All new classes live in os-me (plugins/os-me/lib/os_me/). O-9 is ruled: os-sdlc calls the os-me CLIs as a subprocess and gets no new code.

CalibrationPrior#hit_rate(band:, source:) returns band / 100.0. It ignores source until calibration data replaces the prior (S10). This is the claimed band taken as the hit rate.

ReviewItem is a value with id (String), level (Symbol: :low, :medium, :high), source (Symbol), and band (Integer). novel? is true when source is :taste. CostModel#wrong_minutes takes the item, because it reads only level.

ReviewOrder#sort(items) sorts by three keys, in this order:

Expected loss, largest first: cost.wrong_minutes(item) × (1 − calibration.hit_rate(band: item.band, source: item.source)).
Novel items first.
Input position, first first. (Ruby sort_by is not stable, so this key is necessary.)

RandomSample#pick?(key) is true when Digest::SHA256.hexdigest("sample:#{key}").to_i(16) % 100 < 5. The string sample: is the fixed salt. The same key always gives the same answer, as in OsMe::Split. The caller makes the key unique for each row and each run, for example fast-batch/<batch>/<id>.

Two OsMe::CLI commands (bin os-me-eval):

sample KEY: exit 0 when RandomSample#pick?(KEY) is true, else exit 1. It prints nothing.
review-order ITEMS_FILE: each line of the file is JSON {"id":"A","level":"medium","source":"facts","confidence":84}. The command makes the band with ConfidenceBand.of_percent(confidence), sorts with ReviewOrder (CostModel.new(minutes_per_dollar: 1.0, predict_dollars: 0.5), CalibrationPrior.new), and prints the ids, one for each line.

Orchestrator work (after this ticket merges, fast path, not in the pipeline):

In decision-tiers.md "Final judgment round", step 5: write one JSON line for each predicted row (id, level from the os-me-route JSON, source, the predicted confidence) and list the rows under "Predicted rulings (optional review)" in the order that os-me-eval review-order prints.
In decision-tiers.md "What you do with each tier", mechanical: run ruby ~/.claude/plugins/os-me/bin/os-me-eval sample <flow>/<run>/<id>. On exit 0, list the row under "Decided by the run (optional review)" with the mark sampled.
Point the fast-batch report table and the crc-refactor phase 7 Review section at these steps, with the text ReviewOrder.
Run bin/refresh-plugins. Gate: one fast-batch run lists the predicted rows in the new order and writes route-log.jsonl.

Required work:

Add hit_rate(band:, source:) to CalibrationPrior — check: present def hit_rate(band:, source:) in plugins/os-me/lib/os_me/calibration_prior.rb
Add ReviewItem with novel? — check: present def novel? in plugins/os-me/lib/os_me/review_item.rb
Add ReviewOrder with sort — check: present def sort(items) in plugins/os-me/lib/os_me/review_order.rb
Add RandomSample with pick? — check: present "sample:#{key}" in plugins/os-me/lib/os_me/random_sample.rb
Add the commands sample KEY and review-order ITEMS_FILE to OsMe::CLI — check: present def review_order(items_file) in plugins/os-me/lib/os_me/cli.rb

Acceptance criteria

Given CalibrationPrior.new, when hit_rate(band: 80, source: :facts) runs, then it returns 0.8.
Given items A (:medium, :facts, band 80), B (:high, :precedent, band 90), and C (:low, :taste, band 50), in that order, when ReviewOrder#sort runs with CostModel.new(minutes_per_dollar: 1.0, predict_dollars: 0.5) and CalibrationPrior.new, then the ids are ["B", "A", "C"] (expected losses about 24, 6, and 2.5).
Given items D (:medium, :facts, band 80) and E (:medium, :taste, band 80), in that order, when ReviewOrder#sort runs, then the ids are ["E", "D"].
Given items F and G, both :medium, :facts, band 80, in that order, when ReviewOrder#sort runs, then the ids are ["F", "G"].
Given the keys F1 to F100, when RandomSample#pick? runs on each, then it returns true for F17 only.
Given the key fast-batch/b1/35, when RandomSample#pick? runs two times, then it returns true both times; for fast-batch/b1/34 it returns false both times.
Given the key F17, when os-me-eval sample F17 runs, then the exit status is 0 and stdout is empty; for F16, the exit status is 1 and stdout is empty.
Given an items file with the three lines {"id":"A","level":"medium","source":"facts","confidence":84}, {"id":"B","level":"high","source":"precedent","confidence":90}, {"id":"C","level":"low","source":"taste","confidence":50}, when os-me-eval review-order ITEMS_FILE runs, then stdout is exactly B\nA\nC\n and the exit status is 0.
Given an items file with the line {"id":"A","level":"huge","source":"facts","confidence":84}, when os-me-eval review-order ITEMS_FILE runs, then the exit status is not 0 and stdout is empty.

Owner sketch

class OsMe::CalibrationPrior
  def hit_rate(band:, source:) = band / 100.0
end

class OsMe::ReviewItem < Data.define(:id, :level, :source, :band)
  def novel? = source == :taste
end

class OsMe::ReviewOrder
  def initialize(cost_model, calibration); end
  def sort(items); end
end

class OsMe::RandomSample
  PERCENT = 5
  def pick?(key) = Digest::SHA256.hexdigest("sample:#{key}").to_i(16) % 100 < PERCENT
end

class OsMe::CLI < Thor
  desc 'sample KEY', 'exit 0 if the row joins the 5% review sample, exit 1 if not'
  def sample(key); end
  desc 'review-order ITEMS_FILE', 'print the item ids in review order'
  def review_order(items_file); end
end

Non-goals

Hit rates from calibration data (S10). The prior uses the band only.
OneThingSelector and SessionLedger (S9).
Sorting low-stakes rows. They have no lever labels; they keep their order.
A sample of judgment or low-stakes rows. Those rows are listed already.
os-sdlc code. The skill text and decision-tiers.md change as orchestrator work.

Blocked by: #1247 (CalibrationPrior).

Origin

Trigger: User asked for the decision-system build tickets this session. Rewritten 2026-10-23 with exact criteria after the #1245 lesson.
Improvised this session: none
Chain: off-design S3 build ← acceptance criteria too loose ← first ticket draft. Stop at DESIGN (docs/decision-system/review.md).
Root candidate: none found; searched ReviewOrder, RandomSample, optional review, fast-batch
Where: docs/decision-system/build-plan.md S5; docs/decision-system/review.md; plugins/os-me/lib/os_me/; plugins/os-me/lib/os_me/cli.rb
Session: 4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5.jsonl

VERDICT: CREATE (user-approved; backlog search only)

## Problem The reports of os-sdlc flows list predicted rulings for optional review in the order the run made them. The row with the largest expected loss can be last. Rows that a run decides alone and does not list (mechanical rows) never get a check, so no one sees an error at high confidence. The design is in `docs/decision-system/review.md` (`ReviewOrder`, `RandomSample`). This ticket is the second of two S5 tickets. It uses `CalibrationPrior` from S5. ## Approach All new classes live in os-me (`plugins/os-me/lib/os_me/`). O-9 is ruled: os-sdlc calls the os-me CLIs as a subprocess and gets no new code. `CalibrationPrior#hit_rate(band:, source:)` returns `band / 100.0`. It ignores `source` until calibration data replaces the prior (S10). This is the claimed band taken as the hit rate. `ReviewItem` is a value with `id` (String), `level` (Symbol: `:low`, `:medium`, `:high`), `source` (Symbol), and `band` (Integer). `novel?` is true when `source` is `:taste`. `CostModel#wrong_minutes` takes the item, because it reads only `level`. `ReviewOrder#sort(items)` sorts by three keys, in this order: 1. Expected loss, largest first: `cost.wrong_minutes(item) × (1 − calibration.hit_rate(band: item.band, source: item.source))`. 2. Novel items first. 3. Input position, first first. (Ruby `sort_by` is not stable, so this key is necessary.) `RandomSample#pick?(key)` is true when `Digest::SHA256.hexdigest("sample:#{key}").to_i(16) % 100 < 5`. The string `sample:` is the fixed salt. The same key always gives the same answer, as in `OsMe::Split`. The caller makes the key unique for each row and each run, for example `fast-batch/<batch>/<id>`. Two `OsMe::CLI` commands (bin `os-me-eval`): - `sample KEY`: exit 0 when `RandomSample#pick?(KEY)` is true, else exit 1. It prints nothing. - `review-order ITEMS_FILE`: each line of the file is JSON `{"id":"A","level":"medium","source":"facts","confidence":84}`. The command makes the band with `ConfidenceBand.of_percent(confidence)`, sorts with `ReviewOrder` (`CostModel.new(minutes_per_dollar: 1.0, predict_dollars: 0.5)`, `CalibrationPrior.new`), and prints the ids, one for each line. Orchestrator work (after this ticket merges, fast path, not in the pipeline): - In `decision-tiers.md` "Final judgment round", step 5: write one JSON line for each predicted row (id, `level` from the `os-me-route` JSON, `source`, the predicted confidence) and list the rows under "Predicted rulings (optional review)" in the order that `os-me-eval review-order` prints. - In `decision-tiers.md` "What you do with each tier", mechanical: run `ruby ~/.claude/plugins/os-me/bin/os-me-eval sample <flow>/<run>/<id>`. On exit 0, list the row under "Decided by the run (optional review)" with the mark `sampled`. - Point the `fast-batch` report table and the `crc-refactor` phase 7 Review section at these steps, with the text `ReviewOrder`. - Run `bin/refresh-plugins`. Gate: one fast-batch run lists the predicted rows in the new order and writes `route-log.jsonl`. Required work: 1. Add `hit_rate(band:, source:)` to `CalibrationPrior` — check: present `def hit_rate(band:, source:)` in `plugins/os-me/lib/os_me/calibration_prior.rb` 2. Add `ReviewItem` with `novel?` — check: present `def novel?` in `plugins/os-me/lib/os_me/review_item.rb` 3. Add `ReviewOrder` with `sort` — check: present `def sort(items)` in `plugins/os-me/lib/os_me/review_order.rb` 4. Add `RandomSample` with `pick?` — check: present `"sample:#{key}"` in `plugins/os-me/lib/os_me/random_sample.rb` 5. Add the commands `sample KEY` and `review-order ITEMS_FILE` to `OsMe::CLI` — check: present `def review_order(items_file)` in `plugins/os-me/lib/os_me/cli.rb` ## Acceptance criteria 1. Given `CalibrationPrior.new`, when `hit_rate(band: 80, source: :facts)` runs, then it returns `0.8`. 2. Given items `A` (`:medium`, `:facts`, band 80), `B` (`:high`, `:precedent`, band 90), and `C` (`:low`, `:taste`, band 50), in that order, when `ReviewOrder#sort` runs with `CostModel.new(minutes_per_dollar: 1.0, predict_dollars: 0.5)` and `CalibrationPrior.new`, then the ids are `["B", "A", "C"]` (expected losses about 24, 6, and 2.5). 3. Given items `D` (`:medium`, `:facts`, band 80) and `E` (`:medium`, `:taste`, band 80), in that order, when `ReviewOrder#sort` runs, then the ids are `["E", "D"]`. 4. Given items `F` and `G`, both `:medium`, `:facts`, band 80, in that order, when `ReviewOrder#sort` runs, then the ids are `["F", "G"]`. 5. Given the keys `F1` to `F100`, when `RandomSample#pick?` runs on each, then it returns true for `F17` only. 6. Given the key `fast-batch/b1/35`, when `RandomSample#pick?` runs two times, then it returns true both times; for `fast-batch/b1/34` it returns false both times. 7. Given the key `F17`, when `os-me-eval sample F17` runs, then the exit status is 0 and stdout is empty; for `F16`, the exit status is 1 and stdout is empty. 8. Given an items file with the three lines `{"id":"A","level":"medium","source":"facts","confidence":84}`, `{"id":"B","level":"high","source":"precedent","confidence":90}`, `{"id":"C","level":"low","source":"taste","confidence":50}`, when `os-me-eval review-order ITEMS_FILE` runs, then stdout is exactly `B\nA\nC\n` and the exit status is 0. 9. Given an items file with the line `{"id":"A","level":"huge","source":"facts","confidence":84}`, when `os-me-eval review-order ITEMS_FILE` runs, then the exit status is not 0 and stdout is empty. ## Owner sketch ```ruby class OsMe::CalibrationPrior def hit_rate(band:, source:) = band / 100.0 end class OsMe::ReviewItem < Data.define(:id, :level, :source, :band) def novel? = source == :taste end class OsMe::ReviewOrder def initialize(cost_model, calibration); end def sort(items); end end class OsMe::RandomSample PERCENT = 5 def pick?(key) = Digest::SHA256.hexdigest("sample:#{key}").to_i(16) % 100 < PERCENT end class OsMe::CLI < Thor desc 'sample KEY', 'exit 0 if the row joins the 5% review sample, exit 1 if not' def sample(key); end desc 'review-order ITEMS_FILE', 'print the item ids in review order' def review_order(items_file); end end ``` ## Non-goals - Hit rates from calibration data (S10). The prior uses the band only. - `OneThingSelector` and `SessionLedger` (S9). - Sorting low-stakes rows. They have no lever labels; they keep their order. - A sample of judgment or low-stakes rows. Those rows are listed already. - os-sdlc code. The skill text and `decision-tiers.md` change as orchestrator work. Blocked by: #1247 (`CalibrationPrior`). ## Origin - Trigger: User asked for the decision-system build tickets this session. Rewritten 2026-10-23 with exact criteria after the #1245 lesson. - Improvised this session: none - Chain: off-design S3 build ← acceptance criteria too loose ← first ticket draft. Stop at `DESIGN` (`docs/decision-system/review.md`). - Root candidate: none found; searched ReviewOrder, RandomSample, optional review, fast-batch - Where: docs/decision-system/build-plan.md S5; docs/decision-system/review.md; plugins/os-me/lib/os_me/; plugins/os-me/lib/os_me/cli.rb - Session: 4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5 - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5.jsonl VERDICT: CREATE (user-approved; backlog search only)

jared added the

project/cc-os

filed-by/user

category/code-change

labels

2026-10-23 17:46:12 +00:00

jared referenced this issue

2026-10-23 18:01:00 +00:00

decision-system S5: route judgment rows in os-sdlc flows #1247

jared commented

2026-10-23 18:04:01 +00:00

Author

Owner

Work started on branch ticket-1265 (/os-sdlc:implement, S5b).

jared commented

2026-10-23 18:10:49 +00:00

Author

Owner

Resolution

Done: S5b built: CalibrationPrior#hit_rate, ReviewItem, ReviewOrder, RandomSample, os-me-eval sample and review-order; implementation 132 complete.

Evidence: Branch ticket-1265, Runner commit; one test-review round; suite, lint and required-work gates green; contract audit passed first time; 10 dispatches, 6 min 30 s.

Follow-ups: Skill text (decision-tiers.md final judgment round, fast-batch, crc-refactor) to call os-me-route and the review-order/sample CLIs is orchestrator work for S5, done next on main.

## Resolution **Done:** S5b built: CalibrationPrior#hit_rate, ReviewItem, ReviewOrder, RandomSample, os-me-eval sample and review-order; implementation 132 complete. **Evidence:** Branch ticket-1265, Runner commit; one test-review round; suite, lint and required-work gates green; contract audit passed first time; 10 dispatches, 6 min 30 s. **Follow-ups:** Skill text (decision-tiers.md final judgment round, fast-batch, crc-refactor) to call os-me-route and the review-order/sample CLIs is orchestrator work for S5, done next on main.

jared closed this issue

2026-10-23 18:10:49 +00:00

jared referenced this issue from a commit

2026-10-23 18:11:47 +00:00

feat(os-me): Audited POODR boundaries for S5b review order and sample CLIs; design and contracts hold, all five WORK items present. (#1265)

jared referenced this issue from a commit

2026-10-23 18:14:13 +00:00

docs(os-sdlc): final judgment round routes rows through os-me-route (S5, #1247, #1265)