decision-system S3: calibration report

jared commented

2026-10-23 15:17:25 +00:00

Owner

Problem

No report shows if a stated confidence is true. A memo that says "80%" must be right about 80% of the time. The moment log (S2) and the prediction log hold the outcomes, but no object turns them into hit rates per confidence band. Design: docs/decision-system/learning-loops.md, "Loop 1: calibration" and the TuneSplit sketch.

Approach

Points come from two sources, in one Point shape. The report does not join the two sources.

Source	Band	Outcome	Source label
Moment (`moments.jsonl`)	`ConfidenceBand.of_percent(stated_percent)`	`outcome`	the moment's `source`, else its `kind` (`memo_reply`)
Prediction record (`prediction-log.jsonl`)	`ConfidenceBand.of_percent(confidence)`	verdict `hit` = `accept`, `partial` = `modify`, `miss` = `override`	`prediction`

A moment with outcome none, a moment with no stated_percent, a record with no confidence, and a record with no verdict give no point. Hit value: accept 1.0, modify 0.5, override 0.0. Brier uses band / 100.0 as the forecast.

ConfidenceBand does not exist yet. Add only LEVELS = [50, 65, 80, 90] and self.of_percent(percent) = LEVELS.select { |l| l <= percent }.max || 50; S4 adds the rest (levers.md, "ConfidenceBand").

TuneSplit splits train moments into tune and holdout by a hash of the moment id, as in learning-loops.md: holdout?(moment_id) = Digest::SHA256.hexdigest("tune:#{moment_id}").to_i(16) % 100 < 20. It does not call OsMe::Split.

Required work:

Add ConfidenceBand.of_percent — check: present def self.of_percent in plugins/os-me/lib/os_me/confidence_band.rb
Add TuneSplit#holdout? that hashes tune: plus the moment id — check: present "tune:#{moment_id}" in plugins/os-me/lib/os_me/tune_split.rb
Add CalibrationReport, Point, and the command os-me-eval calibration — check: present class CalibrationReport in plugins/os-me/lib/os_me/calibration_report.rb

Acceptance criteria

Given a stated percent of 72, when ConfidenceBand.of_percent runs, then it returns 65; given 45, it returns 50; given 95, it returns 90.
Given the moment id abc:7, when TuneSplit#holdout? runs, then it returns the value of Digest::SHA256.hexdigest("tune:abc:7").to_i(16) % 100 < 20.
Given 1,000 moment ids s:0 to s:999, when TuneSplit#holdout? runs on each, then between 150 and 250 of them are holdout.
Given 19 points in band 65, when the report renders, then the band 65 row reads "too few".
Given 20 points in band 65 whose outcome is modify, when the report renders, then the band 65 hit rate is 0.50.
Given one point in band 80 with outcome accept and one point in band 65 with outcome override, when the report renders, then the Brier line reads 0.23.
Given a Store with 20 moments that state 72% with outcome accept, when os-me-eval calibration runs, then the output has a band 65 row with hit rate 1.00 and a memo_reply source row with hit rate 1.00.
Given a Store with 20 prediction records with confidence 85 and verdict partial, when os-me-eval calibration runs, then the output has a band 80 row with hit rate 0.50 and a prediction source row with hit rate 0.50.
Given a Store whose moments have outcome none or no stated percent, when os-me-eval calibration runs, then it counts none of them and prints Brier: n/a.

Owner sketch

class OsMe::ConfidenceBand
  LEVELS = [50, 65, 80, 90].freeze
  def self.of_percent(percent); end
end
class OsMe::TuneSplit
  HOLDOUT_PERCENT = 20
  def holdout?(moment_id); end
end
OsMe::Point = Data.define(:band, :source, :outcome)
class OsMe::CalibrationReport
  MIN_POINTS = 20
  HIT = { 'accept' => 1.0, 'modify' => 0.5, 'override' => 0.0 }.freeze
  def self.from_store(store_path); end
  def render; end
end

Non-goals

Per-route and per-labeler tables, ECE, p_wrong, and hit_rate(band:, source:) for other objects (S4, S5). Lever changes (S4). Tuning (S10).

Origin

Trigger: User asked for the decision-system build tickets this session. Rewritten 2026-10-23 after implementation 124: the first acceptance criteria did not pin the band rule, the two point sources, or TuneSplit, and the run built a session join with free band labels.
Improvised this session: none
Chain: off-design S3 build ← acceptance criteria too loose ← first ticket draft. Stop at DESIGN (docs/decision-system/learning-loops.md).
Root candidate: none found; searched decision-system, os-me build stage
Where: docs/decision-system/build-plan.md S3; plugins/os-me/lib/os_me/
Session: 4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5.jsonl

## Problem No report shows if a stated confidence is true. A memo that says "80%" must be right about 80% of the time. The moment log (S2) and the prediction log hold the outcomes, but no object turns them into hit rates per confidence band. Design: `docs/decision-system/learning-loops.md`, "Loop 1: calibration" and the `TuneSplit` sketch. ## Approach Points come from two sources, in one `Point` shape. The report does not join the two sources. | Source | Band | Outcome | Source label | |---|---|---|---| | Moment (`moments.jsonl`) | `ConfidenceBand.of_percent(stated_percent)` | `outcome` | the moment's `source`, else its `kind` (`memo_reply`) | | Prediction record (`prediction-log.jsonl`) | `ConfidenceBand.of_percent(confidence)` | verdict `hit` = `accept`, `partial` = `modify`, `miss` = `override` | `prediction` | A moment with outcome `none`, a moment with no `stated_percent`, a record with no `confidence`, and a record with no verdict give no point. Hit value: `accept` 1.0, `modify` 0.5, `override` 0.0. Brier uses `band / 100.0` as the forecast. `ConfidenceBand` does not exist yet. Add only `LEVELS = [50, 65, 80, 90]` and `self.of_percent(percent) = LEVELS.select { |l| l <= percent }.max || 50`; S4 adds the rest (`levers.md`, "ConfidenceBand"). `TuneSplit` splits train moments into tune and holdout by a hash of the moment id, as in `learning-loops.md`: `holdout?(moment_id) = Digest::SHA256.hexdigest("tune:#{moment_id}").to_i(16) % 100 < 20`. It does not call `OsMe::Split`. Required work: 1. Add `ConfidenceBand.of_percent` — check: present `def self.of_percent` in `plugins/os-me/lib/os_me/confidence_band.rb` 2. Add `TuneSplit#holdout?` that hashes `tune:` plus the moment id — check: present `"tune:#{moment_id}"` in `plugins/os-me/lib/os_me/tune_split.rb` 3. Add `CalibrationReport`, `Point`, and the command `os-me-eval calibration` — check: present `class CalibrationReport` in `plugins/os-me/lib/os_me/calibration_report.rb` ## Acceptance criteria 1. Given a stated percent of 72, when `ConfidenceBand.of_percent` runs, then it returns 65; given 45, it returns 50; given 95, it returns 90. 2. Given the moment id `abc:7`, when `TuneSplit#holdout?` runs, then it returns the value of `Digest::SHA256.hexdigest("tune:abc:7").to_i(16) % 100 < 20`. 3. Given 1,000 moment ids `s:0` to `s:999`, when `TuneSplit#holdout?` runs on each, then between 150 and 250 of them are holdout. 4. Given 19 points in band 65, when the report renders, then the band 65 row reads "too few". 5. Given 20 points in band 65 whose outcome is `modify`, when the report renders, then the band 65 hit rate is 0.50. 6. Given one point in band 80 with outcome `accept` and one point in band 65 with outcome `override`, when the report renders, then the Brier line reads 0.23. 7. Given a Store with 20 moments that state 72% with outcome `accept`, when `os-me-eval calibration` runs, then the output has a band 65 row with hit rate 1.00 and a `memo_reply` source row with hit rate 1.00. 8. Given a Store with 20 prediction records with confidence 85 and verdict `partial`, when `os-me-eval calibration` runs, then the output has a band 80 row with hit rate 0.50 and a `prediction` source row with hit rate 0.50. 9. Given a Store whose moments have outcome `none` or no stated percent, when `os-me-eval calibration` runs, then it counts none of them and prints `Brier: n/a`. ## Owner sketch ```ruby class OsMe::ConfidenceBand LEVELS = [50, 65, 80, 90].freeze def self.of_percent(percent); end end class OsMe::TuneSplit HOLDOUT_PERCENT = 20 def holdout?(moment_id); end end OsMe::Point = Data.define(:band, :source, :outcome) class OsMe::CalibrationReport MIN_POINTS = 20 HIT = { 'accept' => 1.0, 'modify' => 0.5, 'override' => 0.0 }.freeze def self.from_store(store_path); end def render; end end ``` ## Non-goals Per-route and per-labeler tables, ECE, `p_wrong`, and `hit_rate(band:, source:)` for other objects (S4, S5). Lever changes (S4). Tuning (S10). ## Origin - Trigger: User asked for the decision-system build tickets this session. Rewritten 2026-10-23 after implementation 124: the first acceptance criteria did not pin the band rule, the two point sources, or `TuneSplit`, and the run built a session join with free band labels. - Improvised this session: none - Chain: off-design S3 build ← acceptance criteria too loose ← first ticket draft. Stop at `DESIGN` (`docs/decision-system/learning-loops.md`). - Root candidate: none found; searched decision-system, os-me build stage - Where: docs/decision-system/build-plan.md S3; plugins/os-me/lib/os_me/ - Session: 4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5 - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5.jsonl

jared added the

project/cc-os

filed-by/user

category/code-change

labels

2026-10-23 15:17:25 +00:00

jared referenced this issue

2026-10-23 15:17:26 +00:00

decision-system S4: levers and os-me-route in predict #1246

jared referenced this issue

2026-10-23 15:17:29 +00:00

decision-system S9: demotion and the One Thing question #1251

jared referenced this issue

2026-10-23 15:17:29 +00:00

decision-system S10: lever tuning loops #1252

jared referenced this issue

2026-10-23 16:00:58 +00:00

os-me-extract: structured DecisionModel answers, per-session cursor, Haiku model #1256

jared commented

2026-10-23 16:23:32 +00:00

Author

Owner

Work started on branch ticket-1245 (os-sdlc implement pipeline). S2c extraction rerun is in progress; this run builds the S3 objects against fixtures.

jared commented

2026-10-23 16:33:11 +00:00

Author

Owner

Implementation 124 failed at suite-check (gate_skip): the orchestrator passed the previous agent's dispatch ID in the prompt. The green work also missed the design in learning-loops.md (band labels joined by session, TuneSplit reused the held-out split, outcome none counted as a miss). The off-design work is discarded. The body now pins the band rule, the two point sources, and TuneSplit in nine acceptance criteria. A fresh implementation follows.

jared commented

2026-10-23 16:40:27 +00:00

Author

Owner

Resolution

Done: ConfidenceBand.of_percent (50/65/80/90), TuneSplit#holdout? (SHA256 of tune:, 20%), Point, CalibrationReport (moments and prediction records as two point sources; too few under 20 points; modify counts half; Brier from band/100), and os-me-eval calibration. Built by implementation 125 after implementation 124 failed (orchestrator passed a stale dispatch ID; loose criteria let the build miss learning-loops.md). Body rewritten with nine acceptance criteria.

Evidence: Implementation 125 complete: 11 dispatches, 6 min 37 s; suite-check pass twice; contract audit pass after one refactor. Runner commit on ticket-1245 with 9 paths under plugins/os-me.

Follow-ups: none; the S3 data gate (three band cells with 20 or more points) runs on the real Store after the S2c extraction rerun, as orchestrator work under #1244

## Resolution **Done:** ConfidenceBand.of_percent (50/65/80/90), TuneSplit#holdout? (SHA256 of tune:<moment id>, 20%), Point, CalibrationReport (moments and prediction records as two point sources; too few under 20 points; modify counts half; Brier from band/100), and os-me-eval calibration. Built by implementation 125 after implementation 124 failed (orchestrator passed a stale dispatch ID; loose criteria let the build miss learning-loops.md). Body rewritten with nine acceptance criteria. **Evidence:** Implementation 125 complete: 11 dispatches, 6 min 37 s; suite-check pass twice; contract audit pass after one refactor. Runner commit on ticket-1245 with 9 paths under plugins/os-me. **Follow-ups:** none; the S3 data gate (three band cells with 20 or more points) runs on the real Store after the S2c extraction rerun, as orchestrator work under #1244

jared closed this issue

2026-10-23 16:40:28 +00:00

jared referenced this issue from a commit

2026-10-23 16:41:27 +00:00

feat(os-me): Audited POODR boundaries for os-me calibration; design and contracts are sound, all WORK items present. (#1245)

jared referenced this issue

2026-10-23 17:10:38 +00:00

decision-system S4b: evidence-loop levers (ConfidenceBand.claimed, EvidenceLadder, StopRule, StakeReducer, CostModel) #1262

jared referenced this issue

2026-10-23 17:10:39 +00:00

decision-system S4c: lever fields in PredictionParser and PredictionLog, and the os-me-route CLI #1263

jared referenced this issue