os-me: bin/os-me-report renders prediction statistics

jared commented

2026-10-17 11:54:36 +00:00

Owner

Problem

On 2026-10-17 four views were computed by hand. They are the daily hit table, hit rate per trigger id, reliability by confidence bucket, and accept rate by origin. Jared wants them at his fingertips.

Reproduce

n/a: feature work. No report command exists today.

Approach

Required work:

Add class OsMe::PredictionReport. It reads predictions.jsonl and builds four tables. Per day: hit, partial, miss, unscored. Per trigger id: n, hit rate, last-20 hit rate. Per confidence bucket of 10 points: n, hit rate. Per origin and autopilot: n, accepted-verbatim share, discussed share. — check: present class PredictionReport in plugins/os-me/lib/os_me/prediction_report.rb
Add the CLI bin/os-me-report [--since DATE] [--format table|json]. It prints Markdown tables on stdout. It prints JSON on request. — check: present os-me-report in plugins/os-me/bin/os-me-report

Acceptance criteria

Given a fixture log, when the report runs, then the per-day table shows the right counts.
Given a fixture log, when the report runs, then the per-trigger table shows n, hit rate, and last-20 hit rate.
Given a fixture log, when the report runs, then the per-bucket table shows n and hit rate for each 10-point bucket.
Given a fixture log, when the report runs, then the per-origin table shows n and both shares.
Given --since DATE, when the report runs, then it omits records before that date.
Given --format json, when the report runs, then stdout parses as JSON.

Owner sketch

module OsMe
  class PredictionReport
    def initialize(records:, since: nil); end
    def to_markdown; end
    def to_h; end
  end
end

Non-goals

Writing or scoring records.

Blocked by #1074

Origin

Trigger: Jared asked for this ticket on 2026-10-17 while designing os-me prediction records.
Improvised this session: none
Chain: statistics were hand-computed ← no tool reads the record log. DESIGN (plugins/os-me/lib/os_me/prediction_report.rb)
Root candidate: this ticket
Where: plugins/os-me/lib/os_me/prediction_report.rb
Session: 2fc707d3-1acb-4f67-98d1-3685e5283f89
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/2fc707d3-1acb-4f67-98d1-3685e5283f89.jsonl

## Problem On 2026-10-17 four views were computed by hand. They are the daily hit table, hit rate per trigger id, reliability by confidence bucket, and accept rate by origin. Jared wants them at his fingertips. ## Reproduce n/a: feature work. No report command exists today. ## Approach Required work: 1. Add class OsMe::PredictionReport. It reads predictions.jsonl and builds four tables. Per day: hit, partial, miss, unscored. Per trigger id: n, hit rate, last-20 hit rate. Per confidence bucket of 10 points: n, hit rate. Per origin and autopilot: n, accepted-verbatim share, discussed share. — check: present `class PredictionReport` in `plugins/os-me/lib/os_me/prediction_report.rb` 2. Add the CLI `bin/os-me-report [--since DATE] [--format table|json]`. It prints Markdown tables on stdout. It prints JSON on request. — check: present `os-me-report` in `plugins/os-me/bin/os-me-report` ## Acceptance criteria 1. Given a fixture log, when the report runs, then the per-day table shows the right counts. 2. Given a fixture log, when the report runs, then the per-trigger table shows n, hit rate, and last-20 hit rate. 3. Given a fixture log, when the report runs, then the per-bucket table shows n and hit rate for each 10-point bucket. 4. Given a fixture log, when the report runs, then the per-origin table shows n and both shares. 5. Given --since DATE, when the report runs, then it omits records before that date. 6. Given --format json, when the report runs, then stdout parses as JSON. ## Owner sketch ```ruby module OsMe class PredictionReport def initialize(records:, since: nil); end def to_markdown; end def to_h; end end end ``` ## Non-goals - Writing or scoring records. Blocked by #1074 ## Origin - Trigger: Jared asked for this ticket on 2026-10-17 while designing os-me prediction records. - Improvised this session: none - Chain: statistics were hand-computed ← no tool reads the record log. DESIGN (plugins/os-me/lib/os_me/prediction_report.rb) - Root candidate: this ticket - Where: plugins/os-me/lib/os_me/prediction_report.rb - Session: 2fc707d3-1acb-4f67-98d1-3685e5283f89 - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/2fc707d3-1acb-4f67-98d1-3685e5283f89.jsonl

jared added the

project/cc-os

filed-by/user

category/code-change

labels

2026-10-17 11:54:36 +00:00

jared referenced this issue

2026-10-17 11:54:38 +00:00

os-me: leaf History row for the record log and a pointer to bin/os-me-report #1080

jared referenced this issue

2026-10-17 12:31:36 +00:00

os-me: prediction records carry triggers, verdict, origin, and autopilot #1081

jared commented

2026-10-17 12:31:38 +00:00

Author

Owner

Also blocked by #1081: records must carry origin/autopilot (1075) and triggers/verdict (1077) first.

jared referenced this issue

2026-10-17 12:34:49 +00:00

os-me: bin/os-me-record appends and scores prediction records #1074

jared commented

2026-10-17 13:37:01 +00:00

Author

Owner

Takedown relevance: not relevant

Harm: Jared has to hand-compute the prediction statistics (daily hit table, per-trigger hit rate, per-bucket reliability, per-origin accept rate) because no tool reads plugins/os-me's prediction log and prints them.
Protects: Calibrated self-prediction: Jared can see how often os-me's predictions hit and whether confidence means anything, without manual tallying that is slow and error-prone.
Harm path: The ticket's new entry point, plugins/os-me/bin/os-me-report, which a person would run to print the four tables. The only bin script in the plugin is os-me-eval, and nothing registers or calls an os-me-report command. (live: false)

Evidence:

plugins/os-me/bin/os-me-report: File does not exist; Glob of plugins/os-me/bin/* returns only os-me-eval.
plugins/os-me/lib/os_me/prediction_report.rb: File does not exist; no PredictionReport class anywhere under plugins/os-me.
plugins/os-me/lib/os_me/cli.rb:51-53: The only report output in the CLI is the scorer report from os-me-eval; no prediction-statistics command is registered.
.takedown/history.txt:6962: The only history line naming a plugins/os-me/bin path is os-me-eval; no commit adds or mentions os-me-report or prediction_report.
b8460ae9: Commit records predictions on Stop with origin and autopilot, so the log the report would read is written, but no reader for it exists.

Close candidate for Jared's batch ruling. The ticket stays open.

Takedown relevance: not relevant Harm: Jared has to hand-compute the prediction statistics (daily hit table, per-trigger hit rate, per-bucket reliability, per-origin accept rate) because no tool reads plugins/os-me's prediction log and prints them. Protects: Calibrated self-prediction: Jared can see how often os-me's predictions hit and whether confidence means anything, without manual tallying that is slow and error-prone. Harm path: The ticket's new entry point, plugins/os-me/bin/os-me-report, which a person would run to print the four tables. The only bin script in the plugin is os-me-eval, and nothing registers or calls an os-me-report command. (live: false) Evidence: - plugins/os-me/bin/os-me-report: File does not exist; Glob of plugins/os-me/bin/* returns only os-me-eval. - plugins/os-me/lib/os_me/prediction_report.rb: File does not exist; no PredictionReport class anywhere under plugins/os-me. - plugins/os-me/lib/os_me/cli.rb:51-53: The only report output in the CLI is the scorer report from os-me-eval; no prediction-statistics command is registered. - .takedown/history.txt:6962: The only history line naming a plugins/os-me/bin path is os-me-eval; no commit adds or mentions os-me-report or prediction_report. - b8460ae9: Commit records predictions on Stop with origin and autopilot, so the log the report would read is written, but no reader for it exists. Close candidate for Jared's batch ruling. The ticket stays open.

jared commented

2026-10-17 13:37:36 +00:00

Author

Owner

Work started on branch ticket-1077 (os-sdlc poodr-ticket-implementation). The takedown relevance step judged this feature ticket not relevant (run 1077-20261017-093602: the new entry point does not exist yet), so the ticket's own Approach drives the build.

jared commented

2026-10-17 13:46:37 +00:00

Author

Owner

Resolution

Done: OsMe::PredictionReport builds the per-day, per-trigger, per-confidence-bucket and per-origin/autopilot tables; bin/os-me-report prints Markdown or --format json, with --since DATE.

Evidence: Implementation 56 (poodr-ticket-implementation, 15 dispatches, 2 retries): all 6 acceptance criteria verified, suite and contract audit pass; Runner commit on ticket-1077. Smoke run on the live Store prints four empty tables because predictions.jsonl does not exist yet (#1079 backfill creates it). Takedown relevance judged this feature ticket not relevant (run 1077-20261017-093602), so the ticket's own Approach drove the build.

Follow-ups: none

## Resolution **Done:** OsMe::PredictionReport builds the per-day, per-trigger, per-confidence-bucket and per-origin/autopilot tables; bin/os-me-report prints Markdown or --format json, with --since DATE. **Evidence:** Implementation 56 (poodr-ticket-implementation, 15 dispatches, 2 retries): all 6 acceptance criteria verified, suite and contract audit pass; Runner commit on ticket-1077. Smoke run on the live Store prints four empty tables because predictions.jsonl does not exist yet (#1079 backfill creates it). Takedown relevance judged this feature ticket not relevant (run 1077-20261017-093602), so the ticket's own Approach drove the build. **Follow-ups:** none

jared closed this issue

2026-10-17 13:46:37 +00:00

jared commented

2026-10-17 13:54:25 +00:00

Author

Owner

Skeptic DROP (no ticket): takedown is built around defect tickets (relevance, research and grader assume a defect and a cause). #1077 is a feature ticket; the misfit was routing it to takedown, not the relevance card.

jared referenced this issue

2026-10-17 14:14:15 +00:00

os-me: leaf History row for the record log and a pointer to bin/os-me-report #1080

jared referenced this issue from a commit

2026-10-17 14:47:02 +00:00

feat(os-me): Audited POODR boundaries for the 1077 green change; PredictionReport matches the owner sketch and the report seam is tested through the public CLI. (#1077)