jared/cc-os

Watch

Fork 0

Record every Prediction in a session, not only the first #1233

New issue

Closed

opened 2026-10-22 15:38:06 +00:00 by jared · 3 comments

jared commented

2026-10-22 15:38:06 +00:00

Owner

Problem

A second /os-me:predict in one session is never recorded. PredictionParser#parse gives each ruling the id session:<n>, numbered from 1 inside one Prediction. PredictionLog#append drops a record whose id and session_id are already in the log. So the first Prediction of a session takes session:1, and every later Prediction in that session is dropped without a message. The log then under-counts predictions, and the second Prediction's verdict can never be scored.

Observed

Session 8468f10b showed two Predictions on 2026-10-22 (correct, 70%; redirect, 60%). The log holds one record for it.

ruby -rjson -e 'File.foreach(File.expand_path("~/dev/os-me-store/prediction-log.jsonl")){|l| r=JSON.parse(l); puts [r["id"],r["confidence"],r["recorded_at"]].inspect if r["session_id"].to_s.start_with?("8468f10b")}'

["session:1", 70, "2026-10-22T14:42:10Z"]

Reproduce

cd ~/dev/cc-os/plugins/os-me && bundle exec ruby -Itest -e 'require "test_helper"; puts "write a test: two Predictions in one transcript, append twice, expect two records"'

Approach

Give each Prediction in a session its own id, so the dedupe keeps its meaning (one record per shown ruling) without dropping later Predictions. One shape: the id carries the Prediction's position in the transcript, for example the index of the notification entry, or its timestamp. The parser owns the id; the log owns the dedupe. Keep score-record ID working for old ids.

Required work:

A second Prediction in one session is appended as its own record — check: present recorded_at in plugins/os-me/lib/os_me/prediction_log.rb

Acceptance criteria

Given a transcript with two Predictions, when the hook appends after each, then the log holds two records for that session with different ids.
Given the same Prediction appended twice, when the hook runs again, then the log still holds one record for it.
Given an old record with id session:1, when score-record session:1 --verdict hit runs with its session, then it scores.

Owner sketch

class OsMe::PredictionParser
  def parse(text, position:)
end
class OsMe::PredictionLog
  def append(incoming)
end

Origin

Trigger: checking the prediction log after the 2026-10-22 fixes (d4fa00d7, 9530bb66); this session's second Prediction was missing.
Improvised this session: none.
Chain: second Prediction missing ← PredictionLog#append dedupe on (id, session_id) ← ids are session:<n> per Prediction, not per session ← DESIGN (#1079 append dedupe by id and session; #1081 id assignment in the parser).
Root candidate: none found; searched os-me, prediction-log, PredictionLog, PredictionParser, dedupe, session:1, second Prediction, append
Where: plugins/os-me/lib/os_me/prediction_log.rb append; plugins/os-me/lib/os_me/prediction_parser.rb parse
Session: 8468f10b-0466-49d3-b0b5-d1d9f8d47abc
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/8468f10b-0466-49d3-b0b5-d1d9f8d47abc.jsonl

VERDICT: CREATE

## Problem A second `/os-me:predict` in one session is never recorded. `PredictionParser#parse` gives each ruling the id `session:<n>`, numbered from 1 inside one Prediction. `PredictionLog#append` drops a record whose id and session_id are already in the log. So the first Prediction of a session takes `session:1`, and every later Prediction in that session is dropped without a message. The log then under-counts predictions, and the second Prediction's verdict can never be scored. ## Observed Session 8468f10b showed two Predictions on 2026-10-22 (correct, 70%; redirect, 60%). The log holds one record for it. ``` ruby -rjson -e 'File.foreach(File.expand_path("~/dev/os-me-store/prediction-log.jsonl")){|l| r=JSON.parse(l); puts [r["id"],r["confidence"],r["recorded_at"]].inspect if r["session_id"].to_s.start_with?("8468f10b")}' ``` ``` ["session:1", 70, "2026-10-22T14:42:10Z"] ``` ## Reproduce ``` cd ~/dev/cc-os/plugins/os-me && bundle exec ruby -Itest -e 'require "test_helper"; puts "write a test: two Predictions in one transcript, append twice, expect two records"' ``` ## Approach Give each Prediction in a session its own id, so the dedupe keeps its meaning (one record per shown ruling) without dropping later Predictions. One shape: the id carries the Prediction's position in the transcript, for example the index of the notification entry, or its timestamp. The parser owns the id; the log owns the dedupe. Keep `score-record ID` working for old ids. Required work: 1. A second Prediction in one session is appended as its own record — check: present `recorded_at` in `plugins/os-me/lib/os_me/prediction_log.rb` ## Acceptance criteria 1. Given a transcript with two Predictions, when the hook appends after each, then the log holds two records for that session with different ids. 2. Given the same Prediction appended twice, when the hook runs again, then the log still holds one record for it. 3. Given an old record with id `session:1`, when `score-record session:1 --verdict hit` runs with its session, then it scores. ## Owner sketch ```ruby class OsMe::PredictionParser def parse(text, position:) end class OsMe::PredictionLog def append(incoming) end ``` ## Origin - Trigger: checking the prediction log after the 2026-10-22 fixes (d4fa00d7, 9530bb66); this session's second Prediction was missing. - Improvised this session: none. - Chain: second Prediction missing ← `PredictionLog#append` dedupe on (id, session_id) ← ids are `session:<n>` per Prediction, not per session ← DESIGN (#1079 append dedupe by id and session; #1081 id assignment in the parser). - Root candidate: none found; searched os-me, prediction-log, PredictionLog, PredictionParser, dedupe, session:1, second Prediction, append - Where: `plugins/os-me/lib/os_me/prediction_log.rb` `append`; `plugins/os-me/lib/os_me/prediction_parser.rb` `parse` - Session: 8468f10b-0466-49d3-b0b5-d1d9f8d47abc - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/8468f10b-0466-49d3-b0b5-d1d9f8d47abc.jsonl VERDICT: CREATE

jared added the

project/cc-os

filed-by/agent

category/code-change

labels

2026-10-22 15:38:06 +00:00

jared referenced this issue

2026-11-13 20:10:22 +00:00

os-me: score applied rulings from later corrections, with no reply from Jared #1753

jared referenced this issue

2026-11-13 20:10:58 +00:00

os-me: the prediction parser reads every Prediction block shape and every block of a message #1758

jared referenced this issue

2026-11-13 20:24:47 +00:00

Define the ruled_by vocabulary, a prediction id, and a session id for docs/decisions/rulings.tsv #1757

jared commented

2026-11-13 20:24:48 +00:00

Author

Owner

Findings of the backfill run on 2026-11-13. Checked against the source on that date.

Cause that this ticket owns: the id.

YamlPrediction#row (plugins/os-me/lib/os_me/prediction_parser/yaml_prediction.rb:25) always gives the id session:1. The body above says that ids are numbered session:<n>. That is true only for LegacyPrediction (letters a, b, c). Each YAML Prediction gets session:1.
PredictionLog#identity (plugins/os-me/lib/os_me/prediction_log.rb:36-39) is the pair id and session_id. PredictionLog#append (prediction_log.rb:14-19) drops a record whose pair is in the log, with no message.
So a session can have one YAML row only. All 21 records since 2026-11-04 have the id session:1.

Evidence sessions.

f8d22c34: one message with 3 YAML blocks, then two more Predictions. The log has 1 row.
47ffa777: two Predictions (invocations at 2026-11-05T17:55:35Z and 18:25:56Z). The log has 1 row.

Split with #1758, so the two tickets do not overlap.

#1233 (this ticket) owns the id: each shown prediction of a session gets its own id, and the dedupe keeps one row for each. This includes several blocks in one message: they need different ids.
#1758 owns the other causes: (a) the block shapes that the parser rejects, (b) PredictionParser#yaml_block keeps only the last block of a message, (c) CliAppender#prediction_shown? reads the whole transcript, so an older Prediction marks a new invocation as recorded, (d) append cannot name an invocation and the hook path is silent, (e) no held-out check.
This ticket did not own b or c before; nothing moved here. Causes b and c cannot add rows until this ticket ships. Implement this ticket first, or with #1758 in one change.

Note for acceptance criterion 2. After #1758 limits the read to the pending invocation, the same Prediction is no longer re-appended by a later Stop. The dedupe must still keep one row when append runs two times over one transcript.

Findings of the backfill run on 2026-11-13. Checked against the source on that date. **Cause that this ticket owns: the id.** - `YamlPrediction#row` (`plugins/os-me/lib/os_me/prediction_parser/yaml_prediction.rb:25`) always gives the id `session:1`. The body above says that ids are numbered `session:<n>`. That is true only for `LegacyPrediction` (letters a, b, c). Each YAML Prediction gets `session:1`. - `PredictionLog#identity` (`plugins/os-me/lib/os_me/prediction_log.rb:36-39`) is the pair id and session_id. `PredictionLog#append` (`prediction_log.rb:14-19`) drops a record whose pair is in the log, with no message. - So a session can have one YAML row only. All 21 records since 2026-11-04 have the id `session:1`. **Evidence sessions.** - f8d22c34: one message with 3 YAML blocks, then two more Predictions. The log has 1 row. - 47ffa777: two Predictions (invocations at 2026-11-05T17:55:35Z and 18:25:56Z). The log has 1 row. **Split with #1758, so the two tickets do not overlap.** - #1233 (this ticket) owns the id: each shown prediction of a session gets its own id, and the dedupe keeps one row for each. This includes several blocks in one message: they need different ids. - #1758 owns the other causes: (a) the block shapes that the parser rejects, (b) `PredictionParser#yaml_block` keeps only the last block of a message, (c) `CliAppender#prediction_shown?` reads the whole transcript, so an older Prediction marks a new invocation as recorded, (d) `append` cannot name an invocation and the hook path is silent, (e) no held-out check. - This ticket did not own b or c before; nothing moved here. Causes b and c cannot add rows until this ticket ships. Implement this ticket first, or with #1758 in one change. **Note for acceptance criterion 2.** After #1758 limits the read to the pending invocation, the same Prediction is no longer re-appended by a later Stop. The dedupe must still keep one row when `append` runs two times over one transcript.

jared commented

2026-11-13 20:25:50 +00:00

Author

Owner

Work started on 2026-11-13 through /os-sdlc:implement. Branch: ticket-1233. Cause and evidence sessions are in the comment of 2026-11-13; #1758 owns the parser and Stop hook faults.

jared commented

2026-11-13 20:43:26 +00:00

Author

Owner

Resolution

Done: Each Prediction in a session now gets its own positioned id, so the log keeps a second and later Prediction of the same session. The parser owns the positioned id; the log keeps its dedupe on id and session.

Evidence: os-sdlc implementation 291, 2026-11-13: 18 dispatches, suite gate green, survivor gate green, contract audit pass. Branch ticket-1233; paths plugins/os-me/lib/os_me/prediction_parser.rb, plugins/os-me/lib/os_me/cli/append_request.rb, plugins/os-me/test/prediction_repeat_cli_test.rb, plugins/os-me/test/prediction_cli_test.rb.

Follow-ups: #1758 (already filed): the parser block shapes, the last-block rule, the whole-transcript ready check, and the held-out check. No new follow-up.

## Resolution **Done:** Each Prediction in a session now gets its own positioned id, so the log keeps a second and later Prediction of the same session. The parser owns the positioned id; the log keeps its dedupe on id and session. **Evidence:** os-sdlc implementation 291, 2026-11-13: 18 dispatches, suite gate green, survivor gate green, contract audit pass. Branch ticket-1233; paths plugins/os-me/lib/os_me/prediction_parser.rb, plugins/os-me/lib/os_me/cli/append_request.rb, plugins/os-me/test/prediction_repeat_cli_test.rb, plugins/os-me/test/prediction_cli_test.rb. **Follow-ups:** #1758 (already filed): the parser block shapes, the last-block rule, the whole-transcript ready check, and the held-out check. No new follow-up.

jared closed this issue

2026-11-13 20:43:26 +00:00

jared referenced this issue

2026-11-13 20:44:39 +00:00

os-me: the prediction parser reads every Prediction block shape and every block of a message #1758

jared referenced this issue from a commit

2026-11-13 20:49:46 +00:00

feat(os-me): Audited POODR boundaries for ticket 1233; the parser owns the positioned id, the log keeps the dedupe, contracts protected. (#1233)

jared referenced this issue

2026-11-13 21:24:50 +00:00

os-me: the prediction parser makes rows from Prediction blocks that an agent report only quotes #1763

jared referenced this issue