os-me: the prediction parser makes rows from Prediction blocks that an agent report only quotes #1763

New issue

Open

opened 2026-11-13 21:24:50 +00:00 by jared · 0 comments

jared commented

2026-11-13 21:24:50 +00:00

Owner

Problem

os-me-eval append makes rows from prediction: blocks that an agent report only quotes. These rows are not Predictions of a predict run. They are false rows in the candidate log, and they change the scores that os-me computes from the log.

The cause is in plugins/os-me/lib/os_me/ (branch ticket-1758):

Entry#prediction_message (entry.rb:40-46) returns [:notification, text] for every main-thread user entry whose text contains <task-notification> (Entry#notification_text, entry.rb:31-38).
TranscriptReader#prediction_messages (transcript_reader.rb:17-19) gives all of them to the parser (cli/append_request.rb:12).
PredictionParser::Transcript#unseen (prediction_parser/transcript.rb:16-22) makes rows from each fenced block that loads (PredictionParser::Message#predictions, prediction_parser/message.rb:26-29). The fence pattern YAML_BLOCK (prediction_parser/message.rb:6) matches a fence at the start of a line.
No object checks that the notification comes from a predict dispatch.

The fault class is older than #1758. Before #1758, the parser took the last block of the text: def yaml_block(text) = text.to_enum(:scan, YAML_BLOCK).map { Regexp.last_match[:body] }.last (prediction_parser.rb:46 at commit 43bdc0800). A quoted block then replaced the real Prediction. After #1758, a quoted block adds rows.

Scope of harm: sessions where an agent report quotes a Prediction block with the fence at the start of a line. In practice, these are sessions that work on os-me.

Observed

Real-data check of branch ticket-1758 on 2026-11-13 (implementation 292, scratch log, Store not written).

Session 2bb44e9c-522c-4c97-8449-bd46bed898e0 has one real predict run. os-me-eval append gave 6 rows.
Row 1 is correct. It comes from the task-notification of the predict agent. The assistant copy of that Prediction gives no row.
Row 2 comes from the task-notification of an implementation agent. Its report quotes one prediction: block with the fence at column 0.
Rows 3 to 6 come from the task-notification of a ticket-drafting agent. Its report quotes five example block shapes. Four of them give a row.
13 other real sessions in the same check gave correct rows.

Reproduce

os-me-eval append

Run it against a scratch log with the transcript of session 2bb44e9c-522c-4c97-8449-bd46bed898e0. It gives 6 rows; the session has 1 predict run.

Approach

This is a candidate direction. It is not decided.

Join each task-notification to its dispatch. Accept only the notifications of a predict dispatch.

Transcript facts, checked on 2026-11-13 in the transcript of session 2bb44e9c-522c-4c97-8449-bd46bed898e0 (48 Agent calls, 46 task-notifications on the main thread):

The notification entry is a type: user line. Its message.content is a String. The tool-use id is not a JSON field. It is the <tool-use-id> tag in the text, next to the <task-id> tag. The entry-level origin field is {kind: "task-notification", producer: "session-task", runId}. The runId value is not the tool-use id.
The Agent call is a type: assistant line. Its message.content[] has a block with type: "tool_use", name: "Agent", id, and input.prompt. Entry#agent_call_ids (entry.rb:48-52) reads this id now.
The <tool-use-id> value is equal to the id of an Agent tool_use block for 43 of the 46 notifications.
2 notifications have no <tool-use-id> tag. 1 notification has the id of a SendMessage call (a continued agent), not of an Agent call. The design must rule on these three cases.
The text ENTRY.md in the prompt is too weak as the test. 8 Agent prompts in this session contain it, because work on os-me names the file. Only 1 prompt starts with the brief line Read /home/jared/dev/os-me-store/ENTRY.md and follow it in <predict|respond> mode. (~/dev/os-me-store/brief.md:6; plugins/os-me/skills/predict/SKILL.md:17). That prompt is the real predict dispatch.

Required work: none beyond the acceptance criteria.

Acceptance criteria

Given a transcript with one predict dispatch and two later agent reports that quote Prediction blocks, when os-me-eval append runs, then it adds one row.
Given the 13 sessions of the 2026-11-13 check, when os-me-eval append runs on each, then each session keeps its row count.

Non-goals

#1758 owns the block shapes that the parser reads. This defect was found in its real-data check.
#1762 owns the Stop hook.
#1233 is closed.

Origin

Trigger: real-data check of branch ticket-1758 on 2026-11-13: os-me-eval append against a scratch log for 14 real sessions (implementation 292).
Improvised this session: none
Chain: 6 rows for 1 predict run (check of 2026-11-13) ← PredictionParser::Transcript#unseen makes rows from each block that loads (plugins/os-me/lib/os_me/prediction_parser/transcript.rb:16-22) ← Entry#prediction_message accepts every main-thread task-notification (plugins/os-me/lib/os_me/entry.rb:40-46) ← DESIGN: the parser identifies a Prediction by its block shape only, not by its dispatch (plugins/os-me/lib/os_me/prediction_parser/message.rb:6).
Root candidate: none found; searched prediction, parser, quote, task-notification, Entry#prediction_message, PredictionParser::Transcript, #1758, #1762, #1753
Where: OsMe::Entry#prediction_message (plugins/os-me/lib/os_me/entry.rb), OsMe::PredictionParser::Transcript (plugins/os-me/lib/os_me/prediction_parser/transcript.rb)
Session: 2bb44e9c-522c-4c97-8449-bd46bed898e0
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-ticket-1758/2bb44e9c-522c-4c97-8449-bd46bed898e0.jsonl

VERDICT: CREATE (ticket-skeptic, 2026-11-13). Reason: observed false rows in the prediction log; #1758 and #1762 do not have this fault; the data-validity floor applies.

## Problem `os-me-eval append` makes rows from `prediction:` blocks that an agent report only quotes. These rows are not Predictions of a predict run. They are false rows in the candidate log, and they change the scores that os-me computes from the log. The cause is in `plugins/os-me/lib/os_me/` (branch `ticket-1758`): - `Entry#prediction_message` (`entry.rb:40-46`) returns `[:notification, text]` for every main-thread user entry whose text contains `<task-notification>` (`Entry#notification_text`, `entry.rb:31-38`). - `TranscriptReader#prediction_messages` (`transcript_reader.rb:17-19`) gives all of them to the parser (`cli/append_request.rb:12`). - `PredictionParser::Transcript#unseen` (`prediction_parser/transcript.rb:16-22`) makes rows from each fenced block that loads (`PredictionParser::Message#predictions`, `prediction_parser/message.rb:26-29`). The fence pattern `YAML_BLOCK` (`prediction_parser/message.rb:6`) matches a fence at the start of a line. - No object checks that the notification comes from a predict dispatch. The fault class is older than #1758. Before #1758, the parser took the last block of the text: `def yaml_block(text) = text.to_enum(:scan, YAML_BLOCK).map { Regexp.last_match[:body] }.last` (`prediction_parser.rb:46` at commit 43bdc0800). A quoted block then replaced the real Prediction. After #1758, a quoted block adds rows. Scope of harm: sessions where an agent report quotes a Prediction block with the fence at the start of a line. In practice, these are sessions that work on os-me. ## Observed Real-data check of branch `ticket-1758` on 2026-11-13 (implementation 292, scratch log, Store not written). - Session `2bb44e9c-522c-4c97-8449-bd46bed898e0` has one real predict run. `os-me-eval append` gave 6 rows. - Row 1 is correct. It comes from the task-notification of the predict agent. The assistant copy of that Prediction gives no row. - Row 2 comes from the task-notification of an implementation agent. Its report quotes one `prediction:` block with the fence at column 0. - Rows 3 to 6 come from the task-notification of a ticket-drafting agent. Its report quotes five example block shapes. Four of them give a row. - 13 other real sessions in the same check gave correct rows. ## Reproduce ``` os-me-eval append ``` Run it against a scratch log with the transcript of session `2bb44e9c-522c-4c97-8449-bd46bed898e0`. It gives 6 rows; the session has 1 predict run. ## Approach This is a candidate direction. It is not decided. Join each task-notification to its dispatch. Accept only the notifications of a predict dispatch. Transcript facts, checked on 2026-11-13 in the transcript of session `2bb44e9c-522c-4c97-8449-bd46bed898e0` (48 Agent calls, 46 task-notifications on the main thread): - The notification entry is a `type: user` line. Its `message.content` is a String. The tool-use id is not a JSON field. It is the `<tool-use-id>` tag in the text, next to the `<task-id>` tag. The entry-level `origin` field is `{kind: "task-notification", producer: "session-task", runId}`. The `runId` value is not the tool-use id. - The Agent call is a `type: assistant` line. Its `message.content[]` has a block with `type: "tool_use"`, `name: "Agent"`, `id`, and `input.prompt`. `Entry#agent_call_ids` (`entry.rb:48-52`) reads this `id` now. - The `<tool-use-id>` value is equal to the `id` of an Agent `tool_use` block for 43 of the 46 notifications. - 2 notifications have no `<tool-use-id>` tag. 1 notification has the id of a `SendMessage` call (a continued agent), not of an Agent call. The design must rule on these three cases. - The text `ENTRY.md` in the prompt is too weak as the test. 8 Agent prompts in this session contain it, because work on os-me names the file. Only 1 prompt starts with the brief line `Read /home/jared/dev/os-me-store/ENTRY.md and follow it in <predict|respond> mode.` (`~/dev/os-me-store/brief.md:6`; `plugins/os-me/skills/predict/SKILL.md:17`). That prompt is the real predict dispatch. Required work: none beyond the acceptance criteria. ## Acceptance criteria 1. Given a transcript with one predict dispatch and two later agent reports that quote Prediction blocks, when `os-me-eval append` runs, then it adds one row. 2. Given the 13 sessions of the 2026-11-13 check, when `os-me-eval append` runs on each, then each session keeps its row count. ## Non-goals - #1758 owns the block shapes that the parser reads. This defect was found in its real-data check. - #1762 owns the Stop hook. - #1233 is closed. ## Origin - Trigger: real-data check of branch `ticket-1758` on 2026-11-13: `os-me-eval append` against a scratch log for 14 real sessions (implementation 292). - Improvised this session: none - Chain: 6 rows for 1 predict run (check of 2026-11-13) ← `PredictionParser::Transcript#unseen` makes rows from each block that loads (`plugins/os-me/lib/os_me/prediction_parser/transcript.rb:16-22`) ← `Entry#prediction_message` accepts every main-thread task-notification (`plugins/os-me/lib/os_me/entry.rb:40-46`) ← `DESIGN`: the parser identifies a Prediction by its block shape only, not by its dispatch (`plugins/os-me/lib/os_me/prediction_parser/message.rb:6`). - Root candidate: none found; searched prediction, parser, quote, task-notification, Entry#prediction_message, PredictionParser::Transcript, #1758, #1762, #1753 - Where: `OsMe::Entry#prediction_message` (`plugins/os-me/lib/os_me/entry.rb`), `OsMe::PredictionParser::Transcript` (`plugins/os-me/lib/os_me/prediction_parser/transcript.rb`) - Session: 2bb44e9c-522c-4c97-8449-bd46bed898e0 - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-ticket-1758/2bb44e9c-522c-4c97-8449-bd46bed898e0.jsonl VERDICT: CREATE (ticket-skeptic, 2026-11-13). Reason: observed false rows in the prediction log; #1758 and #1762 do not have this fault; the data-validity floor applies.

jared added the

project/cc-os

filed-by/agent

category/code-change

labels

2026-11-13 21:24:50 +00:00

jared referenced this issue

2026-11-13 21:25:10 +00:00

os-me: the prediction parser reads every Prediction block shape and every block of a message #1758

jared referenced this issue

2026-11-13 21:45:35 +00:00

os-me: the Stop hook records only a Prediction shown after the pending invocation, and skips held-out sessions #1762