jared/cc-os

Watch

Fork 0

os-me: score applied rulings from later corrections, with no reply from Jared #1753

New issue

Open

opened 2026-11-13 20:10:22 +00:00 by jared · 2 comments

jared commented

2026-11-13 20:10:22 +00:00

Owner

Problem

An applied ruling gets a score only when Jared replies to the Prediction. ~/dev/os-me-store/learning/loop.md step 3 says: "A prediction with no JARED: line in its block gets no score." and "Only a back-and-forth is evidence. Silence is not." The ladder in plugins/os-context/prompts/session-start/40-decisions.md says: "No rung waits for Jared." So a session applies the Prediction and continues, Jared does not reply, and the record stays without a verdict. The learning loop then has almost no material.

docs/decision-system/vision.md defines the sample as a chain, not a point. It says of silence: "Label it detected, weight it low, and never let it count as a confirmation." No machine step builds this chain verdict today.

Observed

Counts on 2026-11-13:

~/dev/os-me-store/prediction-log.jsonl has 21 records with recorded_at from 2026-11-04 to 2026-11-13. 17 have no verdict. The other 4 are 3 partial and 1 miss.
docs/decisions/rulings.tsv has 164 rows in the same period. 110 rows have a ruled_by value that names a prediction.
Store commits 14a7b37, 1a77761, 3c52424, b3efb8c, 713a8d9, and d2aed2a each clear sessions from the queue with the reason "no reply" and no score.

Gaps found by the manual backfill on 2026-11-13 (source checked on that date):

No detected verdict exists. plugins/os-me/lib/os_me/cli.rb:70 permits hit partial miss only; score-record --verdict detected exits 1. ~/dev/os-me-store/learning/loop.md steps 3 and 6 give silence no score. docs/decision-system/vision.md line 195 labels silence detected, with low weight.
loop.md step 3 says: "Judge each prediction against its own block only". So a correction in a later session cannot give a score.
score-record requires --depth (cli.rb:71: accepted-verbatim discussed overridden). loop.md does not say which value goes with which case, and the command in step 6 has no --depth.
No CLI command makes a log record from a row of docs/decisions/rulings.tsv. os-me-eval append reads a transcript. os-me-eval backfill reads the Store review log.
score-record cannot remove a verdict. It can replace one: a second score-record on the same id writes the new verdict with no message (PredictionLog#score, plugins/os-me/lib/os_me/prediction_log.rb:21-25). So nothing protects a verdict that came from a reply of Jared.

Result of the manual backfill on 2026-11-13:

The backfill run reported: 1 of 15 readable records with no verdict since 2026-11-04 got a verdict; 14 stay with no verdict because detected does not exist. The log after the run agrees with the 1: b10276a0 session:1 got partial on 2026-11-13 (Store commit 3c191c4). The log now has 21 records since 2026-11-04: 4 partial, 1 miss, 16 with no verdict. The 16 include 2 held-out sessions (ae4be489, 9d8c2e15) and 2 sessions that were live during the run (fdf2ac7d, 2bb44e9c).
A join of the predicted rows of rulings.tsv that have no log record proposed, for 71 rows: detected 44, partial 5, hit 2, miss 1, held out 18, session not found 1 (file line 83, id 79).
The join file is docs/decision-system/backfill/2026-11-13-rulings-chain-join.tsv. It has 72 rows: the 71 predicted rows and 1 row that Jared ruled (file line 111, proposed hit, outside the predicted pool). The 18 held-out rows carry no question text and no quote.

Reproduce

n/a: feature ticket.

Approach

The SessionStart extraction (plugins/os-me/hooks/extract_moments.rb) joins each predicted row of docs/decisions/rulings.tsv to later corrections. A correction is a later row that supersedes or reverts the ruling, or a later message from Jared that corrects it. The step writes the chain verdict through os-me-eval score-record:

A correction gives miss or partial.
No later row or message on the ruling gives a detected label with low weight.
Silence never gives hit.
The step skips held-out sessions.

A manual backfill for 2026-11-04 to 2026-11-13 runs on 2026-11-13. This ticket makes that backfill a machine step. Depends on #1757. The join needs a prediction id and a session id on each predicted row; #1757 defines them. Without them the join uses date and text, and it failed for file line 83.

Required scope, from the gaps in Observed:

Add the detected verdict to score-record and to each reader of verdict (PredictionReport::Rows, CalibrationReport). detected never counts as a hit.
Change loop.md step 3 so that a correction in a later session can give a score, and steps 3 and 6 so that silence gives detected.
State in loop.md which --depth value goes with each case, with the chain cases included.
Add a CLI path that makes a log record from a predicted rulings.tsv row that has no record.
Give score-record a way to remove a verdict, and a rule for replacement. The step must not replace a verdict that came from a reply of Jared (criterion 4).
loop.md is in the Store repository (~/dev/os-me-store). Edit it there and commit there.

Required work: none beyond the acceptance criteria.

Acceptance criteria

Given a predicted row in docs/decisions/rulings.tsv and a later row that supersedes it, when the next session starts, then the prediction record has the verdict miss or partial and no person ran a command.
Given a predicted row that no later row and no later message from Jared changes, when the next session starts, then the record has the detected label with low weight and does not have the verdict hit.
Given a predicted row from a held-out session, when the next session starts, then the step writes nothing for that row.
Given a record that Jared scored through a reply, when the next session starts, then the step does not change its verdict.
Given docs/decision-system/backfill/2026-11-13-rulings-chain-join.tsv, when the backfill runs again after this ticket ships, then the log has the proposed verdict of each row that is not held out and has a session, and no person ran a command or edited a file.
Given a record with a verdict, when score-record runs with the removal option, then the record has no verdict.

Non-goals

Where the learning loop runs (#958, #959). Recording of a second Prediction in one session (#1233). Promotion of confirmed predictions into lessons (#1240).

Origin

Trigger: the os-me check of 2026-11-13 in session 2bb44e9c. Jared asked how the os-me outcome function works and said "We definitely need to get this back on track. /os-me:predict desired outcome, how to get there, then do it!" The applied Prediction (docs/decisions/rulings.tsv row 161) sends this work to a ticket.
Improvised this session: a manual backfill of scores for 2026-11-04 to 2026-11-13 from rulings.tsv and transcripts
Chain: 17 of 21 records have no verdict ← the loop scores only on a reply from Jared (~/dev/os-me-store/learning/loop.md step 3) ← the ladder tells sessions not to wait for Jared (40-decisions.md, 2026-11-03 ruling) and the loop text is from 2026-10-05 ← DESIGN (docs/decision-system/vision.md chain sample, not built)
Root candidate: none found; searched #958, #959, #1233, #1240, #1544, score-record, "no reply", silence, detected, loop.md, extract_moments
Where: ~/dev/os-me-store/learning/loop.md; plugins/os-me/hooks/extract_moments.rb; plugins/os-me/lib/os_me/cli.rb (score-record)
Session: 2bb44e9c-522c-4c97-8449-bd46bed898e0
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/2bb44e9c-522c-4c97-8449-bd46bed898e0.jsonl

VERDICT: CREATE (ticket-skeptic, 2026-11-13): no open ticket builds a machine step that scores applied rulings from later corrections without a reply.

## Problem An applied ruling gets a score only when Jared replies to the Prediction. `~/dev/os-me-store/learning/loop.md` step 3 says: "A prediction with no `JARED:` line in its block gets no score." and "Only a back-and-forth is evidence. Silence is not." The ladder in `plugins/os-context/prompts/session-start/40-decisions.md` says: "No rung waits for Jared." So a session applies the Prediction and continues, Jared does not reply, and the record stays without a verdict. The learning loop then has almost no material. `docs/decision-system/vision.md` defines the sample as a chain, not a point. It says of silence: "Label it `detected`, weight it low, and never let it count as a confirmation." No machine step builds this chain verdict today. ## Observed Counts on 2026-11-13: - `~/dev/os-me-store/prediction-log.jsonl` has 21 records with `recorded_at` from 2026-11-04 to 2026-11-13. 17 have no verdict. The other 4 are 3 `partial` and 1 `miss`. - `docs/decisions/rulings.tsv` has 164 rows in the same period. 110 rows have a `ruled_by` value that names a prediction. - Store commits 14a7b37, 1a77761, 3c52424, b3efb8c, 713a8d9, and d2aed2a each clear sessions from the queue with the reason "no reply" and no score. Gaps found by the manual backfill on 2026-11-13 (source checked on that date): - No `detected` verdict exists. `plugins/os-me/lib/os_me/cli.rb:70` permits `hit partial miss` only; `score-record --verdict detected` exits 1. `~/dev/os-me-store/learning/loop.md` steps 3 and 6 give silence no score. `docs/decision-system/vision.md` line 195 labels silence `detected`, with low weight. - `loop.md` step 3 says: "Judge each prediction against its own block only". So a correction in a later session cannot give a score. - `score-record` requires `--depth` (`cli.rb:71`: `accepted-verbatim discussed overridden`). `loop.md` does not say which value goes with which case, and the command in step 6 has no `--depth`. - No CLI command makes a log record from a row of `docs/decisions/rulings.tsv`. `os-me-eval append` reads a transcript. `os-me-eval backfill` reads the Store review log. - `score-record` cannot remove a verdict. It can replace one: a second `score-record` on the same id writes the new verdict with no message (`PredictionLog#score`, `plugins/os-me/lib/os_me/prediction_log.rb:21-25`). So nothing protects a verdict that came from a reply of Jared. Result of the manual backfill on 2026-11-13: - The backfill run reported: 1 of 15 readable records with no verdict since 2026-11-04 got a verdict; 14 stay with no verdict because `detected` does not exist. The log after the run agrees with the 1: b10276a0 `session:1` got `partial` on 2026-11-13 (Store commit 3c191c4). The log now has 21 records since 2026-11-04: 4 `partial`, 1 `miss`, 16 with no verdict. The 16 include 2 held-out sessions (ae4be489, 9d8c2e15) and 2 sessions that were live during the run (fdf2ac7d, 2bb44e9c). - A join of the predicted rows of `rulings.tsv` that have no log record proposed, for 71 rows: `detected` 44, `partial` 5, `hit` 2, `miss` 1, held out 18, session not found 1 (file line 83, id 79). - The join file is `docs/decision-system/backfill/2026-11-13-rulings-chain-join.tsv`. It has 72 rows: the 71 predicted rows and 1 row that Jared ruled (file line 111, proposed `hit`, outside the predicted pool). The 18 held-out rows carry no question text and no quote. ## Reproduce n/a: feature ticket. ## Approach The SessionStart extraction (`plugins/os-me/hooks/extract_moments.rb`) joins each predicted row of `docs/decisions/rulings.tsv` to later corrections. A correction is a later row that supersedes or reverts the ruling, or a later message from Jared that corrects it. The step writes the chain verdict through `os-me-eval score-record`: - A correction gives `miss` or `partial`. - No later row or message on the ruling gives a `detected` label with low weight. - Silence never gives `hit`. - The step skips held-out sessions. A manual backfill for 2026-11-04 to 2026-11-13 runs on 2026-11-13. This ticket makes that backfill a machine step. Depends on #1757. The join needs a prediction id and a session id on each predicted row; #1757 defines them. Without them the join uses date and text, and it failed for file line 83. Required scope, from the gaps in Observed: - Add the `detected` verdict to `score-record` and to each reader of `verdict` (`PredictionReport::Rows`, `CalibrationReport`). `detected` never counts as a hit. - Change `loop.md` step 3 so that a correction in a later session can give a score, and steps 3 and 6 so that silence gives `detected`. - State in `loop.md` which `--depth` value goes with each case, with the chain cases included. - Add a CLI path that makes a log record from a predicted `rulings.tsv` row that has no record. - Give `score-record` a way to remove a verdict, and a rule for replacement. The step must not replace a verdict that came from a reply of Jared (criterion 4). - `loop.md` is in the Store repository (`~/dev/os-me-store`). Edit it there and commit there. Required work: none beyond the acceptance criteria. ## Acceptance criteria 1. Given a predicted row in `docs/decisions/rulings.tsv` and a later row that supersedes it, when the next session starts, then the prediction record has the verdict `miss` or `partial` and no person ran a command. 2. Given a predicted row that no later row and no later message from Jared changes, when the next session starts, then the record has the `detected` label with low weight and does not have the verdict `hit`. 3. Given a predicted row from a held-out session, when the next session starts, then the step writes nothing for that row. 4. Given a record that Jared scored through a reply, when the next session starts, then the step does not change its verdict. 5. Given `docs/decision-system/backfill/2026-11-13-rulings-chain-join.tsv`, when the backfill runs again after this ticket ships, then the log has the proposed verdict of each row that is not held out and has a session, and no person ran a command or edited a file. 6. Given a record with a verdict, when `score-record` runs with the removal option, then the record has no verdict. ## Non-goals Where the learning loop runs (#958, #959). Recording of a second Prediction in one session (#1233). Promotion of confirmed predictions into lessons (#1240). ## Origin - Trigger: the os-me check of 2026-11-13 in session 2bb44e9c. Jared asked how the os-me outcome function works and said "We definitely need to get this back on track. /os-me:predict desired outcome, how to get there, then do it!" The applied Prediction (docs/decisions/rulings.tsv row 161) sends this work to a ticket. - Improvised this session: a manual backfill of scores for 2026-11-04 to 2026-11-13 from rulings.tsv and transcripts - Chain: 17 of 21 records have no verdict ← the loop scores only on a reply from Jared (~/dev/os-me-store/learning/loop.md step 3) ← the ladder tells sessions not to wait for Jared (40-decisions.md, 2026-11-03 ruling) and the loop text is from 2026-10-05 ← DESIGN (docs/decision-system/vision.md chain sample, not built) - Root candidate: none found; searched #958, #959, #1233, #1240, #1544, score-record, "no reply", silence, detected, loop.md, extract_moments - Where: ~/dev/os-me-store/learning/loop.md; plugins/os-me/hooks/extract_moments.rb; plugins/os-me/lib/os_me/cli.rb (score-record) - Session: 2bb44e9c-522c-4c97-8449-bd46bed898e0 - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/2bb44e9c-522c-4c97-8449-bd46bed898e0.jsonl VERDICT: CREATE (ticket-skeptic, 2026-11-13): no open ticket builds a machine step that scores applied rulings from later corrections without a reply.

jared added the

project/cc-os

filed-by/agent

category/code-change

labels

2026-11-13 20:10:22 +00:00

jared referenced this issue

2026-11-13 20:10:50 +00:00

Define the ruled_by vocabulary, a prediction id, and a session id for docs/decisions/rulings.tsv #1757

jared referenced this issue

2026-11-13 20:24:45 +00:00

os-me: the prediction parser reads every Prediction block shape and every block of a message #1758

jared referenced this issue from a commit

2026-11-13 20:49:46 +00:00

fix(os-context): Define outcome node project name

jared referenced this issue

2026-11-13 21:24:50 +00:00

os-me: the prediction parser makes rows from Prediction blocks that an agent report only quotes #1763

jared commented

2026-11-13 21:50:36 +00:00

Author

Owner

Decompose, phase Decide (2026-11-13). a. One change: one step that joins each predicted row of docs/decisions/rulings.tsv to its log record by prediction_id and session_id (ADR-0313, #1757 done) and writes the chain verdict. b. Owner: no existing object owns that join; a new object is needed, whose secret is how a predicted row and the later rows on the same question become a verdict. c. Route: a new object, and the work touches the CLI, the reports, the SessionStart hook and the Store text, so this ticket goes to /os-sdlc:ticket-takedown 1753. Takedown writes the Owner sketch.

jared referenced this issue

2026-11-13 21:51:28 +00:00

Define the ruled_by vocabulary, a prediction id, and a session id for docs/decisions/rulings.tsv #1757

jared commented

2026-11-13 22:34:16 +00:00

Author

Owner

Occurrence: Score a ruling from a correction that Jared gives only in chat
Seen: plugins/os-me/lib/os_me/rulings_chain.rb; plugins/os-me/hooks/extract_moments.rb
Run: session 2bb44e9c-522c-4c97-8449-bd46bed898e0, hand-finish of the takedown design docs/takedown/1753-20261113-165135
Adds: the ticket stays open for this half only. Shipped on 2026-11-13 (commit 2617b07a5 on main, ADR-0314, Store 08c0867): the SessionStart step hooks/score_chain.rb scores from the corrects column (miss, partial), writes detected with low weight on silence, skips held-out sessions and rows with no full session id, never replaces a reply verdict, loads the chain-join backfill, and score-record has detected and --remove; loop.md steps 3 and 6 are updated. Acceptance criteria 1, 3, 4, 5, 6 pass (os-me suite 525 runs, 0 failures; one design audit, all findings fixed or answered). First real run: 196 to 251 records, 56 chain verdicts, 0 existing verdicts changed. Not done: a correction that Jared gives in a later message with no corrects row still leaves the ruling as detected (criterion 2 treats it as silence). Material for it: moments.jsonl labels corrections; ChainWriter lets a later verdict replace its own detected. Known limits: 16 Stop-hook records since 2026-11-04 have no verdict because their table rows carry no prediction_id; up to 17 ruling: records sit in sessions that already have a Stop-hook record; a removed verdict returns as detected at the next session start; the --depth meanings in loop.md step 6 are inferred and need a check by Jared. The takedown pipeline run failed at green-suite on one lint line; the build was done test-first by hand.

Occurrence: Score a ruling from a correction that Jared gives only in chat Seen: plugins/os-me/lib/os_me/rulings_chain.rb; plugins/os-me/hooks/extract_moments.rb Run: session 2bb44e9c-522c-4c97-8449-bd46bed898e0, hand-finish of the takedown design docs/takedown/1753-20261113-165135 Adds: the ticket stays open for this half only. Shipped on 2026-11-13 (commit 2617b07a5 on main, ADR-0314, Store 08c0867): the SessionStart step hooks/score_chain.rb scores from the corrects column (miss, partial), writes detected with low weight on silence, skips held-out sessions and rows with no full session id, never replaces a reply verdict, loads the chain-join backfill, and score-record has detected and --remove; loop.md steps 3 and 6 are updated. Acceptance criteria 1, 3, 4, 5, 6 pass (os-me suite 525 runs, 0 failures; one design audit, all findings fixed or answered). First real run: 196 to 251 records, 56 chain verdicts, 0 existing verdicts changed. Not done: a correction that Jared gives in a later message with no corrects row still leaves the ruling as detected (criterion 2 treats it as silence). Material for it: moments.jsonl labels corrections; ChainWriter lets a later verdict replace its own detected. Known limits: 16 Stop-hook records since 2026-11-04 have no verdict because their table rows carry no prediction_id; up to 17 ruling:<id> records sit in sessions that already have a Stop-hook record; a removed verdict returns as detected at the next session start; the --depth meanings in loop.md step 6 are inferred and need a check by Jared. The takedown pipeline run failed at green-suite on one lint line; the build was done test-first by hand.

jared referenced this issue from a commit

2026-11-13 22:34:19 +00:00

docs(adr): Supersede ADR-0313 with ADR-0314

jared referenced this issue from a commit

2026-11-13 22:34:19 +00:00

feat(os-me): Score rulings from later corrections

jared referenced this issue from a commit

2026-11-13 22:34:19 +00:00

docs(os-me): Record chain scoring and rulings columns