jared/cc-os

Watch

Fork 0

os-me: bin/os-me-record appends and scores prediction records #1074

New issue

Closed

opened 2026-10-17 11:54:30 +00:00 by jared · 2 comments

jared commented

2026-10-17 11:54:30 +00:00

Owner

Problem

The LLM hand-writes the rows of review-log.md. Only 18% of the 131 rows name a trigger id. The stated Confidence percent is never stored. Hit rate per trigger, calibration, and trust by origin cannot be computed.

Design

/home/jared/dev/cc-os/docs/takedown/1074-20261017-082123/final-design.json

Approach

PredictionLog owns JSONL per-ruling records (append, session-keyed lookup, atomic depth rewrite), fed by PredictionParser and ReplyDepth, exposed as Thor commands append and score-record in OsMe::CLI; the one non-red test now asserts the specific error message.

Required work:

Write plugins/os-me/lib/os_me/reply_depth.rb
Write plugins/os-me/lib/os_me/cli.rb
Write plugins/os-me/lib/os_me/prediction_parser.rb
Write plugins/os-me/lib/os_me/prediction_log.rb
Write test plugins/os-me/test/prediction_parser_test.rb
Write test plugins/os-me/test/prediction_cli_test.rb

Acceptance criteria

Lettered prediction, 5 rulings: 5 records session:a..e
Unlettered prediction: 1 record session:1
No Confidence line: confidence null
Confidence: 80%: 80 on every record of that prediction
Accept, reply at most 15 words: accepted-verbatim
Longer reply or no --accept: discussed
Unknown id: Thor::Error naming id, exit 1
Id in more than one session, no --session: Thor::Error naming id and sessions, exit 1
Same id repeated within one session: latest one scored
No assistant text: Thor::Error "no assistant message in PATH", exit 1
Missing transcript: ENOENT, loud

Owner sketch

Responsibility	Owner
Split prediction text into rulings (session:a.. or session:1) and read the Confidence percent	OsMe::PredictionParser (new)
Store records as JSONL, resolve id plus optional session, update depth atomically, fail on unknown or ambiguous id	OsMe::PredictionLog (new)
Depth from the reply: accepted-verbatim if accept and at most 15 words, else discussed	OsMe::ReplyDepth (new)
Last assistant text of a transcript	Existing TranscriptReader and Entry#assistant_text
CLI wiring, errors to nonzero exit	OsMe::CLI (`append`, `score-record`)

plugins/os-me/test/prediction_parser_test.rb

# frozen_string_literal: true

require_relative 'test_helper'

class PredictionParserTest < Minitest::Test
  LETTERED = <<~TEXT
    Prediction
    a. Accept the plan
    b. Split the ticket
    c. Skip the hook
    d. Rename the file
    e. Merge it
    Confidence: 80%
  TEXT

  def parse(text)
    OsMe::PredictionParser.new.parse(text)
  end

  def test_lettered_rulings_get_letter_ids
    rows = parse(LETTERED)

    assert_equal(%w[session:a session:b session:c session:d session:e], rows.map { |row| row[:id] })
    assert_equal 'Accept the plan', rows.first[:text]
  end

  def test_confidence_percent_is_stored_as_integer
    assert_equal([80] * 5, parse(LETTERED).map { |row| row[:confidence] })
  end

  def test_unlettered_prediction_is_one_record
    rows = parse("Jared would accept the plan.\nConfidence: 60%")

    assert_equal(['session:1'], rows.map { |row| row[:id] })
    assert_equal 'Jared would accept the plan.', rows.first[:text]
  end

  def test_missing_confidence_is_nil
    assert_nil parse('Jared would accept the plan.').first[:confidence]
  end

  def test_confidence_over_100_raises
    assert_raises(RuntimeError) { parse("a. x\nConfidence: 150%") }
  end
end

plugins/os-me/test/prediction_cli_test.rb

# frozen_string_literal: true

require_relative 'test_helper'

class PredictionCliTest < Minitest::Test
  include TranscriptFixtures

  def setup
    @dir = Dir.mktmpdir
    @log = File.join(@dir, 'log.jsonl')
  end

  def teardown
    FileUtils.remove_entry(@dir)
  end

  def run_cli(*args)
    status = 0
    out, err = capture_io do
      OsMe::CLI.start([*args, '--log', @log])
    rescue SystemExit => e
      status = e.status
    end
    [status, out, err]
  end

  def append(text, session: 'sess-1')
    path = write_transcript(@dir, 'proj', session, [typed('q'), assistant(text)])
    run_cli('append', path)
  end

  def rows
    File.readlines(@log).map { |line| JSON.parse(line) }
  end

  def test_append_lettered_prediction_logs_five_records
    append("a. one\nb. two\nc. three\nd. four\ne. five\nConfidence: 70%")

    assert_equal(%w[session:a session:b session:c session:d session:e], rows.map { |row| row['id'] })
    assert_equal(['sess-1'], rows.map { |row| row['session_id'] }.uniq)
    assert_equal([70], rows.map { |row| row['confidence'] }.uniq)
  end

  def test_append_unlettered_prediction_logs_one_record
    append('Jared would accept.')

    assert_equal(['session:1'], rows.map { |row| row['id'] })
    assert_nil rows.first['confidence']
  end

  def test_append_without_assistant_text_fails_naming_the_cause
    path = write_transcript(@dir, 'proj', 'sess-2', [typed('q')])
    status, _out, err = run_cli('append', path)

    refute_equal 0, status
    assert_includes err, 'no assistant message'
    refute_path_exists @log
  end

  def test_score_short_accept_is_accepted_verbatim
    append('Jared would accept.')
    run_cli('score-record', 'session:1', 'yes, go ahead', '--accept')

    assert_equal 'accepted-verbatim', rows.first['depth']
  end

  def test_score_longer_reply_is_discussed
    append('Jared would accept.')
    run_cli('score-record', 'session:1', "ok but #{(['word'] * 20).join(' ')}", '--accept')

    assert_equal 'discussed', rows.first['depth']
  end

  def test_score_unknown_id_exits_nonzero_naming_the_id
    append('Jared would accept.')
    status, _out, err = run_cli('score-record', 'session:zz', 'yes', '--accept')

    refute_equal 0, status
    assert_includes err, 'session:zz'
    assert_nil rows.first['depth']
  end

  def test_same_id_in_two_sessions_is_ambiguous_without_session
    append('first', session: 'sess-1')
    append('second', session: 'sess-2')
    status, _out, err = run_cli('score-record', 'session:1', 'yes', '--accept')

    refute_equal 0, status
    assert_includes err, 'session:1'
    assert_equal([nil, nil], rows.map { |row| row['depth'] })
  end

  def test_session_option_scores_the_matching_session
    append('first', session: 'sess-1')
    append('second', session: 'sess-2')
    run_cli('score-record', 'session:1', 'yes', '--accept', '--session', 'sess-1')

    assert_equal(['accepted-verbatim', nil], rows.map { |row| row['depth'] })
  end
end

## Problem The LLM hand-writes the rows of review-log.md. Only 18% of the 131 rows name a trigger id. The stated Confidence percent is never stored. Hit rate per trigger, calibration, and trust by origin cannot be computed. ## Design /home/jared/dev/cc-os/docs/takedown/1074-20261017-082123/final-design.json ## Approach PredictionLog owns JSONL per-ruling records (append, session-keyed lookup, atomic depth rewrite), fed by PredictionParser and ReplyDepth, exposed as Thor commands append and score-record in OsMe::CLI; the one non-red test now asserts the specific error message. Required work: 1. Write `plugins/os-me/lib/os_me/reply_depth.rb` 2. Write `plugins/os-me/lib/os_me/cli.rb` 3. Write `plugins/os-me/lib/os_me/prediction_parser.rb` 4. Write `plugins/os-me/lib/os_me/prediction_log.rb` 5. Write test `plugins/os-me/test/prediction_parser_test.rb` 6. Write test `plugins/os-me/test/prediction_cli_test.rb` ## Acceptance criteria - Lettered prediction, 5 rulings: 5 records session:a..e - Unlettered prediction: 1 record session:1 - No Confidence line: confidence null - Confidence: 80%: 80 on every record of that prediction - Accept, reply at most 15 words: accepted-verbatim - Longer reply or no --accept: discussed - Unknown id: Thor::Error naming id, exit 1 - Id in more than one session, no --session: Thor::Error naming id and sessions, exit 1 - Same id repeated within one session: latest one scored - No assistant text: Thor::Error "no assistant message in PATH", exit 1 - Missing transcript: ENOENT, loud ## Owner sketch | Responsibility | Owner | |---|---| | Split prediction text into rulings (session:a.. or session:1) and read the Confidence percent | OsMe::PredictionParser (new) | | Store records as JSONL, resolve id plus optional session, update depth atomically, fail on unknown or ambiguous id | OsMe::PredictionLog (new) | | Depth from the reply: accepted-verbatim if accept and at most 15 words, else discussed | OsMe::ReplyDepth (new) | | Last assistant text of a transcript | Existing TranscriptReader and Entry#assistant_text | | CLI wiring, errors to nonzero exit | OsMe::CLI (`append`, `score-record`) | `plugins/os-me/test/prediction_parser_test.rb` ```ruby # frozen_string_literal: true require_relative 'test_helper' class PredictionParserTest < Minitest::Test LETTERED = <<~TEXT Prediction a. Accept the plan b. Split the ticket c. Skip the hook d. Rename the file e. Merge it Confidence: 80% TEXT def parse(text) OsMe::PredictionParser.new.parse(text) end def test_lettered_rulings_get_letter_ids rows = parse(LETTERED) assert_equal(%w[session:a session:b session:c session:d session:e], rows.map { |row| row[:id] }) assert_equal 'Accept the plan', rows.first[:text] end def test_confidence_percent_is_stored_as_integer assert_equal([80] * 5, parse(LETTERED).map { |row| row[:confidence] }) end def test_unlettered_prediction_is_one_record rows = parse("Jared would accept the plan.\nConfidence: 60%") assert_equal(['session:1'], rows.map { |row| row[:id] }) assert_equal 'Jared would accept the plan.', rows.first[:text] end def test_missing_confidence_is_nil assert_nil parse('Jared would accept the plan.').first[:confidence] end def test_confidence_over_100_raises assert_raises(RuntimeError) { parse("a. x\nConfidence: 150%") } end end ``` `plugins/os-me/test/prediction_cli_test.rb` ```ruby # frozen_string_literal: true require_relative 'test_helper' class PredictionCliTest < Minitest::Test include TranscriptFixtures def setup @dir = Dir.mktmpdir @log = File.join(@dir, 'log.jsonl') end def teardown FileUtils.remove_entry(@dir) end def run_cli(*args) status = 0 out, err = capture_io do OsMe::CLI.start([*args, '--log', @log]) rescue SystemExit => e status = e.status end [status, out, err] end def append(text, session: 'sess-1') path = write_transcript(@dir, 'proj', session, [typed('q'), assistant(text)]) run_cli('append', path) end def rows File.readlines(@log).map { |line| JSON.parse(line) } end def test_append_lettered_prediction_logs_five_records append("a. one\nb. two\nc. three\nd. four\ne. five\nConfidence: 70%") assert_equal(%w[session:a session:b session:c session:d session:e], rows.map { |row| row['id'] }) assert_equal(['sess-1'], rows.map { |row| row['session_id'] }.uniq) assert_equal([70], rows.map { |row| row['confidence'] }.uniq) end def test_append_unlettered_prediction_logs_one_record append('Jared would accept.') assert_equal(['session:1'], rows.map { |row| row['id'] }) assert_nil rows.first['confidence'] end def test_append_without_assistant_text_fails_naming_the_cause path = write_transcript(@dir, 'proj', 'sess-2', [typed('q')]) status, _out, err = run_cli('append', path) refute_equal 0, status assert_includes err, 'no assistant message' refute_path_exists @log end def test_score_short_accept_is_accepted_verbatim append('Jared would accept.') run_cli('score-record', 'session:1', 'yes, go ahead', '--accept') assert_equal 'accepted-verbatim', rows.first['depth'] end def test_score_longer_reply_is_discussed append('Jared would accept.') run_cli('score-record', 'session:1', "ok but #{(['word'] * 20).join(' ')}", '--accept') assert_equal 'discussed', rows.first['depth'] end def test_score_unknown_id_exits_nonzero_naming_the_id append('Jared would accept.') status, _out, err = run_cli('score-record', 'session:zz', 'yes', '--accept') refute_equal 0, status assert_includes err, 'session:zz' assert_nil rows.first['depth'] end def test_same_id_in_two_sessions_is_ambiguous_without_session append('first', session: 'sess-1') append('second', session: 'sess-2') status, _out, err = run_cli('score-record', 'session:1', 'yes', '--accept') refute_equal 0, status assert_includes err, 'session:1' assert_equal([nil, nil], rows.map { |row| row['depth'] }) end def test_session_option_scores_the_matching_session append('first', session: 'sess-1') append('second', session: 'sess-2') run_cli('score-record', 'session:1', 'yes', '--accept', '--session', 'sess-1') assert_equal(['accepted-verbatim', nil], rows.map { |row| row['depth'] }) end end ```

jared added the

project/cc-os

filed-by/user

category/code-change

labels

2026-10-17 11:54:30 +00:00

~~jared referenced this issue 2026-10-17 11:54:35 +00:00~~

os-me: Stop hook appends prediction records; invocation hook records event and arguments #1075

jared referenced this issue

2026-10-17 11:54:36 +00:00

os-me: bin/os-me-report renders prediction statistics #1077

jared referenced this issue

2026-10-17 11:54:37 +00:00

os-me Store: learning loop scores through bin/os-me-record and learns only from discussed sessions #1078

jared referenced this issue

2026-10-17 11:54:38 +00:00

os-me: backfill the 131 review-log rows into predictions.jsonl once #1079

jared referenced this issue

2026-10-17 11:54:38 +00:00

os-me: leaf History row for the record log and a pointer to bin/os-me-report #1080

jared commented

2026-10-17 12:30:20 +00:00

Author

Owner

Work started on branch ticket-1074 via /os-sdlc:implement with map takedown-implement; design from docs/takedown/1074-20261017-082123.

jared referenced this issue

2026-10-17 12:31:36 +00:00

os-me: prediction records carry triggers, verdict, origin, and autopilot #1081

jared commented

2026-10-17 12:34:49 +00:00

Author

Owner

Resolution

Done: OsMe::PredictionParser parses the last assistant message into one record per lettered ruling (id, session_id, text, confidence, depth); OsMe::PredictionLog appends JSONL records and rewrites depth atomically; OsMe::ReplyDepth classifies the next user reply; CLI gains append and score-record; UnknownId and AmbiguousId errors fail loudly.

Evidence: Implementation 53 on map takedown-implement, 8 dispatches, 1 retry, all gates green; commit d720c79a on branch ticket-1074 (merged to main by worktree finish); design docs/takedown/1074-20261017-082123; tests plugins/os-me/test/prediction_parser_test.rb and prediction_cli_test.rb red at base then green.

Follow-ups: #1081 captured: records must also carry triggers, verdict, origin, and autopilot (named by the takedown grade as remaining; blocks #1075 and #1077). Lens line from dispatch 463 under challenge; a Lens proposal ticket follows if justified.

## Resolution **Done:** OsMe::PredictionParser parses the last assistant message into one record per lettered ruling (id, session_id, text, confidence, depth); OsMe::PredictionLog appends JSONL records and rewrites depth atomically; OsMe::ReplyDepth classifies the next user reply; CLI gains append and score-record; UnknownId and AmbiguousId errors fail loudly. **Evidence:** Implementation 53 on map takedown-implement, 8 dispatches, 1 retry, all gates green; commit d720c79a on branch ticket-1074 (merged to main by worktree finish); design docs/takedown/1074-20261017-082123; tests plugins/os-me/test/prediction_parser_test.rb and prediction_cli_test.rb red at base then green. **Follow-ups:** #1081 captured: records must also carry triggers, verdict, origin, and autopilot (named by the takedown grade as remaining; blocks #1075 and #1077). Lens line from dispatch 463 under challenge; a Lens proposal ticket follows if justified.

jared closed this issue

2026-10-17 12:34:49 +00:00

jared referenced this issue

2026-10-17 12:35:59 +00:00

os-me: prediction records carry triggers, verdict, origin, and autopilot #1081

jared referenced this issue

2026-10-17 14:14:15 +00:00

os-me: leaf History row for the record log and a pointer to bin/os-me-report #1080

jared referenced this issue from a commit

2026-10-17 14:47:02 +00:00

docs(takedown): record run 1074-20261017-082123 (final design for #1074)

jared referenced this issue from a commit