jared/cc-os

Watch

Fork 0

decision-system S7b: extract question_answer, correction, direction and over_escalation moments #1277

New issue

Closed

opened 2026-10-23 19:16:42 +00:00 by jared · 2 comments

jared commented

2026-10-23 19:16:42 +00:00

Owner

Problem

MomentExtractor writes only memo_reply moments. It drops each pair that MemoDetector does not find. Question-answer, correction, direction, and over-escalation moments hold tier and ping-cost signals (docs/decision-system/extraction.md, "Lever → training data"), and no run stores them. S7a (#1249) adds MomentKind and ActiveStretch, but no code calls MomentKind yet.

Approach

Ruling (2026-10-23, /os-me:predict 80%, AFK): budget per model call; stop mid-session; no new cursor; ids of classified exchanges, none included, are the resume point.

MomentExtractor sends each new non-memo pair to MomentKind and gives each result to a consumer that writes one DecisionMoment of that kind, none included.

Per pair, in transcript order:

The id (DecisionMoment.id_for(pair)) is in the log already: skip it. No model call.
The calls of this run reached the budget: stop the run here, also in the middle of a session.
MemoDetector#memo? is true: the current memo path (ReplyShape, one call).
Else: MomentKind#classify (one call). Each answer, none included, goes to DecisionMoment.of_kind.

The consumer is one factory, DecisionMoment.of_kind(pair, kind:, split:, extracted_at:). It accepts each key of MomentKind::CRITERIA (the four kinds and none). It fills the same pointer, time, and latency fields as memo_reply (with the S7a ActiveStretch filter), sets kind, and leaves headline as pair.headline. stated_percent, band, outcome, and outcome_p stay null, because these kinds have no recommendation for ReplyShape to read. It raises ArgumentError for a kind that is not a key of MomentKind::CRITERIA, memo_reply included. labeled_by(model) sets the labeler, as for memo moments.

none is recorded as a row with kind none in the log of the session split (moments.jsonl or eval/moments-held-out.jsonl). No consumer output changes: CalibrationReport (and os-me-eval calibration) skips rows without stated_percent; MomentLog reads only ids for known_ids; RowLabels, os-me-eval report, score, exchanges, and backfill do not read the moments log; the ReplyShape eval harness (eval/reply_shape/run.rb) reads its own labels file. Thus known_ids skips each pair already classified, none included, and there is no new cursor object.

The budget counts model calls, not rows: each ReplyShape call and each MomentKind call counts 1, and a none answer counts too. Today extract returns batch.size; it must return the number of calls. The check happens before each call. When the run stops in the middle of a session, it appends the rows of that session so far and does not advance the extract cursor for that session, so the next run reads the session again and makes calls only for pairs whose ids are not in the log. A session advances the cursor only when no pair in it still needs a call. Held-out sessions go only to eval/moments-held-out.jsonl, as today. ActiveStretch does not choose which pairs to classify, and no pair is dropped past a cap.

MomentExtractor.new gets a required moment_kind: keyword. MomentExtractor.build gives MomentKind.new(client: LabelerClient.for(adapter)) with the same Sonnet adapter as ReplyShape. Existing tests that call MomentExtractor.new give a MomentKind on a Decision::Model::Adapters::Fixed adapter. Tests with a fake claude get a kind_answers(choice) helper in GemReplies, next to shape_answers.

Required work:

Send non-memo pairs to MomentKind in the extractor — check: present moment_kind in plugins/os-me/lib/os_me/moment_extractor.rb
Add the consumer factory for the four kinds and none — check: present def self.of_kind in plugins/os-me/lib/os_me/decision_moment.rb
Add the fake-reply helper for kind answers — check: present def kind_answers in plugins/os-me/test/gem_replies.rb

The fixture pairs are the five pairs in the S7a (#1249) Approach table. The tests put each pair at ai_at 2026-10-01T10:00:00Z and reply_at 2026-10-01T10:05:00Z unless a criterion gives other times. New tests go only under plugins/os-me/test.

Acceptance criteria

Given a Store and, for each of the four kinds question_answer, correction, direction, over_escalation, a train session whose only pair is that kind's fixture pair, and a fake claude on PATH that answers kind with that kind, when os-me-extract --store <store> --transcript-root <root> --budget 20 runs, then it exits 0 and moments.jsonl holds exactly one row for that session with kind equal to that kind, id "<session_id>:0", split train, stated_percent null, outcome null, latency_s 300, and labeler.model sonnet.
Given a train session whose only pair is the none fixture pair and a fake claude that answers kind none, when os-me-extract --budget 20 runs, then the fake claude is called exactly once, moments.jsonl holds exactly one row with kind none, id "<session_id>:0", stated_percent null, and outcome null, and queue/extract-cursor.json holds that session id.
Given a train session whose only pair is the S2 fixture memo with reply yes, go ahead, and a fake claude that answers shape accept, when os-me-extract --budget 20 runs, then the fake claude is called exactly once and moments.jsonl holds one row with kind memo_reply and outcome accept.
Given a train session with the S2 fixture memo pair first and the correction fixture pair second, and a fake claude that answers shape accept on call 1 and kind correction on call 2, when os-me-extract --budget 20 runs, then moments.jsonl holds two rows whose kind values are ['memo_reply', 'correction'] in that order.
Given three train sessions, each with two pairs of the none fixture, and a fake claude that always answers kind none, when os-me-extract --budget 2 runs, then the fake claude is called exactly 2 times, moments.jsonl holds exactly 2 rows, both with kind none, and queue/extract-cursor.json holds exactly 1 session id.
Given one train session with the correction, none, and direction fixture pairs in that order, and a fake claude that answers kind correction on call 1, none on call 2, and direction on call 3, when os-me-extract --budget 2 runs, then the fake claude is called exactly 2 times, moments.jsonl holds two rows whose kind values are ['correction', 'none'] in that order, and queue/extract-cursor.json has no entry for that session id.
Given the Store and transcript after criterion 6, with the transcript unchanged, and a fake claude that answers kind direction, when os-me-extract --budget 20 runs again, then the fake claude is called exactly once in that run, moments.jsonl holds three rows whose kind values are ['correction', 'none', 'direction'] in that order, and queue/extract-cursor.json holds that session id.
Given a held-out session (held_out_id) whose only pair is the correction fixture pair and a fake claude that answers kind correction, when os-me-extract --budget 20 runs, then eval/moments-held-out.jsonl holds one row with that session id, kind correction, and split held_out, and no row in moments.jsonl has that session id.
Given a train session whose only pair is the correction fixture pair with ai_at 2026-10-01T10:00:00Z and reply_at 2026-10-01T12:00:00Z, when os-me-extract --budget 20 runs, then its row has latency_s null and reply_chars 59.
Given the correction fixture pair, when OsMe::DecisionMoment.of_kind(pair, kind: 'memo_reply', split: 'train', extracted_at: '2026-10-01T13:00:00Z') runs, then it raises ArgumentError; with kind: 'none' it returns a moment with kind none and outcome nil.

Owner sketch

module OsMe
  class DecisionMoment
    def self.of_kind(pair, kind:, split:, extracted_at:)
      raise ArgumentError, "unknown moment kind: #{kind}" unless MomentKind::CRITERIA.key?(kind)
      # same pointer, time, and ActiveStretch-filtered latency fields as memo_reply; label fields nil
    end
  end

  class MomentExtractor
    def initialize(catalog:, store_path:, reply_shape:, moment_kind:, split: Split.new, labeler_model: nil); end
    # extract counts each ReplyShape and MomentKind call, stops before the call that would pass the budget,
    # and reports if the session finished, so the cursor advances only for a finished session
  end
end

Non-goals

Tuning the MomentKind criteria. The S7 gate is orchestrator work: an agent checks a sample of each new kind from train sessions only, and each kind must reach 80% agreement.
ReplyShape, AnswerSource, or WrongCost::Assessor labels on the new kinds.
A text pre-filter before MomentKind.
A new per-exchange cursor object, a cap on pairs per session, and use of ActiveStretch to choose pairs.
Changes to CalibrationReport. It skips rows without stated_percent, so the new kinds and none rows do not change its output.
Demotion (S9).

Blocked by: #1249

Origin

Trigger: User asked for the decision-system build tickets this session; S7 is split into S7a (#1249) and S7b (this ticket) because it needs more than 10 acceptance criteria
Improvised this session: none
Chain: no ticket for this stage ← the build plan names the stage and the user asked for the tickets. Stop at DESIGN (docs/decision-system/build-plan.md S7).
Root candidate: none found; searched MomentKind, moment kinds, question_answer, over_escalation, consumer, os-me
Where: docs/decision-system/build-plan.md S7; docs/decision-system/extraction.md; plugins/os-me/lib/os_me/moment_extractor.rb; plugins/os-me/lib/os_me/decision_moment.rb
Session: 4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-ticket-1250/4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5.jsonl

VERDICT: CREATE (user-approved; backlog search only)

## Problem `MomentExtractor` writes only `memo_reply` moments. It drops each pair that `MemoDetector` does not find. Question-answer, correction, direction, and over-escalation moments hold tier and ping-cost signals (`docs/decision-system/extraction.md`, "Lever → training data"), and no run stores them. S7a (#1249) adds `MomentKind` and `ActiveStretch`, but no code calls `MomentKind` yet. ## Approach Ruling (2026-10-23, /os-me:predict 80%, AFK): budget per model call; stop mid-session; no new cursor; ids of classified exchanges, `none` included, are the resume point. `MomentExtractor` sends each new non-memo pair to `MomentKind` and gives each result to a consumer that writes one `DecisionMoment` of that kind, `none` included. Per pair, in transcript order: - The id (`DecisionMoment.id_for(pair)`) is in the log already: skip it. No model call. - The calls of this run reached the budget: stop the run here, also in the middle of a session. - `MemoDetector#memo?` is true: the current memo path (`ReplyShape`, one call). - Else: `MomentKind#classify` (one call). Each answer, `none` included, goes to `DecisionMoment.of_kind`. The consumer is one factory, `DecisionMoment.of_kind(pair, kind:, split:, extracted_at:)`. It accepts each key of `MomentKind::CRITERIA` (the four kinds and `none`). It fills the same pointer, time, and latency fields as `memo_reply` (with the S7a `ActiveStretch` filter), sets `kind`, and leaves `headline` as `pair.headline`. `stated_percent`, `band`, `outcome`, and `outcome_p` stay null, because these kinds have no recommendation for `ReplyShape` to read. It raises `ArgumentError` for a kind that is not a key of `MomentKind::CRITERIA`, `memo_reply` included. `labeled_by(model)` sets the labeler, as for memo moments. `none` is recorded as a row with `kind` `none` in the log of the session split (`moments.jsonl` or `eval/moments-held-out.jsonl`). No consumer output changes: `CalibrationReport` (and `os-me-eval calibration`) skips rows without `stated_percent`; `MomentLog` reads only ids for `known_ids`; `RowLabels`, `os-me-eval report`, `score`, `exchanges`, and `backfill` do not read the moments log; the `ReplyShape` eval harness (`eval/reply_shape/run.rb`) reads its own labels file. Thus `known_ids` skips each pair already classified, `none` included, and there is no new cursor object. The budget counts model calls, not rows: each `ReplyShape` call and each `MomentKind` call counts 1, and a `none` answer counts too. Today `extract` returns `batch.size`; it must return the number of calls. The check happens before each call. When the run stops in the middle of a session, it appends the rows of that session so far and does not advance the extract cursor for that session, so the next run reads the session again and makes calls only for pairs whose ids are not in the log. A session advances the cursor only when no pair in it still needs a call. Held-out sessions go only to `eval/moments-held-out.jsonl`, as today. `ActiveStretch` does not choose which pairs to classify, and no pair is dropped past a cap. `MomentExtractor.new` gets a required `moment_kind:` keyword. `MomentExtractor.build` gives `MomentKind.new(client: LabelerClient.for(adapter))` with the same Sonnet adapter as `ReplyShape`. Existing tests that call `MomentExtractor.new` give a `MomentKind` on a `Decision::Model::Adapters::Fixed` adapter. Tests with a fake `claude` get a `kind_answers(choice)` helper in `GemReplies`, next to `shape_answers`. Required work: 1. Send non-memo pairs to `MomentKind` in the extractor — check: present `moment_kind` in `plugins/os-me/lib/os_me/moment_extractor.rb` 2. Add the consumer factory for the four kinds and `none` — check: present `def self.of_kind` in `plugins/os-me/lib/os_me/decision_moment.rb` 3. Add the fake-reply helper for kind answers — check: present `def kind_answers` in `plugins/os-me/test/gem_replies.rb` The fixture pairs are the five pairs in the S7a (#1249) Approach table. The tests put each pair at `ai_at` `2026-10-01T10:00:00Z` and `reply_at` `2026-10-01T10:05:00Z` unless a criterion gives other times. New tests go only under `plugins/os-me/test`. ## Acceptance criteria 1. Given a Store and, for each of the four kinds `question_answer`, `correction`, `direction`, `over_escalation`, a train session whose only pair is that kind's fixture pair, and a fake `claude` on `PATH` that answers `kind` with that kind, when `os-me-extract --store <store> --transcript-root <root> --budget 20` runs, then it exits 0 and `moments.jsonl` holds exactly one row for that session with `kind` equal to that kind, `id` `"<session_id>:0"`, `split` `train`, `stated_percent` `null`, `outcome` `null`, `latency_s` `300`, and `labeler.model` `sonnet`. 2. Given a train session whose only pair is the `none` fixture pair and a fake `claude` that answers `kind` `none`, when `os-me-extract --budget 20` runs, then the fake `claude` is called exactly once, `moments.jsonl` holds exactly one row with `kind` `none`, `id` `"<session_id>:0"`, `stated_percent` `null`, and `outcome` `null`, and `queue/extract-cursor.json` holds that session id. 3. Given a train session whose only pair is the S2 fixture memo with reply `yes, go ahead`, and a fake `claude` that answers shape `accept`, when `os-me-extract --budget 20` runs, then the fake `claude` is called exactly once and `moments.jsonl` holds one row with `kind` `memo_reply` and `outcome` `accept`. 4. Given a train session with the S2 fixture memo pair first and the `correction` fixture pair second, and a fake `claude` that answers shape `accept` on call 1 and `kind` `correction` on call 2, when `os-me-extract --budget 20` runs, then `moments.jsonl` holds two rows whose `kind` values are `['memo_reply', 'correction']` in that order. 5. Given three train sessions, each with two pairs of the `none` fixture, and a fake `claude` that always answers `kind` `none`, when `os-me-extract --budget 2` runs, then the fake `claude` is called exactly 2 times, `moments.jsonl` holds exactly 2 rows, both with `kind` `none`, and `queue/extract-cursor.json` holds exactly 1 session id. 6. Given one train session with the `correction`, `none`, and `direction` fixture pairs in that order, and a fake `claude` that answers `kind` `correction` on call 1, `none` on call 2, and `direction` on call 3, when `os-me-extract --budget 2` runs, then the fake `claude` is called exactly 2 times, `moments.jsonl` holds two rows whose `kind` values are `['correction', 'none']` in that order, and `queue/extract-cursor.json` has no entry for that session id. 7. Given the Store and transcript after criterion 6, with the transcript unchanged, and a fake `claude` that answers `kind` `direction`, when `os-me-extract --budget 20` runs again, then the fake `claude` is called exactly once in that run, `moments.jsonl` holds three rows whose `kind` values are `['correction', 'none', 'direction']` in that order, and `queue/extract-cursor.json` holds that session id. 8. Given a held-out session (`held_out_id`) whose only pair is the `correction` fixture pair and a fake `claude` that answers `kind` `correction`, when `os-me-extract --budget 20` runs, then `eval/moments-held-out.jsonl` holds one row with that session id, `kind` `correction`, and `split` `held_out`, and no row in `moments.jsonl` has that session id. 9. Given a train session whose only pair is the `correction` fixture pair with `ai_at` `2026-10-01T10:00:00Z` and `reply_at` `2026-10-01T12:00:00Z`, when `os-me-extract --budget 20` runs, then its row has `latency_s` `null` and `reply_chars` `59`. 10. Given the `correction` fixture pair, when `OsMe::DecisionMoment.of_kind(pair, kind: 'memo_reply', split: 'train', extracted_at: '2026-10-01T13:00:00Z')` runs, then it raises `ArgumentError`; with `kind: 'none'` it returns a moment with `kind` `none` and `outcome` `nil`. ## Owner sketch ```ruby module OsMe class DecisionMoment def self.of_kind(pair, kind:, split:, extracted_at:) raise ArgumentError, "unknown moment kind: #{kind}" unless MomentKind::CRITERIA.key?(kind) # same pointer, time, and ActiveStretch-filtered latency fields as memo_reply; label fields nil end end class MomentExtractor def initialize(catalog:, store_path:, reply_shape:, moment_kind:, split: Split.new, labeler_model: nil); end # extract counts each ReplyShape and MomentKind call, stops before the call that would pass the budget, # and reports if the session finished, so the cursor advances only for a finished session end end ``` ## Non-goals - Tuning the `MomentKind` criteria. The S7 gate is orchestrator work: an agent checks a sample of each new kind from train sessions only, and each kind must reach 80% agreement. - `ReplyShape`, `AnswerSource`, or `WrongCost::Assessor` labels on the new kinds. - A text pre-filter before `MomentKind`. - A new per-exchange cursor object, a cap on pairs per session, and use of `ActiveStretch` to choose pairs. - Changes to `CalibrationReport`. It skips rows without `stated_percent`, so the new kinds and `none` rows do not change its output. - Demotion (S9). Blocked by: #1249 ## Origin - Trigger: User asked for the decision-system build tickets this session; S7 is split into S7a (#1249) and S7b (this ticket) because it needs more than 10 acceptance criteria - Improvised this session: none - Chain: no ticket for this stage ← the build plan names the stage and the user asked for the tickets. Stop at `DESIGN` (`docs/decision-system/build-plan.md` S7). - Root candidate: none found; searched MomentKind, moment kinds, question_answer, over_escalation, consumer, os-me - Where: docs/decision-system/build-plan.md S7; docs/decision-system/extraction.md; plugins/os-me/lib/os_me/moment_extractor.rb; plugins/os-me/lib/os_me/decision_moment.rb - Session: 4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5 - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-ticket-1250/4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5.jsonl VERDICT: CREATE (user-approved; backlog search only)

jared added the

project/cc-os

filed-by/user

category/code-change

labels

2026-10-23 19:16:42 +00:00

jared commented

2026-10-23 19:16:49 +00:00

Author

Owner

Work started on branch ticket-1277 (os-sdlc implement, S7b).

jared commented

2026-10-23 19:30:33 +00:00

Author

Owner

Resolution

Done: MomentExtractor classifies each non-memo exchange with MomentKind and writes one row per exchange, none included. The budget counts model calls, is checked per call, and a partial session leaves the cursor in place.

Evidence: Commits 0d21c538, 0551d5f3, 06668695 on ticket-1277. Suite: 285 runs, 0 failures. Rubocop clean. Opus design audit: pass; F6 fixed. Hand-finished after escalation 68 (test defects); implementation 138 abandoned with that reason.

Follow-ups: S7 gate (80% agreement on a checked sample of each new kind) is recorded as open in the build-plan S7 row. Dropped: uncounted calls when a transcript vanishes after classification (pre-existing, rare, overrun is at most one session). Dropped: pair-labeler extraction from MomentExtractor (audit F8 says acceptable at current size).

## Resolution **Done:** MomentExtractor classifies each non-memo exchange with MomentKind and writes one row per exchange, none included. The budget counts model calls, is checked per call, and a partial session leaves the cursor in place. **Evidence:** Commits 0d21c538, 0551d5f3, 06668695 on ticket-1277. Suite: 285 runs, 0 failures. Rubocop clean. Opus design audit: pass; F6 fixed. Hand-finished after escalation 68 (test defects); implementation 138 abandoned with that reason. **Follow-ups:** S7 gate (80% agreement on a checked sample of each new kind) is recorded as open in the build-plan S7 row. Dropped: uncounted calls when a transcript vanishes after classification (pre-existing, rare, overrun is at most one session). Dropped: pair-labeler extraction from MomentExtractor (audit F8 says acceptable at current size).

jared closed this issue

2026-10-23 19:30:33 +00:00

jared referenced this issue from a commit

2026-10-23 19:31:38 +00:00

feat(os-me): Classify non-memo exchanges with MomentKind under a per-call budget (#1277)

jared referenced this issue from a commit

2026-10-23 19:31:38 +00:00

test(os-me): Share train_ids in the test helper; drop a repeated cursor assertion (#1277 audit F6)

jared referenced this issue from a commit

2026-10-23 19:31:38 +00:00

docs(decision-system): S7 built, gate open (#1277)