jared/cc-os

Watch

Fork 0

S10 T2: label-agreement eval for AnswerSource and WrongCost::Assessor #1285

New issue

Closed

opened 2026-10-23 20:25:36 +00:00 by jared · 2 comments

jared commented

2026-10-23 20:25:36 +00:00

Owner

S10 T2: label-agreement eval for AnswerSource and WrongCost::Assessor

Problem

S10 tunes two row levers, and nothing measures them. OsMe::AnswerSource (plugins/os-me/lib/os_me/answer_source.rb) asks one choice question with 5 CRITERIA lines. OsMe::WrongCost::Assessor (plugins/os-me/lib/os_me/wrong_cost/assessor.rb) asks three yes/no (noul) questions from its INSTRUCTIONS. An S10 iteration changes one of those lines. Without a harness, no iteration can be kept or discarded on a number, and no baseline exists.

The source, hard_to_undo, compounds, and stays_hidden fields in ~/dev/os-me-store/moments.jsonl are null on all 355 rows, so no label exists to compare with. A separate labeling agent writes one label file for both levers. This ticket adds the harness that reads that file.

Approach

One harness with a lever switch. One script, plugins/os-me/eval/row_labels/run.rb --lever source|wrong_cost, over shared lib objects. Reason: both levers take the same input (OsMe::Row.new(text:), the same call OsMe::RowLabels.of makes), read the same label file, and use the same metrics, reps, and threads. The only differences are the questions, the label keys, and how to read the answer. Two small lever-check objects hold those differences. Two scripts would repeat the option parsing, the rep loop, and the output.

The harness runs the real lever. It calls AnswerSource#classify and WrongCost::Assessor#assess, so a tuning iteration edits AnswerSource::CRITERIA or Assessor::INSTRUCTIONS in lib and runs again. There is no candidate YAML, so the tuned text does not have to be copied back into lib later (the reply_shape/tuned.yaml step).

Label file. ~/dev/os-me-store/samples/row-labels-YYYY-MM-DD.jsonl, not committed. One JSONL row per memo, with exactly these keys. One labeling pass labels both levers.

Key	Type	Meaning
`id`	string	`<session_id>:<ai_index>`, the `DecisionMoment.id_for` form, unique in the file
`split`	`"tune"` or `"holdout"`	Must equal `OsMe::TuneSplit.new.holdout?(id) ? "holdout" : "tune"`
`text`	string	The `Entry#assistant_text` of the memo line, verbatim. The harness sends it as `Row#text` and reads no transcript.
`sure`	boolean	`true` when the labeler is sure of all four answers
`source`	string	One of `rule`, `precedent`, `facts`, `experiment`, `taste` (the `AnswerSource::CRITERIA` keys)
`hard_to_undo`	boolean	Checked answer to the `hard_to_undo` instruction
`compounds`	boolean	Checked answer to the `compounds` instruction
`stays_hidden`	boolean	Checked answer to the `stays_hidden` instruction

Malformed rows abort the run. A row is malformed when it is not a JSON object, its keys are not exactly the eight above, id is not a string or repeats an earlier row, split differs from TuneSplit, text is not a string, source is not one of the five values, or sure, hard_to_undo, compounds, or stays_hidden is not true or false. The error names the line number.

Shared read. ConfidenceLabel.read already numbers lines, refuses held-out sessions in a tune run, and selects the split. Move that loop and the refusal into OsMe::LabelFile, and call it from ConfidenceLabel.read and LeverLabel.read. Each label class keeps its own row checks. The ConfidenceLabel tests stay unchanged and green. Without this step, the second label class repeats those lines.

Held-out refusal. The same as ConfidenceLabel. With --split tune, a row whose session OsMe::Split#held_out? is true aborts the run, and the error names its id. With --split holdout, the run accepts it. After the checks, only rows of the requested split remain.

Questions per lever. The answers are strings, so one comparison serves both levers.

Lever	Question	Classes	Label	Predicted
`source`	`source`	`rule precedent facts experiment taste`	`source`	`classify(row).source.to_s`
`wrong_cost`	`hard_to_undo`, `compounds`, `stays_hidden`	`true false`	the boolean `.to_s`	`assess(row)` field `.to_s`
`wrong_cost`	`level`	`low medium high`	`WrongCost.new(**labels).level.to_s`	`assess(row).level.to_s`

One model call per row and rep, for each lever. When the call raises Decision::Model::ResponseError, every question of that lever gets the prediction unreadable for that row and rep. An unreadable answer is a miss.

Metrics per question, over all reps of the requested split:

n = rows x reps. agreement = matching answers / n.
kappa = Cohen's kappa over the same n pairs, with the expected agreement summed over the question's classes (as in reply_shape/run.rb). An unreadable prediction is in no class. When the expected agreement is 1.0, kappa is nil and prints n/a.
unreadable = count of unreadable predictions.
majority = rows whose most common prediction over the reps equals the label / rows. A tie is a miss.
sure_n and sure_agreement = n and agreement on the rows with sure: true.
Percents print as %.1f%%, kappa as %.3f, and a zero denominator as n/a.

Output. One stdout line per question, in the order of the table above, for example:
split=tune lever=source question=source rows=2 reps=3 n=6 agreement=33.3% kappa=-0.143 unreadable=1 majority=50.0% sure_n=3 sure_agreement=66.7%
and a TSV at --out with the header id rep question label predicted, one row per answer, sorted by id, then rep, then question order. Flags: --lever, --labels, --split, --out (required), --reps (default 3), --jobs (default 8), --model (default OsMe::MomentExtractor::LABELER_MODEL). The script builds the client with OsMe::LabelerClient.for(OsMe::HeadlessClaude.new(runner: OsMe::HeadlessRunner.new, model:)). Bad arguments, an unknown lever, a malformed row, or a held-out refusal print error: <message> on stderr and exit 2, before any model call.

Required work:

Add the script that parses the flags, reads the labels, runs the chosen lever with LeverRun, writes the TSV, and prints one AgreementScore#summary line per question. check: present LeverCheck.for in plugins/os-me/eval/row_labels/run.rb
Add a README beside the script with the run command, the label row keys, the question table, the metric definitions, the cost (one Sonnet call per row and rep), and this rule: the holdout split is read only when a tune change is a candidate to keep. check: present stays_hidden in plugins/os-me/eval/row_labels/README.md
Make ConfidenceLabel.read call OsMe::LabelFile for the line loop, the held-out refusal, and the split selection. check: present LabelFile in plugins/os-me/lib/os_me/confidence_label.rb

Acceptance criteria

Each test writes its label file to a Dir.mktmpdir directory. It builds ids from generated session ids so that OsMe::Split and OsMe::TuneSplit give the state the test needs (as confidence_label_fixtures.rb does). The client is OsMe::LabelerClient.for(Decision::Model::Adapters::Fixed.new(...)). No test calls a model or the network.

Given a file with two valid tune rows and one valid holdout row, all from train sessions, When OsMe::LeverLabel.read(path, split: 'holdout') runs, Then it returns only the holdout row, and its id, text, sure, source, hard_to_undo, compounds, and stays_hidden equal the row values.
Given a file whose second line has a missing sure key, or source "guess", or hard_to_undo "yes", or a split that differs from TuneSplit, or an id that repeats line 1, When OsMe::LeverLabel.read runs, Then it raises OsMe::Error whose message holds line 2.
Given a file with one row whose session OsMe::Split#held_out? is true, When LeverLabel.read(path, split: 'tune') runs, Then it raises OsMe::Error whose message holds that row's id; and When LeverLabel.read(path, split: 'holdout') runs on the same file, Then it does not raise.
Given two tune rows labeled source facts and taste, and a Fixed adapter that answers source with choice facts, When OsMe::LeverRun.new(OsMe::LeverCheck.for('source', client), reps: 1, jobs: 1).results(labels) is scored by OsMe::AgreementScore for the question source, Then agreement is 0.5, kappa is 0.0, and unreadable is 0.
Given two tune rows labeled (hard_to_undo, compounds, stays_hidden) = (true, false, true) and (false, false, false), and a Fixed adapter that answers noul 0.9, 0.2, and 0.9, When the wrong_cost lever runs with reps 1, Then agreement is 0.5 for hard_to_undo, stays_hidden, and level, agreement is 1.0 for compounds, and kappa for compounds is nil.
Given one tune row and a Fixed adapter with no answers (so the client raises Decision::Model::ResponseError), When the wrong_cost lever runs with reps 2, Then each of the four questions has 2 results, all with the prediction unreadable, and the AgreementScore of each has unreadable 2 and agreement 0.0.
Given one tune row and the source lever with reps 3, When LeverRun#results runs with jobs 2, Then it returns 3 results whose rep values are 1, 2, and 3.
Given hand-built source results: row A (sure: true, label facts) predicted facts, facts, taste in reps 1 to 3, and row B (sure: false, label taste) predicted facts, unreadable, facts, When AgreementScore#summary(split: 'tune', lever: 'source') is called, Then it returns split=tune lever=source question=source rows=2 reps=3 n=6 agreement=33.3% kappa=-0.143 unreadable=1 majority=50.0% sure_n=3 sure_agreement=66.7%.
Given the lever name cost, When OsMe::LeverCheck.for('cost', client) runs, Then it raises OsMe::Error whose message holds cost.
Given a label file with a malformed row, or the flag --lever cost with a valid file, When ruby -I plugins/os-me/lib plugins/os-me/eval/row_labels/run.rb --lever LEVER --labels FILE --split tune --out OUT runs, Then it exits 2 and stderr starts with error:.

Owner sketch

module OsMe
  class LeverLabel < Data.define(:id, :split, :text, :sure, :source, :hard_to_undo, :compounds, :stays_hidden)
    SOURCES = AnswerSource::CRITERIA.keys.map(&:to_s).freeze

    def self.read(path, split:, sessions: Split.new, tune_split: TuneSplit.new)
      LabelFile.new(path, sessions: sessions).read(split) { |line, seen_ids| parse(line, tune_split, seen_ids) }
    end

    def session_id = id.rpartition(':').first
    def row = Row.new(text: text)
  end

  class LeverCheck
    def self.for(name, client)
      { 'source' => Source, 'wrong_cost' => Cost }.fetch(name) { raise Error, "unknown lever #{name}" }.new(client)
    end

    class Source
      QUESTIONS = { 'source' => AnswerSource::CRITERIA.keys.map(&:to_s) }.freeze

      def initialize(client) = @client = client
      def name = 'source'
      def expected(label) = { 'source' => label.source }
      def predicted(label) = { 'source' => AnswerSource.new(@client).classify(label.row).source.to_s }
    end

    class Cost
      def predicted(label)
        cost = OsMe::WrongCost::Assessor.new(@client).assess(label.row)
        cost.to_h.transform_keys(&:to_s).transform_values(&:to_s).merge('level' => cost.level.to_s)
      end
    end
  end

  LeverResult = Data.define(:id, :rep, :question, :label, :predicted, :sure)

  class LeverRun
    def initialize(check, reps:, jobs:)
    def results(labels) # one LeverResult per label, rep, and question; ResponseError -> 'unreadable'
  end

  class AgreementScore
    def initialize(results, question:, classes:)
    def agreement; def kappa; def unreadable; def majority
    def summary(split:, lever:)
  end
end

Scope and blocking

In scope: OsMe::LeverLabel, OsMe::LabelFile, OsMe::LeverCheck (with Source and Cost), OsMe::LeverResult, OsMe::LeverRun, OsMe::AgreementScore, the move of the shared read out of ConfidenceLabel, plugins/os-me/eval/row_labels/{run.rb,README.md}, and tests under plugins/os-me/test.
Out of scope: changes to AnswerSource::CRITERIA or Assessor::INSTRUCTIONS (the S10 tuning loop's job after this lands); the label file, which a separate agent writes in the Store; a baseline run; changes to reply_shape/run.rb; a reversal-rate signal; .gitignore changes in the Store.
Blocked by: nothing. The tuning loops for both levers are blocked by this ticket and by the label file.

Origin

Trigger: S10 plan (AFK decision-system build): AnswerSource and WrongCost::Assessor have no checked labels and no harness, so S10 cannot tune them
Improvised this session: none
Chain: S10 cannot tune AnswerSource or WrongCost ← no harness scores them against labels ← S4 built the levers without an eval. Stop at DESIGN (docs/decision-system/build-plan.md S10).
Root candidate: none found; searched AnswerSource eval, WrongCost eval, reply_shape/run.rb
Where: plugins/os-me/eval/; plugins/os-me/lib/os_me/answer_source.rb; plugins/os-me/lib/os_me/wrong_cost/
Session: 4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5.jsonl

VERDICT: CREATE (user-approved; backlog search only)

# S10 T2: label-agreement eval for AnswerSource and WrongCost::Assessor ## Problem S10 tunes two row levers, and nothing measures them. `OsMe::AnswerSource` (`plugins/os-me/lib/os_me/answer_source.rb`) asks one choice question with 5 `CRITERIA` lines. `OsMe::WrongCost::Assessor` (`plugins/os-me/lib/os_me/wrong_cost/assessor.rb`) asks three yes/no (`noul`) questions from its `INSTRUCTIONS`. An S10 iteration changes one of those lines. Without a harness, no iteration can be kept or discarded on a number, and no baseline exists. The `source`, `hard_to_undo`, `compounds`, and `stays_hidden` fields in `~/dev/os-me-store/moments.jsonl` are `null` on all 355 rows, so no label exists to compare with. A separate labeling agent writes one label file for both levers. This ticket adds the harness that reads that file. ## Approach **One harness with a lever switch.** One script, `plugins/os-me/eval/row_labels/run.rb --lever source|wrong_cost`, over shared lib objects. Reason: both levers take the same input (`OsMe::Row.new(text:)`, the same call `OsMe::RowLabels.of` makes), read the same label file, and use the same metrics, reps, and threads. The only differences are the questions, the label keys, and how to read the answer. Two small lever-check objects hold those differences. Two scripts would repeat the option parsing, the rep loop, and the output. **The harness runs the real lever.** It calls `AnswerSource#classify` and `WrongCost::Assessor#assess`, so a tuning iteration edits `AnswerSource::CRITERIA` or `Assessor::INSTRUCTIONS` in lib and runs again. There is no candidate YAML, so the tuned text does not have to be copied back into lib later (the `reply_shape/tuned.yaml` step). **Label file.** `~/dev/os-me-store/samples/row-labels-YYYY-MM-DD.jsonl`, not committed. One JSONL row per memo, with exactly these keys. One labeling pass labels both levers. | Key | Type | Meaning | |---|---|---| | `id` | string | `<session_id>:<ai_index>`, the `DecisionMoment.id_for` form, unique in the file | | `split` | `"tune"` or `"holdout"` | Must equal `OsMe::TuneSplit.new.holdout?(id) ? "holdout" : "tune"` | | `text` | string | The `Entry#assistant_text` of the memo line, verbatim. The harness sends it as `Row#text` and reads no transcript. | | `sure` | boolean | `true` when the labeler is sure of all four answers | | `source` | string | One of `rule`, `precedent`, `facts`, `experiment`, `taste` (the `AnswerSource::CRITERIA` keys) | | `hard_to_undo` | boolean | Checked answer to the `hard_to_undo` instruction | | `compounds` | boolean | Checked answer to the `compounds` instruction | | `stays_hidden` | boolean | Checked answer to the `stays_hidden` instruction | **Malformed rows abort the run.** A row is malformed when it is not a JSON object, its keys are not exactly the eight above, `id` is not a string or repeats an earlier row, `split` differs from `TuneSplit`, `text` is not a string, `source` is not one of the five values, or `sure`, `hard_to_undo`, `compounds`, or `stays_hidden` is not `true` or `false`. The error names the line number. **Shared read.** `ConfidenceLabel.read` already numbers lines, refuses held-out sessions in a tune run, and selects the split. Move that loop and the refusal into `OsMe::LabelFile`, and call it from `ConfidenceLabel.read` and `LeverLabel.read`. Each label class keeps its own row checks. The `ConfidenceLabel` tests stay unchanged and green. Without this step, the second label class repeats those lines. **Held-out refusal.** The same as `ConfidenceLabel`. With `--split tune`, a row whose session `OsMe::Split#held_out?` is true aborts the run, and the error names its `id`. With `--split holdout`, the run accepts it. After the checks, only rows of the requested split remain. **Questions per lever.** The answers are strings, so one comparison serves both levers. | Lever | Question | Classes | Label | Predicted | |---|---|---|---|---| | `source` | `source` | `rule precedent facts experiment taste` | `source` | `classify(row).source.to_s` | | `wrong_cost` | `hard_to_undo`, `compounds`, `stays_hidden` | `true false` | the boolean `.to_s` | `assess(row)` field `.to_s` | | `wrong_cost` | `level` | `low medium high` | `WrongCost.new(**labels).level.to_s` | `assess(row).level.to_s` | One model call per row and rep, for each lever. When the call raises `Decision::Model::ResponseError`, every question of that lever gets the prediction `unreadable` for that row and rep. An unreadable answer is a miss. **Metrics per question,** over all reps of the requested split: - `n` = rows x reps. `agreement` = matching answers / n. - `kappa` = Cohen's kappa over the same n pairs, with the expected agreement summed over the question's classes (as in `reply_shape/run.rb`). An `unreadable` prediction is in no class. When the expected agreement is 1.0, kappa is `nil` and prints `n/a`. - `unreadable` = count of `unreadable` predictions. - `majority` = rows whose most common prediction over the reps equals the label / rows. A tie is a miss. - `sure_n` and `sure_agreement` = n and agreement on the rows with `sure: true`. - Percents print as `%.1f%%`, kappa as `%.3f`, and a zero denominator as `n/a`. **Output.** One stdout line per question, in the order of the table above, for example: `split=tune lever=source question=source rows=2 reps=3 n=6 agreement=33.3% kappa=-0.143 unreadable=1 majority=50.0% sure_n=3 sure_agreement=66.7%` and a TSV at `--out` with the header `id rep question label predicted`, one row per answer, sorted by `id`, then `rep`, then question order. Flags: `--lever`, `--labels`, `--split`, `--out` (required), `--reps` (default 3), `--jobs` (default 8), `--model` (default `OsMe::MomentExtractor::LABELER_MODEL`). The script builds the client with `OsMe::LabelerClient.for(OsMe::HeadlessClaude.new(runner: OsMe::HeadlessRunner.new, model:))`. Bad arguments, an unknown lever, a malformed row, or a held-out refusal print `error: <message>` on stderr and exit 2, before any model call. Required work: 1. Add the script that parses the flags, reads the labels, runs the chosen lever with `LeverRun`, writes the TSV, and prints one `AgreementScore#summary` line per question. check: present `LeverCheck.for` in `plugins/os-me/eval/row_labels/run.rb` 2. Add a README beside the script with the run command, the label row keys, the question table, the metric definitions, the cost (one Sonnet call per row and rep), and this rule: the holdout split is read only when a tune change is a candidate to keep. check: present `stays_hidden` in `plugins/os-me/eval/row_labels/README.md` 3. Make `ConfidenceLabel.read` call `OsMe::LabelFile` for the line loop, the held-out refusal, and the split selection. check: present `LabelFile` in `plugins/os-me/lib/os_me/confidence_label.rb` ## Acceptance criteria Each test writes its label file to a `Dir.mktmpdir` directory. It builds ids from generated session ids so that `OsMe::Split` and `OsMe::TuneSplit` give the state the test needs (as `confidence_label_fixtures.rb` does). The client is `OsMe::LabelerClient.for(Decision::Model::Adapters::Fixed.new(...))`. No test calls a model or the network. 1. Given a file with two valid tune rows and one valid holdout row, all from train sessions, When `OsMe::LeverLabel.read(path, split: 'holdout')` runs, Then it returns only the holdout row, and its `id`, `text`, `sure`, `source`, `hard_to_undo`, `compounds`, and `stays_hidden` equal the row values. 2. Given a file whose second line has a missing `sure` key, or `source` `"guess"`, or `hard_to_undo` `"yes"`, or a `split` that differs from `TuneSplit`, or an `id` that repeats line 1, When `OsMe::LeverLabel.read` runs, Then it raises `OsMe::Error` whose message holds `line 2`. 3. Given a file with one row whose session `OsMe::Split#held_out?` is true, When `LeverLabel.read(path, split: 'tune')` runs, Then it raises `OsMe::Error` whose message holds that row's `id`; and When `LeverLabel.read(path, split: 'holdout')` runs on the same file, Then it does not raise. 4. Given two tune rows labeled `source` `facts` and `taste`, and a Fixed adapter that answers `source` with choice `facts`, When `OsMe::LeverRun.new(OsMe::LeverCheck.for('source', client), reps: 1, jobs: 1).results(labels)` is scored by `OsMe::AgreementScore` for the question `source`, Then agreement is 0.5, kappa is 0.0, and unreadable is 0. 5. Given two tune rows labeled (`hard_to_undo`, `compounds`, `stays_hidden`) = (true, false, true) and (false, false, false), and a Fixed adapter that answers `noul` 0.9, 0.2, and 0.9, When the `wrong_cost` lever runs with reps 1, Then agreement is 0.5 for `hard_to_undo`, `stays_hidden`, and `level`, agreement is 1.0 for `compounds`, and kappa for `compounds` is nil. 6. Given one tune row and a Fixed adapter with no answers (so the client raises `Decision::Model::ResponseError`), When the `wrong_cost` lever runs with reps 2, Then each of the four questions has 2 results, all with the prediction `unreadable`, and the `AgreementScore` of each has unreadable 2 and agreement 0.0. 7. Given one tune row and the `source` lever with reps 3, When `LeverRun#results` runs with jobs 2, Then it returns 3 results whose `rep` values are 1, 2, and 3. 8. Given hand-built `source` results: row A (`sure: true`, label `facts`) predicted `facts`, `facts`, `taste` in reps 1 to 3, and row B (`sure: false`, label `taste`) predicted `facts`, `unreadable`, `facts`, When `AgreementScore#summary(split: 'tune', lever: 'source')` is called, Then it returns `split=tune lever=source question=source rows=2 reps=3 n=6 agreement=33.3% kappa=-0.143 unreadable=1 majority=50.0% sure_n=3 sure_agreement=66.7%`. 9. Given the lever name `cost`, When `OsMe::LeverCheck.for('cost', client)` runs, Then it raises `OsMe::Error` whose message holds `cost`. 10. Given a label file with a malformed row, or the flag `--lever cost` with a valid file, When `ruby -I plugins/os-me/lib plugins/os-me/eval/row_labels/run.rb --lever LEVER --labels FILE --split tune --out OUT` runs, Then it exits 2 and stderr starts with `error:`. ## Owner sketch ```ruby module OsMe class LeverLabel < Data.define(:id, :split, :text, :sure, :source, :hard_to_undo, :compounds, :stays_hidden) SOURCES = AnswerSource::CRITERIA.keys.map(&:to_s).freeze def self.read(path, split:, sessions: Split.new, tune_split: TuneSplit.new) LabelFile.new(path, sessions: sessions).read(split) { |line, seen_ids| parse(line, tune_split, seen_ids) } end def session_id = id.rpartition(':').first def row = Row.new(text: text) end class LeverCheck def self.for(name, client) { 'source' => Source, 'wrong_cost' => Cost }.fetch(name) { raise Error, "unknown lever #{name}" }.new(client) end class Source QUESTIONS = { 'source' => AnswerSource::CRITERIA.keys.map(&:to_s) }.freeze def initialize(client) = @client = client def name = 'source' def expected(label) = { 'source' => label.source } def predicted(label) = { 'source' => AnswerSource.new(@client).classify(label.row).source.to_s } end class Cost def predicted(label) cost = OsMe::WrongCost::Assessor.new(@client).assess(label.row) cost.to_h.transform_keys(&:to_s).transform_values(&:to_s).merge('level' => cost.level.to_s) end end end LeverResult = Data.define(:id, :rep, :question, :label, :predicted, :sure) class LeverRun def initialize(check, reps:, jobs:) def results(labels) # one LeverResult per label, rep, and question; ResponseError -> 'unreadable' end class AgreementScore def initialize(results, question:, classes:) def agreement; def kappa; def unreadable; def majority def summary(split:, lever:) end end ``` ## Scope and blocking - In scope: `OsMe::LeverLabel`, `OsMe::LabelFile`, `OsMe::LeverCheck` (with `Source` and `Cost`), `OsMe::LeverResult`, `OsMe::LeverRun`, `OsMe::AgreementScore`, the move of the shared read out of `ConfidenceLabel`, `plugins/os-me/eval/row_labels/{run.rb,README.md}`, and tests under `plugins/os-me/test`. - Out of scope: changes to `AnswerSource::CRITERIA` or `Assessor::INSTRUCTIONS` (the S10 tuning loop's job after this lands); the label file, which a separate agent writes in the Store; a baseline run; changes to `reply_shape/run.rb`; a reversal-rate signal; `.gitignore` changes in the Store. - Blocked by: nothing. The tuning loops for both levers are blocked by this ticket and by the label file. ## Origin - Trigger: S10 plan (AFK decision-system build): AnswerSource and WrongCost::Assessor have no checked labels and no harness, so S10 cannot tune them - Improvised this session: none - Chain: S10 cannot tune AnswerSource or WrongCost ← no harness scores them against labels ← S4 built the levers without an eval. Stop at `DESIGN` (`docs/decision-system/build-plan.md` S10). - Root candidate: none found; searched AnswerSource eval, WrongCost eval, reply_shape/run.rb - Where: plugins/os-me/eval/; plugins/os-me/lib/os_me/answer_source.rb; plugins/os-me/lib/os_me/wrong_cost/ - Session: 4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5 - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5.jsonl VERDICT: CREATE (user-approved; backlog search only)

jared added the

project/cc-os

filed-by/user

category/code-change

labels

2026-10-23 20:25:36 +00:00

jared commented

2026-10-23 20:25:43 +00:00

Author

Owner

Work started on branch ticket-1285 (os-sdlc implement, AFK decision-system build, S10 T2).

jared commented

2026-10-23 20:34:40 +00:00

Author

Owner

Resolution

Done: eval/row_labels/run.rb scores AnswerSource and WrongCost::Assessor against checked labels: agreement, kappa and unreadable count per question and split, over reps.

Evidence: Implementation 147: 8 dispatches, 0 retries, 8 min 36 s, implementation_complete. Suite gate green. Label file of 100 rows (83 tune, 17 holdout) written to the Store samples by an Opus labeler from train sessions.

Follow-ups: Baselines for both levers run next under S10 (eval-run, no ticket); tuning loops wait for Jared because each iteration costs about 300 Sonnet calls.

## Resolution **Done:** eval/row_labels/run.rb scores AnswerSource and WrongCost::Assessor against checked labels: agreement, kappa and unreadable count per question and split, over reps. **Evidence:** Implementation 147: 8 dispatches, 0 retries, 8 min 36 s, implementation_complete. Suite gate green. Label file of 100 rows (83 tune, 17 holdout) written to the Store samples by an Opus labeler from train sessions. **Follow-ups:** Baselines for both levers run next under S10 (eval-run, no ticket); tuning loops wait for Jared because each iteration costs about 300 Sonnet calls.

jared closed this issue

2026-10-23 20:34:40 +00:00

jared referenced this issue from a commit

2026-10-23 20:35:48 +00:00

feat(os-me): Audited POODR boundaries for LabelFile, LeverLabel, LeverCheck (Source, Cost), LeverRun, AgreementScore, and eval/row_labels/run.rb; design and contracts hold, and R1, R2, and R3 are present. (#1285)

jared referenced this issue from a commit

2026-10-23 20:35:48 +00:00

docs(os-me): S10 T2 label-agreement eval History row (#1285)

jared referenced this issue

2026-10-23 21:44:55 +00:00

os-me: eval adapter choice moves to one shared object; row_labels gains --adapter #1291