decision-system S10: lever tuning loops #1252

New issue

Open

opened 2026-10-23 15:17:29 +00:00 by jared · 1 comment

jared commented

2026-10-23 15:17:29 +00:00

Owner

Problem

The levers use first-draft criteria text, thresholds, and anchors. ConfidenceExtractor and ReplyShape feed every other number, so their errors spread. Tuning loops must improve them against labels without touching held-out sessions.

Approach

Required work:

Record each kept change in the lever object with both numbers (tune and holdout) in the commit message — check: present ConfidenceExtractor in plugins/os-me/lib/**/*.rb

Orchestrator work: run /autoresearch per lever. Before each loop, read ~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md and docs/eval-discipline.md. Use the metric and labels in the tuning table of docs/decision-system/learning-loops.md. Order: ConfidenceExtractor and ReplyShape first, then AnswerSource, WrongCostAssessor, and the ConfidenceBand anchors. A loop keeps a change only if the tune split improves and the holdout split does not lose. Read only train sessions. Follow the AFK rules in docs/decision-system/build-plan.md: never wait for Jared, use the AFK default of each open item, and list each default used in the report.

Acceptance criteria

Given a loop run, when a change improves tune and does not lose on holdout, then the loop commits it with both numbers in the message.
Given a loop run, when a change fails either test, then the loop discards it and commits nothing.
Given a finished loop, when its labels are listed, then no label comes from a held-out session.

Owner sketch

# No new object. Loops change criteria text, thresholds, and anchors in existing lever objects.

Scope and blocking

Blocked by: #1245
Category: category/eval-run. The work is a set of tuning loops, not one behavior change. Do not run it through /os-sdlc:implement.

Blocked by S3 for the calibration numbers. S2c supplies the checked sample. Eval discipline applies: do not read held-out Task blocks or reserve sets.

Non-goals: new levers (S4).

Origin

Trigger: User asked for the decision-system build tickets this session
Improvised this session: none
Chain: no ticket for this stage ← the build plan names the stage and the user asked for the tickets. Stop at DESIGN (docs/decision-system/build-plan.md).
Root candidate: none found; searched lever tuning, autoresearch, ConfidenceExtractor
Where: docs/decision-system/build-plan.md S10; docs/decision-system/learning-loops.md; lever objects in plugins/os-me/lib/os_me/
Session: 4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5.jsonl

VERDICT: CREATE (user-approved; backlog search only)

## Problem The levers use first-draft criteria text, thresholds, and anchors. `ConfidenceExtractor` and `ReplyShape` feed every other number, so their errors spread. Tuning loops must improve them against labels without touching held-out sessions. ## Approach Required work: 1. Record each kept change in the lever object with both numbers (tune and holdout) in the commit message — check: present `ConfidenceExtractor` in `plugins/os-me/lib/**/*.rb` Orchestrator work: run `/autoresearch` per lever. Before each loop, read `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md` and `docs/eval-discipline.md`. Use the metric and labels in the tuning table of `docs/decision-system/learning-loops.md`. Order: `ConfidenceExtractor` and `ReplyShape` first, then `AnswerSource`, `WrongCostAssessor`, and the `ConfidenceBand` anchors. A loop keeps a change only if the tune split improves and the holdout split does not lose. Read only train sessions. Follow the AFK rules in `docs/decision-system/build-plan.md`: never wait for Jared, use the AFK default of each open item, and list each default used in the report. ## Acceptance criteria 1. Given a loop run, when a change improves tune and does not lose on holdout, then the loop commits it with both numbers in the message. 2. Given a loop run, when a change fails either test, then the loop discards it and commits nothing. 3. Given a finished loop, when its labels are listed, then no label comes from a held-out session. ## Owner sketch ```ruby # No new object. Loops change criteria text, thresholds, and anchors in existing lever objects. ``` ## Scope and blocking Blocked by: #1245 Category: `category/eval-run`. The work is a set of tuning loops, not one behavior change. Do not run it through `/os-sdlc:implement`. Blocked by S3 for the calibration numbers. S2c supplies the checked sample. Eval discipline applies: do not read held-out `Task` blocks or reserve sets. Non-goals: new levers (S4). ## Origin - Trigger: User asked for the decision-system build tickets this session - Improvised this session: none - Chain: no ticket for this stage ← the build plan names the stage and the user asked for the tickets. Stop at `DESIGN` (`docs/decision-system/build-plan.md`). - Root candidate: none found; searched lever tuning, autoresearch, ConfidenceExtractor - Where: docs/decision-system/build-plan.md S10; docs/decision-system/learning-loops.md; lever objects in plugins/os-me/lib/os_me/ - Session: 4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5 - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/4f3f95ea-3c6f-4d23-a0ea-e349d6ab02a5.jsonl VERDICT: CREATE (user-approved; backlog search only)

jared added the

project/cc-os

filed-by/user

category/eval-run

labels

2026-10-23 15:17:29 +00:00

jared referenced this issue

2026-10-23 17:30:59 +00:00

decision-system S2c: moment record, log, cursor, extract CLI #1244

jared commented

2026-10-23 17:31:06 +00:00

Author

Owner

Input from S2 (#1244), 2026-10-23: the 355 moments in moments.jsonl carry outcomes from the Haiku labeler with the old ReplyShape criteria (frozen-50 agreement 66-73%). The tuned criteria with Sonnet reach 83.3%. ExtractCursor and MomentLog#known_ids skip every known moment, so no current command relabels them. Before S10 tunes on these outcomes, relabel the existing moments with the current LABELER_MODEL and criteria (each moment has labeler_model, so stale rows are findable). S3 calibration numbers (Brier 0.16, band hit rates) also change after the relabel.