Build the held-out set and the fixed replay score for os-me #940

New issue

Closed

opened 2026-10-04 20:15:26 +00:00 by jared · 2 comments

jared commented

2026-10-04 20:15:26 +00:00

Owner

Problem

The os-me skill cannot improve without a way to measure how well it predicts Jared's responses.

Question

How does a fixed script, outside the Store, score a Prediction against Jared's real reply?

Context

Map: #936

Build in plugins/os-me/ (Shell):

A sampler that finds back-and-forth exchanges in ~/.claude/projects/*/ transcripts: an assistant output, then a typed reply from Jared that accepts, corrects, or redirects. Skip sessions with no reply from Jared.
A deterministic split by session id. The held-out session ids go to a list. The skill and the kickoff run must skip those sessions; the Shell skills state this rule.
A score script: for each held-out exchange, give the skill the context before Jared's reply, get its Prediction, and compare it with the real reply. A model judge returns structured output (JSON read by code, no regex over prose): same action or not, and what the Prediction missed. The script prints the agreement rate and appends it to a score log with the date and the Store commit.
Follow docs/eval-discipline.md. Read ~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md before you design the judge.

AFK.

## Problem The os-me skill cannot improve without a way to measure how well it predicts Jared's responses. ## Question How does a fixed script, outside the Store, score a Prediction against Jared's real reply? ## Context Map: #936 Build in `plugins/os-me/` (Shell): - A sampler that finds back-and-forth exchanges in `~/.claude/projects/*/` transcripts: an assistant output, then a typed reply from Jared that accepts, corrects, or redirects. Skip sessions with no reply from Jared. - A deterministic split by session id. The held-out session ids go to a list. The skill and the kickoff run must skip those sessions; the Shell skills state this rule. - A score script: for each held-out exchange, give the skill the context before Jared's reply, get its Prediction, and compare it with the real reply. A model judge returns structured output (JSON read by code, no regex over prose): same action or not, and what the Prediction missed. The script prints the agreement rate and appends it to a score log with the date and the Store commit. - Follow `docs/eval-discipline.md`. Read `~/Documents/SecondBrain/howto/running-autoresearch-skill-evals.md` before you design the judge. AFK.

jared added the

project/cc-os

filed-by/user

labels

2026-10-04 20:15:26 +00:00

jared referenced this issue

2026-10-04 20:15:53 +00:00

Run the os-me kickoff: the skill builds its first Store from transcripts #941

jared commented

2026-10-04 20:21:37 +00:00

Author

Owner

Start: building the fixed eval (bin/os-me-eval, lib, test, eval) on branch os-me.

jared commented

2026-10-04 20:59:26 +00:00

Author

Owner

Resolution

Done: bin/os-me-eval built: deterministic 15 percent held-out split by SHA-256 of the session id, exchange finder, context renderer, headless predictor and judge with JSON verdicts, score log. Review bug fixed: predictor reads are limited to the Store and the context directory.

Evidence: main 5e3ffbc5; 32 runs 76 assertions 0 failures; rubocop clean; cc-os counts: 159 train sessions, 23 held-out sessions, 74 held-out exchanges; a real call proved a transcript read is denied.

Follow-ups: none; the score-measure question for Jared is listed on #942

## Resolution **Done:** bin/os-me-eval built: deterministic 15 percent held-out split by SHA-256 of the session id, exchange finder, context renderer, headless predictor and judge with JSON verdicts, score log. Review bug fixed: predictor reads are limited to the Store and the context directory. **Evidence:** main 5e3ffbc5; 32 runs 76 assertions 0 failures; rubocop clean; cc-os counts: 159 train sessions, 23 held-out sessions, 74 held-out exchanges; a real call proved a transcript read is denied. **Follow-ups:** none; the score-measure question for Jared is listed on #942

jared closed this issue

2026-10-04 20:59:26 +00:00

jared referenced this issue

2026-10-04 20:59:41 +00:00

Map: os-me, a skill that predicts and gives Jared's response #936

jared referenced this issue from a commit

2026-10-05 13:50:50 +00:00

feat(os-me): Shell, held-out replay score, ADR-0272 (#937 #938 #940)

jared referenced this issue from a commit

2026-10-05 13:50:50 +00:00

test(os-me): exchange finder and renderer cover slash-command replies and injected entries (#940)

jared referenced this issue from a commit