Instrument os-sdlc agent dispatch metrics to detect cost regressions #270

New issue

Open

opened 2026-08-04 19:23:21 +00:00 by jared · 1 comment

jared commented

2026-08-04 19:23:21 +00:00

Owner

Context

Investigated a suspected os-sdlc programmer regression (54-65k tokens, 4.5-8.5 min per dispatch on 2026-08-04). Transcript + diff review showed no regression: per-dispatch cost tracks ticket complexity (a 78k-token programmer run predates the suspect lint-fixer commit 936428d), and per-ticket totals grew by design with the sharded-programming pipeline (12d76a2) and wider retry routing (f5a4961). But we had no baseline data to check — diagnosis required manual transcript archaeology. Add per-dispatch metrics capture in the os-sdlc pipeline (agent type, tokens, duration, tool uses, shard count, ticket id) plus a simple outlier flag vs rolling baseline, so future 'this feels slow' moments can be confirmed or dismissed as they happen.

## Context Investigated a suspected os-sdlc programmer regression (54-65k tokens, 4.5-8.5 min per dispatch on 2026-08-04). Transcript + diff review showed no regression: per-dispatch cost tracks ticket complexity (a 78k-token programmer run predates the suspect lint-fixer commit 936428d), and per-ticket totals grew by design with the sharded-programming pipeline (12d76a2) and wider retry routing (f5a4961). But we had no baseline data to check — diagnosis required manual transcript archaeology. Add per-dispatch metrics capture in the os-sdlc pipeline (agent type, tokens, duration, tool uses, shard count, ticket id) plus a simple outlier flag vs rolling baseline, so future 'this feels slow' moments can be confirmed or dismissed as they happen.

jared added the

label

2026-08-04 19:23:21 +00:00

jared added the

plugin/os-sdlc

label

2026-08-13 18:12:38 +00:00

jared added the

frozen

label

2026-08-16 14:25:06 +00:00

jared commented