Instrument os-sdlc agent dispatch metrics to detect cost regressions #270

New issue

Closed

opened 2026-08-04 19:23:21 +00:00 by jared · 2 comments

jared commented

2026-08-04 19:23:21 +00:00

Owner

Context

Investigated a suspected os-sdlc programmer regression (54-65k tokens, 4.5-8.5 min per dispatch on 2026-08-04). Transcript + diff review showed no regression: per-dispatch cost tracks ticket complexity (a 78k-token programmer run predates the suspect lint-fixer commit 936428d), and per-ticket totals grew by design with the sharded-programming pipeline (12d76a2) and wider retry routing (f5a4961). But we had no baseline data to check — diagnosis required manual transcript archaeology. Add per-dispatch metrics capture in the os-sdlc pipeline (agent type, tokens, duration, tool uses, shard count, ticket id) plus a simple outlier flag vs rolling baseline, so future 'this feels slow' moments can be confirmed or dismissed as they happen.

## Context Investigated a suspected os-sdlc programmer regression (54-65k tokens, 4.5-8.5 min per dispatch on 2026-08-04). Transcript + diff review showed no regression: per-dispatch cost tracks ticket complexity (a 78k-token programmer run predates the suspect lint-fixer commit 936428d), and per-ticket totals grew by design with the sharded-programming pipeline (12d76a2) and wider retry routing (f5a4961). But we had no baseline data to check — diagnosis required manual transcript archaeology. Add per-dispatch metrics capture in the os-sdlc pipeline (agent type, tokens, duration, tool uses, shard count, ticket id) plus a simple outlier flag vs rolling baseline, so future 'this feels slow' moments can be confirmed or dismissed as they happen.

jared added the

label

2026-08-04 19:23:21 +00:00

jared added the

plugin/os-sdlc

label

2026-08-13 18:12:38 +00:00

jared added the

frozen

label

2026-08-16 14:25:06 +00:00

jared commented

2026-08-16 14:26:03 +00:00

Author

Owner

Frozen in the 2026-08-16 backlog reset — see #419 for the expiry procedure. Do not work unless a live run rediscovers this issue.

jared commented

2026-09-23 20:23:37 +00:00

Author

Owner

Resolution

Done: Closed in the 2026-09-23 relevance pass of retired-template tickets: SUPERSEDED.

Evidence: ADR-0175 (agents.model_id + dispatches token columns) and ADR-0193 (sweep timing) persist per-dispatch tokens/model; core gap closed.

Follow-ups: none

## Resolution **Done:** Closed in the 2026-09-23 relevance pass of retired-template tickets: SUPERSEDED. **Evidence:** ADR-0175 (agents.model_id + dispatches token columns) and ADR-0193 (sweep timing) persist per-dispatch tokens/model; core gap closed. **Follow-ups:** none

jared closed this issue

2026-09-23 20:23:37 +00:00