Make system-model opportunities output deterministic #1505

New issue

Closed

opened 2026-11-01 22:07:20 +00:00 by jared · 2 comments

jared commented

2026-11-01 22:07:20 +00:00

Owner

Problem

system-model opportunities gives different output for the same code. The coverage.touched figure changed from 92 to 91 between two runs with no code change. Component sizes depend on journey coverage, so finding scores and counts are not reproducible. Proof figures and before/after comparisons are then not reliable.

Observed

Two agents on 2026-11-01 reported this independently. One saw coverage.touched go from 92 to 91 with no code change. Another saw opportunities JSON differ between a "before" run during concurrent activity and a clean rerun.

Reproduce

for i in 1 2 3; do BUNDLE_GEMFILE=plugins/os-sdlc/Gemfile bundle exec plugins/os-sdlc/bin/system-model opportunities plugins/os-sdlc --json | ruby -rjson -e 'p JSON.parse($stdin.read).dig("coverage","touched")'; done

Approach

Find the source of the variation in journey tracing (for example a timing-dependent trace subprocess, a shared temp path, or an unordered set). Make the coverage result depend only on the code under analysis.

Required work: none beyond the acceptance criteria.

Acceptance criteria

Given unchanged code, when system-model opportunities plugins/os-sdlc --json runs several times, including while other suites run, then every run gives the same output.

Origin

Trigger: before/after measurements for #1501 and the leaks rule, 2026-11-01
Improvised this session: agents reran "before" measurements on a quiet machine
Chain: unstable findings ← unstable component sizes ← unstable coverage.touched ← unknown (IMPROVISATION: reruns)
Root candidate: none found; searched determin, nondetermin, opportunit, coverage.touched, journey coverage, system-model, order
Where: plugins/os-sdlc/lib/os_sdlc/system_model (journey coverage)
Session: 60a0ffdb-3a9b-4ff4-8f09-f798b484e41e
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/60a0ffdb-3a9b-4ff4-8f09-f798b484e41e.jsonl

VERDICT: CREATE (ticket-skeptic, 2026-11-01: reproduced on a clean tree, coverage.touched 91, 91, 92 over three runs)

## Problem `system-model opportunities` gives different output for the same code. The `coverage.touched` figure changed from 92 to 91 between two runs with no code change. Component sizes depend on journey coverage, so finding scores and counts are not reproducible. Proof figures and before/after comparisons are then not reliable. ## Observed Two agents on 2026-11-01 reported this independently. One saw `coverage.touched` go from 92 to 91 with no code change. Another saw `opportunities` JSON differ between a "before" run during concurrent activity and a clean rerun. ## Reproduce ``` for i in 1 2 3; do BUNDLE_GEMFILE=plugins/os-sdlc/Gemfile bundle exec plugins/os-sdlc/bin/system-model opportunities plugins/os-sdlc --json | ruby -rjson -e 'p JSON.parse($stdin.read).dig("coverage","touched")'; done ``` ## Approach Find the source of the variation in journey tracing (for example a timing-dependent trace subprocess, a shared temp path, or an unordered set). Make the coverage result depend only on the code under analysis. Required work: none beyond the acceptance criteria. ## Acceptance criteria 1. Given unchanged code, when `system-model opportunities plugins/os-sdlc --json` runs several times, including while other suites run, then every run gives the same output. ## Origin - Trigger: before/after measurements for #1501 and the leaks rule, 2026-11-01 - Improvised this session: agents reran "before" measurements on a quiet machine - Chain: unstable findings ← unstable component sizes ← unstable `coverage.touched` ← unknown (IMPROVISATION: reruns) - Root candidate: none found; searched determin, nondetermin, opportunit, coverage.touched, journey coverage, system-model, order - Where: plugins/os-sdlc/lib/os_sdlc/system_model (journey coverage) - Session: 60a0ffdb-3a9b-4ff4-8f09-f798b484e41e - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/60a0ffdb-3a9b-4ff4-8f09-f798b484e41e.jsonl VERDICT: CREATE (ticket-skeptic, 2026-11-01: reproduced on a clean tree, coverage.touched 91, 91, 92 over three runs)

jared added the

plugin/os-sdlc

filed-by/agent

category/code-change

labels

2026-11-01 22:07:20 +00:00

jared commented

2026-11-01 22:07:40 +00:00

Author

Owner

Work started on branch worktree-agent-a6cbc6cbefffeaf7a

jared commented

2026-11-01 22:13:16 +00:00

Author

Owner

Resolution

Done: Tracer runs the trace subprocess with a pinned Minitest seed (Tracer::SEED = 0), so journey order and coverage are stable

Evidence: merged on main; cause: random seed let the first journey absorb 39 Boot lazy-load messages (355 vs 316); 5 sequential + 8 parallel runs under suite load gave one md5; suite 0 failures

Follow-ups: none: lazy-load calls still land in the first journey's trace; noted in the session handoff for the next judged re-measure

## Resolution **Done:** Tracer runs the trace subprocess with a pinned Minitest seed (Tracer::SEED = 0), so journey order and coverage are stable **Evidence:** merged on main; cause: random seed let the first journey absorb 39 Boot lazy-load messages (355 vs 316); 5 sequential + 8 parallel runs under suite load gave one md5; suite 0 failures **Follow-ups:** none: lazy-load calls still land in the first journey's trace; noted in the session handoff for the next judged re-measure

jared closed this issue

2026-11-01 22:13:16 +00:00

jared referenced this issue from a commit

2026-11-01 22:13:17 +00:00

fix(system-model): pin the trace subprocess Minitest seed so journeys are deterministic (#1505)

jared referenced this issue

2026-11-02 22:35:20 +00:00

Tag one use_case test per os-backlog command for journeys #1514