Reset behavior-verifier loop bound on deterministic progress (bound_exhausted fires during healthy progress)

jared commented

2026-08-25 19:42:43 +00:00

Owner

Context

Ticket #508's run (session 4b7df35f, implementation 1, 2026-08-25) failed with bound_exhausted:behavior-verifier/fail after 27 dispatches while making verifiable progress every loop. The behavior-verifier fail edge (poodr-implementation.yaml ~line 173, bound: 2) conflates two concerns: infinite-loop guard and scope budget. Each verifier fail means 'ticket not fully covered yet' and loops to test-writer for one micro-gap; a 3-AC ticket burned the whole bound inside AC1. Any ticket with more increments than the bound is a guaranteed fail. Ticket body reached all briefs — not a context problem; slicing + fixed bound is the defect.

Intelligence pass 2026-08-26 (code-probe verified): LoopLedger#spend! updates only spent (updated_at bug confirmed); behavior-verifier verdict is agent self-report on dispatches.verdict with no persisted test artifact; step_results.output is raw gate stdout; MinitestSummary parses aggregate counts only (runs/assertions/failures/errors), no test names; no per-implementation dispatch counter exists (dispatches.implementation_id FK makes one a single count query); terminal outcomes flow through NextStep#terminal_payload → CLI#emit, the single JSON surface.

Tasks

[ ] Progress reset (option A, count witness — decided 2026-08-26): new read-only ProgressWitness object (lib/os_sdlc/runner/progress_witness.rb) answering progressed_since_last_visit?(from_node) by comparing green-suite run counts (MinitestSummary over existing step_results rows) at the last two visits to the node; no schema change, no gate-command change. First visit (no baseline) → false. Inject into LoopLedger.new at the single construction site NextStep#route!; LoopLedger#spend! resets spent to 0 when the edge is flagged and the witness answers true, else spends as today. Public message shape (:ok | :exhausted) unchanged; RoutingApplier#edge_spent_ok? untouched. Edges opt in via a map-level flag: fail: { to: test-writer, bound: 2, progress_reset: true } + Map::Loader edge predicate. Judgment ping-pong edges never carry the flag — the non-goal enforces itself.
[ ] Global dispatch cap as escalation: DispatchBudget with DEFAULT_CAP = 60 in code (backstop is never opt-in), overridable per map via a dispatch_cap: key. Check in RoutingApplier#spent_decision before edge spend (must interrupt landing, before advance_and_stamp!). On firing: route a new escalate path to a needs-orchestrator terminal with outcome type escalation — never implementation_failed. Payload (via existing terminal_payload/CLI#emit surface) attaches a deterministic StateDump from pipeline.db: edge_spends rows, last N dispatches (agent, verdict, timestamps), suite-count timeline, ticket body, current branch/diff summary.
[ ] Escalation procedure reference file: plugins/os-sdlc/references/escalation-procedure.md owns the orchestrator playbook — read the state dump before touching anything, diagnose and redirect with full agency, all code changes re-enter through the normal gates (never merge freeform work), capture a ticket per escalation naming the cause. The implement skill gains ONE line only, e.g.: "In case of emergency (escalation message received), read and follow the escalation procedure: references/escalation-procedure.md." Rationale: keeps the diagnosis playbook out of every session's context and behind an explicit trigger, so the mechanical path stays the only visible path; file is reusable by other maps/skills.
[ ] Fix cosmetic bug: edge_spends.updated_at not bumped on spend (EdgeSpend#spend! update hash gains updated_at: Sequel::CURRENT_TIMESTAMP; Sequel does not auto-touch it, no timestamps plugin loaded).
[ ] ADR: progress-based termination semantics for progress_reset edges + cap-as-escalation (amends ADR-0133, does not supersede it). Non-goals recorded there: test-writer↔test-reviewer bounds unchanged; count witness not identity sets (option A).

Acceptance criteria

[ ] A ticket with N>bound increments completes when every loop lands a new green behavioral test (green-suite run count rises between verifier visits)
[ ] A loop producing no new green test for bound consecutive verifier fails still terminates with bound_exhausted
[ ] The global dispatch cap (default 60, map-overridable) halts the mechanical loop on any pathological reset cycle and emits an escalation outcome with the state dump; it never silently terminates the implementation, and it applies even when a map declares no dispatch_cap key
[ ] The implement skill contains only the one-line escalation pointer; the full procedure lives in references/escalation-procedure.md

Design rationale (option B loop-guard, decided 2026-08-25)

The loop guard must not depend on any model's self-report. An increments table keyed on model-declared increment identity is gameable by the drift it exists to catch (verifier renames the same gap → infinite resets). A new green test is a deterministic, hard-to-fake progress witness already produced by every healthy loop. Bound then means 'consecutive no-progress loops', which is exactly the infinite-loop condition and nothing else.

Progress witness = counts, not identity sets (option A, decided 2026-08-26): test identities are not recoverable from existing evidence (MinitestSummary parses counts only; gate stdout has no names). Count-increase and identity-set witnesses resist gaming equally — both are defeated only by landing a genuinely new green test — so identity recording would buy schema + gate-command changes for no robustness gain. Known weakness accepted: DiffedTestPaths churn can wobble counts across visits; revisit identity recording only if that produces observed false verdicts.

Cap-as-escalation (decided 2026-08-26): the cap fires precisely when the pipeline has hit something outside its mechanical competence and the most tokens are already sunk in mostly green, gate-verified code. Discarding that to a terminal implementation_failed throws away the expensive part. Instead the orchestrator — a stronger model with full tools — gets the diagnosis job, via a reference-file procedure loaded only on escalation (keeps tokens out of normal sessions and denies the mechanical agents any standing excuse to deviate). Guards: orchestrator changes re-enter via gates; every escalation files a ticket. Default cap 60 ≈ 2× #508's healthy 27-dispatch multi-AC run. Dropped from scope: per-increment metrics ride-along.

Origin

• Trigger: /os-sdlc:implement 508 → implementation_failed bound_exhausted:behavior-verifier/fail (dispatch 27, 15:04)
• Improvised this session: none (failure protocol followed; #508 marked waiting)
• Chain: guaranteed fail on multi-AC tickets ← bound spent on healthy loops ← fail-edge bound counts total visits, not no-progress visits ← DESIGN (poodr-implementation.yaml behavior-verifier fail edge bound:2; LoopLedger#spend!)
• Root candidate: this ticket

## Context Ticket #508's run (session 4b7df35f, implementation 1, 2026-08-25) failed with bound_exhausted:behavior-verifier/fail after 27 dispatches while making verifiable progress every loop. The behavior-verifier fail edge (poodr-implementation.yaml ~line 173, bound: 2) conflates two concerns: infinite-loop guard and scope budget. Each verifier fail means 'ticket not fully covered yet' and loops to test-writer for one micro-gap; a 3-AC ticket burned the whole bound inside AC1. Any ticket with more increments than the bound is a guaranteed fail. Ticket body reached all briefs — not a context problem; slicing + fixed bound is the defect. Intelligence pass 2026-08-26 (code-probe verified): LoopLedger#spend! updates only `spent` (updated_at bug confirmed); behavior-verifier verdict is agent self-report on dispatches.verdict with no persisted test artifact; step_results.output is raw gate stdout; MinitestSummary parses aggregate counts only (runs/assertions/failures/errors), no test names; no per-implementation dispatch counter exists (dispatches.implementation_id FK makes one a single count query); terminal outcomes flow through NextStep#terminal_payload → CLI#emit, the single JSON surface. ## Tasks [ ] Progress reset (option A, count witness — decided 2026-08-26): new read-only ProgressWitness object (lib/os_sdlc/runner/progress_witness.rb) answering progressed_since_last_visit?(from_node) by comparing green-suite run counts (MinitestSummary over existing step_results rows) at the last two visits to the node; no schema change, no gate-command change. First visit (no baseline) → false. Inject into LoopLedger.new at the single construction site NextStep#route!; LoopLedger#spend! resets spent to 0 when the edge is flagged and the witness answers true, else spends as today. Public message shape (:ok | :exhausted) unchanged; RoutingApplier#edge_spent_ok? untouched. Edges opt in via a map-level flag: `fail: { to: test-writer, bound: 2, progress_reset: true }` + Map::Loader edge predicate. Judgment ping-pong edges never carry the flag — the non-goal enforces itself. [ ] Global dispatch cap as escalation: DispatchBudget with DEFAULT_CAP = 60 in code (backstop is never opt-in), overridable per map via a `dispatch_cap:` key. Check in RoutingApplier#spent_decision before edge spend (must interrupt landing, before advance_and_stamp!). On firing: route a new `escalate` path to a `needs-orchestrator` terminal with outcome type `escalation` — never implementation_failed. Payload (via existing terminal_payload/CLI#emit surface) attaches a deterministic StateDump from pipeline.db: edge_spends rows, last N dispatches (agent, verdict, timestamps), suite-count timeline, ticket body, current branch/diff summary. [ ] Escalation procedure reference file: plugins/os-sdlc/references/escalation-procedure.md owns the orchestrator playbook — read the state dump before touching anything, diagnose and redirect with full agency, all code changes re-enter through the normal gates (never merge freeform work), capture a ticket per escalation naming the cause. The implement skill gains ONE line only, e.g.: "In case of emergency (escalation message received), read and follow the escalation procedure: references/escalation-procedure.md." Rationale: keeps the diagnosis playbook out of every session's context and behind an explicit trigger, so the mechanical path stays the only visible path; file is reusable by other maps/skills. [ ] Fix cosmetic bug: edge_spends.updated_at not bumped on spend (EdgeSpend#spend! update hash gains updated_at: Sequel::CURRENT_TIMESTAMP; Sequel does not auto-touch it, no timestamps plugin loaded). [ ] ADR: progress-based termination semantics for progress_reset edges + cap-as-escalation (amends ADR-0133, does not supersede it). Non-goals recorded there: test-writer↔test-reviewer bounds unchanged; count witness not identity sets (option A). ## Acceptance criteria [ ] A ticket with N>bound increments completes when every loop lands a new green behavioral test (green-suite run count rises between verifier visits) [ ] A loop producing no new green test for bound consecutive verifier fails still terminates with bound_exhausted [ ] The global dispatch cap (default 60, map-overridable) halts the mechanical loop on any pathological reset cycle and emits an `escalation` outcome with the state dump; it never silently terminates the implementation, and it applies even when a map declares no dispatch_cap key [ ] The implement skill contains only the one-line escalation pointer; the full procedure lives in references/escalation-procedure.md ## Design rationale (option B loop-guard, decided 2026-08-25) The loop guard must not depend on any model's self-report. An increments table keyed on model-declared increment identity is gameable by the drift it exists to catch (verifier renames the same gap → infinite resets). A new green test is a deterministic, hard-to-fake progress witness already produced by every healthy loop. Bound then means 'consecutive no-progress loops', which is exactly the infinite-loop condition and nothing else. Progress witness = counts, not identity sets (option A, decided 2026-08-26): test identities are not recoverable from existing evidence (MinitestSummary parses counts only; gate stdout has no names). Count-increase and identity-set witnesses resist gaming equally — both are defeated only by landing a genuinely new green test — so identity recording would buy schema + gate-command changes for no robustness gain. Known weakness accepted: DiffedTestPaths churn can wobble counts across visits; revisit identity recording only if that produces observed false verdicts. Cap-as-escalation (decided 2026-08-26): the cap fires precisely when the pipeline has hit something outside its mechanical competence and the most tokens are already sunk in mostly green, gate-verified code. Discarding that to a terminal implementation_failed throws away the expensive part. Instead the orchestrator — a stronger model with full tools — gets the diagnosis job, via a reference-file procedure loaded only on escalation (keeps tokens out of normal sessions and denies the mechanical agents any standing excuse to deviate). Guards: orchestrator changes re-enter via gates; every escalation files a ticket. Default cap 60 ≈ 2× #508's healthy 27-dispatch multi-AC run. Dropped from scope: per-increment metrics ride-along. ## Origin • Trigger: /os-sdlc:implement 508 → implementation_failed bound_exhausted:behavior-verifier/fail (dispatch 27, 15:04) • Improvised this session: none (failure protocol followed; #508 marked waiting) • Chain: guaranteed fail on multi-AC tickets ← bound spent on healthy loops ← fail-edge bound counts total visits, not no-progress visits ← DESIGN (poodr-implementation.yaml behavior-verifier fail edge bound:2; LoopLedger#spend!) • Root candidate: this ticket

jared added the

plugin/os-sdlc

label

2026-08-25 19:42:43 +00:00

jared commented

2026-08-25 21:11:22 +00:00

Author

Owner

Data points from the #508 live run (session 4b7df35f): the run completed at dispatch 66 — a hardcoded cap of 60 would have killed a healthy run; make the cap generous or map-configurable before merge. Also: implementation branch worktree-agent-aac9c5cd71a72ccd3 hand-authored ADR-0150, which now collides with cc-os ADR-0150 created on the #508 branch (adr-new CLI); renumber the #511 ADR at merge. Two of the three #508 bound-exhaustions were healthy-progress cases the progress-reset would have absorbed; the third (contradictory test, 3 no-progress loops) correctly would not.

jared referenced this issue

2026-08-25 21:11:40 +00:00

Build worktree Thor CLI (bin/worktree start|finish) and merge skills into /os:worktree #508

jared commented

2026-08-26 12:24:47 +00:00

Author

Owner

This was generated by AI during triage.

Ticket body rewritten 2026-08-26 after a triage review session. Changes: tasks collapsed 7→4 (test-identity recording folded into the progress-reset task; the test-writer↔test-reviewer non-goal moved into the ADR task; the optional metrics ride-along dropped from scope). The global dispatch cap changed from terminal failure to escalation: on firing, the runner hands the orchestrator a deterministic state dump from pipeline.db; the orchestrator diagnoses and redirects, its code changes re-enter via gates, and every escalation files a ticket. AC3 updated to match. Progress-reset core design (option B) unchanged.

> *This was generated by AI during triage.* Ticket body rewritten 2026-08-26 after a triage review session. Changes: tasks collapsed 7→4 (test-identity recording folded into the progress-reset task; the test-writer↔test-reviewer non-goal moved into the ADR task; the optional metrics ride-along dropped from scope). The global dispatch cap changed from terminal failure to escalation: on firing, the runner hands the orchestrator a deterministic state dump from pipeline.db; the orchestrator diagnoses and redirects, its code changes re-enter via gates, and every escalation files a ticket. AC3 updated to match. Progress-reset core design (option B) unchanged.

jared commented

2026-08-26 13:07:32 +00:00

Author

Owner

This was generated by AI during triage.

Design session 2026-08-26 (change-interpreter + code-probe + seam-designer intelligence pass): (1) Progress witness decided as option A — green-suite run-count comparison via MinitestSummary over existing step_results; no schema or gate changes; identity recording rejected (equal gaming resistance, extra cost). (2) Dispatch cap: DEFAULT_CAP=60 in code, map-overridable via dispatch_cap: key — backstop is never opt-in. (3) Escalation orchestrator playbook moved to plugins/os-sdlc/references/escalation-procedure.md; implement skill carries a one-line pointer only (token cost + keeps mechanical agents from deviating). Body tasks/ACs/rationale updated to match; verified code facts added to Context.

> *This was generated by AI during triage.* Design session 2026-08-26 (change-interpreter + code-probe + seam-designer intelligence pass): (1) Progress witness decided as option A — green-suite run-count comparison via MinitestSummary over existing step_results; no schema or gate changes; identity recording rejected (equal gaming resistance, extra cost). (2) Dispatch cap: DEFAULT_CAP=60 in code, map-overridable via dispatch_cap: key — backstop is never opt-in. (3) Escalation orchestrator playbook moved to plugins/os-sdlc/references/escalation-procedure.md; implement skill carries a one-line pointer only (token cost + keeps mechanical agents from deviating). Body tasks/ACs/rationale updated to match; verified code facts added to Context.

jared commented

2026-08-26 13:10:23 +00:00

Author

Owner

This was generated by AI during triage.

Work started on branch 511-progress-based-bound (TDD, seams per the 2026-08-26 design comment).

> *This was generated by AI during triage.* Work started on branch 511-progress-based-bound (TDD, seams per the 2026-08-26 design comment).

jared referenced this issue

2026-08-26 13:30:40 +00:00

Enforce escalate routes in Map::Validator; add them to coding.yaml #513

jared commented

2026-08-26 13:30:50 +00:00

Author

Owner

Resolution

Done: Progress-based loop bound + cap-as-escalation implemented TDD on branch 511-progress-based-bound, merged to main 85d5326 (commit 413dee2), ADR-0152 amends ADR-0133. progress_reset edge flag + ProgressWitness (green-suite run-count comparison) resets LoopLedger spend on progress; DispatchBudget DEFAULT_CAP 60 / map dispatch_cap override escalates via needs-orchestrator terminal with StateDump payload (Implementation.escalate!, escalated_at, SCHEMA_VERSION 18); references/escalation-procedure.md + one-line implement-skill pointer; edge_spends.updated_at bump fixed.

Evidence: Full suite 1137 runs, 2923 assertions, 0 failures (baseline 1111). AC1: cli_next_instruction boundary test — 3+ fails on a bound-2 progress_reset edge with rising green evidence keeps routing to test-writer. AC2: existing bound-exhaustion test untouched green. AC3: escalation boundary tests — capped implementation yields type escalation, reason dispatch_cap:, state_dump attached, not marked failed; default cap applies with no map key. AC4: SKILL.md carries only the one-line pointer; procedure in references/escalation-procedure.md. Plugin cache refreshed.

Follow-ups: #513 (Map::Validator escalate-route enforcement + coding.yaml edges, ticket-skeptic CREATE); the UnroutableVerdict-generic-exit-3 surfacing is folded into #513; no others.

## Resolution **Done:** Progress-based loop bound + cap-as-escalation implemented TDD on branch 511-progress-based-bound, merged to main 85d5326 (commit 413dee2), ADR-0152 amends ADR-0133. progress_reset edge flag + ProgressWitness (green-suite run-count comparison) resets LoopLedger spend on progress; DispatchBudget DEFAULT_CAP 60 / map dispatch_cap override escalates via needs-orchestrator terminal with StateDump payload (Implementation.escalate!, escalated_at, SCHEMA_VERSION 18); references/escalation-procedure.md + one-line implement-skill pointer; edge_spends.updated_at bump fixed. **Evidence:** Full suite 1137 runs, 2923 assertions, 0 failures (baseline 1111). AC1: cli_next_instruction boundary test — 3+ fails on a bound-2 progress_reset edge with rising green evidence keeps routing to test-writer. AC2: existing bound-exhaustion test untouched green. AC3: escalation boundary tests — capped implementation yields type escalation, reason dispatch_cap:<cap>, state_dump attached, not marked failed; default cap applies with no map key. AC4: SKILL.md carries only the one-line pointer; procedure in references/escalation-procedure.md. Plugin cache refreshed. **Follow-ups:** #513 (Map::Validator escalate-route enforcement + coding.yaml edges, ticket-skeptic CREATE); the UnroutableVerdict-generic-exit-3 surfacing is folded into #513; no others.

jared closed this issue

2026-08-26 13:30:51 +00:00

jared referenced this issue from a commit

2026-08-26 13:48:53 +00:00

feat(os-sdlc): progress-based loop bound + dispatch-cap escalation (#511)

jared referenced this issue from a commit

2026-08-26 13:48:53 +00:00

merge 511-progress-based-bound: #511 progress-based loop bound + dispatch-cap escalation