Ticket map escalates on a documentation-only case: behavior-verifier fails, test-writer unexpected_pass

jared commented

2026-09-15 13:55:41 +00:00

Owner

Problem

Implementation 17 on ticket #587 ran the poodr-ticket-implementation.yaml map.
Case 5 of the ticket's ## Cases section asked for a prose change to SKILL.md.
The programmer made that edit in round 1, together with the code change.

Observed

behavior-verifier (dispatch 230, 25 s, 2 tool uses) returned fail. It named outstanding tasks.
The map routed the run to test-writer (dispatch 232).
test-writer wrote tests/skills/implement_skill_default_map_test.rb. The test asserts on the raw SKILL.md text.
Its diff-test gate exited 0 at once.

verdict: escalate
reason: unexpected_pass:test-writer/diff-test

Rubocop then flagged that test three times with Sdlc/Structural/RawFileContentAssertion.
The orchestrator finished the case by hand. It deleted the prose test, committed, and closed the run.

Reproduce

n/a: pipeline-behavior defect, not a single reproducible command

Expected

A documentation-only case must not sit under ## Cases. Per ADR-0190, that section holds one observable given/when/then per line.
The TicketShape check must flag a documentation-only line under ## Cases.
As an alternative or additional fix, the behavior-verifier charter should gain a rule: an outstanding task whose only evidence is a text-file diff is verified by reading, not by routing to test-writer.
Triage picks between these two candidates.

Origin

Trigger: implementation-open run 17 on ticket #587, poodr-ticket-implementation.yaml map, dispatch 230/232
Improvised this session: orchestrator hand-deleted the prose test, committed, and closed the run
Chain: unexpected_pass:test-writer/diff-test ← test-writer wrote a raw-text-assertion test for a documentation case ← behavior-verifier failed a documentation-only task instead of verifying it by reading ← DESIGN (ADR-0190 does not exclude documentation-only lines from ## Cases)
Root candidate: this ticket, candidate (a): extend the capture template's agent-ready variant and the TicketShape check to exclude documentation/prose edits from ## Cases
Where: docs/memory-system ticket-map shape rules (ADR-0190), os-sdlc behavior-verifier charter
Session: b1c83b81-7be1-4f17-9e8e-cae36a934f25
Transcript: (session id above; no separate transcript path provided)
VERDICT: CREATE (ticket-skeptic: named defect class with a complete Origin chain ending in a real DESIGN gap; ADR-0190 does not exclude prose lines from ## Cases; root candidate itself, no in-session duplicate)

## Problem Implementation 17 on ticket #587 ran the poodr-ticket-implementation.yaml map. Case 5 of the ticket's `## Cases` section asked for a prose change to SKILL.md. The programmer made that edit in round 1, together with the code change. ## Observed behavior-verifier (dispatch 230, 25 s, 2 tool uses) returned fail. It named outstanding tasks. The map routed the run to test-writer (dispatch 232). test-writer wrote `tests/skills/implement_skill_default_map_test.rb`. The test asserts on the raw SKILL.md text. Its diff-test gate exited 0 at once. ``` verdict: escalate reason: unexpected_pass:test-writer/diff-test ``` Rubocop then flagged that test three times with `Sdlc/Structural/RawFileContentAssertion`. The orchestrator finished the case by hand. It deleted the prose test, committed, and closed the run. ## Reproduce n/a: pipeline-behavior defect, not a single reproducible command ## Expected A documentation-only case must not sit under `## Cases`. Per ADR-0190, that section holds one observable given/when/then per line. The `TicketShape` check must flag a documentation-only line under `## Cases`. As an alternative or additional fix, the behavior-verifier charter should gain a rule: an outstanding task whose only evidence is a text-file diff is verified by reading, not by routing to test-writer. Triage picks between these two candidates. ## Origin - Trigger: implementation-open run 17 on ticket #587, poodr-ticket-implementation.yaml map, dispatch 230/232 - Improvised this session: orchestrator hand-deleted the prose test, committed, and closed the run - Chain: unexpected_pass:test-writer/diff-test ← test-writer wrote a raw-text-assertion test for a documentation case ← behavior-verifier failed a documentation-only task instead of verifying it by reading ← DESIGN (ADR-0190 does not exclude documentation-only lines from `## Cases`) - Root candidate: this ticket, candidate (a): extend the capture template's agent-ready variant and the TicketShape check to exclude documentation/prose edits from `## Cases` - Where: docs/memory-system ticket-map shape rules (ADR-0190), os-sdlc behavior-verifier charter - Session: b1c83b81-7be1-4f17-9e8e-cae36a934f25 - Transcript: (session id above; no separate transcript path provided) - VERDICT: CREATE (ticket-skeptic: named defect class with a complete Origin chain ending in a real DESIGN gap; ADR-0190 does not exclude prose lines from `## Cases`; root candidate itself, no in-session duplicate)

jared added the

P2

plugin/os-sdlc

filed-by/agent

labels

2026-09-15 13:55:41 +00:00

jared referenced this issue

2026-09-15 13:56:42 +00:00

TemplateValidator requires Owner sketch and Cases on ready-for-agent tickets; capture template and ticket-decompose emit them (ADR-0190) #588

jared referenced this issue

2026-09-15 14:21:11 +00:00

TemplateValidator requires Owner sketch and Cases on ready-for-agent tickets; capture template and ticket-decompose emit them (ADR-0190) #588

jared referenced this issue

2026-09-18 13:40:13 +00:00

os-backlog validator: optional sections list for ## Retired tests and ## Approach #612

jared commented

2026-09-19 18:02:51 +00:00

Author

Owner

Second manifestation, from hyperthrive-websites #55 (session 593440fa): after a hand-fix re-entry on an already-green tree, test-writer escalate/unexpected_pass leaves the Runner in await_completion with no dispatch and no failure; there is no re-entry path.

Observed, in order:

Implementation 2 failed with reason verdict:error after programmer-repair (dispatches 10-12) because a stale test the ticket did not list under ## Retired tests asserted the old behavior; the programmer cannot edit tests, so the gate could not go green.
Following references/escalation-procedure.md step 4 ("hand-fix a test ... then re-enter through the normal gates from the affected node"), the operator hand-fixed the stale test; full suite green (493 runs, 0 failures). There is no resume command, so re-entry means a new implementation-open.
implementation-open then refused: harness error: OsSdlc::HarnessError: ticket body missing: Retired tests (not found: ...) for all five retired tests, because implementation 2 had already deleted them. Workaround: retitle the heading to ## Retired tests (removed in implementation 2).
implementation-open then succeeded (implementation 3). change-auditor pass, seam-designer pass. test-writer (dispatch 14) found every Case already covered, added nothing, ran os-sdlc-runner gate 14: verdict escalate, classification unexpected_pass (34 runs, 87 assertions, 0 failures). subagent-stop recorded verdict escalate. next_step: {"type":"await_completion","implementation_id":3}. os-sdlc-runner next returns the same await_completion indefinitely. No dispatch, no escalation payload, no implementation_failed.

Consequence: the escalation procedure's documented recovery (hand-fix, re-enter) cannot be executed: the map has no edge from test-writer escalate/unexpected_pass and the Runner stalls. The operator had to move the green work to a side branch, reset to main, and rerun the whole map (implementation 4) at full cost.

Suggested fix for the maintainer to judge: either (a) give the map an edge for test-writer unexpected_pass that proceeds to behavior-verifier when the tree is already green for the ticket's Cases, or (b) add a resume/re-enter command that opens at a named node against the current tree, and make implementation-open tolerate retired tests that are already absent.

Second manifestation, from hyperthrive-websites #55 (session 593440fa): after a hand-fix re-entry on an already-green tree, test-writer escalate/unexpected_pass leaves the Runner in await_completion with no dispatch and no failure; there is no re-entry path. **Observed, in order:** 1. Implementation 2 failed with reason verdict:error after programmer-repair (dispatches 10-12) because a stale test the ticket did not list under `## Retired tests` asserted the old behavior; the programmer cannot edit tests, so the gate could not go green. 2. Following references/escalation-procedure.md step 4 ("hand-fix a test ... then re-enter through the normal gates from the affected node"), the operator hand-fixed the stale test; full suite green (493 runs, 0 failures). There is no resume command, so re-entry means a new implementation-open. 3. implementation-open then refused: `harness error: OsSdlc::HarnessError: ticket body missing: Retired tests (not found: ...)` for all five retired tests, because implementation 2 had already deleted them. Workaround: retitle the heading to `## Retired tests (removed in implementation 2)`. 4. implementation-open then succeeded (implementation 3). change-auditor pass, seam-designer pass. test-writer (dispatch 14) found every Case already covered, added nothing, ran `os-sdlc-runner gate 14`: verdict `escalate`, classification `unexpected_pass` (34 runs, 87 assertions, 0 failures). subagent-stop recorded verdict `escalate`. next_step: `{"type":"await_completion","implementation_id":3}`. `os-sdlc-runner next` returns the same await_completion indefinitely. No dispatch, no escalation payload, no implementation_failed. **Consequence:** the escalation procedure's documented recovery (hand-fix, re-enter) cannot be executed: the map has no edge from test-writer `escalate/unexpected_pass` and the Runner stalls. The operator had to move the green work to a side branch, reset to main, and rerun the whole map (implementation 4) at full cost. **Suggested fix for the maintainer to judge:** either (a) give the map an edge for test-writer `unexpected_pass` that proceeds to behavior-verifier when the tree is already green for the ticket's Cases, or (b) add a resume/re-enter command that opens at a named node against the current tree, and make implementation-open tolerate retired tests that are already absent.

jared referenced this issue

2026-09-23 16:46:03 +00:00

Ticket sections have no shared model, and pipeline roles work from silos of the ticket #643

jared commented

2026-09-24 20:19:31 +00:00

Author

Owner

2026-09-24 blast-radius re-evaluation: the fix belongs in TicketShape as a doc-only-case rule with an ADR-0190 amendment. The ticket's candidate (b), "behavior-verifier verifies by reading," reverses ADR-0173 and is not allowed without a superseding ADR. Removed from the quick-win batch; needs design.

jared referenced this issue

2026-10-01 15:26:43 +00:00

Add the poodr-guard-implementation map for green guard tests #876

jared added the

category/owner-ruling

label

2026-10-10 13:32:07 +00:00

jared referenced this issue

2026-10-18 18:58:41 +00:00

no pipeline path for guard/characterization tests (unexpected_pass escalation) #859

jared commented

2026-10-18 18:59:20 +00:00