Collaborator pass signal-delta criterion needs a mechanical gate, not a verifier judgment #1089

New issue

Closed

opened 2026-10-17 16:32:22 +00:00 by jared · 2 comments

jared commented

2026-10-17 16:32:22 +00:00

Owner

Problem

The loop's ticket-body writes acceptance criterion 1 as a signal delta: "when bin/collaborator-pass measure runs after the change, then finding X scores at most N". On #1088 the behavior-verifier marked it done and the run completed, but the post-merge measure shows the finding unchanged (copied:defn:b47f392a still 4) plus a new IDENTICAL copied entry of mass 450 across the seven commands, and the argv clump back to 49. No pipeline node runs the measure, so the criterion is judged, not checked, and the loop's stop-check caught the regression only after merge.

Observed

stop-check after run 20261017T163118Z: copied_entry 24 -> 49 (regressed), data_clump 296 -> 309 (regressed); loop stopped.

Reproduce

plugins/os-sdlc/bin/collaborator-pass stop-check --runs docs/crc/os-sdlc/collaborator-runs --rulings docs/crc/os-sdlc/collaborator-rulings.tsv

Approach

A typed check the pipeline can run before merge: either a new Required-work check form that runs collaborator-pass measure in the worktree and compares one finding's score, or a gate step registered for tickets that carry a recipe (detect by the Trigger: bin/collaborator-pass run Origin line). The verifier then has a mechanical result to read instead of a prose claim.

Required work: none beyond the acceptance criteria.

Acceptance criteria

Given a ticket whose Origin names a collaborator-pass run, when the pipeline reaches suite-check, then a gate step runs measure in the worktree and fails when the named finding's score exceeds the ticket's score_after.
Given #1088's diff replayed, when that gate runs, then it fails on copied:defn:b47f392a at 4 and on the new IDENTICAL defn group.

Origin

Trigger: collaborator pass loop iteration 2, ticket #1088, implementation 59, 2026-10-17
Improvised this session: none
Chain: regression merged ← signal-delta criterion judged by behavior-verifier without a measurement ← DESIGN (loop brief stage 7: ticket-body emits the delta as a prose acceptance criterion; no gate step runs measure)
Root candidate: this ticket
Where: plugins/os-sdlc/lib/os_sdlc/collaborator_pass/ticket_draft/acceptance.rb
Session: f1eaf768-77b4-48c7-afde-b0344f519273
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/f1eaf768-77b4-48c7-afde-b0344f519273.jsonl

VERDICT: CREATE
REASON: Observed failure in a complete run; #1088 merged with its signal-delta criterion unmet and no pipeline node runs measure. A gate before merge fixes the cause.

## Problem The loop's ticket-body writes acceptance criterion 1 as a signal delta: "when bin/collaborator-pass measure runs after the change, then finding X scores at most N". On #1088 the behavior-verifier marked it done and the run completed, but the post-merge measure shows the finding unchanged (copied:defn:b47f392a still 4) plus a new IDENTICAL copied entry of mass 450 across the seven commands, and the argv clump back to 49. No pipeline node runs the measure, so the criterion is judged, not checked, and the loop's stop-check caught the regression only after merge. ## Observed stop-check after run 20261017T163118Z: copied_entry 24 -> 49 (regressed), data_clump 296 -> 309 (regressed); loop stopped. ## Reproduce ``` plugins/os-sdlc/bin/collaborator-pass stop-check --runs docs/crc/os-sdlc/collaborator-runs --rulings docs/crc/os-sdlc/collaborator-rulings.tsv ``` ## Approach A typed check the pipeline can run before merge: either a new Required-work check form that runs `collaborator-pass measure` in the worktree and compares one finding's score, or a gate step registered for tickets that carry a recipe (detect by the `Trigger: bin/collaborator-pass run` Origin line). The verifier then has a mechanical result to read instead of a prose claim. Required work: none beyond the acceptance criteria. ## Acceptance criteria 1. Given a ticket whose Origin names a collaborator-pass run, when the pipeline reaches suite-check, then a gate step runs `measure` in the worktree and fails when the named finding's score exceeds the ticket's score_after. 2. Given #1088's diff replayed, when that gate runs, then it fails on copied:defn:b47f392a at 4 and on the new IDENTICAL defn group. ## Origin - Trigger: collaborator pass loop iteration 2, ticket #1088, implementation 59, 2026-10-17 - Improvised this session: none - Chain: regression merged ← signal-delta criterion judged by behavior-verifier without a measurement ← DESIGN (loop brief stage 7: ticket-body emits the delta as a prose acceptance criterion; no gate step runs measure) - Root candidate: this ticket - Where: plugins/os-sdlc/lib/os_sdlc/collaborator_pass/ticket_draft/acceptance.rb - Session: f1eaf768-77b4-48c7-afde-b0344f519273 - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/f1eaf768-77b4-48c7-afde-b0344f519273.jsonl VERDICT: CREATE REASON: Observed failure in a complete run; #1088 merged with its signal-delta criterion unmet and no pipeline node runs measure. A gate before merge fixes the cause.

jared added the

plugin/os-sdlc

filed-by/agent

category/code-change

labels

2026-10-17 16:32:22 +00:00

jared commented

2026-10-17 16:33:44 +00:00

Author

Owner

Work started 2026-10-17 on branch ticket-1089 (same AFK session). Design: a delta-check subcommand on bin/collaborator-pass that measures the current tree against a baseline run and a recipe, and the loop skill runs it before worktree finish and refuses to merge on regression. No pipeline map or check-vocabulary change.

jared commented

2026-10-17 16:38:18 +00:00

Author

Owner

Resolution

Done: bin/collaborator-pass delta-check: measures the current tree against a baseline run and a recipe; fails when the recipe's finding exceeds score_after or any shape's score rises. The loop skill runs it after implementation_complete and before worktree finish and refuses to merge on exit 1.

Evidence: Commit e141d2e5 merged to main. Suite 2040 runs, 0 failures. Rubocop clean on new files. Replayed on #1088's recipe against run 20261017T161455Z: exit 1, copied_entry 24 -> 50 and data_clump 296 -> 312 reported as regressions.

Follow-ups: The pipeline itself still has no measure gate; this check runs at the orchestrator step before merge, which is where the loop merges. If a pipeline-level gate is wanted later it is a new ticket; none filed.

## Resolution **Done:** bin/collaborator-pass delta-check: measures the current tree against a baseline run and a recipe; fails when the recipe's finding exceeds score_after or any shape's score rises. The loop skill runs it after implementation_complete and before worktree finish and refuses to merge on exit 1. **Evidence:** Commit e141d2e5 merged to main. Suite 2040 runs, 0 failures. Rubocop clean on new files. Replayed on #1088's recipe against run 20261017T161455Z: exit 1, copied_entry 24 -> 50 and data_clump 296 -> 312 reported as regressions. **Follow-ups:** The pipeline itself still has no measure gate; this check runs at the orchestrator step before merge, which is where the loop merges. If a pipeline-level gate is wanted later it is a new ticket; none filed.

jared closed this issue

2026-10-17 16:38:18 +00:00

jared referenced this issue

2026-10-17 16:38:24 +00:00

Move method to owner: CommandController superclass for the seven commands (copied:defn:8a8c27d3) #1090

jared referenced this issue from a commit

2026-10-17 16:55:28 +00:00

feat(os-sdlc): collaborator-pass delta-check guards the merge (#1089)