behavior-verifier can fail a dispatch without running a probe #950

New issue

Open

opened 2026-10-05 12:56:38 +00:00 by jared · 0 comments

jared commented

2026-10-05 12:56:38 +00:00

Owner

Problem

The behavior-verifier can fail a dispatch with a false claim about the code. Its charter bars it from reading files or running commands, so it states code behavior without proof. A false fail costs extra test-writer rounds and an escalation.

Reproduce

n/a: workflow defect seen in one run; the evidence is the recorded dispatch 260 of implementation 29.

Approach

The charter in plugins/os-sdlc/agents/behavior-verifier.md says: do not read files, do not search, and run no command except the handoff command. Dispatch 260 failed AC 2 and claimed that role_by_const misses constants wrapped in module OsSdlc. That claim is false: NodeOccurrence#nested_namespace in plugins/os-sdlc/lib/os_sdlc/lint/inward_dependencies.rb strips OsSdlc. Its correction scenario also reversed the layer direction. Two extra test-writer rounds and escalation 10 followed.

Fix direction: a criterion may be marked outstanding only with output from a probe the verifier ran, quoted in the item. A claim about code behavior that the diff does not show needs that quote.

Required work:

Change the behavior-verifier charter so an outstanding criterion cites quoted probe output — check: present probe in plugins/os-sdlc/agents/behavior-verifier.md
Remove the charter rule that bars all commands, or narrow it to allow one read-only probe per outstanding criterion — check: absent do not run commands other than in plugins/os-sdlc/agents/behavior-verifier.md

Rejected: Tell the verifier to be more careful — a prose request does not stop an unproven claim.

Acceptance criteria

Given a green increment where the diff meets a criterion, when behavior-verifier cannot show a failing probe for it, then the criterion is not marked outstanding.
Given a criterion marked outstanding, when the handoff is read, then the item quotes the probe output that shows the failure.

Owner sketch

# agents/behavior-verifier.md charter: probe-before-outstanding rule

Origin

Trigger: os-sdlc implementation 29 of ticket #947, dispatch 260 (behavior-verifier fail on AC 2).
Improvised this session: orchestrator rerouted back to behavior-verifier after escalation.
Chain: false fail ← verifier judged code behavior without a probe ← DESIGN (plugins/os-sdlc/agents/behavior-verifier.md bars commands).
Where: plugins/os-sdlc/agents/behavior-verifier.md
Session: ab599e53-dbf5-4953-b8bf-c598a9c5c16e
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-crc-os-adr-2026-10-04/ab599e53-dbf5-4953-b8bf-c598a9c5c16e.jsonl

VERDICT: CREATE

## Problem The behavior-verifier can fail a dispatch with a false claim about the code. Its charter bars it from reading files or running commands, so it states code behavior without proof. A false fail costs extra test-writer rounds and an escalation. ## Reproduce n/a: workflow defect seen in one run; the evidence is the recorded dispatch 260 of implementation 29. ## Approach The charter in `plugins/os-sdlc/agents/behavior-verifier.md` says: do not read files, do not search, and run no command except the handoff command. Dispatch 260 failed AC 2 and claimed that `role_by_const` misses constants wrapped in `module OsSdlc`. That claim is false: `NodeOccurrence#nested_namespace` in `plugins/os-sdlc/lib/os_sdlc/lint/inward_dependencies.rb` strips `OsSdlc`. Its correction scenario also reversed the layer direction. Two extra test-writer rounds and escalation 10 followed. Fix direction: a criterion may be marked outstanding only with output from a probe the verifier ran, quoted in the item. A claim about code behavior that the diff does not show needs that quote. Required work: 1. Change the behavior-verifier charter so an outstanding criterion cites quoted probe output — check: present `probe` in `plugins/os-sdlc/agents/behavior-verifier.md` 2. Remove the charter rule that bars all commands, or narrow it to allow one read-only probe per outstanding criterion — check: absent `do not run commands other than` in `plugins/os-sdlc/agents/behavior-verifier.md` Rejected: Tell the verifier to be more careful — a prose request does not stop an unproven claim. ## Acceptance criteria 1. Given a green increment where the diff meets a criterion, when behavior-verifier cannot show a failing probe for it, then the criterion is not marked outstanding. 2. Given a criterion marked outstanding, when the handoff is read, then the item quotes the probe output that shows the failure. ## Owner sketch ```ruby # agents/behavior-verifier.md charter: probe-before-outstanding rule ``` ## Origin - Trigger: os-sdlc implementation 29 of ticket #947, dispatch 260 (behavior-verifier fail on AC 2). - Improvised this session: orchestrator rerouted back to behavior-verifier after escalation. - Chain: false fail ← verifier judged code behavior without a probe ← DESIGN (plugins/os-sdlc/agents/behavior-verifier.md bars commands). - Where: plugins/os-sdlc/agents/behavior-verifier.md - Session: ab599e53-dbf5-4953-b8bf-c598a9c5c16e - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-crc-os-adr-2026-10-04/ab599e53-dbf5-4953-b8bf-c598a9c5c16e.jsonl VERDICT: CREATE

jared added the

plugin/os-sdlc

filed-by/agent

labels

2026-10-05 12:56:38 +00:00

jared referenced this issue

2026-10-05 12:56:49 +00:00

inward-dependencies lint crashes on reroute/; take roles from a map, not the folder #947

jared added the

category/prose-artifact

label

2026-10-10 13:31:46 +00:00