jared/cc-os

Watch

Fork 0

Gate output in briefs is cut at 4000 chars, so test-reviewer cannot see every red test #670

New issue

Closed

opened 2026-09-24 13:41:43 +00:00 by jared · 2 comments

jared commented

2026-09-24 13:41:43 +00:00

Owner

Problem

BriefFacts caps the diff-test gate output at MAX_GATE_OUTPUT_CHARS = 4000
(plugins/os-sdlc/lib/os_sdlc/pipeline/brief_facts.rb:8). A ticket with many
acceptance criteria produces more red output than that. The test-reviewer
then cannot confirm which new tests are red, and it adds a GAP item every
round instead of a verdict it can stand behind.

On #655 implementation 19 (2026-09-24), the test-reviewer reported in three
consecutive rounds (dispatches 216, 218, 220):

FAILURE: diff-test output is truncated (10 of 22 red entries visible).
GAP: gate_results diff-test output is truncated at 4000 of 14380 chars.

Gate output is truncated at 4000 of 9186 chars, so red status of the AC3-

AC5, AC7, AC9, AC15, AC18, AC19, AC20 tests is not visible

Output is truncated at 4000 of 8954 chars, so the status of the remaining

tests is not visible.

Test-reviewer, last 30 days: 22 pass / 25 fail / 1 error, 12 of the fails
mention truncation.

The largest stored diff-test output on record (dispatch 240) is 24,627
chars. Of that, stack-trace lines (---Backtrace--- blocks and lines
matching \S+:\d+:in ) are 71% of the bytes; the 24 N) Failure:/N) Error: headers together are 292 chars. The reviewer needs the header plus
message per failure, not the trace.

Approach

Required work:

Write OsSdlc::Pipeline::RedTests in plugins/os-sdlc/lib/os_sdlc/pipeline/red_tests.rb.
RedTests.parse(output) returns one string per N) Failure: / N) Error: block, using this regex sketch.
BLOCK = /^\s*\d+\) (Failure|Error):\n(.+?)\n(.*?)(?=^\s+\S+:\d+:in |^\s*\d+\) |\z)/m
entries = output.scan(BLOCK).map { |kind, name, message| "#{kind[0]} #{name.strip}\n#{message.strip}" }
Add "red_tests" => :red_tests to FACT_RESOLVERS in plugins/os-sdlc/lib/os_sdlc/pipeline/brief_facts.rb.
Make the resolver return the latest dispatch's step summaries transformed to { step => RedTests.parse(summary["output"]) }.
Remove MAX_GATE_OUTPUT_CHARS and truncate from BriefFacts.
Add GATE_OUTPUT_WARN_CHARS = 20_000 and a sized(text) helper that returns text unchanged at or under the line.
Over the line, sized prefixes one line, "BLOAT: gate output is N chars, over the 20000 watch line. Report it as a BLOAT item in your handoff.", followed by the whole text.
Apply sized at the gate_results and lint_offense_history call sites; drop no character of either fact.
Edit plugins/os-sdlc/maps/poodr-implementation.yaml and plugins/os-sdlc/maps/poodr-ticket-implementation.yaml so the test-reviewer node's enrich list uses red_tests in place of gate_results.
Edit plugins/os-sdlc/agents/test-reviewer.md: replace "A test named in an Error: entry of gate_results is red. A test named in a Failure: entry is red. Reject a test as unexpectedly green only when it appears in neither." with "A test named in a red_tests entry is red; E marks an error, F a failure. Reject a test as unexpectedly green only when no entry names it."
In the same file, change the "If test_changes or gate_results is absent or empty" line to name red_tests instead of gate_results.
Replace plugins/os-sdlc/tests/pipeline/brief_facts_gate_results_truncation_marker_test.rb (asserts TRUNCATED) with BLOAT-prefix coverage.
Add a RedTests test using the Fixture blocks below.
Add a map test asserting both maps' test-reviewer enrich lists contain red_tests and not gate_results.
Update any map fixture test that lists the enrich set.
Run bin/refresh-plugins after the charter and map edits — check: bin/refresh-plugins

Rejected: persisting red_tests as a step_results column — a SCHEMA_VERSION bump archives and recreates the live pipeline.db (db.rb archive_and_recreate), and the parse is a pure function of the stored output, so no schema change is needed.

Acceptance criteria

Given the dispatch-240 fixture output, when OsSdlc::Pipeline::RedTests.parse runs against it, then it returns 24 entries and no entry contains a backtrace line.
Given the codebase after this change, when BriefFacts::FACTS is inspected, then it includes red_tests, and both map files validate without UnknownFact or InvalidMapError.
Given any BriefFacts fact call path, when its output is inspected, then it never emits a TRUNCATED marker and never drops a character of the underlying text.
Given a gate_results value over 20,000 chars, when BriefFacts builds the fact, then the value carries the BLOAT: prefix line and the full text follows it.
Given plugins/os-sdlc/agents/test-reviewer.md, when its red-status rule is read, then it names red_tests and does not name gate_results.
Given the stored dispatch-240 output, when it is parsed into the red_tests fact, then all 24 of its failure/error test names appear in the fact.

Owner sketch

class OsSdlc::Pipeline::RedTests
  def self.parse(output)
end

class OsSdlc::Pipeline::BriefFacts
  def red_tests
  def gate_results
  def sized(text)
end

Non-goals

Persisting the parsed list: the step_results.output column is unchanged and nothing new is stored.
The programmer node keeps gate_results in both maps.
plugins/os-sdlc/tests/pipeline/lint_offense_line_test.rb: its 200-char cut belongs to LintOffenseLine, not MAX_GATE_OUTPUT_CHARS.
#630
#662
The enrich audit (#676)
The amending ADR for #630/#662/#670, to be written once after all three land, outside this run
Persisting the parsed red_tests list

Fixture

Pulled from .sdlc/pipeline.db, dispatch 240, step diff-test, by running
the design's BLOCK regex against the stored output column. It found 24
blocks; first Failure and first Error shown here with traces trimmed to 3
lines:

First Failure block:

1) Failure:
OsSdlc::Pipeline::CompletionIntakeDispatchOwnershipTest#test_a_subagent_stop_for_a_dispatch_owned_by_another_implementation_is_rejected [plugins/os-sdlc/tests/pipeline/completion_intake_dispatch_ownership_test.rb:36]:
[OsSdlc::Model::Implementation::StaleDispatch] exception expected, not
Class: <NoMethodError>
Message: <"undefined method 'db' for nil">
---Backtrace---
/home/jared/dev/cc-os/.claude/worktrees/ticket-657/plugins/os-sdlc/lib/os_sdlc/pipeline/completion_intake.rb:21:in 'OsSdlc::Pipeline::CompletionIntake#initialize'
/home/jared/dev/cc-os/.claude/worktrees/ticket-657/plugins/os-sdlc/lib/os_sdlc/pipeline/completion_intake.rb:16:in 'OsSdlc::Pipeline::CompletionIntake.call'
/home/jared/dev/cc-os/.claude/worktrees/ticket-657/plugins/os-sdlc/tests/pipeline/completion_intake_dispatch_ownership_test.rb:36:in '...'
---------------

A later Error block:

N) Error:
OsSdlc::ProjectTestsForTest#test_tests_for_returns_nil_for_an_unknown_file:
ArgumentError: missing keyword: :path

The parser's own output for this file (RedTests.entries, message clipped
at the first stack-trace line) is:

F OsSdlc::Pipeline::CompletionIntakeDispatchOwnershipTest#test_a_subagent_stop_for_a_dispatch_owned_by_another_implementation_is_rejected [plugins/os-sdlc/tests/pipeline/completion_intake_dispatch_ownership_test.rb:36]:
[OsSdlc::Model::Implementation::StaleDispatch] exception expected, not
Class: <NoMethodError>
Message: <"undefined method 'db' for nil">

E OsSdlc::ProjectTestsForTest#test_tests_for_returns_nil_for_an_unknown_file:
ArgumentError: missing keyword: :path

Use this file (or dispatch 240's full output) as the fixture for
RedTests parser tests; it is real gate output, not synthesized.

Origin

Trigger: /os-sdlc:implement 655, dispatches 216, 218, 220
Improvised this session: none; a fourth test-writer round was dispatched
with the four open items named in the prompt
Chain: reviewer GAP every round ← gate evidence cut at 4000 chars ←
BriefFacts::MAX_GATE_OUTPUT_CHARS (DESIGN, brief_facts.rb:8; ADR-0206 caps
the hook payload, not the brief-fetch path)
Root candidate: this ticket
Where: OsSdlc::Pipeline::BriefFacts, plugins/os-
sdlc/lib/os_sdlc/pipeline/brief_facts.rb
Session: 557d47dd-277f-4747-a5a1-56dc14e19cf5
Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/557d47dd-
277f-4747-a5a1-56dc14e19cf5.jsonl

Skeptic gate: not run; the running goal (session 557d47dd) directs a follow-
up ticket for each pipeline defect.

## Problem BriefFacts caps the diff-test gate output at MAX_GATE_OUTPUT_CHARS = 4000 (plugins/os-sdlc/lib/os_sdlc/pipeline/brief_facts.rb:8). A ticket with many acceptance criteria produces more red output than that. The test-reviewer then cannot confirm which new tests are red, and it adds a GAP item every round instead of a verdict it can stand behind. On #655 implementation 19 (2026-09-24), the test-reviewer reported in three consecutive rounds (dispatches 216, 218, 220): FAILURE: diff-test output is truncated (10 of 22 red entries visible). GAP: gate_results diff-test output is truncated at 4000 of 14380 chars. Gate output is truncated at 4000 of 9186 chars, so red status of the AC3- AC5, AC7, AC9, AC15, AC18, AC19, AC20 tests is not visible Output is truncated at 4000 of 8954 chars, so the status of the remaining tests is not visible. Test-reviewer, last 30 days: 22 pass / 25 fail / 1 error, 12 of the fails mention truncation. The largest stored diff-test output on record (dispatch 240) is 24,627 chars. Of that, stack-trace lines (`---Backtrace---` blocks and lines matching `\S+:\d+:in `) are 71% of the bytes; the 24 `N) Failure:`/`N) Error:` headers together are 292 chars. The reviewer needs the header plus message per failure, not the trace. ## Approach Required work: 1. Write `OsSdlc::Pipeline::RedTests` in plugins/os-sdlc/lib/os_sdlc/pipeline/red_tests.rb. 2. `RedTests.parse(output)` returns one string per `N) Failure:` / `N) Error:` block, using this regex sketch. `BLOCK = /^\s*\d+\) (Failure|Error):\n(.+?)\n(.*?)(?=^\s+\S+:\d+:in |^\s*\d+\) |\z)/m` `entries = output.scan(BLOCK).map { |kind, name, message| "#{kind[0]} #{name.strip}\n#{message.strip}" }` 3. Add `"red_tests" => :red_tests` to FACT_RESOLVERS in plugins/os-sdlc/lib/os_sdlc/pipeline/brief_facts.rb. 4. Make the resolver return the latest dispatch's step summaries transformed to `{ step => RedTests.parse(summary["output"]) }`. 5. Remove MAX_GATE_OUTPUT_CHARS and truncate from BriefFacts. 6. Add `GATE_OUTPUT_WARN_CHARS = 20_000` and a `sized(text)` helper that returns text unchanged at or under the line. 7. Over the line, `sized` prefixes one line, "BLOAT: gate output is N chars, over the 20000 watch line. Report it as a BLOAT item in your handoff.", followed by the whole text. 8. Apply `sized` at the gate_results and lint_offense_history call sites; drop no character of either fact. 9. Edit plugins/os-sdlc/maps/poodr-implementation.yaml and plugins/os-sdlc/maps/poodr-ticket-implementation.yaml so the test-reviewer node's enrich list uses red_tests in place of gate_results. 10. Edit plugins/os-sdlc/agents/test-reviewer.md: replace "A test named in an `Error:` entry of `gate_results` is red. A test named in a `Failure:` entry is red. Reject a test as unexpectedly green only when it appears in neither." with "A test named in a `red_tests` entry is red; `E` marks an error, `F` a failure. Reject a test as unexpectedly green only when no entry names it." 11. In the same file, change the "If `test_changes` or `gate_results` is absent or empty" line to name `red_tests` instead of `gate_results`. 12. Replace plugins/os-sdlc/tests/pipeline/brief_facts_gate_results_truncation_marker_test.rb (asserts TRUNCATED) with BLOAT-prefix coverage. 13. Add a RedTests test using the Fixture blocks below. 14. Add a map test asserting both maps' test-reviewer enrich lists contain red_tests and not gate_results. 15. Update any map fixture test that lists the enrich set. 16. Run bin/refresh-plugins after the charter and map edits — check: `bin/refresh-plugins` Rejected: persisting red_tests as a step_results column — a SCHEMA_VERSION bump archives and recreates the live pipeline.db (db.rb archive_and_recreate), and the parse is a pure function of the stored output, so no schema change is needed. ## Acceptance criteria 1. Given the dispatch-240 fixture output, when `OsSdlc::Pipeline::RedTests.parse` runs against it, then it returns 24 entries and no entry contains a backtrace line. 2. Given the codebase after this change, when `BriefFacts::FACTS` is inspected, then it includes `red_tests`, and both map files validate without `UnknownFact` or `InvalidMapError`. 3. Given any `BriefFacts` fact call path, when its output is inspected, then it never emits a `TRUNCATED` marker and never drops a character of the underlying text. 4. Given a `gate_results` value over 20,000 chars, when `BriefFacts` builds the fact, then the value carries the `BLOAT:` prefix line and the full text follows it. 5. Given `plugins/os-sdlc/agents/test-reviewer.md`, when its red-status rule is read, then it names `red_tests` and does not name `gate_results`. 6. Given the stored dispatch-240 output, when it is parsed into the `red_tests` fact, then all 24 of its failure/error test names appear in the fact. ## Owner sketch ```ruby class OsSdlc::Pipeline::RedTests def self.parse(output) end class OsSdlc::Pipeline::BriefFacts def red_tests def gate_results def sized(text) end ``` ## Non-goals - Persisting the parsed list: the step_results.output column is unchanged and nothing new is stored. - The programmer node keeps gate_results in both maps. - plugins/os-sdlc/tests/pipeline/lint_offense_line_test.rb: its 200-char cut belongs to LintOffenseLine, not MAX_GATE_OUTPUT_CHARS. - #630 - #662 - The enrich audit (#676) - The amending ADR for #630/#662/#670, to be written once after all three land, outside this run - Persisting the parsed red_tests list ## Fixture Pulled from `.sdlc/pipeline.db`, dispatch 240, step `diff-test`, by running the design's `BLOCK` regex against the stored `output` column. It found 24 blocks; first Failure and first Error shown here with traces trimmed to 3 lines: First Failure block: ``` 1) Failure: OsSdlc::Pipeline::CompletionIntakeDispatchOwnershipTest#test_a_subagent_stop_for_a_dispatch_owned_by_another_implementation_is_rejected [plugins/os-sdlc/tests/pipeline/completion_intake_dispatch_ownership_test.rb:36]: [OsSdlc::Model::Implementation::StaleDispatch] exception expected, not Class: <NoMethodError> Message: <"undefined method 'db' for nil"> ---Backtrace--- /home/jared/dev/cc-os/.claude/worktrees/ticket-657/plugins/os-sdlc/lib/os_sdlc/pipeline/completion_intake.rb:21:in 'OsSdlc::Pipeline::CompletionIntake#initialize' /home/jared/dev/cc-os/.claude/worktrees/ticket-657/plugins/os-sdlc/lib/os_sdlc/pipeline/completion_intake.rb:16:in 'OsSdlc::Pipeline::CompletionIntake.call' /home/jared/dev/cc-os/.claude/worktrees/ticket-657/plugins/os-sdlc/tests/pipeline/completion_intake_dispatch_ownership_test.rb:36:in '...' --------------- ``` A later Error block: ``` N) Error: OsSdlc::ProjectTestsForTest#test_tests_for_returns_nil_for_an_unknown_file: ArgumentError: missing keyword: :path ``` The parser's own output for this file (`RedTests.entries`, message clipped at the first stack-trace line) is: ``` F OsSdlc::Pipeline::CompletionIntakeDispatchOwnershipTest#test_a_subagent_stop_for_a_dispatch_owned_by_another_implementation_is_rejected [plugins/os-sdlc/tests/pipeline/completion_intake_dispatch_ownership_test.rb:36]: [OsSdlc::Model::Implementation::StaleDispatch] exception expected, not Class: <NoMethodError> Message: <"undefined method 'db' for nil"> E OsSdlc::ProjectTestsForTest#test_tests_for_returns_nil_for_an_unknown_file: ArgumentError: missing keyword: :path ``` Use this file (or dispatch 240's full output) as the fixture for `RedTests` parser tests; it is real gate output, not synthesized. ## Origin * Trigger: /os-sdlc:implement 655, dispatches 216, 218, 220 * Improvised this session: none; a fourth test-writer round was dispatched with the four open items named in the prompt * Chain: reviewer GAP every round ← gate evidence cut at 4000 chars ← BriefFacts::MAX_GATE_OUTPUT_CHARS (DESIGN, brief_facts.rb:8; ADR-0206 caps the hook payload, not the brief-fetch path) * Root candidate: this ticket * Where: OsSdlc::Pipeline::BriefFacts, plugins/os- sdlc/lib/os_sdlc/pipeline/brief_facts.rb * Session: 557d47dd-277f-4747-a5a1-56dc14e19cf5 * Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os/557d47dd- 277f-4747-a5a1-56dc14e19cf5.jsonl Skeptic gate: not run; the running goal (session 557d47dd) directs a follow- up ticket for each pipeline defect.

jared added the

plugin/os-sdlc

filed-by/agent

labels

2026-09-24 13:41:43 +00:00

jared referenced this issue

2026-09-24 14:13:34 +00:00

code.path is the target path; no lib/ default #655

jared referenced this issue

2026-09-24 14:30:28 +00:00

code.path is the target path; no lib/ default #655

jared referenced this issue

2026-09-24 15:44:24 +00:00

CLI invocation seam + in-process CLI tests #631

jared referenced this issue

2026-09-24 15:55:42 +00:00

os-sdlc: read-only judgment charters (behavior-verifier and peers) fail with verdict:error when the brief exceeds the hook cap #662

jared referenced this issue

2026-09-24 15:56:01 +00:00

Audit each os-sdlc map node's enrich facts against what its role acts on #676

jared referenced this issue

2026-09-24 16:03:49 +00:00

HookPayload fallback can still exceed cap on large handoffs #630

jared referenced this issue

2026-09-24 16:12:08 +00:00

HookPayload fallback can still exceed cap on large handoffs #630

jared referenced this issue

2026-09-24 16:18:51 +00:00

os-sdlc: read-only judgment charters (behavior-verifier and peers) fail with verdict:error when the brief exceeds the hook cap #662

jared commented

2026-09-24 16:18:59 +00:00

Author

Owner

Work started on branch ticket-670 via /os-sdlc:implement.

jared commented

2026-09-24 16:38:30 +00:00

Author

Owner

Resolution

Done: RedTests parses Minitest output into one entry per red test (kind, name, location, message; no traces). red_tests is a BriefFacts fact; the test-reviewer node enriches with it in place of gate_results in both maps. MAX_GATE_OUTPUT_CHARS truncation removed; facts over 20,000 chars carry a BLOAT warning prefix and stay whole.

Evidence: main ed5073f7 via os-sdlc implementation 23: 8 dispatches, 0 retries, all pass; RedTests parses the dispatch-240 fixture to 24 entries; suite green (dispatch 263); worktree finish refreshed plugins.

Follow-ups: Amending ADR for ADR-0204 (gate cap) and ADR-0206 (envelope, seven charters) written next in this session. Enrich audit is #676 (blocked by this ticket, now unblocked). No other follow-ups.

## Resolution **Done:** RedTests parses Minitest output into one entry per red test (kind, name, location, message; no traces). red_tests is a BriefFacts fact; the test-reviewer node enriches with it in place of gate_results in both maps. MAX_GATE_OUTPUT_CHARS truncation removed; facts over 20,000 chars carry a BLOAT warning prefix and stay whole. **Evidence:** main ed5073f7 via os-sdlc implementation 23: 8 dispatches, 0 retries, all pass; RedTests parses the dispatch-240 fixture to 24 entries; suite green (dispatch 263); worktree finish refreshed plugins. **Follow-ups:** Amending ADR for ADR-0204 (gate cap) and ADR-0206 (envelope, seven charters) written next in this session. Enrich audit is #676 (blocked by this ticket, now unblocked). No other follow-ups.

jared closed this issue

2026-09-24 16:38:30 +00:00

jared referenced this issue from a commit

2026-09-24 16:58:03 +00:00

feat(os-sdlc): red_tests fact replaces gate_results for the test-reviewer; gate output warns at 20k chars instead of truncating at 4k (#670)

jared referenced this issue

2026-09-24 17:11:05 +00:00

reviewer receives full gate output it is told not to re-check; trim gate_results to gate verdicts #679

jared referenced this issue

2026-09-24 17:11:11 +00:00

TEST ISSUE for debug #685

jared referenced this issue