Revisit test fixture and factory strategy across plugin test suites #1019

Closed
opened 2026-10-16 14:34:05 +00:00 by jared · 7 comments
Owner

Problem

Test fixture practice is scattered across the plugin test suites.
Suites share fixtures per concern, not per domain object, and sharing stops at each plugin boundary.
No lens or convention document prescribes fixture shape.
The only test-lens text on fixtures warns about bare count pins.

Evidence of duplication, from a survey on 2026-10-16:

  • .sdlc/project.yaml with targets is hand-built at 15 sites: 11 os-sdlc tests, 4 os-sdlc tests/support files, and all 4 takedown-eval tests that need it. Meanwhile 48 os-sdlc tests use ProjectScenario.
  • TicketFixture::COMPLETE_BODY exists in 4 copies: tests/support/ticket_fixture.rb:7, command/implementation/open_ticket_shape_test.rb:10, command/implementation/open_map_name_test.rb:8, parser/ticket_shape_parity_test.rb:7.
  • FakeRunner is defined 4 times with different signatures: os-sdlc tests/test_helper.rb:114, takedown test/run_support.rb:12, os-me test/test_helper.rb:72, tests/adapter/probe/lsp_test.rb:12.
  • Target.new(name: is hand-built in 3 files with no helper.
  • The takedown eval test test/command/final_suite_test.rb holds the same two-target Hash in two setup methods (lines 38 and 206).

The one place that matches the per-domain-object model is os-adr: AdrFixtures#with_project and sample_adr(id:, title:, status:, ...).

The takedown eval cannot reach plugins/os-sdlc/tests/support, because ADR-0297 keeps the eval to its own difference from the os-sdlc library. Shared test support has no home that both suites can load.

Reproduce

n/a: design decision ticket; the evidence above comes from a survey, not a failing command.

Approach

This is a decision ticket, not a build ticket. Expected output: a decision memo to Jared, then ADR(s) through /os-adr:create if a one-way door is chosen, then build tickets.

Open questions:

  1. Fixtures or factories: static fixture data versus builders with keyword overrides.
  2. Well-regarded gems (for example FactoryBot and Faker) versus hand-written builders. Judge on Minitest fit, no Rails, and the one-constant-per-file and Zeitwerk conventions already in place.
  3. A standard: one or two default fixtures or factories per domain object, each in its own file, pulled into a test as needed, so most tests stay DRY and readable. Decide where shared test support lives when two suites need it (the ADR-0297 boundary). Decide whether the standard becomes lens text.

Required work: none beyond the acceptance criteria.

Acceptance criteria

  1. Given the survey evidence, when Jared reviews the decision memo, then he rules on questions 1 to 3.
  2. Given a ruling that is a one-way door, when the memo is accepted, then an ADR records it through /os-adr:create.
  3. Given the rulings, when the work is decomposed, then build tickets exist for the standard and for the duplicate sites.

Non-goals

  • Fixing the 15 hand-built sites now.
  • The lint-lesson review.

Origin

  • Trigger: a fixture survey run on 2026-10-16 while ruling on Layout/LineLength Q3 in the lint-lesson review.
  • Improvised this session: none
  • Chain: duplicated hand-built fixtures ← no fixture standard and no shared test-support home ← DESIGN (ADR-0297 boundary).
  • Root candidate: none
  • Where: docs/research/lint-lessons/layout-line-length.md section 10
  • Session: 983d18f5-eb86-409f-a7ca-daea277fedf5
  • Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-lint-lessons/983d18f5-eb86-409f-a7ca-daea277fedf5.jsonl
## Problem Test fixture practice is scattered across the plugin test suites. Suites share fixtures per concern, not per domain object, and sharing stops at each plugin boundary. No lens or convention document prescribes fixture shape. The only test-lens text on fixtures warns about bare count pins. Evidence of duplication, from a survey on 2026-10-16: - `.sdlc/project.yaml` with targets is hand-built at 15 sites: 11 os-sdlc tests, 4 os-sdlc `tests/support` files, and all 4 takedown-eval tests that need it. Meanwhile 48 os-sdlc tests use `ProjectScenario`. - `TicketFixture::COMPLETE_BODY` exists in 4 copies: `tests/support/ticket_fixture.rb:7`, `command/implementation/open_ticket_shape_test.rb:10`, `command/implementation/open_map_name_test.rb:8`, `parser/ticket_shape_parity_test.rb:7`. - `FakeRunner` is defined 4 times with different signatures: os-sdlc `tests/test_helper.rb:114`, takedown `test/run_support.rb:12`, os-me `test/test_helper.rb:72`, `tests/adapter/probe/lsp_test.rb:12`. - `Target.new(name:` is hand-built in 3 files with no helper. - The takedown eval test `test/command/final_suite_test.rb` holds the same two-target Hash in two `setup` methods (lines 38 and 206). The one place that matches the per-domain-object model is os-adr: `AdrFixtures#with_project` and `sample_adr(id:, title:, status:, ...)`. The takedown eval cannot reach `plugins/os-sdlc/tests/support`, because ADR-0297 keeps the eval to its own difference from the os-sdlc library. Shared test support has no home that both suites can load. ## Reproduce n/a: design decision ticket; the evidence above comes from a survey, not a failing command. ## Approach This is a decision ticket, not a build ticket. Expected output: a decision memo to Jared, then ADR(s) through `/os-adr:create` if a one-way door is chosen, then build tickets. Open questions: 1. Fixtures or factories: static fixture data versus builders with keyword overrides. 2. Well-regarded gems (for example FactoryBot and Faker) versus hand-written builders. Judge on Minitest fit, no Rails, and the one-constant-per-file and Zeitwerk conventions already in place. 3. A standard: one or two default fixtures or factories per domain object, each in its own file, pulled into a test as needed, so most tests stay DRY and readable. Decide where shared test support lives when two suites need it (the ADR-0297 boundary). Decide whether the standard becomes lens text. Required work: none beyond the acceptance criteria. ## Acceptance criteria 1. Given the survey evidence, when Jared reviews the decision memo, then he rules on questions 1 to 3. 2. Given a ruling that is a one-way door, when the memo is accepted, then an ADR records it through `/os-adr:create`. 3. Given the rulings, when the work is decomposed, then build tickets exist for the standard and for the duplicate sites. ## Non-goals - Fixing the 15 hand-built sites now. - The lint-lesson review. ## Origin - Trigger: a fixture survey run on 2026-10-16 while ruling on Layout/LineLength Q3 in the lint-lesson review. - Improvised this session: none - Chain: duplicated hand-built fixtures ← no fixture standard and no shared test-support home ← DESIGN (ADR-0297 boundary). - Root candidate: none - Where: docs/research/lint-lessons/layout-line-length.md section 10 - Session: 983d18f5-eb86-409f-a7ca-daea277fedf5 - Transcript: /home/jared/.claude/projects/-home-jared-dev-cc-os--claude-worktrees-lint-lessons/983d18f5-eb86-409f-a7ca-daea277fedf5.jsonl
Author
Owner

Decision memo and implementation plan (2026-10-28)

Rulings by Jared (prediction accepted):

  1. Builders over static constants where a record is costly to write. Where the constructor already defaults every field (e.g. OsSdlc::Project::Target), no fixture.
  2. Custom builders, no FactoryBot, no Faker. Patterns: Object Mother (Schuh/Punke) for named presets; Test Data Builder (Pryce, GOOS ch. 22) / Parameterized Creation Method (Meszaros) for build(**overrides).
  3. The standard lives in tests/support, modules named <Record>Fixture, one constant per file, Zeitwerk-loaded (ADR-0241). Delivery: a ~30-line reference under lens-testing, one link sentence in lens-testing SKILL.md, one index line in the test-writer agent. No cop up front; re-measure after the first pipeline batch that writes tests. Reference stays general so other plugins adopt it unchanged. No ADR: none of these is a one-way door. The ADR-0297 premise in this ticket is stale: 0297 is retired.

Rules (go in the reference):

  1. A fixture returns a value and does no I/O. Scenarios own the disk. Fakes own behavior.
  2. build(**overrides) passes overrides to the real constructor and never re-states a constructor default.
  3. Variants are named methods, never .sub on a returned value.
  4. A test asserts only on values it passed or on a preset's documented property.
  5. One record per file, one constant, Zeitwerk-loaded.

Re-verified evidence 2026-10-28 (os-sdlc): 16 test files hand-build a complete ticket body; nothing references TicketFixture::COMPLETE_BODY; TicketFixture.seed does disk I/O (20 callers); 10 hand-built project.yaml-with-targets sites (6 os-sdlc, 4 takedown eval) while ProjectScenario#write_multi_target_project_yaml exists; three unrelated fakes named FakeRunner.

Measurement (run from plugins/os-sdlc; baseline at d5cf9f96 yaml=10 body=16):

printf 'yaml=%s body=%s\n' "$(grep -rlE '"targets" *=> *\{|^ *targets:$|\\ntargets:|targets:\\n' --include='*.rb' tests eval/ticket-takedown/test | xargs -r grep -l 'project\.yaml' | grep -vc 'support/project_scenario\.rb')" "$(grep -rl '## Problem' --include='*.rb' tests | xargs -r grep -l '## Acceptance criteria' | grep -vc 'support/ticket_fixture\.rb')"

Cop decision rule: after the first pipeline batch that writes tests following ticket 5, re-run. If yaml or body rises above the post-ticket-5 value, file a cop for that record via mine-lints. If flat, no cop; close #1019.

Tickets (filed via capture; numbers in the follow-up comment):

  1. Record-fixture reference + lens link + test-writer index line (by hand).
  2. Split TicketFixture into value-only fixture (complete_body, without(section), with_required_work(item)) and TicketScenario owning the disk writes (fast-batch; depends on 1).
  3. Move HomeAndStart out of project_scenario.rb (fast-batch).
  4. Replace 16 hand-built bodies and 6 hand-built project.yaml Hashes in os-sdlc tests (fast-batch; depends on 2, 3).
  5. Takedown eval require_relatives home_and_start.rb + project_scenario.rb and replaces its 4 hand-built Hashes (fast-batch; depends on 3).

Out of scope follow-ups: rename the three FakeRunner fakes; audit other *_fixture.rb for I/O (git_repo_fixture, changed_project_fixture, gate_commands_project_fixture, open_implementation_fixture); convert os-adr, os-me, os-backlog; finding-builder index line if builders keep hand-building records.

## Decision memo and implementation plan (2026-10-28) Rulings by Jared (prediction accepted): 1. Builders over static constants where a record is costly to write. Where the constructor already defaults every field (e.g. `OsSdlc::Project::Target`), no fixture. 2. Custom builders, no FactoryBot, no Faker. Patterns: Object Mother (Schuh/Punke) for named presets; Test Data Builder (Pryce, GOOS ch. 22) / Parameterized Creation Method (Meszaros) for `build(**overrides)`. 3. The standard lives in `tests/support`, modules named `<Record>Fixture`, one constant per file, Zeitwerk-loaded (ADR-0241). Delivery: a ~30-line reference under lens-testing, one link sentence in lens-testing SKILL.md, one index line in the test-writer agent. No cop up front; re-measure after the first pipeline batch that writes tests. Reference stays general so other plugins adopt it unchanged. No ADR: none of these is a one-way door. The ADR-0297 premise in this ticket is stale: 0297 is retired. Rules (go in the reference): 1. A fixture returns a value and does no I/O. Scenarios own the disk. Fakes own behavior. 2. `build(**overrides)` passes overrides to the real constructor and never re-states a constructor default. 3. Variants are named methods, never `.sub` on a returned value. 4. A test asserts only on values it passed or on a preset's documented property. 5. One record per file, one constant, Zeitwerk-loaded. Re-verified evidence 2026-10-28 (os-sdlc): 16 test files hand-build a complete ticket body; nothing references `TicketFixture::COMPLETE_BODY`; `TicketFixture.seed` does disk I/O (20 callers); 10 hand-built project.yaml-with-targets sites (6 os-sdlc, 4 takedown eval) while `ProjectScenario#write_multi_target_project_yaml` exists; three unrelated fakes named `FakeRunner`. Measurement (run from plugins/os-sdlc; baseline at d5cf9f96 `yaml=10 body=16`): ``` printf 'yaml=%s body=%s\n' "$(grep -rlE '"targets" *=> *\{|^ *targets:$|\\ntargets:|targets:\\n' --include='*.rb' tests eval/ticket-takedown/test | xargs -r grep -l 'project\.yaml' | grep -vc 'support/project_scenario\.rb')" "$(grep -rl '## Problem' --include='*.rb' tests | xargs -r grep -l '## Acceptance criteria' | grep -vc 'support/ticket_fixture\.rb')" ``` Cop decision rule: after the first pipeline batch that writes tests following ticket 5, re-run. If `yaml` or `body` rises above the post-ticket-5 value, file a cop for that record via mine-lints. If flat, no cop; close #1019. Tickets (filed via capture; numbers in the follow-up comment): 1. Record-fixture reference + lens link + test-writer index line (by hand). 2. Split `TicketFixture` into value-only fixture (`complete_body`, `without(section)`, `with_required_work(item)`) and `TicketScenario` owning the disk writes (fast-batch; depends on 1). 3. Move `HomeAndStart` out of `project_scenario.rb` (fast-batch). 4. Replace 16 hand-built bodies and 6 hand-built project.yaml Hashes in os-sdlc tests (fast-batch; depends on 2, 3). 5. Takedown eval `require_relative`s `home_and_start.rb` + `project_scenario.rb` and replaces its 4 hand-built Hashes (fast-batch; depends on 3). Out of scope follow-ups: rename the three `FakeRunner` fakes; audit other `*_fixture.rb` for I/O (`git_repo_fixture`, `changed_project_fixture`, `gate_commands_project_fixture`, `open_implementation_fixture`); convert os-adr, os-me, os-backlog; finding-builder index line if builders keep hand-building records.
Author
Owner

Filed from the 1019 memo: #1458 Add record-fixture reference, lens link, and test-writer index line; #1459 Split TicketFixture into a value-only fixture and TicketScenario; #1460 Move HomeAndStart out of project_scenario.rb into its own file; #1461 Replace hand-built ticket bodies and project.yaml Hashes in os-sdlc tests with fixtures. The fifth ticket (Use ProjectScenario in the takedown eval tests) is held: the skeptic found open #1011 with the same mechanism.

Filed from the 1019 memo: #1458 Add record-fixture reference, lens link, and test-writer index line; #1459 Split TicketFixture into a value-only fixture and TicketScenario; #1460 Move HomeAndStart out of project_scenario.rb into its own file; #1461 Replace hand-built ticket bodies and project.yaml Hashes in os-sdlc tests with fixtures. The fifth ticket (Use ProjectScenario in the takedown eval tests) is held: the skeptic found open #1011 with the same mechanism.
Author
Owner

fast-batch-21 shipped on main c74e1bde: #1460 #1459 #1461 closed. Measurement after the batch (from plugins/os-sdlc): yaml=4 (all in eval/ticket-takedown, #1011 occurrence) body=3 (kept parser-input files only). Baseline was yaml=10 body=16. Next: after the first pipeline batch that writes tests, re-run the measurement; if either count rises, file a cop for that record via mine-lints; if flat, close this ticket.

fast-batch-21 shipped on main c74e1bde: #1460 #1459 #1461 closed. Measurement after the batch (from plugins/os-sdlc): yaml=4 (all in eval/ticket-takedown, #1011 occurrence) body=3 (kept parser-input files only). Baseline was yaml=10 body=16. Next: after the first pipeline batch that writes tests, re-run the measurement; if either count rises, file a cop for that record via mine-lints; if flat, close this ticket.
Author
Owner

Conversion tickets filed: #1463 Rename the nine I/O-doing *_fixture.rb support files to scenarios; #1464 WorkOrderFixture.build for the 73 tests that hand-build WorkOrder; #1465 Record fixtures for ProcessIO, Finding, Signal, StepOutcome, SourceTree, Location; #1466 Consolidate git identity, git init, and rubocop AllCops literals in tests; #1467 Split AdrFixtures into AdrFixture (value) and AdrScenario (disk); #1468 Rename the two os-sdlc FakeRunner fakes after their collaborator. Order: #1463 → #1466; #1464 → #1465; #1467, #1468 independent. Held: the os-me Zeitwerk and FakeRunner ticket (skeptic verdict OCCURRENCE 1019).

Conversion tickets filed: #1463 Rename the nine I/O-doing *_fixture.rb support files to scenarios; #1464 WorkOrderFixture.build for the 73 tests that hand-build WorkOrder; #1465 Record fixtures for ProcessIO, Finding, Signal, StepOutcome, SourceTree, Location; #1466 Consolidate git identity, git init, and rubocop AllCops literals in tests; #1467 Split AdrFixtures into AdrFixture (value) and AdrScenario (disk); #1468 Rename the two os-sdlc FakeRunner fakes after their collaborator. Order: #1463 → #1466; #1464 → #1465; #1467, #1468 independent. Held: the os-me Zeitwerk and FakeRunner ticket (skeptic verdict OCCURRENCE 1019).
Author
Owner

os-me conversion filed as #1469 (user-approved; skeptic had marked it an occurrence of this ruling ticket, which holds no build work). Full conversion set: #1463 → #1466, #1464 → #1465, #1467 (os-adr, by hand), #1468, #1469 (os-me, by hand). Re-measure step still pending the first test-writing pipeline batch.

os-me conversion filed as #1469 (user-approved; skeptic had marked it an occurrence of this ruling ticket, which holds no build work). Full conversion set: #1463 → #1466, #1464 → #1465, #1467 (os-adr, by hand), #1468, #1469 (os-me, by hand). Re-measure step still pending the first test-writing pipeline batch.
Author
Owner

Resolution

Done: Re-measure done; no cop (ruling M1, predicted 72%). The convention shipped (#1458) and every conversion ticket is closed (#1463 #1464 #1465 #1466 #1467 #1468 #1469 #1269, #1011 occurrence).

Evidence: Main 2fe1b7f0, measurement from plugins/os-sdlc: yaml=0 body=4. The post-fast-batch-21 baseline on main was yaml=4 body=4: fast-batch-21 measured body=3 on its branch, before #1450 (5ccc305f, pre-convention) added tests/implementation/acceptance_test.rb BODY. #1453 (7246f609), the first test-writing batch after the convention, left the counts flat. The 4 body files are kept literals that are input under test; acceptance_test.rb was checked in fast-batch-22 fix 9 (no TicketFixture call yields typed required work plus a Retired tests section; ruling L1b). Note: M1 assumed body=3 after L1; the count is still flat. Rulings: docs/crc/os-sdlc/fast-batch-22/rulings.tsv, memo rulings/M1.md.

Follow-ups: none (ruling M1: no re-run habit)

## Resolution **Done:** Re-measure done; no cop (ruling M1, predicted 72%). The convention shipped (#1458) and every conversion ticket is closed (#1463 #1464 #1465 #1466 #1467 #1468 #1469 #1269, #1011 occurrence). **Evidence:** Main 2fe1b7f0, measurement from plugins/os-sdlc: yaml=0 body=4. The post-fast-batch-21 baseline on main was yaml=4 body=4: fast-batch-21 measured body=3 on its branch, before #1450 (5ccc305f, pre-convention) added tests/implementation/acceptance_test.rb BODY. #1453 (7246f609), the first test-writing batch after the convention, left the counts flat. The 4 body files are kept literals that are input under test; acceptance_test.rb was checked in fast-batch-22 fix 9 (no TicketFixture call yields typed required work plus a Retired tests section; ruling L1b). Note: M1 assumed body=3 after L1; the count is still flat. Rulings: docs/crc/os-sdlc/fast-batch-22/rulings.tsv, memo rulings/M1.md. **Follow-ups:** none (ruling M1: no re-run habit)
jared closed this issue 2026-10-28 19:00:30 +00:00
Author
Owner

Skeptic DROP for follow-up 'os-sdlc test-support leftovers after fast-batch-22': nothing fails today, no observed failure (speculative-hardening test). Leftovers, fix on next touch: redundant require_relative "cop_helpers" in plugins/os-sdlc/tests/support/rescue_source.rb; SeededGitRepoHelpers#run_git ignores exit status (GitRepoScenario#run_git! raises); CrcLedgerScenario has class and instance methods both named write.

Skeptic DROP for follow-up 'os-sdlc test-support leftovers after fast-batch-22': nothing fails today, no observed failure (speculative-hardening test). Leftovers, fix on next touch: redundant require_relative "cop_helpers" in plugins/os-sdlc/tests/support/rescue_source.rb; SeededGitRepoHelpers#run_git ignores exit status (GitRepoScenario#run_git! raises); CrcLedgerScenario has class and instance methods both named write.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
jared/cc-os#1019
No description provided.