Split by test examples

caution

Only RSpec is supported, let us know if you use a different test runner.

As an alternative, consider spreading test examples into multiple files.

By default, Knapsack Pro parallelizes tests across CI nodes by example (it/specify). This is useful when you have slow test files but don't want to manually split test examples into smaller test files. You can disable this feature with KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES=false.

As an example, imagine you have two test files in your suite:

Test file path (2 files)	Time execution
spec/controllers/api/v3/books_controller_spec.rb	6 minutes and 30 seconds
spec/features/books_spec.rb	2 minutes and 30 seconds

On your Knapsack Pro dashboard, you can see the yellow highlight because of the bottleneck: if you run those tests on 2 parallel CI nodes, the total execution time would be 6.5 minutes instead of the optimal 4.5 minutes (2.5 minutes + 6.5 minutes divided by 2 CI nodes).

When splitting by test examples, the bottleneck disappears because Knapsack Pro distributes individual tests so that each CI node is balanced (e.g., 4.5 minutes + 4.5 minutes):

Test file path (5 files)	Time execution
spec/controllers/api/v3/books_controller_spec.rb[1:1]	2 minutes
spec/controllers/api/v3/books_controller_spec.rb[1:2]	2 minutes
spec/controllers/api/v3/books_controller_spec.rb[1:3]	2 minutes
spec/controllers/api/v3/books_controller_spec.rb[1:4]	30 seconds
spec/features/books_spec.rb	2 minutes and 30 seconds

Knapsack Pro knows the optimal time per parallel CI node is 4.5 minutes, so it looks for test files slower than 3.15 minutes (70% of 4.5 minutes) and splits them by test examples across parallel CI nodes. We use 70% as a threshold for two reasons:

• Test execution time varies–especially for end-to-end tests–and we want to split all the potential bottlenecks.
• RSpec consumes a lot of memory running individual test examples, so Knapsack Pro only splits potential bottlenecks and parallelizes the rest of your test suite by file.

In other words, files are split by test examples just enough to guarantee all the parallel CI nodes finish at a similar time to maximize performance and minimize memory consumption.

We recommend running at least 2 CI builds after you enable this feature or change the number of CI nodes to allow the Knapsack Pro API to learn about your test suite.

caution

Does not support run_all_when_everything_filtered

Troubleshooting

Why aren’t some test files split by examples?

When many test files are skipped (e.g., with --tag ~skip), the actual CI node time may fall short of estimates, marking fewer test files as slow and unbalancing the CI build.

You can manually set a threshold for splitting test files by examples with KNAPSACK_PRO_SLOW_TEST_FILE_THRESHOLD. This should help with unbalanced CI builds.

Why are some of my test files not executed?

Read the answer on the RSpec page.

How do I fix LoadError: cannot load such file -- MY_RUBY_GEM?

Probably, you load MY_RUBY_GEM in Rakefile so when knapsack_pro runs a rake task it fails. To fix the problem either:

• Don't load MY_RUBY_GEM when RAILS_ENV=test
• Add MY_RUBY_GEM in the :test group in the Gemfile

How do I fix Don't know how to build task 'knapsack_pro:rspec_test_example_detector'?

Try to remove the default prefix bundle exec used by knapsack_pro by setting KNAPSACK_PRO_RSPEC_TEST_EXAMPLE_DETECTOR_PREFIX="".