Using Knapsack Pro with RSpec
Retry failed/flaky tests
You have a couple of options:
- rspec-retry (recommended)
- RSpec's
--only-failures
bundle exec rake knapsack_pro:queue:rspec
bundle exec rspec --only-failures
--fail-fast
You can use the Rake argument syntax to fail fast:
# Stop when 1 test failed
bundle exec rake "knapsack_pro:queue:rspec[--fail-fast]"
# Stop when 3 tests failed
bundle exec rake "knapsack_pro:queue:rspec[--fail-fast 3]"
Run a subset of tests
To run a subset of your test suite you have a couple of options:
KNAPSACK_PRO_TEST_FILE_*
environment variables (recommended)- RSpec's
--tag MY_TAG
,--tag ~MY_TAG
,--tag type:feature
, or--tag ~type:feature
If you are seeking faster performance on your CI, you may want to read Parallelize test examples (instead of files)
Parallelize test examples (instead of files)
Formatters (rspec_junit_formatter
, json
)
Format stdout with the documentation
formatter and file output with any RSpec supported formatter.
Regular Mode
- JUnit XML
- JSON
You need to install the rspec_junit_formatter
gem.
bundle exec rake "knapsack_pro:rspec[--format documentation --format RspecJunitFormatter --out tmp/rspec.xml]"
bundle exec rake "knapsack_pro:rspec[--format documentation --format json --out tmp/rspec.json]"
Queue Mode
- JUnit XML
- JSON
You need to install the rspec_junit_formatter
gem.
# Refer to your CI docs for `$MY_CI_NODE_INDEX`
export KNAPSACK_PRO_CI_NODE_INDEX=$MY_CI_NODE_INDEX
bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format RspecJunitFormatter --out tmp/rspec_$KNAPSACK_PRO_CI_NODE_INDEX.xml]"
# spec_helper.rb or rails_helper.rb
# `TMP_REPORT` must be the same path as `--out`
# `TMP_REPORT` must be a full path (no `~`)
TMP_REPORT = "tmp/tmp_rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.xml"
FINAL_REPORT = "tmp/final_rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.xml"
KnapsackPro::Hooks::Queue.after_subset_queue do |queue_id, subset_queue_id|
if File.exist?(TMP_REPORT)
FileUtils.mv(TMP_REPORT, FINAL_REPORT)
end
end
FINAL_REPORT
will contain all the tests run on the CI node (not just the last subset). For more information, you can read this Github issue.
If your CI nodes write to the same disk, you need to append the CI node index to the solution presented above to avoid conflicts:
TMP_REPORT = "tmp/tmp_rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.xml"
FINAL_REPORT = "tmp/final_rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.xml"
This applies also if you are running parallel test processes on each CI node (see our page on to integrate Knapsack Pro with parallel_tests
for an example).
# Refer to your CI docs for `$MY_CI_NODE_INDEX`
export KNAPSACK_PRO_CI_NODE_INDEX=$MY_CI_NODE_INDEX
bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format json --out tmp/rspec_$KNAPSACK_PRO_CI_NODE_INDEX.json]"
# spec_helper.rb or rails_helper.rb
# `TMP_REPORT` must be the same path as `--out`
# `TMP_REPORT` must be a full path (no `~`)
TMP_REPORT = "tmp/tmp_rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.json"
FINAL_REPORT = "tmp/final_rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.json"
KnapsackPro::Hooks::Queue.after_subset_queue do |queue_id, subset_queue_id|
if File.exist?(TMP_REPORT)
FileUtils.mv(TMP_REPORT, FINAL_REPORT)
end
end
FINAL_REPORT
will contain all the tests run on the CI node (not just the last subset). For more information, you can read this Github issue.
If your CI nodes write to the same disk, you need to append the CI node index to the solution presented above to avoid conflicts:
TMP_REPORT = "tmp/tmp_rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.json"
FINAL_REPORT = "tmp/final_rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.json"
This applies also if you are running parallel test processes on each CI node (see our page on to integrate Knapsack Pro with parallel_tests
for an example).
Rswag gem configuration
If you use the Rswag gem that extends rspec-rails "request specs" with a Swagger-based DSL, then you can configure Knapsack Pro to run Rswag tests with the Rswag formatter (Rswag::Specs::SwaggerFormatter
) and run other tests without it.
Run Rswag tests with KNAPSACK_PRO_TEST_FILE_PATTERN
:
export KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=api-token-for-rswag-tests
export KNAPSACK_PRO_TEST_FILE_PATTERN={spec/requests/**/*_spec.rb,spec/api/**/*_spec.rb,spec/integration/**/*_spec.rb}
bundle exec rake "knapsack_pro:queue:rspec[--format Rswag::Specs::SwaggerFormatter]"
Run non-Rswag tests by skipping Rswag tests with KNAPSACK_PRO_TEST_FILE_EXCLUDE_PATTERN
:
export KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=api-token-for-non-rswag-tests
export KNAPSACK_PRO_TEST_FILE_EXCLUDE_PATTERN={spec/requests/**/*_spec.rb,spec/api/**/*_spec.rb,spec/integration/**/*_spec.rb}
bundle exec rake "knapsack_pro:queue:rspec"
Troubleshooting
If you cannot find what you are looking for in this section, please refer to the Ruby troubleshooting page.
Some of my test files are not executed
First, check if the RSpec output mentions any filtering like the following:
Run options: include {:focus=>true, :ids=>{"./spec/example_spec.rb"=>["1:1:2"]}}
Second, you may want to grep the codebase (including .rspec
) for:
--tag MY_TAG
,-t MY_TAG
fit
,fdescribe
, orfcontext
- test examples or groups tagged with
:focus
Do not use run_all_when_everything_filtered
Make sure to use filter_run_when_matching
instead of the deprecated run_all_when_everything_filtered
. The latter may cause skipping some of your tests.
# ⛔️ Bad
RSpec.configure do |c|
c.filter_run :focus => true
c.run_all_when_everything_filtered = true
end
# ✅ Good
RSpec.configure do |c|
c.filter_run_when_matching :focus
end
# 🤘 FYeah
CI=true # Refer to your CI docs
RSpec.configure do |c|
unless ENV['CI']
c.filter_run_when_matching :focus
end
end
Some tests are failing in Queue Mode
Since Knapsack Pro ignores .rspec
and many projects use it to require spec_helper.rb
or rails_helper.rb
, some tests may be falling. Make sure you either require the correct helper at the top of each test file or pass it as an argument:
bundle exec rake "knapsack_pro:queue:rspec[--require rails_helper]"
Also, please make sure you have explicitly set RAILS_ENV=test
on your CI nodes.
.rspec
is ignored in Queue Mode
The .rspec
file is ignored in Queue Mode because knapsack_pro
needs to pass arguments explicitly to RSpec::Core::Runner
. You can inline them with the Rake argument syntax instead.
Something is wrong with my custom formatter
Try with KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS=false
.
I see the summary of failed/pending tests multiple times in Queue Mode
It may happen if you use:
- a custom RSpec formatter
knapsack_pro
< 0.33.0KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS=false
This is due to the fact that Knapsack Pro in Queue Mode runs tests in batches, and RSpec accumulates failures/pending tests for all batches.
TypeError: superclass mismatch for class MyClass
in Queue Mode
Probably, you are in the following situation:
# spec/a_spec.rb
class BaseClassA
end
module Mock
module FakeModels
class MyClass < BaseClassA
def args
end
end
end
end
describe 'A test of something' do
it do
end
end
# spec/b_spec.rb
class BaseClassB
end
module Mock
module FakeModels
# 👇 Base class is different
class MyClass < BaseClassB
def args
end
end
end
end
describe 'B test of something' do
it do
end
end
Use RSpec's stub_const
instead.
My tests fail in Queue Mode
Knapsack Pro uses RSpec::Core::Runner
in Queue Mode to run tests without reloading Ruby/Rails for each batch of tests. If you monkey-patch RSpec or mutate its global state, the test runner may not be able to clean up properly after each batch.
Also, you can try to use the knapsack_pro
binary instead of bundle exec rake knapsack_pro:queue:rspec
.