Skip to main content

Using Knapsack Pro with RSpec

Retry failed/flaky tests​

You have a couple of options:

--fail-fast​

You can use the Rake argument syntax to fail fast:

# Stop when 1 test failed
bundle exec rake "knapsack_pro:queue:rspec[--fail-fast]"

# Stop when 3 tests failed
bundle exec rake "knapsack_pro:queue:rspec[--fail-fast 3]"

Run a subset of tests​

To run a subset of your test suite you have a couple of options:

  • KNAPSACK_PRO_TEST_FILE_* environment variables (recommended)
  • RSpec's --tag MY_TAG, --tag ~MY_TAG, --tag type:feature, or --tag ~type:feature

If you are seeking faster performance on your CI, you may want to read Parallelize test examples (instead of files)

Parallelize test examples (instead of files)​

See Split by test examples.

Formatters (rspec_junit_formatter, json)​

Format stdout with the documentation formatter and file output with any RSpec supported formatter.

You need to install the rspec_junit_formatter gem.

JUnit formatter​

bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format RspecJunitFormatter --out tmp/rspec.xml]"

You need to append the CI node index to the report name to avoid conflicts if your CI nodes write to the same disk:

# Refer to your CI docs for `$MY_CI_NODE_INDEX`
bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format RspecJunitFormatter --out tmp/rspec_$MY_CI_NODE_INDEX.xml]"

This also applies if you are running parallel test processes on each CI node (see how to integrate Knapsack Pro with parallel_tests here).

For legacy versions of knapsack_pro older than 7.0, please click here.
# Refer to your CI docs for `$MY_CI_NODE_INDEX`
bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format RspecJunitFormatter --out tmp/rspec_$MY_CI_NODE_INDEX.xml]"
spec_helper.rb or rails_helper.rb
# `TMP_REPORT` must be the same path as `--out`
# `TMP_REPORT` must be a full path (no `~`)
TMP_REPORT = "tmp/rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.xml"
FINAL_REPORT = "tmp/final_rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.xml"

KnapsackPro::Hooks::Queue.after_subset_queue do |queue_id, subset_queue_id|
  if File.exist?(TMP_REPORT)
    FileUtils.mv(TMP_REPORT, FINAL_REPORT)
  end
end

FINAL_REPORT will contain all the tests run on the CI node (not just the last subset). For more information, you can read this Github issue.

If your CI nodes write to the same disk, you need to append the CI node index to the solution presented above to avoid conflicts:

TMP_REPORT = "tmp/rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.xml"
FINAL_REPORT = "tmp/final_rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.xml"

This applies also if you are running parallel test processes on each CI node (see our page on to integrate Knapsack Pro with parallel_tests for an example).

JSON formatter​

bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format json --out tmp/rspec.json]"

You need to append the CI node index to the report name to avoid conflicts if your CI nodes write to the same disk:

# Refer to your CI docs for `$MY_CI_NODE_INDEX`
bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format json --out tmp/rspec_$MY_CI_NODE_INDEX.json]"

This also applies if you are running parallel test processes on each CI node (see how to integrate Knapsack Pro with parallel_tests here).

For legacy versions of knapsack_pro older than 7.0, please click here.
# Refer to your CI docs for `$MY_CI_NODE_INDEX`
bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format json --out tmp/rspec_$MY_CI_NODE_INDEX.json]"
spec_helper.rb or rails_helper.rb
# `TMP_REPORT` must be the same path as `--out`
# `TMP_REPORT` must be a full path (no `~`)
TMP_REPORT = "tmp/rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.json"
FINAL_REPORT = "tmp/final_rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.json"

KnapsackPro::Hooks::Queue.after_subset_queue do |queue_id, subset_queue_id|
  if File.exist?(TMP_REPORT)
    FileUtils.mv(TMP_REPORT, FINAL_REPORT)
  end
end

FINAL_REPORT will contain all the tests run on the CI node (not just the last subset). For more information, you can read this Github issue.

If your CI nodes write to the same disk, you need to append the CI node index to the solution presented above to avoid conflicts:

TMP_REPORT = "tmp/rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.json"
FINAL_REPORT = "tmp/final_rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.json"

This applies also if you are running parallel test processes on each CI node (see our page on to integrate Knapsack Pro with parallel_tests for an example).

Rswag gem configuration​

If you use the Rswag gem that extends rspec-rails "request specs" with a Swagger-based DSL, then you can configure Knapsack Pro to run Rswag tests with the Rswag formatter (Rswag::Specs::SwaggerFormatter) and run other tests without it.

Run Rswag tests with KNAPSACK_PRO_TEST_FILE_PATTERN:

export KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=api-token-for-rswag-tests
export KNAPSACK_PRO_TEST_FILE_PATTERN={spec/requests/**/*_spec.rb,spec/api/**/*_spec.rb,spec/integration/**/*_spec.rb}
bundle exec rake "knapsack_pro:queue:rspec[--format Rswag::Specs::SwaggerFormatter]"

Run non-Rswag tests by skipping Rswag tests with KNAPSACK_PRO_TEST_FILE_EXCLUDE_PATTERN:

export KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=api-token-for-non-rswag-tests
export KNAPSACK_PRO_TEST_FILE_EXCLUDE_PATTERN={spec/requests/**/*_spec.rb,spec/api/**/*_spec.rb,spec/integration/**/*_spec.rb}
bundle exec rake "knapsack_pro:queue:rspec"

Load code only once for a specific type of specs​

In large projects that use RSpec, it is common to have some expensive setup logic that is only needed when certain kinds of specs have been loaded. You would prefer to avoid the setup cost if that kind of spec has not been loaded.

The when_first_matching_example_defined hook makes it easy to conditionally perform some logic when the first test example is defined with matching metadata, allowing you to ensure the necessary setup is only performed when needed.

It is not advisable to combine before(:suite) with the RSpec when_first_matching_example_defined method in Queue Mode if your goal is to run MyExpensiveService.start just once for specific test types upon their first load.

This setup risks the before(:suite) hook not being executed if a test of a particular type (e.g., system) is not included in the initial batch fetched from the Queue API.

To guarantee that MyExpensiveService.start is executed only once and precisely when a specific test type is loaded for the first time, you need to use the following helper:

spec_helper.rb
def when_first_matching_example_defined(type:)
  env_var_name = "WHEN_FIRST_MATCHING_EXAMPLE_DEFINED_FOR_#{type}"

  RSpec.configure do |config|
    config.when_first_matching_example_defined(type: type) do
      config.before(:context) do
        unless ENV[env_var_name]
          yield
        end
        ENV[env_var_name] = 'hook_called'
      end
    end
  end
end

when_first_matching_example_defined(type: :system) do
  MyExpensiveService.start
end

Troubleshooting​

If you cannot find what you are looking for in this section, please refer to the Ruby troubleshooting page.

Some of my test files are not executed​

tip

Use KNAPSACK_PRO_TEST_FILE_* to filter the tests files to run in Knapsack Pro.

First, check if the RSpec output mentions any filtering like the following:

Run options: include {:focus=>true, :ids=>{"./spec/example_spec.rb"=>["1:1:2"]}}

Second, you may want to grep the codebase (including .rspec) for:

  • --tag MY_TAG, -t MY_TAG
  • fit, fdescribe, or fcontext
  • test examples or groups tagged with :focus
caution

Do not use run_all_when_everything_filtered

Make sure to use filter_run_when_matching instead of the deprecated run_all_when_everything_filtered. The latter may cause skipping some of your tests.

# ⛔️ Bad
RSpec.configure do |c|
  c.filter_run :focus => true
  c.run_all_when_everything_filtered = true
end

# ✅ Good
RSpec.configure do |c|
  c.filter_run_when_matching :focus
end

# 🤘 FYeah
CI=true # Refer to your CI docs

RSpec.configure do |c|
  unless ENV['CI']
    c.filter_run_when_matching :focus
  end
end

If you are using RSpec in Queue Mode and Split by test example, --tag is not supported.

Some tests are failing in Queue Mode​

Since Knapsack Pro ignores .rspec and many projects use it to require spec_helper.rb or rails_helper.rb, some tests may be falling. Make sure you either require the correct helper at the top of each test file or pass it as an argument:

bundle exec rake "knapsack_pro:queue:rspec[--require rails_helper]"

Also, please make sure you have explicitly set RAILS_ENV=test on your CI nodes.

.rspec is ignored in Queue Mode​

The .rspec file is ignored in Queue Mode because knapsack_pro needs to pass arguments explicitly to RSpec::Core::Runner. You can inline them with the Rake argument syntax instead.

Something is wrong with my custom formatter​

Please ensure the knapsack_pro gem is updated to version 7.0 or higher.

For legacy versions of knapsack_pro older than 7.0, please click here.

I see the summary of failed/pending tests multiple times in Queue Mode​

Please ensure the knapsack_pro gem is updated to version 7.0 or higher.

For legacy versions of knapsack_pro older than 7.0, please click here.

It may happen if you use:

This is due to the fact that Knapsack Pro in Queue Mode runs tests in batches, and RSpec accumulates failures/pending tests for all batches.

TypeError: superclass mismatch for class MyClass in Queue Mode​

Probably, you are in the following situation:

# spec/a_spec.rb

class BaseClassA
end

module Mock
  module FakeModels
    class MyClass < BaseClassA
      def args
      end
    end
  end
end

describe 'A test of something' do
  it do
  end
end


# spec/b_spec.rb

class BaseClassB
end

module Mock
  module FakeModels
    # 👇 Base class is different
    class MyClass < BaseClassB
      def args
      end
    end
  end
end

describe 'B test of something' do
  it do
  end
end

Use RSpec's stub_const instead.

My tests fail in Queue Mode​

Knapsack Pro in Queue Mode runs tests without reloading Ruby/Rails for each batch of tests. If you monkey-patch RSpec or mutate its global state, the test runner may not be able to clean up properly after each batch. Please let us know if you have any issues.

Also, you can try to use the knapsack_pro binary instead of bundle exec rake knapsack_pro:queue:rspec.

before(:suite) / after(:suite) are executed multiple times in Queue Mode​

This issue has been fixed in knapsack_pro version 7.0. Learn more about Knapsack Pro hooks.

The before(:suite) hooks defined in test files are not executed in Queue Mode​

It is recommended that you define your before(:suite) hooks in spec_helper.rb or rails_helper.rb. These files should be loaded before any test files so that the hook is registered by RSpec.

The before(:suite) hook is executed first. After that, test files are dynamically loaded in multiple batches from the Knapsack Pro Queue API. The before(:suite) hooks defined in test files won't be executed because it is too late!

If you need to have something that is similar to before(:suite) and you want to define it in a test file, then you can use this:

spec/features/a_spec.rb
RSpec.configure do |config|
  config.before(:context) do
    unless ENV['MY_HOOK_NAME']
      # your code to run in the hook
    end
    ENV['MY_HOOK_NAME'] = 'hook_called'
  end
end

Alternatively, if you need to load code only once for a specific type of specs you can check this.