Using Knapsack Pro with RSpec

Retry failed/flaky tests

You have a couple of options:

• rspec-retry (recommended)
• RSpec's --only-failures

  bundle exec rake knapsack_pro:queue:rspec
  bundle exec rspec --only-failures

--fail-fast

You can use the Rake argument syntax to fail fast:

# Stop when 1 test failed
bundle exec rake "knapsack_pro:queue:rspec[--fail-fast]"

# Stop when 3 tests failed
bundle exec rake "knapsack_pro:queue:rspec[--fail-fast 3]"

Run a subset of tests

To run a subset of your test suite you have a couple of options:

• KNAPSACK_PRO_TEST_FILE_* environment variables (recommended)
• RSpec's --tag MY_TAG, --tag ~MY_TAG, --tag type:feature, or --tag ~type:feature

If you are seeking faster performance on your CI, you may want to read Parallelize test examples (instead of files)

Parallelize test examples (instead of files)

See Split by test examples.

Formatters (rspec_junit_formatter, json)

Format stdout with the documentation formatter and file output with any RSpec supported formatter.

You need to install the rspec_junit_formatter gem.

JUnit formatter

Queue Mode

bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format RspecJunitFormatter --out tmp/rspec.xml]"

You need to append the CI node index to the report name to avoid conflicts if your CI nodes write to the same disk:

# Refer to your CI docs for `$MY_CI_NODE_INDEX`
export KNAPSACK_PRO_CI_NODE_INDEX=$MY_CI_NODE_INDEX

bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format RspecJunitFormatter --out tmp/rspec_$KNAPSACK_PRO_CI_NODE_INDEX.xml]"

This also applies if you are running parallel test processes on each CI node (see how to integrate Knapsack Pro with parallel_tests here).

For legacy versions of knapsack_pro older than 7.0, please click here.

# Refer to your CI docs for `$MY_CI_NODE_INDEX`
export KNAPSACK_PRO_CI_NODE_INDEX=$MY_CI_NODE_INDEX

bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format RspecJunitFormatter --out tmp/rspec_$KNAPSACK_PRO_CI_NODE_INDEX.xml]"

spec_helper.rb or rails_helper.rb

# `TMP_REPORT` must be the same path as `--out`
# `TMP_REPORT` must be a full path (no `~`)
TMP_REPORT = "tmp/rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.xml"
FINAL_REPORT = "tmp/final_rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.xml"

KnapsackPro::Hooks::Queue.after_subset_queue do |queue_id, subset_queue_id|
  if File.exist?(TMP_REPORT)
    FileUtils.mv(TMP_REPORT, FINAL_REPORT)
  end
end

FINAL_REPORT will contain all the tests run on the CI node (not just the last subset). For more information, you can read this Github issue.

If your CI nodes write to the same disk, you need to append the CI node index to the solution presented above to avoid conflicts:

TMP_REPORT = "tmp/rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.xml"
FINAL_REPORT = "tmp/final_rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.xml"

This applies also if you are running parallel test processes on each CI node (see our page on to integrate Knapsack Pro with parallel_tests for an example).

Regular Mode

bundle exec rake "knapsack_pro:rspec[--format documentation --format RspecJunitFormatter --out tmp/rspec.xml]"

JSON formatter

Queue Mode

bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format json --out tmp/rspec.json]"

You need to append the CI node index to the report name to avoid conflicts if your CI nodes write to the same disk:

# Refer to your CI docs for `$MY_CI_NODE_INDEX`
export KNAPSACK_PRO_CI_NODE_INDEX=$MY_CI_NODE_INDEX

bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format json --out tmp/rspec_$KNAPSACK_PRO_CI_NODE_INDEX.json]"

This also applies if you are running parallel test processes on each CI node (see how to integrate Knapsack Pro with parallel_tests here).

For legacy versions of knapsack_pro older than 7.0, please click here.

# Refer to your CI docs for `$MY_CI_NODE_INDEX`
export KNAPSACK_PRO_CI_NODE_INDEX=$MY_CI_NODE_INDEX

bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format json --out tmp/rspec_$KNAPSACK_PRO_CI_NODE_INDEX.json]"

spec_helper.rb or rails_helper.rb

# `TMP_REPORT` must be the same path as `--out`
# `TMP_REPORT` must be a full path (no `~`)
TMP_REPORT = "tmp/rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.json"
FINAL_REPORT = "tmp/final_rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.json"

KnapsackPro::Hooks::Queue.after_subset_queue do |queue_id, subset_queue_id|
  if File.exist?(TMP_REPORT)
    FileUtils.mv(TMP_REPORT, FINAL_REPORT)
  end
end

FINAL_REPORT will contain all the tests run on the CI node (not just the last subset). For more information, you can read this Github issue.

If your CI nodes write to the same disk, you need to append the CI node index to the solution presented above to avoid conflicts:

TMP_REPORT = "tmp/rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.json"
FINAL_REPORT = "tmp/final_rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.json"

This applies also if you are running parallel test processes on each CI node (see our page on to integrate Knapsack Pro with parallel_tests for an example).

Regular Mode

bundle exec rake "knapsack_pro:rspec[--format documentation --format json --out tmp/rspec.json]"

Rswag gem configuration

If you use the Rswag gem that extends rspec-rails "request specs" with a Swagger-based DSL, then you can configure Knapsack Pro to run Rswag tests with the Rswag formatter (Rswag::Specs::SwaggerFormatter) and run other tests without it.

Run Rswag tests with KNAPSACK_PRO_TEST_FILE_PATTERN:

export KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=api-token-for-rswag-tests
export KNAPSACK_PRO_TEST_FILE_PATTERN={spec/requests/**/*_spec.rb,spec/api/**/*_spec.rb,spec/integration/**/*_spec.rb}
bundle exec rake "knapsack_pro:queue:rspec[--format Rswag::Specs::SwaggerFormatter]"

Run non-Rswag tests by skipping Rswag tests with KNAPSACK_PRO_TEST_FILE_EXCLUDE_PATTERN:

export KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=api-token-for-non-rswag-tests
export KNAPSACK_PRO_TEST_FILE_EXCLUDE_PATTERN={spec/requests/**/*_spec.rb,spec/api/**/*_spec.rb,spec/integration/**/*_spec.rb}
bundle exec rake "knapsack_pro:queue:rspec"

Load code only once for a specific type of specs

In large projects that use RSpec, it is common to have some expensive setup logic that is only needed when certain kinds of specs have been loaded. You would prefer to avoid the setup cost if that kind of spec has not been loaded.

The when_first_matching_example_defined hook makes it easy to conditionally perform some logic when the first test example is defined with matching metadata, allowing you to ensure the necessary setup is only performed when needed.

It is not advisable to combine before(:suite) with the RSpec when_first_matching_example_defined method in Queue Mode if your goal is to run MyExpensiveService.start just once for specific test types upon their first load.

This setup risks the before(:suite) hook not being executed if a test of a particular type (e.g., system) is not included in the initial batch fetched from the Queue API.

To guarantee that MyExpensiveService.start is executed only once and precisely when a specific test type is loaded for the first time, you need to use the following helper:

spec_helper.rb

def when_first_matching_example_defined(type:)
  env_var_name = "WHEN_FIRST_MATCHING_EXAMPLE_DEFINED_FOR_#{type}"

  RSpec.configure do |config|
    config.when_first_matching_example_defined(type: type) do
      config.before(:context) do
        unless ENV[env_var_name]
          yield
        end
        ENV[env_var_name] = 'hook_called'
      end
    end
  end
end

when_first_matching_example_defined(type: :system) do
  MyExpensiveService.start
end

Troubleshooting

If you cannot find what you are looking for in this section, please refer to the Ruby troubleshooting page.

Some of my test files are not executed

tip

Use KNAPSACK_PRO_TEST_FILE_* to filter the tests files to run in Knapsack Pro.

First, check if the RSpec output mentions any filtering like the following:

Run options: include {:focus=>true, :ids=>{"./spec/example_spec.rb"=>["1:1:2"]}}

Second, you may want to grep the codebase (including .rspec) for:

• --tag MY_TAG, -t MY_TAG
• fit, fdescribe, or fcontext
• test examples or groups tagged with :focus

caution

Do not use run_all_when_everything_filtered

Make sure to use filter_run_when_matching instead of the deprecated run_all_when_everything_filtered. The latter may cause skipping some of your tests.

# ⛔️ Bad
RSpec.configure do |c|
  c.filter_run :focus => true
  c.run_all_when_everything_filtered = true
end

# ✅ Good
RSpec.configure do |c|
  c.filter_run_when_matching :focus
end

# 🤘 FYeah
CI=true # Refer to your CI docs

RSpec.configure do |c|
  unless ENV['CI']
    c.filter_run_when_matching :focus
  end
end

If you are using RSpec in Queue Mode and Split by test example, --tag is not supported.

Some tests are failing in Queue Mode

Since Knapsack Pro ignores .rspec and many projects use it to require spec_helper.rb or rails_helper.rb, some tests may be falling. Make sure you either require the correct helper at the top of each test file or pass it as an argument:

bundle exec rake "knapsack_pro:queue:rspec[--require rails_helper]"

Also, please make sure you have explicitly set RAILS_ENV=test on your CI nodes.

.rspec is ignored in Queue Mode

The .rspec file is ignored in Queue Mode because knapsack_pro needs to pass arguments explicitly to RSpec::Core::Runner. You can inline them with the Rake argument syntax instead.

Something is wrong with my custom formatter

Please ensure the knapsack_pro gem is updated to version 7.0 or higher.

For legacy versions of knapsack_pro older than 7.0, please click here.

Try with KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS=false.

I see the summary of failed/pending tests multiple times in Queue Mode

Please ensure the knapsack_pro gem is updated to version 7.0 or higher.

For legacy versions of knapsack_pro older than 7.0, please click here.

It may happen if you use:

• a custom RSpec formatter
• knapsack_pro < 0.33.0
• KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS=false

This is due to the fact that Knapsack Pro in Queue Mode runs tests in batches, and RSpec accumulates failures/pending tests for all batches.

TypeError: superclass mismatch for class MyClass in Queue Mode

Probably, you are in the following situation:

# spec/a_spec.rb

class BaseClassA
end

module Mock
  module FakeModels
    class MyClass < BaseClassA
      def args
      end
    end
  end
end

describe 'A test of something' do
  it do
  end
end


# spec/b_spec.rb

class BaseClassB
end

module Mock
  module FakeModels
    # 👇 Base class is different
    class MyClass < BaseClassB
      def args
      end
    end
  end
end

describe 'B test of something' do
  it do
  end
end

Use RSpec's stub_const instead.

My tests fail in Queue Mode

Knapsack Pro uses RSpec::Core::Runner in Queue Mode to run tests without reloading Ruby/Rails for each batch of tests. If you monkey-patch RSpec or mutate its global state, the test runner may not be able to clean up properly after each batch.

Also, you can try to use the knapsack_pro binary instead of bundle exec rake knapsack_pro:queue:rspec.

before(:suite) / after(:suite) are executed multiple times in Queue Mode

This issue has been fixed in knapsack_pro version 7.0. Learn more about Knapsack Pro hooks.