Skip to main content

Using Knapsack Pro with RSpec

Retry failed/flaky tests​

You have a couple of options:

--fail-fast​

You can use the Rake argument syntax to fail fast:

# Stop when 1 test failed
bundle exec rake "knapsack_pro:queue:rspec[--fail-fast]"

# Stop when 3 tests failed
bundle exec rake "knapsack_pro:queue:rspec[--fail-fast 3]"

Run a subset of tests​

To run a subset of your test suite you have a couple of options:

  • KNAPSACK_PRO_TEST_FILE_* environment variables (recommended)
  • RSpec's --tag MY_TAG, --tag ~MY_TAG, --tag type:feature, or --tag ~type:feature

If you are seeking faster performance on your CI, you may want to read Parallelize test examples (instead of files)

Parallelize test examples (instead of files)​

See Split by test examples.

Formatters (rspec_junit_formatter, json)​

Format stdout with the documentation formatter and file output with any RSpec supported formatter.

Regular Mode​

You need to install the rspec_junit_formatter gem.

bundle exec rake "knapsack_pro:rspec[--format documentation --format RspecJunitFormatter --out tmp/rspec.xml]"

Queue Mode​

You need to install the rspec_junit_formatter gem.

# Refer to your CI docs for `$MY_CI_NODE_INDEX`
export KNAPSACK_PRO_CI_NODE_INDEX=$MY_CI_NODE_INDEX

bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format RspecJunitFormatter --out tmp/rspec_$KNAPSACK_PRO_CI_NODE_INDEX.xml]"
# spec_helper.rb or rails_helper.rb

# `TMP_REPORT` must be the same path as `--out`
# `TMP_REPORT` must be a full path (no `~`)
TMP_REPORT = "tmp/tmp_rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.xml"
FINAL_REPORT = "tmp/final_rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.xml"

KnapsackPro::Hooks::Queue.after_subset_queue do |queue_id, subset_queue_id|
  if File.exist?(TMP_REPORT)
    FileUtils.mv(TMP_REPORT, FINAL_REPORT)
  end
end

FINAL_REPORT will contain all the tests run on the CI node (not just the last subset). For more information, you can read this Github issue.

If your CI nodes write to the same disk, you need to append the CI node index to the solution presented above to avoid conflicts:

TMP_REPORT = "tmp/tmp_rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.xml"
FINAL_REPORT = "tmp/final_rspec_#{ENV['KNAPSACK_PRO_CI_NODE_INDEX']}.xml"

This applies also if you are running parallel test processes on each CI node (see our page on to integrate Knapsack Pro with parallel_tests for an example).

Rswag gem configuration​

If you use the Rswag gem that extends rspec-rails "request specs" with a Swagger-based DSL, then you can configure Knapsack Pro to run Rswag tests with the Rswag formatter (Rswag::Specs::SwaggerFormatter) and run other tests without it.

Run Rswag tests with KNAPSACK_PRO_TEST_FILE_PATTERN:

export KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=api-token-for-rswag-tests
export KNAPSACK_PRO_TEST_FILE_PATTERN={spec/requests/**/*_spec.rb,spec/api/**/*_spec.rb,spec/integration/**/*_spec.rb}
bundle exec rake "knapsack_pro:queue:rspec[--format Rswag::Specs::SwaggerFormatter]"

Run non-Rswag tests by skipping Rswag tests with KNAPSACK_PRO_TEST_FILE_EXCLUDE_PATTERN:

export KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=api-token-for-non-rswag-tests
export KNAPSACK_PRO_TEST_FILE_EXCLUDE_PATTERN={spec/requests/**/*_spec.rb,spec/api/**/*_spec.rb,spec/integration/**/*_spec.rb}
bundle exec rake "knapsack_pro:queue:rspec"

Troubleshooting​

If you cannot find what you are looking for in this section, please refer to the Ruby troubleshooting page.

Some of my test files are not executed​

First, check if the RSpec output mentions any filtering like the following:

Run options: include {:focus=>true, :ids=>{"./spec/example_spec.rb"=>["1:1:2"]}}

Second, you may want to grep the codebase (including .rspec) for:

  • --tag MY_TAG, -t MY_TAG
  • fit, fdescribe, or fcontext
  • test examples or groups tagged with :focus
caution

Do not use run_all_when_everything_filtered

Make sure to use filter_run_when_matching instead of the deprecated run_all_when_everything_filtered. The latter may cause skipping some of your tests.

# ⛔️ Bad
RSpec.configure do |c|
  c.filter_run :focus => true
  c.run_all_when_everything_filtered = true
end

# ✅ Good
RSpec.configure do |c|
  c.filter_run_when_matching :focus
end

# 🤘 FYeah
CI=true # Refer to your CI docs

RSpec.configure do |c|
  unless ENV['CI']
    c.filter_run_when_matching :focus
  end
end

Some tests are failing in Queue Mode​

Since Knapsack Pro ignores .rspec and many projects use it to require spec_helper.rb or rails_helper.rb, some tests may be falling. Make sure you either require the correct helper at the top of each test file or pass it as an argument:

bundle exec rake "knapsack_pro:queue:rspec[--require rails_helper]"

Also, please make sure you have explicitly set RAILS_ENV=test on your CI nodes.

.rspec is ignored in Queue Mode​

The .rspec file is ignored in Queue Mode because knapsack_pro needs to pass arguments explicitly to RSpec::Core::Runner. You can inline them with the Rake argument syntax instead.

Something is wrong with my custom formatter​

Try with KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS=false.

I see the summary of failed/pending tests multiple times in Queue Mode​

It may happen if you use:

This is due to the fact that Knapsack Pro in Queue Mode runs tests in batches, and RSpec accumulates failures/pending tests for all batches.

TypeError: superclass mismatch for class MyClass in Queue Mode​

Probably, you are in the following situation:

# spec/a_spec.rb

class BaseClassA
end

module Mock
  module FakeModels
    class MyClass < BaseClassA
      def args
      end
    end
  end
end

describe 'A test of something' do
  it do
  end
end


# spec/b_spec.rb

class BaseClassB
end

module Mock
  module FakeModels
    # 👇 Base class is different
    class MyClass < BaseClassB
      def args
      end
    end
  end
end

describe 'B test of something' do
  it do
  end
end

Use RSpec's stub_const instead.

My tests fail in Queue Mode​

Knapsack Pro uses RSpec::Core::Runner in Queue Mode to run tests without reloading Ruby/Rails for each batch of tests. If you monkey-patch RSpec or mutate its global state, the test runner may not be able to clean up properly after each batch.

Also, you can try to use the knapsack_pro binary instead of bundle exec rake knapsack_pro:queue:rspec.

before(:suite) / after(:suite) are executed multiple times in Queue Mode​

See Hooks: Before and After