Knapsack gem README

knapsack gem on GitHub / this readme source code on GitHub

Knapsack splits tests across CI nodes and makes sure that tests will run comparable time on each node.

Parallel tests across CI server nodes based on each test file’s time execution. Knapsack generates a test time execution report and uses it for future test runs.

The knapsack gem supports:

Without Knapsack - bad test suite split

Without Knapsack gem

With Knapsack - better test suite split

With Knapsack gem

With Knapsack Pro - optimal test suite split

Watch 1 minute video how Queue Mode works

To better understand it you can check examples for one of popular CI providers like:

CircleCI - Improve CircleCI parallelisation for RSpec, Minitest, Cypress
Heroku CI - How to run parallel dynos on Heroku CI to complete tests faster
Buildkite - Auto balancing 7 hours tests between 100 parallel agents
Jenkins - Jenkins Pipeline how to run parallel tests in your workflow stages
CodeShip - How to run CodeShip Parallel Test Pipelines efficiently - optimal CI parallelization
Semaphore CI - Run parallel jobs on Semaphore CI 2.0 to get faster CI build time
Travis CI - How to run Travis CI parallel jobs with build matrix feature fast

JavaScript examples for test runners:

Cypress, Jest - Run javascript E2E tests faster with Cypress on parallel CI nodes

Features in knapsack vs knapsack_pro

Feature	knapsack gem	knapsack_pro gem
Queue mode designed for optimal dynamic test suite split	✘	✔
Test suite split based on tests time execution	✔	✔
Automated tests time execution recording	✘	✔
Test suite split based on most up to date tests time execution data	✘	✔
Show all features

How knapsack_pro makes my life easier as opposed to regular knapsack gem?

The knapsack_pro version has queue mode designed for optimal test suite split thanks to dynamic tests allocation.
With the knapsack_pro version the setup and the ongoing work is easier because you don’t have to manually generate knapsack json report with test files time execution for each test suite like rspec, cucumber etc.
The knapsack_pro version tracks all you branches so when your test code changes the knapsack_pro tries to provide you as optimal test suite split as possible.
When your test codebase changes over time you need to manually generate a new knapsack report in free gem version which is extra overhead and waste of time of the developer.

Only knapsack_pro gem has Queue Mode feature that saves optimal amount of time? Please visit Knapsack Pro.

Presentations about knapsack gem

Requirements

>= Ruby 2.1.0

Update gem
Installation
Usage
Setup your CI server
FAQ
Gem tests
- Spec
- Spec examples
Contributing
Acknowledgements
Mentions

Update gem

Please check changelog before update gem. Knapsack follows semantic versioning.

Installation

Add those lines to your application’s Gemfile:

group :test, :development do
  gem 'knapsack'
end

And then execute:

$ bundle

Add this line at the bottom of Rakefile:

Knapsack.load_tasks if defined?(Knapsack)

Usage

You can find here example of rails app with already configured knapsack.

https://github.com/KnapsackPro/rails-app-with-knapsack

Step for RSpec

Add at the beginning of your spec_helper.rb:

require 'knapsack'

# CUSTOM_CONFIG_GOES_HERE

Knapsack::Adapters::RSpecAdapter.bind

Step for Cucumber

Create file features/support/knapsack.rb and add there:

require 'knapsack'

# CUSTOM_CONFIG_GOES_HERE

Knapsack::Adapters::CucumberAdapter.bind

Step for Minitest

Add the Knapsack code after you load the app environment in the test/test_helper.rb file:

ENV['RAILS_ENV'] ||= 'test'
require File.expand_path('../../config/environment', __FILE__)
require 'rails/test_help'

require 'knapsack'

# CUSTOM_CONFIG_GOES_HERE

knapsack_adapter = Knapsack::Adapters::MinitestAdapter.bind
knapsack_adapter.set_test_helper_path(__FILE__)

Step for Spinach

Create file features/support/env.rb and add there:

require 'knapsack'

# CUSTOM_CONFIG_GOES_HERE

Knapsack::Adapters::SpinachAdapter.bind

Custom configuration

You can change default Knapsack configuration for RSpec, Cucumber, Minitest or Spinach tests. Here are examples what you can do. Put below configuration instead of CUSTOM_CONFIG_GOES_HERE.

Knapsack.tracker.config({
  enable_time_offset_warning: true,
  time_offset_in_seconds: 30
})

Knapsack.report.config({
  test_file_pattern: 'spec/**{,/*/**}/*_spec.rb', # default value based on adapter
  report_path: 'knapsack_custom_report.json'
})

# you can use your own logger
require 'logger'
Knapsack.logger = Logger.new(STDOUT)
Knapsack.logger.level = Logger::INFO

Common step

Generate time execution report for your test files. Run below command on one of your CI nodes.

# Step for RSpec
$ KNAPSACK_GENERATE_REPORT=true bundle exec rspec spec

# Step for Cucumber
$ KNAPSACK_GENERATE_REPORT=true bundle exec cucumber features

# Step for Minitest
$ KNAPSACK_GENERATE_REPORT=true bundle exec rake test

# If you use Rails 5.0.x then run this instead:
$ KNAPSACK_GENERATE_REPORT=true bundle exec rake test

# If you use Rails >= 5.1's SystemTest, run both unit and system tests
$ KNAPSACK_GENERATE_REPORT=true bundle exec rake test test:system

# Step for Spinach
$ KNAPSACK_GENERATE_REPORT=true bundle exec spinach

Commit generated report knapsack_rspec_report.json, knapsack_cucumber_report.json, knapsack_minitest_report.json or knapsack_spinach_report.json into your repository.

This report should be updated only after you add a lot of new slow tests or you change existing ones which causes a big time execution difference between CI nodes. Either way, you will get time offset warning at the end of the rspec/cucumber/minitest results which reminds you when it’s a good time to regenerate the knapsack report.

KNAPSACK_GENERATE_REPORT is truthy when "true" or 0. All other values are falsy, though "false", and 1 are semantically preferrable.

Adding or removing tests

There is no need to regenerate the report every time when you add/remove test file. If you remove a test file then Knapsack will ignore its entry in report. In case when you add a new file and it doesn’t already exist in report, the test file will be assigned to one of the CI node.

You’ll want to regenerate your execution report whenever you remove or add a test file with a long time execution time that would affect one of the CI nodes. You will get Knapsack notification whenever is good time to regenerate report.

Setup your CI server

On your CI server run this command for the first CI node. Update CI_NODE_INDEX for the next one.

# Step for RSpec
$ CI_NODE_TOTAL=2 CI_NODE_INDEX=0 bundle exec rake knapsack:rspec

# Step for Cucumber
$ CI_NODE_TOTAL=2 CI_NODE_INDEX=0 bundle exec rake knapsack:cucumber

# Step for Minitest
$ CI_NODE_TOTAL=2 CI_NODE_INDEX=0 bundle exec rake knapsack:minitest

# Step for Spinach
$ CI_NODE_TOTAL=2 CI_NODE_INDEX=0 bundle exec rake knapsack:spinach

You can add KNAPSACK_TEST_FILE_PATTERN if your tests are not in default directory. For instance:

# Step for RSpec
$ KNAPSACK_TEST_FILE_PATTERN="directory_with_specs/**{,/*/**}/*_spec.rb" CI_NODE_TOTAL=2 CI_NODE_INDEX=0 bundle exec rake knapsack:rspec

# Step for Cucumber
$ KNAPSACK_TEST_FILE_PATTERN="directory_with_features/**/*.feature" CI_NODE_TOTAL=2 CI_NODE_INDEX=0 bundle exec rake knapsack:cucumber

# Step for Minitest
$ KNAPSACK_TEST_FILE_PATTERN="directory_with_tests/**{,/*/**}/*_spec.rb" CI_NODE_TOTAL=2 CI_NODE_INDEX=0 bundle exec rake knapsack:minitest

# Step for Spinach
$ KNAPSACK_TEST_FILE_PATTERN="directory_with_features/**/*.feature" CI_NODE_TOTAL=2 CI_NODE_INDEX=0 bundle exec rake knapsack:spinach

You can set KNAPSACK_REPORT_PATH if your knapsack report was saved in non default location. Example:

# Step for RSpec
$ KNAPSACK_REPORT_PATH="knapsack_custom_report.json" CI_NODE_TOTAL=2 CI_NODE_INDEX=0 bundle exec rake knapsack:rspec

# Step for Cucumber
$ KNAPSACK_REPORT_PATH="knapsack_custom_report.json" CI_NODE_TOTAL=2 CI_NODE_INDEX=0 bundle exec rake knapsack:cucumber

# Step for Minitest
$ KNAPSACK_REPORT_PATH="knapsack_custom_report.json" CI_NODE_TOTAL=2 CI_NODE_INDEX=0 bundle exec rake knapsack:minitest

# Step for Spinach
$ KNAPSACK_REPORT_PATH="knapsack_custom_report.json" CI_NODE_TOTAL=2 CI_NODE_INDEX=0 bundle exec rake knapsack:spinach

Info about ENV variables

CI_NODE_TOTAL - total number CI nodes you have.

CI_NODE_INDEX - index of current CI node starts from 0. Second CI node should have CI_NODE_INDEX=1.

Note some CI providers like GitLab CI have the same name of environment variable like CI_NODE_INDEX which starts from 1 instead of 0. Knapsack gem will automatically pick it up and change from 1 to 0 to make knapsack work.

Passing arguments to rake task

Passing arguments to rspec

Knapsack allows you to pass arguments through to rspec. For example if you want to run only specs that have the tag focus. If you do this with rspec directly it would look like:

$ bundle exec rake rspec --tag focus

To do this with Knapsack you simply add your rspec arguments as parameters to the knapsack rake task.

$ bundle exec rake "knapsack:rspec[--tag focus]"

Remember that using tags to limit which specs get run will affect the time each file takes to run. One solution to this is to generate a new knapsack_rspec_report.json for the commonly run scenarios.

Passing arguments to cucumber

Add arguments to knapsack cucumber task like this:

$ bundle exec rake "knapsack:cucumber[--name feature]"

Passing arguments to minitest

Add arguments to knapsack minitest task like this:

$ bundle exec rake "knapsack:minitest[--arg_name value]"

For instance to run verbose tests:

$ bundle exec rake "knapsack:minitest[--verbose]"

Passing arguments to spinach

Add arguments to knapsack spinach task like this:

$ bundle exec rake "knapsack:spinach[--name feature]"

Knapsack binary

You can install knapsack globally and use binary. For instance:

$ knapsack rspec "--tag custom_tag_name --profile"
$ knapsack cucumber
$ knapsack minitest "--verbose --pride"
$ knapsack spinach "-f spinach_examples"

Here you will find example when it might be useful.

Info for CircleCI users

If you are using circleci.com you can omit CI_NODE_TOTAL and CI_NODE_INDEX. Knapsack will use CIRCLE_NODE_TOTAL and CIRCLE_NODE_INDEX provided by CircleCI.

Here is an example for test configuration in your .circleci/config.yml file.

Step 1

For the first time run all tests on a single CI node with enabled report generator.

# CircleCI 2.0
- run:
  name: Step for RSpec
  command: |
    # export word is important here!
    export KNAPSACK_GENERATE_REPORT=true
    bundle exec rspec spec

- run:
  name: Step for Cucumber
  command: |
    # export word is important here!
    export KNAPSACK_GENERATE_REPORT=true
    bundle exec cucumber features

- run:
  name: Step for Minitest
  command: |
    # export word is important here!
    export KNAPSACK_GENERATE_REPORT=true
    bundle exec rake test
    # For Rails 5.1 runs unit and system tests
    bundle exec rake test test:system

- run:
  name: Step for Spinach
  command: |
    # export word is important here!
    export KNAPSACK_GENERATE_REPORT=true
    bundle exec rspec spinach

After tests pass on your CircleCI machine you should copy knapsack json report which is rendered at the end of rspec/cucumber/minitest results. Save it into your repository as knapsack_rspec_report.json, knapsack_cucumber_report.json, knapsack_minitest_report.json or knapsack_spinach_report.json file and commit.

Step 2

Now you should update test command and enable parallel. Please remember to add additional containers for your project in CircleCI settings.

# CircleCI 2.0
- run:
  name: Step for RSpec
  command: bundle exec rake knapsack:rspec

- run:
  name: Step for Cucumber
  command: bundle exec rake knapsack:cucumber

- run:
  name: Step for Minitest
  command: bundle exec rake knapsack:minitest

- run:
  name: Step for Spinach
  command: bundle exec rake knapsack:spinach

Now everything should work. You will get a warning at the end of rspec/cucumber/minitest results if time execution takes too long.

Info for Travis users

Step 1

For the first time run all tests at once with enabled report generator. Edit .travis.yml

script:
  # Step for RSpec
  - "KNAPSACK_GENERATE_REPORT=true bundle exec rspec spec"

  # Step for Cucumber
  - "KNAPSACK_GENERATE_REPORT=true bundle exec cucumber features"

  # Step for Minitest
  - "KNAPSACK_GENERATE_REPORT=true bundle exec rake test"
  - "KNAPSACK_GENERATE_REPORT=true bundle exec rake test test:system" # For Rails 5.1 runs unit and system tests

  # Step for Spinach
  - "KNAPSACK_GENERATE_REPORT=true bundle exec spinach"

After tests pass you should copy knapsack json report which is rendered at the end of rspec/cucumber/minitest results. Save it into your repository as knapsack_rspec_report.json, knapsack_cucumber_report.json, knapsack_minitest_report.json or knapsack_spinach_report.json file and commit.

Step 2

You can parallel your builds across virtual machines with travis matrix feature. Edit .travis.yml

script:
  # Step for RSpec
  - "bundle exec rake knapsack:rspec"

  # Step for Cucumber
  - "bundle exec rake knapsack:cucumber"

  # Step for Minitest
  - "bundle exec rake knapsack:minitest"

  # Step for Spinach
  - "bundle exec rake knapsack:spinach"

env:
  - CI_NODE_TOTAL=2 CI_NODE_INDEX=0
  - CI_NODE_TOTAL=2 CI_NODE_INDEX=1

If you want to have some global ENVs and matrix of ENVs then do it like this:

script:
  # Step for RSpec
  - "bundle exec rake knapsack:rspec"

  # Step for Cucumber
  - "bundle exec rake knapsack:cucumber"

  # Step for Minitest
  - "bundle exec rake knapsack:minitest"

  # Step for Spinach
  - "bundle exec rake knapsack:spinach"

env:
  global:
    - RAILS_ENV=test
    - MY_GLOBAL_VAR=123
    - CI_NODE_TOTAL=2
  jobs:
    - CI_NODE_INDEX=0
    - CI_NODE_INDEX=1

Such configuration will generate matrix with 2 following ENV rows:

CI_NODE_TOTAL=2 CI_NODE_INDEX=0 RAILS_ENV=test MY_GLOBAL_VAR=123
CI_NODE_TOTAL=2 CI_NODE_INDEX=1 RAILS_ENV=test MY_GLOBAL_VAR=123

More info about global and matrix ENV configuration in travis docs.

Info for semaphoreapp.com users

Step 1

For the first time run all tests at once with enabled report generator. Set up your build command:

# Step for RSpec
KNAPSACK_GENERATE_REPORT=true bundle exec rspec spec

# Step for Cucumber
KNAPSACK_GENERATE_REPORT=true bundle exec cucumber features

# Step for Minitest
KNAPSACK_GENERATE_REPORT=true bundle exec rake test
KNAPSACK_GENERATE_REPORT=true bundle exec rake test test:system # For Rails 5.1 runs unit and system tests

# Step for Spinach
KNAPSACK_GENERATE_REPORT=true bundle exec spinach

After tests pass you should copy knapsack json report which is rendered at the end of rspec/cucumber/test results. Save it into your repository as knapsack_rspec_report.json, knapsack_cucumber_report.json, knapsack_minitest_report.json or knapsack_spinach_report.json file and commit.

Step 2

Semaphore 2.0

knapsack gem supports environment variables provided by Semaphore CI 2.0 to run your tests. You will have to define a few things in .semaphore/semaphore.yml config file.

# .semaphore/semaphore.yml
# Use the latest stable version of Semaphore 2.0 YML syntax:
version: v1.0

# Name your pipeline. In case you connect multiple pipelines with promotions,
# the name will help you differentiate between, for example, a CI build phase
# and delivery phases.
name: Demo Rails 5 app

# An agent defines the environment in which your code runs.
# It is a combination of one of available machine types and operating
# system images.
# See https://docs.semaphoreci.com/article/20-machine-types
# and https://docs.semaphoreci.com/article/32-ubuntu-1804-image
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804

# Blocks are the heart of a pipeline and are executed sequentially.
# Each block has a task that defines one or more jobs. Jobs define the
# commands to execute.
# See https://docs.semaphoreci.com/article/62-concepts
blocks:
  - name: Setup
    task:
      env_vars:
        - name: RAILS_ENV
          value: test
      jobs:
        - name: bundle
          commands:
          # Checkout code from Git repository. This step is mandatory if the
          # job is to work with your code.
          # Optionally you may use --use-cache flag to avoid roundtrip to
          # remote repository.
          # See https://docs.semaphoreci.com/article/54-toolbox-reference#libcheckout
          - checkout
          # Restore dependencies from cache.
          # Read about caching: https://docs.semaphoreci.com/article/54-toolbox-reference#cache
          - cache restore gems-$SEMAPHORE_GIT_BRANCH-$(checksum Gemfile.lock),gems-$SEMAPHORE_GIT_BRANCH-,gems-master-
          # Set Ruby version:
          - sem-version ruby 2.6.1
          - bundle install --jobs=4 --retry=3 --path vendor/bundle
          # Store the latest version of dependencies in cache,
          # to be used in next blocks and future workflows:
          - cache store gems-$SEMAPHORE_GIT_BRANCH-$(checksum Gemfile.lock) vendor/bundle

  - name: RSpec tests
    task:
      env_vars:
        - name: RAILS_ENV
          value: test
        - name: PGHOST
          value: 127.0.0.1
        - name: PGUSER
          value: postgres
      # This block runs two jobs in parallel and they both share common
      # setup steps. We can group them in a prologue.
      # See https://docs.semaphoreci.com/article/50-pipeline-yaml#prologue
      prologue:
        commands:
          - checkout
          - cache restore gems-$SEMAPHORE_GIT_BRANCH-$(checksum Gemfile.lock),gems-$SEMAPHORE_GIT_BRANCH-,gems-master-
          # Start Postgres database service.
          # See https://docs.semaphoreci.com/article/54-toolbox-reference#sem-service
          - sem-service start postgres
          - sem-version ruby 2.6.1
          - bundle install --jobs=4 --retry=3 --path vendor/bundle
          - bundle exec rake db:setup

      jobs:
      - name: Run tests with Knapsack
        parallelism: 2
        commands:
          # Step for RSpec
          - bundle exec rake knapsack:rspec
          # Step for Cucumber
          - bundle exec rake knapsack:cucumber
          # Step for Minitest
          - bundle exec rake knapsack:minitest
          # Step for Spinach
          - bundle exec rake knapsack:spinach

You may also find useful article about spliting tests in a dynamic way with knapsack_pro Queue Mode: run parallel jobs on Semaphore CI 2.0 to get faster CI build time.

Semaphore 1.0

Knapsack supports semaphoreapp ENVs SEMAPHORE_THREAD_COUNT and SEMAPHORE_CURRENT_THREAD. The only thing you need to do is set up knapsack rspec/cucumber/minitest command for as many threads as you need. Here is an example:

# Thread 1
## Step for RSpec
bundle exec rake knapsack:rspec
## Step for Cucumber
bundle exec rake knapsack:cucumber
## Step for Minitest
bundle exec rake knapsack:minitest
## Step for Spinach
bundle exec rake knapsack:spinach

# Thread 2
## Step for RSpec
bundle exec rake knapsack:rspec
## Step for Cucumber
bundle exec rake knapsack:cucumber
## Step for Minitest
bundle exec rake knapsack:minitest
## Step for Spinach
bundle exec rake knapsack:spinach

Tests will be split across threads.

Info for buildkite.com users

Step 1

For the first time run all tests at once with enabled report generator. Run the following commands locally:

# Step for RSpec
KNAPSACK_GENERATE_REPORT=true bundle exec rspec spec

# Step for Cucumber
KNAPSACK_GENERATE_REPORT=true bundle exec cucumber features

# Step for Minitest
KNAPSACK_GENERATE_REPORT=true bundle exec rake test
KNAPSACK_GENERATE_REPORT=true bundle exec rake test test:system # For Rails 5.1 runs unit and system tests

# Step for Spinach
KNAPSACK_GENERATE_REPORT=true bundle exec spinach

Step 2

Knapsack supports buildkite ENVs BUILDKITE_PARALLEL_JOB_COUNT and BUILDKITE_PARALLEL_JOB. The only thing you need to do is to configure the parallelism parameter in your build step and run the appropiate command in your build

# Step for RSpec
bundle exec rake knapsack:rspec

# Step for Cucumber
bundle exec rake knapsack:cucumber

# Step for Minitest
bundle exec rake knapsack:minitest

# Step for Spinach
bundle exec rake knapsack:spinach

When using the docker-compose plugin on Buildkite, you have to tell it which environment variables to pass to the docker container:

steps:
  - label: "Test"
    parallelism: 2
    plugins:
      - docker-compose#3.0.3:
        run: app
        # use here proper knapsack command for your test runner
        command: bundle exec rake knapsack:rspec
        config: docker-compose.test.yml
        env:
          - BUILDKITE_PARALLEL_JOB_COUNT
          - BUILDKITE_PARALLEL_JOB

Info for GitLab CI users

If you are using GitLab 11.5 or later you can omit CI_NODE_TOTAL and CI_NODE_INDEX. Knapsack will use CI_NODE_TOTAL and CI_NODE_INDEX provided by GitLab if you use the parallel option in GitLab CI.

Step 1

For the first time run all tests on a single CI node with enabled report generator.

test:
  script: KNAPSACK_GENERATE_REPORT=true bundle exec rspec spec

Here are other commands you could use instead of RSpec.

# Step for Cucumber
KNAPSACK_GENERATE_REPORT=true bundle exec cucumber features

# Step for Minitest
KNAPSACK_GENERATE_REPORT=true bundle exec rake test
KNAPSACK_GENERATE_REPORT=true bundle exec rake test test:system # For Rails 5.1 runs unit and system tests

# Step for Spinach
KNAPSACK_GENERATE_REPORT=true bundle exec spinach

After tests pass on your GitLab CI you should copy knapsack json report which is rendered at the end of rspec/cucumber/minitest results. Save it into your repository as knapsack_rspec_report.json, knapsack_cucumber_report.json, knapsack_minitest_report.json or knapsack_spinach_report.json file and commit.

Step 2

Now you should update test command and enable parallel. Please remember to set proper parallel value for your project.

Here you can find info how to configure the parallel CI nodes.

test:
  script: bundle exec rake knapsack:rspec
  parallel: 2

Here are other commands you could use instead of knapsack for RSpec.

# Step for Cucumber
bundle exec rake knapsack:cucumber

# Step for Minitest
bundle exec rake knapsack:minitest

# Step for Spinach
bundle exec rake knapsack:spinach

Info for snap-ci.com users

Step 1

For the first time run all tests at once with enabled report generator. Run the following commands locally:

# Step for RSpec
KNAPSACK_GENERATE_REPORT=true bundle exec rspec spec

# Step for Cucumber
KNAPSACK_GENERATE_REPORT=true bundle exec cucumber features

# Step for Minitest
KNAPSACK_GENERATE_REPORT=true bundle exec rake test
KNAPSACK_GENERATE_REPORT=true bundle exec rake test test:system # For Rails 5.1 runs unit and system tests

# Step for Spinach
KNAPSACK_GENERATE_REPORT=true bundle exec spinach

Step 2

Knapsack supports snap-ci.com ENVs SNAP_WORKER_TOTAL and SNAP_WORKER_INDEX. The only thing you need to do is to configure number of workers for your project in configuration settings in order to enable parallelism. Next thing is to set below commands to be executed in your stage:

# Step for RSpec
bundle exec rake knapsack:rspec

# Step for Cucumber
bundle exec rake knapsack:cucumber

# Step for Minitest
bundle exec rake knapsack:minitest

# Step for Spinach
bundle exec rake knapsack:spinach

Info for Jenkins

In order to run parallel jobs with Jenkins you should use Jenkins Pipeline. You can learn basics about it in the article Parallelism and Distributed Builds with Jenkins.

Here is an example Jenkinsfile working with Jenkins Pipeline and knapsack gem.

You may want to read article Knapsack with Jenkins Pipeline from Michał Knapik.

More tips can be found in the issue.

Info for BitBucket Pipelines

Step 1

For the first time run all tests at once with enabled report generator. Run the following commands locally:

# Step for RSpec
KNAPSACK_GENERATE_REPORT=true bundle exec rspec spec

# Step for Cucumber
KNAPSACK_GENERATE_REPORT=true bundle exec cucumber features

# Step for Minitest
KNAPSACK_GENERATE_REPORT=true bundle exec rake test
KNAPSACK_GENERATE_REPORT=true bundle exec rake test test:system # For Rails 5.1 runs unit and system tests

# Step for Spinach
KNAPSACK_GENERATE_REPORT=true bundle exec spinach

Step 2

Knapsack supports BitBucket Pipelines ENVs BITBUCKET_PARALLEL_STEP_COUNT and BITBUCKET_PARALLEL_STEP. The only thing you need to do is to configure the parallelism parameter in your build step and run the appropiate command in your build

# Step for RSpec
bundle exec rake knapsack:rspec

# Step for Cucumber
bundle exec rake knapsack:cucumber

# Step for Minitest
bundle exec rake knapsack:minitest

# Step for Spinach
bundle exec rake knapsack:spinach

FAQ

How Knapsack Pro makes my life easier as opposed to knapsack free gem?

Knapsack Pro version has Queue Mode designed for optimal test suite split thanks to dynamic tests allocation. Learn more
Knapsack Pro version tracks all you branches so when your test code changes the Knapsack Pro tries to provide you optimal test suite split.
With Knapsack Pro version the setup and the ongoing work is easier because you don’t have to manually generate knapsack json report with test files time execution for each test suite like RSpec, Cucumber etc.
When your test codebase changes over time you need to manually generate a new knapsack json report in free gem version which is extra overhead and waste of developer time.

What time offset warning means?

At the end of tests execution results you can see warning like this:

========= Knapsack Time Offset Warning ==========
Time offset: 30s
Max allowed node time execution: 02m 30s
Exceeded time: 37s

Time offset: 30s - this is the current time offset value, by default it’s 30s. Let’s assume whole test suite takes 4 minutes and you do split across 2 CI nodes so the optimal split is 2 minutes per node. Time offset 30s means when tests on single CI node will take longer than 2 minutes and 30s then you see warning about regenerating report because probably test suite files changed and the knapsack report contains old time execution data about each test file so regenerating knapsack report should help you provide a more optimal test suite split.

Max allowed node time execution: 02m 30s - it’s average time execution of tests per CI node + time offset. In this case average tests time execution per CI node is 2 minutes.

Exceeded time: 37s - it means tests on particular CI node took 37s longer than max allowed node time execution. Sometimes this value is negative when tests executed faster than max allowed node time execution.

How to generate knapsack report?

If you want to regenerate report take a look here.

KNAPSACK_GENERATE_REPORT=true bundle exec rspec spec

If you run command like this on your development machine then test suite time execution might be different than if you generate a report on CI machine (for instance tests might be faster on your machine then on CI node) so that might be a reason why you see warning about regenerating report. You can generate the report on single CI node which should give you result specific for your CI node instead of your development machine. In case you don’t want to bother about manually regenerating knapsack report please take a look on knapsack_pro gem.

What does “leftover specs” mean?

When you run your specs with knapsack rake task then you will see in the output something like:

Report specs:
spec/models/user_spec.rb
spec/controllers/users_controller_spec.rb

Leftover specs:
spec/models/book_spec.rb
spec/models/author_spec.rb

The leftover specs mean we don’t have recorded time execution for those test files. The reason might be:

that someone added a new test file after knapsack report was generated
another reason might be an empty spec file with no test cases
or you run in RSpec only subset of tests using tags like --tag type:my-custom-tag then if you recorded json report for such a tag then only tagged specs will be in json report and all other specs will be named as “leftover specs”

If leftover specs will be distributed across CI nodes then it will happen based on file name instead of the test file execution time which is missing for them.

If you have a lot of leftover specs then you can generate knapsack report again to improve you test distribution across CI nodes.

Why some of test files are still in “leftover specs” after I generate a new report?

If test file is empty or has only pending tests then it cannot be recorded so it will end up in leftovers specs list.

How can I run tests from multiple directories?

The test file pattern config option supports any glob pattern handled by Dir.glob and can be configured to pull test files from multiple directories. An example of this when using RSpec would be "{spec,engines/**/spec}/**{,/*/**}/*_spec.rb". For complex cases like this, the test directory can’t be extracted and must be specified manually using the KNAPSACK_TEST_DIR environment variable:

$ KNAPSACK_TEST_DIR=spec KNAPSACK_TEST_FILE_PATTERN="{spec,engines/**/spec}/**{,/*/**}/*_spec.rb" bundle exec rake knapsack:rspec

KNAPSACK_TEST_DIR will be your default path for rspec so you should put there your spec_helper.rb. Please ensure you will require it in your test files this way:

# good
require_relative 'spec_helper'

# bad - won't work
require 'spec_helper'

How to update existing knapsack report for a few test files?

You may want to look at monkey patch in this issue. Take into account that there are some cons of this approach.

How to run tests for particular CI node in your development environment

In your development environment you can debug tests that were run on the particular CI node. For instance to run subset of tests for the first CI node with specified seed you can do.

CI_NODE_TOTAL=2 \
CI_NODE_INDEX=0 \
bundle exec rake "knapsack:rspec[--seed 123]"

Above example is for RSpec. You can use respectively rake task name and token environment variable when you want to run tests for minitest, cucumber or spinach.

How can I change log level?

You can change log level by specifying the KNAPSACK_LOG_LEVEL environment variable.

KNAPSACK_LOG_LEVEL=warn bundle exec rake knapsack:rspec

Available values are debug, info, and warn. The default log level is info.

Gem tests

Spec

To run specs for Knapsack gem type:

$ bundle exec rspec spec

Spec examples

Directory spec_examples contains examples of fast and slow specs. There is a spec_example/spec_helper.rb with binded Knapsack.

To generate a new knapsack report for specs with focus tag (only specs in spec_examples/leftover directory have no focus tag), please type:

$ KNAPSACK_GENERATE_REPORT=true bundle exec rspec --default-path spec_examples --tag focus

Warning: Current knapsack_rspec_report.json file was generated for spec_examples except spec_examples/leftover directory. Just for testing reason to see how leftover specs will be distribute in a dumb way across CI nodes.

To see specs distributed for the first CI node type:

$ CI_NODE_TOTAL=2 CI_NODE_INDEX=0 KNAPSACK_SPEC_PATTERN="spec_examples/**{,/*/**}/*_spec.rb" bundle exec rake knapsack:rspec

Specs in spec_examples/leftover take more than 3 seconds. This should cause a Knapsack time offset warning because we set time_offset_in_seconds to 3 in spec_examples/spec_helper.rb. Type below to see warning:

$ bundle exec rspec --default-path spec_examples

Contributing

Fork it ( https://github.com/KnapsackPro/knapsack/fork )
Create your feature branch (git checkout -b my-new-feature)
Commit your changes (git commit -am 'Add some feature')
Push to the branch (git push origin my-new-feature)
You can create example tests in related repository with example of rails application and knapsack gem usage.
Create a new Pull Request

Acknowledgements

Many thanks to Małgorzata Nowak for beautiful logo.

Mentions

Lunar Logic Blog Parallel your specs and don’t waste time
Travis CI Parallelizing RSpec and Cucumber on multiple VMs
Semaphore Running RSpec specs in parallel
Semaphore Running Cucumber scenarios in parallel
Buildkite Libraries
Snap CI Knapsack: optimal test suite split based on time execution
CircleCI Test splitting documentation
Michał Knapik Blog Knapsack with Jenkins Pipeline