Skip to main content

PREDICT

caution

This feature is limited to beta-testers.

caution

Only Ruby >= 3.2.0 & RSpec are supported.

A substantial percentage of your CI builds is spent running tests that would never fail. For example, updating a README.md should not require any testing. And changing a bunch of files should not run the entire test suite.

PREDICT runs only the tests that are likely to fail. The subset of tests is calculated by combining the test code coverage collected over time with the changes you are working on.

How it works​

PREDICT collects test coverage into a report that looks like the following:

{
  "spec/models/user_spec.rb[1:1]": [
    "app/models/user.rb"
  ],
  "spec/features/dashboard_spec.rb[1:1:1]": [
    "app/models/user.rb",
    "app/models/organization.rb",
    "app/controllers/application_controller.rb",
    "app/controllers/dashboard_controller.rb",
    "app/views/dashboard/index.html.erb"
  ]
}

The test prediction is based on the following steps:

  1. Fetch from the API the most recent test coverage report
  2. Calculate what files changed between the current commit and the commit that generated the report
  3. Find in the report which test examples cover the changed files

For example, if app/models/user.rb changed, PREDICT would run:

  • spec/models/user_spec.rb[1:1]
  • spec/features/dashboard_spec.rb[1:1:1]

If app/views/dashboard/index.html.erb changed instead, PREDICT would only run:

  • spec/features/dashboard_spec.rb[1:1:1]

Configuration​

To enable PREDICT, set any of the following ENV variables:

  • KNAPSACK_PRO_PREDICT_DISABLED_ON_BRANCHES (comma-separated strings or regular expressions): on what branches to run all the tests
  • KNAPSACK_PRO_PREDICT_DISABLED_ON_CHANGED_FILES (Ruby glob): what files, if modified, cause all the tests to run
  • KNAPSACK_PRO_PREDICT_ALWAYS_RUN_TESTS (Ruby glob): what tests to always run

If PREDICT finds the string [skip predict] / [predict skip] in the commit message, it will run all the tests.

Given the dynamic nature of Ruby, PREDICT will likely not be correct 100% of the time. So we recommend you run all the tests before deploying to production. You can do that in multiple ways:

  • Include [skip predict] / [predict skip] in the last commit message before you merge a pull request
  • Include your production branch in KNAPSACK_PRO_PREDICT_DISABLED_ON_BRANCHES

Example​

Given the following configuration on CI (use single quotes to prevent shell expansion):

KNAPSACK_PRO_PREDICT_DISABLED_ON_BRANCHES='main,staging,/release-.*/i' \
KNAPSACK_PRO_PREDICT_DISABLED_ON_CHANGED_FILES='{Gemfile.lock,package-lock.json}' \
KNAPSACK_PRO_PREDICT_ALWAYS_RUN_TESTS='spec/features/**/*_spec.rb' \
bundle exec rake "knapsack_pro:queue:rspec"

You can expect the following behaviour:

  • On main, staging, and branches starting with release-, all tests are run
  • On the other branches, the tests subset is run (including spec/features/**/*_spec.rb) unless:
    • Gemfile or Gemfile.lock were modified
    • The current commit message contains [skip predict] / [predict skip]
    • The per-test coverage does not exist on the Knapsack Pro API (e.g., you just enabled PREDICT)

Ruby glob​

Both KNAPSACK_PRO_PREDICT_DISABLED_ON_CHANGED_FILES and KNAPSACK_PRO_PREDICT_ALWAYS_RUN_TESTS accept a Ruby glob as a value.

Ensure the glob matches the correct files by running the following command in the root of your project:

ruby -e "puts (Dir.glob '{Gemfile*,package*.json}')"

# Gemfile
# Gemfile.lock
# package-lock.json
# package.json

Troubleshooting​

Sidekiq throws a segmentation fault​

If you are using Sidekiq in your project, be aware that versions >=7.3.3 are affected by an issue that causes segmentation faults.

On CI, before running your tests you can downgrade to the latest version that works correctly with:

bundle config --local deployment false # "Unfreezes" the lock file. May not be necessary.
bundle remove sidekiq && bundle add sidekiq --version=7.3.2

Using with SimpleCov or Coverage​

If you are using SimpleCov or Coverage, start them before Knapsack Pro and enable both line and eval coverage:

require 'simplecov'
SimpleCov.start do
  # line coverage is the default
  enable_coverage_for_eval
end

require 'knapsack_pro'
# ...