PREDICT

caution

This feature is limited to beta-testers.

caution

Only RSpec is supported for now.

A substantial percentage of your CI builds is spent running tests that would never fail. For example, updating a README.md should not require any testing. And changing a bunch of files should not run the entire test suite.

PREDICT runs only the tests that are likely to fail. The subset of tests is calculated by combining the test code coverage collected over time with the changes you are working on.

How it works

PREDICT collects test coverage into a report that looks like the following:

{
  "spec/models/user_spec.rb[1:1]": [
    "app/models/user.rb"
  ],
  "spec/features/dashboard_spec.rb[1:1:1]": [
    "app/models/user.rb",
    "app/models/organization.rb",
    "app/controllers/application_controller.rb",
    "app/controllers/dashboard_controller.rb",
    "app/views/dashboard/index.html.erb"
  ]
}

The test prediction is based on the following steps:

1. Fetch from the API the most recent test coverage report
2. Calculate what files changed between the current commit and the commit that generated the report
3. Find in the report which test examples cover the changed files

For example, if app/models/user.rb changed, PREDICT would run:

• spec/models/user_spec.rb[1:1]
• spec/features/dashboard_spec.rb[1:1:1]

If app/views/dashboard/index.html.erb changed instead, PREDICT would only run:

• spec/features/dashboard_spec.rb[1:1:1]

Configuration

To enable PREDICT, set any of the following ENV variables:

• KNAPSACK_PRO_PREDICT_DISABLED_ON_BRANCHES (comma-separated strings or regular expressions): on what branches to run all the tests
• KNAPSACK_PRO_PREDICT_DISABLED_ON_CHANGED_FILES (Ruby glob): what files, if modified, cause all the tests to run
• KNAPSACK_PRO_PREDICT_ALWAYS_RUN_TESTS (Ruby glob): what tests to always run

If PREDICT finds the string [skip predict] / [predict skip] in the commit message, it will run all the tests.

Given the dynamic nature of Ruby, PREDICT will likely not be correct 100% of the time. So we recommend you run all the tests before deploying to production. You can do that in multiple ways:

• Include [skip predict] / [predict skip] in the last commit message before you merge a pull request
• Include your production branch in KNAPSACK_PRO_PREDICT_DISABLED_ON_BRANCHES

Example

Given the following configuration on CI (use single quotes to prevent shell expansion):

KNAPSACK_PRO_PREDICT_DISABLED_ON_BRANCHES='main,staging,/release-.*/i' \
KNAPSACK_PRO_PREDICT_DISABLED_ON_CHANGED_FILES='{Gemfile.lock,package-lock.json}' \
KNAPSACK_PRO_PREDICT_ALWAYS_RUN_TESTS='spec/features/**/*_spec.rb' \
bundle exec rake "knapsack_pro:queue:rspec"

You can expect the following behaviour:

• On main, staging, and branches starting with release-, all tests are run
• On the other branches, the tests subset is run (including spec/features/**/*_spec.rb) unless:
  • Gemfile or Gemfile.lock were modified
  • The current commit message contains [skip predict] / [predict skip]
  • The test coverage does not yet exist on the Knapsack Pro API (e.g., the first run after enabling PREDICT)

Ruby glob

Both KNAPSACK_PRO_PREDICT_DISABLED_ON_CHANGED_FILES and KNAPSACK_PRO_PREDICT_ALWAYS_RUN_TESTS accept a Ruby glob as a value.

Ensure the glob matches the correct files by running the following command in the root of your project:

ruby -e "puts (Dir.glob '{Gemfile*,package*.json}')"

# Gemfile
# Gemfile.lock
# package-lock.json
# package.json

Custom rules

KNAPSACK_PRO_PREDICT_DISABLED_ON_CHANGED_FILES and KNAPSACK_PRO_PREDICT_ALWAYS_RUN_TESTS allow you to control what tests PREDICT runs. But you may want more precision.

In some situations, you might configure PREDICT to ensure specific tests are run based on some conditions. This is useful in those cases where test coverage would not detect some files (e.g., React components).

PREDICT provides KnapsackPro::Predictor::AssociatedSpecs and KnapsackPro::Predictor::ModifiedSupportSpecs out of the box, but you can define your rules too.

Associated specs

Given the following configuration:

predict = KnapsackPro::Predict.new
predict.use KnapsackPro::Predictor::AssociatedSpecs.new(from: %r{app/react/(.*).js}, to: "spec/features/%s_spec.rb")
predict.use KnapsackPro::Predictor::AssociatedSpecs.new(from: %r{app/react/(.*).js}, to: "spec/features/a_specific_spec.rb")
predict.instrument

If app/react/foo.js changed, PREDICT would add spec/features/foo_spec.rb and spec/features/a_specific_spec.rb to the prediction.

Modified support specs

When using RSpec shared_contexts and shared_examples, configure them with:

predict = KnapsackPro::Predict.new
predict.use KnapsackPro::Predictor::ModifiedSupportSpecs.new(%r{spec/.*_examples\.rb\z})
predict.use KnapsackPro::Predictor::ModifiedSupportSpecs.new(%r{spec/.*_contexts\.rb\z})
predict.instrument

When tests examples are added to the support files, all the specs including them are added to the prediction.

Define your custom rules

You can create custom rules based on your specific conditions by giving predict.use an object that responds to #call(git_diff, test_coverage) and returns an array of paths (e.g., [spec/models/foo_spec.rb[1:1], spec/featurs/bar_spec.rb]) to be added to the prediction.

Check the source code for KnapsackPro::Predictor::AssociatedSpecs or KnapsackPro::Predictor::ModifiedSupportSpecs for a starting point.

Using PREDICT with SimpleCov (or Coverage)

If you are using SimpleCov or Coverage, start them before Knapsack Pro:

SimpleCov

require 'simplecov'
SimpleCov.start do
  # ...
end

require 'knapsack_pro'
# ...

Coverage

Coverage.start

require 'knapsack_pro'
# ...