Troubleshooting
NameError: uninitialized constant MyModule::MyModelName
Try with full namespacing ::SomeModule::MyModule::MyModelName
.
Debug Knapsack Pro on your development environment/machine
Regular Mode
To reproduce what Knapsack Pro executed on a specific CI node, check out the same branch and run:
KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=MY_TOKEN \
KNAPSACK_PRO_CI_NODE_INDEX=MY_INDEX \
KNAPSACK_PRO_CI_NODE_TOTAL=MY_TOTAL \
KNAPSACK_PRO_BRANCH=MY_BRANCH \
KNAPSACK_PRO_COMMIT_HASH=MY_COMMIT \
KNAPSACK_PRO_CI_NODE_BUILD_ID=MY_BUILD_ID \
bundle exec rake "knapsack_pro:rspec[--seed MY_SEED]"
KNAPSACK_PRO_CI_NODE_BUILD_ID
must be the same as the CI build you are trying to reproduce (if it helps, take a look at what Knapsack Pro uses as node_build_id
for your CI provider).
You can also run the same subset of tests without Knapsack Pro: in the logs, find the command that Knapsack Pro used to invoke the test runner:
rspec --default-path spec "spec/models/user_spec.rb" "spec/models/comment_spec.rb"
Queue Mode
To reproduce what Knapsack Pro executed on a specific CI node, check out the same branch and run:
KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=MY_TOKEN \
KNAPSACK_PRO_CI_NODE_INDEX=MY_INDEX \
KNAPSACK_PRO_CI_NODE_TOTAL=MY_TOTAL \
KNAPSACK_PRO_BRANCH=MY_BRANCH \
KNAPSACK_PRO_COMMIT_HASH=MY_COMMIT \
KNAPSACK_PRO_CI_NODE_BUILD_ID=MY_BUILD_ID \
KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true \
bundle exec rake "knapsack_pro:queue:rspec[--seed MY_SEED]"
KNAPSACK_PRO_CI_NODE_BUILD_ID
must be the same as the CI build you are trying to reproduce (if it helps, take a look at what Knapsack Pro uses as node_build_id
for your CI provider).
You can also run the same subset of tests without Knapsack Pro: in the logs, find the command that Knapsack Pro used to invoke the test runner.
You will find multiple commands to reproduce each batch:
[knapsack_pro] To retry the last batch of tests fetched from the API Queue, please run the following command on your machine:
[knapsack_pro] bundle exec rspec --format documentation --format RspecJunitFormatter --out tmp/rspec.xml --default-path spec --seed 24098 "spec/features/dashboard/billing/subscription_error_path_1_spec.rb[1:1:2:1:1:1]"
Or a single command at the end to execute all the batches the CI node executed:
[knapsack_pro] To retry all the tests assigned to this CI node, please run the following command on your machine:
[knapsack_pro] bundle exec rspec --format documentation --format RspecJunitFormatter --out tmp/rspec.xml --default-path spec --seed 24098 "spec/features/dashboard/billing/subscription_error_path_1_spec.rb[1:1:2:1:1:1]" "spec/features/dashboard/builds/build_distribution_for_build_spec.rb[1:1:1:10:2:1]" "spec/features/dashboard/builds/build_distribution_for_build_spec.rb[1:1:2:1]" "spec/features/dashboard/admin_statistics_spec.rb[1:6:1:1]" "spec/features/dashboard/identity_providers_spec.rb[1:6:1]" "spec/features/dashboard/onboarding_spec.rb[1:2:1:1:1]"
Knapsack Pro hangs
Some users reported frozen CI nodes with:
Test runners producing too much output (e.g., Codeship and Queue Mode): reduce it with
KNAPSACK_PRO_LOG_LEVEL=warn
Test runners producing no output: make sure to use a formatter (like RSpec's
--format progress
)Timecop (e.g., Travis): ensure
Timecop.return
is executed after all examples# spec/spec_helper.rb
RSpec.configure do |c|
c.after(:all) do
Timecop.return
end
endChrome 83+ prevents downloads in sandboxed iframes: add an
allow-downloads
keyword in the sandbox attribute listTests freeze and CI node timeouts (the Ruby process is killed by a CI provider). Add the following script to
spec_helper.rb
. Thanks to that, you will see the backtrace showing what code line causes the hanging problem. Backtrace is visible only if the Ruby process got kill withUSR1
signal. Learn more from this GitHub issue.# source https://github.com/rspec/rspec-rails/issues/1353#issuecomment-93173691
puts "Process pid: #{Process.pid}"
trap 'USR1' do
threads = Thread.list
puts
puts "=" * 80
puts "Received USR1 signal; printing all #{threads.count} thread backtraces."
threads.each do |thr|
description = thr == Thread.main ? "Main thread" : thr.inspect
puts
puts "#{description} backtrace: "
puts thr.backtrace.join("\n")
end
puts "=" * 80
endIf the CI provider does not kill the frozen process, you can do it by running the below command in the terminal (when you are ssh-ed into the CI node):
kill -USR1 <process pid>
Knapsack Pro does not work on a forked repository
Make sure your Knapsack Pro API token is set up as a secret in your CI (not hardcoded in the repository) to avoid leaking it.
Since the token won't be available on forked CI builds, you can use a script to run:
- Knapsack Pro in Fallback Mode(static split by file names) on forked builds
- Knapsack Pro in Queue Mode or Regular Mode on internal builds
Create bin/knapsack_pro_tests
, make it executable chmod u+x
, and use it on CI to run your tests:
#!/bin/bash
if [ "$KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC" = "" ]; then
KNAPSACK_PRO_ENDPOINT=https://api-disabled-for-fork.knapsackpro.com \
KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=disabled-for-fork \
KNAPSACK_PRO_MAX_REQUEST_RETRIES=0 \
KNAPSACK_PRO_CI_NODE_RETRY_COUNT=0 \
bundle exec rake knapsack_pro:queue:rspec
else
bundle exec rake knapsack_pro:queue:rspec
fi
Error commit_hash
parameter is required
Knapsack Pro takes KNAPSACK_PRO_COMMIT_HASH
and KNAPSACK_PRO_BRANCH
from the CI environment (see supported CIs). If your CI is not supported you may see the following error:
ERROR -- : [knapsack_pro] {"errors"=>[{"commit_hash"=>["parameter is required"]}]}
In such a case, you have two options:
- If you have
git
installed on CI, setKNAPSACK_PRO_REPOSITORY_ADAPTER=git
andKNAPSACK_PRO_PROJECT_DIR
- Otherwise, set
KNAPSACK_PRO_COMMIT_HASH
andKNAPSACK_PRO_BRANCH
yourself
LoadError: cannot load such file -- spec_helper
If you are using a complex KNAPSACK_PRO_TEST_FILE_PATTERN
, Knapsack Pro could have problems finding the directory containing the spec_helper.rb
file. Please set KNAPSACK_PRO_TEST_DIR
.
CI builds fail with Test::Unit
but all tests passed
Please ensure you are using Test::Unit
only and not loading minitest too.
Tests hitting external APIs fail
Most likely, you have global shared state between your tests.
For example, you may have a hook that deletes Stripe data while a parallel CI node is testing the Stripe integration:
before(:each) do
Stripe::Subscription.all.each { |subscription| subscription.delete }
Stripe::Customer.all.each { |customer| customer.delete }
end
You can fix it by either:
- Using a gem like VCR to record/replay your HTTP interactions
- Write your tests so that they can run in parallel (e.g., each test creates/deletes its own Stripe data)
OpenSSL::SSL::SSLError
VCR, WebMock, or FakeWeb may trigger the following error when Knapsack Pro interacts with the Knapsack Pro API:
OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: tlsv1 alert internal error
Make sure to follow the steps to configure VCR/WebMock/FakeWeb.
ActiveRecord::SubclassNotFound
If files are changing during a tests run, you may get the following error:
ActiveRecord::SubclassNotFound:
Invalid single-table inheritance type: AuthenticationProviders::AnAuthenticationProvider is not a subclass of AuthenticationProvider
Please make sure classes are loaded once in the test environment:
# environments/test.rb
config.eager_load = true
Adding parallel CI nodes/jobs makes test slower
You can verify if the tests are getting slower on your Knapsack Pro dashboard:
- Execution times of your CI builds are increasing:
Recorded CI builds > Show (build) > Test Files > Total execution time
- Individual test stats are trending up:
Statistics of test files history > Stats (test file) > History of the test file (chart)
Here are the most common reasons:
- CI nodes share resources (CPU/RAM/IO)
- Tests are accessing the same resource (e.g., Stripe sandbox, database)
- Your CI gives you a limited pool of parallel CI nodes (and runs the rest serially)
Some tests were skipped in Regular Mode
There is an unlikely scenario where some CI nodes may start in Fallback Mode but others in Regular Mode resulting in some tests being skipped. Our recommendations are either
- Switch to Queue Mode and enjoy faster CI builds too
- Disable Fallback mode with
KNAPSACK_PRO_FALLBACK_MODE_ENABLED=false
- Increase the request attempts with
KNAPSACK_PRO_MAX_REQUEST_RETRIES
FactoryBot/FactoryGirl raises in Queue Mode
Use the
knapsack_pro
binary:bundle exec knapsack_pro queue:rspec
Avoid implicit associations:
# ⛔️ Bad
FactoryBot.define do
factory :assignment do
task
end
end
# ✅ Good
FactoryBot.define do
factory :assignment do
association :task
end
end
One CI node run the test suite again in Queue Mode
Most likely, that CI node started when all the others finished running and initialized a new queue.
You have a couple of options:
- Make sure
KNAPSACK_PRO_CI_NODE_BUILD_ID
is set (recommended) - Set
KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true
Rake tasks under tests are run more than once in Queue Mode
Make sure the Rake task is loaded once:
before do
Rake::Task[MY_TASK_NAME].clear if Rake::Task.task_defined?(MY_TASK_NAME)
Rake.application.rake_require(MY_TASK_FILENAME)
Rake::Task.define_task(:environment)
end
Tests distribution is unbalanced in Queue Mode
Please consider:
- Splitting by test examples if you have a bottleneck file that is packed with test examples
- If it's a retry, remember that
KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true
uses a cached split