Recently I’ve been looking into the source code of Minitest to find out if I can run some tests and then dynamically run another set of tests once the previous run is done. This would allow me to provide dynamically a list of tests to execute on my parallel CI nodes to run CI builds faster.

Minitest

Something similar exists in RSpec thanks to RSpec::Core::Runner feature that allows running specs multiple times with different runner options in the same process.

In RSpec flow looks like:

require "spec_helper"

RSpec::Core::Runner.run([... some parameters ...])

RSpec.clear_examples

RSpec::Core::Runner.run([... different parameters ...])

As you can see one of the steps is to clear examples with RSpec.clear_examples for the previous run to ensure the executed tests won’t affect the next list of tests we will run.

I was also looking if something similar exists in Minitest to ensure we have a pristine state of test runner before we run another set of test files. I step on Minitest::Runnable.reset method that could do it.

Digging into Minitest source code

I found out that Minitest comes with class method run that runs the loaded test files.

# minitest/lib/minitest.rb
module Minitest
  ##
  # This is the top-level run method. Everything starts from here. It
  # tells each Runnable sub-class to run, and each of those are
  # responsible for doing whatever they do.
  #
  # The overall structure of a run looks like this:
  #
  #   Minitest.autorun
  #     Minitest.run(args)
  #       Minitest.__run(reporter, options)
  #         Runnable.runnables.each
  #           runnable.run(reporter, options)
  #             self.runnable_methods.each
  #               self.run_one_method(self, runnable_method, reporter)
  #                 Minitest.run_one_method(klass, runnable_method)
  #                   klass.new(runnable_method).run

  def self.run args = []

Knowing that I could run tests with it. The first step thou was to ensure we will be able to load test files but I realized at the top of each of test file I have a line like:

require 'test_helper'

and the test_helper.rb file was not found while I attempt to load test file so I had to first add a directory with my tests to load path to make above require work.

# add test directory to load path to make require 'test_helper' work
$LOAD_PATH.unshift('test')

# now load test files
require './test/models/user_test.rb'
require './test/models/article_test.rb'

# if all tests pass we want to exit process with 0 exit code
final_exit_code = 0

# run tests loaded into memory
args  = ['--verbose']
# We need to duplicate the args because the run method will change the Array object.
# We will reuse args later.
tests_passed? = Minitest.run(args.dup)

# now the tests will be executed

# the variable tests_passed? will be true if tests passed. Otherwise would be false
final_exit_code = 1 unless tests_passed?

# Before we run another set of test files we need to reset the test runner state
Minitest::Runnable.reset


# Let's load another set of test files
require './test/controllers/users_controller_test.rb'
require './test/controllers/articles_controller_test.rb'

# we can run new set of test files
tests_passed? = Minitest.run(args.dup)
final_exit_code = 1 unless tests_passed?

# now the second set of test files will be executed

# once the tests files finished run then we can exit process with proper exit code
# 0 - when all tests are green
# 1 - when at least one test failed. Exit code 1 tells our CI provider
#     that process running tests failed.
exit(final_exit_code)

Running Minitest continuously and fetching test files from the Queue in a dynamic way

Digging into the source code of Minitest helped me to find out a way to run my tests in a more efficient way. I applied this to the knapsack_pro gem I’m working on.