Jenkins Pipeline is a suite of plugins that allows creating simple-to-complex build stages for your testing environment on CI. We can use Jenkins Pipeline to run a few stages at the same time and thanks to that parallelize test suite across a few stages to complete tests faster.

Jenkins

In order to run parallel stages with Jenkins Pipeline, we will need a proper Jenkinsfile which represents our delivery pipeline as code via the Pipeline domain-specific language (DSL) syntax.

Another thing we will have to figure out is the problem how to divide our test suite across parallel stages in a way that each subset of test suite executed across all stages will complete work at the same time. It’s important to complete the tests on all stages at a similar time to run our CI build as fast as possible and eliminate bottleneck stage.

How to split test suite evenly across parallel Jenkins stages

To divide our tests across parallel stages we can use Knapsack Pro which allows to dynamically allocate tests across stages (also known as CI nodes). This way we will run our parallelised tests in optimal time.

Here you can learn more how dynamic test suite allocation works and with what else problems it can help.

Jenkinsfile for parallel pipeline

In this example, I will show you how to split your Ruby and JavaScript tests across parallel Jenkins stages.

This is Ruby example how to split Cucumber and RSpec test suite. Other test runners like Minitest, Test::Unit etc are available as well at Knapsack Pro:

timeout(time: 60, unit: 'MINUTES') {
  node() {
    stage('Checkout') {
      checkout([/* checkout code from git */])

      // determine git commit hash because we need to pass it to knapsack_pro
      COMMIT_HASH = sh(returnStdout: true, script: 'git rev-parse HEAD').trim()

      stash 'source'
    }
  }

  def num_nodes = 4; // define your total number of CI nodes (how many parallel jobs will be executed)
  def nodes = [:]

  for (int i = 0; i < num_nodes; i++) {
    def index = i;
    nodes["ci_node_${i}"] = {
      node() {
        stage('Setup') {
          unstash 'source'
          // other setup steps
        }

        def knapsack_options = """\
            KNAPSACK_PRO_CI_NODE_TOTAL=${num_nodes}\
            KNAPSACK_PRO_CI_NODE_INDEX=${index}\
            KNAPSACK_PRO_COMMIT_HASH=${COMMIT_HASH}\
            KNAPSACK_PRO_BRANCH=${env.BRANCH_NAME}\
        """

        // example how to run cucumber tests in Knapsack Pro Regular Mode
        stage('Run cucumber') {
          sh """${knapsack_options} bundle exec rake knapsack_pro:cucumber"""
        }

        // example how to run rspec tests in Knapsack Pro Queue Mode
        // Queue Mode should be as a last stage so it can autobalance build if tests in regular mode were not perfectly distributed
        stage('Run rspec') {
          sh """KNAPSACK_PRO_CI_NODE_BUILD_ID=${env.BUILD_TAG} ${knapsack_options} bundle exec rake knapsack_pro:queue:rspec"""
        }
      }
    }
  }

  parallel nodes // run CI nodes in parallel
}

Here is JavaScript example how to split Cypress tests. You can learn more about splitting E2E tests for Cypress test runner here.

timeout(time: 60, unit: 'MINUTES') {
  node() {
    stage('Checkout') {
      checkout([/* checkout code from git */])

      // determine git commit hash because we need to pass it to Knapsack Pro
      COMMIT_HASH = sh(returnStdout: true, script: 'git rev-parse HEAD').trim()

      stash 'source'
    }
  }

  def num_nodes = 4; // define your total number of CI nodes (how many parallel jobs will be executed)
  def nodes = [:]

  for (int i = 0; i < num_nodes; i++) {
    def index = i;
    nodes["ci_node_${i}"] = {
      node() {
        stage('Setup') {
          unstash 'source'
          // other setup steps
        }

        def knapsack_options = """\
            KNAPSACK_PRO_CI_NODE_TOTAL=${num_nodes}\
            KNAPSACK_PRO_CI_NODE_INDEX=${index}\
            KNAPSACK_PRO_COMMIT_HASH=${COMMIT_HASH}\
            KNAPSACK_PRO_BRANCH=${env.BRANCH_NAME}\
            KNAPSACK_PRO_CI_NODE_BUILD_ID=${env.BUILD_TAG}\
        """

        // example how to run tests with Knapsack Pro
        stage('Run tests') {
          sh """${knapsack_options} $(npm bin)/knapsack-pro-cypress"""
        }
      }
    }
  }

  parallel nodes // run CI nodes in parallel
}

Summary

Jenkins Pipeline gets you a continuous delivery (CD) pipeline which is an automated expression of your process for getting software from version control right through to your users. It’s important to save the time of your engineering team by running CI build fast. One way of doing it is test suite parallelisation that can be done in an optimal way with Knapsack Pro for Ruby and JavaScript tests.

If you are currently considering the choice between Jenkins and other CI, check out our popular comparison pages: Jenkins vs Github Actions, Drone vs Jenkins, AWS CodeBuild vs Jenkins, and Cloud Build vs Jenkins.