Blog about testing & documentation for KnapsackPro.com

Have you ever used Ruby's Object#method?

Tue, 27 Apr 2021 08:00:00 +0000

Object#method is a little-known, yet very interesting Ruby method. Let’s see how it works and how you can use it in your code.

The basics

Let’s imagine we have some Ruby class, e.g.

class Animal
  def speak(sound)
    puts "I say #{sound}!"
  end
end

Calling the #method on the instance of this class, would return a Method object. This Method object acts as a closure in the context of the associated object.

animal = Animal.new
met = animal.method(:speak)

met.class
#=> Method
met.inspect
=> "#<Method: Animal#speak(sound)>"

met.call # we forget to pass the argument
#=> ArgumentError (wrong number of arguments (given 0, expected 1))
met.call('meow')
# I say meow!
# => nil

One thing that’s interesting in Method objects, is the fact they implement the #to_proc method. To see how this could be useful, let’s first remind ourselves of a widely-used Ruby shortcut.

The ampersand operator

Consider this common Ruby shortcut:

[1, 2, 3, 4, 5, 6].select { |num| num.even? }
#=> [2, 4, 6]

# commonly simplified with:

[1, 2, 3, 4, 5, 6].select(&:even?)
#=> [2, 4, 6]

The reason this works is two-fold. First, there is the & (ampersand) operator. When used at the beginning of a method argument, it transforms its operant into a Proc object (by calling #to_proc on it), and passes it to the method as if it was a block. Secondly, the Symbol class implements #to_proc. The way it’s implemented effectively allows us to send a given symbol to the provided argument. See Symbol#to_proc in action below:

is_even = :even?.to_proc

is_even.call(1)
#=> false
is_even.call(2)
#=> true

This proc could also be passed to the #select method from the previous example. We still need to use the ampersand operator (so that Ruby knows to treat it like a block).

[1, 2, 3, 4, 5, 6].select(&is_even)
#=> [2, 4, 6]

Ampersand with the Method object

Given the fact that the Method object implements the #to_proc method as well, we can use it much the same way as we did the symbol. Let’s circle back to our previous example to see this in action.

class Animal
  def speak(sound)
    puts "I say #{sound}!"
  end
end

met = Animal.new.method(:speak)
animal_sounds = ['meow', 'woof', 'moo', 'oink', 'hee-haw']

animal_sounds.each(&met)
# I say meow!
# I say woof!
# I say moo!
# I say oink!
# I say hee-haw!
# => ["meow", "woof", "moo", "oink", "hee-haw"]

So when is this useful?

One reason I used #method combined with the ampersand operator in the past was to get rid of repetitive, one-dimensional blocks. This is especially valuable, when you are chaining multiple methods together. Let’s say we have multiple filters with the same interface (all returning true or false) that we want to use on a collection:

class AboveMinThresholdChecker
  MIN_THRESHOLD = 200

  def self.call(number)
    new.call(number)
  end

  def call(number)
    number > MIN_THRESHOLD
  end
end

class BelowMaxThresholdChecker
  # ...
end

class PrimeNumberChecker
  # ...
end

Using explicit blocks, we would then do something like:

numbers = (1..400).to_a

numbers.select { |num| AboveMinThresholdChecker.call(num) }.
        select { |num| BelowMaxThresholdChecker.call(num) }.
        select { |num| PrimeNumberChecker.call(num) }

With our new knowledge, we can simplify the above in the following way using the #method:

numbers.select(&AboveMinThresholdChecker.method(:call)).
        select(&BelowMaxThresholdChecker.method(:call)).
        select(&PrimeNumberChecker.method(:call))

Summary

Have you used #method in Ruby before? Can you think of other interesting situations it could prove handy? Please share in the comments below!

If your project suffers from long CI builds, you can address this with Knapsack Pro. Be the hero in your team by streamlining your CI process and improving the developer productivity!

Estimate database connections pool size for Rails application

Mon, 26 Apr 2021 20:00:00 +0000

Configuring the database connections pool for the Rails app might not be a straightforward task for many programmers. There is a constraint of max opened connections on a database level. Your server environment configuration can change in time and affect the number of connections to the database required. For instance number of servers you use can change when you autoscale it based on the web traffic. It means that the number of web processes/threads running for Puma or Unicorn servers could change. All this adds additional complexity. When you use two databases (e.g. Postgres + Redis), everything gets more complex. In this article, we will address that. You will learn how to estimate needed database connections for your Ruby on Rails production application.

Why do available database connections matter?

The first question is, why do you need to care about available database connections? The answer is simple. Suppose you configured your Ruby application to open too many DB connections. In that case, it could happen that you will get ActiveRecord::ConnectionTimeoutError exceptions from the application when the database cannot handle more new connections from your Rails app. It can result in 500 errors visible for your web app users.

This problem might not be apparent immediately. Often you will find out about it in production. Your application might work just fine until specific circumstances cause the Rails app to need more DB connections, which can trigger exception flood. Let’s see how to avoid it.

RoR application configuration step by step

Let’s break a typical Ruby on Rails application down into smaller components that use databases.

We have a Rails application that uses the Postgres database for ActiveRecord usage.
We also use the Redis database for background workers like Sidekiq.

It looks simple, isn’t it? Let’s start with that, and later on, we will add more complexity to the mix :)

Postgres database connections - how to check the limit?

How to check how many available connections do you have for Postgres?

If you use a dedicated server with Postgres installed, then most likely you have a default max_connections which is typically 100 connections.
If you use a Postgres instance on the AWS, then you need to check the AWS documentation to find out what’s the max allowed connections to your database instance (it depends on if you use Amazon RDS or Aurora and what is server instance class)
If you use Heroku, you can check the Connection Limit for the Postgres Heroku add-on to check max acceptable connections.

ActiveRecord connection pool

In your Rails application, the config/database.yml file contains the pool option. As explained in the Rails docs:

Active Record database connections are managed by ActiveRecord::ConnectionAdapters::ConnectionPool, which ensures that a connection pool synchronizes the amount of thread access to a limited number of database connections.

Since the connection pooling is handled inside of Active Record by default, all application servers (Thin, Puma, Unicorn, etc.) should behave the same. The database connection pool is initially empty. As demand for connections increases, it will create them until it reaches the connection pool limit.

Any one request will check out a connection the first time it requires access to the database. At the end of the request, it will check the connection back in. This means that the additional connection slot will be available again for the next request in the queue.

The pool can be defined this way:

production:
  adapter: postgresql
  database: blog_production
  pool: 5

or as a part of a URL to the database:

development:
  url: postgresql://localhost/blog_production?pool=5

The URL option is popular when you host a database on an external server like Amazon RDS. Then you could define the URL this way:

production:
  url: postgres://blog_production:PASSWORD@blog-production.abcdefgh.eu-west-1.rds.amazonaws.com/blog_production?sslca=config/rds-combined-ca-bundle.pem&pool=5

Please note that for the production, you should not commit credentials in the config/database.yml file. Instead, store it in environment variables and then read the value at your Rails app’s runtime.

production:
  url: <%= ENV['DB_URL'] %>

How does ActiveRecord connection pool affects Postgres max connections?

Let’s start with a simple example. Your application may use one of the application servers like Puma or Unicorn. Let’s focus on Puma because it’s more complex as it has a separate configuration for several processes (known as workers in Puma terms) and threads. Unicorn runs in a single thread only. It works exactly like Puma with a single thread setting.

Puma config: 1 process and 1 thread

Let’s say you use the Puma server to run the Rails application. The Puma is configured to run 1 process (worker) and it has only 1 thread.

The puma process can open up to 5 connections to the database because the pool option is defined as 5 in config/database.yml. Typically, there are fewer connections than that because when you run 1 process and only 1 thread, only 1 connection to the Postgres database will be needed to make a database query.

Sometimes the database connection might be dead. In such a case, ActiveRecord can open a new connection, and then you may end up with 2 active connections. In the worst-case scenario when 4 connections would be dead, then Rails can open 5 connections max.

Puma config: 1 process and 2 threads

If you use 2 threads in a single Puma process (worker) then it means those 2 threads can use the same pool of DB connections within the Puma process.

It means that 2 DB connections will be open out of 5 possible. If any connection is dead, then more connections can be opened until the 5 connection pool limit is reached.

Puma config: 2 processes and 2 threads per process

If you run 2 Puma processes (workers) and each process has 2 threads then it means that each single process will open 2 DB connections because you have 2 threads per process. You have 2 processes so it means at the start of your application, there might be 4 DB connections open. Each process has its pool, so you have 2 pools. Each pool can open up to 5 DB connections. It means that in the worst-case scenario, there can be even 10 connections created to the database.

Assuming you use 2 threads per Puma process, it’s good to have the pool option set to 2 + some spare connections. It allows ActiveRecord to open a new connection if one of the DB connections is dead.

Puma config: 2 processes and 2 threads, and 2 web dynos on Heroku

If you use Heroku to host your application, it allows scaling your web application horizontally by adding more servers (dynos). Assume you run your application on 2 servers (2 Heroku dynos), each dyno is running 2 Puma processes, and each process has 2 threads. It means at the start, your application may open 6 connections to the database. Here is why:

2 dynos X 2 Puma processes X 2 Puma threads = 6 DB connections

2 dynos X 2 Puma process X Pool size (5) = Total pool size 20

It means that in the worst-case your application may open 20 DB connections.

Autoscaling web application

If you autoscale your web servers by adding more servers during the peak web traffic, you need to be careful. Ensure your application stays within the Postgres max connections limit. The above example shows you how to calculate expected opened DB connections and the worst-case scenario. Please adjust your pool size to ensure that you will be below the max connections limit for your database engine in the worst-case scenario.

What else can open DB connections?

We just talked about a webserver like Puma that can open connections and consume your max DB connections limit. But other non-web processes can do it as well:

You run Rails console on production in a Heroku dyno heroku run bin/rails console --app=my-app-name. It runs an instance of your Rails app, and 1 DB connection will be open. In the worst-case scenario, the number of connections defined in the pool can be opened. But it’s unlikely that your DB connections would go dead. So the whole pool limit shouldn’t be used.
You run scheduled rake tasks via Heroku Scheduler (cron-like tool). If the rake tasks are performed periodically, they need to open a connection to the DB so that at least 1 DB connection is used from the pool per rake task. Imagine you have 10 rake tasks that are started every hour. It means you need 10 available DB connections every hour. It can be easy to miss this if you base your estimation on just the web connections.
You use background workers like Sidekiq to perform async jobs. Your jobs may open DB connections. We will talk about it later.

Background worker - Sidekiq and ActiveRecord pool

Sidekiq process will use the pool defined in config/database.yml similarly as Puma. All Sidekiq threads in a Sidekiq process can use a common pool of connections.

If you run multiple servers (Heroku dynos), then it works similarly to the Puma example.

2 servers (dynos) X 1 Sidekiq process X 10 Sidekiq threads = 20 DB connections will be open.

You need to have a pool size of at least 10 in config/database.yml because Sidekiq by default, uses 10 threads.

If you use a pool size smaller than 10, Sidekiq threads will contend the access to the limited connections in the pool. It may work out fine, but be aware this can increase your job’s processing time. It can also lead to this problem.

Sidekiq and Redis database connections

Sidekiq uses the Redis database to store async jobs. It would be best if you calculate DB connections to Redis as well as Postgres connections. A Sidekiq server process requires at least (concurrency + 5) connections. The concurrency option is the number of Sidekiq threads per Sidekiq process.

Using the previous example:

2 servers (dynos) X 1 Sidekiq process X 10 Sidekiq threads = 2 servers (dynos) X 1 Sidekiq process X (10 + 5) = 30 Redis connections required.

Redis database connections

If you use Redis for processing background jobs, then it’s not just the Sidekiq process that is using Redis connections. Your Puma process and threads can use Redis to add new jobs to the Sidekiq queue as well. Typically you will have 1 Redis connection per 1 Puma thread.

If you explicitly open a new Redis connection with Redis.new, this can create a new connection per the Puma thread as well.

Summary

We covered a few examples of Postgres and Redis on calculating DB connections needed by your Rails application. I hope this will give you a better understanding of how to estimate how many DB connections you need on your database level to serve your application’s demands properly.

If you are looking to improve your Rails application workflow please consider checking how to run automated tests in parallel on your CI server with Knapsack Pro.

Querying single columns in Rails Active Record using the #pluck method

Sat, 17 Apr 2021 08:00:00 +0000

Have you heard of the #pluck method from Rails’ Active Record? Read on to see what it does and how it can be used to easily query single columns from your database!

The use case

Sometimes when retrieving data from the database, we are only interested in certain attributes (e.g., just emails of all the users, instead of data residing in all columns of the users table).

The naïve approach would be to load all the data into the memory and then map it so that only the subset of the data is left, e.g.:

User.all.map(&:email)
# SELECT "users".* FROM "users"

#=> ["kate@example.com", "john@example.org", "janet@example.com"]

This is quite inefficient.

First of all, the SQL query is retrieving data from all columns from the table. Secondly, the ActiveRecrodCollection object is created in the memory. Its members are initialized with all the data retrieved from the DB. The map at the end returns a regular Ruby Array object, making this intermediate step unnecessary.

Knowing we only really need to load the data from the email column, we could improve the above by chaining the Active Record’s #select method:

User.all.select(:email).map(&:email)
# SELECT "users"."email" FROM "users"

#=> ["kate@example.com", "john@example.org", "janet@example.com"]

The SQL query is limited to one column now, but the ActiveRecord objects are still being created in the intermediate step before map. Granted, they contain less data now (as the attributes we didn’t select are omitted), but it’s still quite unnecessary in this use case.

Using the #pluck

The good news is, we can use #pluck instead to solve this issue. It’s as simple as:

User.pluck(:email)
# SELECT "users"."email" FROM "users"

#=> ["kate@example.com", "john@example.org", "janet@example.com"]

The SQL is still retrieving just the data we need. The data is then returned in the Plain Old Ruby Array, and no ActiveRecord object is being created in between!

Retrieving data from multiple columns

The #pluck method works with multiple DB columns, too. Just pass their names as arguments.

Instead of this:

User.select(:id, :email).map { |user| [user.id, user.email] }
# SELECT "users"."id", "users"."email" FROM "users"

#=> [[1, "kate@example.com"], [4, "john@example.org"], [5, "janet@example.com"]]

Just do this:

User.pluck(:id, :email)
# SELECT "users"."id", "users"."email" FROM "users"

#=> [[1, "kate@example.com"], [4, "john@example.org"], [5, "janet@example.com"]]

Summary

When querying the database, it’s important to be mindful about what data we retrieve and what is being done with it. The #pluck method both keeps the SQL query to the minimum and avoids the ActiveRecord object creation overhead when it’s not needed in a given scenario.

Have you used the #pluck method in your project? Please share in the comments.

And if your project could benefit from faster CI builds, consider using Knapsack Pro to achieve that!

How BitBucket Pipeline with parallel Cypress tests can speed up CI build

Tue, 13 Apr 2021 18:00:00 +0000

Do you use BitBucket Pipeline as your CI server? Are you struggling with slow E2E tests in Cypress? Did you know BitBucket Pipeline can run parallel steps? You can use it to distribute your browser tests across several parallel steps to execute end-to-end Cypress tests in a short amount of time.

How to run tests in parallel

Distributing tests across parallel steps to spread the workload and run tests faster might be more challenging than you think. The question is how to divide Cypress test files across the parallel jobs in order to ensure the work is distributed evenly? But… is distributing work evenly what you actually want?

To get the shortest CI build time you want to utilize the available CI resources to the fullest. You want to avoid wasting time. This means that you want to ensure the parallel steps will finish work at a similar time as this would mean there are no bottlenecks in CI machines utilization.

Many things are unknown and unpredictable. This can affect how long it will take to execute tests on BitBucket Pipeline. There are things like:

boot time - time spent on loading your CI docker container
loading npm/yarn dependencies from cache
running Cypress tests
tests can run against different browsers and this can affect how long the tests are executed
sometimes tests can fail and their execution time is different
other times you may have flaky tests randomly failing and you could use Test Retries in Cypress to automatically rerun failed test cases. This results in running a test file for longer.

All of the above contribute to the uncertainty around execution time. It’s hard to know how best to divide test files across the parallel steps to ensure the steps complete work at a similar time. But there is a solution to that - a dynamic test suite split during runtime.

Queue Mode - a dynamic tests split

To distribute tests work across BitBucket Pipeline parallel steps you can use Knapsack Pro with a Queue Mode. You can use @knapsack-pro/cypress npm package that will generate a Queue with a list of test files on the Knapsack Pro API side and then all parallel steps can connect to the queue to consume test files and execute them. This way parallel steps ask for more tests only after they finish executing a set of tests previously fetched from the Knapsack Pro API. You can learn about the details of Queue Mode from the article.

BitBucket Pipeline YML config

Here is an example of a BitBucket Pipeline config in YML. As you can see, there are 3 parallel steps to run Cypress tests via Knapsack Pro. If you would like to run your tests on more parallel jobs you simply need to add more steps.

image: cypress/base:10
options:
  max-time: 30

# job definition for running E2E tests in parallel with KnapsackPro.com
e2e: &e2e
  name: Run E2E tests with @knapsack-pro/cypress
  caches:
    - node
    - cypress
  script:
    # run web application in the background
    - npm run start:ci &
    # env vars from https://support.atlassian.com/bitbucket-cloud/docs/variables-and-secrets/
    - export KNAPSACK_PRO_CI_NODE_BUILD_ID=$BITBUCKET_BUILD_NUMBER
    - export KNAPSACK_PRO_COMMIT_HASH=$BITBUCKET_COMMIT
    - export KNAPSACK_PRO_BRANCH=$BITBUCKET_BRANCH
    - export KNAPSACK_PRO_CI_NODE_TOTAL=$BITBUCKET_PARALLEL_STEP
    - export KNAPSACK_PRO_CI_NODE_INDEX=$BITBUCKET_PARALLEL_STEP_COUNT
    # https://docs.knapsackpro.com/cypress/guide/#configuration-steps
    - export KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true
    - npx @knapsack-pro/cypress
  artifacts:
    # store any generated images and videos as artifacts
    - cypress/screenshots/**
    - cypress/videos/**

pipelines:
  default:
  - step:
      name: Install dependencies
      caches:
        - npm
        - cypress
        - node
      script:
        - npm ci
  - parallel:
    # run N steps in parallel
    - step:
        <<: *e2e
    - step:
        <<: *e2e
    - step:
        <<: *e2e

definitions:
  caches:
    npm: $HOME/.npm
    cypress: $HOME/.cache/Cypress

If you are looking for an example with a custom docker container for a parallel step please see this one.

Please remember to add your API token in KNAPSACK_PRO_TEST_SUITE_TOKEN_CYPRESS environment variable as secure variable.

Summary

BitBucket Pipeline is a CI server that allows running scripts in parallel. You can use parallel steps to distribute your Cypress tests with Knapsack Pro to save time and run CI build as fast as possible.

Using Ruby's 2.7 new #tally method

Sat, 10 Apr 2021 08:00:00 +0000

Ruby 2.7 introduced the Enumberable#tally method. It allows to easily count elements’ occurrences in a given collection. In other words, it literally tallies them up. :)

The use case

Say we have a collection of words, and we’d like to count the occurrences of each word within it.

words = %w(the be to of and a in that have I it for not on with he as you do at
           the not for it be we her so up and a to for on with)
# some of the most common English words based on https://en.wikipedia.org/wiki/Most_common_words_in_English

Pre Ruby 2.7

In previous versions of Ruby, whenever we encountered a logic counting occurrences of collection elements (here: words in the array), we would most likely see a code similar to this:

word_counts = {}
words.each do |word|
  if word_counts[word]
    word_counts[word] += 1
  else
    word_counts[word] = 1
  end
end

word_counts
# => {"the"=>2, "be"=>2, "to"=>2, "of"=>1, "and"=>2, "a"=>2, "in"=>1, "that"=>1, "have"=>1, "I"=>1, "it"=>2, "for"=>3, "not"=>2, "on"=>2, "with"=>2, "he"=>1, "as"=>1, "you"=>1, "do"=>1, "at"=>1, "we"=>1, "her"=>1, "so"=>1, "up"=>1}

This could be simplified by initializing the Hash with a default value of 0 (which makes total sense in this case):

word_counts = Hash.new(0)
words.each { |word| word_counts[word] += 1 }

word_counts
# => {"the"=>2, "be"=>2, "to"=>2, "of"=>1, "and"=>2, "a"=>2, "in"=>1, "that"=>1, "have"=>1, "I"=>1, "it"=>2, "for"=>3, "not"=>2, "on"=>2, "with"=>2, "he"=>1, "as"=>1, "you"=>1, "do"=>1, "at"=>1, "we"=>1, "her"=>1, "so"=>1, "up"=>1}

We could even transform it further to a one-liner using #each_with_object:

words.each_with_object(Hash.new(0)) { |word, word_counts| word_counts[word] += 1 }
# => {"the"=>2, "be"=>2, "to"=>2, "of"=>1, "and"=>2, "a"=>2, "in"=>1, "that"=>1, "have"=>1, "I"=>1, "it"=>2, "for"=>3, "not"=>2, "on"=>2, "with"=>2, "he"=>1, "as"=>1, "you"=>1, "do"=>1, "at"=>1, "we"=>1, "her"=>1, "so"=>1, "up"=>1}

I personally like the second solution the most. I think it strikes the best balance between simplicity and ease of understanding.

As we can see, all of the above approaches rely on iterating through the collection to tally up its members.

There is also another approach that could be chosen: relying on Enumerable#group_by. If I were to use this one, the code I would write would probably look something like this:

words.group_by(&:itself). # this produces a structure like: {"the"=>["the", "the"], "be"=>["be", "be"], ... }
      map { |word, occurrences| [word, occurrences.count] }.
      to_h
# => {"the"=>2, "be"=>2, "to"=>2, "of"=>1, "and"=>2, "a"=>2, "in"=>1, "that"=>1, "have"=>1, "I"=>1, "it"=>2, "for"=>3, "not"=>2, "on"=>2, "with"=>2, "he"=>1, "as"=>1, "you"=>1, "do"=>1, "at"=>1, "we"=>1, "her"=>1, "so"=>1, "up"=>1}

The new way

Since Ruby 2.7, we could get the same result (a Hash containing numbers of occurrences of each element) by simply invoking the #tally method on the collection.

words.tally
# => {"the"=>2, "be"=>2, "to"=>2, "of"=>1, "and"=>2, "a"=>2, "in"=>1, "that"=>1, "have"=>1, "I"=>1, "it"=>2, "for"=>3, "not"=>2, "on"=>2, "with"=>2, "he"=>1, "as"=>1, "you"=>1, "do"=>1, "at"=>1, "we"=>1, "her"=>1, "so"=>1, "up"=>1}

It’s not only a nice shortcut, it also expresses the intent in a clear way - something many of us love in Ruby code.

Summary

Have you had a chance to use the Enumerable#tally method in your production code yet? Please share in the comments below!

Do you use continuous integration in your Ruby project? Consider using Knapsack Pro to increase your team’s productivity by shortening build times on your CI server!

How to run Ruby on Rails tests on Github Actions using RSpec

Thu, 08 Apr 2021 06:00:00 +0000

Are you thinking about migrating a Ruby on Rails project CI pipeline to Github Actions? You will learn how to configure the Rails app to run RSpec tests using Github Actions.

This article covers a few things for Github Actions YAML config:

how to use ruby/setup-ruby action to install Ruby gems with bundler and automatically cache gems. This way you can load Ruby gems for your project from the cache and run CI build fast.
how to use Postgres on Github Actions
how to use Redis on Github Actions
how to use Github Actions build matrix to run parallel jobs and execute RSpec tests spread across multiple jobs to save time

Github Actions YML config for Rails application

ruby/setup-ruby action

ruby/setup-ruby is an action that you can use to install a particular Ruby programming language version. It allows you to cache Ruby gems based on your Gemfile.lock out of the box.

- name: Set up Ruby
  uses: ruby/setup-ruby@v1
  with:
    # Not needed with a .ruby-version file
    ruby-version: 2.7
    # runs 'bundle install' and caches installed gems automatically
    bundler-cache: true

How to configure Postgres on Github Actions

To use Postgres on Github Actions you need to set up a service for Postgres. I recommend using additional options that will configure Postgres to use RAM instead of disk. This way your database can run faster in a testing environment.

In the config below, we also pass the settings for doing a health check to ensure the database is up and running before you start running tests.

# If you need DB like PostgreSQL, Redis then define service below.

# https://github.com/actions/example-services/tree/main/.github/workflows

services:
  postgres:
    image: postgres:10.8
    env:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: ""
      POSTGRES_DB: postgres
    ports:
      - 5432:5432
    # needed because the postgres container does not provide a healthcheck
    # tmpfs makes DB faster by using RAM
    options: >-
      --mount type=tmpfs,destination=/var/lib/postgresql/data
      --health-cmd pg_isready
      --health-interval 10s
      --health-timeout 5s
      --health-retries 5

How to configure Redis on Github Actions

You can use Redis Docker container to start Redis server on Github Actions. See how simple it is:

services:
  redis:
    image: redis
    ports:
      - 6379:6379
    options: --entrypoint redis-server

How to use Github Actions build matrix to run tests with parallel jobs

You can use the build matrix in Github Actions to run multiple jobs at the same time.

You will need to split test files between these parallel jobs. For that, you can use Knapsack Pro with Queue Mode to distribute tests evenly between the jobs. This way you can ensure the proper amount of tests is executed on each job and the workload is well balanced between the jobs. Simply speaking this way you can make sure the CI build is as fast as possible - it has optimal execution time.

- name: Run tests
  env:
    KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC: ${{ secrets.KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC }}
    KNAPSACK_PRO_CI_NODE_TOTAL: ${{ matrix.ci_node_total }}
    KNAPSACK_PRO_CI_NODE_INDEX: ${{ matrix.ci_node_index }}
    KNAPSACK_PRO_FIXED_QUEUE_SPLIT: true
    KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES: true
    KNAPSACK_PRO_LOG_LEVEL: info
  run: |
    bundle exec rake knapsack_pro:queue:rspec

You can see that for RSpec we also use a knapsack_pro Ruby gem flag KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES. It allows to automatically detect slow test files and split them between parallel jobs.

You can learn more about it in a separate article explaining how to run slow RSpec files on Github Actions with parallel jobs by doing an auto split of the spec file by test examples.

Full YML config for Github Actions and Ruby on Rails project

Here is the full configuration of the CI pipeline for Github Actions. You can use it to run tests for your Rails project.

name: Main

on: [push]

jobs:
  test:
    runs-on: ubuntu-latest

    # If you need DB like PostgreSQL, Redis then define service below.
    # https://github.com/actions/example-services/tree/main/.github/workflows
    services:
      postgres:
        image: postgres:10.8
        env:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: ""
          POSTGRES_DB: postgres
        ports:
          - 5432:5432
        # needed because the postgres container does not provide a healthcheck
        # tmpfs makes DB faster by using RAM
        options: >-
          --mount type=tmpfs,destination=/var/lib/postgresql/data
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
      redis:
        image: redis
        ports:
          - 6379:6379
        options: --entrypoint redis-server

    # https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#jobsjob_idstrategymatrix
    strategy:
      fail-fast: false
      matrix:
        # [n] - where the n is a number of parallel jobs you want to run your tests on.
        # Use a higher number if you have slow tests to split them between more parallel jobs.
        # Remember to update the value of the `ci_node_index` below to (0..n-1).
        ci_node_total: [8]
        # Indexes for parallel jobs (starting from zero).
        # E.g. use [0, 1] for 2 parallel jobs, [0, 1, 2] for 3 parallel jobs, etc.
        ci_node_index: [0, 1, 2, 3, 4, 5, 6, 7]

    env:
      RAILS_ENV: test
      GEMFILE_RUBY_VERSION: 2.7.2
      PGHOST: localhost
      PGUSER: postgres
      # Rails verifies the time zone in DB is the same as the time zone of the Rails app
      TZ: "Europe/Warsaw"

    steps:
      - uses: actions/checkout@v2

      - name: Set up Ruby
        uses: ruby/setup-ruby@v1
        with:
          # Not needed with a .ruby-version file
          ruby-version: 2.7
          # runs 'bundle install' and caches installed gems automatically
          bundler-cache: true

      - name: Create DB
        run: |
          bin/rails db:prepare
      - name: Run tests
        env:
          KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC: ${{ secrets.KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC }}
          KNAPSACK_PRO_CI_NODE_TOTAL: ${{ matrix.ci_node_total }}
          KNAPSACK_PRO_CI_NODE_INDEX: ${{ matrix.ci_node_index }}
          KNAPSACK_PRO_LOG_LEVEL: info
          # if you use Knapsack Pro Queue Mode you must set below env variable
          # to be able to retry CI build and run previously recorded tests
          # https://docs.knapsackpro.com/ruby/queue-mode/#dynamic-split-vs-fixed-split
          KNAPSACK_PRO_FIXED_QUEUE_SPLIT: true
          # RSpec split test files by test examples feature - it's optional
          # https://knapsackpro.com/faq/question/how-to-split-slow-rspec-test-files-by-test-examples-by-individual-it
          KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES: true
        run: |
          bundle exec rake knapsack_pro:queue:rspec

Summary

You’ve just learned how to set up your Rails application on Github Actions. I hope this will help you if you migrate your project from a different CI server to Github Actions.

You can learn more about Knapsack Pro and how it can help you run tests fast using parallel jobs on CI. It works with RSpec, Cucumber, Minitest, and other Ruby test runners. Knapsack Pro can also work with JavaScript test runners and has a native API integration.

Run Minitest on Github Actions with parallel jobs using build matrix

Sun, 04 Apr 2021 13:00:00 +0000

How to run Ruby on Rails tests in Minitest on Github Actions? What to do if tests are slow? How to manage complex workflows? You can use Github Actions build matrices to divide Minitest files between jobs and run the test suite much faster.

If your Minitest tests are taking dozens of minutes and you would like to save some time for your Ruby engineering team then you could use tests parallelization on your CI server.

To run tests as fast as possible you need to split them into equal buckets (into parallel jobs). But how to do it? Some of the test files can be super fast to execute, other Minitest files can take minutes if they run system tests (E2E tests).

There is also an aspect of preparing the test environment for each parallel job. By preparing I mean you need to clone a repository, install ruby gems or load them from a cache, maybe you need to load some docker container, etc. This can take various amounts of time on each parallel job. Random network errors happen like network delay to load cached gems, or maybe Github Actions from time to time will start one of your jobs late compared to others. It’s an inevitable issue in the network environment and can cause your tests to run for a different amount of time on each parallel job. This is visible on the graph below and it causes the CI build to be slower.

In a perfect scenario you would like to cover all these problems and no matter what still be able to split Minitest work in parallel jobs in a way that ensures the tests on each parallel job completes at a similar time. This guarantees no bottlenecks. The perfect tests split is on the below graph.

Split tests in a dynamic way with Queue Mode

You can use Knapsack Pro Queue Mode to split tests in a dynamic way between parallel jobs. This way each job consumes tests from a queue until the queue is empty. Simply speaking this allows you to utilize your CI server resources efficiently and run tests in optimal time.

I described how Queue Mode splits Ruby and JavaScript tests in parallel with a dynamic test suite split. You can learn from that article about it.

Github Actions build matrix to run parallel tests

Github Actions has a build matrix feature that allows running many jobs at the same time. You can use it to run your Minitest tests between parallel jobs.

Below is a full Github Actions YML config for a Rails project and Minitest. The tests are split with knapsack_pro Ruby gem and Queue Mode.

name: Main

on: [push]

jobs:
  test:
    runs-on: ubuntu-latest

    # If you need DB like PostgreSQL, Redis then define service below.
    # https://github.com/actions/example-services/tree/main/.github/workflows
    services:
      postgres:
        image: postgres:10.8
        env:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: ""
          POSTGRES_DB: postgres
        ports:
          - 5432:5432
        # needed because the postgres container does not provide a healthcheck
        # tmpfs makes DB faster by using RAM
        options: >-
          --mount type=tmpfs,destination=/var/lib/postgresql/data
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
      redis:
        image: redis
        ports:
          - 6379:6379
        options: --entrypoint redis-server

    # https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#jobsjob_idstrategymatrix
    strategy:
      fail-fast: false
      matrix:
        # Set N number of parallel jobs you want to run tests on.
        # Use higher number if you have slow tests to split them on more parallel jobs.
        # Remember to update ci_node_index below to 0..N-1
        ci_node_total: [8]
        # set N-1 indexes for parallel jobs
        # When you run 2 parallel jobs then first job will have index 0, the second job will have index 1 etc
        ci_node_index: [0, 1, 2, 3, 4, 5, 6, 7]

    env:
      RAILS_ENV: test
      PGHOST: localhost
      PGUSER: postgres
      # Rails verifies Time Zone in DB is the same as time zone of the Rails app
      TZ: "Europe/Warsaw"

    steps:
      - uses: actions/checkout@v2

      - name: Set up Ruby
        uses: ruby/setup-ruby@v1
        with:
          # Not needed with a .ruby-version file
          ruby-version: 2.7
          # runs 'bundle install' and caches installed gems automatically
          bundler-cache: true

      - name: Create DB
        run: |
          bin/rails db:prepare
      - name: Run tests
        env:
          KNAPSACK_PRO_TEST_SUITE_TOKEN_MINITEST: ${{ secrets.KNAPSACK_PRO_TEST_SUITE_TOKEN_MINITEST }}
          KNAPSACK_PRO_CI_NODE_TOTAL: ${{ matrix.ci_node_total }}
          KNAPSACK_PRO_CI_NODE_INDEX: ${{ matrix.ci_node_index }}
          KNAPSACK_PRO_FIXED_QUEUE_SPLIT: true
          KNAPSACK_PRO_LOG_LEVEL: info
        run: |
          bundle exec rake knapsack_pro:queue:minitest

Summary

As you can see the slow Minitest test suite doesn’t need to be an issue for you. QA, Testers, or Automation Engineers could benefit from improving the CI build speed and allowing their software developers team to deliver products faster. You can learn more at Knapsack Pro.

Fix: 'warning: Using the last argument as keyword parameters is deprecated'

Sat, 03 Apr 2021 07:00:00 +0000

Have you encountered the warning: Using the last argument as keyword parameters is deprecated message in your Ruby project? You probably started seeing this after upgrading your project to Ruby 2.7. This version introduced the deprecation as a means to prepare users for a breaking change in Ruby 3. Let’s learn how to fix this!

The explanation

In Ruby 2, you could pass a Hash as the last argument to the method that expects keyword arguments, and the key-value pairs from the hash would be treated as keyword arguments.

def greet(greeting, name:)
  puts "#{greeting}, #{name}!"
end

options = { name: "John" }

# Ruby 2.6
greet("Hello", options)
#=> "Hello, John!"

Ruby 2.7 still allows this, but now warns about this usage.

# Ruby 2.7
greet("Hello", options)
warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
#=> "Hello, John!"

And Ruby 3 raises an ArgumentError.

# Ruby 3
greet("Hello", options)
ArgumentError (wrong number of arguments (given 2, expected 1; required keyword: name))

In the above example, greeting is a “positional argument” of the #greet method, while name: is a keyword argument. As we can see, in Ruby 2 we could invoke the method as if we were just passing two regular arguments to it, using a hash object as the last one. Ruby’s interpreter would then automagically “deconstruct” the hash as needed and match its key-value pairs to the keyword argument expected by the method.

The Ruby team believed that this behavior is prone to be quite confusing and decided to alter it in Ruby 3. That’s why you see these deprecation warnings in Ruby 2.7, and the usage is no longer valid since Ruby 3.

So how to fix it?

There are two ways to resolve the warnings in Ruby 2.7 (and prepare your code for Ruby 3).

The simple solution

The most non-invasive way to fix the warning is to do what the warning itself suggests - just use the double splat (**) operator before the hash. This way, we explicitly say that the Hash’s contents are meant to be “exploded” to match the required keyword arguments.

def greet(greeting, name:)
  puts "#{greeting}, #{name}!"
end

options = { name: "John" }

# Ruby 2.7
greet("Hello", **options)
#=> "Hello, John!"

# Ruby 3
greet("Hello", **options)
#=> "Hello, John!"

Works like a charm!

Not using the Hash

Another option would obviously be to abstain from using the hash object in the first place. This may require a bigger refactor in your code or simply be undesired. Still, it might be something to consider, especially in simple cases.

def greet(greeting, name:)
  puts "#{greeting}, #{name}!"
end

# Ruby 2.7
greet("Hello", name: "John")
#=> "Hello, John!"

# Ruby 3
greet("Hello", name: "John")
#=> "Hello, John!"

Avoiding the shortcuts

As with every warning, we might get an idea to ignore or even suppress it. I advise against this. The warning is there for a reason, and it’s best to fix it and avoid troubles with future version upgrades. This is even more true given how straightforward this warning is to fix in most cases.

Conclusion

Have you encountered this warning in your code? Please let us know in the comments!

And if your Ruby project suffers from slow CI builds, consider using Knapsack Pro to improve the productivity of your team!

How and when to use Ruby case statements

Thu, 01 Apr 2021 11:00:00 +0000

Using the case expression in Ruby is a great way to write conditionals in a clear, and succinct way. Despite that, it’s not hard to encounter convoluted if constructs in places where refactoring to the case statement would result in huge improvements. Let’s learn more about the case expression and when it’s best to use it.

Expressing logic simply

Consider this piece of code:

number = 6

if number > 0 && number <= 3
  'low value'
elsif number > 3 && number <= 7
  'medium value'
elsif number > 7 && number <= 10
  ' high value'
else
  'invalid value'
end

It seems like quite a complicated way to express a pretty straightforward logic, doesn’t it? Still, I bet that you have seen a construct like that many times. I know I have.

This logic could be greatly simplified with the case expression:

case number
when (0..3)
  'low value'
when (4..7)
  'medium value'
when (8..10)
  'high value'
else
  'invalid value'
end

We could also use the then keyword to make this even more succinct:

case number
when (0..3)  then 'low value'
when (4..7)  then 'medium value'
when (8..10) then 'high value'
else              'invalid value'
end

This code effectively works like:

if (0..3) === number
  'low value'
elsif (4..7) === number
  'medium value'
elsif (8..10) === number
  'high value'
else
  'invalid value'
end

As you can see, the case statements are most powerful when our logic is concerned with a value of a single object (the number in the above examples), which is provided next to the case keyword. What comes after each when acts as a pattern. It doesn’t have to be a Range necessarily - this would work with any object. Similar to how the #grep method behaves, the case expression results in invoking the #=== method on the given patterns. In fact, the #=== method in Ruby is often called the “case equality”, as it’s mostly known for being used in case statements.

You can consult the Ruby documentation and look into how the #=== method is implement in various classes. On the Object class, it works the same as the regular equality method, the #==. It’s then up to descending classes to implement their own interpretation of the “case equality”.

Patterns in the case statements

Let’s look at some examples of objects being used as patterns in the case statements.

The case statement with integers as patterns:

case number
when 0 then false
when 1 then true
end

With strings:

case answer
when 'Y' then true
when 'N' then false
end

And with regexes:

case string
when (/^a/) then 'starts with an A'
when (/^z/) then 'starts with a Z'
end

You can of course mix the types of patterns within one statement:

case number
when 0       then 'zero'
when (1..3)  then 'low value'
when (4..7)  then 'medium value'
when (8..10) then 'high value'
end

One last tip. You can provide multiple patterns for each when. This would effectively mimic the logical || operator, with each pattern being compared against. To illustrate this, let’s expand on a previous example:

case answer.downcase
when 'y', 'yes' then true
when 'n', 'no' then false
end

Under the hood, this works just like:

if 'y' === answer.downcase  || 'yes' === answer.downcase
  true
elsif 'n' === answer.downcase || 'no' === answer.downcase
  false
end

Summary

I hope that you find this quick recap of the case expression useful. How often are you using case statements in your projects? Please share in the comments.

If your Ruby project suffers from slow CI build times, consider using Knapsack Pro to improve the productivity of your team.

How to load Ruby gems from cache on Github Actions

Thu, 25 Mar 2021 07:00:00 +0000

How to start CI build faster by loading Ruby gems from cache on Github Actions? You can start running your tests for a Ruby on Rails project quicker if you manage to set up all dependencies in a short amount of time. Caching can be helpful with that. Ruby gems needed for your project can be cached by Github Actions and thanks to that they can be loaded much faster when you run a new CI build.

You will learn how to configure Github Actions using:

actions/cache - it’s a popular solution to cache Ruby gems.
ruby/setup-ruby - it’s a solution to install a specific Ruby version and cache Ruby gems with bundler. Two features in one action.

actions/cache - just cache dependencies

Actions/cache is a popular solution that can be used to save data into the cache and restore it during the next CI build. It’s often used for Ruby on Rails projects that also use actions/setup-ruby for managing the Ruby version on Github Actions.

Let’s look at the Github Actions caching config example using actions/cache.

# .github/workflows/main.yml
name: Main
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - uses: actions/cache@v2
        with:
          path: vendor/bundle
          key: ${{ runner.os }}-gems-${{ hashFiles('**/Gemfile.lock') }}
          restore-keys: |
            ${{ runner.os }}-gems-

      - name: Bundle install
        env:
          RAILS_ENV: test
        run: |
          bundle config path vendor/bundle
          bundle install --jobs 4 --retry 3

You need to specify a directory path that will be cached. It’s vendor/bundle in our case.
You also generate a unique cache key based on the OS version and Gemfile.lock file. When you change the operating system version or you install a new gem and Gemfile.lock changes then as a result the new key value will be generated.
You need to configure the bundler to install all your Ruby gems to the directory vendor/bundle.
You can use bundler options:
- --jobs 4 - install gems using parallel workers. This allows faster gems installation.
- --retry 3 - makes 3 attempts to connect to Rubygems if there is a network issue (for instance temporary downtime of Rubygems.org)

If you would like to see the full YAML config for the Github Actions and Rails project you can take a look at some of our articles:

ruby/setup-ruby - install Ruby and cache gems

In the previous section, we mentioned the actions/setup-ruby is often used with Ruby on Rails projects. The actions/setup-ruby has been deprecated so it’s recommended to use ruby/setup-ruby action nowadays. It already has caching feature that you could use. Let’s see how.

# .github/workflows/main.yml
name: Main
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2

    - uses: ruby/setup-ruby@v1
      with:
        # Not needed with a .ruby-version file
        ruby-version: 2.7
        # runs 'bundle install' and caches installed gems automatically
        bundler-cache: true

    # run RSpec tests
    - run: bundle exec rspec

As you can see using ruby/setup-ruby for managing the Ruby version and gems caching is much simpler. You just add an option bundler-cache: true and that’s it.

You can read in ruby/setup-ruby documentation:

“It is also possible to cache gems manually, but this is not recommended because it is verbose and very difficult to do correctly. There are many concerns which means using actions/cache is never enough for caching gems (e.g., incomplete cache key, cleaning old gems when restoring from another key, correctly hashing the lockfile if not checked in, OS versions, ABI compatibility for ruby-head, etc). So, please use bundler-cache: true instead…”

Summary

You saw 2 ways of caching Ruby gems on Github Actions. There are also other ways to make your CI build faster like running tests in parallel. You can learn more about test parallelisation here or simply check the Knapsack Pro homepage.

Faster Cypress + RSpec test suite for Rails apps on GitHub Actions using Knapsack Pro

Tue, 23 Mar 2021 18:00:00 +0000

Cypress is an amazing tool for end to end testing Rails applications, but large test suites can quickly take upwards of 20 minutes to run. That’s where Knapsack Pro comes in. Knapsack Pro Queue Mode to intelligently split your test suite into jobs that can be run in parallel, reducing run time to only a few minutes. In this article we’ll show how to quickly implement Knapsack Pro Queue Mode to speed up both Cypress & RSpec test suites in a Ruby on Rails app on Github Actions

Set up Knapsack Pro API Keys

First step is to go to your Knapsack dashboard and grab your API keys for both RSpec and Cypress. Once you have those, go to your Github Repo’s settings, for example:

Set up your GitHub Actions config file

Once you’ve added your KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC and KNAPSACK_PRO_TEST_SUITE_TOKEN_CYPRESS secrets, the next step is setting up your GitHub Actions configuration file to use the Knapsack Runner in place of the normal commands to run RSpec and Cypress.

Existing GitHub Actions config

For those that already have a GH actions config file setup (e.g. .github/workflows/ci.yml), here’s all that you should need to change to get Knapsack Pro Queue Mode working for both Cypress and RSpec.

Change your RSpec run command to use Knapsack:

+      strategy:
+        fail-fast: false
+        matrix:
+          # Set N number of parallel jobs you want to run tests on.
+          # Use higher number if you have slow tests to split them on more parallel jobs.
+          # Remember to update ci_node_index below to 0..N-1
+          ci_node_total: [2]
+          # set N-1 indexes for parallel jobs
+          # When you run 2 parallel jobs then first job will have index 0, the second job will have index 1 etc
+          ci_node_index: [0, 1]
      - name: Run RSpec Tests
+      env:
+        KNAPSACK_PRO_CI_NODE_TOTAL: ${{ matrix.ci_node_total }}
+        KNAPSACK_PRO_CI_NODE_INDEX: ${{ matrix.ci_node_index }}
+        KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC: ${{ secrets.KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC }}
+        KNAPSACK_PRO_FIXED_QUEUE_SPLIT: true
+    run: bin/rake knapsack_pro:queue:rspec  # Run RSpec using Knapsack Pro Queue Mode
-    run: bin/rspec spec

Change your cypress run command to use Knapsack as well:

+      strategy:
+        fail-fast: false
+        matrix:
+          ci_node_total: [5]
+          # set N-1 indexes for parallel jobs
+          # When you run 2 parallel jobs then first job will have index 0, the second job will have index 1 etc
+          ci_node_index: [0, 1, 2, 3, 4]
      - name: Run cypress tests
+        env:
+          KNAPSACK_PRO_TEST_SUITE_TOKEN_CYPRESS: ${{ secrets.KNAPSACK_PRO_TEST_SUITE_TOKEN_CYPRESS }}
+          KNAPSACK_PRO_CI_NODE_TOTAL: ${{ matrix.ci_node_total }}
+          KNAPSACK_PRO_CI_NODE_INDEX: ${{ matrix.ci_node_index }}
+          KNAPSACK_PRO_FIXED_QUEUE_SPLIT: true
+          KNAPSACK_PRO_TEST_FILE_PATTERN: '{cypress/**/*,app/javascript/**/*.component}.spec.{js,ts,tsx}'
+        run: yarn @knapsack-pro/cypress
-        run: yarn cypress run

New Github Actions config file

For those starting from scratch, here’s a full example .github/workflows/ci.yaml for a Rails app with Cypress + RSpec with Knapsack tokens for RSpec and Cypress already added.

name: ci
on: [push]
jobs:
  # OPTIONAL: Cancel any previous CI runs to save your GH Actions minutes
  cancel:
    name: "Cancel Previous Runs"
    runs-on: ubuntu-20.04
    timeout-minutes: 3
    steps:
      - uses: styfle/cancel-workflow-action@0.8.0
        with:
          workflow_id: 3553203
  yarn:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-node@v2-beta
        with:
          node-version: "12"
      - uses: actions/cache@v2
        with:
          path: "**/node_modules"
          key: ${{ runner.os }}-yarn-${{ hashFiles('yarn.lock') }}
          restore-keys: |
            ${{ runner.os }}-yarn-
      - name: Yarn install
        run: yarn install --frozen-lockfile
        env:
          CYPRESS_INSTALL_BINARY: 0 # Prevent installing Cypress binary until later when it's needed
  bundle:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: ruby/setup-ruby@v1
        with:
          bundler-cache: true
  rspec:
    timeout-minutes: 3 # Adjust as needed, just here to prevent accidentally using up all your minutes from a silly infinite loop of some kind
    env:
      RAILS_ENV: test
    runs-on: ubuntu-latest
    needs: [bundle]
    strategy:
      fail-fast: false
      matrix:
        # Set N number of parallel jobs you want to run tests on.
        # Use higher number if you have slow tests to split them on more parallel jobs.
        # Remember to update ci_node_index below to 0..N-1
        ci_node_total: [2]
        # set N-1 indexes for parallel jobs
        # When you run 2 parallel jobs then first job will have index 0, the second job will have index 1 etc
        ci_node_index: [0, 1]
    steps:
      - uses: actions/checkout@v2
      - uses: ruby/setup-ruby@v1
        with:
          bundler-cache: true
      - name: Build DB
        run: bin/rails db:schema:load
      - name: Run RSpec Tests
        env:
          PGPORT: ${{ job.services.postgres.ports[5432] }} # get randomly assigned published port
          KNAPSACK_PRO_CI_NODE_TOTAL: ${{ matrix.ci_node_total }}
          KNAPSACK_PRO_CI_NODE_INDEX: ${{ matrix.ci_node_index }}
          KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC: ${{ secrets.KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC }}
          KNAPSACK_PRO_FIXED_QUEUE_SPLIT: true
        run: bin/rake knapsack_pro:queue:rspec # Run RSpec using Knapsack Pro Queue Mode
  cypress:
    timeout-minutes: 20 # Adjust as needed, just here to prevent accidentally using up all your minutes from a silly infinite loop of some kind
    env:
      RAILS_ENV: test
      RACK_ENV: test
      GITHUB_TOKEN: ${{ github.token }}
    runs-on: ubuntu-latest
    needs: [bundle, yarn]
    strategy:
      fail-fast: false
      matrix:
        ci_node_total: [5]
        # set N-1 indexes for parallel jobs
        # When you run 5 parallel jobs then first job will have index 0, the second job will have index 1 etc
        ci_node_index: [0, 1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-node@v2-beta
        with:
          node-version: "12"
      - uses: actions/cache@v2
        with:
          path: "**/node_modules"
          key: ${{ runner.os }}-yarn-${{ hashFiles('yarn.lock') }}
      - uses: ruby/setup-ruby@v1
        with:
          bundler-cache: true
      - name: Build DB
        run: bin/rails db:schema:load
      - name: Run Rails Server in background
        run: bin/rails server -p 3000 &
      - run: npx cypress -v > .cypress-version
      - uses: actions/cache@v2
        with:
          path: ~/.cache/Cypress
          key: cypress-cache-v3-${{ runner.os }}-${{ hashFiles('.cypress-version') }}
      - run: yarn cypress install
      - uses: actions/setup-node@v2-beta
        with:
          node-version: "12"
      - run: yarn wait-on 'http-get://localhost:3000' -t 30000
      - name: Run tests
        env:
          KNAPSACK_PRO_TEST_SUITE_TOKEN_CYPRESS: ${{ secrets.KNAPSACK_PRO_TEST_SUITE_TOKEN_CYPRESS }}
          KNAPSACK_PRO_CI_NODE_TOTAL: ${{ matrix.ci_node_total }}
          KNAPSACK_PRO_CI_NODE_INDEX: ${{ matrix.ci_node_index }}
          KNAPSACK_PRO_FIXED_QUEUE_SPLIT: true
          KNAPSACK_PRO_TEST_FILE_PATTERN: "{cypress/**/*,app/javascript/**/*.component}.spec.{js,ts,tsx}"
        run: yarn @knapsack-pro/cypress # Run Cypress using Knapsack Pro Queue Mode
      # Save screenshots and videos of failed tests and make them available as Github build artifacts
      - uses: actions/upload-artifact@v2
        if: failure()
        with:
          name: cypress-screenshots
          path: cypress/screenshots
      - uses: actions/upload-artifact@v2
        if: failure()
        with:
          name: cypress-videos
          path: cypress/videos
      - uses: actions/upload-artifact@v2
        if: failure()
        with:
          name: cypress-logs
          path: cypress/logs

Add Knapsack Pro gem and npm package

Add the Knapsack Pro gem to your Gemfile:

group :development, :test do
  #...
  gem 'knapsack_pro'
end

Add the Knapsack Pro npm package with yarn add --dev @knapsack-pro/cypress

Run your tests & view your results

Once you’ve completed the above steps, trigger a test run on your repo. You should see multiple jobs for both RSpec and Cypress like so:

Now check your Knapsack Dashboard for the results 🚀

The complete example Rails app can be found in this repository. Happy testing!

Auto-scaling Buildkite CI build agents for RSpec (run parallel jobs in minutes instead of hours)

Fri, 19 Mar 2021 07:00:00 +0000

If your RSpec test suite runs for hours, you could shorten that to just minutes with parallel jobs using Buildkite agents. You will learn how to run parallel tests in optimal CI build time for your Ruby on Rails project. I will also show you a few useful things for Buildkite CI like:

A real RSpec test suite taking 13 hours and 32 minutes executed in only 5 minutes 20 seconds by using 151 parallel Buildkite agents with knapsack_pro Ruby gem.
How to distribute test files between parallel jobs using Queue Mode in Knapsack Pro to utilize CI machines optimally.
A simple example of CI Buildkite parallelism config.
An advanced example of Buildkite config with Elastic CI Stack for AWS.
Why you might want to use AWS Spot Instances
How to automatically split slow RSpec test files by test examples (test cases) between parallel Buildkite agents

A real RSpec test suite taking 13 hours and executed in only 5 minutes

I’d like to show you the results from a real project for running RSpec parallel tests. The project we are looking at here is huge and its RSpec tests run time is 13 hours and 32 minutes. It’s super slow. You can imagine creating a git commit and waiting 13 hours to find out the next day that your code breaks something else in the project. You can’t work like that!

The solution for this is to run tests in parallel on many CI machines using Buildkite agents. Each CI machine has a Buildkite agent installed that will run a chunk of the RSpec test suite. Below you can see an example of running ~13 hours test suite across 151 parallel Buildkite agents. This allows running the whole RSpec test suite in just 5 minutes 20 seconds!

The above graph comes from the Knapsack Pro user dashboard. 151 parallel jobs are a lot of machines. It would take the whole screen to show you 151 bars. You can only see the last few bars on the graph. The bars are showing how the RSpec test files were split between parallel machines.

You can see that each parallel machine finishes work at a similar time. The right edges of all of the bars are pretty close to each other. This is the important part. You want to ensure the RSpec work is distributed evenly between parallel jobs. This way you can avoid a bottleneck - a slow job running too many test files. I’ll show you how to do it.

How to distribute test files between parallel jobs using Queue Mode in Knapsack Pro to utilize CI machines optimally

To run CI build as fast as possible we need to utilize our available resources as much as we can. This means the work of running RSpec tests should be split between parallel machines evenly.

The bigger the test suite, the longer it takes to run it and more edge cases can happen when you split running tests among many machines in the network. Some of the possible edge cases:

some of the test files take longer than others to run (for instance E2E test files)
some of the test cases fail and run quicker, some don’t and run longer. This affects the overall time spent by the CI machine on running your tests.
some of the test cases take longer because they must connect with network/external API etc - this adds uncertainty to their execution time
some of the parallel machines spend more time on boot time
- installing Ruby gems takes longer
- loading Ruby gems from cache is slow
- or simply the CI provider has not started your job yet
- or maybe you have not enough available machines in the pool of available agents

Multiple things can disrupt the spread of work between parallel nodes.

Our ultimate goal is to ensure all machines finish work at a similar time because this means every machine received a workload that was suitable to its available capabilities. This means that, if a machine started work very late it will run only a small part of the tests. If another machine started work very early it will run more tests. This will even out the ending time between parallel machines. All this is possible thanks to Queue Mode in knapsack_pro Ruby gem, it will take care of running tests in parallel for you. Queue Mode splits test files dynamically between parallel jobs to ensure the jobs completes at the same time.

You can see an example of running a small RSpec test suite across 2 parallel Buildkite agents for the Ruby on Rails project.

A simple example of CI Buildkite parallelism config

Here is a very simple example of Buildkite config to run 2 parallel jobs as you can see on the screenshot.

# .buildkite/pipeline.yml

env:
  # You should hide you secrets like API token
  # Please follow https://buildkite.com/docs/pipelines/security/secrets/managing
  KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC: "204abb31f698a6686120a40efeff31e5"
  # allow to run the same set of test files on job retry
  # https://github.com/KnapsackPro/knapsack_pro-ruby#knapsack_pro_fixed_queue_split-remember-queue-split-on-retry-ci-node
  KNAPSACK_PRO_FIXED_QUEUE_SPLIT: true

steps:
  - command: "bundle exec rake knapsack_pro:queue:rspec"
    parallelism: 2

Please note that you should hide your credentials like the Knapsack Pro API token and not commit it into your repository. You can refer to the Buildkite secrets documentation.

An advanced Buildkite config with Elastic CI Stack for AWS

If you want to run your RSpec project on dozen or even hundreds of parallel machines, you need powerful resources. In such a case, you can follow the Buildkite tutorial about AWS setup. The Elastic CI Stack for AWS gives you a private, autoscaling Buildkite Agent cluster in your own AWS account.

AWS Spot Instances can save you money

AWS offers Spot Instances. These machines are cheap but they can be withdrawn by AWS at any time. This means that you can run cheap machines for your CI but from time to time the AWS may kill one of your parallel machines. Such a scenario can be handled by the Knapsack Pro. It remembers the set of test files allocated to the AWS machine that was running the tests. When the machine is withdrawn and later on retried by the Buildkite retry feature then the proper test files will be executed as you would expect.

Buildkite retry feature

Buildkite config allows for automatic retry of your job. This can be helpful when you use AWS Spot Instances. When AWS shuts down your machine during test runtime due to withdrawal then Buildkite can automatically run a new job on a new machine.

Another use case for the automatic retry is when you have flaky Ruby tests that sometimes pass green or fail red. You can use Buildkite to retry the failing job in such a case.

My recommendation is to use the rspec-retry gem as a first choice. RSpec-retry gem will retry only failing test cases instead of all test files assigned to the parallel machine. The second option you can try is to rely on the Buildkite retry feature. It will retry the CI node and all tests assigned to it by Knapsack Pro API.

How to automatically split large slow RSpec test files by test examples (test cases) between parallel Buildkite agents

Slow RSpec test files are often related to E2E tests, the browser tests like capybara feature specs. They can run for a few or sometimes even dozens of minutes. They could become a bottleneck if the parallel job has to run a single test file for 10 minutes while other parallel jobs complete a few smaller test files in 5 minutes.

There is a solution for that! You can use Knapsack Pro with RSpec split by examples feature that will automatically detect slow RSpec test files in your project and split them between parallel Buildkite agents by test examples (test cases).

Summary

As you can see a combination of a few elements like Buildkite CI with cloud infrastructure solutions like AWS and an optimal split of test files using Knapsack Pro can improve significantly the work of your team. With Knapsack Pro you can achieve great results and super fast CI builds. Feel free to try it and join other happy Buildkite users.

If you are looking for a Docker config you can also see repository examples at the end of the article: Auto balancing 7 hours tests between 100 parallel jobs on Buildkite CI.

Understanding and using Ruby's powerful #grep method

Sat, 13 Mar 2021 11:00:00 +0000

Ruby is known for its many handy methods. I’d like to take a look at one that is a little bit less known, but quite powerful nonetheless: the #grep method.

About the method

#grep is one of many instance methods defined on the Enumerable module. This essentially means that you can use it on all Ruby classes that include this module, e.g. Array or Hash. The Enumerable module is the place where many other popular “collection” methods are defined - #map, #select, and the like. Whatever Ruby object you can use the #map on, you should be able to use the #grep on as well.

“Getting a #grep”

Let’s see an example usage of the #grep method:

array = ['foo', 'bar', 'baz']

array.grep(/^ba/)
# => ["bar", "baz"]

Seems pretty handy, right? And that’s just one way of using it. But before showing more examples, let’s deconstruct this to understand exactly how it works.

What happens under the hood here could be mimicked with a code like this:

array.select { |element| /regexp/ === element }

Or, more generally:

collection.select { |element| pattern === element }

Notice a few things here:

The return value will always be an array. The length of this array will be between 0 (empty), and the size of the original collection (which is exactly how #select behaves too).
We are using the triple equals (#===) method (“operator”). The #=== is probably worthy of a separate article. But for now, we can just recall that it’s implemented in classes as sort of a “lighter” (meaning: less strict) equality method (also called “case” equality due to its usage in case statements). We can illustrate the difference between regular and “case” equality with an example:

# the two objects are obviously not the same
(1..2) == 2 #=> false
# 2 falls within the range of 1..2
(1..2) === 2 #=> true

We don’t have to limit ourselves to regexes. Any object could be passed as a “pattern” to compare against.
The triple equals method is invoked on the given pattern, not the elements of the collection. (The pattern === element bit is of course just a syntactical sugar for pattern.=== element - stripping out the ‘sugar’ exposes the method invocation). Invoking the method on the pattern makes sense, as it’s this object that should “know” when something is passing its “light” equality requirements. Notice the difference here:

"foobar" === /foo/ # invoking the `String#===` method
#=> false

/foo/ === "foobar" # invoking the `Regexp#===` method
#=> true

When and how to use it

Enough theory. Let’s get down to it and look at some concrete situations where the #grep method might come in very handy to you.

#grep with a Regexp

You can easily match strings with a Regexp pattern.

Just use:

string_collection.grep(/^pattern/)

instead of:

string_collection.select { |str| /^pattern/ === str }

Apart from handling collections of strings in the code, I find the #grep method to be very handy for… recalling things (e.g. by performing a quick lookup in the terminal). These examples should illustrate the idea:

Module.constants.grep(/RUBY/)
#=>[:RUBY_DESCRIPTION, :RUBY_VERSION, :RUBY_RELEASE_DATE, :RUBY_PLATFORM,
#   :RUBY_PATCHLEVEL, :RUBY_REVISION, :RUBY_COPYRIGHT, :RUBY_ENGINE,
#   :RUBY_ENGINE_VERSION, :RUBYGEMS_ACTIVATION_MONITOR]

Array.instance_methods.grep(/gr/)
#=> [:grep, :grep_v, :group_by]

(The Regexp#=== works with symbols just as it does with strings)

#grep with a Range

Range#=== checks the inclusion of Numeric values in the given range. This means that we can pass a range to grep to quickly filter values falling within the range.

numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9]

numbers.grep(3..5)
#=> [3, 4, 5]

#grep with a class name Constant

all_sorts_of_things = [1, 2, 'apple', 3, { foo: 4 }, 'banana', nil]

all_sorts_of_things.grep(Hash)
#=> [{:foo=>4}]

all_sorts_of_things.grep(String)
#=> ["apple", "banana"]

#grep with any object

Since the #=== method in Ruby is implemented on an Object, you can pass any descendant of this class to the #grep method. In the Object class, the #=== method behaves the same way as #==. It’s up to descending class to implement (or not) a different meaning between the two.

all_sorts_of_things = [1, 2, 'apple', 3, { foo: 4 }, 'banana', nil]

all_sorts_of_things.grep('apple')
#=> ['apple']

all_sorts_of_things.grep(3.0)
#=> [3]

The opposite: #grep_v

Just like #select has an opposite #reject method, there is also a #grep_v method that behaves like you would imagine - it rejects the matched objects, similar to how this code would:

string_collection.reject { |str| pattern === str }

Example usage:

all_sorts_of_things = [1, 2, 'apple', 3, { foo: 4 }, 'banana', nil]

all_sorts_of_things.grep_v(Integer)
#=> ["apple", {:foo=>4}, "banana", nil]

Passing a block

Both #grep, and #grep_v are utilizing a block if one is supplied. The block is used to transform the matched values before returning them in an array (just like chaining a #map after #select would). E.g.

IO.constants.grep(/SEEK/) { |cons| IO.const_get(cons) }
#=> [1, 0, 3, 2, 4] # (values of IO::SEEK_CUR, IO::SEEK_SET, IO::SEEK_DATA, IO::SEEK_END, and IO::SEEK_HOLE)

The same operation without a #grep would probably look something like that:

IO.constants.select { |cons| cons =~ /SEEK/ }.
             map { |cons| IO.const_get(cons) }
#=> [1, 0, 3, 2, 4]

Summary

As you can see, the #grep method is pretty powerful! If you haven’t already, add it to your developer’s arsenal and see where it can help you in your work.

Are you using the #grep method in your project? Please share in the comments below!

And if your Ruby project suffers from slow build times, consider using Knapsack Pro to improve the productivity and delivery times of your team!

Run Jest tests on GitHub Actions with optimal parallelization

Fri, 05 Mar 2021 08:00:00 +0000

Jest is a powerful testing framework used in JavaScript projects. Besides vanilla JS, it’s often used for React, NodeJS, Angular or Vue.js projects, among others. I am going to help you configure running your Jest test suite on GitHub Actions. We are going to use CI parallelization with Knapsack Pro, for maximum effectiveness. Let’s begin.

Why parallelization

Splitting the test suites on multiple machines is a smart way to achieve short build times and increase your team’s productivity. A naïve division of work between multiple machines might result in fewer gains than anticipated though, due to possible bottlenecks (please refer to this article to learn more about the reasons behind it). This is where a dynamic test distribution with Knapsack Pro (@knapsack-pro/jest) comes into play. It ensures you utilize your parallel CI nodes (a.k.a. parallel jobs) in the optimal way.

Configuring GitHub Actions: Build Matrix

GitHub Actions is configured through a yaml config file residing in your repository. It should be placed in the: .github/workflows directory, e.g. .github/workflows/main.yml. When properly configured, your GH Actions builds will be visible in the Actions tab in your GitHub repository page.

The Build Matrix is a powerful feature in GitHub Actions, which allows you to easily configure a combination of settings for running multiple jobs. Since we are concerned with simple parallelism for the purpose of this demonstration, our build matrix config is pretty straightforward:

strategy:
  fail-fast: false
  matrix:
    ci_node_total: [4]
    ci_node_index: [0, 1, 2, 3]

For the ci_node_total key, we always provide a single-element list. The value should be equal to the number of parallel nodes you are going to use.

The ci_node_index key should contain a list of indexes of your CI nodes. We are using zero-based numbering, so the list will essentially be a 0..n-1 range, where n is the number of your parallel nodes (used as a value in ci_node_total). For ci_node_total: [2], the ci_node_index would be [0, 1]. For ci_node_total: [3], it would be [0, 1, 2], and so on.

The above config would result in running 4 parallel nodes (jobs) on GitHub Actions. GitHub combines the provided lists into a matrix (hence the name) to determine this. As a result, the 4 parallel nodes would have these settings:

{matrix.ci_node_total: 4, matrix.ci_node_index: 0}
{matrix.ci_node_total: 4, matrix.ci_node_index: 1}
{matrix.ci_node_total: 4, matrix.ci_node_index: 2}
{matrix.ci_node_total: 4, matrix.ci_node_index: 3}

This is exactly what we need to pass to the Knapsack Pro API to run the Queue correctly. Here’s the config step for running your Jest test suite on each node:

- name: Run tests with Knapsack Pro
  env:
    KNAPSACK_PRO_TEST_SUITE_TOKEN_JEST: ${{ secrets.KNAPSACK_PRO_TEST_SUITE_TOKEN_JEST }}
    KNAPSACK_PRO_CI_NODE_TOTAL: ${{ matrix.ci_node_total }}
    KNAPSACK_PRO_CI_NODE_INDEX: ${{ matrix.ci_node_index }}
    KNAPSACK_PRO_FIXED_QUEUE_SPLIT: true
  run: |
    npx @knapsack-pro/jest

Notice how we use the previously defined values of matrix.ci_node_total and matrix.ci_node_index to set the KNAPSACK_PRO_CI_NODE_TOTAL and KNAPSACK_PRO_CI_NODE_INDEX variables, respectively.

Apart from these two, there are two additional variables defined: KNAPSACK_PRO_TEST_SUITE_TOKEN_JEST and KNAPSACK_PRO_FIXED_QUEUE_SPLIT.

KNAPSACK_PRO_TEST_SUITE_TOKEN_JEST is a token needed for Knapsack API authorization. Configure it using secrets (Actions & Dependabot) as described in GitHub Actions’ docs.

To view or generate Knapsack API token for your project, head over to Knapsack User’s Dashboard (you may need to sign up for an account if you don’t have one). Save your API token in your GitHub Secrets as KNAPSACK_PRO_TEST_SUITE_TOKEN_JEST.

The other variable, KNAPSACK_PRO_FIXED_QUEUE_SPLIT needs to be set to true. This will ensure the retry functionality on GitHub Actions works correctly with Knapsack API.

Here’s a full example config for Jest test suite on GitHub Actions:

# .github/workflows/main.yaml
name: Main

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest

    strategy:
      fail-fast: false
      matrix:
        node-version: [8.x]
        # [n] - where the n is a number of parallel jobs you want to run your tests on.
        # Use a higher number if you have slow tests to split them between more parallel jobs.
        # Remember to update the value of the `ci_node_index` below to (0..n-1).
        ci_node_total: [4]
        # Indexes for parallel jobs (starting from zero).
        # E.g. use [0, 1] for 2 parallel jobs, [0, 1, 2] for 3 parallel jobs, etc.
        ci_node_index: [0, 1, 2, 3]

    steps:
      - uses: actions/checkout@v1

      - name: Use Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v1
        with:
          node-version: ${{ matrix.node-version }}

      - name: npm install and build
        run: |
          npm install
          npm run build --if-present

      - name: Run tests with Knapsack Pro
        env:
          KNAPSACK_PRO_TEST_SUITE_TOKEN_JEST: ${{ secrets.KNAPSACK_PRO_TEST_SUITE_TOKEN_JEST }}
          KNAPSACK_PRO_CI_NODE_TOTAL: ${{ matrix.ci_node_total }}
          KNAPSACK_PRO_CI_NODE_INDEX: ${{ matrix.ci_node_index }}
          # necessary for rerunning the same build to work correctly
          KNAPSACK_PRO_FIXED_QUEUE_SPLIT: true
        run: |
          npx @knapsack-pro/jest

Running your builds

When your GitHub Actions is properly set up, it’s going to run your build when you push new changes to the repository. You can now enjoy the optimal parallelization of your Jest tests.

To see how Knapsack is splitting your test suite, visit the User Dashboard. The GitHub Actions builds will also be visible under the Actions tab in your project’s repository on GH.

Anything unclear? Leave a comment below or go ahead and contact us about your problem. We’ll be happy to help!

Cucumber BDD testing using Github Actions parallel jobs to run tests quicker

Wed, 03 Mar 2021 07:00:00 +0000

Cucumber employs Behavior-Driven Development (BDD) for testing your application. This type of test is often time-consuming when running in the browser. You will learn how to run Cucumber tests on Github Actions using parallel jobs to execute the test suite much faster.

Github Actions matrix strategy

You can use the Github Actions matrix strategy to run parallel jobs. You will need to divide your Cucumber test files between the parallel jobs in a way that work will be balanced out between the jobs.

It’s not that simple to do because often Cucumber tests can take a different amount of time. One test file can have many test cases, the other can have only a few but very complex ones, etc.

There are often more steps in your CI pipeline like installing dependencies, loading data from the cache and each step can take a different amount of time per parallel job before even Cucumber tests are started. The steps affect the overall CI build speed.

What you would like to achieve is to run parallel jobs in a way that they always finish the execution of Cucumber tests at a similar time. Thanks to that you will avoid lagging jobs that could be a bottleneck in your CI build.

Dynamically split Cucumber tests using Queue Mode

To get optimal CI build execution time you need to ensure the work between parallel jobs is split in such a way as to avoid bottleneck slow job. To achieve that you can split Cucumber test files in a dynamic way between the parallel jobs using Knapsack Pro Queue Mode and knapsack_pro ruby gem.

Knapsack Pro API will take care of coordinating how tests are divided between parallel jobs. On the API side, there is a Queue with a list of your test files and each parallel job on Github Actions is running Cucumber tests via the knapsack_pro Ruby gem. The knapsack_pro gem asks Queue API for a set of test files to run and after it gets executed then the gem asks for another set of test files until the Queue is consumed. This ensures that all parallel jobs finish running tests at a very similar time so that you can avoid bottleneck jobs.

You can learn more about the dynamic tests suite split in Queue Mode or check the video below.

Github Actions parallel jobs config for Cucumber

Here is the full Github Actions YAML config example for the Cucumber test suite in a Ruby on Rails project using knapsack_pro gem to run Cucumber tests between parallel jobs.

# .github/workflows/main.yml
name: Main

on: [push]

jobs:
  test:
    runs-on: ubuntu-latest

    # If you need DB like PostgreSQL, Redis then define service below.
    # https://github.com/actions/example-services/tree/main/.github/workflows
    services:
      postgres:
        image: postgres:10.8
        env:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: ""
          POSTGRES_DB: postgres
        ports:
          - 5432:5432
        # needed because the postgres container does not provide a healthcheck
        # tmpfs makes DB faster by using RAM
        options: >-
          --mount type=tmpfs,destination=/var/lib/postgresql/data
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

      redis:
        image: redis
        ports:
          - 6379:6379
        options: --entrypoint redis-server

    # https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#jobsjob_idstrategymatrix
    strategy:
      fail-fast: false
      matrix:
        # Set N number of parallel jobs you want to run tests on.
        # Use higher number if you have slow tests to split them on more parallel jobs.
        # Remember to update ci_node_index below to 0..N-1
        ci_node_total: [8]
        # set N-1 indexes for parallel jobs
        # When you run 2 parallel jobs then first job will have index 0, the second job will have index 1 etc
        ci_node_index: [0, 1, 2, 3, 4, 5, 6, 7]

    steps:
      - uses: actions/checkout@v2

      - name: Set up Ruby
        uses: actions/setup-ruby@v1
        with:
          ruby-version: 2.7

      - uses: actions/cache@v2
        with:
          path: vendor/bundle
          key: ${{ runner.os }}-gems-${{ hashFiles('**/Gemfile.lock') }}
          restore-keys: |
            ${{ runner.os }}-gems-

      - name: Bundle install
        env:
          RAILS_ENV: test
        run: |
          bundle config path vendor/bundle
          bundle install --jobs 4 --retry 3

      - name: Create DB
        env:
          # use localhost for the host here because we have specified a container for the job.
          # If we were running the job on the VM this would be postgres
          PGHOST: localhost
          PGUSER: postgres
          RAILS_ENV: test
        run: |
          bin/rails db:prepare

      - name: Run tests
        env:
          PGHOST: localhost
          PGUSER: postgres
          RAILS_ENV: test
          KNAPSACK_PRO_TEST_SUITE_TOKEN_CUCUMBER: ${{ secrets.KNAPSACK_PRO_TEST_SUITE_TOKEN_CUCUMBER }}
          KNAPSACK_PRO_CI_NODE_TOTAL: ${{ matrix.ci_node_total }}
          KNAPSACK_PRO_CI_NODE_INDEX: ${{ matrix.ci_node_index }}
          KNAPSACK_PRO_FIXED_QUEUE_SPLIT: true
          KNAPSACK_PRO_LOG_LEVEL: info
        run: |
          bundle exec rake knapsack_pro:queue:cucumber

Here is the view from Github Actions showing that we run 8 parallel jobs for the CI build.

Summary

I hope you find this example useful. If you would like to learn more about Knapsack Pro please check our homepage and see a list of supported test runners for parallel testing in Ruby, JavaScript, etc.

Faster pipelines with Knapsack Pro by parallelizing Cypress tests on GitLab CI

Tue, 02 Mar 2021 15:20:00 +0000

When the runtime of our build pipeline had reached 20 minutes on average for each commit on each Merge Request, we knew we needed a solution to speed things up. We wanted to optimize everything we could within the pipeline, a way of doing this was to parallelize our tests with Knapsack Pro to speed up our Cypress integration tests which would save us a sizable amount of time.

What is our setup?

Our team is working on Kiwi.com, which is a travel company selling tickets for flights, trains, buses and any other kinds of transportation. More specifically, the team is responsible for the Help Center, where users can go to find articles and chat with support before, during and after their trips.

Our application is deployed in two distinct ways: as an npm package integrated into other modules within Kiwi.com as a sidebar (called sidebar), and as a standalone web application (called full-page) which is dockerized and then deployed on GKE (Google Kubernetes Engine) for our production environment and using GCR (Google Cloud Run) for staging envs .

Most of the code is shared between these units but there are some minor differences in design and major differences in flows/functionality, so they need to be tested separately.

We use GitLab for code reviews and version control and GitLab CI to run our build pipelines.

We use Cypress for end-to-end and integration testing with separate tests for both types of deployment.

What problem are we trying to solve?

We want our pipelines and tests to run faster but not spend too much time nor money on doing it, so running each test on its own server is probably not the best way to go, not to mention the overhead of starting up the runners and checking out the source code.

If our pipelines are slow, developers could forget that they had something running in the background and jump between tasks or worse, somebody else merges code into the main branch and the pipeline has to be run again…

If our pipelines are fast, then we have to wait less while they are running, after applying some of the changes suggested during code reviews and to get hotfixes out to production if necessary.

Manual splitting of Cypress tests

After seeing that running tests in series takes extremely long, we tried to split some areas up and run them on separate Cypress runners and this worked for a while. But, as the tests’ size grew asymmetrically, we had to shift around which suites should run together continuously.

List of Cypress test files, showing that tests are nested two levels: by module and by feature or area

List of Cypress test Gitlab CI jobs, showing that there are many separate jobs for each of the test subfolders

List of Cypress test Gitlab CI jobs, showing that job runtimes vary from 53 seconds to 5 minutes

Pipeline summary

What is Knapsack Pro?

Knapsack Pro is a tool that helps split up test suites in a dynamic way across parallel CI runners to ensure that each runner finishes work simultaneously.

Its name comes from the “knapsack problem”, a problem in combinatorial optimization, to figure out how to take things (test suites) with certain attributes (number of tests, time it took on average to run, etc) and determine how best to divide them to get the most value (all CI nodes are running tests from boot until shutdown and they all finish at the same time).

It comes with a dashboard to see the distribution of test subsets for each parallel run of CI runners, which is useful when splitting test suites or to see if something is going wrong with the runs. The dashboard also suggests if you’d benefit from less or more parallel runners as well.

The founder of Knapsack Pro, Artur Trzop has been helping us proactively, sometimes noticing and notifying us that something went wrong with our pipeline even before we did. Whenever we had an issue, we could just write to him and we would figure out a solution together.

Plugging in Knapsack Pro

To try out Knapsack Pro, all we did was follow the docs and afterward merge all separate Cypress jobs into one and set up some reporterOptions in Cypress to ensure that we’ll collect all of the test results as JUnit reports from all the parallel runs.

Here’s a short example of our full-page test jobs before the conversion:

.cypress_fp: &cypress_fp
  stage: cypress
  image: "cypress/browsers:node14.15.0-chrome86-ff82"
  retry: 2
  before_script:
    - yarn install--frozen-lockfile --production=false
  artifacts: # save screenshots as an artifact
    name: fp-screenshots
    expire_in: 3 days
    when: on_failure
    paths:
      - cypress/screenshots/

fp contact:
  <<: *cypress_fp
  script:
    - yarn cypress:run-ci -s "./cypress/tests/full-page/contact/*.js"

fp login:
  <<: *cypress_fp
  script:
    - yarn cypress:run-ci-s "./cypress/tests/full-page/login/*.js"

fp search:
  <<: *cypress_fp
  script:
    - yarn cypress:run-ci -s "./cypress/tests/full-page/search/*.js"

# etc ...

Here is the new full-page test job after the conversion:

.cypress_parallel_defaults: &cypress_parallel_defaults

full-page:
  stage: cypress
  image: "cypress/browsers:node14.15.0-chrome86-ff82"
  retry: 0
  parallel: 4
  before_script:
    - yarn install--frozen-lockfile --production=false
  script:
    - yarn @knapsack-pro/cypress --record false --reporter junit --reporter-options "mochaFile=cypress/results/junit-[hash].xml"
  after_script: # collect all artifacts
    - mkdir -p cypress/screenshots/full-page && cp -r cypress/screenshots/full-page $CI_PROJECT_DIR
    - mkdir -p cypress/videos/full-page && cp -r cypress/videos/full-page $CI_PROJECT_DIR
    - mkdir -p cypress/results && cp -r cypress/results $CI_PROJECT_DIR
    # all the copying above is done to make it easy to browse the artifacts in the UI (single folder)
  artifacts: # save videos and screenshots as artifacts
    name: "$CI_JOB_NAME_$CI_NODE_INDEX-of-$CI_NODE_TOTAL"
    when: on_failure
    expire_in: 3 days
    paths:
      - full-page/
    reports:
      junit: results/junit-*.xml
    expose_as: cypress_full-page
  variables:
    CYPRESS_BASE_URL: https://$CI_ENVIRONMENT_SLUG.$CLOUDRUN_CUSTOM_DOMAIN_SUFFIX/en/help
    KNAPSACK_PRO_TEST_SUITE_TOKEN_CYPRESS: $KNAPSACK_PRO_TEST_SUITE_TOKEN_CYPRESS_FULL_PAGE
    KNAPSACK_PRO_CI_NODE_BUILD_ID: full-page-$CI_COMMIT_REF_SLUG-$CI_PIPELINE_ID
    KNAPSACK_PRO_TEST_FILE_PATTERN: cypress/full-page/*.js

This was the result we saw on the Knapsack Pro Dashboard, showing that our current separation of suites needs to be worked on to gain all the benefits of parallelization:

Knapsack Pro dashboard showing the slowest test files, slowest being almost four minutes while the second slowest is only 30 seconds

After some tweaking, splitting, and moving suites around, this is what we finally had:

List of Cypress test files, showing that tests are a single level: by module

List of Cypress test Gitlab CI jobs, showing that there are only two separate jobs parallelized four times for each of the modules

List of Cypress test Gitlab CI jobs, showing that job runtimes are almost equal for each batch of parallel jobs (3:18 - 3:24 and 2:07 - 2:17)

Pipeline summary

It is still not perfectly balanced but with this granularity, we can utilize four runners at the same time and there isn’t too much idle time, so we’re not wasting CPU cycles.

Knapsack Pro dashboard showing the slowest test files, slowest being 21.803 seconds while the second slowest being 21.001 seconds (some other tests are only 1-6 seconds short)

What else can we do?

15 minutes still feels long, pipeline runtimes around 10 minutes are a good goal to reach but we have more optimization to do around build times and Docker compilation times, which are out of scope for this article. If you’re interested in reading more about how to do that, I can recommend the article Docker for JavaScript Devs: How to Containerize Node.js Apps Efficiently for more details.

Results

The switch to Knapsack Pro took less than a day. Most of the work afterward was to split up tasks and optimize other parts of GitLab CI. The whole process took two weeks for a single developer.

By splitting up the test suites into smaller, equal sized chunks, running tests parallelized on four CI runners (for each of our target builds) and some other general pipeline fixes (better caching between jobs/steps, only running parts of the pipeline for actual changes), we’ve managed to decrease our pipeline runtime of ~20 minutes to ~15 minutes.

When we add more test suites, Knapsack Pro will take care of sorting the tests to the proper CI runners to ensure that pipeline runtimes don’t increase if necessary.

We sped up our pipeline by 25%, or 5 minutes on average, and seeing that each of our three developers starts about 6-10 pipelines a day (depending on the workload and type of work), we’ve essentially saved between 90-150 minutes of development time a day, as well as increased morale because nobody likes to wait.

Setting up Knapsack Pro in your Ruby RSpec project

Thu, 25 Feb 2021 08:00:00 +0000

Knapsack Pro offers two modes for RSpec tests in your Ruby application: the Regular Mode and the Queue Mode. They differ in their approach towards dividing your test files between parallel CI nodes.

Regular Mode vs Queue Mode

The Regular Mode uses timing data for every test file and splits all tests based on that. For example, if you use 6 parallel CI nodes, it will split your tests into 6 sets, roughly equal by their combined timings (based on previously collected timing data). These sets will then be distributed across parallel machines and run by them. We call this a static distribution of tests.

With the Queue Mode, there is one central queue comprising all of the test files. Each parallel node receives a small subset of tests from the queue and runs them. The pattern repeats until the whole queue is empty. (BTW, The Queue Mode also uses the timing data - but this is used only to order the tests in the queue in a way that ensures the best overall performance.) We don’t know which tests will end up distributed to which node beforehand - that’s why this distribution is dynamic.

The way the Queue Mode works helps mitigate unstable test run times and parallel node performance. When the metrics can vary, the queue mode is the answer.

The Regular vs Queue Mode distinction is covered more in-depth in a separate post.

After helping many different projects over the years, we can say, that the Queue Mode is usually the best solution. In the real world, the performance of individual machines and tests is subject to all sorts of unpredictability. Using Queue Mode results in the most robust optimization despite these constraints.

Always start your setup with Regular Mode

Since we are claiming that the Queue Mode is the ultimate best for most projects, you might be tempted to jump right in and give it a try. This is not what we advise, however. Let me explain why.

Non-trivial setup

We are doing our best to make the Knapsack setup experience as smooth as possible for every project giving our solution a try. Some things just lie outside of our control, though.

The combination of your specific CI pipeline configuration and your RSpec usage might require some adjustments along the way.

What we know from experience is that usually the Queue Mode uncovers different adjustment needs than the Regular Mode. The reason behind that is that in the Queue Mode, each one of your parallel CI nodes will usually run the RSpec command multiple times (as compared to just one run per node in the Regular Mode). This distinction might result in additional challenges to solve.

As with all debugging, it’s best to narrow down (isolate) possible issues we are tackling at a given time. This is why we strongly suggest to set up the Regular Mode first. When you ensure it works as expected, and confirm that in your User Dashboard, you are ready to proceed to the Queue Mode setup.

And now for the good news: we have documented all of the common issues you might encounter during your setup in our FAQ. Keep that in mind during your onboarding.

Head over to our installation guide and get the best out of Knapsack for your project today! We would very happy to assist you along the way, so do not hesitate to contact us if you need anything.

Conclusion

Following the described order of steps (using Regular Mode before Queue Mode) should result in the smoothest Knapsack Pro setup experience. In case you are stuck at any point, please consult our installation guide, FAQ, or simply contact us. We’ll be more than happy to help!

Best Heroku add-ons for Ruby on Rails project

Thu, 25 Feb 2021 07:00:00 +0000

After working for over 8 years with Heroku and Ruby on Rails projects I have my own favorite set of Heroku add-ons that work great with Rails apps. You are about to learn about the add-ons that come in handy in your daily Ruby developer life.

Heroku add-ons

Here it is, a list of my favorite Heroku add-ons from Heroku Marketplace and why I choose them for my Ruby on Rails projects hosted on Heroku.

Heroku Scheduler

Heroku Scheduler can run scheduled tasks every 10 minutes, every hour, or every day. I use it to run my scheduled rake tasks. For instance every day I run a rake task that will send a summary of users who signed up in the last 24 hours to my mailbox.

Heroku Scheduler add-on is free. The only limitation is that it has fewer options than the cron in the Unix system. If you need to run a rake task every Monday then you need to set up a rake task as a daily task in Heroku Scheduler and do a check of the day in the rake task itself to skip it when needed.

# lib/tasks/schedule/notify_users_about_past_due_subscription.rake
namespace :schedule do
  desc 'Send notification about past due subscriptions to users'
  task notify_users_about_past_due_subscription: :environment do
    if Time.current.monday?
      Billing::NotifyUsersAboutPastDueSubscriptionWorker.perform_async
    else
      Rails.logger.info("Skip schedule:notify_users_about_past_due_subscription task.")
    end
  end
end

New Relic APM

New Relic add-on does application performance monitoring. It’s one of my favorite add-ons. It allows to track each process like puma/unicorn/sidekiq per dyno and its performance. You can see which Rails controller actions take the most time. You can see your API endpoints with the highest throughput and those which are time-consuming. New Relic helped me many times to debug bottlenecks in my app and thanks to that I was able to make Knapsack Pro API with an average 50ms response time. Who said the Rails app has to be slow? :)

Rollbar

Rollbar allows for exception tracking in your Ruby code and also in JS code on the front end side. It has a generous free plan with a 5000 exception limit per month.

You can easily ignore some common Rails exceptions to stay within the free plan limit.

# config/initializers/rollbar.rb
Rollbar.configure do |config|
  config.access_token = ENV['ROLLBAR_ACCESS_TOKEN']

  if Rails.env.test? || Rails.env.development?
    config.enabled = false
  end

  # Add exception class names to the exception_level_filters hash to
  # change the level that exception is reported at. Note that if an exception
  # has already been reported and logged the level will need to be changed
  # via the rollbar interface.
  # Valid levels: 'critical', 'error', 'warning', 'info', 'debug', 'ignore'
  # 'ignore' will cause the exception to not be reported at all.
  config.exception_level_filters.merge!('ActionController::RoutingError' => 'ignore')
  config.exception_level_filters.merge!('ActionController::InvalidAuthenticityToken' => 'ignore')
  config.exception_level_filters.merge!('ActionController::BadRequest' => 'ignore')
  config.exception_level_filters.merge!('ActiveRecord::RecordNotFound' => 'ignore')
  config.exception_level_filters.merge!('Rack::Timeout::RequestTimeoutException' => 'ignore')
  config.exception_level_filters.merge!('Rack::QueryParser::InvalidParameterError' => 'ignore')
  config.exception_level_filters.merge!('ActionDispatch::Http::MimeNegotiation::InvalidType' => 'ignore')

  config.environment = ENV['ROLLBAR_ENV'].presence || Rails.env
end

Redis Cloud

If you use Redis in your Ruby on Rails app then Redis Cloud is your add-on. It has a free plan and paid plans are more affordable than other add-ons have.

Redis Cloud add-on does automatic backups of your data and offers a nice web UI to preview the live Redis usage and historical usage of your database instance.

I like to use Redis Cloud + sidekiq gem in my Rails apps. Also, Redis is useful if you need to cache some data quickly in the memory and expire it after some time.

redis_connection = Redis.new(
  # use REDISCLOUD_URL when app is running on Heroku,
  # or fallback to local Redis (useful for development)
  url: ENV.fetch('REDISCLOUD_URL', 'redis://localhost:6379/0'),
  # tune network timeouts to be a little more lenient when you are seeing occasional timeout
  # errors for Heroku Redis Cloud addon
  # https://github.com/sidekiq/sidekiq/wiki/Using-Redis#life-in-the-cloud
  timeout: 5
)

redis_connection.setex('my-key-name', 1.hour, 'this value will expire in 1 hour')

Twilio SendGrid

SendGrid is a free add-on that allows you to start sending emails from your Ruby on Rails. You can even connect your domain to it so your users get emails from your domain.

There are free 12,000 emails per month in the free plan.

Heroku Add-on to save you time & money

Here are a few of my favorite add-ons that will help you save money and time in your project.

AutoIdle

AutoIdle lets you save money by automatically putting your staging and review apps to sleep on Heroku. I use it to turn off my web and worker dyno for the staging app when there is no traffic to the app. No more paying for Heroku resources during the night and weekends. ;)

Rails Autoscale

Rails Autoscale is a powerful add-on that will help you save money on Heroku. It will measure requests queue time and based on that add or remove dynos for your web processes. If you have higher traffic during the day it will add more dynos. During the night when the traffic is low, it will remove dynos.

Rails Autoscale can also track your worker queue. For instance, if you have a lot of jobs scheduled in Sidekiq then Rails Autoscale will add more worker dynos to process your job queue faster. It can even shut down worker dyno when there are no jobs to be processed which can save you even more money.

Knapsack Pro

Knapsack Pro is a Heroku add-on and ruby gem that can run your Rails tests in RSpec, Cucumber, Minitest, etc, and automatically split the tests between parallel machines on any CI server. It works with Heroku CI, CircleCI, Buildkite, Travis CI, etc. It will help you save time by doing a dynamic split of tests with Queue Mode to ensure all parallel jobs finish work at a similar time. This way you optimize your CI build runs and save the most time.

Below you can see an example of the optimal distribution of tests, where each parallel CI machine performs tests for 10 minutes, thanks to which the entire CI build lasts only 10 minutes instead of 40 if you would run tests on a single CI server only.

I’ve been working on the Knapsack Pro add-on and I’d love to hear your feedback if you give it a try.

RSpec testing parallel jobs with CircleCI and JUnit XML report

Tue, 23 Feb 2021 07:00:00 +0000

You will learn how to run RSpec tests for your Ruby on Rails project on CircleCI with parallel jobs to shorten the running time of your CI build. Moreover, you will learn how to configure JUnit formatter to generate an XML report for your tests to show failing RSpec test examples nicely in CircleCI web UI. Finally, you will see how to automatically detect slow spec files and divide their test examples between parallel jobs to eliminate the bottleneck job that’s taking too much time to run tests.

Ruby gems to configure your RoR project

Here are the key elements you need:

rspec_junit_formatter - it’s a ruby gem that generates an XML report for executed tests with information about test failures. This report can be automatically read by CircleCI to present it in CircleCI web UI. No more browsing through long RSpec output - just look at highlighted failing specs in the TESTS tab :)

knapsack_pro - it’s a Ruby gem for running tests on parallel CI jobs to ensure all jobs finish work at a similar time to save you as much time as possible and eliminate bottlenecks.
- It uses the Queue Mode to dynamically split test files between parallel jobs.
- Knapsack Pro can also detect your slow RSpec test files and divide them between parallel jobs by test examples. You don’t have to manually split your big spec file into smaller files if you want to split work between parallel container on CircleCI :)

Just add the above gems to your Gemfile.

group :test do
  gem 'rspec'
  gem 'rspec_junit_formatter'
end

group :test, :development do
  gem 'knapsack_pro'
end

For Knapsack Pro you will need an API token and you need to follow the installation guide to configure your project.

If you use knapsack_pro gem in Queue Mode with CircleCI you may want to collect metadata like JUnit XML report about your RSpec test suite.

It’s important to copy the XML report to /tmp/test-results. Learn more about collecting metadata on CircleCI in the docs.

CircleCI YML configuration for RSpec

Here is the complete CircleCI YML config file for RSpec, Knapsack Pro and JUnit formatter.

# Ruby CircleCI 2.0 configuration file
#
# Check https://circleci.com/docs/2.0/language-ruby/ for more details
#
version: 2
jobs:
  build:
    parallelism: 10
    # https://circleci.com/docs/2.0/configuration-reference/#resource_class
    resource_class: small
    docker:
      # specify the version you desire here
      - image: circleci/ruby:2.7.1-node-browsers
        environment:
          PGHOST: 127.0.0.1
          PGUSER: my_db_user
          RAILS_ENV: test
          # Split slow RSpec test files by test examples
          # https://knapsackpro.com/faq/question/how-to-split-slow-rspec-test-files-by-test-examples-by-individual-it
          KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES: true

      # Specify service dependencies here if necessary
      # CircleCI maintains a library of pre-built images
      # documented at https://circleci.com/docs/2.0/circleci-images/
      - image: circleci/postgres:10.6-alpine-ram
        environment:
          POSTGRES_DB: my_db_name
          POSTGRES_PASSWORD: ""
          POSTGRES_USER: my_db_user
          # Rails verifies Time Zone in DB is the same as time zone of the Rails app
          TZ: "Europe/Warsaw"

      - image: redis:6.0.7

    working_directory: ~/repo
    environment:
      TZ: "Europe/Warsaw"

    steps:
      - checkout

      # Download and cache dependencies
      - restore_cache:
          keys:
          - v2-dependencies-bundler-{{ checksum "Gemfile.lock" }}-{{ checksum ".ruby-version" }}
          # fallback to using the latest cache if no exact match is found
          - v2-dependencies-bundler-

      - run:
          name: install ruby dependencies
          command: |
            bundle install --jobs=4 --retry=3 --path vendor/bundle

      - save_cache:
          paths:
            - ./vendor/bundle
          key: v2-dependencies-bundler-{{ checksum "Gemfile.lock" }}-{{ checksum ".ruby-version" }}

      # Database setup
      - run: bin/rails db:prepare

      - run:
          name: run tests
          command: |
            mkdir -p /tmp/test-results
            bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format RspecJunitFormatter --out /tmp/test-results/rspec.xml]"

      # collect reports
      - store_test_results:
          path: /tmp/test-results
      - store_artifacts:
          path: /tmp/test-results
          destination: test-results

Summary

You’ve just learned how to make your CircleCI builds way faster! Now your RSpec tests can be automatically run on many parallel machines to save you time. Please let us know if it was helpful or if you have any questions. Feel free to sign up at Knapsack Pro or down below and try it yourself.

How to build Knapsack Pro API client from scratch in any programming language

Thu, 18 Feb 2021 07:00:00 +0000

You will learn how to integrate with Knapsack Pro API to run parallel tests in any programming language for any testing framework. You will see what’s needed to build from scratch Knapsack Pro API client similar to our existing clients like knapsack_pro Ruby gem, @knapsack-pro/jest for Jest in JavaScript, or @knapsack-pro/cypress for Cypress test runner (also in JavaScript).

Here you can find the list of existing Knapsack Pro clients to run your tests in parallel for programming languages like Ruby, JavaScript, and their testing frameworks.

Introduction - learn basics

First, you need to understand what Knapsack Pro does and how it splits test files in parallel CI nodes to run your CI build fast.

Learn what Regular Mode and Queue Mode are in Knapsack Pro and how they work.

Please see below the dictionary of terms we will use in this article:

Knapsack Pro API - it’s an API responsible for deciding how to split test files between parallel CI nodes. Your API client is going to send recorded time execution of your test files to the API to see results in the Knapsack Pro user dashboard. Knapsack Pro API will use the data to better predict how to split your test files in future CI build runs. Here is the documentation for all API endpoints.
Knapsack Pro client - is a library that you install in your project. It contains business logic responsible for connecting with Knapsack Pro API. The client knows how to read environment variables for various CI providers to automatically detect git commit hash, branch name, number of total parallel CI nodes, and CI node index. Knapsack Pro client connects with the Knapsack Pro API to fetch a list of test files to run a proper set of tests on a given parallel CI node. Knapsack Pro client also knows how to integrate with a test runner in a given programming language. For instance, the Knapsack Pro client in Ruby programming language is a knapsack_pro ruby gem. It knows how to run tests for test runners like RSpec, Cucumber, Minitest, etc. Simply speaking, Knapsack Pro client is a wrapper around test runner (testing framework) in a given programing language. Here is a list of existing Knapsack Pro clients.
Test runner (testing framework) - each programming language has its own testing framework. For instance, in Ruby programming language there are test runners like RSpec, Cucumber, Minitest. In JavaScript, you can find Jest, Puppeteer, Karma, Jasmine, Cypress, TestCafe, etc. In Python, there are pytest, unittest.
Knapsack Pro Regular Mode - it’s a static split of tests between parallel CI nodes (performed deterministically). Basically, before starting tests we know up front what set of test files should be run on each parallel CI node.
Knapsack Pro Queue Mode - it’s a dynamic way of splitting tests between parallel CI nodes. In this case, each parallel CI node asks Knapsack Pro API for a set of tests and runs it. Once completed, it asks for another set of tests. It’s repeated until all tests are executed and the Knapsack Pro API has no more test files in the Queue. Please read the article about the difference between Regular Mode and Queue Mode to learn about it in detail and see some graphs showing the difference.

Now you know a few useful terms. Before we start learning how to build a Knapsack Pro client from scratch in your favorite programming language, let’s check how such a client looks like in JavaScript. In order to integrate with Knapsack Pro API in JavaScript, we created an NPM package called @knapsack-pro/core. This package knows how to communicate with Knapsack Pro API and how to read environment variables for various CI providers.

As you probably know, there are many testing frameworks written in JavaScript, e.g. Jest, Cypress, etc. For the Jest testing framework we created another package @knapsack-pro/jest for the Jest testing framework that uses @knapsack-pro/core. The @knapsack-pro/jest NPM package contains business logic responsible for integration with the Jest library so you can run Jest tests in parallel using Knapsack Pro API.

Before you start building your own Knapsack Pro client in your programming language I highly recommend reading the article where we covered how @knapsack-pro/core and @knapsack-pro/jest work. Those are lightweight NPM packages and the source code is easy to understand. You can get some ideas about code organization and technical requirements for building a Knapsack Pro client from scratch.

How to build Knapsack Pro client

You will see how to build a Knapsack Pro client from scratch based on the JavaScript example. Knapsack Pro client is built as 2 packages:

Knapsack Pro Core - (i.e. @knapsack-pro/core) which is responsible for:
- connecting with the Knapsack Pro API. It can respond to, and handle common response types and errors coming from the API.
- reading environment variables specific to Knapsack Pro client like API token, log level, API endpoint URL, etc.
- CI providers environment variables integration - Knapsack Pro Core library can read environment variables for popular CI providers. Thanks to that it can automatically detect git commit hash, branch name, number of parallel CI nodes, etc.
- Logger - it can log useful tips for the output or warnings.
- Fallback Mode - it knows how to run tests in parallel when there is a network issue and the connection with Knapsack Pro API is not working.
Knapsack Pro Test Runner - (i.e. @knapsack-pro/jest) is responsible for:
- integration of Knapsack Pro Core with your test runner (testing framework) like Jest, etc.
- knowing how to run tests for a given test runner, how to record time execution, and report it back to Knapsack Pro Core so the recorded test files can be saved on the Knapsack Pro API side.
- reading environment variables specific for the test runner, for instance how to detect a list of Jest test files residing on the disk.

Knapsack Pro Core

The core functionality of the Knapsack Pro client is in the Core package (@knapsack-pro/core). We will review a few main elements and describe how they work.

Environment variables integration

Knapsack Pro Core client should understand a few environment variables. See example for @knapsack-pro/core environment variables. Users can define those environment variables in their CI server settings to control the behavior of the Knapsack Pro client.

KNAPSACK_PRO_LOG_LEVEL - it determines how much debugging info should be produced by the Knapsack Pro client to the output during the runtime of tests. The default is info. If you set the debug value then the Knapsack Pro client should show in the output a payload of requests and responses from the Knapsack Pro API.
KNAPSACK_PRO_ENDPOINT - it’s the URL of the Knapsack Pro API. The default value is https://api.knapsackpro.com which is production API. You can use our production API and your API token from the user dashboard for testing purposes.
KNAPSACK_PRO_TEST_SUITE_TOKEN - it’s an API token that you can use to connect with Knapsack Pro API. If the value is not defined then an error should be raised.
KNAPSACK_PRO_FIXED_QUEUE_SPLIT - it’s a flag to control the behavior of Queue Mode. The default value is false.
- If the value is true then the API will cache the way test files were split between parallel CI nodes. So when you retry the CI build the tests won’t be dynamically split. Instead, they will be split in the same order as during the very first run (which was a dynamic tests split).
- Do you want to use “retry single failed parallel CI node” feature for your CI? For instance, some of CI providers like Travis CI, Buildkite or Codeship allow you to retry only one of failed parallel CI nodes instead of retrying the whole CI build with all parallel CI nodes. If you want to be able to retry only a single failed parallel CI node then you need to tell Knapsack Pro API to remember the way test files were allocated across parallel CI nodes by adding to your CI environment variables KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true.
- The default is KNAPSACK_PRO_FIXED_QUEUE_SPLIT=false which means that when you want to retry the whole failed CI build then a new dynamic test suite split will happen across all retried parallel CI nodes. Some people may prefer to retry the whole failed CI build with test files allocated across parallel CI nodes in the same order as it happened for the failed CI build - in such a case you should set KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true.
- To learn more about this flag you can also see examples in knapsack_pro ruby gem related to the KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true.
KNAPSACK_PRO_CI_NODE_TOTAL - the default value conveying the number of parallel CI nodes used.
- If KNAPSACK_PRO_CI_NODE_TOTAL has a value then it should be used.
- If KNAPSACK_PRO_CI_NODE_TOTAL has no value then Knapsack Pro client should read CI provider environment variables to determine CI node total number.
- If no value is detected then an error should be raised. Please see the source code of @knapsack-pro/core.
KNAPSACK_PRO_CI_NODE_INDEX - it is the index of the parallel CI node (parallel job). It should start from 0 to KNAPSACK_PRO_CI_NODE_TOTAL - 1. If you use 2 parallel CI nodes in total then indexes should be 0 and 1.
- If KNAPSACK_PRO_CI_NODE_INDEX has a value then it should be used.
- If KNAPSACK_PRO_CI_NODE_INDEX has no value then Knapsack Pro client should read CI provider environment variables to determine CI node index.
- If no value is detected then an error should be raised. Please see the source code of @knapsack-pro/core.
KNAPSACK_PRO_CI_NODE_BUILD_ID - a user of your Knapsack Pro client can define CI build ID with this environment variable. For instance, if the user uses Jenkins as a CI provider then Jenkins has no autogenerated CI build ID out of the box. In such a case, the user should create a custom value (unique for every CI build) and assign it to the KNAPSACK_PRO_CI_NODE_BUILD_ID environment variable.
- CI build has many parallel CI nodes. Each parallel CI node should have the same KNAPSACK_PRO_CI_NODE_BUILD_ID value. This means the parallel CI nodes belong to the same CI build.
- Knapsack Pro client by default should try to detect CI build ID for popular CI providers by looking for it in the environment variables.
- If KNAPSACK_PRO_CI_NODE_BUILD_ID value was defined by the user then it should be used during a request to the Knapsack Pro API. It has higher priority than detected CI build ID from a CI provider environment variables.
- If the user did not define KNAPSACK_PRO_CI_NODE_BUILD_ID then a default value missing-build-id should be used.
  - Knapsack Pro API understands missing-build-id string and knows the CI build has an undefined CI build ID then. In such a case only one parallel CI build can be run at a time for a given set of values (git commit hash AND branch name AND number of parallel CI nodes) - otherwise, tests could be accidentally split between 2 CI builds.
    - Why this set of values matter? From the Knapsack Pro API perspective, a unique CI build is a set of test files that belongs to a git commit hash, branch name and it is split across a certain number of parallel CI nodes. When the user will run a few CI builds at the same time for the same git commit, branch name and on the same number of parallel CI nodes then we need a way to distinguish CI builds from each other. That’s why CI build ID is useful and recommended to be pass in request to Knapsack Pro API.
- It might be easier to understand this logic, just check the source code of @knapsack-pro/core.
KNAPSACK_PRO_COMMIT_HASH - it’s a commit hash.
- If KNAPSACK_PRO_COMMIT_HASH has a value then it should be used.
- If KNAPSACK_PRO_COMMIT_HASH has no value then Knapsack Pro client should read CI provider environment variables to determine git commit hash.
- If no value is detected then a git rev-parse HEAD command should be run to determine the commit hash.
- If git is not installed then raise an error. Please see the source code of @knapsack-pro/core.
KNAPSACK_PRO_BRANCH - it’s a branch name.
- If KNAPSACK_PRO_BRANCH has a value then it should be used.
- If KNAPSACK_PRO_BRANCH has no value then Knapsack Pro client should read CI provider environment variables to determine a branch name.
- If no value is detected then a git rev-parse --abbrev-ref HEAD command should be run to determine the branch name.
- If git is not installed then raise an error. Please see the source code of @knapsack-pro/core.
KNAPSACK_PRO_USER_SEAT - A user name that started the CI build. It is usually the same person that made the git commit. It must be masked with a regexp pattern /(?<=\w{2})[a-zA-Z]/g. For example, John Doe <john.doe@example.com> should be masked to Jo** Do* <jo**.do*@ex*****.co*>.
CI providers environment variables integration - Knapsack Pro client should try to read environment variables for popular CI providers. Thanks to that user have to do less work to set up the Knapsack Pro client with his project.
- Here can be found a list of supported CI providers.
- Here is a list of environment variables for each CI provider.

Fallback Mode

Knapsack Pro Core should have implemented business logic for running tests in Fallback Mode. When Knapsack Pro API is not reachable because of downtime then tests should be run in Fallback Mode without the need to use the API.

How Fallback Mode works? The service responsible for Fallback Mode should take a list of test files and the number of total parallel CI nodes. You can sort the test files and divide them by the total parallel CI nodes number.

It’s also possible that during tests runtime in Queue Mode the connection with Knapsack Pro API will be lost. This could mean that some of the test files were already executed based on the set of test files fetched from Queue API and then the connection was lost. In such a case Fallback Mode should exclude test files that were already executed. In Queue Mode the Fallback Mode guarantees each of the test files is run at least once across parallel CI nodes to make sure we never skip a test file.

Here you can see the source code of Fallback Test Distributor.

Logger

Knapsack Pro Core should have a logger with a default info log level. A user should be able to control log level with the environment variable KNAPSACK_PRO_LOG_LEVEL. You use the logger to produce useful tips to the output during tests runtime:

info when Fallback Mode was started
when log level is debug then show request payload
when log level is debug then show response body

Here is an example service for the logger.

Knapsack Pro API integration

Knapsack Pro Core should have implemented business logic for making requests to Knapsack Pro API. There are a few basic elements you need to cover:

Send headers with the client name and client version in each request to the Knapsack Pro API. You should add KNAPSACK-PRO-CLIENT-NAME and KNAPSACK-PRO-CLIENT-VERSION headers in each request. Note that the Knapsack Pro Core (@knapsack-pro/core) is just a core library, so it means the actual client name and version should be defined in the Knapsack Pro Test Runner client (@knapsack-pro/jest) and provided as an argument to the Knapsack Pro Core. Please use semantic versioning.
Send the CI provider name as a header in each request to the Knapsack Pro API with the name KNAPSACK-PRO-CI-PROVIDER.
When a request to the Knapsack Pro API fails then it should be repeated 3 times.
- There are exceptions when a response status indicates a failure - in these cases the request should never be repeated:
  - When response status is 400 then it means request attributes error.
  - When response status is 422 then it means validation error.
  - When the response status is 403 then a free trial period ended.
- For all above 4xx response statuses you should show the error body response to the output and stop running tests. Ensure that the process has exit code 1 - thanks to that CI provider will know the CI build failed.
When the Knapsack Pro API returns different response status than listed above. For instance when you get 500 status then you should repeat the request 3 times. If the 3rd response has a non-2xx status as well, then you should run test files in Fallback Mode.
Ensure you set max request timeout to 15 seconds. When Knapsack Pro API won’t send a response within 15 seconds then it’s better to cancel the request and wait some time before repeating the request. You can wait 8 seconds, and increase by another 8 seconds each consequent request that must be repeated (e.g. wait for 8s, then 16s, then 24s).

Knapsack Pro API - Queue Mode

Knapsack Pro Core should contain logic for making requests to Knapsack Pro API for Queue Mode. Here is described the Queue Mode API endpoint.

Please read the API documentation. Especially an example of the request body. There are 3 types of requests to ensure we can connect with the Queue on the API side in a fast way by sending a request payload as small as possible.

I’ll describe below an example covering all 3 types of request payloads. There are different types of payloads because the Knapsack Pro client runs at the same time on parallel CI nodes and we don’t know which one will be connected with the Knapsack Pro API first. We need to deal with the parallel request problem. For instance, the very first request to Knapsack Pro Queue API should initialize a new Queue with the test files on the API side. But we don’t know which parallel CI nodes will connect to the API first. There is mutex protection on the API side to detect the very first request but we also need to take care of things on the Knapsack Pro Core side.

Let’s start with a simple example. You have 2 parallel CI nodes. The first CI node has node index 0. The second parallel CI node has index 1. Note that the convention is to start index number from 0 to N-1 (N is a total number of parallel CI nodes).

Let’s assume that only the first parallel CI node (CI node index 0) sends requests to the Knapsack Pro API because the CI machine for the first CI node started work earlier than the second CI node.

The first CI node sends the below request. Its purpose is to attempt to connect to the existing Queue on the API side.

// 1st type of request to Queue API should set attributes:
// can_initialize_queue: true AND attempt_connect_to_queue: true
// Note that there is no test_files parameter in the payload to make the request fast and keep the payload small.

{
  "can_initialize_queue": true,
  "attempt_connect_to_queue": true,
  "fixed_queue_split": false,
  "commit_hash": "6e3396177d9f8ca87e2b93b4b0a25babd09d574d",
  "branch": "main",
  "node_total": "2",
  "node_index": "0",
  "node_build_id": "1234",
  "user_seat": "Jo** Do* <jo**.do*@ex*****.co*>"
}

The above request was the very first request sent to the API and on the API side the Queue does not exist yet. It means the API response returns an error informing us about the Queue not existing.

// 1st type of response to the 1st type of request (when can_initialize_queue: true AND attempt_connect_to_queue: true)
// It can happen only when the queue does not exist on the API side or cannot be read from the cache on the API side

{
  "queue_name": "1:6baacadcdd493c1a6024ee7e51f018f5",
  "message": "A queue with a list of test files does not exist on the API side yet. If you see this message, everything works as expected. Now Knapsack Pro client will initialize a queue on the API side with a list of test files you want to run. The request to initialize the queue will have attributes like can_initialize_queue=true, attempt_connect_to_queue=false, and test_files, etc.",
  "code": "ATTEMPT_CONNECT_TO_QUEUE_FAILED"
}

You need to make a second request. This time, it should contain a list of test files residing on the disk. These will be used to create the Queue.

// 2nd type of request to Queue API should happen only if the API response for 1st type of request has:
// "code": "ATTEMPT_CONNECT_TO_QUEUE_FAILED"
// it means an attempt to connect to the queue failed because the queue does not exist on the API side yet.
// You must initialize a new queue with the below request.
// It should set attributes:
// can_initialize_queue: true AND attempt_connect_to_queue: false
// Note that there is a test_files attribute in the payload to initialize a queue based on the list of test_files from your disk.
// This request can be slow if you provide a large number of test files (~1000+).
// That is why we did 1st request to try to connect to the existing queue first (as one of the other parallel CI nodes could have already initialized it).

{
  "can_initialize_queue": true,
  "attempt_connect_to_queue": false,
  "fixed_queue_split": false,
  "commit_hash": "6e3396177d9f8ca87e2b93b4b0a25babd09d574d",
  "branch": "main",
  "node_total": "2",
  "node_index": "0",
  "node_build_id": "1234",
  "user_seat": "Jo** Do* <jo**.do*@ex*****.co*>",
  "test_files": [
    {
      "path": "test/fast/a_test.rb"
    },
    {
      "path": "test/fast/b_test.rb"
    },
    {
      "path": "test/slow/c_test.rb"
    },
    {
      "path": "test/slow/d_test.rb"
    }
  ]
}

API should return a set of test files assigned to the first CI node (CI node index 0). You should run the test files with your test runner now (using the Knapsack Pro Test Runner client - we will describe it later).

// 2nd type of response can happen for all types of request
// It returns a list of test files that should be run with your test runner

{
  "queue_name": "1:6baacadcdd493c1a6024ee7e51f018f5",
  "build_subset_id": null,
  "test_files": [
    {
      "path": "test/slow/d_test.rb",
      "time_execution": 3.14
    },
    {
      "path": "test/fast/b_test.rb",
      "time_execution": null
    }
  ]
}

After you execute the test files, you should ask the API for another set of test files until the API response contains an empty list of test files. This signifies that the whole Queue has been consumed.

// 3rd type of request to Queue API should happen only if 1st or 2nd type of request returned a list of test_files.
// With the below request you can continue fetching test files from the queue to run them with your test runner.
// Request payload should have attributes:
// can_initialize_queue: false AND attempt_connect_to_queue: false
// Note there is no test_files attribute in the payload to make the request fast and keep the payload small.

{
  "can_initialize_queue": false,
  "attempt_connect_to_queue": false,
  "fixed_queue_split": false,
  "commit_hash": "6e3396177d9f8ca87e2b93b4b0a25babd09d574d",
  "branch": "main",
  "node_total": "2",
  "node_index": "0",
  "node_build_id": "1234",
  "user_seat": "Jo** Do* <jo**.do*@ex*****.co*>"
}

When the API response has no test files it means the Queue was consumed and all test files were executed.

{
  "queue_name": "1:6baacadcdd493c1a6024ee7e51f018f5",
  "build_subset_id": null,
  "test_files": []
}

After test files have been run and their execution time has been recorded you can send the test files timing data to the Knapsack Pro API. You need to create a build subset record on the API side.

You can consult the source code of @knapsack-pro/core responsible for making requests to Knapsack Pro API for Queue Mode and for a request to create a build subset.

Please also see how Knapsack Pro Core (@knapsack-pro/core) uses a service for the API to run tests, record tests execution time, and save recorded test files time for a given CI node as a build subset in the API.

Knapsack Pro Test Runner integration

In this section, you will learn what’s need to be covered in the Knapsack Pro Test Runner source code (e.g. @knapsack-pro/jest).

You need to recognize environment variables defined by user of Knapsack Pro client. The user define them in CI environment variables settings.

KNAPSACK_PRO_TEST_SUITE_TOKEN_JEST - the value of it should override the KNAPSACK_PRO_TEST_SUITE_TOKEN so that Knapsack Pro Core (@knapsack-pro/core) can use the API token during requests.
KNAPSACK_PRO_TEST_FILE_PATTERN - it should contain a default test file pattern that can be used to detect test files on the disk in a directory specific to your test runner. We use a glob function to detect test files on the disk.
There should be a test file finder service that can recognize the pattern and find the list of test files on the disk. We use this list of test files to send them in request to the API so that the API server can split those test files into parallel CI nodes.
KNAPSACK_PRO_TEST_FILE_EXCLUDE_PATTERN - is an exclude pattern. If a user wants to ignore some of the test files she can provide a pattern for it.

Knapsack Pro Test Runner library (e.g. @knapsack-pro/jest) should have their name and version and it should be passed to Knapsack Pro Core (@knapsack-pro/core) when you will use core functionality to connect with the API (for instance to run tests in Queue Mode).

Please note that Knapsack Pro Test Runner should track recorded test files time execution in seconds and pass it back to Knapsack Pro Core. It should also pass info whether tests are green or red (failing). Thanks to that Knapsack Pro Core will set proper process exit status. When at least 1 test fails then the process exit status should be 1 so the CI provider will mark your CI build as a failed one.

Testing your Knapsack Pro client

For testing your Knapsack Pro client I recommend creating a new project with tests in your testing framework (test runner). Here is an example project with Jest tests.

You can create a bin script that runs tests for a given CI node index using the Knapsack Pro client.

For instance use:

bin/knapsack_pro_jest 0 2 - run tests on CI node index 0. The total number of CI nodes is 2.
bin/knapsack_pro_jest 1 2 - run tests on CI node index 1.

README

It’s good to create a well-documented README for your packages. You can get inspired by checking documentation for:

Extras - Regular Mode integration

If you would like to build the Knapsack Pro client that uses Regular Mode instead of Queue Mode you need to replace the step with using Queue Mode and just use Regular Mode API instead. It’s much simpler than Queue API.

In Regular Mode, you need to send a list of existing test files on the disk to the API. The API returns a set of test files to run. Once you execute the tests you need to create a build subset record in the API.

KNAPSACK_PRO_FIXED_TEST_SUITE_SPLIT - Regular Mode has a flag to control whether tests split should be cached on the API side. It’s true by default. Learn more about it from knapsack_pro ruby gem documentation.
- Value of KNAPSACK_PRO_FIXED_TEST_SUITE_SPLIT should be sent as attribute fixed_test_suite_split in request to the API.

Summary

We covered how to build Knapsack Pro client integration from scratch based on the example of the existing JavaScript/TypeScript client built from 2 NPM packages @knapsack-pro/core and @knapsack-pro/jest.

I hope you find it useful. I recommend digging into the source code of the above packages. They are lightweight and should be easy to understand. You can replicate their behavior to build your integration with Knapsack Pro API for your favorite programming language and your test runner (testing framework).

Fix intermittently failing CI builds due to flaky tests in RSpec

Mon, 15 Feb 2021 08:00:00 +0000

Randomly failing CI builds can be frustrating. One run is green, another is red (with no changes in the code!). Disheartening. Let’s see what can be done with this.

Random failures

When builds are randomly failing, flaky tests are often the culprit.

A flaky test is a test that can both fail and pass for the exact same code. In other (and more fancy) words, the test is nondeterministic. Its outcome varies, even if (seemingly) nothing else does.

Trust issues

Nondeterministic tests erode trust. Keeping them in the mix can quickly cause the whole test suite to lose its value. Think about it. The failed build is useful when the failure actually signifies something. It needs to make us stop. To signal that something needs fixing. The more it happens for trivial reasons, the less alert we become to it.

It’s better to not run the test case at all if it can’t be trusted to provide useful feedback.

First things first

When you encounter this problem, think if you need a given test at all. If it wasn’t there already, would you write it? Ask other team members for their opinion.

If the test is deemed important, the correct (and safest) solution would be to debug and solve the underlying issues right away.

Not possible at the moment? At least make sure that it’s the failure of the test which is random, and not its passing. Then I strongly suggest documenting the issue, creating a ticket, or planning to solve it in some other way.

Raise the issue inside your team. If they care about quality assurance and continuous delivery (or their own sanity, really), they will understand.

When this is taken care of, we have a few short-term options.

Rerunning failed builds

When the failure of a CI build is attributed to randomness, the natural reaction is to retry the build. If the rerun build passes, it would be best to still document what happened.

Some CI providers let you configure auto-retrying failed builds. It might be tempting to “solve” randomness this way. I advise against using it (for flaky tests anyway).

Here at Knapsack we are all about decreasing build times. We strongly believe that the shorter the build times, the more productive the team becomes. Frequently rerunning the whole build results in just the opposite. It wastes a lot of time.

If you don’t trust it, x it!

In RSpec, we can change the it method (denoting a test example) to xit to skip a given test case [1]. By default, RSpec’s output shows how many tests were skipped. This way, they stay visible. This increases the chance that they will be fixed. That’s what makes this solution advantageous over just commenting the problematic code out.

Rerunning failed test case

Some flaky tests are caused by inadvertent order dependence or timing issues. Rerunning the individual example could result in a pass. rspec-retry is a library for RSpec that allows doing just that. With it, you can configure which tests should be retried on failure, and how many times. It’s also possible to specify wait time before subsequent retries.

Retrying on a test case level limits the amount of time wasted. If you use Knapsack Pro in the Queue Mode, the waste should be negligible.

If you don’t use RSpec, just search for a similar solution for your test runner. It’s likely to be built-in or provided as an extension by an additional library.

Conclusion

By understanding the consequences of each solution, we are well-equipped to tackle flaky tests.

How does your team respond to random failures? Please share in the comments below!

[1] The ‘x’ prefix works with other methods as well. Please consult the RSpec docs for details.

How to merge SimpleCov results with parallel Rails specs on Semaphore CI

Thu, 05 Nov 2020 21:00:00 +0000

If you are running Rails specs on parallel machines with Knapsack Pro, one challenge you will run into is combining the code coverage results generated by SimpleCov.

This post will show you how to generate a report of the total combined code coverage after all the tests have executed. Here’s what the pipeline diagram looks like at a high level with unrelated sections blurred out:

Configure

SimpleCov

First lets look at the config file used for SimpleCov below. Note no minimum_coverage configuration for failing the build. This is because each node most likely will not meet the minimum coverage threshold on its own so it could lead to the build failing erroneously.

Also note, before_queue hook for Knapsack Pro. This is the important piece, it will set a command name based on the CI node index so that the results are recorded against it.

.simplecov:

SimpleCov.start do
  add_filter %r{^/config/}
  add_filter %r{^/db/}
  add_filter %r{^/spec/}

  add_group 'Admin', 'app/admin'
  add_group 'Controllers', 'app/controllers'
  add_group 'Helpers', 'app/helpers'
  add_group 'Jobs', 'app/jobs'
  add_group 'Libraries', 'lib/'
  add_group 'Mailers', 'app/mailers'
  add_group 'Models', 'app/models'
  add_group 'Policies', 'app/policies'
  add_group 'Serializers', 'app/serializers'
end
Rails.application.eager_load!

KnapsackPro::Hooks::Queue.before_queue do
  SimpleCov.command_name("rspec_ci_node_#{KnapsackPro::Config::Env.ci_node_index}")
end

So now when SimpleCov creates a .resultset.json it will have a specific key depending on which CI node it was run in like the example below. This will be useful down the line when it comes to combining the results.

{
  "rspec_ci_node_0": {
    "coverage": { ... }
  }
}

{
  "rspec_ci_node_1": {
    "coverage": { ... }
  }
}
...

Semaphore CI

Below is the relevant portions of the Semaphore CI configuration. It runs the Rails tests and then uploads the coverage results as a Semaphore workflow artifact. After all the parallel tests have completed, it will run a job to collate the coverage results from all the machines.

.semaphore/semaphore.yml:

- name: "Rails Tests"
    task:
      jobs:
        - name: Rails
          parallelism: 10
          commands:
            - ./.semaphore/helpers/rails_tests.sh
      epilogue:
        always:
          commands:
            - ./.semaphore/helpers/upload_test_artifacts.sh
      secrets:
        - name: docker-hub
        - name: knapsack-pro-rails

- name: "Code Coverage"
  dependencies: ["Rails Tests"]
  task:
    env_vars:
      - name: SEMAPHORE_RAILS_JOB_COUNT
        value: "10"
    jobs:
      - name: Collate Results
        commands:
          - ./.semaphore/helpers/calc_code_coverage.sh
    secrets:
      - name: docker-hub

Below is the bash file which executes the Rails tests on each parallel machine. It sets up the Rails environment and then runs Knapsack Pro in Queue Mode.

.semaphore/helpers/rails_tests.sh:

#!/bin/bash
set -euo pipefail

docker-compose -f docker-compose.semaphore.yml --no-ansi run \
  -e KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES=true \
  -e KNAPSACK_PRO_TEST_FILE_PATTERN="spec/**/*_spec.rb" \
  ci bash -c "bin/rake ci:setup db:create db:structure:load knapsack_pro:queue:rspec['--no-color --format progress --format RspecJunitFormatter --out tmp/rspec-junit/rspec.xml']"

If you want you can also learn how to use RspecJunitFormatter with KnapsackPro Queue Mode.

This is the bash file which is responsible for uploading the SimpleCov results from each machine. It compresses the coverage directory and uploads it to Semaphore.

.semaphore/helpers/upload_test_artifacts.sh:

if [ -d "tmp/rspec-junit" ]
then
  echo "Pushing rspec junit results"
  artifact push job tmp/rspec-junit --destination semaphore/test-results/
fi

if [ -d "coverage" ]
then
  echo "Pushing simplecov results"
  tar czf coverage_${SEMAPHORE_JOB_INDEX}.tgz -C coverage .
  artifact push workflow coverage_${SEMAPHORE_JOB_INDEX}.tgz
fi

Lastly, this is the bash file for collating all the results from Semaphore. It will download the coverage artifacts from each parallel machine and run a rake task which will collate them and then upload the results into a combined total_coverage.tgz file as shown below:

.semaphore/helpers/calc_code_coverage.sh:

#!/bin/bash
set -euo pipefail

for i in $(eval echo "{1..$SEMAPHORE_RAILS_JOB_COUNT}"); do
  artifact pull workflow coverage_$i.tgz;
  mkdir coverage_$i
  tar -xzf coverage_$i.tgz -C coverage_$i
done

docker-compose -f docker-compose.semaphore.yml --no-ansi run ci bash -c "bin/rake coverage:report"
tar czf total_coverage.tgz -C coverage .
artifact push workflow total_coverage.tgz

This coverage:report rake task will simply call SimpleCov.collate which will go through the coverage results in each folder and combine them into a single .resultset.json shown below

lib/task/coverage_report.rake:

namespace :coverage do
  desc 'Collates all result sets generated by the different test runners'
  task report: :environment do
    require 'simplecov'

    SimpleCov.collate Dir['coverage_*/.resultset.json']
  end
end

.resultset.json:

{
  "rspec_ci_node_0, rspec_ci_node_1, rspec_ci_node_2, rspec_ci_node_3, rspec_ci_node_4, rspec_ci_node_5, rspec_ci_node_6, rspec_ci_node_7, rspec_ci_node_8, rspec_ci_node_9": {
    "coverage": { ... }
}

Summary

Finally here’s what your Semaphore workflow artifacts will look like. It will have a compressed coverage file generated on each machine and a total coverage file that we created at the very end:

This approach can also be easily ported over to other CI providers by simply changing the artifact push and artifact pull commands to S3 or another CI specific artifact upload command.

I hope this article was useful to you. Let me know if you have any questions or feedback. If you liked this post, please share it on social media and follow me on Twitter!

How to build native integration with Knapsack Pro API to run tests in parallel for any test runner (testing framework)

Thu, 24 Sep 2020 22:00:00 +0000

Do you know that Knapsack Pro API can work with any test runner in any programming language?

If your test runner is not listed here as one of the supported test runners out of the box in Knapsack Pro, then you can use @knapsack-pro/core npm package to directly integrate with Knapsack Pro API and build your test runner integration with Knapsack Pro API.

We have users who did that, for instance, for the TestCafe test runner.

Knapsack Pro offers out of the box support for test runners like Cypress and Jest with these packages:

They both use @knapsack-pro/core which is a wrapper around Knapsack Pro API.

@knapsack-pro/core provides support for Knapsack Pro Queue Mode API. Thanks to that, you can run tests in parallel CI nodes using a dynamic test suite split with Queue Mode. To learn more about how the Queue Mode works, you can see the section Dynamic tests split of the article describing the difference between Regular Mode and Queue Mode.

How Queue Mode works with Knapsack Pro API

Here is the general idea behind Queue Mode in Knapsack Pro.

There are parallel CI nodes on your CI server. Each CI node is running the Knapsack Pro client command to run tests.

The very first request from the Knapsack Pro client command (example npx @knapsack-pro/cypress) sends a list of all test files existing on the disk to Knapsack Pro API Queue. Then API returns the proper set of tests for the CI node.
There is a Queue with a list of test files on the Knapsack Pro API side. The Queue is build based on a list of tests sent to the API and based on historically recorded data about your tests execution time in order to sort tests in the Queue from slowest to fastest.
Each Knapsack Pro client command connects with the Knapsack Pro API Queue and consumes a set of tests fetched from the Queue. API returns a set of tests from the top of the Queue (slowest first).
Once the set of tests is executed on the CI node then the Knapsack Pro client command asks for another set of tests from the Queue. This is repeated until the Queue is empty.
Once all tests are executed and their execution time is recorded then the Knapsack Pro client command sends recorded time of each test file to Knapsack Pro API (this creates a Build Subset on the API side).

Thanks to the Queue Mode, tests are allocated between parallel CI nodes in a dynamic way to ensure that all CI nodes finish their work at a similar time. This allows getting optimal CI build time (as fast as possible).

Build your own integration with Knapsack Pro API

You can fork one of the existing integrations like Cypress (@knapsack-pro/cypress) or Jest (@knapsack-pro/jest) and replace the Cypress/Jest test runner with your own test runner to build the integration.

This article explains how to do it based on @knapsack-pro/cypress npm package.

First, you need to clone @knapsack-pro/core repository which is a core library that allows you to connect with Knapsack Pro API. In the README file, you can find instructions on how to configure and build the project in the development environment.

The next step is to clone @knapsack-pro/cypress repository. You will have to replace Cypress with your test runner npm package.

Here is some basic info about the project structure. It’s written in TypeScript. The TypeScript source code is in src directory. The lib directory contains TypeScript code compiled to JavaScript. You should not modify files in the lib directory because they are overridden during compilation. You can find tips on how to compile the project in the README file.

You can rename forked project @knapsack-pro/cypress to @knapsack-pro/my-test-runner and update the info in package.json.

Update name to @knapsack-pro/my-test-runner and version to 1.0.0. See in code.
Add your package to the peerDependency. This enforces your users to install a specific version of the test runner in their project alongside your package. See in code.
Add your test runner package to devDependencies. This allows using a specific version of the test runner in development for testing your @knapsack-pro/my-test-runner with another local project using an example test suite supported by my-test-runner npm package. For example, to test @knapscak-pro/cypress we have a separate repository with an example test suite written in Cypress. We verify with it that our @knapsack-pro/cypress package (See in code) works fine. In order to do, we’ve created an example bin script to connect with Knapsack Pro API. Please remove from it the ENDPOINT environment variable - this way the @knapsack-pro/core will connect to the production API (https://api.knapsackpro.com) by default.

Note how we pass to KnapsackProCore the list of all existing test files on the disk - see in code. This is needed to initialize the Queue on the API side with the very first request to the API (as mentioned earlier). Those test files will be used to run your tests.

The most important place in the code is running your test runner and passing recorded tests timing data and info if tests are green or red back to the @knapsack-pro/core. See how it’s done for @knapsack-pro/cypress - see in code.

Summary

The description above should allow you to use @knapsack-pro/core and build your own integration for your test runner like TestCafe, etc.

You can fork @knapsack-pro/cypress or check @knapsack-pro/jest which is even thinner than @knapsack-pro/cypress. Just take a look at the source code and README for those projects to learn more.

Note that using @knapsack-pro/core instead of directly writing requests to Knapsack Pro API has the benefit of being able to use @knapsack-pro/core features, like the Fallback Mode. When the library is not able to connect to the API then it can auto-retry requests and show warnings in the logger and also run the tests in the Fallback Mode. As you can see, using the library can help you avoid dealing with many hassles along the way!

I hope this article was useful to you. Let us know if you have any questions or if you would like to see an out of the box integration for your favorite test runner. We’d like to add more test runners to our list of supported out of the box test runners in the future.

How to build a custom Knapsack Pro API client from scratch in any programming language

How to run slow RSpec files on Github Actions with parallel jobs by doing an auto split of the spec file by test examples

Mon, 15 Jun 2020 06:00:00 +0000

Splitting your CI build jobs between multiple machines running in parallel is a great way to make the process fast, which results in more time for building features. Github Actions allows running parallel jobs easily. In a previous article, we explained how you can use Knapsack Pro to split your RSpec test files efficiently between parallel jobs on GitHub Actions. Today we’d like to show how to address the problem of slow test files negatively impacting the whole build times.

Consider the split

Imagine you have a project with 30 RSpec spec files. Each file contains multiple test examples (RSpec “its”). Most of the files are fairly robust, fast unit tests. Let’s say there are also some slower files, like feature specs. Perhaps one such feature spec file takes approximately 5 minutes to execute.

When we run different spec files on different parallel machines, we strive for similar execution time on all of them. In a described scenario, even if we run 30 parallel jobs (each one running just one test file), the 5 minute feature spec would be a bottleneck of the whole build. 29 machines may finish their work in a matter of seconds, but the whole build won’t be complete until the 1 remaining node finishes executing its file.

Divide and conquer

To solve the problem of a slow test file, we need to split what’s inside it. We could refactor it and ensure the test examples live in separate, smaller test files. There are two problems with that though:

First, it needs work. Although admittedly quite plausible in a described scenario, in real life it’s usually not just the one file that’s causing problems. Oftentimes there is a number of slow and convoluted test files, with their own complex setups, like nested before blocks, lets, etc. We’ve all seen them (and probably contributed to them ending-up this way), haven’t we? ;-) Refactoring files like that is no fun, and there seem to always be more higher priority work to be done, at least from our experience.

Second, we believe that the code organization should be based on other considerations. How you create your files and classes is most likely a result of following some approach agreed upon in your project. Dividing classes into smaller ones so that the CI build can run faster encroaches on your conventions. It might be more disturbing to some than the others, but we feel it’s fair to say it’d be best to avoid - if there was a better way to achieve the same…

Enter split by test examples

As you certainly know, RSpec allows us to run individual examples instead of whole files. We decided to take advantage of that, and solve the problem of bottleneck test files by gathering information about individual examples from such slower files. Such examples are then dynamically distributed between your parallel nodes and run individually, so no individual file can be a bottleneck for the whole build. What’s important, no additional work is needed - this is done automatically by the knapsack_pro gem. Each example is run in its correct context that’s set-up exactly the same as if you had run the whole file.

If you are already using Knapsack Pro in queue mode, you can enable this feature just by adding one ENV variable to your GitHub Actions workflow config: KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES: true (please make sure you’re running the newest version of knapsack_pro gem). After a few runs, Knapsack Pro will start automatically splitting your slowest test files by individual examples.

Here’s a full example GitHub Actions workflow config for a Rails project using RSpec:

# .github/workflows/main.yaml

name: Main

on: [push]

jobs:
  test:
    runs-on: ubuntu-latest

    # If you need DB like PostgreSQL, Redis then define service below.
    # https://github.com/actions/example-services/tree/main/.github/workflows
    services:
      postgres:
        image: postgres:10.8
        env:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: ""
          POSTGRES_DB: postgres
        ports:
          - 5432:5432
        # needed because the postgres container does not provide a healthcheck
        # tmpfs makes DB faster by using RAM
        options: >-
          --mount type=tmpfs,destination=/var/lib/postgresql/data
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
      redis:
        image: redis
        ports:
          - 6379:6379
        options: --entrypoint redis-server

    # https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#jobsjob_idstrategymatrix
    strategy:
      fail-fast: false
      matrix:
        # Set N number of parallel jobs you want to run tests on.
        # Use higher number if you have slow tests to split them on more parallel jobs.
        # Remember to update ci_node_index below to 0..N-1
        ci_node_total: [8]
        # set N-1 indexes for parallel jobs
        # When you run 2 parallel jobs then first job will have index 0, the second job will have index 1 etc
        ci_node_index: [0, 1, 2, 3, 4, 5, 6, 7]

    steps:
      - uses: actions/checkout@v2

      - name: Set up Ruby
        uses: actions/setup-ruby@v1
        with:
          ruby-version: 2.6

      - uses: actions/cache@v2
        with:
          path: vendor/bundle
          key: ${{ runner.os }}-gems-${{ hashFiles('**/Gemfile.lock') }}
          restore-keys: |
            ${{ runner.os }}-gems-
      - name: Bundle install
        env:
          RAILS_ENV: test
        run: |
          bundle config path vendor/bundle
          bundle install --jobs 4 --retry 3
      - name: Create DB
        env:
          # use localhost for the host here because we have specified a container for the job.
          # If we were running the job on the VM this would be postgres
          PGHOST: localhost
          PGUSER: postgres
          RAILS_ENV: test
        run: |
          bin/rails db:prepare
      - name: Run tests
        env:
          PGHOST: localhost
          PGUSER: postgres
          RAILS_ENV: test
          KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC: ${{ secrets.KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC }}
          KNAPSACK_PRO_CI_NODE_TOTAL: ${{ matrix.ci_node_total }}
          KNAPSACK_PRO_CI_NODE_INDEX: ${{ matrix.ci_node_index }}
          KNAPSACK_PRO_FIXED_QUEUE_SPLIT: true
          KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES: true
          KNAPSACK_PRO_LOG_LEVEL: info
        run: |
          bundle exec rake knapsack_pro:queue:rspec

You can find more details in the video below:

Let us know in the comments what you think about this solution. If you’d like to give this setup a try, you can also consult our FAQ entry explaining how to split slow RSpec test files.

As always, don’t hesitate to ask questions if you encounter any troubles with configuring GitHub Actions in your project - we’d be happy to help!

How to merge Cypress test reports generated by Mochawesome on Github Actions

Thu, 05 Mar 2020 11:00:00 +0000

Looking through CI logs to find out which of your Cypress tests failed can be time consuming and error-prone, especially when your tests are spread out across several machines. With the Mochawesome test report generator and Github Actions, you can easily generate a beautiful, easy-to-read report of your test run, which merges results from all parallel nodes.

Cypress Setup

First, add Mochawesome as a dependency of your project:

yarn install --save-dev mochawesome or

npm install --save-dev mochawesome.

Next, tell Cypress to use Mochawesome by adding the following to your cypress.json:

{
  "reporter": {
    "reporterOptions": {
      "overwrite": false,
      "html": false,
      "json": true
    }
  }
}

Now, when you run a Cypress test suite, a .json test report file will be generated for each .spec file ran, inside the mochawesome-report directory. Since Knapsack Pro runs tests one at a time, we need "overwrite": false to ensure that previous test reports are kept during subsequent test runs.

Note that we wrote "html": false and "json": true. Since we’re running tests one at a time, generating an HTML report now will be useless, since it will only show the results of one spec file. These options tell Mochawesome to only generate a JSON report. After all the tests are done, we will merge all the JSON reports together, then generate an HTML report, so that we have one report with all of our test results.

Github Actions Workflow

Now we can configure our Github Actions workflow. The first job will run our Cypress tests with Knapsack Pro:

  test_e2e:
    ...
    strategy:
      fail-fast: false
      matrix:
        node-version: [10.16.2]
        # Knapsack Pro parallelization
        ci_node_total: [4]
        ci_node_index: [0, 1, 2, 3]
    steps:
      ...
      - name: Configure Knapsack Pro
        run: |
          echo ::set-env name=KNAPSACK_PRO_TEST_SUITE_TOKEN_CYPRESS::${{ secrets.KNAPSACK_PRO_TEST_SUITE_TOKEN_CYPRESS }}
          echo ::set-env name=KNAPSACK_PRO_TEST_FILE_PATTERN::"cypress/**/*.spec.{js,ts}"
          echo ::set-env name=KNAPSACK_PRO_CI_NODE_TOTAL::${{ matrix.ci_node_total }}
          echo ::set-env name=KNAPSACK_PRO_CI_NODE_INDEX::${{ matrix.ci_node_index }}

In this case, I am using 4 parallel CI nodes, but you should modify this accordingly if you’re using more or less. We will use the $KNAPSACK_PRO_CI_NODE_INDEX environment variable later, as this tells us the index of the currently running parallel node. Since I’m using 4 machines, this will be a number from 0 - 3.

Now we can run @knapsack-pro/cypress as usual:

      - name: Run E2E tests
        run: npx @knapsack-pro/cypress
      - name: Upload E2E test reports
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: test-reports-${{ env.KNAPSACK_PRO_CI_NODE_INDEX }}
          path: mochawesome-report

Running the tests will create the mochawesome-report directory, containing all our individual test reports. To pass these along to the next job, which will take care of merging them, we use Github’s upload-artifact action. Note that we provide the condition if: always(), to ensure that test reports will always be uploaded, even if some of our tests failed.

When naming the artifact, we’re using the node index environment variable from earlier. When running on 4 machines for example, this means that 4 artifacts will be uploaded: test-reports-0, test-reports-1, test-reports-2, test-reports-3.

The next job, which runs after all the parallel tests finish, will download the test reports, merge all of them, and upload a final HTML report.

  gen_report:
    name: Generate test report
    needs: test_e2e
    if: always()
    runs-on: ubuntu-latest
    steps:
      - name: Install dependencies
        run: npm install mochawesome-merge mochawesome-report-generator

The if: always() ensures that reports will be generated even if any tests fail. Next, we install the dependencies needed to merge and generate the report.

Now we need to download all our test artifacts, which were uploaded as test-reports-0, test-reports-1, etc. Unfortunately, since Github Actions doesn’t support YAML anchors, this part requires a bit of copy and paste. Just like before, tweak this part depending on how many parallel nodes you’re using:

      - name: Create reports directory
        run: mkdir reports
      - name: Download test-reports-0
        uses: actions/download-artifact@v4
        with:
          name: test-reports-0
          path: reports/0
      - name: Download test-reports-1
        uses: actions/download-artifact@v4
        with:
          name: test-reports-1
          path: reports/1
      - name: Download test-reports-2
        uses: actions/download-artifact@v4
        with:
          name: test-reports-2
          path: reports/2
      - name: Download test-reports-3
        uses: actions/download-artifact@v4
        with:
          name: test-reports-3
          path: reports/3

Now we have a reports directory, with subdirectories containing reports from each node. Our directory structure will now look like this:

❯ tree reports
reports
├── 0
│   ├── mochawesome_001.json
│   ├── mochawesome_002.json
│   ├── mochawesome_003.json
│   ├── mochawesome_004.json
│   ├── mochawesome_005.json
│   └── mochawesome.json
├── 1
│   ├── mochawesome_001.json
│   ├── mochawesome_002.json
│   ├── mochawesome_003.json
│   └── mochawesome.json
├── 2
│   ├── mochawesome_001.json
│   ├── mochawesome_002.json
│   ├── mochawesome_003.json
│   └── mochawesome.json
└── 3
    ├── mochawesome_001.json
    ├── mochawesome_002.json
    ├── mochawesome_003.json
    └── mochawesome.json

4 directories, 18 files

To merge these reports, we want to have them all in one directory. The next step is a short Bash script:

      - name: Move all reports into one directory
        run: |
          mkdir -p mochawesome-report
          i=0
          for file in $(find reports -type f -name mochawesome\*.json); do
            filename=$(basename "$file" .json)-"$i".json
            mv "$file" mochawesome-report/"$filename"
            ls mochawesome-report
            i=$((i + 1))
          done

This script finds all the .json report files and moves them to the mochawesome-report folder, which will look like this:

❯ tree mochawesome-report 
mochawesome-report
├── mochawesome_001-10.json
├── mochawesome_001-16.json
├── mochawesome_001-2.json
├── mochawesome_001-6.json
├── mochawesome_002-15.json
├── mochawesome_002-1.json
├── mochawesome_002-5.json
├── mochawesome_002-9.json
├── mochawesome_003-0.json
├── mochawesome_003-14.json
├── mochawesome_003-4.json
├── mochawesome_003-8.json
├── mochawesome_004-13.json
├── mochawesome_005-12.json
├── mochawesome-11.json
├── mochawesome-17.json
├── mochawesome-3.json
└── mochawesome-7.json

0 directories, 18 files

Note that a number is appended to the end of each filename. This is only to prevent filename conflicts when we’re moving all the reports into one directory. The name of the file won’t have any affect on the generated report.

Now we can actually generate the final report:

      - name: Merge and generate reports
        run: |
          mkdir report
          npx mochawesome-merge > report/report.json
          npx marge --inline report/report.json -o report/

We ran these commands to:

mkdir report: Create the directory that will hold the final report
npx mochawesome-merge > report/report.json: Merge all of our .json files from the mochawesome-report directory into one file, report/report.json
npx marge --inline report/report.json -o report/: Take the merged report and generate a pretty HTML report from it

Now we upload this report:

      - name: Upload report
        uses: actions/upload-artifact@v4
        with:
          name: E2E Test Reports
          path: report/

Now you will be able to download your report from the build artifacts:

If you want to learn more about how to test your Javascript applications faster then check out the blog and website about the Knapsack Pro tool for CI parallelisation.

How to speed up Ruby and JavaScript tests with CI parallelisation

Sun, 02 Feb 2020 15:30:00 +0000

When working on a larger project, you may struggle with the problem of an increasingly growing set of tests, which over time begins to perform slower on your continuous integration (CI) server. I had this problem while working on a project in Ruby on Rails, where RSpec tests on CircleCI took about 15 minutes.

As it was bothering me, I decided to do something about it, which resulted in building an open-source Knapsack Ruby gem library (the name derives from the knapsack problem), which deals with distributing tests between parallel CI servers. In this article, you will learn about two approaches to split tests on parallel continuous integration servers - static and dynamic.

If you have in your project a test suite that takes to execute on a CI server a dozen or so minutes, or maybe even a few hours, you know how inconvenient this is for programmers. When you are working on some new feature and pushing a new git commit into the repository, you have to wait a long time for your CI server until it executes CI build.

Waiting a few minutes or an hour is delaying the feedback you can get from the CI server about tests that may have not been completed (red tests). After all, we all want to get information about whether our CI build is green or red as soon as possible so that the work of programmers is not blocked.

Problem with running parallel tests on the CI server

To speed up the execution of the CI build, you can use parallelism on the CI server, i.e. launching several parallel CI machines (CI containers, e.g. in Docker), where each parallel server will perform a part of the test set. However, there is a problem as to which tests should be run on which servers (CI nodes) so that their distribution is fairly even and you don’t have to wait for a CI node that is a bottleneck.

Below you can see an example of a non-optimal distribution of tests on 4 CI servers, where the second server marked in red is a bottleneck, so the waiting time for the completion of the entire CI build is up to 20 minutes.

Optimal distribution of tests on parallel CI servers

In an ideal scenario, the tests should be distributed in such a way that all parallel CI servers end operations at a similar time. In the following part, I will show how this can be achieved.

Static split of tests in a deterministic way - Regular Mode

One way to determine how to divide tests between parallel machines on a CI server so that each server completes tests at a similar time is to use the measured runtime of the files in the test suite. This was the first approach I implemented in Knapsack Ruby gem.

After measuring the test execution time, we can assign individual test files between parallel CI servers to make sure that the CI build does not have a bottleneck.

With the help of the knapsack library, you can run tests for many test runners in Ruby, such as RSpec, Minitest, Cucumber, Spinach, and Turnip. Using test runtime, Knapsack gem can build a list of tests to be performed on a specific CI node.

I improved this way of dividing tests by measuring test files timing per git commit and branches. In the below video I show how Regular Mode a static split of tests in a deterministic way works in Knapsack Pro. In the next section, you will learn about some of the edge cases of this approach and how to solve it.

Problem with the static split of tests

While collecting information from users, I found out that the distribution of tests in a static way is not always a good solution. Sometimes some tests have a random execution time, which depends, for example, on how busy the CI server is or on the fact that the test does not pass due to a software error, quitting work faster than usual, etc.

For example, tests using a browser can have fluctuations in execution runtime (tests in Capybara in Ruby or E2E tests in JavaScript).

The problem also grows depending on what CI server you use. Does each of the parallel CI machines have similar performance or does it share resources like a CPU or RAM? Does the CI container run in a shared environment? If the CI node is overloaded, then our tests may, of course, be slower.

Besides, there will be problems with whether all parallel machines start at a similar time or not. If you have purchased a pool of parallel CI servers, someone else might be using it too, e.g. another CI build from the current project or another project from your organization.

If not all CI nodes start at the same time or the boot time of certain steps in the middle of the CI node execution can take a random time, then we would like to be able to make sure that all CI machines finish their work at a similar moment. Slow CI machines or those which started work late should do fewer tests, and those machines that have started work earlier can easily do more.

All parallel CI nodes must stop working at a similar time to avoid a bottleneck, that is, overloading the machine with tests.

Dynamic tests split - Queue Mode

The solution to the above problem is to dynamically divide tests between parallel machines within one CI build. This is a problem I have been working on in recent years, creating the Knapsack Pro library and the Queue Mode for Ruby and JavaScript with support for several popular test runners like Jest or Cypress.

The idea is simple. We have a set of tests that are queued on the Knapsack Pro server. Individual parallel CI machines consume the queue with Knapsack Pro API until the queue is over. Thanks to this, the tests are optimally distributed among CI servers, helping you to avoid a bottleneck in the form of an overloaded (too slow) CI server. Below you can see an example:

Dynamic test suite split solves our problem with random test execution time, with slow running CI servers or with servers that are overloaded which work is slower. No matter when they start or finish work - it’s important that they don’t take too many tests to execute until they finish their current work.

See how dynamic test suite split works in Queue Mode for Knapsack Pro.

Implementation of Knapsack Pro in Ruby and JavaScript

Knapsack Pro has native support for many popular CI servers. It is also an agnostic CI tool, so you can use any CI server. All you have to do is configure the Knapsack Pro command for each parallel CI server running within one CI build. Below you can see a general example of how config YAML might look for a CI server with Knapsack Pro:

If you use RSpec and you have very slow test files you can auto split them. Knapsack Pro detects slow RSpec test files to split it by test examples on parallel jobs.

Conclusions

Knapsack Pro supports Ruby and several test runners in JavaScript such as Jest and Cypress, but there are plans to add support for more test runners and programming languages. I would love to hear what you use to test applications and which CI servers. In case you are considering changing your CI provider, check out our list of CI servers features. You can contact me on LinkedIn, and you can find more information about the described solution at KnapsackPro.com. I hope this article was useful to you. :)

URI.escape is obsolete. Percent-encoding your query string

Sun, 19 Jan 2020 12:00:00 +0000

Have you encountered one of those warnings in your Ruby 2.7.0 project?

warning: URI.escape is obsolete
warning: URI.encode is obsolete

Find out how to fix it!

A little bit of history

Ruby 2.7.0 shows a warning when invoking URI.escape or its alias, URI.encode. It might look like a fresh deprecation, but the fact is, these methods have been marked as obsolete for… over 10 years now! If you’re wondering how come you’ve never encountered the warning before, here’s the answer: previously it was displayed only if you run your script in a verbose mode, and this has changed just recently.

So why is `URI.escape` obsolete, anyway?

The trouble with a concept of “escaping the URI” is that URI consists of many components (like path or query), and we don’t want to escape them in the same way. For example, it’s fine for a # character to appear at the end of the URI (when it’s used as what’s usually called an anchor, or in URI parlance - a fragment component) - but when the same # is part of user’s input (like in a search query), we want to encode it to ensure correct interpretation. Similarly, if the query string value contains other reserved characters, like = or &, we do want to escape them so that they are not incorrectly interpreted as delimiters, as if they were used as reserved characters.

URI.escape relies on a simple gsub operation for the whole string and doesn’t differentiate between distinct components, which doesn’t take into account intricacies like those mentioned above.

How to fix it?

Since I haven’t found an existing solution that would, when given a whole-URI string, interpret distinct components and apply different escaping rules in a desired way on its own, my advice is to encode different components separately. The most common (and most sensitive) use-case is probably encoding of the query string in the query component, so I’ll focus on this case. And Ruby’s URI module itself provides two handy methods that will help us achieve just that!

Percent-encoding your query string

`URI.encode_www_form_component(string, enc=nil)`

This method will percent-encode every reserved character, leaving only *, -, ., 0-9, A-Z, _, a-z intact. It also substitues space with +. Examples:

query = "Tom & Jerry"
query = URI.encode_www_form_component(query)
query # => "Tom+%26+Jerry"

post = "So scared, but let's do this! #yolo"
post = URI.encode_www_form_component(post)
post # => "So+scared%2C+but+let%27s+do+this%21+%23yolo"

As you can see, we can now safely build our query strings with values escaped this way. But if this seems like too much of manual work, there is a handy way to handle the whole query for you. Introducing…

`URI.encode_www_form(enum, enc=nil)`

Slightly different interface. It takes an Enumerable (usually a nested Array or a Hash) and builds the query accordingly. It uses .encode_www_form_component internally, so the encoding rules stay the same. The difference lies in the way you use it. Examples:

URI.encode_www_form([["search", "Tom & Jerry"], ["page", "3"]])
# => "search=Tom+%26+Jerry&page=3"

URI.encode_www_form(["search", "2 + 2 = 5"])
# => "search=2+%2B+2+%3D+5"

URI.encode_www_form(search: "why is URI.escape obsolete", category: "meta")
#=> "search=why+is+URI.escape+obsolete&category=meta"

# Shameless plug
URI.encode_www_form(q: "how to speed up your CI?", tag: ["#devops", "#productivity"], lang: "en")
#=> "q=how+to+speed+up+your+CI%3F&tag=%23devops&tag=%23productivity&lang=en"

Quite straightforward, isn’t it?

Rails project?

`Hash#to_query`

(also aliased as Hash.to_param)

Rails extends the Hash class with this handy method. It will return a query string in a correct format, with its values correctly encoded where needed.

query_data = { q: "how to speed up your CI?", tag: ["#devops", "#productivity"], lang: "en" }
query_data.to_query
#=> "lang=en&q=how+to+speed+up+your+CI%3F&tag%5B%5D=%23devops&tag%5B%5D=%23productivity"

(Notice how it also sorts the keys alphabetically.)

You can also pass an optional namespace to this method, which would enclose the key names in the returned string, resulting in a format like search[q]="how to speed up your CI?" (but obviously percent-encoded):

query_data = { q: "how to speed up your CI?", tag: ["#devops", "#productivity"], lang: "en" }
query_data.to_query("search")
#=> "search%5Blang%5D=en&search%5Bq%5D=how+to+speed+up+your+CI%3F&search%5Btag%5D%5B%5D=%23devops&search%5Btag%5D%5B%5D=%23productivity"

Summing up

I hope that short article cleared up any confusion you might have had about encoding your strings for use in URI. Please let us know in the comments if you use any other ways of dealing with percent-encoding in your Ruby/Rails projects!

And if your project is struggling with slow CI builds, check out Knapsack Pro, which can help you solve that problem.

How to run Cypress parallel tests on Codefresh CI server

Sat, 19 Oct 2019 16:00:00 +0000

When you work with end to end tests in Cypress you may notice they quickly get time-consuming for running the whole test suite. The more complex the project the more test cases you may end up with and this can take a long time to run - dozens of minutes or even hours.

One way of executing faster tests in Cypress is to use CI parallelism which is essentially running a few chunks of test suite spread across parallel jobs on CI server. In this article, you will learn how to configure Codefresh.io server with matrix feature to run parallel steps for your Cypress tests and this way save time on faster CI builds.

Configure Codefresh.io to run Cypress.io tests

You need in your project repository a Dockerfile that will be used by Codefresh to build Docker image which will be used to run Docker container with your Cypress tests inside of it.

FROM cypress/base:10

RUN apt-get update && \
  apt-get install -y \
  python3-dev \
  python3-pip

# Install AWS CLI
RUN pip3 install awscli

# Install Codefresh CLI
RUN wget https://github.com/codefresh-io/cli/releases/download/v0.31.1/codefresh-v0.31.1-alpine-x64.tar.gz
RUN tar -xf codefresh-v0.31.1-alpine-x64.tar.gz -C /usr/local/bin/

COPY . /src

WORKDIR /src

RUN npm install

Next thing is to add .codefresh/codefresh.yml file with instructions to run your application HTTP server in the background and then run Cypress tests against the application.

In the below example, you will see that you can leverage matrix environment configuration for Codefresh. Set a few environmental variables with the node index for each parallel step. This is required by Knapsack Pro tool. Knapsack Pro is a wrapper around Cypress test runner which is responsible for running a proper set of tests across parallel jobs (parallel steps on Codefresh).

Knapsack Pro for Cypress has built-in Queue Mode mechanism which splits test files in a dynamic way across parallel CI nodes (parallel jobs) and thanks to that ensure each parallel step takes a similar time so your CI build is executed in optimal time. You save time by avoiding a not optimal split of tests. You simply eliminate bottleneck slow job that could happen when you allocate too many slow test files to the same parallel step.

You will learn more how exactly Queue Mode works at the end of this article but for now, let’s take a look at Codefresh YAML configuration.

You also need to add an API token for the Cypress test suite. You can get the API token at Knapsack Pro.

You need to set an API token KNAPSACK_PRO_TEST_SUITE_TOKEN_CYPRESS in Codefresh dashboard, see left menu Pipelines -> settings (cog icon next to the pipeline) -> Variables tab (see a vertical menu on the right side).
Set where Codefresh YAML file can be found. In Codefresh dashboard, see left menu Pipelines -> settings (cog icon next to pipeline) -> Workflow tab (horizontal menu on the top) -> Path to YAML (set there ./.codefresh/codefresh.yml).
Set how many parallel jobs (parallel CI nodes) you want to run with KNAPSACK_PRO_CI_NODE_TOTAL environment variable in .codefresh/codefresh.yml file.
Ensure in the matrix section you listed all KNAPSACK_PRO_CI_NODE_INDEX environment variables with a value from 0 to KNAPSACK_PRO_CI_NODE_TOTAL-1. Codefresh will generate a matrix of parallel jobs where each job has a different value for KNAPSACK_PRO_CI_NODE_INDEX. Thanks to that Knapsack Pro knows what tests should be run on each parallel job.

How dynamic test suite split works in Queue Mode

You can learn how dynamic test suite split with Knapsack Pro Queue Mode works in the below video. There are examples of a few edge cases that Queue Mode helps with.

I hope you find this article helpful. You can also check other articles. For instance, I wrote an article about Codefresh and Ruby on Rails project setup with a video showing how it works.

If you are considering using Cypress in your project, check out our comparison of Cypress.io with other solutions. And to compare Codefresh CI to other CIs, you can visit our comparison pages: Codefresh vs CircleCI, Codefresh vs Jenkins, Github Actions vs Codefresh CI, and Codefresh vs Other CI providers.

How to use Codefresh CI parallel steps to run RSpec a few times faster for Rails project

Fri, 18 Oct 2019 17:00:00 +0000

Codefresh.io seems to be a very nice CI solution to work with if you are a Ruby developer. I’ve tested one of my projects on Codefresh to see how it allows running fast tests. Codefresh has a matrix feature that lets you run parallel steps for the CI build. In this article, you will see how to leverage Codefresh matrix configuration and Knapsack Pro client library for testing in parallel your Ruby on Rails project with RSpec test suite.

Configure Rails project on Codefresh.io

In order to run CI builds for Rails project on Codefresh you need YAML configuration file and Docker image that will be used as your base Docker container running your tests.

We also need DB on the CI server. In my Ruby on Rails project, I use PostgreSQL so I have to configure Codefresh service to run Postgres.

Let’s start with creating .codefresh/codefresh.yml file and Test.Dockerfile that you need to add to your repository.

As you can see I use knapsack_pro ruby gem as a command to run my RSpec tests. It will know what set of tests should be executed on a particular parallel step based on KNAPSACK_PRO_CI_NODE_INDEX value. The node index is generated based on the list of KNAPSACK_PRO_CI_NODE_INDEX variables defined for the matrix.

Knapsack Pro will automatically split tests across parallel jobs (steps) to ensure each step takes a similar time. It’s possible thanks to Knapsack Pro Queue Mode which does dynamic test suite split across parallel steps. You can learn more about technical details on how Queue Mode works from the video at the end of this article but let’s check now how Codefresh works in action. You can see my CI builds in Codefresh web dashboard on the below video.

If you forgot steps from the video then I listed things you need to set up in Codefresh dashboard and in YAML config file.

You need to set an API token like KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC in Codefresh dashboard, see left menu Pipelines -> settings (cog icon next to the pipeline) -> Variables tab (see a vertical menu on the right side). Add there new API token depending on the test runner you use:
- KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC
- KNAPSACK_PRO_TEST_SUITE_TOKEN_CUCUMBER
- KNAPSACK_PRO_TEST_SUITE_TOKEN_MINITEST
- KNAPSACK_PRO_TEST_SUITE_TEST_UNIT
- KNAPSACK_PRO_TEST_SUITE_TOKEN_SPINACH
Set where Codefresh YAML file can be found. In Codefresh dashboard, see left menu Pipelines -> settings (cog icon next to pipeline) -> Workflow tab (horizontal menu on the top) -> Path to YAML (set there ./.codefresh/codefresh.yml).
Set how many parallel jobs (parallel CI nodes) you want to run with KNAPSACK_PRO_CI_NODE_TOTAL environment variable in .codefresh/codefresh.yml file.
Ensure in the matrix section you listed all KNAPSACK_PRO_CI_NODE_INDEX environment variables with a value from 0 to KNAPSACK_PRO_CI_NODE_TOTAL-1. Codefresh will generate a matrix of parallel jobs where each job has a different value for KNAPSACK_PRO_CI_NODE_INDEX. Thanks to that Knapsack Pro knows what tests should be run on each parallel job.

How Queue Mode works

In this video, I explained technical details for a dynamic test suite split and what problems it solves to help you run fast CI builds in optimal time.

There are more test runners in Ruby like Cucumber, Minitest, etc that are supported by Knapsack Pro. Even JavaScript tools like Cypress.io or Jest can be launched with Knapsack Pro wrapper (see installation guide).

I hope you find this tutorial useful and you can benefit from faster CI builds for your Rails project thanks to running parallel steps on Codefresh.io.

If you are currently considering a choice between Codefresh and other solutions, check out our comparisons: Codefresh vs CircleCI, Codefresh vs Jenkins, Github Actions vs Codefresh CI, and Codefresh vs Other CI providers.

GitHub Actions CI config for Ruby on Rails project with MySQL, Redis, Elasticsearch - how to run parallel tests

Fri, 27 Sep 2019 14:00:00 +0000

You will learn how to configure Ruby on Rails project on GitHub Actions. This specific Rails project has MySQL and Redis database. There is also Elasticsearch service running on CI. If your project is close to that setup below GitHub Actions yaml configuration will allow you to run tests on GitHub CI server.

If you happen to use PostgreSQL you can check our previous article about Rails app configuration with Postgres on GitHub Actions.

GitHub Actions configuration for Rails

In your repository, you need to create file .github/workflows/main.yaml Thanks to it GitHub Actions will run your CI builds. You can find results of executed CI builds in Actions tab on your GitHub repository page.

In this case, Rails application has MySQL, Redis, and Elasticsearch databases. You need to set up services with docker container to run each. In the below config, there is also a step for health check the MySQL and Elasticsearch to ensure both are up and running before you can start running tests.

The tests are executed across parallel jobs thanks to matrix feature in GitHub Actions and the Knapsack Pro ruby gem that will auto-balance tests distribution across jobs. Auto balancing tests using Knapsack Pro Queue Mode will ensure each parallel job finish work at a similar time. Thanks to that there is no bottleneck (no slow job with too many tests to run) and you can enjoy fast CI build time because you get optimal tests split across parallel tasks.

How Knapsack Pro Queue Mode works

In this video, you will learn how dynamic test suite spilt across parallel jobs (parallel CI nodes) work.

How Knapsack Pro Regular Mode works

Knapsack Pro has also a deterministic way of splitting tests. Tests are split only once before running tests. This is the most simple way that you will try for the first time to record your CI build before you switch to Queue Mode.

You can learn more about Knapsack Pro in the installation guide.

Summary

If you would like to learn more about GitHub Actions check our previous article about testing Rails app with PostgreSQL on GitHub Actions. There is also a video showing it in action.

Learn how to split slow RSpec spec files by test examples on GitHub Actions.

And if you are still considering using GitHub Actions, then our comparison of GitHub Actions vs other CIs may be helpful for you. The most commonly visited comparisons are: GitHub Actions vs Circle CI, GitHub Actions vs Jenkins, and GitHub Actions vs Travis CI.

How to run RSpec on GitHub Actions for Ruby on Rails app using parallel jobs

Mon, 16 Sep 2019 20:00:00 +0000

GitHub introduced their own CI server solution called GitHub Actions. You will learn how to set up your Ruby on Rails application on GitHub Actions with YAML config file. To run your RSpec test suite faster you will configure parallel jobs with matrix strategy on GitHub Actions.

Automate your workflow on GitHub Actions

GitHub Actions makes it easy to automate all your software workflows with world-class CI/CD. Building, testing, and deploying your code right from GitHub became available with simple YAML configuration.

You can even create a few YAML config files to run a different set of rules on your CI like scheduling daily CI builds. But let’s focus strictly on how to get running tests for Rails app on GitHub Actions.

Setup Ruby on Rails on GitHub Actions with YAML config

In your project repository, you need to create file .github/workflows/main.yaml Thanks to it GitHub will run your CI build. You can find results of CI builds in Actions tab for your GitHub repository.

In our case Rails application has Postgres database so you need to set up service with docker container to run Postgres DB.

# If you need DB like PostgreSQL then define service below.
# Example for Redis can be found here:
# https://github.com/actions/example-services/tree/main/.github/workflows
services:
  postgres:
    image: postgres:10.8
    env:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: ""
      POSTGRES_DB: postgres
    ports:
    # will assign a random free host port
    - 5432/tcp
    # needed because the postgres container does not provide a healthcheck
    options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5

To be able to install pg ruby gem from project Gemfile you will need libpq-dev library in Ubuntu system hence the step to install it. libpq is a set of library functions that allow client programs to pass queries to the PostgreSQL backend server and to receive the results of these queries. We need it to compile pg gem. Next step will be installing our Ruby gems.

# required to compile pg ruby gem
- name: install PostgreSQL client
  run: sudo apt-get install libpq-dev

- name: Build and create DB
  env:
    # use localhost for the host here because we have specified a container for the job.
    # If we were running the job on the VM this would be postgres
    PGHOST: localhost
    PGUSER: postgres
    PGPORT: ${ { job.services.postgres.ports[5432] }} # get randomly assigned published port
    RAILS_ENV: test
  run: |
    gem install bundler
    bundle config path vendor/bundle
    bundle install --jobs 4 --retry 3
    bin/rails db:setup

To run RSpec tests across parallel jobs you need to set up matrix feature and thanks to that run the whole test suite distributed across jobs.

Configuring a build matrix

A build matrix provides different configurations for the virtual environment to test. For instance, a workflow can run a job for more than one supported version of a language, operating system, etc. For each configuration, a copy of the job runs and reports status.

In case of running parallel tests, you want to run the Rails application on the same Ruby version and Ubuntu system. But you want to split RSpec test suite into 2 sets so half of the tests go to a first parallel job and the second half to another job.

To split tests you can use Ruby gem Knapsack Pro that will split tests across parallel GitHub jobs in a dynamic way. Thanks to that each parallel job will be consuming a set of tests fetched from Knapsack Pro API Queue to ensure each parallel job finishes work at a similar time. This allows for evenly distributed tests and no bottleneck in parallel jobs (no slow job). Your CI build will be as fast as possible.

In our case, you split tests across 2 parallel jobs so you need to set 2 as matrix.ci_node_total. Then each parallel job should have assigned index to matrix.ci_node_index starting from 0. The first parallel job gets index 0 and the second job gets index 1. This allows Knapsack Pro to know what tests should be executed on a particular job.

# https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#jobsjob_idstrategymatrix
strategy:
  fail-fast: false
  matrix:
    # Set N number of parallel jobs you want to run tests on.
    # Use higher number if you have slow tests to split them on more parallel jobs.
    # Remember to update ci_node_index below to 0..N-1
    ci_node_total: [2]
    # set N-1 indexes for parallel jobs
    # When you run 2 parallel jobs then first job will have index 0,
    # the second job will have index 1 etc
    ci_node_index: [0, 1]

You need to specify also API token for Knapsack Pro, for RSpec it will be KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC.

Then you can run tests with Knapsack Pro in Queue Mode for RSpec:

bundle exec rake knapsack_pro:queue:rspec

Full GitHub Actions config file for Rails tests

Here you can find the full YAML configuration file for GitHub Actions and Ruby on Rails project.

I also recorded video showing how it all works and how CI builds with parallel jobs are configured on GitHub Actions.

Dynamic test suite split with Queue Mode

If you would like to better understand how Queue Mode works in Knapsack Pro and what else problems it solves you will find a few useful information in below video.

Also check out another article describing the parallelisation setup for GitHub Actions for Rails with MySQL, Redis and Elasticsearch or learn how to auto split slow RSpec test files by test examples.

I hope you find this helpful.

If you are currently considering moving to GitHub Actions, definitely check out our Comparison of GitHub Actions to other CI solutions. The most popular ones include Github Actions vs Circle CI, Github Actions vs Jenkins, and Github Actions vs Travis CI.

How to run parallel jobs for RSpec tests on GitLab CI Pipeline and speed up Ruby & JavaScript testing

Fri, 05 Jul 2019 15:00:00 +0000

GitLab CI allows you to run tests much faster thanks to CI parallelisation feature. You can run parallel jobs across multiple GitLab Runners. In order to do it, you will learn how to split tests in a dynamic way across parallel tasks to ensure there is no bottleneck in GitLab Pipeline. Thanks to that CI build can be run as fast as possible so your Ruby & JS tests can be finely fast.

GitLab CI parallelisation

The common problem, when you want to run tests in parallel to complete your 1-hour test suite in a few minutes instead of waiting hours, is to find a way how to split tests on parallel jobs. Some of your Ruby or JavaScript tests can take milliseconds and some even a few minutes per test file (for instance when using Capybara in RSpec features testing). Problem with slow tests also occurs in E2E (end to end testing) when using Cypress test runner as browser testing can take quite a long time to execute.

If you add more parallel GitLab Runners you also may notice that some runners can start work later or not all jobs can be started at the same time (for instance when you run GitLab Runners on your own infrastructure and other CI builds occupies some of the runners).

Dynamic test suite split to eliminate CI build bottlenecks

A solution to optimal tests distribution across parallel jobs (parallel CI nodes) is to distribute test files in smaller chunks across parallel GitLab runners. This ensures active runners can consume and execute your tests while too busy runners with slow tests would run fewer test cases. What is important is to ensure that all parallel runners will finish work at a similar time and thanks to that you won’t see stuck GitLab runner with too much work to process.

To split tests in a dynamic way for Ruby & JavaScript tests you can use Queue Mode in Knapsack Pro. Below I will explain how Queue Mode works and what problems it solves.

GitLab YAML config for parallel testing

Here you can find an example config .gitlab-ci.yml for Ruby on Rails project that has RSpec tests executed across 2 parallel jobs using Knapsack Pro Queue Mode. The similar configuration would be for JavaScript projects with Jest or Cypress tests (full list of supported test runners in Knapsack Pro can be found here).

Please remember to set API token for Knapsack Pro as environment variable name KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC in GitLab Settings -> CI/CD -> Variables (Expand) as a masked variable.

# .gitlab-ci.yml
image: "ruby:2.6"

services:
  - postgres

variables:
  RAILS_ENV: test
  POSTGRES_DB: database_name
  POSTGRES_USER: postgres
  POSTGRES_PASSWORD: ""
  DATABASE_URL: "postgresql://postgres:postgres@postgres:5432/database_name"
  # KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC: it is set in Settings -> CI/CD -> Variables (Expand) as masked variable

before_script:
  - apt-get update -qq && apt-get install -y -qq nodejs
  - ruby -v
  - which ruby
  - gem install bundler --no-document
  - bundle install --jobs $(nproc)  "${FLAGS[@]}"

  # Database setup
  - bin/rails db:setup

rspec:
  parallel: 2
  script:
    - bundle exec rake knapsack_pro:queue:rspec

Note you can run dozens of parallel jobs by changing parallel option and thanks to that run the very long test suite in a few minutes instead of waiting hour.

Summary

GitLab with its CI/CD tool allows to run fast CI builds thanks to parallelisation of your tests. By using Knapsack Pro Queue Mode you can ensure your tests are split across parallel jobs in an optimal way so your team gets test results as fast as possible.

If you are looking for an optimal CI solution for your project, check out our comparisons: GitLab CI vs Github Actions, GitLab vs Circle CI, Jenkins vs GitLab CI, or GitLab vs other CI providers.

Cucumber testing with Jenkins parallel pipeline to get down CI build time

Sat, 15 Jun 2019 15:00:00 +0000

Cucumber is a popular automation testing tool for Behaviour-Driven Development (BDD) but when you use it for some time in your work project then the amount of automated tests adds up and you can spend dozens of minutes to run your Cucumber test suite. Sometimes complex projects can have a few hours of execution time for the Cucumber tests. To save time and speed up your Cucumber builds on CI (Continuous Integration) you can use CI parallelization. In this article, you will see how to do it for Jenkins using Jenkins parallel pipeline.

How to split Cucumber tests in parallel

Jenkins allows you to configure pipeline as code and use Jenkins pipeline stages to define tasks that will be executed in parallel (at the same time).

In this example, you will use continuous integration tools like cucumber ruby gem and knapsack_pro gem to split tests across parallel Jenkins stages.

When you are testing your Ruby on Rails project with Cucumber you want to ensure tests executed in parallel will finish work at a similar time to fully utilize your CI server resources. As you know some test files can have very little tests and other much more test cases. Depend on the test file’s content the execution time for each test file may vary which makes harder to allocate tests in an optimal way across parallel tasks.

It’s important to split test files from the test suite across each Jenkins parallel stage in a way that you eliminate bottleneck (a slow stage that is running tests longer than other stages). To do that you can split test suite in a dynamic way across parallel stages using Knapsack Pro Queue Mode for Cucumber test runner. In below video you will learn how Queue Mode works:

Jenkins Pipeline parallel Cucumber testing

This is a configuration for Jenkinsfile where you run 4 parallel stages with Cucumber tests using knapsack_pro plugin.

timeout(time: 60, unit: 'MINUTES') {
  node() {
    stage('Checkout') {
      checkout([/* checkout code from git */])

      // determine git commit hash because we need to pass it to knapsack_pro
      COMMIT_HASH = sh(returnStdout: true, script: 'git rev-parse HEAD').trim()

      stash 'source'
    }
  }

  def num_nodes = 4; // define your total number of CI nodes (how many parallel jobs will be executed)
  def nodes = [:]

  for (int i = 0; i < num_nodes; i++) {
    def index = i;
    nodes["ci_node_${i}"] = {
      node() {
        stage('Setup') {
          unstash 'source'
          // other setup steps
        }

        def knapsack_options = """\
            KNAPSACK_PRO_CI_NODE_TOTAL=${num_nodes}\
            KNAPSACK_PRO_CI_NODE_INDEX=${index}\
            KNAPSACK_PRO_COMMIT_HASH=${COMMIT_HASH}\
            KNAPSACK_PRO_BRANCH=${env.BRANCH_NAME}\
        """

        // Example how to run Cucumber tests in Knapsack Pro Queue Mode
        // Queue Mode should be a last stage if you have other stages in your pipeline
        // thanks to that it can autobalance CI build time if other tests were not perfectly distributed
        stage('Run Cucumber tests') {
          sh """KNAPSACK_PRO_CI_NODE_BUILD_ID=${env.BUILD_TAG} ${knapsack_options} bundle exec rake knapsack_pro:queue:cucumber"""
        }
      }
    }
  }

  parallel nodes // run CI nodes in parallel
}

Summary

Testing big projects is a consuming task. Automation testing with Cucumber can help with that. Your CI server like Jenkins can help even more when you can leverage parallel test execution with Jenkins stages and by doing optimal test suite split with Knapsack Pro Queue Mode. The Queue Mode is working also for other Ruby or JavaScript test runners.

Are you considering using Jenkins for your project? If so, check out our popular comparison pages: Jenkins vs Github Actions, Drone vs Jenkins, AWS CodeBuild vs Jenkins, and Cloud Build vs Jenkins.

Cypress parallel testing with Jenkins Pipeline stages

Sun, 02 Jun 2019 10:00:00 +0000

In this tutorial for JavaScript end to end testing, you will learn about Cypress test runner for UI automation testing and how to use it with Jenkins CI server. Cypress helps with frontend automation testing using headless browser or just regular browser. E2E tests often take a long time to run and for bigger projects, those type of tests can take dozens of minutes or even hours. To save developers time you want to load balancing Cypress tests across Jenkins parallel pipeline stages. Thanks to that you can run your 1-hour test suite in a few minutes.

How to speed up Cypress tests

Cypress is a Javascript End to End testing framework that has built in parallelisation but in this article, we will cover Cypress parallel without dashboard service. You want to be able to run tests also when external Cypress dashboard service API is down. It’s possible thanks to Knapsack Pro client for Cypress with built-in Fallback Mode.

Run Cypress tests in parallel with Jenkins Pipeline

You may want to run Cypress concurrent tests but for that, we will use declarative Jenkins Pipeline with defined Jenkins stages. This allows us to run Cypress parallel tests.

Let’s look at this Jenkins Pipeline as a code to understand Cypress Jenkins integration:

timeout(time: 60, unit: 'MINUTES') {
  node() {
    stage('Checkout') {
      checkout([/* checkout code from git */])

      // determine git commit hash because we need to pass it to Knapsack Pro
      COMMIT_HASH = sh(returnStdout: true, script: 'git rev-parse HEAD').trim()

      stash 'source'
    }
  }

  def num_nodes = 4; // define your total number of CI nodes (how many parallel jobs will be executed)
  def nodes = [:]

  for (int i = 0; i < num_nodes; i++) {
    def index = i;
    nodes["ci_node_${i}"] = {
      node() {
        stage('Setup') {
          unstash 'source'
          // other setup steps
        }

        def knapsack_options = """\
            KNAPSACK_PRO_CI_NODE_TOTAL=${num_nodes}\
            KNAPSACK_PRO_CI_NODE_INDEX=${index}\
            KNAPSACK_PRO_COMMIT_HASH=${COMMIT_HASH}\
            KNAPSACK_PRO_BRANCH=${env.BRANCH_NAME}\
        """

        // Example how to run Cypress tests in Knapsack Pro Queue Mode
        // Queue Mode should be a last stage if you have other stages in your pipeline
        // thanks to that it can autobalance CI build time if other tests were not perfectly distributed
        stage('Run Cypress in parallel') {
          sh """KNAPSACK_PRO_CI_NODE_BUILD_ID=${env.BUILD_TAG} ${knapsack_options} npx @knapsack-pro/cypress"""
        }
      }
    }
  }

  parallel nodes // run CI nodes in parallel
}

How dynamic tests split works in Knapsack Pro Queue Mode

To better understand how test suite split works when you allocate tests in a dynamic way across parallel stages please see below video. You will find here more edge cases that can be solved when running tests in parallel and how Knapsack Pro Queue Mode helps with CI parallelisation. Ensuring each of your parallel tasks run a similar amount of time is important to get optimal CI build time and thanks to that save you as much time as possible.

To compare Jenkins with other CI solutions, check out our popular comparison pages: Jenkins vs Github Actions, Drone vs Jenkins, AWS CodeBuild vs Jenkins, Cloud Build vs Jenkins, Jenkins vs Other CI providers.

Split RSpec tests with Jenkins Parallel Pipeline to run specs faster

Sat, 01 Jun 2019 11:00:00 +0000

Jenkins CI server has a declarative pipeline that allows you to set Jenkins parallel stages. You can use the stages to run them at the same time (parallel run) to execute your RSpec test suite in a few smaller faster chunks instead of one long test suite run.

Running stages in parallel with Jenkins workflow pipeline

You will use Jenkinsfile and pipeline syntax to get parallel execution of tasks. RSpec tests need to be split in equal time across stages and to do that you need to ensure the time of each RSpec spec file won’t compound on one of the stages because that could lead to bottleneck - a stage that takes more time to run tests than other stages.

To split RSpec tests evenly you can use knapsack_pro gem with its Queue Mode that will dynamically split specs on parallel Jenkins stages to ensure each stage takes similar amount of time. You can learn more from below video about knapsack_pro Queue Mode and what kind of edge cases it solves when you split tests on CI server.

Stages in Declarative Jenkins Pipeline

Pipeline as Code in Jenkins is a very popular way to define your configuration. Here is example for running RSpec tests with knapsack_pro ruby gem to ensure your Ruby on Rails tests are split in optimal way across parallel stages.

timeout(time: 60, unit: 'MINUTES') {
  node() {
    stage('Checkout') {
      checkout([/* checkout code from git */])

      // determine git commit hash because we need to pass it to knapsack_pro
      COMMIT_HASH = sh(returnStdout: true, script: 'git rev-parse HEAD').trim()

      stash 'source'
    }
  }

  def num_nodes = 4; // define your total number of CI nodes (how many parallel jobs will be executed)
  def nodes = [:]

  for (int i = 0; i < num_nodes; i++) {
    def index = i;
    nodes["ci_node_${i}"] = {
      node() {
        stage('Setup') {
          unstash 'source'
          // other setup steps
        }

        def knapsack_options = """\
            KNAPSACK_PRO_CI_NODE_TOTAL=${num_nodes}\
            KNAPSACK_PRO_CI_NODE_INDEX=${index}\
            KNAPSACK_PRO_COMMIT_HASH=${COMMIT_HASH}\
            KNAPSACK_PRO_BRANCH=${env.BRANCH_NAME}\
        """

        // Example how to run RSpec tests in Knapsack Pro Queue Mode
        // Queue Mode should be a last stage if you have other stages in your pipeline
        // thanks to that it can autobalance CI build time if other tests were not perfectly distributed
        stage('Run rspec') {
          sh """KNAPSACK_PRO_CI_NODE_BUILD_ID=${env.BUILD_TAG} ${knapsack_options} bundle exec rake knapsack_pro:queue:rspec"""
        }
      }
    }
  }

  parallel nodes // run CI nodes in parallel
}

Summary

Automatic splitting of tests to speed up test stages is a way to ensure your CI builds for Ruby on Rails project are finally as fast as possible. You can learn about Jenkins parallel pipeline for other test runners in Ruby or JavaScript like Cypress or Jest and how CI parallelisation can help save time with faster testing.

If you are currently considering using Jenkins among other solutions, here are some comparison pages that may prove useful to you: Jenkins vs Github Actions, Drone vs Jenkins, AWS CodeBuild vs Jenkins, Cloud Build vs Jenkins, Jenkins vs Other CI providers.

How to run tests faster on Heroku CI with parallel dynos

Sat, 25 May 2019 16:00:00 +0000

Heroku provides a CI solution out of the box for teams. Heroku CI can run tests in dyno instance for your project. What is more interesting you can run parallel dynos as part of your CI build. This allows you to split tests on parallel dynos to complete CI build faster and save time.

Heroku charges you for seconds spent in runtime for each dyno. It means instead of running your slow test suite on a single dyno you could split it across multiple dynos and pay more or less the same and significantly reduce the CI build time for your project.

How to start with Heroku CI

If you already have a team in Heroku dashboard then you can use Heroku CI and run tests for your project.

In Heroku, you can open your team and particular pipeline for one of your projects. You can find there a Tests tab where you can enable Heroku CI.

You also need an app.json file in a repository of your project. The file contains information about what is needed to run the project on Heroku. You can add to the app.json file additional configuration needed for Heroku CI.

In order to use Heroku CI Parallel Test Runs, we need to have it enabled. You can ask Heroku support to activate it for your project. This feature allows to run up to 32 parallel dynos for your CI build.

You can also watch all the steps on more detailed video or copy some examples from this article.

Below you find examples how to split tests on Heroku CI for Ruby and JavaScript projects.

How to run parallel dynos for test suite in Ruby on Rails project

You can split Ruby tests written in RSpec, Minitest, Cucumber or other tests runners across parallel dynos in a dynamic way where all dynos will finish work at similar time so you get results about your test suite being green or red as soon as possible. To do it you can use Knapsack Pro tool with its Queue Mode for dynamic test suite split. With quantity key, you can set how many parallel dynos you want to use to run your scripts test command.

{
  "environments": {
    "test": {
      "formation": {
        "test": {
          "quantity": 2
        }
      },
      "addons": [
        "heroku-postgresql:in-dyno",
        "heroku-redis:in-dyno"
      ],
      "scripts": {
        "test": "bundle exec rake knapsack_pro:queue:rspec"
      },
      "env": {
        "KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC": "rspec-token"
      }
    }
  }
}

You can also change dyno size for CI build. If you run complex tests that need more CPU or memory then you could add size parameter to app.json to define dyno type.

{
  "environments": {
    "test": {
      "formation": {
        "test": {
          "quantity": 1,
          "size": "performance-l"
        }
      }
   }
}

Run JavaScript tests across parallel Heroku CI dynos for Cypress E2E test suite

End-to-end tests (E2E) often takes a lot of time because clicking through multiple scenarios of your website is time-consuming. Splitting Cypress test suite on multiple dynos will help us save a lot of time and keep CI build fast.

We can use @knapsack-pro/cypress project for that. It uses Knapsack Pro Queue Mode.

Here is a config for your app.json

{
  "environments": {
    "test": {
      "formation": {
        "test": {
          "quantity": 2
        }
      },
      "addons": [
        "heroku-postgresql:in-dyno",
        "heroku-redis:in-dyno"
      ],
      "scripts": {
        "test": "npx @knapsack-pro/cypress"
      },
      "env": {
        "KNAPSACK_PRO_TEST_SUITE_TOKEN_CYPRESS": "example"
      }
    }
  }
}

If you would like to learn more about Cypress then check the video in an article about running javascript E2E tests with Cypress on parallel CI nodes.

Run Jest tests in JavaScript across parallel Heroku CI dynos

We can use @knapsack-pro/jest client library to split your Jest tests. It uses Knapsack Pro Queue Mode.

Here is a config for your app.json

{
  "environments": {
    "test": {
      "formation": {
        "test": {
          "quantity": 2
        }
      },
      "addons": [
        "heroku-postgresql:in-dyno",
        "heroku-redis:in-dyno"
      ],
      "scripts": {
        "test": "npx @knapsack-pro/jest"
      },
      "env": {
        "KNAPSACK_PRO_TEST_SUITE_TOKEN_JEST": "example"
      }
    }
  }
}

Summary

You can find all integrations for various test runners in installation guide.
See supported test runners in Knapsack Pro.

Heroku CI is a great addition to Heroku. If you already use Heroku you can easily leverage parallel dynos and keep low costs of your CI as you pay only for dyno usage. The great thing is that you can run many parallel CI dynos with a subset of your test suite to save a ton of time for your engineering team. You can learn more about how Knapsack Pro can help you save time with faster CI builds.

Learn more about Queue Mode and what problems it solves in below video.

If you are currently considering a choice between Heroku CI and other CI solutions, check out our comparison pages: Heroku CI vs Circle CI, Github Actions vs Heroku CI, Netlify vs Heroku, and Heroku CI vs Other CIs

CodeClimate and CircleCI 2.0 parallel builds config for RSpec with SimpleCov and JUnit formatter

Fri, 24 May 2019 18:00:00 +0000

How to merge CodeClimate reports for your RSpec test suite executed with parallel builds on CircleCI 2.0? You will learn how to run RSpec parallel tests for your for Ruby on Rails project using CircleCI and how to send test coverage merged from parallel jobs into CodeClimate. We will cover config examples for:

How to use SimpleCov needed by CodeClimate Test Reporter to prepare RSpec test coverage summary on each parallel job and then how to merge it so you will be able to send it to CodeClimate dashboard.
How to use JUnit formatter for RSpec running on parallel jobs. Thanks to JUnit formatter you can see nice tests results in CircleCI UI view. For instance, when your tests fail then CircleCI will show you failing tests at the top of your CI build steps.
How to split your RSpec tests across parallel jobs using Knapsack Pro Queue Mode. At the end of the article, you will see a video explaining how Queue Mode in knapsack_pro ruby gem dynamically distributes specs across parallel jobs to ensure each job takes a similar amount of time to ensure CI build is as fast as possible (to get you optimal CI build time).

CodeClimate and parallel builds on CircleCI 2.0

Let’s start with the full YAML config for CircleCI 2.0. You will find comment for each important step and what it does. You may have already configured some of the stuff in your project so looking at a full example below might be more familiar to you. If you are just adding CodeClimate to your project for the first time then except below config you will need to setup simplecov gem and it’s covered in next section.

Here is the full CircleCI 2.0 example for parallel builds using RSpec and CodeClimate.

# .circleci/config.yml

version: 2
jobs:
  build:
    # Specify the number of parallel jobs to run. Increasing the parallelism speeds your CI build.
    parallelism: 2
    docker:
      # specify the version you desire here
      - image: circleci/ruby:2.6.3-node-browsers
        environment:
          PGHOST: 127.0.0.1
          PGUSER: rails-app-with-knapsack_pro
          RAILS_ENV: test
          RACK_ENV: test

          # API token should be set in CircleCI environment variables settings instead of here
          # KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC: rspec-token

      # Specify service dependencies here if necessary
      # CircleCI maintains a library of pre-built images
      # documented at https://circleci.com/docs/2.0/circleci-images/
      - image: circleci/postgres:9.4.12-alpine
        environment:
          POSTGRES_DB: rails-app-with-knapsack_pro_test
          POSTGRES_PASSWORD: ""
          POSTGRES_USER: rails-app-with-knapsack_pro

    working_directory: ~/repo

    steps:
      - checkout

      - run:
          name: Install Code Climate Test Reporter
          command: |
            curl -L https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-amd64 > ./cc-test-reporter
            chmod +x ./cc-test-reporter
      - run: ./cc-test-reporter before-build

      # Download and cache dependencies
      - restore_cache:
          keys:
          # NOTE: remove space between { { here and in all below examples
          - v2-dependencies-bundler-{ { checksum "Gemfile.lock" }}
          # fallback to using the latest cache if no exact match is found
          - v2-dependencies-bundler-

      - run:
          name: install ruby dependencies
          command: |
            bundle install --jobs=4 --retry=3 --path vendor/bundle

      - save_cache:
          paths:
            - ./vendor/bundle
          # NOTE: remove space between { { here
          key: v2-dependencies-bundler-{ { checksum "Gemfile.lock" }}

      # wait for postgres to be available
      - run: dockerize -wait tcp://localhost:5432 -timeout 1m
      # Database setup
      - run: bin/rails db:setup

      # Run RSpec tests with knapsack_pro Queue Mode and use junit formatter
      # junit formatter must be configured as described in FAQ for knapsack_pro Queue Mode
      # this is also described in this article later
      # https://docs.knapsackpro.com/ruby/circleci/#collect-metadata-in-queue-mode
      - run:
          name: run tests
          command: |
            mkdir -p /tmp/test-results
            bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format RspecJunitFormatter --out /tmp/test-results/rspec.xml]"

      - run:
          name: Code Climate Test Coverage
          command: |
            ./cc-test-reporter format-coverage -t simplecov -o "coverage/codeclimate.$CIRCLE_NODE_INDEX.json"

      # store coverage directory with CodeClimate reports prepared based on simplecov reports
      # it's special step used to persist a temporary file to be used by another job in the workflow
      - persist_to_workspace:
          root: coverage
          paths:
            - codeclimate.*.json

      # store test reports created with junit formatter in order to allow CircleCI
      # show info about executed tests in UI on top of CI build steps
      - store_test_results:
          path: /tmp/test-reports

      # store test reports created with junit formatter in order to allow CircleCI
      # let you browse recorded xml files in Artifacts tab
      - store_artifacts:
          path: /tmp/test-reports

  upload-coverage:
    docker:
      - image: circleci/ruby:2.6.3-node
    environment:
      # you can add your CodeClimate test report ID here or in CircleCI
      # settings for environment variables
      CC_TEST_REPORTER_ID: use-here-your-codeclimate-test-report-id
    working_directory: ~/repo

    steps:
      # This will restore files from persist_to_workspace step
      # Thanks to it we will have access to CodeClimate test coverage files from
      # each parallel job. We need them in order to merge it into one file in next step.
      - attach_workspace:
          at: ~/repo
      - run:
          name: Install Code Climate Test Reporter
          command: |
            curl -L https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-amd64 > ./cc-test-reporter
            chmod +x ./cc-test-reporter
      - run:
          # merge CodeClimate files from each parallel job into sum coverage
          # and then upload it to CodeClimate dashboard
          command: |
            ./cc-test-reporter sum-coverage --output - codeclimate.*.json | ./cc-test-reporter upload-coverage --debug --input -

workflows:
  version: 2

  commit:
    jobs:
      # run our CI build with tests
      - build
      # once CI build is completed then we merge CodeClimate reports
      # from each parallel job and upload summary coverage to CodeClimate
      - upload-coverage:
          requires:
             - build

SimpleCov configuration for RSpec

When you use simplecov with RSpec, you have to set a unique name for its report with SimpleCov.command_name:

# spec/rails_helper.rb or spec/spec_helper.rb

require 'simplecov'
SimpleCov.start

# this is needed when you use knapsack_pro Queue Mode
KnapsackPro::Hooks::Queue.before_queue do
  SimpleCov.command_name("rspec_ci_node_#{KnapsackPro::Config::Env.ci_node_index}")
end

JUnit formatter for RSpec

In order to show in CircleCI UI info about your test suite like failed tests, you need to generate xml report for your RSpec test suite using JUnit formatter.

You can use junit formatter for RSpec thanks to gem rspec_junit_formatter.

# knapsack_pro Queue Mode

bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format RspecJunitFormatter --out /tmp/test-reports/rspec.xml]"

Learn more about collecting metadata in CircleCI in the docs.

Summary and Queue Mode to do dynamic test suite split

CI builds can be much faster thanks to leveraging parallel jobs on Circle CI 2.0 and CI parallelisation on any CI provider (see more parallelisation examples for your CI providers). You can check Knapsack Pro tool for CI parallelisation and learn more about Queue Mode and what problems it solves in below video.

Installation guide for knapsack_pro gem can be found here in order to setup your RSpec test suite.

How to merge CodeClimate reports for parallel jobs (CI nodes)

Wed, 27 Mar 2019 21:00:00 +0000

If you run tests on parallel jobs (using CI parallelisation) you need to merge CodeClimate reports from each parallel job (CI node) into a unified report.

An additional problem may happen that some of your parallel jobs (one of your parallel CI node) may not have reported CodeClimate result when tests were never executed on that CI node.

It could happen when you run tests with dynamic tests allocation across parallel jobs using Knapsack Pro Queue Mode. For instance if once of CI node started running tests after other CI nodes already executed the whole test suite distributed for particular CI build then the node has no CodeClimate report.

Here is a list of steps we need to follow to ensure we can correctly prepare a final report for CodeClimate.

run tests split across parallel jobs (parallel CI nodes) with dynamic test suite split called Knapsack Pro Queue Mode
if one of CI node did not execute tests then knapsack_pro will log info to log file. We should grep the log file and if there were no tests executed then we don’t generate CodeClimate report based on simple-cov
as last step we merge all generated CodeClimate reports

Below is the full example for Semaphore CI 2.0 config but it applies to any other CI provider, just use syntax specifically for your CI tool.

version: v1.0
name: My app
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804

blocks:
  - name: Install dependencies
    task:
      agent:
        machine:
          type: e1-standard-4
          os_image: ubuntu1804

      env_vars:
        - name: RAILS_ENV
          value: "test"

      secrets:
        - name: semaphore_secrets

      jobs:
        - name: Build gems and assets
          commands:
            - checkout
            # ...
            - cache restore $SEMAPHORE_PROJECT_NAME-cc-test-reporter
            - if [ -d 'cc-reporter' ]; then
              echo 'Found cc-reporter in cache';
              else
              mkdir cc-reporter;
              curl -L https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-amd64 > ./cc-reporter/cc-test-reporter;
              chmod +x ./cc-reporter/cc-test-reporter;
              cache store $SEMAPHORE_PROJECT_NAME-cc-test-reporter cc-reporter;
              fi

  - name: Run specs
    task:
      env_vars:
        - name: RAILS_ENV
          value: "test"
        # write knapsack_pro logs to file instead of stdout:
        # log/knapsack_pro_node_0.log
        # log/knapsack_pro_node_1.log
        - name: KNAPSACK_PRO_LOG_DIR
          value: "log"
        # text we expect to find in logs when there was no tests executed for particular parallel job (CI node)
        - name: NO_TEST_EXECUTED_STR
          value: "[knapsack_pro] No test files were executed on this CI node"

      secrets:
        - name: semaphore_secrets

      prologue:
        commands:
          - checkout
          # ...
          - cache restore $SEMAPHORE_PROJECT_NAME-cc-test-reporter
          - mkdir -p coverage/final
          - ./cc-reporter/cc-test-reporter before-build

      # run 2 parallel jobs with Knapsack Pro Queue Mode
      jobs:
        - name: RSpec node 0 - Knapsack Pro
          commands:
            - KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:queue:rspec
            - STR_COUNT=`grep -F "$NO_TEST_EXECUTED_STR" log/knapsack_pro_node_0.log | wc -l | tr -d ' '`
            - if [ $STR_COUNT -eq 0 ]; then ./cc-reporter/cc-test-reporter format-coverage -t simplecov --output "coverage/final/codeclimate.$SEMAPHORE_JOB_ID.json"; fi
            - cache store coverage-0-$SEMAPHORE_WORKFLOW_ID coverage/final

        - name: RSpec node 1 - Knapsack Pro
          commands:
            - KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:queue:rspec
            - STR_COUNT=`grep -F "$NO_TEST_EXECUTED_STR" log/knapsack_pro_node_1.log | wc -l | tr -d ' '`
            - if [ $STR_COUNT -eq 0 ]; then ./cc-reporter/cc-test-reporter format-coverage -t simplecov --output "coverage/final/codeclimate.$SEMAPHORE_JOB_ID.json"; fi
            - cache store coverage-1-$SEMAPHORE_WORKFLOW_ID coverage/final

  - name: Post-test tasks
    task:
      secrets:
        - name: semaphore_secrets

      jobs:
        - name: CodeClimate
          commands:
            - cache restore $SEMAPHORE_PROJECT_NAME-cc-test-reporter
            # we have 2 parallel jobs so 0..1 range
            - for i in {0..1}; do cache restore coverage-$i-$SEMAPHORE_WORKFLOW_ID; done
            - FILES_COUNT=`ls -1 coverage/final | wc -l | tr -d ' '`
            - ./cc-reporter/cc-test-reporter sum-coverage --output - --parts $FILES_COUNT coverage/final/codeclimate.*.json | ./cc-reporter/cc-test-reporter upload-coverage --input -
            - for i in {0..1}; do cache delete coverage-$i-$SEMAPHORE_WORKFLOW_ID; done

Summary and Queue Mode for dynamic test suite split

CI builds can be much faster thanks to leveraging parallel jobs on Semaphore CI 2.0 and CI parallelisation on any CI provider (see more parallelisation examples for your CI providers). You can check Knapsack Pro tool for CI parallelisation and learn more about Queue Mode and what problems it solves in below video.

Run parallel jobs on Semaphore CI 2.0 to get faster CI build time

Sat, 16 Mar 2019 14:00:00 +0000

Semaphore CI 2.0 allows configuring your CI build task with parallel jobs. This way you can run simultaneously a few different commands that do not depend on each other. But we could also use parallel jobs to split your test suite across a few jobs and this way save time. I will show you how to speed up your CI build for Ruby or JavaScript project (Rails / Node project).

With Semaphore CI 2.0 you don’t pay for a reserved amount of containers that can be run in parallel as in some other CI providers. Instead, they count the amount of work time spent on running containers. This creates an incentive to run more parallel jobs to execute our tests fast and still keep bill at a similar level as if we would just run all tests in single container waisting our own time.

Let’s save time with parallel jobs

In order to run parallel jobs with our tests in an optimal way we need to ensure each job will finish work at a similar time. This way there will be no bottleneck like job executing too many tests or too slow tests. The slow job could affect and made our whole CI build slower. Especially end to end tests (E2E) can be very slow and their time execution can vary.

You can split tests across parallel jobs in a dynamic way to ensure all jobs complete work at a similar time using the Knapsack Pro Queue Mode. You can learn more about what else problems can be solved with Queue Mode in the video at the very end of this article but right now let’s jump to the Semaphore CI 2.0 demo example and the config examples we could use.

Here you can find Semaphore CI 2.0 config for projects using:

Ruby on Rails (RSpec, other tests runners like Minitest, Cucumber and so on are also supported)
JavaScript tests in Cypress.io end to end test runner
JavaScript tests in Jest

Ruby on Rails config for Semaphore 2.0

knapsack_pro gem supports environment variables provided by Semaphore CI 2.0 to run your tests. You will have to define a few things in .semaphore/semaphore.yml config file.

You need to set KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC. If you don’t want to commit secrets in yml file then you can follow this guide.
You should create as many parallel jobs as you need with parallelism property. If your test suite is long you should use more parallel jobs.

Below you can find full Semaphore CI 2.0 config for Rails project.

# .semaphore/semaphore.yml
# Use the latest stable version of Semaphore 2.0 YML syntax:
version: v1.0

# Name your pipeline. In case you connect multiple pipelines with promotions,
# the name will help you differentiate between, for example, a CI build phase
# and delivery phases.
name: Demo Rails 5 app

# An agent defines the environment in which your code runs.
# It is a combination of one of available machine types and operating
# system images.
# See https://docs.semaphoreci.com/article/20-machine-types
# and https://docs.semaphoreci.com/article/32-ubuntu-1804-image
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804

# Blocks are the heart of a pipeline and are executed sequentially.
# Each block has a task that defines one or more jobs. Jobs define the
# commands to execute.
# See https://docs.semaphoreci.com/article/62-concepts
blocks:
  - name: Setup
    task:
      env_vars:
        - name: RAILS_ENV
          value: test
      jobs:
        - name: bundle
          commands:
          # Checkout code from Git repository. This step is mandatory if the
          # job is to work with your code.
          # Optionally you may use --use-cache flag to avoid roundtrip to
          # remote repository.
          # See https://docs.semaphoreci.com/article/54-toolbox-reference#libcheckout
          - checkout
          # Restore dependencies from cache.
          # Read about caching: https://docs.semaphoreci.com/article/54-toolbox-reference#cache
          - cache restore gems-$SEMAPHORE_GIT_BRANCH-$(checksum Gemfile.lock),gems-$SEMAPHORE_GIT_BRANCH-,gems-main-
          # Set Ruby version:
          - sem-version ruby 2.6.1
          - bundle install --jobs=4 --retry=3 --path vendor/bundle
          # Store the latest version of dependencies in cache,
          # to be used in next blocks and future workflows:
          - cache store gems-$SEMAPHORE_GIT_BRANCH-$(checksum Gemfile.lock) vendor/bundle

  - name: RSpec tests
    task:
      env_vars:
        - name: RAILS_ENV
          value: test
        - name: PGHOST
          value: 127.0.0.1
        - name: PGUSER
          value: postgres
        - name: KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC
          value: your_api_token_here
      # This block runs two jobs in parallel and they both share common
      # setup steps. We can group them in a prologue.
      # See https://docs.semaphoreci.com/article/50-pipeline-yaml#prologue
      prologue:
        commands:
          - checkout
          - cache restore gems-$SEMAPHORE_GIT_BRANCH-$(checksum Gemfile.lock),gems-$SEMAPHORE_GIT_BRANCH-,gems-main-
          # Start Postgres database service.
          # See https://docs.semaphoreci.com/article/54-toolbox-reference#sem-service
          - sem-service start postgres
          - sem-version ruby 2.6.1
          - bundle install --jobs=4 --retry=3 --path vendor/bundle
          - bundle exec rake db:setup

      jobs:
      - name: Run tests with Knapsack Pro
        parallelism: 2
        commands:
        # Step for RSpec in Queue Mode
        - bundle exec rake knapsack_pro:queue:rspec
        # Step for Cucumber in Queue Mode
        - bundle exec rake knapsack_pro:queue:cucumber

        # Step for RSpec in Regular Mode
        - bundle exec rake knapsack_pro:rspec
        # Step for Cucumber in Regular Mode
        - bundle exec rake knapsack_pro:cucumber
        # Step for Minitest in Regular Mode
        - bundle exec rake knapsack_pro:minitest
        # Step for test-unit in Regular Mode
        - bundle exec rake knapsack_pro:test_unit
        # Step for Spinach in Regular Mode
        - bundle exec rake knapsack_pro:spinach

Cypress.io config for Semaphore 2.0

@knapsack-pro/cypress supports environment variables provided by Semaphore CI 2.0 to run your tests. You will have to define a few things in .semaphore/semaphore.yml config file.

You need to set KNAPSACK_PRO_TEST_SUITE_TOKEN_CYPRESS. If you don’t want to commit secrets in yml file then you can follow this guide.
You should create as many parallel jobs as you need with parallelism property. If your test suite is long you should use more parallel jobs.

Below you can find example part of Semaphore CI 2.0 config.

blocks:
  - name: Cypress tests
    task:
      env_vars:
        - name: KNAPSACK_PRO_TEST_SUITE_TOKEN_CYPRESS
          value: your_api_token_here
      prologue:
        commands:
          - checkout
          - nvm install --lts carbon
          - sem-version node --lts carbon

      jobs:
        - name: Run tests with Knapsack Pro
          parallelism: 2
          commands:
            - npx @knapsack-pro/cypress

Jest config for Semaphore 2.0

@knapsack-pro/jest supports environment variables provided by Semaphore CI 2.0 to run your tests. You will have to define a few things in .semaphore/semaphore.yml config file.

You need to set KNAPSACK_PRO_TEST_SUITE_TOKEN_JEST. If you don’t want to commit secrets in yml file then you can follow this guide.
You should create as many parallel jobs as you need with parallelism property. If your test suite is long you should use more parallel jobs.

Below you can find example part of Semaphore CI 2.0 config.

blocks:
  - name: Cypress tests
    task:
      env_vars:
        - name: KNAPSACK_PRO_TEST_SUITE_TOKEN_JEST
          value: your_api_token_here
      prologue:
        commands:
          - checkout
          - nvm install --lts carbon
          - sem-version node --lts carbon

      jobs:
        - name: Run tests with Knapsack Pro
          parallelism: 2
          commands:
            - npx @knapsack-pro/jest

The Queue Mode and summary

As you can see your CI builds can be much faster thanks to leveraging parallel jobs on Semaphore CI 2.0. You can check Knapsack Pro tool for CI parallelisation and learn more about Queue Mode and what problems it solves in below video.

Handle invalid JSON payload in Rails 5+ API. Catch JSON parse error

Sun, 10 Feb 2019 11:00:00 +0000

When someone post invalid JSON payload to your Ruby on Rails URL endpoint then Rails crashes with 500 error. We could render nice message instead of the error. For instance if you have public API you may want to show reason of the error to explain to your API client why the request failed.

We can catch ActionDispatch::Http::Parameters::ParseError in ApplicationController. If you want to just catch this exception only for your API endpoints exposed by Rails then put the below code in proper API base controller.

module API
  class BaseController < ActionController::Base
    # TODO remove this when a new version of Rails > 5.2.2 will be released
    # https://github.com/rails/rails/issues/34244#issuecomment-433365579
    def process_action(*args)
      super
    rescue ActionDispatch::Http::Parameters::ParseError => exception
      render status: 400, json: { errors: [ exception.message ] }
    end

    # This will work for Rails > 5.2.2
    # https://github.com/rails/rails/pull/34341#issuecomment-434727301
    rescue_from ActionDispatch::Http::Parameters::ParseError do |exception|
      render status: 400, json: { errors: [ exception.cause.message ] }
    end
  end
end

Above tip works for Rails 5+. Hope you will find this useful. If you want to learn more how to faster test your API and Rails application then check out blog and website about knapsack_pro gem for CI parallelisation.

Jenkins Pipeline how to run parallel tests in your workflow stages

Sun, 16 Dec 2018 16:00:00 +0000

Jenkins Pipeline is a suite of plugins that allows creating simple-to-complex build stages for your testing environment on CI. We can use Jenkins Pipeline to run a few stages at the same time and thanks to that parallelize test suite across a few stages to complete tests faster.

In order to run parallel stages with Jenkins Pipeline, you need a proper Jenkinsfile which represents our delivery pipeline as code via the Pipeline DSL.

Another thing we will have to figure out is the problem how to divide our test suite across parallel stages in a way that each subset of test suite executed across all stages will complete work at the same time. It’s important to complete the tests on all stages at a similar time to run our CI build as fast as possible and eliminate bottleneck stage.

How to split test suite evenly across parallel Jenkins stages

To divide our tests across parallel stages we can use Knapsack Pro which allows to dynamically allocate tests across stages (also known as CI nodes). This way we will run our parallelised tests in optimal time.

Here you can learn more how dynamic test suite allocation works and with what else problems it can help.

Jenkinsfile for parallel pipeline

In this example, I will show you how to split your Ruby and JavaScript tests across parallel Jenkins stages.

This is Ruby example how to split Cucumber and RSpec test suite. Other test runners like Minitest, Test::Unit etc are available as well at Knapsack Pro:

timeout(time: 60, unit: 'MINUTES') {
  node() {
    stage('Checkout') {
      checkout([/* checkout code from git */])

      // determine git commit hash because we need to pass it to knapsack_pro
      COMMIT_HASH = sh(returnStdout: true, script: 'git rev-parse HEAD').trim()

      stash 'source'
    }
  }

  def num_nodes = 4; // define your total number of CI nodes (how many parallel jobs will be executed)
  def nodes = [:]

  for (int i = 0; i < num_nodes; i++) {
    def index = i;
    nodes["ci_node_${i}"] = {
      node() {
        stage('Setup') {
          unstash 'source'
          // other setup steps
        }

        def knapsack_options = """\
            KNAPSACK_PRO_CI_NODE_TOTAL=${num_nodes}\
            KNAPSACK_PRO_CI_NODE_INDEX=${index}\
            KNAPSACK_PRO_COMMIT_HASH=${COMMIT_HASH}\
            KNAPSACK_PRO_BRANCH=${env.BRANCH_NAME}\
        """

        // example how to run cucumber tests in Knapsack Pro Regular Mode
        stage('Run cucumber') {
          sh """${knapsack_options} bundle exec rake knapsack_pro:cucumber"""
        }

        // example how to run rspec tests in Knapsack Pro Queue Mode
        // Queue Mode should be as a last stage so it can autobalance build if tests in regular mode were not perfectly distributed
        stage('Run rspec') {
          sh """KNAPSACK_PRO_CI_NODE_BUILD_ID=${env.BUILD_TAG} ${knapsack_options} bundle exec rake knapsack_pro:queue:rspec"""
        }
      }
    }
  }

  parallel nodes // run CI nodes in parallel
}

Here is JavaScript example how to split Cypress tests. You can learn more about splitting E2E tests for Cypress test runner here.

timeout(time: 60, unit: 'MINUTES') {
  node() {
    stage('Checkout') {
      checkout([/* checkout code from git */])

      // determine git commit hash because we need to pass it to Knapsack Pro
      COMMIT_HASH = sh(returnStdout: true, script: 'git rev-parse HEAD').trim()

      stash 'source'
    }
  }

  def num_nodes = 4; // define your total number of CI nodes (how many parallel jobs will be executed)
  def nodes = [:]

  for (int i = 0; i < num_nodes; i++) {
    def index = i;
    nodes["ci_node_${i}"] = {
      node() {
        stage('Setup') {
          unstash 'source'
          // other setup steps
        }

        def knapsack_options = """\
            KNAPSACK_PRO_CI_NODE_TOTAL=${num_nodes}\
            KNAPSACK_PRO_CI_NODE_INDEX=${index}\
            KNAPSACK_PRO_COMMIT_HASH=${COMMIT_HASH}\
            KNAPSACK_PRO_BRANCH=${env.BRANCH_NAME}\
            KNAPSACK_PRO_CI_NODE_BUILD_ID=${env.BUILD_TAG}\
        """

        // example how to run tests with Knapsack Pro
        stage('Run tests') {
          sh """${knapsack_options} npx @knapsack-pro/cypress"""
        }
      }
    }
  }

  parallel nodes // run CI nodes in parallel
}

Summary

Jenkins Pipeline gets you a continuous delivery (CD) pipeline which is an automated expression of your process for getting software from version control right through to your users. It’s important to save the time of your engineering team by running CI build fast. One way of doing it is test suite parallelisation that can be done in an optimal way with Knapsack Pro for Ruby and JavaScript tests.

If you are currently considering the choice between Jenkins and other CI, check out our popular comparison pages: Jenkins vs Github Actions, Drone vs Jenkins, AWS CodeBuild vs Jenkins, and Cloud Build vs Jenkins.

How to run CodeShip Parallel Test Pipelines efficiently - optimal CI parallelization

Sat, 08 Dec 2018 19:00:00 +0000

When you use CodeShip as your CI server you can significantly increase the speed of your CI builds with Parallel Test Pipelines. Pipelines allow you to run multiple commands at the same time, for instance, you can split test suite across a few pipelines and complete the CI build much faster.

How to run parallel commands on CodeShip

Setup via CodeShip interface

One way is to define commands via the CodeShip interface. Once parallel test pipelines are enabled, each project can have multiple test pipelines that run in parallel.

In order to run CI build as fast as possible we need to ensure the parallel commands will run subset of the test suite in a way that all the commands complete at the same time to avoid slow pipeline bottleneck. To split test suite we will use Knapsack Pro with Queue Mode which does dynamic test suite split across pipelines for Ruby and JavaScript tests to keep running our tests in an optimal way across parallel pipelines (also known as CI nodes).

Example for test suite in RSpec for Ruby on Rails project

Configure test pipelines (1/2 used)

# first CI node running in parallel

# RSpec tests in Knapsack Pro Queue Mode (dynamic test suite split)
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:queue:rspec

Configure test pipelines (2/2 used)

# second CI node running in parallel

# RSpec tests in Knapsack Pro Queue Mode (dynamic test suite split)
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:queue:rspec

Example of test suite split for Cypress test runner in JavaScript

Configure test pipelines (1/2 used)

# first CI node running in parallel
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 npx @knapsack-pro/cypress

Configure test pipelines (2/2 used)

# second CI node running in parallel
KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 npx @knapsack-pro/cypress

You can learn more about Cypress test runner for E2E tests in JavaScript in this article.

Setup via codeship-services.yml

If you use CodeShip Pro then a parallel step group is defined by using the type: parallel header and then nesting all steps you want to be parallelized, as seen in this example:

- type: parallel
  steps:
  - service: app
    command: KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 bundle exec rake knapsack_pro:queue:rspec
  - service: app
    command: KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=1 bundle exec rake knapsack_pro:queue:rspec

More examples on how to configure CodeShip Pro parallelism

How dynamic test suite split works

If you would like to better understand how dynamic test suite split works and what problems it can solve check the video:

Summary

Running tests in parallel is a fast way to lower time of your CI build. To make it more efficient we can split test suite in an optimal way across CI nodes with Knapsack Pro to keep CI nodes auto balanced and run CI build as fast as possible.

CircleCI 2.0 cache Ruby gems or npm dependencies

Sat, 08 Dec 2018 12:00:00 +0000

CircleCI 2.0 allows us to cache specific files or folders. We can use that to cache ruby gems installed with bundler and restore them when we will run another CI build. This way new CI build could run faster by using cached files from the previous build. This article shows you how to cache npm dependencies as well.

Cache ruby gems

One of the first steps is to check if an available cache exists v1-dependencies-bundler-{ { checksum "Gemfile.lock" }} for our Gemfile.lock. That’s why we include as part of cache key the checksum of the Gemfile.lock file. This way if you will have different content of Gemfile.lock for different git commits we will use proper cached files when a new CI build would have the same checksum of Gemfile.lock.

When you will install new gems and push a new git commit with updated content of Gemfile.lock then the cache key for that new Gemfile.lock won’t exist. In such case, we want to use fallback cache key matching pattern v1-dependencies-bundler-.

After the cache is restored we can install gems with bundler to a specified directory vendor/bundle and then we can cache it under specific cache key v1-dependencies-bundler-{ { checksum "Gemfile.lock" }}

# .circleci/config.yml
version: 2
jobs:
  build:
    # other config

    steps:
      - checkout

      # Restore bundle cache
      - type: cache-restore
        keys:
          # NOTE: remove space between { { here and in all below examples
          - v1-dependencies-bundler-{ { checksum "Gemfile.lock" }}
          # fallback to using the latest cache if no exact match is found
          - v1-dependencies-bundler-

      # Bundle install dependencies
      - run: bundle install --jobs=4 --retry=3 --path vendor/bundle

      # Store bundle cache
      - type: cache-save
        key: v1-dependencies-bundler-{ { checksum "Gemfile.lock" }}
        paths:
          - vendor/bundle

      # other config

If you would like to see a full example of CircleCI 2.0 config file for Ruby on Rails project check this article.

Cache npm dependencies

The flow for caching npm packages is similar. Here you can see an example.

# .circleci/config.yml
version: 2
jobs:
  build:
    # other config

    steps:
      # other config

      - restore_cache:
          keys:
          - v1-dependencies-{ { checksum "package.json" }}
          # fallback to using the latest cache if no exact match is found
          - v1-dependencies-

      - run: npm install

      - save_cache:
          paths:
            - node_modules
          key: v1-dependencies-{ { checksum "package.json" }}

      # other config

Summary

Caching available for CircleCI 2.0 can be very helpful and helps us cut down the time of CI build. There are also more options to speed up our CI build. One of it is CI parallelization for CircleCI 2.0, you can learn more in the article or check demo on Knapsack Pro.

How to run parallel dynos on Heroku CI to complete tests faster

Sat, 01 Dec 2018 19:00:00 +0000

Heroku provides a CI solution out of the box for teams. They can run tests in dyno instance for your project. What’s more interesting you can run parallel dynos as part of your CI build. This allows you to split tests on parallel dynos to complete CI build faster and save time.

How to start with Heroku CI

If you or your company has already created a team in Heroku then you can use Heroku CI and run tests for your project there.

In Heroku, you can open your team and particular pipeline for one of your projects. You will find there a Tests tab where you can enable Heroku CI.

You will also need an app.json file in a repository of your project. The file contains information about what’s needed to run the project on Heroku. We will add to the app.json file additional configuration needed for Heroku CI.

In order to use Heroku CI parallel test runs, we need to have it enabled. You will have to ask Heroku support to activate it for your project. This feature allows to run up to 32 parallel dynos for your CI build.

You can also watch all the steps on more detailed video or copy some examples from this article.

Below I will show you examples how to split tests on Heroku CI for Ruby and JavaScript projects.

How to run parallel dynos for test suite in Ruby on Rails project

We can split Ruby tests written in RSpec, Minitest or other tests runners across parallel dynos in a dynamic way where all dynos will finish work at similar time so we will get results about our test suite being green or red as soon as possible. To do it we will use Knapsack Pro tool with its Queue Mode for dynamic test suite split. With quantity key, we can set how many parallel dynos we want to use to run our scripts test command.

{
  "environments": {
    "test": {
      "formation": {
        "test": {
          "quantity": 2
        }
      },
      "addons": [
        "heroku-postgresql:in-dyno",
        "heroku-redis:in-dyno"
      ],
      "scripts": {
        "test": "bundle exec rake knapsack_pro:rspec"
      },
      "env": {
        "KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC": "rspec-token"
      }
    }
  }
}

We also can change dyno size for our CI build. If you run complex tests that need more CPU or memory you could add size parameter to app.json to define dyno type.

{
  "environments": {
    "test": {
      "formation": {
        "test": {
          "quantity": 1,
          "size": "performance-l"
        }
      }
   }
}

Run JavaScript tests across parallel Heroku CI dynos for Cypress E2E test suite

We can use @knapsack-pro/cypress project for that. It uses Knapsack Pro Queue Mode.

Here is a config for your app.json

{
  "environments": {
    "test": {
      "formation": {
        "test": {
          "quantity": 2
        }
      },
      "addons": [
        "heroku-postgresql:in-dyno",
        "heroku-redis:in-dyno"
      ],
      "scripts": {
        "test": "npx @knapsack-pro/cypress"
      },
      "env": {
        "KNAPSACK_PRO_TEST_SUITE_TOKEN_CYPRESS": "example"
      }
    }
  }
}

If you would like to learn more about Cypress then check the video in an article about running javascript E2E tests with Cypress on parallel CI nodes.

Summary

Heroku CI is a great addition to Heroku. If you already use Heroku you can easily leverage dynos and keep low costs of your CI as you pay only for dyno usage. The great thing is that you can run many parallel CI dynos with a subset of your test suite to save a ton of time for your engineering team. You can learn more about how Knapsack Pro can help you save time with faster CI builds.

If you’d like to compare Heroku CI with other solutions, check out our comparison pages: Heroku CI vs Circle CI, Github Actions vs Heroku CI, Netlify vs Heroku, and Heroku CI vs Other CIs

How to run Travis CI parallel jobs with build matrix feature fast

Fri, 30 Nov 2018 18:00:00 +0000

Travis CI allows you to run multiple concurrent jobs as part of the same CI build. They even allow for up to 200 parallel jobs for open source projects (the same for private repositories). You can leverage that using Travis build matrix feature to run your project way faster by splitting tests into many smaller jobs that will run a subset of your test suite.

How build matrix feature works?

Build matrix feature allows to automatically create a matrix of all possible combinations of language and environment dependent set of configuration options. For instance, when you want to test your project on 2 different programming language versions and with 2 different browsers then Travis would generate 4 parallel jobs running your tests for each programming language and browser.

Split tests across parallel jobs

We could use build matrix feature to split tests by adding to your project Knapsack Pro that will split your Ruby or Javascript tests automatically across parallel jobs in a way that all concurrent jobs would run subset of test suite and finish work in similar time so you would get the result of your test suite passing or not as fast as possible without waiting for the slowest job.

How to run Ruby tests on parallel jobs with knapsack_pro ruby gem:

# .travis.yml
script:
  # Step for RSpec
  - "bundle exec rake knapsack_pro:rspec"

  # Step for Cucumber
  - "bundle exec rake knapsack_pro:cucumber"

  # Step for Minitest
  - "bundle exec rake knapsack_pro:minitest"

  # Step for test-unit
  - "bundle exec rake knapsack_pro:test_unit"

  # Step for Spinach
  - "bundle exec rake knapsack_pro:spinach"

env:
  global:
    # tokens should be set in travis settings in web interface to avoid expose tokens in build logs
    - KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=rspec-token
    - KNAPSACK_PRO_TEST_SUITE_TOKEN_CUCUMBER=cucumber-token
    - KNAPSACK_PRO_TEST_SUITE_TOKEN_MINITEST=minitest-token
    - KNAPSACK_PRO_TEST_SUITE_TOKEN_TEST_UNIT=test-unit-token
    - KNAPSACK_PRO_TEST_SUITE_TOKEN_SPINACH=spinach-token

    - KNAPSACK_PRO_CI_NODE_TOTAL=2
  jobs:
    - KNAPSACK_PRO_CI_NODE_INDEX=0
    - KNAPSACK_PRO_CI_NODE_INDEX=1

How to run Cypress tests in JavaScript on concurrent jobs with @knapsack-pro/cypress:

# .travis.yml
script:
  - "npx @knapsack-pro/cypress"

env:
  global:
    - KNAPSACK_PRO_CI_NODE_TOTAL=2
    # allows to be able to retry failed tests on one of parallel job (CI node)
    - KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true

  jobs:
    - KNAPSACK_PRO_CI_NODE_INDEX=0
    - KNAPSACK_PRO_CI_NODE_INDEX=1

By doing test suite split in a dynamic way across Travis parallel jobs we save more time and keep our CI build fast. I also call parallel jobs as CI nodes because they are part of a single CI build. Here on the video, I describe a few more problems that can be solved with dynamic test suite split.

Travis CI build matrix in action

The OSS Consul - Open Government and E-Participation Web Software is a great example of how Knapsack Pro helps split tests across parallel jobs.

I've got cup gift for GitHub contribution to #Consul - Open Government and E-Participation Web Software https://t.co/NNIAgO3uXX that empowers https://t.co/eGO3aj0slM The Consul team uses @KnapsackPro to run #ruby tests faster :) Thanks @bertocq @voodoorai2000 pic.twitter.com/8sowbeXlAJ
— Artur Trzop (@ArturTrzop) June 28, 2018

Summary

If you would like to learn more about testing with Knapsack Pro you can check other articles on our blog like testing with Cypress test runner.

In case you are contemplating using Travis CI, then this comparison of Travis CI with other solutions can be helpful to you. The comparison of Travis to Github Actions garners the most interest. Other popular pages include Travis vs AppVeyor and Travis vs Bitbucket Pipelines.

Improve CircleCI parallelisation for RSpec, Minitest, Cypress

Sat, 24 Nov 2018 11:00:00 +0000

Maybe you use CircleCI parallelisation to run your test suite across multiple CI nodes but you noticed that some CI nodes take more time to complete tests than the others.

This can happen when your tests have random time. Often E2E (end-to-end) tests have a more random time of execution because the browser has to wait for some elements to be loaded on the website or maybe your app is depended on external API and processing requests have different duration. Other reason can be a delay with starting one of your parallel CI nodes.

Default CircleCI test suite split

Here is an example of running tests with default CircleCI parallelisation. As you can see the whole CI build finished work in 29 minutes 37 seconds. Tests were executed on 15 parallel CI nodes and some of them run tests for 14 minutes and the slowest one for almost 30 minutes.

If we could auto-balance the split of test suite across CI nodes in a way that all CI nodes do work in similar time then we could get shorter CI build.

Dynamic test suite split

We can split tests in a dynamic way across CI nodes using Knapsack Pro. For instance, we can split tests for RSpec or Minitest in Ruby. If you run E2E tests with Cypress test runner then you can split Javascript tests as well.

Here is graph after adding Knapsack Pro Queue Mode. Knapsack Pro Queue Mode keeps your tests auto-balanced across CI nodes in order to allow all CI nodes finish work in similar time. As you can see CI build took 22 minutes 50 seconds instead of almost 30 minutes. It means we saved 7 minutes per each CI build.

When your team runs 20 CI builds per day you could save 2 hours 20 minutes every day. It’s over 46 hours saved during a month.

In below video I show what else problems can be solved with dynamic test suite split.

If you use RSpec then you can even auto split slow RSpec test files by test examples on parallel jobs thanks to knapsack_pro gem. It means your single slow spec file can run in parallel!

You can also read more about it at Knapsack Pro and see support for more test runners there.

Other tips

If you are curious about Cypress test suite split.

If you would like to run your CircleCI 2.0 tests with Chrome headless.

Tech details how to run tests in Minitest continuously with dynamic test files loading.

If you are looking for an optimal CI solution for your project, check out our comparisons of Circle CI vs other CI providers. Most popular comparisons include Circle CI vs GitHub Actions, Circle CI vs CodeFresh CI, Circle CI vs Gitlab CI, and Circle CI vs Bitbucket Pipelines.

Run javascript E2E tests faster with Cypress on parallel CI nodes

Sat, 17 Nov 2018 18:00:00 +0000

Cypress test runner is a great alternative to Selenium in end-to-end testing space. When it comes to E2E tests they tend to grow with time and running them is slow and becomes time wasting or just coffee break for developers. ;)

To save the time and give quick feedback to developers about CI builds passing or not we could run tests across parallel CI nodes. Many CI providers allow you to set up how many parallel machines can be run within your CI build.

Problem with splitting tests on parallel machines

If you want to run your test suite on multiple CI nodes at the same time you need to decide which test file goes to particular CI nodes.

In a perfect scenario, we would like to run each parallel CI machine equal time. Thanks to that we won’t have to wait for slow CI node that runs too many tests and is the bottleneck for the whole CI build to complete.

The test suite is a living animal, there are constant changes in it, new tests are added or removed. The time of running particular tests can change and sometimes even be random when for instance your tests need to wait for some data to be loaded from external API.

Even CI provider can cause an additional problem. Sometimes boot time of your parallel CI nodes can vary. One CI node starts work later than the others or maybe it’s stuck in the queue due running some other CI build in your organization account.

Some of CI solutions allows you to use a big amount of parallel CI nodes to speed up your tests by leveraging your own server infrastructure like AWS with very cheap preeemted resources - Amazon EC2 Spot Instances (in case of Google, you have Google Cloud Preemptible VMs). The downside of this is when the AWS or Google wants to take away from your the server during the running your tests. Suddenly you lose one of your parallel CI nodes and you have to retry it and it becomes your bottleneck.

How to allocate tests in a dynamic way across parallel CI nodes to save time?

I’ve been working on the parallelization problem for last few years. With help and feedback from many people, we came up with a solution that helps us speed up our test suite thanks to allocating tests across parallel CI nodes in a dynamic way.

Basically, Knapsack Pro orchestrate the parallelization of your test suite. On Knapsack Pro API you have Queue of test files in descending order of test file run duration. Your parallel CI nodes connect with the Queue and fetch set of test files to run them on CI node. The Knapsack Pro API is taking care of allocating the tests in proper order and to remembering the time execution of your test files so we could leverage that in future CI build runs.

Knapsack Pro API can also remember tests assigned to each parallel CI node. When you lost one of parallel CI node during runtime due AWS or Google preempting your machine then you can just retry the killed CI node and run only those tests that were lost. The other parallel CI nodes would consume more tests from Queue before you retry killed CI node so you won’t waste additional time on running tests that were not lost.

Run Cypress tests with Knapsack Pro Queue Mode

To start running your tests faster you can add to your project the @knapsack-pro/cypress package. It works with many CI providers out of the box. Here you can find the installation guide.

Basically, we will run a single command on all parallel CI nodes and Knapsack Pro will take care of running your tests fast.

npx @knapsack-pro/cypress

In below video, I show you an example Cypress project and how to run it on parallel CI nodes.

Hope this will help you. I’d like to hear feedback from you and exchange ideas about how we could run our tests even faster!

Clean RSpec configuration directory structure for Ruby on Rails gems needed in testing

Thu, 08 Nov 2018 11:00:00 +0000

When your Ruby on Rails project is getting bigger your test suite as well. You need to test more of your business logic and sometimes you will use other gems that can help you with that. Most of the time you may need something like database_cleaner, capybara for feature tests or rspec-sidekiq to test your workers.

Adding new gems needed for testing often requires changes in RSpec configuration. You add a new line of config here and there in the spec_helper.rb or rails_helper.rb file and suddenly you have huge and hard to understand config file for RSpec.

I will show you how I organize my RSpec configuration directory structure to easily add or modify the RSpec configuration in a clean way.

Prepare RSpec config directory structure

I keep all of my configuration code related to RSpec in directory spec/support/config. You can create it.

Next step is to ensure the RSpec will read files from the config directory. In order to do it please ensure you have below line in your spec/rails_helper.rb file.

Dir[Rails.root.join('spec/support/**/*.rb')].each { |f| require f }

By default, it is commented out so please uncomment it.

Separate config files for different testing gems

Here I will show you examples of popular gems and how to keep their configuration clean. A few examples are on the video and many more code examples are in this article.

Configuration for Database Cleaner

For database_cleaner gem you can just create config file spec/support/config/database_cleaner.rb:

# spec/support/config/database_cleaner.rb
require 'database_cleaner'

RSpec.configure do |config|
  config.before(:suite) do
    DatabaseCleaner.strategy = :deletion
    DatabaseCleaner.clean_with(:deletion)
  end

  config.around(:each) do |example|
    DatabaseCleaner.cleaning do
      example.run
    end
  end
end

Configuration for Capybara

Here is my configuration of Capybara placed at spec/support/config/capybara.rb.

# spec/support/config/capybara.rb
require 'capybara/rspec'

JS_DRIVER = :selenium_chrome_headless
DEFAULT_MAX_WAIT_TIME = ENV['CI'] ? 5 : 2

Capybara.default_driver = :rack_test
Capybara.javascript_driver = JS_DRIVER
Capybara.default_max_wait_time = DEFAULT_MAX_WAIT_TIME

RSpec.configure do |config|
  config.before(:each) do |example|
    Capybara.current_driver = JS_DRIVER if example.metadata[:js]
    Capybara.current_driver = :selenium if example.metadata[:selenium]
    Capybara.current_driver = :selenium_chrome if example.metadata[:selenium_chrome]

    Capybara.default_max_wait_time = example.metadata[:max_wait_time] if example.metadata[:max_wait_time]
  end

  config.after(:each) do |example|
    Capybara.use_default_driver if example.metadata[:js] || example.metadata[:selenium]
    Capybara.default_max_wait_time = DEFAULT_MAX_WAIT_TIME if example.metadata[:max_wait_time]
  end
end

I have a few custom things here like easy option to switch between different browsers executing my tests by just adding tag like :selenium_chrome to test:

# spec/features/example_feature_spec.rb
it 'something', :selenium_chrome do
  visit '/'
  expect(page).to have_content 'Welcome'
end

You can learn more how to configure Capybara with Chrome headless here.

Configuration for Sidekiq

I like keep my Sidekiq configuration with other useful Sidekiq related gems in one place spec/support/config/sidekiq.rb:

# spec/support/config/sidekiq.rb
require 'rspec-sidekiq'
require 'sidekiq/testing'
require 'sidekiq_unique_jobs/testing'

RSpec.configure do |config|
  config.before(:each) do
    Sidekiq::Worker.clear_all

    # https://github.com/sidekiq/sidekiq/wiki/Testing#testing-worker-queueing-fake
    if RSpec.current_example.metadata[:sidekiq_fake]
      Sidekiq::Testing.fake!
    end

    # https://github.com/sidekiq/sidekiq/wiki/Testing#testing-workers-inline
    if RSpec.current_example.metadata[:sidekiq_inline]
      Sidekiq::Testing.inline!
    end
  end

  config.after(:each) do
    if RSpec.current_example.metadata[:sidekiq_fake] || RSpec.current_example.metadata[:sidekiq_inline]
      Sidekiq::Testing.disable!
    end
  end
end

RSpec::Sidekiq.configure do |config|
  config.warn_when_jobs_not_processed_by_sidekiq = false
end

I use gems like sidekiq-unique-jobs and rspec-sidekiq. Here you can read more about testing Sidekiq.

Configuration for FactoryBot (known as FactoryGirl)

FactoryBot config file for Ruby on Rails can be isolated at spec/support/config/factory_bot.rb:

# spec/support/config/factory_bot.rb
RSpec.configure do |config|
  config.include FactoryBot::Syntax::Methods
end

Configuration for JSON Spec

If you have API endpoints in your application you may like the json_spec gem that can help you test JSON responses. Here is my config at spec/support/config/json_spec.rb

# spec/support/config/json_spec.rb
require 'json_spec'

RSpec.configure do |config|
  config.include JsonSpec::Helpers
end

Configuration for RSpec Retry

Sometimes big projects have painful tests that randomly fail. The last rescue when we cannot make them stable and always green is to retry them a few times before marking them as failed. This is one option how to deal with flaky tests and we can use for that rspec-retry gem.

# spec/support/config/rspec_retry.rb
RSpec.configure do |config|
  # show retry status in spec process
  config.verbose_retry = true
  # show exception that triggers a retry if verbose_retry is set to true
  config.display_try_failure_messages = true

  # run retry only on features
  config.around :each, :js do |ex|
    # retry test 3 times on CI but do not retry when testing locally
    ex.run_with_retry retry: (ENV['CI'] ? 3 : 1)
  end

  # callback to be run between retries
  config.retry_callback = proc do |ex|
    # run some additional clean up task - can be filtered by example metadata
    if ex.metadata[:js]
      Capybara.reset!
    end
  end
end

Here you can learn more about how to deal with flaky tests:

Configuration for Shoulda Matchers

Shoulda Matchers provides RSpec and Minitest-compatible one-liners that test common Rails functionality. Here is my config spec/support/config/shoulda_matchers.rb:

# spec/support/config/shoulda_matchers.rb
require 'shoulda/matchers'

Shoulda::Matchers.configure do |config|
  config.integrate do |with|
    # Choose a test framework:
    with.test_framework :rspec
    #with.test_framework :minitest
    #with.test_framework :minitest_4
    #with.test_framework :test_unit

    # Choose one or more libraries:
    #with.library :active_record
    #with.library :active_model
    #with.library :action_controller
    # Or, choose the following (which implies all of the above):
    with.library :rails
  end
end

Configuration for VCR and WebMock

You can record your requests in testing with VCR or mock request with WebMock. This is my config spec/support/config/vcr.rb:

# spec/support/config/vcr.rb
require 'vcr'

VCR.configure do |config|
  config.cassette_library_dir = 'spec/fixtures/vcr_cassettes'
  config.hook_into :webmock
  config.allow_http_connections_when_no_cassette = true
  config.ignore_hosts(
    'localhost',
    '127.0.0.1',
    '0.0.0.0',
    'example.com',
  )
end

WebMock.disable_net_connect!(allow: [
  'example.com',
])

Configuration for Knapsack Pro to split test suite across parallel CI nodes

If you have a large test suite taking a dozen of minutes or maybe even hours then you may want to run tests in parallel across multiple CI node with Knapsack Pro to get the fastest CI build.

This is example config file.

# spec/support/config/knapsack_pro.rb
require 'knapsack_pro'
KnapsackPro::Adapters::RSpecAdapter.bind

Here you can learn more about how it works.

Configuration for Page Objects

You can keep your page objects in a separate directory. Page objects can be later reused in multiple tests. Create directory spec/support/page_objects. Here is example page object for billing page:

# spec/support/page_objects/billing_page.rb
class BillingPage
  def self.fill_company_details(
    first_name: 'Kayleigh',
    last_name: 'Johnston',
    company: 'Example Company',
    vat_id: 'UK1234567890',
    email: 'kayleigh.johnston@example.com',
    street_address: '59 Botley Road',
    locality: 'Midtown',
    region: '', # can be blank
    postal_code: 'IV27 5LL',
    country_name: 'United Kingdom',
    website: 'http://example.com'
  )
    Capybara.fill_in 'first_name', with: first_name
    Capybara.fill_in 'last_name', with: last_name
    Capybara.fill_in 'company', with: company
    Capybara.fill_in 'vat_id', with: vat_id
    Capybara.fill_in 'email', with: email
    Capybara.fill_in 'street_address', with: street_address
    Capybara.fill_in 'locality', with: locality
    Capybara.fill_in 'region', with: region
    Capybara.fill_in 'postal_code', with: postal_code
    Capybara.select country_name, from: 'country_name'
    Capybara.fill_in 'website', with: website
  end
end

Then you can use the page object in your feature spec.

# spec/features/billing_spec.rb
it 'fills company details on billing page' do
  BillingPage.fill_company_details
end

Configuration for shared examples

You can organize your RSpec shared examples in directory spec/support/shared_examples that will be autoloaded.

Summary

As you can see there are a lot of different useful gems for testing in RSpec. If we would keep all their configuration just in spec_helper.rb we would quickly get a messy file. Separation of config files can help us keep it clean and easy to maintain. Let me know in comments how you keep your config files sane.

How to run tests in Minitest continuously with dynamic test files loading

Sun, 27 May 2018 10:00:00 +0000

Recently I’ve been looking into the source code of Minitest to find out if I can run some tests and then dynamically run another set of tests once the previous run is done. This would allow me to provide dynamically a list of tests to execute on my parallel CI nodes to run CI builds faster.

Something similar exists in RSpec thanks to RSpec::Core::Runner feature that allows running specs multiple times with different runner options in the same process.

In RSpec flow looks like:

require "spec_helper"

RSpec::Core::Runner.run([... some parameters ...])

RSpec.clear_examples

RSpec::Core::Runner.run([... different parameters ...])

As you can see one of the steps is to clear examples with RSpec.clear_examples for the previous run to ensure the executed tests won’t affect the next list of tests we will run.

I was also looking if something similar exists in Minitest to ensure we have a pristine state of test runner before we run another set of test files. I step on Minitest::Runnable.reset method that could do it.

Digging into Minitest source code

I found out that Minitest comes with class method run that runs the loaded test files.

# minitest/lib/minitest.rb
module Minitest
  ##
  # This is the top-level run method. Everything starts from here. It
  # tells each Runnable sub-class to run, and each of those are
  # responsible for doing whatever they do.
  #
  # The overall structure of a run looks like this:
  #
  #   Minitest.autorun
  #     Minitest.run(args)
  #       Minitest.__run(reporter, options)
  #         Runnable.runnables.each
  #           runnable.run(reporter, options)
  #             self.runnable_methods.each
  #               self.run_one_method(self, runnable_method, reporter)
  #                 Minitest.run_one_method(klass, runnable_method)
  #                   klass.new(runnable_method).run

  def self.run args = []

Knowing that I could run tests with it. The first step thou was to ensure we will be able to load test files but I realized at the top of each of test file I have a line like:

require 'test_helper'

and the test_helper.rb file was not found while I attempt to load test file so I had to first add a directory with my tests to load path to make above require work.

# add test directory to load path to make require 'test_helper' work
$LOAD_PATH.unshift('test')

# now load test files
require './test/models/user_test.rb'
require './test/models/article_test.rb'

# if all tests pass we want to exit process with 0 exit code
final_exit_code = 0

# run tests loaded into memory
args  = ['--verbose']
# We need to duplicate the args because the run method will change the Array object.
# We will reuse args later.
tests_passed? = Minitest.run(args.dup)

# now the tests will be executed

# the variable tests_passed? will be true if tests passed. Otherwise would be false
final_exit_code = 1 unless tests_passed?

# Before we run another set of test files we need to reset the test runner state
Minitest::Runnable.reset


# Let's load another set of test files
require './test/controllers/users_controller_test.rb'
require './test/controllers/articles_controller_test.rb'

# we can run new set of test files
tests_passed? = Minitest.run(args.dup)
final_exit_code = 1 unless tests_passed?

# now the second set of test files will be executed

# once the tests files finished run then we can exit process with proper exit code
# 0 - when all tests are green
# 1 - when at least one test failed. Exit code 1 tells our CI provider
#     that process running tests failed.
exit(final_exit_code)

Running Minitest continuously and fetching test files from the Queue in a dynamic way

Digging into the source code of Minitest helped me to find out a way to run my tests in a more efficient way. I applied this to the knapsack_pro gem I’m working on.

How to export test suite timing data from Knapsack Pro API

Sat, 28 Apr 2018 15:00:00 +0000

Knapsack Pro tracks your CI builds and how long each test files took to run on one of parallel CI nodes. Thanks to that Knapsack Pro can prepare the optimal test suite split for your future test suite runs on your CI provider.

You may find useful to export data about your CI builds and timing for your test files in case you would like to analyze it and observe trends how your test files change over time.

How to get a list of all CI builds

You can get the list of all CI builds recorded by Knapsack Pro tool. You need test suite API token. You can get it from user dashboard.

curl -X GET \
  https://api.knapsackpro.com/v1/builds \
  -H 'cache-control: no-cache' \
  -H 'KNAPSACK-PRO-TEST-SUITE-TOKEN: e5311882cbba506223ee8036fa68dc13'

You will get a response from API with the list of CI builds.

{
    "meta": {
        "records_per_page": 100,
        "total_pages": 1,
        "prev_page": null,
        "current_page": 1,
        "next_page": null,
        "is_first_page": true,
        "is_last_page": true,
        "is_page_out_of_range": false
    },
    "links": {
        "first": "https://api.knapsackpro.com/v1/builds?page=1",
        "prev": null,
        "self": "https://api.knapsackpro.com/v1/builds?page=1",
        "next": null,
        "last": "https://api.knapsackpro.com/v1/builds?page=1"
    },
    "data": [
        {
          "id": "651efcce-cc5f-4cfc-b8fa-f49b4e5fb4af",
          "commit_hash": "347f33f598e5c66727e36b6f0c13b034f6a057f0",
          "branch": "main",
          "node_total": 3,
          "created_at": "2018-04-21T10:57:42.439Z",
          "updated_at": "2018-04-21T10:57:54.989Z"
        },
        {
          "id": "ba2190af-1bb1-4e3a-8ce1-37303549a4c3",
          "commit_hash": "347f33f598e5c66727e36b6f0c13b034f6a057f0",
          "branch": "main",
          "node_total": 2,
          "created_at": "2018-04-02T10:00:44.741Z",
          "updated_at": "2018-04-02T10:00:44.763Z"
        }
    ]
}

Note the CI build from Knapsack Pro API perspective means a unique combination of commit_hash, branch, node_total. Even when you run multiple CI builds in your CI provider for the same combination of fields there will be only single CI build on Knapsack Pro API side.

You can use id of the build to fetch detailed info about tests recorded for that CI build by doing request to https://api.knapsackpro.com/v1/builds/:id endpoint. You will learn about it in next section.

You can see summary of GET https://api.knapsackpro.com/v1/builds endpoint here.

Filtering CI build list

You can use optional query params like commit_hash, branch, node_total to filter list of CI builds. See example:

curl -X GET \
  https://api.knapsackpro.com/v1/builds?page=1&branch=main&node_total=2&commit_hash=d1acb81ac1bead703eb6de64d1af24104d5d4b2c \
  -H 'cache-control: no-cache' \
  -H 'KNAPSACK-PRO-TEST-SUITE-TOKEN: e5311882cbba506223ee8036fa68dc13'

How to get the timing of test files for CI build

You can get detailed info about CI build with all build subsets containing the test files recorded timing for that CI build.

curl -X GET \
  https://api.knapsackpro.com/v1/builds/651efcce-cc5f-4cfc-b8fa-f49b4e5fb4af \
  -H 'cache-control: no-cache' \
  -H 'KNAPSACK-PRO-TEST-SUITE-TOKEN: e5311882cbba506223ee8036fa68dc13'

Note the CI build means unique combination of commit_hash, branch, node_total. If you will run your tests multiple times for that combination then you will get multiple recorded build subset records with test files timing for particular CI node indexes.

For instance, you can get a response from Knapsack Pro API:

{
  "id": "651efcce-cc5f-4cfc-b8fa-f49b4e5fb4af",
  "commit_hash": "347f33f598e5c66727e36b6f0c13b034f6a057f0",
  "branch": "main",
  "node_total": 2,
  "created_at": "2018-04-02T10:00:44.741Z",
  "updated_at": "2018-04-02T10:00:44.763Z",
  "build_subsets": [
    {
      "id": "b9f09921-939d-43a6-ae5d-7c95866b4dec",
      "node_index": 0,
      "created_at": "2018-04-21T11:18:34.539Z",
      "updated_at": "2018-04-21T11:18:34.539Z",
      "test_files": [
        {
          "path": "spec/a_spec.rb",
          "time_execution": 0.0008809566497802734
        },
        {
          "path": "spec/b_spec.rb",
          "time_execution": 0.019934892654418945
        }
      ]
    },
    {
      "id": "2fa57fa8-0ff3-4af8-980f-5788c5ab6d21",
      "node_index": 0,
      "created_at": "2018-04-21T11:18:23.750Z",
      "updated_at": "2018-04-21T11:18:23.750Z",
      "test_files": [
        {
          "path": "spec/a_spec.rb",
          "time_execution": 0.011754035949707031
        },
        {
          "path": "spec/b_spec.rb",
          "time_execution": 0.22170114517211914
        }
      ]
    },
    {
      "id": "f7a0bdda-54d6-48b2-965a-4b1a4153043d",
      "node_index": 1,
      "created_at": "2018-04-21T11:18:39.506Z",
      "updated_at": "2018-04-21T11:18:39.506Z",
      "test_files": [
        {
          "path": "spec/c_spec.rb",
          "time_execution": 0.013020992279052734
        },
        {
          "path": "spec/d_spec.rb",
          "time_execution": 0.0174105167388916
        },
        {
          "path": "spec/e_spec.rb",
          "time_execution": 0.23191308975219727
        }
      ]
    },
    {
      "id": "cd424b01-9dfc-4b6d-be5f-e8acbea6c6f3",
      "node_index": 1,
      "created_at": "2018-04-21T11:18:29.156Z",
      "updated_at": "2018-04-21T11:18:29.156Z",
      "test_files": [
        {
          "path": "spec/c_spec.rb",
          "time_execution": 0.011027812957763672
        },
        {
          "path": "spec/d_spec.rb",
          "time_execution": 0.019047812957763879
        },
        {
          "path": "spec/e_spec.rb",
          "time_execution": 0.038027712957363670
        }
      ]
    }
  ]
}

As you can see in the response you have two records for node_index=0 and two for node_index=1 because test suite was executed twice on CI provider.

You can see summary of GET https://api.knapsackpro.com/v1/builds/:id endpoint here.

Hope this helps you to do whatever you like to do with your data. We keep data for last 90 days. If you need the data for a longer period please store them on your side. :)

Manuel, one of our users did great job creating ruby script to fetch data from Knapsack Pro API and calculate popular slow tests across multiple CI builds. Note the script is based on old version of /v1/builds API and needs to be adjusted to work with pagination.

CircleCI 2.0 Capybara feature specs - Selenium webdriver with Chrome headless

Mon, 16 Oct 2017 19:00:00 +0000

I’ve been using Capybara-WebKit for a long time but while switching from CircleCI 1.0 to CircleCI 2.0 I had some problems to use it on the CI.

This triggered to try Chrome Headless with Selenium Webdriver. I will show you how to configure Circle CI 2.0 and your Ruby on Rails project to use capybara/selenium/chrome headless together.

Add capybara and selenium-webdriver

Let’s add capybara and selenium-webdriver gems to Gemfile:

# Gemfile
group :development, :test do
  gem 'capybara'
  gem 'selenium-webdriver'
end

and run bundle install.

If you already had the gems in your Gemfile then ensure you have latest version with bundle update capybara selenium-webdriver.

If you want to make sure Capybara feature specs will work on your development machine:

$ brew install chromedriver

If your feature specs fail then upgrade the driver because you may have installed old one.

$ brew upgrade chromedriver

Configure Capybara

Add config file for Capybara:

# spec/support/config/capybara.rb
JS_DRIVER = :selenium_chrome_headless

Capybara.default_driver = :rack_test
Capybara.javascript_driver = JS_DRIVER
Capybara.default_max_wait_time = 2

RSpec.configure do |config|
  config.before(:each) do |example|
    Capybara.current_driver = JS_DRIVER if example.metadata[:js]
    Capybara.current_driver = :selenium if example.metadata[:selenium]
    Capybara.current_driver = :selenium_chrome if example.metadata[:selenium_chrome]
  end

  config.after(:each) do
    Capybara.use_default_driver
  end
end

Ensure you load config files from spec/support directory:

# spec/rails_helper.rb

# The following line is provided for convenience purposes. It has the downside
# of increasing the boot-up time by auto-requiring all files in the support
# directory. Alternatively, in the individual `*_spec.rb` files, manually
# require only the support files necessary.
Dir[Rails.root.join('spec/support/**/*.rb')].each { |f| require f }

Example feature spec

We can create example feature spec to test if everything works:

# spec/features/home_spec.rb
feature 'Homepage Features' do
  before { visit root_path }

  # it won't run js code but it is fast
  it do
    expect(page).to have_content 'Hello World'
  end

  # it will run js code
  it '', :js do
    expect(page).to have_content 'Hello World'
  end

  # it will open Firefox
  # remove x from xit to run the test in Firefox on your machine to preview
  xit '', :selenium do
    expect(page).to have_content 'Hello World'
  end

  # it will open Chrome
  # remove x from xit to run the test in Chrome on your machine to preview
  xit '', :selenium_chrome do
    expect(page).to have_content 'Hello World'
  end
end

And run tests on your development machine with bin/rspec spec/features/home_spec.rb or bundle exec rspec spec/features/home_spec.rb.

Configure CircleCI 2.0 to run Chrome headless

Here is example .circleci/config.yml:

# .circleci/config.yml
version: 2
jobs:
  build:
    parallelism: 1
    working_directory: ~/project-name
    docker:
      # this is important to use proper image with browsers support
      - image: circleci/ruby:2.4.2-node-browsers
        environment:
          PGHOST: 127.0.0.1
          PGUSER: project-name
          RAILS_ENV: test
      - image: circleci/postgres:9.4.12-alpine
        environment:
          POSTGRES_DB: project-name_test
          POSTGRES_PASSWORD: ""
          POSTGRES_USER: project-name
      - image: redis:3.2.7
    steps:
      - checkout

      # Restore bundle cache
      - type: cache-restore
        # remove space between { {
        key: project-name-{ { checksum "Gemfile.lock" }}

      # Bundle install dependencies
      - run: bundle install --path vendor/bundle

      # Store bundle cache
      - type: cache-save
        # remove space between { {
        key: project-name-{ { checksum "Gemfile.lock" }}
        paths:
          - vendor/bundle

      # Prepare .env, useful if you use dotenv gem
      - run: cp .env.example .env

      # Database setup
      - run: bundle exec rake db:create
      - run: bundle exec rake db:schema:load

      # Run rspec in parallel
      - type: shell
        command: |
          bundle exec rspec --profile 10 \
                            --format RspecJunitFormatter \
                            --out /tmp/test-results/rspec.xml \
                            --format progress

      # Save artifacts
      - type: store_test_results
        path: /tmp/test-results

Speed up your tests with Circle CI parallelisation

If your feature specs are very long you can save some time by running multiple parallel CI nodes. For instance set it to 6 in .circleci/config.yml and use dynamic RSpec specs allocation across CI nodes with knapsack_pro gem and Queue Mode to get optimal test suite split to save as much time as possible.

# .circleci/config.yml
jobs:
  build:
    parallelism: 6

  steps:
    # some tests that are not balanced and executed only on first CI node
    - run: case $CIRCLE_NODE_INDEX in 0) npm test ;; esac

    # auto-balancing CI build time execution to be flat and optimal (as fast as possible).
    # Queue Mode does dynamic tests allocation so the previous not balanced run command won't
    # create a bottleneck on the CI node
    - run:
      name: RSpec via knapsack_pro Queue Mode
      command: |
        # export word is important here!
        export RAILS_ENV=test
        bundle exec rake "knapsack_pro:queue:rspec[--format documentation]"

You can learn how RSpec test suite parallelisation works in 1 minute video.

Summary

Now you are good to push your code to a repository and see how your Capybara feature specs work with Chrome Headless on CircleCI 2.0.

When distributed locks might be helpful in Ruby on Rails application

Mon, 17 Jul 2017 05:00:00 +0000

During this year I noticed 2 similar concurrency problems with my Ruby on Rails application and I solved them with distributed locks. I’m going to show you how to detect if your application might have a concurrency problem and how to solve it.

Let me start with a bit of context before we discuss the problem. I’m running small SaaS application KnapsackPro.com and the application provides API for the gem knapsack_pro. The whole point of the tool is to optimize time execution of your RSpec, Cucumber etc test suite by splitting tests across CI nodes running in parallel.

Imagine a scenario where you have 20 minutes long RSpec test suite and you would like to split it across 2 parallel CI nodes. In the perfect case, you should run half of RSpec tests on the first CI node and the second half on the second CI node. In result, your test suite would run only 10 minutes.

But where is the potential risk of the concurrency problem?

On the Knapsack Pro API side, there is test file queue generated for your CI build. Each CI node periodically requests the Knapsack Pro API via knapsack_pro gem for test files that should be executed next. Thanks to that each CI node will finish tests at the same time.

When both CI nodes start work at the same time to execute your test suite then the knapsack_pro gem does a request to Knapsack Pro API to get the list of test files that should be executed on the particular CI node. The first request coming to the Knapsack Pro API is responsible for creating a test file work queue.

Take a look how code responsible for creating the queue looks like:

def test_files
  create_queue unless queue_exists?

  test_files_from_top_of_the_queue
end

At some point, I started getting emails from a customer that they notice that sometimes their test files are executed twice. At first, I could not reproduce the problem and I was wondering what the root problem might be. When I got the second email from one of the customers I was sure that it was not a single case. I spotted that probably something is wrong with the part of the code you see above. I started asking myself, maybe the checking create_queue unless queue_exists? is not enough?

I knew the production server uses multiple unicorn processes and I was curious what if the checking if the test file queue exists is happening exactly at the same time.

My idea to reproduce this situation was to write a script that would do concurrent requests in separate ruby threads against my API server. I was hoping if the problem is related to concurrency then this way I should be able to reveal it.

I prepared the unicorn configuration that I could use in development to run multiple concurrent unicorn processes.

# bin/api_test/unicorn.rb
worker_processes 5
timeout 40
preload_app true

before_fork do |server, worker|
  Signal.trap 'TERM' do
    puts 'Unicorn main worker intercepting TERM and sending myself QUIT instead'
    Process.kill 'QUIT', Process.pid
  end

  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.connection.disconnect!
end

after_fork do |server, worker|
  Signal.trap 'TERM' do
    puts 'Unicorn worker intercepting TERM and doing nothing. Wait for main worker to send QUIT'
  end

  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.establish_connection
end

Below is my script to run concurrent requests. As you can see I use RSpec here to do expect and ensure there is no problem with API. When the test files in the work queue are duplicated then the test fails and that means the concurrency problem exists and the test file work queue is created twice instead of just once.

#!/usr/bin/env ruby
# bin/api_test/initialize_queue
require 'knapsack_pro'
require 'rspec'

# Start rails server in development with unicorn
# to test concurrent requests
#  $  bundle exec unicorn -p 3000 -c bin/api_test/unicorn.rb
#
# Run this file with rspec
#  $ rspec bin/api_test/initialize_queue

node_total = 5

# use development API
ENV['KNAPSACK_PRO_ENDPOINT'] = 'http://api.knapsackpro.localhost:3000'
ENV['KNAPSACK_PRO_TEST_SUITE_TOKEN'] = '333e7b8d1b64fd6447df34a77e3662eb'
ENV['KNAPSACK_PRO_LOG_LEVEL'] = 'warn'

test_files = [
  {"path"=>"spec/bar_spec.rb"},
  {"path"=>"spec/controllers/articles_controller_spec.rb"},
  {"path"=>"spec/controllers/welcome_controller_spec.rb"},
  {"path"=>"spec/dir with spaces/foobar_spec.rb"},
  {"path"=>"spec/features/calculator_spec.rb"},
  {"path"=>"spec/features/homepage_spec.rb"},
  {"path"=>"spec/foo_spec.rb"},
  {"path"=>"spec/services/calculator_spec.rb"},
  {"path"=>"spec/services/meme_spec.rb"},
  {"path"=>"spec/timecop_spec.rb"},
  {"path"=>"spec/vcr_spec.rb"}
]
expected_test_files = KnapsackPro::TestFilePresenter.paths(test_files).sort

def test_files_from_queue(can_initialize_queue, commit_hash, branch, node_total, node_index, test_files)
  action = KnapsackPro::Client::API::V1::Queues.queue(
    can_initialize_queue: can_initialize_queue,
    commit_hash: commit_hash,
    branch: branch,
    node_total: node_total,
    node_index: node_index,
    node_build_id: 'missing-build-id',
    test_files: test_files,
  )
  connection = KnapsackPro::Client::Connection.new(action)
  response = connection.call
  if connection.success?
    raise ArgumentError.new(response) if connection.errors?
    KnapsackPro::TestFilePresenter.paths(response['test_files'])
  else
    raise ArgumentError.new("Couldn't connect with Knapsack Pro API. Response: #{response}")
  end
end


commit_hash = SecureRandom.hex
all_test_files = []

threads = []

node_total.times do |node_index|
  threads << Thread.new do
    can_initialize_queue = true
    node_all_test_files = []
    while true
      node_subset_test_files = test_files_from_queue(
        can_initialize_queue,
        commit_hash,
        'api_test',
        node_total,
        node_index,
        test_files
      )
      node_all_test_files += node_subset_test_files
      can_initialize_queue = false

      puts
      puts "CI node: #{node_index}"
      puts node_subset_test_files.inspect

      break if node_subset_test_files.empty?
    end
    node_all_test_files
  end
end

threads.each do |thr|
  all_test_files += thr.join.value
end

describe 'Ensure queue API returns all test files without duplicates' do
  it do
    expect(all_test_files.sort).to eq expected_test_files
  end
end

This way I was able to reproduce the problem in development and I had a script thanks to that I could verify if a future fix will be working.

How to deal with concurrency problem

The first thing that came to my mind was that maybe I should write myself some sort of solution. The test file work queue is stored in Redis so I was wondering maybe I could do something on the Redis level to ensure the work queue is created only once.

I quickly realized that none of my ideas sound reasonable to solve the problem then I looked for options how people deal with the concurrency problem.

Basically, the conclusion after my research was:

don’t reinvent the wheel
look for proven solutions because distributed problems are complex

and I found that distributed lock can help to ensure the part of my code will be execute only once at a time.

Distributed locking for the rescue

What distributed lock does?

Simply speaking it synchronize access to shared resources in our case the code responsible for creating the test file work queue. It means different processes (unicorn processes) must operate with shared resources in a mutually exclusive way.

Why you want a lock in a distributed application?

There are 2 main reasons why you want to introduce lock:

Efficiency - because you want to avoid expensive computation and save time and money
Correctness - you would like to prevent data loss or data corruption. You want to avoid data inconsistency.

I found a few gems like redis-mutex, redis-lock, redlock-rb, redis-semaphore. While I was reading about all of those gems and checking their issues and pull requests I learned that distributed problems are even more complex than I thought at first.

The most reasonable tool seemed to be redis-semaphore gem so I picked that. Here is the sample solution I did.

def test_files
  semaphore_name = :ci_build_id # unique ID of CI build
  expire_lock_after = 5 # seconds

  semaphore = Redis::Semaphore.new(semaphore_name, host: "localhost")
  semaphore.lock(expire_lock_after) do
    create_queue unless queue_exists?
  end

  test_files_from_top_of_the_queue
end

As you can see I define semaphore name based on CI build ID because in my business context the concurrency error happened only for the CI build so the ID of the build seems to be the perfect candidate to use it as a semaphore name.

I also set lock timeout to 5 seconds because I assume in most cases creating the work queue takes milliseconds and if something goes wrong and suddenly it will take seconds then it would be better to expire lock and allow the app to fail rather than lock unicorn process forever.

When I had the working fix then I validated if it actually solves the problem by using my script to do concurrent requests in ruby threads. Indeed it solved the problem!

Concurrency problem you most likely have too

A month ago or so I found another concurrency issue in my API and the problematic code looked like:

def save
  build = find_build || new_build # similar to find_or_initialize_by
  do_something_complex_with_build
  build.save
end

This may look familiar to you because it’s similar to what Rails method find_or_initialize_by does. If you have code like that in your codebase then ask yourself what if my code will be executed twice at the same time? Will this cause any problem?

What should you remember?

Locks are hard
Distributed locks are even harder
Don’t reinvent the wheel, use proven solutions
Most web apps are not thread-safe due to missing locks
Expect edge cases while you grow

Final tips

Be aware of trade-off. Do you care about efficiency or correctness?
Test your endpoints with concurrent requests to reproduce problem and to validate fix will work
Remember to use transactions when changing many records but don’t forget DB transactions won’t help with a concurrency problem due to the fact transaction works per DB connection.
If possible use unique index to ensure data consistency in DB. When concurrency problem will happen then at least DB will ensure data correctness.
Use tested distribution lock solutions

I also recommend reading the following article: Differences between transactions and locking.

Auto balancing 7 hours tests between 100 parallel jobs on CI - Buildkite example

Sun, 26 Mar 2017 11:20:00 +0000

I like tests. You may like them too because thanks to them you are not afraid to change the code. Tests can help you catch errors earlier. I like tests, even more, when they are fast.

In large projects tests tend to get bigger and bigger, your test suite starts taking dozens of minutes, then hours. We want to have feedback loop during development and testing short so with big test suites, we go for parallelism.

Test suite parallelism

We can split tests across multiple CI nodes to get faster feedback. There are a few ways how to do it, most common are:

Simple methods of distributing tests across CI nodes:

Split tests based on type. For instance, we run unit tests on first CI node and feature tests on second CI node.
Split tests based on directory or file names, or number of test files

Advanced methods of distributing tests across CI nodes:

Split tests based on time execution and running predetermine subset of tests per CI node.

This way we should get similar time execution on each CI node. But there are cons of this approach. Sometimes tests took different time because of bad CI node performance or the nature of the test like testing external API or feature tests clicking on the website that could take random time.
Dynamic tests split is a way based on recorded tests time execution and work queue.

Each CI node gets subsets of the test suite from the work queue until the queue is empty. This way we have faster, more efficient tests distribution across CI nodes. There is no bottleneck when one of our CI nodes has worse performance. It will just do less work than others but in the end, we developers get feedback about test suite passing or not as soon as it is possible.

The story of knapsack

In 2014 I started working on predetermine test suite split solution based on tests time execution. In 2015 I developed more advanced solution called Regular Mode built into the gem knapsack_pro to track tests time execution across commits and branches.

In late 2016 I created the early version of dynamic test suite split solution based on tests time execution and work queue. Early 2017 I started testing it on large projects with a few companies. It happens the largest projects were using the Buildkite.com CI provider. I’m going to show you why and how to run insanely fast test suite there.

What is Buildkite.com

Bulidkite gives you separation between CI web interface and the build infrastructure. Basically, you can run your tests on any machine with an installed buildkite agent. It can be cloud like AWS or your VPS or even your Mac or Windows. You can save a lot of time of booting your CI nodes with Buildkite because you can keep artifacts like installed gems, migrated DB etc on your machine unlike other CI provider do by uploading them to external store and booting CI node from scratch with every new run.

Another nice thing about Buildkite is the fact that buildkite agent (CI node) starts work as soon as something is available. There are no locked CI nodes that just finished subset of CI build and waiting until whole CI build is completed.

There is one more useful thing which is retry failed CI node feature. You can just retry only single CI node with failed tests instead of scheduling a completely new build and rerunning what already passed.

How to run 7 hours build across 100 parallel jobs with knapsack_pro gem?

We are going to use knapsack_pro gem with Queue Mode for RSpec. You can find here info how to add it to your Gemfile and install knapsack_pro for your project.

Knapsack Pro supports buildkite environment variables to determine git commit, branch name and number of parallel jobs. The only thing you need to do is to configure the parallelism parameter in your pipeline step and run the appropriate knapsack_pro command.

Create pipeline:

If you have an account in Buildkite then you need to create a new pipeline for your project.

You will have to provide a project name and git repository for the code checkout.

In environment variables section you should set:

RACK_ENV=test
RAILS_ENV=test
KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true

KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true flag allows us to use retry failed CI node feature on Buildkite.com. When we retry the CI node then we want to run what was run there previously instead of dynamically allocate tests again hence the fixed queue split flag is true.

Set step command:

Commands to run: bundle install && bundle exec rake db:reset && bundle exec rake knapsack_pro:queue:rspec
Label: RSpec in Queue Mode
Environment Variables: KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=462c48d886ab38ddcdb81d379379e639
Parallelism: 100

In environment variables, we set Knapsack Pro API key KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC per step command. You can obtain API key here.

When everything is filled you can save your pipeline and run your buildkite agents across your CI nodes. Push a new commit to your repository and allow it to pass. The first CI build run will record time execution of your tests and saves it to Knapsack Pro API.

Go to user dashboard and click build metrics link next to your API token. Click show link on the recent build and ensure the time execution data were recorded for all your CI nodes. You should see info that build subsets were collected.

From now on you can run your tests with optimal test suite split thanks to dynamic tests allocation across CI nodes. The second commit pushed to repo should have better auto balancing time because Knapsack Pro API can use time execution recorded in the previous run to prepare better work queue.

I tested this with a company who has a large test suite ~7 hours and the average time execution per CI node was ~4 minutes. That is pretty nice and insanely fast pipeline for such large test suite. I’m curious how it will work with your project.

What else can I learn?

See an example repository of how to run Rails CI with Knapsack Pro and test steps in parallel with Buildkite. Here is an example for Docker, Knapsack Pro and Buildkite.
What is optimal order of test commands?
Learn more about Knapsack Pro parallelisation
Compare Buildkite to other CI solutions, including Buildkite vs Jenkins, Buildkite vs CircleCI, and Buildkite vs GitHub Actions.

Blog about testing & documentation for KnapsackPro.com

Have you ever used Ruby's Object#method?

The basics

The ampersand operator

Ampersand with the Method object

So when is this useful?

Summary

Estimate database connections pool size for Rails application

Why do available database connections matter?

RoR application configuration step by step

Postgres database connections - how to check the limit?

ActiveRecord connection pool

How does ActiveRecord connection pool affects Postgres max connections?

Puma config: 1 process and 1 thread

Puma config: 1 process and 2 threads

Puma config: 2 processes and 2 threads per process

Puma config: 2 processes and 2 threads, and 2 web dynos on Heroku

Autoscaling web application

What else can open DB connections?

Background worker - Sidekiq and ActiveRecord pool

Sidekiq and Redis database connections

Redis database connections

Summary

Querying single columns in Rails Active Record using the #pluck method

The use case

Using the #pluck

Retrieving data from multiple columns

Summary

How BitBucket Pipeline with parallel Cypress tests can speed up CI build

How to run tests in parallel

Queue Mode - a dynamic tests split

BitBucket Pipeline YML config

Summary

Using Ruby's 2.7 new #tally method

The use case

Pre Ruby 2.7

The new way

Summary

How to run Ruby on Rails tests on Github Actions using RSpec

Github Actions YML config for Rails application

ruby/setup-ruby action

How to configure Postgres on Github Actions

How to configure Redis on Github Actions

How to use Github Actions build matrix to run tests with parallel jobs

Full YML config for Github Actions and Ruby on Rails project

Summary

Run Minitest on Github Actions with parallel jobs using build matrix

Split tests in a dynamic way with Queue Mode

Github Actions build matrix to run parallel tests

Summary

Fix: 'warning: Using the last argument as keyword parameters is deprecated'

The explanation

So how to fix it?

The simple solution

Not using the Hash

Avoiding the shortcuts

Conclusion

How and when to use Ruby case statements

Expressing logic simply

Patterns in the case statements

Summary

How to load Ruby gems from cache on Github Actions

actions/cache - just cache dependencies

ruby/setup-ruby - install Ruby and cache gems

Summary

Faster Cypress + RSpec test suite for Rails apps on GitHub Actions using Knapsack Pro

Set up Knapsack Pro API Keys

Set up your GitHub Actions config file

Existing GitHub Actions config

New Github Actions config file

Add Knapsack Pro gem and npm package

Run your tests & view your results

Related resources

Auto-scaling Buildkite CI build agents for RSpec (run parallel jobs in minutes instead of hours)

A real RSpec test suite taking 13 hours and executed in only 5 minutes

How to distribute test files between parallel jobs using Queue Mode in Knapsack Pro to utilize CI machines optimally

A simple example of CI Buildkite parallelism config

An advanced Buildkite config with Elastic CI Stack for AWS

AWS Spot Instances can save you money

Buildkite retry feature