How to merge SimpleCov results with parallel Rails specs on Semaphore CI
If you are running Rails specs on parallel machines with Knapsack Pro, one challenge you will run into is combining the code coverage results generated by SimpleCov.
This post will show you how to generate a report of the total combined code coverage after all the tests have executed. Here’s what the pipeline diagram looks like at a high level with unrelated sections blurred out:
Configure
SimpleCov
First lets look at the config file used for SimpleCov below. Note no minimum_coverage
configuration for failing the build. This is because each node most likely will not meet the minimum coverage threshold on its own so it could lead to the build failing erroneously.
Also note, before_queue
hook for Knapsack Pro. This is the important piece, it will set a command name based on the CI node index so that the results are recorded against it.
.simplecov
:
So now when SimpleCov creates a .resultset.json
it will have a specific key depending on which CI node it was run in like the example below. This will be useful down the line when it comes to combining the results.
Semaphore CI
Below is the relevant portions of the Semaphore CI configuration. It runs the Rails tests and then uploads the coverage results as a Semaphore workflow artifact. After all the parallel tests have completed, it will run a job to collate the coverage results from all the machines.
.semaphore/semaphore.yml
:
Below is the bash file which executes the Rails tests on each parallel machine. It sets up the Rails environment and then runs Knapsack Pro in Queue Mode.
.semaphore/helpers/rails_tests.sh
:
If you want you can also learn how to use RspecJunitFormatter with KnapsackPro Queue Mode.
This is the bash file which is responsible for uploading the SimpleCov results from each machine. It compresses the coverage directory and uploads it to Semaphore.
.semaphore/helpers/upload_test_artifacts.sh
:
Lastly, this is the bash file for collating all the results from Semaphore. It will download the coverage artifacts from each parallel machine and run a rake task which will collate them and then upload the results into a combined total_coverage.tgz
file as shown below:
.semaphore/helpers/calc_code_coverage.sh
:
This coverage:report
rake task will simply call SimpleCov.collate
which will go through the coverage results in each folder and combine them into a single .resultset.json
shown below
lib/task/coverage_report.rake
:
.resultset.json
:
Summary
Finally here’s what your Semaphore workflow artifacts will look like. It will have a compressed coverage file generated on each machine and a total coverage file that we created at the very end:
This approach can also be easily ported over to other CI providers by simply changing the artifact push
and artifact pull
commands to S3 or another CI specific artifact upload command.
I hope this article was useful to you. Let me know if you have any questions or feedback. If you liked this post, please share it on social media and follow me on Twitter!