How to merge Cypress test reports generated by Mochawesome on Github Actions
Looking through CI logs to find out which of your Cypress tests failed can be time consuming and error-prone, especially when your tests are spread out across several machines. With the Mochawesome test report generator and Github Actions, you can easily generate a beautiful, easy-to-read report of your test run, which merges results from all parallel nodes.
Cypress Setup
First, add Mochawesome as a dependency of your project:
yarn install --save-dev mochawesome
or
npm install --save-dev mochawesome
.
Next, tell Cypress to use Mochawesome by adding the following to your cypress.json
:
{
"reporter": {
"reporterOptions": {
"overwrite": false,
"html": false,
"json": true
}
}
}
Now, when you run a Cypress test suite, a .json
test report file will be generated for each .spec
file ran, inside the mochawesome-report
directory. Since Knapsack Pro runs tests one at a time, we need "overwrite": false
to ensure that previous test reports are kept during subsequent test runs.
Note that we wrote "html": false
and "json": true
. Since we’re running tests one at a time, generating an HTML report now will be useless, since it will only show the results of one spec file. These options tell Mochawesome to only generate a JSON report. After all the tests are done, we will merge all the JSON reports together, then generate an HTML report, so that we have one report with all of our test results.
Github Actions Workflow
Now we can configure our Github Actions workflow. The first job will run our Cypress tests with Knapsack Pro:
test_e2e:
...
strategy:
fail-fast: false
matrix:
node-version: [10.16.2]
# Knapsack Pro parallelization
ci_node_total: [4]
ci_node_index: [0, 1, 2, 3]
steps:
...
- name: Configure Knapsack Pro
run: |
echo ::set-env name=KNAPSACK_PRO_TEST_SUITE_TOKEN_CYPRESS::${{ secrets.KNAPSACK_PRO_TEST_SUITE_TOKEN_CYPRESS }}
echo ::set-env name=KNAPSACK_PRO_TEST_FILE_PATTERN::"cypress/**/*.spec.{js,ts}"
echo ::set-env name=KNAPSACK_PRO_CI_NODE_TOTAL::${{ matrix.ci_node_total }}
echo ::set-env name=KNAPSACK_PRO_CI_NODE_INDEX::${{ matrix.ci_node_index }}
In this case, I am using 4 parallel CI nodes, but you should modify this accordingly if you’re using more or less. We will use the $KNAPSACK_PRO_CI_NODE_INDEX
environment variable later, as this tells us the index of the currently running parallel node. Since I’m using 4 machines, this will be a number from 0 - 3.
Now we can run knapsack-pro-cypress
as usual:
- name: Run E2E tests
run: knapsack-pro-cypress
- name: Upload E2E test reports
if: always()
uses: actions/upload-artifact@master
with:
name: test-reports-${{ env.KNAPSACK_PRO_CI_NODE_INDEX }}
path: mochawesome-report
Running the tests will create the mochawesome-report
directory, containing all our individual test reports. To pass these along to the next job, which will take care of merging them, we use Github’s upload-artifact
action. Note that we provide the condition if: always()
, to ensure that test reports will always be uploaded, even if some of our tests failed.
When naming the artifact, we’re using the node index environment variable from earlier. When running on 4 machines for example, this means that 4 artifacts will be uploaded: test-reports-0
, test-reports-1
, test-reports-2
, test-reports-3
.
The next job, which runs after all the parallel tests finish, will download the test reports, merge all of them, and upload a final HTML report.
gen_report:
name: Generate test report
needs: test_e2e
if: always()
runs-on: ubuntu-latest
steps:
- name: Install dependencies
run: npm install mochawesome-merge mochawesome-report-generator
The if: always()
ensures that reports will be generated even if any tests fail. Next, we install the dependencies needed to merge and generate the report.
Now we need to download all our test artifacts, which were uploaded as test-reports-0
, test-reports-1
, etc. Unfortunately, since Github Actions doesn’t support YAML anchors, this part requires a bit of copy and paste. Just like before, tweak this part depending on how many parallel nodes you’re using:
- name: Create reports directory
run: mkdir reports
- name: Download test-reports-0
uses: actions/download-artifact@master
with:
name: test-reports-0
path: reports/0
- name: Download test-reports-1
uses: actions/download-artifact@master
with:
name: test-reports-1
path: reports/1
- name: Download test-reports-2
uses: actions/download-artifact@master
with:
name: test-reports-2
path: reports/2
- name: Download test-reports-3
uses: actions/download-artifact@master
with:
name: test-reports-3
path: reports/3
Now we have a reports
directory, with subdirectories containing reports from each node. Our directory structure will now look like this:
❯ tree reports
reports
├── 0
│ ├── mochawesome_001.json
│ ├── mochawesome_002.json
│ ├── mochawesome_003.json
│ ├── mochawesome_004.json
│ ├── mochawesome_005.json
│ └── mochawesome.json
├── 1
│ ├── mochawesome_001.json
│ ├── mochawesome_002.json
│ ├── mochawesome_003.json
│ └── mochawesome.json
├── 2
│ ├── mochawesome_001.json
│ ├── mochawesome_002.json
│ ├── mochawesome_003.json
│ └── mochawesome.json
└── 3
├── mochawesome_001.json
├── mochawesome_002.json
├── mochawesome_003.json
└── mochawesome.json
4 directories, 18 files
To merge these reports, we want to have them all in one directory. The next step is a short Bash script:
- name: Move all reports into one directory
run: |
mkdir -p mochawesome-report
i=0
for file in $(find reports -type f -name mochawesome\*.json); do
filename=$(basename "$file" .json)-"$i".json
mv "$file" mochawesome-report/"$filename"
ls mochawesome-report
i=$((i + 1))
done
This script finds all the .json
report files and moves them to the mochawesome-report
folder, which will look like this:
❯ tree mochawesome-report
mochawesome-report
├── mochawesome_001-10.json
├── mochawesome_001-16.json
├── mochawesome_001-2.json
├── mochawesome_001-6.json
├── mochawesome_002-15.json
├── mochawesome_002-1.json
├── mochawesome_002-5.json
├── mochawesome_002-9.json
├── mochawesome_003-0.json
├── mochawesome_003-14.json
├── mochawesome_003-4.json
├── mochawesome_003-8.json
├── mochawesome_004-13.json
├── mochawesome_005-12.json
├── mochawesome-11.json
├── mochawesome-17.json
├── mochawesome-3.json
└── mochawesome-7.json
0 directories, 18 files
Note that a number is appended to the end of each filename. This is only to prevent filename conflicts when we’re moving all the reports into one directory. The name of the file won’t have any affect on the generated report.
Now we can actually generate the final report:
- name: Merge and generate reports
run: |
mkdir report
npx mochawesome-merge > report/report.json
npx marge --inline report/report.json -o report/
We ran these commands to:
mkdir report
: Create the directory that will hold the final reportnpx mochawesome-merge > report/report.json
: Merge all of our.json
files from themochawesome-report
directory into one file,report/report.json
npx marge --inline report/report.json -o report/
: Take the merged report and generate a pretty HTML report from it
Now we upload this report:
- name: Upload report
uses: actions/upload-artifact@master
with:
name: E2E Test Reports
path: report/
Now you will be able to download your report from the build artifacts:
If you want to learn more about how to test your Javascript applications faster then check out the blog and website about the Knapsack Pro tool for CI parallelisation.