Skip to main content

Retry only Failures

A substantial percentage of your CI retries are spent running tests that already passed on the previous run. Not only is it a waste of time & resources, it's also an opportunity for flakes to cause troubles.

With Retry only Failures, when you retry one (or all the) nodes, Knapsack Pro executes only the tests that failed last time on that node.

Configuration​

To enable Retry only Failures, npm install @knapsack-pro/playwright to v0.3 or later.

Knapsack Pro uses KNAPSACK_PRO_TEST_QUEUE_ID to uniquely identify a queue on the first run and all its retries.

Additional configuration for CircleCI​

On CircleCI, you need to expose the pipeline number as follows:

# ...
environment:
  CIRCLE_PIPELINE_NUMBER: << pipeline.number >>

Example​

Let's say your test suite contains 3 tests:

  • a.spec.ts
  • b.spec.ts
  • c.spec.ts

If you run your tests on 2 nodes, you could expect the following split:

  • Node 0: executes a.spec.ts and b.spec.ts where a.spec.ts:10 fails and b.spec.ts:20 is skipped
  • Node 1: executes c.spec.ts successfully

If you retry Node 0, Retry only Failures will execute only a.spec.ts:10. If you retry Node 1, Retry only Failures will execute no tests.