[Feature] Support running flaky-tests N times before marking them failed

[rahulait]

Problem Statement

We are planning to use chainsaw for our k8s testing. We have e2e tests written in go which do a lot of system testing using k8s. Since these tests work with a bunch of systems, things can sometimes go wrong due to flakiness in environment. For example, if API is being upgraded and the tests are trying to reach to API or requests are getting rate-limited etc. If a test fails due to some reason, we would like to re-run it automatically just to see if it passes the next time or not than requiring someone to manually trigger tests again. We would like to do this only for the tests which have failed. Chainsaw does have a flag --repeat-count=N but that always runs the test N number of times (irrespective of failure). We would like to have something similar to what ginkgo has (ginkgo.flake-attempts=N) so that flaky tests can be auto-retried.

Solution Description

Similar to --repeat-count, add a flag --flake-attempts or --repeat-failed-count which will make the failed test run max N times or until it succeeds.

Alternatives

No response

Additional Context

No response

Slack discussion

https://kubernetes.slack.com/archives/C067LUFL43U/p1733811046564419

Research

• I have searched other issues in this repository and mine is not recorded.