Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flaky performance test #293

Open
jodersky opened this issue Apr 4, 2019 · 1 comment
Open

Flaky performance test #293

jodersky opened this issue Apr 4, 2019 · 1 comment

Comments

@jodersky
Copy link

jodersky commented Apr 4, 2019

The "not show bad performance characteristics when object keys' hashCodes collide" test in JsonParserSpec is flaky.

collidingTime / regularTime must be < 2L // speed must be in same order of magnitude

In absolute terms, the regularTime is about 3E6 (3ms) on a modern laptop. This time is sufficiently large to make the test mostly pass on a dedicated machine. However, on a shared environment such as a Travis-CI build, it frequently causes the build to fail.

Potential solutions:

  • easy: increase the number of keys by an order of magnitude
  • future-proof: don't assume a number of keys will give a large enough time, but rather incrementally deduce that number for a given system time and then run the test
@raboof
Copy link
Contributor

raboof commented Apr 4, 2019

I think we should not put too much effort in these tests - let's first increase the number of keys (and/or be more lenient in the expectation) - if that's not enough I'd even be open to doing a couple of runs or removing the test wholesale.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants