test seeding and repeat

[IonBazan]

What is the reason for this PR?

This PR tries to rerun the tests without seeding to detect ones that are not generating valid results.

Thanks to this approach, I found following issues:

• regexify does not handle [.] properly (see Regexify dot issue #133)

• old and new optional() behavior does not always match

• PersonTest for mn_MN locale can fail on different regex collations ([А-Я] will not match letter Ё on C collation)

• BiasedTest is failing with random seed - needs fixing?

• An improvement to CI (fixes Test randomness #125)

Author's checklist

• Follow the Contribution Guide

Summary of changes

There is additional test matrix entry that executes each test 100 times, excluding tests tagged seed and external (downloading external resource) to ensure they work properly with different seeds.

setUp() method in base TestCase class checks if current test is in seed group and seeds random number generator with 1 if needed.

verbose mode was disabled by default to limit "skipped tests" output spam in the --repeat 100 job.

Review checklist

• All checks have passed
• Changes are approved by maintainer

[pimjansen]

Can we review last comments?

[localheinz]

@IonBazan

The tests are currently failing, can you take another 👀 , please?

I think that it is a good idea to expose failures in the generators by repeatedly running the tests, but perhaps 100 is a bit too much? What do you think?

With this change, we run *221800 instead of 2236 tests on GitHub Actions, and significantly increase the build time.

Perhaps it would make sense to add this as a target to Makefile, so a developer can run this locally (as well or alternatively)?

[krsriq]

@IonBazan apparently there's another regex-dot-related bug. I'll create a PR to fix this shortly.

[krsriq]

I think I've fixed this now (was not quite as easy as I had hoped), so once #206 is merged we should be green here.

[IonBazan]

Thanks @krsriq - seems that it's working fine now!

@localheinz how about we add another pipeline to run these 100x tests on push only (after merge to main branch) instead? This will reduce number of jobs run for PRs, still providing enough confidence to merge PRs with passing builds. If anything goes wrong, we should be still able to catch that once in a while after merges. I will add the task to Makefile to make it easier for developers to test some random-related features.

We will still need to stabilize BiasedTest behavior - does anyone have any experience with this feature?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 1 week if no further activity occurs. Thank you for your contributions.

[bram-pkg]

Any updates on this, @IonBazan ?

[IonBazan]

@bramceulemans sorry- there is not much to update here because it's just a POC and I believe it's up to the maintainers to decide whether they want to run these checks. I just resolved the conflicts so it's ready to merge if needed.

[bram-pkg]

@GrahamCampbell @pimjansen

Opinion on this?

[pimjansen]

I think its a fine addition but it doesnt directly mean that those implementations are always wrong.

But another important thing is that builds fail until we fix those methods as well.

How do we want to handle that? We can raise issues for those tests and silent the pipeline job or we just fix them directly.

I am a fan of the last one

[bram-pkg]

Closing this as stale for now.