Regression tests not working reliably

[jnsbck]

Despite putting in a lot of work, the regression tests (#475) are not working reliably in the CI. I have done some reading and found a few hints to why this is. Mainly the github CI is not very reliably. See here and ħere, one can expect large deviations (up to 50%) when running comparing the same code against itself in the CI. .

This means we might want to check for regressions differently:

1. only do it on demand (i.e. via github comment)
2. compute baseline at test time

I have also been looking into asv, which looks really cool and would be something to implement during a hackathon maybe.