Add runtime tests

[michaeldeistler]

Note that the tests are disabled in the workflow: pytest -m "not runtime"

[jnsbck]

Awesome! Thanks for implementing this, I think this will be really useful. I had also worked on the same thing, but was pursuing sth slightly different. I have added my proof of concept to this branch. (see also my comments). Feel free to work off of it or remove it. I think having a seperate CI for regressions might be useful in the long run.

Happy to discuss this more.

[github-actions]

New Baselines

test_runtime(num_cells=1, artificial=False, connect=False, connection_prob=0.0, voltage_solver=jaxley.stone)
🆕 build_time: (0.566s).
🆕 compile_time: (18.334s).
🆕 run_time: (2.804s).

test_runtime(num_cells=1, artificial=False, connect=False, connection_prob=0.0, voltage_solver=jax.sparse)
🆕 build_time: (0.394s).
🆕 compile_time: (3.041s).
🆕 run_time: (2.451s).

test_runtime(num_cells=10, artificial=False, connect=True, connection_prob=0.1, voltage_solver=jaxley.stone)
🆕 build_time: (1.836s).
🆕 compile_time: (29.308s).
🆕 run_time: (18.758s).

test_runtime(num_cells=10, artificial=False, connect=True, connection_prob=0.1, voltage_solver=jax.sparse)
🆕 build_time: (1.256s).
🆕 compile_time: (26.581s).
🆕 run_time: (25.064s).

test_runtime(num_cells=1000, artificial=True, connect=True, connection_prob=0.001, voltage_solver=jaxley.stone)
🆕 build_time: (109.680s).
🆕 compile_time: (45.643s).
🆕 run_time: (41.482s).

test_runtime(num_cells=1000, artificial=True, connect=True, connection_prob=0.001, voltage_solver=jax.sparse)
🆕 build_time: (108.686s).
🆕 compile_time: (61.090s).
🆕 run_time: (58.329s).

[jnsbck]

@michaeldeistler This should be ready for review. The following is implemented and works:

• tests/test_regression.py contains the regression tests that are run using pytest -m regression
• automates writing results to database, comparing results stored baselines and producing a report (see above).
• regression_tests.yml runs the regression tests.
• update_regression_baseline.yml to update the baseline
• idea is to trigger this on comment
• comments with updated baseline report
• pushes the updated baselines to the branch in which the workflow is run.

If you want to run this locally do you can do NEW_BASELINE=1 pytest -m regression i.e. on main and then run pytest -m regression on feature.

Some thoughts I had: Even with a 20% tolerance the regression tests frequently fail for some reason. (might need to average across several runs or take the max across several runs for the baseline, but currently the regression tests take quite long. Would be good to merge #489 to speed this up).

Lemme know if anything is unclear

[michaeldeistler]

Wow this is amazing!!!

I don't think I understand when exactly the updating of the times is being run. On a comment to this particular PR (i.e., #475)? Or some other PR/issue? Also, what should the comment be? Anything? Who would the comment have to be by? Anyone? (BTW I am commenting now, so does this trigger the regression tests :D ?)

[michaeldeistler]

I don't get this. Why do we first put it in the gitignore just to then force add it?

[jnsbck]

I agree this appears a bit odd, but I want to prevent users from pushing an updated baseline file from their local machine (which is why I have added it to the gitignore). The only way baselines should be updated is through github actions (to ensure regression tests are run in a consistent environment). That is why the github actions uses -f, while for the user changes to these files are ignored.

EDIT: This also makes running regression tests locally easier, since: running NEW_BASELINE=1 pytest -m regression on main and then pytest -m regression on feature. Ignores the changes to the baseline when switching the branch, so no stashing etc. necessary to run the test on feature with the baseline file from main.

[jnsbck]

Does that make sense?

[michaeldeistler]

Ah, got it, yes makes sense!

[jnsbck]

The event issue_comment gets triggered on every comment that gets posted to an issue or PR. However, the workflow to produce an updated baseline is only triggered if the comment body contains the string /update_regression_baseline (so your comment would not have triggered it a) because the issue_comment workflow has to be merged to main first due to the way the event listener works on github and b) cos you did not use the command above). The workflow is run on the PR branch, which means the new baseline is produced from the updated codebase. The change to the baselines is then merged into the current PR. The command to start the workflow can currently be used by all users, but we could limit this to admins as well.

[jnsbck]

I have verified this to work in a seperate repo, if the workflow to update the baselines exists on the default branch. Upon commenting with /update_regression_baselines on any PR, the workflow gets triggered. First a comment is posted to inform the user the workflow is run, then the new baselines are computed and compared to the ones stored in main, and a report is produced and the prev. comment is updated with the result. If successful, the updated baselines are pushed to the PR branch if the branch is part of the repo or a seperate PR is opened in case the branch is from a fork (The user is also made aware of this in the comment), see below.

Since the workflow can only be triggered on the default branch this PR has to be merged first and needs a seperate PR to update the baselines using /update_regression_baselines. Hence the regression tests above failing.

Side note: The report is always written to the pytest output (above the Failure report), so you can check why the regression test action might have failed in detail.

With this, I think this can be merged @michaeldeistler, unless you have any other reseverations.

[michaeldeistler]

Thanks a lot! Feel free to merge (I cannot approve because it is technically "my" PR).