Add "canary" workflow for HMR simulations [WIP] #22

mattwthompson · 2020-10-07T14:23:13Z

Resolves #21

What the script looks like in action:

(openff-dev) [openforcefields] echo openforcefields/offxml/openff-1.2.0.offxml | python canary/scripts/test_hmr.py
Running HMR with force field openff-1.2.0 and molecule with SMILES [H]C#CC([H])([H])[H]
Running HMR with force field openff-1.2.0 and molecule with SMILES [H]c1c(c(c(c(c1[H])[H])C#CC([H])([H])[H])[H])[H]
Traceback (most recent call last):
  File "canary/scripts/test_hmr.py", line 92, in <module>
    raise HMRCanaryError(failed_runs)
__main__.HMRCanaryError: [[Molecule with name '' and SMILES '[H]c1c(c(c(c(c1[H])[H])C#CC([H])([H])[H])[H])[H]', 'openff-1.2.0']]
(openff-dev) [openforcefields] echo openforcefields/offxml/openff-1.2.1.offxml | python canary/scripts/test_hmr.py
Running HMR with force field openff-1.2.1 and molecule with SMILES [H]C#CC([H])([H])[H]
Running HMR with force field openff-1.2.1 and molecule with SMILES [H]c1c(c(c(c(c1[H])[H])C#CC([H])([H])[H])[H])[H]

I was running into some stochastic test passes with 2 ps runs, so bumped it up to 10 ps.
Many of the "coverage" mols are not covered by Parsley. Haven't dug into them much for more insightful failures.
I intentionally kept these functions out of the module structure of the Python package as I don't believe these are things we should focus on shipping out. These are intentionally not rigorous and I'd like to avoid making long-term guarantees to the users that these tests are stable, or even a part of the API.

mattwthompson · 2020-10-07T15:33:35Z

This will be a little tricky to review, let me try to provide the necessary context:

This branch at 7de9632 is "the real" version
In ab97f08 I added back openff-1.2.0.offxml, which at least does what's expected locally, and should cause CI to fail. It will be rolled back before merging (also goofed some stuff, so 0ddea85 should be used as a point of comparison)

(openff-dev) [openforcefields] git diff upstream/master --name-only | grep "\.offxml" | python canary/scripts/test_hmr.py
Running HMR with force field openff-1.2.0-old-and-bad and molecule with SMILES [H]C#CC([H])([H])[H]
Running HMR with force field openff-1.2.0-old-and-bad and molecule with SMILES [H]c1c(c(c(c(c1[H])[H])C#CC([H])([H])[H])[H])[H]
Traceback (most recent call last):
  File "canary/scripts/test_hmr.py", line 92, in <module>
    raise HMRCanaryError(failed_runs)
__main__.HMRCanaryError: [[Molecule with name '' and SMILES '[H]c1c(c(c(c(c1[H])[H])C#CC([H])([H])[H])[H])[H]', 'openff-1.2.0-old-and-bad']]

j-wags

This is so cool! I think NaNs aren't being caught properly, but an uncaught NaN is still an informative canary, so it's up to you whether to try and catch them. Also, I don't think these sims are actually running for 10ps (rather they're still just at 2), so I left a comment on that.

Before you merge, could you modify step 2 of the release process to include the best way of running the canaries, and the criteria for blocking/not blocking the release? I think it will look like "ensure the new FF was tested by looking for these lines, and then ensure that no tests failed"

j-wags · 2020-10-12T23:17:36Z

canary/scripts/test_hmr.py

+        "rigidWater": True,
+        "removeCMMotion": False,
+        "hydrogenMass": 4
+        * unit.amu,  # Does this also _subtract_ mass from heavy atoms?:w


I think so. So carbons get really light sometimes.

j-wags · 2020-10-12T23:18:15Z

canary/scripts/test_hmr.py

+    context.setPositions(mol.conformers[0])
+
+    # Run for 10 ps
+    integrator.step(500)


This is only 2ps

Also, here's where we could put the try/except to catch NaNs from the simulation.

j-wags · 2020-10-12T23:48:32Z

canary/scripts/test_hmr.py

+        for mol in hmr_mols:
+            try:
+                hmr_driver(mol, ff_name)
+            except NANEnergyError:


On linux, NaN failures don't seem to be getting caught. Not a huge deal since failing is the whole point of the canary, but it may be helpful to make it so that multiple NaNs can be caught and a summary of failures can be shown to users.

Unfortunately, we run into one of those "un-Pythonic" behaviors in OpenMM, as the triggered exception cannot be caught.

>>> from simtk.openmm import OpenMMException >>> try: ... raise OpenMMException ... except OpenMMException: ... print('caught!') ... Traceback (most recent call last): File "<stdin>", line 2, in <module> TypeError: exceptions must derive from BaseException During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<stdin>", line 3, in <module> TypeError: catching classes that do not inherit from BaseException is not allowed

I can just catch it later, but not with the specificity we'd normally prefer (catching more than less is better for these tests, anyway):

(test) root@af8cfd33010c:/openforcefields# echo openforcefields/offxml/openff-1.2.0.offxml | python canary/scripts/test_hmr.py -vs Warning: Unable to load toolkit 'OpenEye Toolkit'. The Open Force Field Toolkit does not require the OpenEye Toolkits, and can use RDKit/AmberTools instead. However, if you have a valid license for the OpenEye Toolkits, consider installing them for faster performance and additional file format support: https://docs.eyesopen.com/toolkits/python/quickstart-python/linuxosx.html OpenEye offers free Toolkit licenses for academics: https://www.eyesopen.com/academic-licensing [16:05:40] WARNING: no name column found on line 0 [16:05:40] WARNING: no name column found on line 1 Running HMR with force field openff-1.2.0 and molecule with SMILES [H][C]#[C][C]([H])([H])[H] /bin/bash: /opt/conda/envs/test/lib/libtinfo.so.6: no version information available (required by /bin/bash) /bin/bash: /opt/conda/envs/test/lib/libtinfo.so.6: no version information available (required by /bin/bash) /bin/bash: /opt/conda/envs/test/lib/libtinfo.so.6: no version information available (required by /bin/bash) /bin/bash: /opt/conda/envs/test/lib/libtinfo.so.6: no version information available (required by /bin/bash) /bin/bash: /opt/conda/envs/test/lib/libtinfo.so.6: no version information available (required by /bin/bash) /bin/bash: /opt/conda/envs/test/lib/libtinfo.so.6: no version information available (required by /bin/bash) Running HMR with force field openff-1.2.0 and molecule with SMILES [H][c]1[c]([H])[c]([H])[c]([C]#[C][C]([H])([H])[H])[c]([H])[c]1[H] /bin/bash: /opt/conda/envs/test/lib/libtinfo.so.6: no version information available (required by /bin/bash) /bin/bash: /opt/conda/envs/test/lib/libtinfo.so.6: no version information available (required by /bin/bash) /bin/bash: /opt/conda/envs/test/lib/libtinfo.so.6: no version information available (required by /bin/bash) /bin/bash: /opt/conda/envs/test/lib/libtinfo.so.6: no version information available (required by /bin/bash) /bin/bash: /opt/conda/envs/test/lib/libtinfo.so.6: no version information available (required by /bin/bash) /bin/bash: /opt/conda/envs/test/lib/libtinfo.so.6: no version information available (required by /bin/bash) Traceback (most recent call last): File "canary/scripts/test_hmr.py", line 94, in <module> raise HMRCanaryError(failed_runs) __main__.HMRCanaryError: [[Molecule with name '' and SMILES '[H][C]#[C][C]([H])([H])[H]', 'openff-1.2.0', 'NaN Position(s)'], [Molecule with name '' and SMILES '[H][c]1[c]([H])[c]([H])[c]([C]#[C][C]([H])([H])[H])[c]([H])[c]1[H]', 'openff-1.2.0', 'NaN Position(s)']]

mattwthompson · 2020-10-13T16:10:19Z

All canary tests failing in 372a96a, rolling back the bad OFFXML and merging

The 1.2.0 OFFXML was used just for testing, as it needed to mimic a new file that not tracked by git on the master branch

mattwthompson added 2 commits October 7, 2020 09:24

Add inital pass of HMR canary workflow

280e54c

Clean up files

a79c760

mattwthompson force-pushed the hmr-canary branch from 891d497 to a79c760 Compare October 7, 2020 14:28

mattwthompson added 3 commits October 7, 2020 09:28

Rename canary workflow

ea9ac27

Ensure canary tests are not called during normaly unit tests

7de9632

Add intentionally bad file to trigger HMR failure

ab97f08

mattwthompson added 2 commits October 7, 2020 10:48

Fix git ref

2102be1

Fix another git ref

0ddea85

j-wags approved these changes Oct 12, 2020

View reviewed changes

Catch NaN particle positions, fix run length

372a96a

mattwthompson added 2 commits October 13, 2020 11:13

Remove "bad" file used in testing

b0a5887

The 1.2.0 OFFXML was used just for testing, as it needed to mimic a new file that not tracked by git on the master branch

Fix setting of new_off_xml var

09da994

mattwthompson merged commit c3d58d8 into master Oct 13, 2020

mattwthompson mentioned this pull request Oct 13, 2020

Automate running HMR "canary" tests before release #21

Open

mattwthompson deleted the hmr-canary branch September 7, 2022 14:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add "canary" workflow for HMR simulations [WIP] #22

Add "canary" workflow for HMR simulations [WIP] #22

mattwthompson commented Oct 7, 2020 •

edited by j-wags

Loading

mattwthompson commented Oct 7, 2020 •

edited

Loading

j-wags left a comment •

edited

Loading

j-wags Oct 12, 2020

j-wags Oct 12, 2020

j-wags Oct 12, 2020

j-wags Oct 12, 2020

mattwthompson Oct 13, 2020 •

edited

Loading

mattwthompson commented Oct 13, 2020

Add "canary" workflow for HMR simulations [WIP] #22

Add "canary" workflow for HMR simulations [WIP] #22

Conversation

mattwthompson commented Oct 7, 2020 • edited by j-wags Loading

mattwthompson commented Oct 7, 2020 • edited Loading

j-wags left a comment • edited Loading

Choose a reason for hiding this comment

j-wags Oct 12, 2020

Choose a reason for hiding this comment

j-wags Oct 12, 2020

Choose a reason for hiding this comment

j-wags Oct 12, 2020

Choose a reason for hiding this comment

j-wags Oct 12, 2020

Choose a reason for hiding this comment

mattwthompson Oct 13, 2020 • edited Loading

Choose a reason for hiding this comment

mattwthompson commented Oct 13, 2020

mattwthompson commented Oct 7, 2020 •

edited by j-wags

Loading

mattwthompson commented Oct 7, 2020 •

edited

Loading

j-wags left a comment •

edited

Loading

mattwthompson Oct 13, 2020 •

edited

Loading