BUG: Network jones2009_3x3_drives.json is incompatible with MPI

[asoplata]

I ran into the following issue when investigating #871. Currently, this bug prevents the default number of cores from being greater than one, in the case of the GUI when MPI is activated.

I was testing the GUI by way of enabling MPI and increasing the default number of cores used. However, weirdly, only some, but not all, of the GUI tests were failing at the actual simulation step. I realized that the problem was with this particular test network itself, not the GUI. Here is an example script that, if you run it from the hnn-core directory, reproduces the error (assuming you have MPI installed):

#!/usr/bin/env python

from pathlib import Path
from hnn_core import jones_2009_model, MPIBackend, simulate_dipole
from hnn_core import hnn_io as hio

hnn_core_root = Path('hnn_core')

###########################
# Network 1: the default case of HNN-Core's API. The config file is shown here
# ONLY for the sake of comparison.
config_net1 = (hnn_core_root / 'param' / 'default.json')
net1 = jones_2009_model()

###########################
# Network 2: the default case for the GUI. Note that this must be loaded
# differently than the classic case above.
config_net2 = (hnn_core_root / 'param' / 'jones2009_base.json')
net2 = hio.read_network_configuration(config_net2)

###########################
# Network 3: the network that is sometimes, but not always, the network used in
# actual GUI tests. Specifically, this is the network used in GUI tests when
# the `setup_gui` fixture is used.
config_net3 = (hnn_core_root / 'tests' / 'assets' / 'jones2009_3x3_drives.json')
net3 = hio.read_network_configuration(config_net3)

########################################################################
# Now let's test that they actually simulate using MPI.
# We will see that Networks 1 and 2 work, but 3 does not. Therefore, any GUI
# tests that use Network 3 and attempt to use MPI will fail.
for net in [net1, net2, net3]:
    try:
        with MPIBackend(n_procs=4):
            dpl = simulate_dipole(net, tstop=20., n_trials=1)
    except RuntimeError as err:
        print(err)

If you run this, you'll notice that net1 (the default model of the API) and net2 (the default model of the GUI) work fine, but net3 fails. The key part of the error appears to be the following:

Building the NEURON model
0 /opt/anaconda3/envs/hc12/bin/nrniv: usable mindelay is 0 (or less than dt for fixed step method)
0  near line 0
0  ^
        0 finitialize()
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
  Proc: [[5323,1],0]
  Errorcode: -1

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[Done]

Looking at the NEURON forums ( https://www.neuron.yale.edu/phpBB/viewtopic.php?t=3090 ), this may be tied to a particular but unknown delay parameter causing problems in net3 but not net2.

More generally, this is a case of inconsistency in what networks we test and when: this issue caused many, but not all, GUI tests which run simulations to fail. As it turns out, the actual network being simulated is inconsistent in test_gui.py: sometimes, it's the default GUI network (such as in test_gui.py::test_gui_add_drives()), but sometimes it's instead the network tied to this issue (any test that uses the test_gui.py::setup_gui() fixture). This also implies that many of our GUI tests are not actually testing the default GUI network!

Therefore, the tasks for this issue should include:

• Diagnose and fix the issue preventing the network of jones2009_3x3_drives.json from using MPI. Based on its creation in test_io.py::jones_2009_network(), and the error output from NEURON and MPI, it is almost certainly an issue with a delay parameter in one of the additional drives added to this network, which are not present in the default network.
• Properly introduce pytest fixture usage to GUI testing, and ensure that, at minimum, all testing tests the default network (except where unreasonable).

[asoplata]

This is almost certainly related to #663