Verifying CVODE performance with upcoming 9.0 release

[pramodk]

Context

I am copying this ticket from one of the BBP internal ticket. As this example depends on BBP internal circuit & configurations, unfortunately it's a bit difficult to provide it as a complete reproducer. But I thought it would be good to copy whatever is available as we are going to loose access to BBP infrastructure soon.

Overview of the issue

This is what Andras provided:

Similar to this: https://github.com/neuronsimulator/nrn/issues/2787 I've also experienced a drop in performance of NEURON 9.0 compared to NEURON 8.2.4 when simulating a single cell using CVode.

My setup is in bluecellulab, but I've put together a minimal example that can be run in neurodamus with the latest module. It's a long simulation (22.5 minutes of biological time) with sparse events: there are 2 * 60 EPSPs (evoked by spike replay) and 75 spikes (evoked by current injection).

I've put it here:
/gpfs/bbp.cscs.ch/project/proj96/home/ecker/simulations/BBPBGLIB-1172

With CVode in blucellulab it took:
33 mins. in NEURON 9.0
20 mins. in NEURON 8.2.4

Although improved, the second number still seems weird, considering that it's just a single cell and I was hoping that CVode will require less compute time than biological time... Giuseppe claims that he ran the same sims <2-3 mins. with NEURON 7.

(With fixed dt, it takes ~5h with NEURON 9.0.)

See attached exported ticket.
[#BBPBGLIB-1172] NEURON (and CVode) performance drop.pdf

Example

The example BBPBGLIB-1172/bluecellulab/runsim.py mentioned in the ticket uses BBP's sonata circuit. But copying the example here for reference:

import os
import time
from libsonata import SpikeReader
from bluepysnap import Simulation
import bluecellulab

bluecellulab.set_verbose(2)
bluecellulab.neuron.h.cvode.atolscale("v", .1)
BASE_DIR = "/gpfs/bbp.cscs.ch/project/proj16/kumbhar/pramod_scratch/andras/BBPBGLIB-1172"


def main(node_pop="S1nonbarrel_neurons", fastforward=750000.):
    # Get necessary params
    sim_config = os.path.join(BASE_DIR, "bluecellulab", "simulation_config.json")
    sim = Simulation(sim_config)
    t_end = sim.config["run"]["tstop"]
    pre_gid, post_gid = sim.node_sets.content["precell"]["node_id"][0], sim.node_sets.content["postcell"]["node_id"][0]
    pre_spikes = SpikeReader(os.path.join(BASE_DIR, "prespikes.h5"))[node_pop].get_dict()["timestamps"]
    # Instantiate postcell in `bluecellulab`
    sim = bluecellulab.CircuitSimulation(sim_config)
    sim.instantiate_gids([(node_pop, post_gid)], add_synapses=True, add_minis=False, add_pulse_stimuli=True,
                         intersect_pre_gids=[(node_pop, pre_gid)],
                         pre_spike_trains={(node_pop, pre_gid): pre_spikes})
    cell = sim.cells[(node_pop, post_gid)]
    for sec in cell.somatic + cell.axonal:
        sec.uninsert("SK_E2")  # fix AHP during high freq. firing, by removing big K+ channel
    # Run sim to fastforward point and move all synases to stable states (to solve fewer eqs.)
    start_time = time.time()
    sim.run(fastforward, cvode=True)
    for syn_id, synapse in cell.synapses.items():
        if synapse.hsynapse.rho_GB >= 0.5:
            synapse.hsynapse.rho_GB = 1.
            synapse.hsynapse.Use = synapse.hsynapse.Use_p
            synapse.hsynapse.gmax_AMPA = synapse.hsynapse.gmax_p_AMPA
        else:
            synapse.hsynapse.rho_GB = 0.
            synapse.hsynapse.Use = synapse.hsynapse.Use_d
            synapse.hsynapse.gmax_AMPA = synapse.hsynapse.gmax_d_AMPA
    # Complete run
    bluecellulab.neuron.h.cvode_active(1)
    bluecellulab.neuron.h.continuerun(t_end)
    print("Elapsed time: %s" % time.strftime("%H:%M:%S", time.gmtime(time.time() - start_time)))


if __name__ == "__main__":
    main()

Expected result/behavior

Performance between v8.0 and 9.0 should be same.

BBPBGLIB-1172.tgz