Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MPI parallelization of dynamics #120

Open
Tracked by #593
athelaf opened this issue Aug 9, 2022 · 4 comments · May be fixed by #663
Open
Tracked by #593

MPI parallelization of dynamics #120

athelaf opened this issue Aug 9, 2022 · 4 comments · May be fixed by #663
Assignees
Labels
ICCS Tasks or reviews for the ICCS team

Comments

@athelaf
Copy link

athelaf commented Aug 9, 2022

No description provided.

@athelaf athelaf self-assigned this Aug 9, 2022
@athelaf athelaf changed the title MPI parallelization of Dynamics MPI parallelization of dynamics Aug 9, 2022
@einola einola added this to the 3 Stand-alone model milestone Aug 15, 2022
@einola einola moved this to Todo in neXtSIM_DG overview Aug 15, 2022
@einola einola moved this from Todo to In Progress in neXtSIM_DG overview Aug 15, 2022
@athelaf athelaf moved this from In Progress to Todo in neXtSIM_DG overview Sep 1, 2022
@draenog draenog added the ICCS Tasks or reviews for the ICCS team label May 15, 2023
@mondus mondus moved this from Todo to In Progress in neXtSIM_DG overview May 15, 2023
@mondus mondus moved this from In Progress to Pending in neXtSIM_DG overview May 15, 2023
@mondus
Copy link

mondus commented May 15, 2023

A decision on this work is required. Either

  1. Merge now and then add the Dynamics (Milestone) parallelisation at a later date
  2. Wait until Dynamics merge, add dynamics parallelisation and merge later.

Preference is perhaps for 1 and then a new PR for the dynamics parallelisation later.

@draenog
Copy link
Collaborator

draenog commented Jun 1, 2023

Requirements:

  1. MPI parallelization should be based on domain decomposition
  2. Decomposition should use rectangular boxes
  3. Decomposition should take into account land mask for load balancing
  4. Decomposition will not change during the run

@draenog
Copy link
Collaborator

draenog commented Jun 1, 2023

  • Write a preprocessing tool to generate domain decomposition
  • MPI parallelization of thermodynamics where all operations, exept for I/O, are local to an MPI rank
  • MPI parallelization of dynamics

@MarionBWeinzierl
Copy link
Contributor

Info from @einola and @timspainNERSC about size of problems to be computed:

Hi! This is to follow up the discussion we had just now. The question was what the problem size is. So I went and did a bit of
checking. The standard development test that I use has about 500k elements. Our coupled run has 300k - but we run it for much longer. A fully coupled climate run at the lowest resolution would have about 100k. We want to be able to run regional setups with much higher resolutions - 10 million would be nice!

So:

  • Standard coupled: 100k, maybe 1 million
  • Regional: 500k to 2 million
  • A dream scenario for regional: 10 million

For reference, the 25kmNH test run has 18k elements, and I don't think we would want the production version of the model to be much slower than that is when running single threaded. So 10k elements per core might be a good rule of thumb.

TomMelt added a commit that referenced this issue Dec 4, 2023
## Background 
This PR is part of #120, where the basis of strategy for MPI
parallelisation is described.

## Change description
As all thermodynamics operations are local to a grid cell all required
MPI communication, is handled by NetCDF4 library. Therefore the only
required steps are:

- [x] Initialize and finalize MPI stack
- [x] Read decomposition metadata on each rank
- [x] Read and write the necessary part of the grid on each rank
(depends on #330)
- [x] Tests for parallel I/O 

**NOTE** PR #432 should be merged into this branch before it is merged
into develop

To run `run_simple_example.sh` you will need to generate the following
netcdf file

```
$ ncdump partition.nc 
netcdf partition {
dimensions:
        P = 1 ;
        L = 1 ;
        globalX = 30 ;
        globalY = 30 ;

group: bounding_boxes {
  variables:
        int global_x(P) ;
        int global_y(P) ;
        int local_extent_x(P) ;
        int local_extent_y(P) ;
  data:

   global_x = 0 ;

   global_y = 0 ;

   local_extent_x = 30 ;

   local_extent_y = 30 ;
  } // group bounding_boxes
}

```
TomMelt added a commit that referenced this issue Aug 21, 2024
# Add MPI support to ParaGrid
closes #534
closes #454
closes #448 

---
# Task List
- [x] modify `make_init_para24x30.py` to use new DG data layout and
remove CG vars
- [x] add MPI doctest support to `ParaGrid_test.cpp`
- [x] move `extent` and `start` variables into `ModelArray`'s
`DimensionSpec`
- [x] add special MPI test case to `ParaGrid_test.cpp`
- [x] add MPI support to `getModelState`
- [x] add MPI support to `dumpModelState`
- [x] add MPI support to `writeDiagnosticTime`
- [x] reinstate test `ConfigOutput_test.cpp` for MPI builds
- [x] add MPI support to `ERA5`/`TOPAZ` tests

---
# Change Description

After #331 added MPI parallelisation for thermodynamics on the RectGrid,
this PR does the same for the Parametric grid. This should then check
off the second task in #120 (_"MPI parallelization of thermodynamics
where all operations, except for I/O, are local to an MPI rank"_)

---
# Test Description

- `ParaGrid_test.cpp` tests core functionality of ParaGrid (serial and
MPI)
- `./nextsim --config-file config_para24x30.cfg` should provide an
integration test (serial and MPI) (based on #506)

---
# Further work (for a future PR)

- add MPI to dynamics (to close PR #120)
- implement halo comms
- implement boundary conditions (as part of MPI) 
- tidy naming of variables in Domain Decomp tool
@TomMelt TomMelt linked a pull request Aug 23, 2024 that will close this issue
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ICCS Tasks or reviews for the ICCS team
Projects
Status: Pending
Development

Successfully merging a pull request may close this issue.

5 participants