Full support for C interface to MPI.

[jmbr]

This is a work in progress PR to add full API support to cl-mpi. AFAICT the only currently missing features from the MPI 3.1 standard are the functions and datatypes intended to interface with Fortran (e.g., MPI_Fint, MPI_*_c2f, and MPI_*_f2c).

The submitted patch passes the test suite with SBCL 1.5.3.114 and Open MPI 4.0.1.

These are remaining tasks:

1. Reintroduce version information for each function.
2. Use knowledge of the underlying MPI library at build time. That way, we could create interfaces to features that are specific to Open MPI (e.g., the communicator split type constants OMPI_COMM_TYPE_*) and MPICH (e.g., MPIX_COMM_TYPE_NEIGHBORHOOD).
3. Open MPI and MPICH differ on how they define some constants. Sometimes it's via enumeration types and other times it is done via preprocessor macros (e.g., MPI_DIST_GRAPH, MPI_CART, MPI_GRAPH, MPI_UNWEIGHTED, MPI_WEIGHTS_EMPTY, MPI_WIN_FLAVOR_CREATE, MPI_WIN_FLAVOR_ALLOCATE, MPI_WIN_FLAVOR_DYNAMIC, MPI_WIN_FLAVOR_SHARED, MPI_WIN_UNIFIED, and MPI_WIN_SEPARATE). This thwarts the groveller and we need a better way to handle this at build time.

Looking forward to hearing your thoughts on this.

[marcoheisig]

Wow, it is great to hear someone is working on making cl-mpi feature complete! I will support you as good as I can. Some remarks:

What happened to my defmpifun macro? Your PR seems to remove it. I understand that defmpifun it is a bit extravagant, but since the MPI standard uses a very consistent naming convention, it works really well in practice.

1. I am not sure what you mean by 'reintroduce version information'. Is that because your patch removes my defmpifun macro?

2. and 3. Yes, the fact that MPICH and OpenMPI use very different representation for many important data structures is a pain. Furthermore, each MPI implementation might choose to change the representation with each new release (luckily, they usually don't do that). And I haven't even mentioned the worst part yet: On OpenMPI, several MPI constants are actually pointers to heap allocated objects. Of course, a C program will never notice that. But in Lisp, you can save-lisp-and-die and restart, which means that a Lisp program can actually load MPI multiple times. That is the reason why I provide kludges like initialize-mpi-constants and reload-mpi-libraries.

I see two ways to handle the OpenMPI/MPICH dilemma.

1. Add even more kludges to define-mpi-constant and the reader macro in mpi/wrap.lisp.

2. Provide two low-level libraries cl-mpi-openmpiand cl-mpi-mpich that export a shared interface and have cl-mpi pick the right one with some ASDF magic.

3. Find a good general solution to the issue of C constants with unknown representation and contribute it to CFFI.

The good thing about 2. would be that one can easily provide access to implementation dependent features.

[jmbr]

Thank you for your thoughtful and quick reply.

What happened to my defmpifun macro? Your PR seems to remove it. I understand that defmpifun it is a bit extravagant, but since the MPI standard uses a very consistent naming convention, it works really well in practice.

I agree that MPI is consistent with respect to names and I see defmpifun as a useful device for incorporating new prototypes by hand. However, the patch I submitted is almost fully mechanically-generated and the interface-generation code (to be cleaned-up and posted soon) can also accommodate new functionality directly, so I saw no need to keep defmpifun around because it became more convoluted to generate bindings based on naming conventions than to use the types directly in an automated manner. In short, the rationale for omitting defmpifun is that the generator can produce defcfun forms directly and can handle new functionality when it becomes available in forthcoming revisions of the MPI standard.

1. I am not sure what you mean by 'reintroduce version information'. Is that because your patch removes my defmpifun macro?

That is right. I don't currently have a good source of information to feed the versioning information automatically, so I'll have to do that by hand later on. On a side note, I wish the MPI forum published the LaTeX sources of the standard document (or the latest draft).

2. and 3. Yes, the fact that MPICH and OpenMPI use very different representation for many important data structures is a pain. Furthermore, each MPI implementation might choose to change the representation with each new release (luckily, they usually don't do that). And I haven't even mentioned the worst part yet: On OpenMPI, several MPI constants are actually pointers to heap allocated objects. Of course, a C program will never notice that. But in Lisp, you can save-lisp-and-die and restart, which means that a Lisp program can actually load MPI multiple times. That is the reason why I provide kludges like initialize-mpi-constants and reload-mpi-libraries.

Yeah, that makes things tricky. In a different version of this patch I was using defcvar to get a hold of the global variables that underpin stuff like MPI_COMM_WORLD in Open MPI.

I see two ways to handle the OpenMPI/MPICH dilemma.

1. Add even more kludges to define-mpi-constant and the reader macro in mpi/wrap.lisp.
2. Provide two low-level libraries cl-mpi-openmpiand cl-mpi-mpich that export a shared interface and have cl-mpi pick the right one with some ASDF magic.
3. Find a good general solution to the issue of C constants with unknown representation and contribute it to CFFI.

The good thing about 2. would be that one can easily provide access to implementation dependent features.

In general I'd prefer to stay away from option 1. The overhead of calling a function might not be much but it is unnecessary. Option 2 is more palatable but IMO makes the code less flexible (i.e., it wouldn't work out of the box for a hypothetical brand new implementation of MPI). I find option 3 appealing but I fear it's out of the scope of CFFI, which leads me to suggest (just for the sake of discussion --I'm not necessarily endorsing this--) to use an external build system such as CMake (or GNU autoconf) to handle the detection of the idiosyncrasies of the MPI implementation and generate a suitable grovel.lisp file. One could even do away with the need to include version information for each constant and function because the auto-configuration step could try everything and only compile interfaces to facilities that are available.

[marcoheisig]

I really hope we can do without CMake or Autoconf. I don't think these build systems add a lot of value compared to ASDF and CFFI.

Here is another idea:

We use a fully table driven approach with the variables *mpi-types*, *mpi-functions* and *mpi-constants*. Each of these variables would contain records with the relevant information from the MPI standard - name, version information, type information, the name of the corresponding Lisp type etc.

Then cl-mpi performs the following steps:

1. It grovels the C type of each MPI type, using black magic that we have yet to invent (probably just a suitable call to defwrapper*).

2. It defines the suitable Lisp wrappers for each type, using the information from 1.

3. It defines all MPI constants, as well as the suitable reload-mpi-constants function.

4. It defines all MPI functions described in *mpi-functions*.

The good thing about this approach is that each step can be implemented as a single macro.

How does that sound?

[jmbr]

Sounds good. I'll get back to you with an update by the end of the week.

[guicho271828]

enumeration vs macro problem can be addressed by defwrapper in CFFI.
https://common-lisp.net/project/cffi/manual/cffi-manual.html#Wrapper-for-Inline_002fStatic-Functions-and-Macros

[jmbr]

enumeration vs macro problem can be addressed by defwrapper in CFFI.
https://common-lisp.net/project/cffi/manual/cffi-manual.html#Wrapper-for-Inline_002fStatic-Functions-and-Macros

Thanks for your suggestion, @guicho271828. As a matter of fact, cl-mpi already uses defwrapper (or the undocumented defwrapper*, to be precise). See mpi/wrap.lisp and the definition of the define-mpi-constant macro for details.

[marcoheisig]

By the way - jmbr, what is the current status of your code? Please tell me when I shall review and merge anything.

[jmbr]

By the way - jmbr, what is the current status of your code? Please tell me when I shall review and merge anything.

I've been using this patch successfully since I posted it here. I still have to get around to re-entering version information on the symbols, though.

Saw your sb-simd work recently. Great stuff!