This page describes how to build MPICH on the 'Summit' machine at Oak Ridge. Summit is a POWER9-based machine with Infiniband interconnect. The 'UCX' device works best here. Performance is on par with the IBM SpectrumScale MPI (based on OpenMPI).
Summit build instructions
Released versions of mpich (as of 3.2) do not know how to deal with the unusually large business cards generated on this platform. You'll need a release containing commit 2bb15c6e875386f.
# Without CUDA support configure --with-device=ch4:ucx # With CUDA support configure --with-device=ch4:ucx --with-cuda=$CUDAPATH
Watch out for system ucx vs mpich built-in ucx. I got some undefined symbols in ucx routines because MPICH was configured to use its own mpich but was picking up system UCX (thanks to spack setting LD_LIBRARY_PATH)
As far as we checked, MPICH (3.4b1) works with 'jsrun'. However, you might need to unload Darshan at present (10/6/2020) (See ).
module unload darshan-runtime
To run it with GPUs, you might need to add '--smpiargs="-disable_gpu_hooks"' for 'jsrun'.