From Mpich
Revision as of 20:34, 6 October 2020 by Siwasaki (talk | contribs)

Jump to: navigation, search

This page describes how to build MPICH on the 'Summit' machine at Oak Ridge. Summit is a POWER9-based machine with Infiniband interconnect. The 'UCX' device works best here. Performance is on par with the IBM SpectrumScale MPI (based on OpenMPI).

Summit build instructions

Released versions of mpich (as of 3.2) do not know how to deal with the unusually large business cards generated on this platform. You'll need a release containing commit 2bb15c6e875386f.

configure options

# Without CUDA support
configure --with-device=ch4:ucx
# With CUDA support
configure --with-device=ch4:ucx --with-cuda=$CUDAPATH

Build notes

Watch out for system ucx vs mpich built-in ucx. I got some undefined symbols in ucx routines because MPICH was configured to use its own mpich but was picking up system UCX (thanks to spack setting LD_LIBRARY_PATH)

Execution notes

As far as we checked, MPICH (3.4b1) works with 'jsrun'. However, you might need to unload Darshan at present (10/6/2020) (See [1]).

module unload darshan-runtime

To run it with GPUs, you might need to add '--smpiargs="-disable_gpu_hooks"' for 'jsrun'.