Difference between revisions of "Summit"
(→Build notes) |
|||
| Line 6: | Line 6: | ||
=== configure options === | === configure options === | ||
| − | configure --with-device=ch4:ucx | + | <pre><nowiki> |
| + | # Without CUDA support | ||
| + | configure --with-device=ch4:ucx | ||
| + | # With CUDA support | ||
| + | configure --with-device=ch4:ucx --with-cuda=$CUDAPATH | ||
| + | </nowiki></pre> | ||
=== Build notes === | === Build notes === | ||
| Line 14: | Line 19: | ||
=== Execution notes === | === Execution notes === | ||
| − | MPICH | + | As far as we checked, MPICH (3.4b1) works with 'jsrun'. However, you might need to unload Darshan at present (10/6/2020) (See [https://github.com/pmodels/mpich/issues/4815]). |
| + | |||
| + | <pre><nowiki> | ||
| + | module unload darshan-runtime | ||
| + | </nowiki></pre> | ||
| + | |||
| + | To run it with GPUs, you might need to add '--smpiargs="-disable_gpu_hooks"' for 'jsrun'. | ||
Revision as of 20:34, 6 October 2020
This page describes how to build MPICH on the 'Summit' machine at Oak Ridge. Summit is a POWER9-based machine with Infiniband interconnect. The 'UCX' device works best here. Performance is on par with the IBM SpectrumScale MPI (based on OpenMPI).
Summit build instructions
Released versions of mpich (as of 3.2) do not know how to deal with the unusually large business cards generated on this platform. You'll need a release containing commit 2bb15c6e875386f.
configure options
# Without CUDA support configure --with-device=ch4:ucx # With CUDA support configure --with-device=ch4:ucx --with-cuda=$CUDAPATH
Build notes
Watch out for system ucx vs mpich built-in ucx. I got some undefined symbols in ucx routines because MPICH was configured to use its own mpich but was picking up system UCX (thanks to spack setting LD_LIBRARY_PATH)
Execution notes
As far as we checked, MPICH (3.4b1) works with 'jsrun'. However, you might need to unload Darshan at present (10/6/2020) (See [1]).
module unload darshan-runtime
To run it with GPUs, you might need to add '--smpiargs="-disable_gpu_hooks"' for 'jsrun'.
