View source for Building MPICH For Development

When developing MPICH itself it is often useful to build the library differently than you would for production use.  This page aggregates several tips and tricks for making your development life easier when working with MPICH.

== Enable Debugging Symbols ==
Configure with <code>--enable-g=dbg</code> to get debugging symbols included in the build.

<pre>
  --enable-g=option - Control the level of debugging support in the MPICH
  implementation.  option is a list of comma separated names including
      none     - No debugging
      mem      - Memory usage tracing
      handle   - Trace handle operations
      dbg      - Add compiler flag, -g, to CFLAGS, CXXFLAGS FFLAGS and F90FLAGS.
      log      - Enable debug event logging
      meminit  - Preinitialize memory associated structures and unions to
                 eliminate access warnings from programs like valgrind
      mutex    - Enable error checking on pthread mutexes
      all      - All of the above choices
</pre>

Personally I (goodell@) usually build with: <code>--enable-g=dbg,mem,meminit,handle</code> or <code>--enable-g=all</code>.  I will drop some or all of these for performance testing.  I also find that it is useful to include <code>CFLAGS=-g3</code> to enable gcc to expand macros.

== Configure With "strict" ==
MPICH is a very portable project and one of the only ways to ensure that we are developing portable code on a permissive compiler like gcc is to configure gcc to be as strict and pedantic as possible.  The MPICH configure provides an option to make this easy: <code>--enable-strict</code>.

I recommend using <code>--enable-strict=all</code> or <code>--enable-strict=posix</code> for those situations where <code>all</code> is too constraining and you know what you are doing.  This strictness will also help to catch general programming errors, not just portability issues.

== Use gdb ==
When debugging MPICH, you may want to use <code>gdb</code> to step into the code to see what's going on inside the code, whether to check out a variable or to track a segmentation fault. Here are several tips for you to use <code>gdb</code> in MPICH development.
=== remove <code>-O2</code> flag ===
The default cflag of MPICH, <code>-O2</code>, will change the instruction order of your application. It will cause <code>gdb</code> jump to unexpected lines in your code when you doing stepping. The solution is to configure with <pre>--enable-fast=O0</pre>

Another way of doing this is to change the flags in 
[http://stackoverflow.com/questions/3116645/default-compiler-flags-with-autotools config.site].

=== Debugging Multiple Processes ===
For general information of how to debug an MPI program, see 
[http://stackoverflow.com/questions/329259/how-do-i-debug-an-mpi-program discussion] here. Another discussion is [http://lists.mcs.anl.gov/pipermail/mpich-discuss/2010-April/006966.html here].

To use gdb to debugging multiple processes. You need to have xterm installed.
Then you can launch multiple MPI processes in gdb with following command:
<pre>
mpiexec -n [number of processes] xterm -e gdb ./your_app
</pre>

Or you want to launch gdb windows only for ranks 0 and 4.
<pre>
mpiexec -n 1 xterm -e gdb ./your_app : -n  3 ./your_app : -n 1 xterm -e gdb ./your_app
</pre>

A very simple way to attach <code>gdb</code> to a MPI process is to
# place a empty loop in your code: <code>while(a){}</code>
# use <code>ps</code> or <code>pgrep</code> to find your process id, attach <code>gdb</code> to it
# use <code>(gdb) set var a = 0</code> to jump out of the loop

Additional note: the default setting of xterm may be ugly. You may want to [[Configure xterm Fonts and Colors for Your Eyeball]].

=== TUI ===
gdb provides a Text User Interface for you to show the source file, the assembly output, the program registers and gdb commands in separate text windows. Check it out in [http://sourceware.org/gdb/onlinedocs/gdb/TUI.html TUI]. Here is a screenshot.
[[File:Gdb-tui-example.png | 600px]]

For mac users, you may need to build gdb from source because the default gdb provided by apple does not support TUI. 

=== Building GDB for OS X ===
* http://sourceware.org/gdb/wiki/BuildingOnDarwin
* http://www.opensource.apple.com/source/lldb/lldb-69/docs/code-signing.txt

=== Use DDD instead of GDB ===
If you are more comfortable with GUI-enabled debuggers, you may want to use [http://www.gnu.org/software/ddd/ DDD] instead of GDB.
The corresponding commands for launching MPI process in ddd are:
<pre>
mpiexec -n [number of processes] ddd ./your_app
mpiexec -n 1 ddd ./your_app : -n  3 ./your_app : -n 1 ddd ./your_app
</pre>

=== CGDB ===
A nice alternative to gdb's TUI mode is [http://cgdb.github.io/ cgdb] "The Curses Debugger".  It's a curses front-end to gdb, with vi-like navigation out of the box.   Use it the same way you'd use gdb: cgdb  passes its command line arguments down to gdb.

=== Compile Programs with "-g" ===
It is a little trick to get the compiled binary support -g. See the make process for cpi as a reference.
<pre>
cd examples
make -n cpi
</pre>

== Use -jN ==
Run <code>make -jN</code> where <code>N</code> is some number like 4 or 8.  This only works starting with MPICH2-1.5, since prior to that the build system used an approach that was not parallel-make safe.

== Use ccache ==
When developing MPICH it's not uncommon to need to <code>make clean</code> or <code>make distclean</code> fairly often for various reasons.  One way to help reduce the time drain that this can be is to use [http://ccache.samba.org/ ccache].  On my desktop this utility cuts my build times in half or better.  For setup just read the docs on the ccache site, they are pretty good.

== Don't Build Unnecessary Code ==
If you know you won't need a particular feature of MPICH then you might be able to disable it to speed up compilation or to prevent configure from complaining.  For example you can pick and choose from the following list of systems that can be disabled:

<pre>
--disable-f77
--disable-fc
--disable-cxx
--disable-romio
</pre>

The first three are useful if you only need the MPI C language bindings.  They disable the Fortran and C++ bindings.  The last one disables MPI-IO support.

== Use VPATH Builds ==
You can build several versions of MPICH from the same source tree by doing what's known as a [http://www.gnu.org/software/make/manual/make.html#General-Search "VPATH build"].  Let's assume you have your source in some directory like: <code>/foo/bar/mpich-trunk</code>.  If you are using an git version of the code you will need to do an ./autogen.sh in this source directory (see [[Getting_And_Building_MPICH#Setting_up_the_build_environment|here]].

Now let's say that you want to build two versions, one with a supported thread level of MPI_THREAD_SINGLE and one with MPI_THREAD_MULTIPLE.  You would create two directories that are '''not''' in the source directory like <code>/foo/bar/single</code> and <code>/foo/bar/multiple</code>.  Change to the <code>single</code> directory and run
<pre>
 ../mpich-trunk/configure --prefix=/foo/bar/single-installed --enable-threads=single
</pre>
Then <code>make</code> and <code>make install</code>.

Next change to the <code>multiple</code> directory and issue the same commands but replace "single" with "multiple" everywhere.  You now have two builds of MPICH from the same source tree.

== Platform specific build instructions ==
The following wiki pages contain additional platform-specific build information.
* [[BGQ|Blue Gene/Q]]
* [[PE_RTE|x86 and POWER using the Parallel Environment Runtime Edition]]

<!-- vim: set ft=wikipedia tw=0 : -->