Testing MPICH

From Mpich
Revision as of 13:59, 25 July 2008 by Gropp (talk | contribs) (Running the configure tests)

Jump to: navigation, search

Testing MPICH2

Using the MPI test suites with MPICH2

Basic tests can be performed using the MPICH2 tests

make testing

The results are in test/summary.xml. The XML style file for these results is in test/TestResults.xsl.

To run the full set of tests and add the results to the [[http://www-unix.mcs.anl.gov/mpi/mpich1/micronotes/mpich2-status/index.htm%7Ctest results web page]], simply run /home/MPI/nightly/basictests . This will not update the web page however. Run /home/MPI/nightly/updatesum to update the test results web page. To customize the tests run by basictests, the following command-line options may be used:

Selects the combinations of process managers and ADI implementations to test. The default is
mpd-ch3 rmpd-ch3 mpd-ch3:ssm mpd-ch3:shm gforker-ch3 random
-cc=name, -cxx=name, -fc=name, -f90=name

Specifies different compilers for C, C++, Fortran 77, and Fortran 90


Sets the "architecture name". This option should be used to ensure that the results are given a unique output name. For example


for IA32 Linux using gcc version 3.1 . The file name for the results is made up of the arch, process manager, device, and date. This option is particularly important because the columns of the web page are determined by this option.


Add the "soft" option. For example, to use the PGI compilers at MCS, you need to execute the command

    soft add +pgi

This option causes basictests to perform this step


Sets an alternate location for the source of MPICH2 (the default is /home/MPI/testing/mpich2/mpich2).


Sets an alternate location for the output files. The default is /home/MPI/nightly/runs.


Selects the tests to run. The default is to run all four test sets and is equivalent to specifying -tests=mpich:mpicxx:intel:mpich2.

For example, to run the tests on your own copy of MPICH2, storing the data in a private directory, for just the mpd process manager and ch3 device, use

    /home/MPI/nightly/basictests -mpich2dir=/home/me/mympich2 \
                                 -datadir=/home/me/myresults \

To do: add information on using all five test suites, including the MPI I/O test suite.

Example uses of basictests

Basic use

Run the tests and let basictests choose the compilers and the test name (used to label a column on the web page)

Alternate Compilers

This example shows how to select the Portland Group compilers. Note the use of the -soft option to ensure that the correct paths are set up for using Portland Group compilers and the use of the -arch option to give this test a unique name

   /home/MPI/nightly/basictests -soft=pgi-5.2 -cc=pgcc -fc=pgf77 -cxx=pgCC -f90=pgf90 -arch=IA32-Linux-PG

Changing the default tests

The tests that are run by default are choosen in the script /home/MPI/nightly/basictests. To add a new architecture to the tests, simply run basictests on a new platform; the results will be automatically added to the web page [[1]].

To add a new device and/or process manager, you need to edit two files. In /home/MPI/nightly/basictests, find the line

tables="mpd-ch3 mpd-ch3:ssm mpd-ch3:shm mpd-ch3:sshm gforker-ch3 random"

Add the process manager and device pair that you want to test, following the form processmanager-device. For a device that has additional options, add these with a colon after the device name. For example, to add testing with the gforker process manager for the shared memory version of the ch3 device, add gforker-ch3:shm to tables:

tables="mpd-ch3 mpd-ch3:ssm mpd-ch3:shm mpd-ch3:sshm gforker-ch3 gforker-ch3:shm random"

You must also update /home/MPI/nightly/updatesum to add the new test to the web page. Add the same entry to the @tables array in updatesum. As this is a perl script, the syntax is slightly different. Continuing with the example above, we add "gforker-ch3:shm" to @tables:

@tables = ( "mpd-ch3",
            "mpd-ch3:ssm", "mpd-ch3:shm", "mpd-ch3:sshm",
            "random" );

That's it. The next time basictests runs, the new process manager and device pair will be added to the tests, and the web page will reflect the new tests.

Location of test output

The output of the tests run by basictests is copied into the directory /home/MPI/nightly/runs in a file with a name that contains the architecture, process manager, device, date, and test name. For example, the files for IA32 under Linux, with the GNU compilers, using the MPD process manager and the ch3 device, and started on April 1st, 2005 at 2200 hours, are


The first of these is the entire output file. The second contains only the output of make that was might contain warnings or errors about the make. The remaining eight files contain the test results and the failure-only results (the files with the -fail extension).

If more details are required, you will need to check the directory in which the tests were run. By default (see basictests), this is in /sandbox/USER/cb, where USER is the name of the person that ran the tests.

Running the special tests

The [tests] are run with the script /home/MPI/nightly/specialtest -dir=rundir, where rundir is the directory in which specialtest writes the log files. The -help option to specialtest describes the options that can be used to run a subset of tests or to select different combinations of process managers or devices.

For example, to run the tests on your own copy of MPICH2 and place the results in a private location, use

    /home/MPI/nightly/specialtest -srcdir=/home/me/mympich2 \
                                  -dir=/sandbox/me \

The results can then be viewed by opening /home/me/mytests/index.htm in a web browser. <p> To update the special tests output on the general web page, simply run


This will use the MPICH2 source in /home/MPI/testing/mpich2/mpich2.

Running the configure tests

The [| configure tests] are run with the script /home/MPI/nightly/testoptions.

To update the special tests output on the general web page, simply run


This will use the MPICH2 source in /home/MPI/testing/mpich2/mpich2.

Running the coding checks

To run the coding checks, change to a directory that contains MPICH2 and execute the following command:

     /home/gropp/projects/software/buildsys/src/codingcheck -conf=mpich2 \
        -skipfiles=src/mpid/globus,src/mpe2,src/mpid/rdma \
        -checktest src examples test

(This assumes that you are running on one of the MCS division Linux systems; you can install the buildsys tools and using codingcheck from your own installation if you need to run the coding check on a different system). The report of problems detected is sent to standard output. The above command will produce a relatively comprehensive report. See the use of the codingcheck command in /home/MPI/nightly/updatetodo for the current command use to update the web page; this command may skip additional directories and may use additional options (such as -rmchecks=cppdefines) to tune the output.

An updated report is generated every morning as one of the steps performed by /home/MPI/nightly/updatetodo and placed into /mcs/www/mpi/mpi/mpich2/todo/coding-problems.txt (accessible as [[2]]).

An explanation of some of the rules that are applied by the style checker are in the coding standards web pages.

Finding Specific Classes of Problems

Currently, the coding checker produces a unified report about all problems. To find specific problems, you can use grep or view the report (it is a text file) in your favorite editor. Some simple examples are:

Find errors

These are uses that are viewed as erroneous. In many cases, these are uses of either MPI or PMPI routines where NMPI names must be used (NMPI routines ensure that errors are properly handled).

     grep 'Error:' /mcs/www/mpi/mpich2/todo/coding-problems.txt
Find non-conforming ifdef names

These find C preprocessor names that may conflict with other system header files

    grep Warning: /mcs/www/mpi/mpich2/todo/coding-problems.txt | egrep 'ifn?def'
Find missing style headers

Each file should have a style header to avoid unnecessary reformatting when the file is edited (e.g., to maintain a consistent level of indentation).

   grep 'style header missing' /mcs/www/mpi/mpich2/todo/coding-problems.txt
Find suspicious function usage

There are a number of functions, such as strcat, that should never appear in good code because of the potential for memory overwrites (even if the code is surrounded by length checks, any reader of the code must take the time to ensure that those checks are correct; it is better to avoid them). Other routines, such as malloc, interfere with debugging tools that are built into the code. Yet other functions, such as setvbuf or snprintf, are either not portable to all operating systems or are only part of C99 or later, and may pose portability problems for users with older (e.g., C90) compilers.

   grep 'Warning: found ' /mcs/www/mpi/mpich2/todo/coding-problems.txt
   grep 'Caution: found ' /mcs/www/mpi/mpich2/todo/coding-problems.txt
Find missing Copyright statements

To get a quick list of files that are missing a copyright statement:

   grep 'Copyright statement missing' /mcs/www/mpi/mpich2/todo/coding-problems.txt
Find files that use a particular function

For example, to find all files that use malloc

   grep malloc /mcs/www/mpi/mpich2/todo/coding-problems.txt | grep -v Warning:

Running the global symbols checks

To update the web page that reports on the global symbols, run


Like the other tests, this works with the source files in /home/MPI/testing/mpich2/mpich2. The program itself is a simple perl script; if you wish to change the tests for yourself, make a copy and update the values of the following variables, found near the top of the file:


Location of MPICH2 source


Directory to be used to build MPICH2 as part of identifying global symbols


Direction into which the result web page is placed


Location of program that is used to check for global symbols, along with any command-line arguments


Set to 1 if the configure and make log files should be copied to $reportdir

There are several easy ways to get a quick look at the global symbols. The easiest is to use the nm program on libmpich.a. However, you will need to filter the output to find any non-conforming names. For example, under Linux,

    nm libmpich.a | grep " C " | grep -v mpipriv

will show the names of any uninitialized global variables.

Finding non-conforming names is harder, primarily because the list of allowed prefixes is fairly large. Instead of trying to list all prefixes and using grep -v, you can use

    /homes/gropp/projects/software/buildsys/src/checkforglobs -mpich2 libmpich.a

When testing for global symbols, make sure that you include tests with weak symbols disabled to ensure that any internal routines that are static only when weak symbols are supported have conforming names.

Running and understanding the coverage tests

The easiest way to run the coverage tests is with the script /home/MPI/nightly/getcoverage. This uses the version of MPICH2 that is updated every night in /home/MPI/testing/mpich2/mpich2, and uses the MPICH2 test suite and the three additional test suites that are in /home/MPI/testing/tsuites. Run this script with

    /home/MPI/nightly/getcoverage -updateweb

The best way to view the coverage data for the entire project is through the <a href="http://www-unix.mcs.anl.gov/mpi/mpich2/todo/coverage/index.htm">web page</a>. Code with a blue background has been ignored in counting covered and uncovered lines; this is typically error handling and reporting code or experimental code that is not expected to be executed. Code with a red background is code that should have been executed by the tests but was not (in some cases, this code was not marked as error handling or reporting code; such code should be fixed by annotating the source code to mark the code as error handling).

To update the coverage information for a single file, the easiest approach is to use /home/MPI/nightly/getcoverage to create the correct versions of the libraries and get the initial coverage results. To add to the results, simply compile a program with the coverage-enabled libraries (by default, this is in /sandbox/$LOGNAME/mpich2-cov) and run it. This will cause the coverage data files (files with extensions da) to be updated. To generate a text file describing the coverage, you can normally use

    gcov foo.c

in the directory that contains the file foo.da (you'll also see the files foo.bb and foo.bbg). This will produce a file foo.c.gcov. Lines that are marked with ##### have not be executed (see man gcov for more information on the gcov tool).

When running the /home/MPI/nightly/getcoverage tool from a cron job, the following options are useful:

  • -quiet - Suppresses all output
  • -logfile=name - Sends all output to the named file.

If -updateweb is also selected, then the -logfile option also copies the contents of the logfile into logfile.txt in the designated web directory.

Troubleshooting the tests

Stopping the tests

Several of the test suites allow you to stop them by creating a file. When the test suite detects the existance of the file, the tests are aborted. A good way to create these files is with date so that it is easy to see when the file was originally created.




.stoptest in the top-level testing directory (e.g., mpich2/test/mpi

Sources for the Test Suites

The test suites should be accessible through the <a href="http://www.mcs.anl.gov/mpi/mpi-test/tsuite.html">test suite web page</a>. Local to Argonne, the test suites are available from the CVS repository:

Test SuiteCVS RepositoryCVS Module
LLNL I/O/home/MPI/cvsMasterTestmpio

Running the LLNL I/O Test

To run the LLNL I/O test, do the following:

  1. Get a current version. For example,
    cvs -d /home/MPI/cvsMaster export -D now Testmpio
  1. Update the Makefile. For MPICH2, all you should need to

do is to reset the value of the variable MPIHOME.

  1. Make a t1 subdirectory in the test directory:
    mkdir t1

Without this step, the tests will fail with "invalid file name" messages.

  1. Run the test with make testing. If you have trouble with

this (for example, it fails if you direct the output into a file), run the test manually as follows (assuming a C-shell):

    setenv MPIO_USER_PATH `pwd`/t1
    mpiexec -n 4 testmpio 1 |& tee test.log
  1. Interpret the results. This test does not provide a pass/fail summary,
so you will need to examine the output for problems.