Nemesis Runtime Netmod Selection

From Mpich
Revision as of 16:18, 10 November 2012 by Balaji (talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

This page explains the mechanism used by Nemesis to select network modules at runtime. With this mechanism the list of network modules is specified in configure and are then statically compiled in to the MPICH library. At runtime the user sets an environment variable to choose which network module to use.

Building and Execution

The --with-device= configure option can be used to specify multiple network modules to be compiled into the library. The list of network modules is specified following ch3:nemesis: as a comma-separated list (with no spaces). The first network module listed will be the default. E.g.:

 ./configure --with-device=ch3:nemesis:tcp,mx

At runtime, the network module is chosen using the MPICH_NEMESIS_NETMOD environment variable. E.g.:

 MPICH_NEMESIS_NETMOD=mx mpiexec -n 4 ./cpi

If MPICH_NEMESIS_NETMOD is not set, the first network module listed in configure is used.


Each network module defines a function pointer table, MPID_nem_netmod_funcs_t. This function pointer table is named MPIDI_nem_modulename_module_funcs. E.g. for the tcp network module, the function pointer table is defined in tcp_init.c as:

   MPID_nem_netmod_funcs_t MPIDI_nem_tcp_funcs = {

MPID_nem_netmod_funcs is an array of the function pointer tables for each configured netmod. It's defined in mpid_nem_net_array.c which is generated by configure from In this file, MPID_nem_netmod_strings is also defined which holds the names of the netmods.

Which netmod to use is determined in MPID_nem_choose_netmod() by reading the MPICH_NEMESIS_NETMOD environment variable and searching for a match in MPID_nem_netmod_strings. Then MPID_nem_netmod_func is set to point to the corresponding entry in the MPID_nem_netmod_funcs array.

Any call to a network module function is done by dereferencing MPID_nem_netmod_func.

The variable MPID_nem_netmod_id is the index of the function pointer table for the current netmod in the MPID_nem_netmod_funcs array: i.e., MPID_nem_netmod_func == MPID_nem_netmod_funcs[MPID_nem_netmod_id].


struct for function pointers for a single netmod (mpid_nem_nets.h)
MPID_nem_netmod_funcs_t MPIDI_nem_MODULENAME_module_funcs
function pointer table for netmod MODULENAME (defined in each netmod)
MPID_nem_netmod_funcs_t *MPID_nem_netmod_funcs[]
array of pointers to function pointer tables from each configured netmod (generated by configure from
char MPID_nem_netmod_strings[][]
Array of string names for each netmod (generated by configure from
MPID_nem_netmod_funcs_t *MPID_nem_netmod_func
global pointer to the function pointer table for the selected netmod. Dereference this to call the netmod function. (mpid_nem_network.c)
environment variable specifying which netmod to use
index of the function pointer table for the current netmod: MPID_nem_netmod_func == MPID_nem_netmod_funcs[MPID_nem_netmod_id] (mpid_nem_network.c)
index of the function pointer table for the netmod MODULENAME (generated by configure from

Future Work

  • Multiple netmods - This can be done having each vc contain a pointer to the appropriate netmod function pointer (that part's easy). How do we decide/specify which netmod to use for which vc?
  • Dynamically loaded netmods