Difference between revisions of "Thread Private Storage"

From Mpich
Jump to: navigation, search
 
Line 1: Line 1:
 
<!-- ported from http://www-unix.mcs.anl.gov/mpi/mpichold/developer/design/threadprivate.htm -->
 
<!-- ported from http://www-unix.mcs.anl.gov/mpi/mpichold/developer/design/threadprivate.htm -->
 +
 +
While the MPI standard is fairly careful to be thread-safe, and in particular to avoid the need for thread-private state, there are a few places where the MPICH2 implementation may need to storage that is specific to each thread. These places include
 +
 +
* Error return from predefined MPI reduction operations
 +
* Nesting count used to ensure that <code>MPI_ERRORS_RETURN</code> is used when an MPI routine is used internally (e.g., if a routine in MPICH2 calls <code>MPI_Send</code>, in case of an error, <code>MPI_Send</code> should return and not invoke the error handler, since we will want the calling routine to handle the error).
 +
* Performance counters
 +
 +
These are only needed in a relatively few places, and not in most of the performance-critical paths.
 +
 +
Unfortunately, access to thread-private (or thread-specific) storage is not easy in C or C++, since the compiler knows nothing about threads. At least one compiler, the C compiler for the NEC SX-4, provides a pragma to mark a variable as thread-private, making it somewhat easier to make use of thread-private data. In most cases, however, it is necessary to call a routine provided by the thread package to access thread-private data; this data is usually identified by an integer id. For performance reasons, we'll want to make only one call to access the thread-private storage within a routine (since the thread-private data will have to be assigned to a variable that is allocated on the stack, since that is the only thread-private data on which we can depend).
 +
 +
These considerations lead to the following design:
 +
 +
# There is one structure, of type <code>MPICH_PerThread_t</code> (see <code>src/include/mpiimpl.h</code>) that contains all thread-private data. The thread-private data that we will have the thread package manage is a pointer to this storage, which will be allocated on first use.
 +
# The id for this storage is in the <code>MPIR_Process</code> structure, of type <code>MPIR_PerProcess_t</code>, which is also defined in <code>src/include/mpiimpl.h</code>. The id is in the thread_storage field.
 +
# In any routine that needs access to the thread-private data, the following macros are used. These are macros so that they can be rendered into null statements when single-threaded versions of MPICH2 are built.
 +
 +
;MPIU_THREADPRIV_DECL
 +
:  Used the declare the variables needed to access the thread-private storage. To enable a check that the thread-private storage has been set, this pointer is initialized to zero.
 +
;MPIU_THREADPRIV_GET
 +
:  Used to acquire the (pointer to) the thread-private structure. This must be called before any of the routines that may access the thread-private storage.
 +
;MPIU_THREADPRIV_FIELD
 +
:  Use to access a field within the per-thread instance of <code>MPICH_PerThread_t</code>. This is used by the implementations of the "nest" macros, the error returns for the collective operations, and the performance statistics collection. This macro allows the single-threaded code to use a statically allocated <code>MPIR_Thread</code> variable while the multi-threaded code must use a pointer to the thread-private storage:
 +
#ifdef MPICH_MULTI_THREADED
 +
#define MPIU_THREADPRIV_FIELD(name) MPIR_Thread->name
 +
#else
 +
#define MPIU_THREADPRIV_FIELD(name) MPIR_Thread.name
 +
#endif
 +
:  This ensures the best performance in the single threaded case by avoiding the extra indirection that is unnecessary in that case.
 +
;MPIU_THREADPRIV_INITKEY
 +
:  This macro is used to initialize the key that is used to identify the thread-private storage. This macro must be invoked early in <code>MPI_Init</code> (or <code>MPI_Init_thread</code>) to ensure that the key is available to all subsequently created threads.
 +
;MPIU_THREADPRIV_INIT
 +
:  This macro is used to allocate and initialize the thread-private storage. It is used within the definition of <code>MPIU_THREADPRIV_GET</code> to handle the first use of thread-private storage within a thread and in the <code>MPI_Init</code> routine to make sure that the storage is allocated for the master thread.
 +
;MPE_Thread_tls_t
 +
:  The type of the key for the thread-private storage. It may be initialized to the value <code>MPE_THREAD_TLS_T_NULL</code>.
 +
 +
A special routine, <code>MPIR_GetPerThread</code>, returns as its argument a pointer to the thread-private storage for <code>MPIR_Thread</code>.
 +
  
 
[[Category:Design Documents]]
 
[[Category:Design Documents]]
  
 
<!-- vim: set ft=wikipedia : -->
 
<!-- vim: set ft=wikipedia : -->

Revision as of 19:03, 9 January 2008


While the MPI standard is fairly careful to be thread-safe, and in particular to avoid the need for thread-private state, there are a few places where the MPICH2 implementation may need to storage that is specific to each thread. These places include

  • Error return from predefined MPI reduction operations
  • Nesting count used to ensure that MPI_ERRORS_RETURN is used when an MPI routine is used internally (e.g., if a routine in MPICH2 calls MPI_Send, in case of an error, MPI_Send should return and not invoke the error handler, since we will want the calling routine to handle the error).
  • Performance counters

These are only needed in a relatively few places, and not in most of the performance-critical paths.

Unfortunately, access to thread-private (or thread-specific) storage is not easy in C or C++, since the compiler knows nothing about threads. At least one compiler, the C compiler for the NEC SX-4, provides a pragma to mark a variable as thread-private, making it somewhat easier to make use of thread-private data. In most cases, however, it is necessary to call a routine provided by the thread package to access thread-private data; this data is usually identified by an integer id. For performance reasons, we'll want to make only one call to access the thread-private storage within a routine (since the thread-private data will have to be assigned to a variable that is allocated on the stack, since that is the only thread-private data on which we can depend).

These considerations lead to the following design:

  1. There is one structure, of type MPICH_PerThread_t (see src/include/mpiimpl.h) that contains all thread-private data. The thread-private data that we will have the thread package manage is a pointer to this storage, which will be allocated on first use.
  2. The id for this storage is in the MPIR_Process structure, of type MPIR_PerProcess_t, which is also defined in src/include/mpiimpl.h. The id is in the thread_storage field.
  3. In any routine that needs access to the thread-private data, the following macros are used. These are macros so that they can be rendered into null statements when single-threaded versions of MPICH2 are built.
MPIU_THREADPRIV_DECL
Used the declare the variables needed to access the thread-private storage. To enable a check that the thread-private storage has been set, this pointer is initialized to zero.
MPIU_THREADPRIV_GET
Used to acquire the (pointer to) the thread-private structure. This must be called before any of the routines that may access the thread-private storage.
MPIU_THREADPRIV_FIELD
Use to access a field within the per-thread instance of MPICH_PerThread_t. This is used by the implementations of the "nest" macros, the error returns for the collective operations, and the performance statistics collection. This macro allows the single-threaded code to use a statically allocated MPIR_Thread variable while the multi-threaded code must use a pointer to the thread-private storage:
#ifdef MPICH_MULTI_THREADED
#define MPIU_THREADPRIV_FIELD(name) MPIR_Thread->name
#else
#define MPIU_THREADPRIV_FIELD(name) MPIR_Thread.name
#endif
This ensures the best performance in the single threaded case by avoiding the extra indirection that is unnecessary in that case.
MPIU_THREADPRIV_INITKEY
This macro is used to initialize the key that is used to identify the thread-private storage. This macro must be invoked early in MPI_Init (or MPI_Init_thread) to ensure that the key is available to all subsequently created threads.
MPIU_THREADPRIV_INIT
This macro is used to allocate and initialize the thread-private storage. It is used within the definition of MPIU_THREADPRIV_GET to handle the first use of thread-private storage within a thread and in the MPI_Init routine to make sure that the storage is allocated for the master thread.
MPE_Thread_tls_t
The type of the key for the thread-private storage. It may be initialized to the value MPE_THREAD_TLS_T_NULL.

A special routine, MPIR_GetPerThread, returns as its argument a pointer to the thread-private storage for MPIR_Thread.