CH3 VC protocol
This document describes the protocol CH3 uses when closing a VC. The close protocol ensures that both sides of the connection are ready to close and all communication has finished.
When a VC is released, and its ref count drops to 0, the VC is closed. Similarly in
MPIDI_PG_Close_VCs() is called which goes through all VCs in all process groups and closes them. A VC is closed by calling
MPIDI_CH3U_VC_WaitForClose() is called to wait for all VCs to finish closing. In
outstanding_close_ops is incremented to keep track of the number of VCs that are in the process of closing. When the close protocol completes for a VC,
outstanding_close_ops is decremented. In this way,
MPIDI_CH3U_VC_WaitForClose() can simply keep calling progress while
outstanding_close_ops is non-zero.
All VCs start in the
INACTIVE state at init time. When a message is sent over a VC the VC is connected by the channel, if necessary, and its state is changed to
Below is the state diagram for the close protocol. There are eight states in the protocol
ACTIVE-- Communication has been initiated and has not yet been completed
LOCAL_CLOSE-- Local process has initiated the close protocol
REMOTE_CLOSE-- Remote process has initiated the close protocol
CLOSE_ACKED-- Local process has initiated the close protocol and has acknowledged the remote process's CLOSE packet
CLOSED-- Local process has received and ack from the remote process
INACTIVE-- Either no communication has been initiated, or the VC has been succesfully closed and resources have been released
INACTIVE_CLOSED-- The process is in finalize and this VC should not be opened
MORIBUND-- This connection has terminated prematurely. There may be outstanding sends or receives on this VC. Currently,
MORIBUNDis a terminal state, but in the future it may be possible to activate the VC again.
In the labels for the transitions, the part above the line indicate the event triggering the transition. The part below the line describes the actions taken in the transition (In the diagram a label for a transition generally appears to the right of the edge). Note that the error transitions are shown with red and green edges to simplify the diagram. A key describing the transitions is shown at the bottom of the diagram.