Capability to post receive buffers at any point in time for the Zerocopy Post API

[nitbhat]

The ZC Post API currently supports posting buffers only after receiving the metadata message, in the Post entry method. This is a limitation that needs to be addressed in order to support the posting of buffers at any point in time during execution, irrespective of the arrival of the metadata message from the sender. This will be a useful feature primarily for AMPI (to support the MPI programming model), but also applicable to Charm++.

The API that I have created in the first draft is as follows:
Method called by the user to post a receiver buffer is called CkPostBuffer.
void CkPostBuffer(T *destBuffer, size_t size, int tag);

Method called by the user to match the sender buffer with an already posted (or to be posted in the future) buffer is called CkMatchBuffer. This will be called in the body of the post entry method on the receiver.
void CkMatchBuffer(CkNcpyBufferPost *ncpyPost, int bufferIndex, int tag);

Example:
.ci file declaration (just as before)
entry void recv_zerocopy(nocopypost int buffer[size], size_t size);

.C file definition of the Post Entry method

void recv_zerocopy(int *&buffer, size_t &size, CkNcpyBufferPost *ncpyPost) {
     // 0 indicates the first buffer. If there were two buffers, there would be another CkMatchBuffer with index 1
     CkMatchBuffer(ncpyPost, 0, 23); 
}

.C file call to post the buffer
CkPostBuffer(myDestBuffer, 4000, 23); // 23 is the matching tag

.C file definition of the Regular Entry method (which is executed after the buffer is posted)

void recv_zerocopy(int *buffer, size_t size) {
    // ready to consume the buffer
    compute(myDestBuffer); // note that buffer and myDestBuffer are the same
}

Notes:

1. Tag is a user passed integer into both CkPostBuffer and CkMatchBuffer. Each CkPostBuffer call should have a corresponding CkMatchBuffer call with the same tag on the same PE.

Open Questions:

1. For nodegroups, since the entry method can execute on any PE of the node, it would be good to support the above semantics at a node level. I was thinking that the best way to do this is with corresponding node level variants of the same methods - CkPostNodeBuffer and CkMatchNodeBuffer. Are there better ways of doing this?
2. The existing semantics allow for posting of the buffer on the arrival of the metadata message by assigning the reference to the pointer. Should we continue supporting this (for backward compatibility) or should we change that to look more like this transformed API?

Let me know your questions/thoughts.

[minitu]

Pasting my comments from slack:
I do agree that being able to post a receive only after the metadata arrives on the receiver is a limitation. I think it is a good idea to provide a user-provided tag based mechanism, because the only information necessary to post a ucp_tag_recv_nb in the UCX machine layer to receive GPU data are the destination pointer and size (which can be obtained on the receiver, e.g. from the post entry method) and the tag. Currently the tag is designated by the sender and sent to the receiver inside the metadata (i.e. CkDeviceBuffer), which I believe is the only thing preventing the UCX receive from being posted earlier.