--- name: "fi_msg" section: "3" platform: "linux" package: "libfabric" version: "2.7.0-1" release: "Arch Linux 2026-11-01" platforms: ["linux"] url: "https://new.carta.tech/fi_msg" --- # fi\_msg(3) Message data transfer operations fi\_send / fi\_sendv / fi\_sendmsg fi\_inject / fi\_senddata : Initiate an operation to send a message ## NAME fi\_msg - Message data transfer operations - fi\_recv / fi\_recvv / fi\_recvmsg Post a buffer to receive an incoming message fi\_send / fi\_sendv / fi\_sendmsg fi\_inject / fi\_senddata : Initiate an operation to send a message - fi\_xpu\_send Initiate a send operation from an XPU. - fi\_xpu\_recv Post a receive buffer from an XPU. ## SYNOPSIS - ``` #include ssize_t fi_recv(struct fid_ep *ep, void * buf, size_t len, void *desc, fi_addr_t src_addr, void *context); ssize_t fi_recvv(struct fid_ep *ep, const struct iovec *iov, void **desc, size_t count, fi_addr_t src_addr, void *context); ssize_t fi_recvmsg(struct fid_ep *ep, const struct fi_msg *msg, uint64_t flags); ssize_t fi_send(struct fid_ep *ep, const void *buf, size_t len, void *desc, fi_addr_t dest_addr, void *context); ssize_t fi_sendv(struct fid_ep *ep, const struct iovec *iov, void **desc, size_t count, fi_addr_t dest_addr, void *context); ssize_t fi_sendmsg(struct fid_ep *ep, const struct fi_msg *msg, uint64_t flags); ssize_t fi_inject(struct fid_ep *ep, const void *buf, size_t len, fi_addr_t dest_addr); ssize_t fi_senddata(struct fid_ep *ep, const void *buf, size_t len, void *desc, uint64_t data, fi_addr_t dest_addr, void *context); ssize_t fi_injectdata(struct fid_ep *ep, const void *buf, size_t len, uint64_t data, fi_addr_t dest_addr); #include int fi_xpu_send(struct fid_xpu_ep *ep, const void *buf, size_t len, void *desc, uint64_t data, void *dest_addr, void *context, uint64_t flags, int scope); int fi_xpu_recv(struct fid_xpu_ep *ep, void *buf, size_t len, void *desc, void *src_addr, void *context, uint64_t flags, int scope); ``` ## ARGUMENTS - *ep* Fabric endpoint on which to initiate send or post receive buffer. - *buf* Data buffer to send or receive. For 0-byte operations, buf may be ignored. - *len* Length of data buffer to send or receive, specified in bytes. Valid transfers are from 0 bytes up to the endpoint’s max\_msg\_size. - *iov* Vectored data buffer. For 0-byte operations, iov may be ignored. - *count* Count of vectored data entries. For 0-byte operations, count may be 0. - *desc* Descriptor associated with the data buffer. For 0-byte operations, desc may be ignored even for FI\_MR\_LOCAL. See [fi\_mr(3)](/fi_mr/3.md). - *data* Remote CQ data to transfer with the sent message. - *dest\_addr* Destination address for connectionless transfers. Ignored for connected endpoints. - *src\_addr* Applies only to connectionless endpoints configured with the FI\_DIRECTED\_RECV. For all other endpoint configurations, src\_addr is ignored. src\_addr defines the source address to receive from. By default, the src\_addr is treated as a source endpoint address (i.e. fi\_addr\_t returned from fi\_av\_insert / fi\_av\_insertsvc / fi\_av\_remove). If the FI\_AUTH\_KEY flag is specified with fi\_recvmsg, src\_addr is treated as a source authorization key (i.e. fi\_addr\_t returned from fi\_av\_insert\_auth\_key). If set to FI\_ADDR\_UNSPEC, any source address may match. - *msg* Message descriptor for send and receive operations. - *flags* Additional flags to apply for the send or receive operation. - *context* User specified pointer to associate with the operation. This parameter is ignored if the operation will not generate a successful completion, unless an op flag specifies the context parameter be used for required input. - *scope* Cooperative threading scope for device-side operations. Specifies the set of threads issuing the same operation collectively. Only used by fi\_xpu\_send and fi\_xpu\_recv. See [fi\_xpu(3)](/fi_xpu/3.md) for details. ## DESCRIPTION The send functions – fi\_send, fi\_sendv, fi\_sendmsg, fi\_inject, and fi\_senddata – are used to transmit a message from one endpoint to another endpoint. The main difference between send functions are the number and type of parameters that they accept as input. Otherwise, they perform the same general function. Messages sent using fi\_msg operations are received by a remote endpoint into a buffer posted to receive such messages. The receive functions – fi\_recv, fi\_recvv, fi\_recvmsg – post a data buffer to an endpoint to receive inbound messages. Similar to the send operations, receive operations operate asynchronously. Users should not touch the posted data buffer(s) until the receive operation has completed. An endpoint must be enabled before an application can post send or receive operations to it. For connected endpoints, receive buffers may be posted prior to connect or accept being called on the endpoint. This ensures that buffers are available to receive incoming data immediately after the connection has been established. Completed message operations are reported to the user through one or more event collectors associated with the endpoint. Users provide context which are associated with each operation, and is returned to the user as part of the event completion. See fi\_cq for completion event details. ### fi\_send The call fi\_send transfers the data contained in the user-specified data buffer to a remote endpoint, with message boundaries being maintained. ### fi\_sendv The fi\_sendv call adds support for a scatter-gather list to fi\_send. The fi\_sendv transfers the set of data buffers referenced by the iov parameter to a remote endpoint as a single message. ### fi\_sendmsg The fi\_sendmsg call supports data transfers over both connected and connectionless endpoints, with the ability to control the send operation per call through the use of flags. The fi\_sendmsg function takes a struct fi\_msg as input. - ``` struct fi_msg { const struct iovec *msg_iov; /* scatter-gather array */ void **desc; /* local request descriptors */ size_t iov_count;/* # elements in iov */ fi_addr_t addr; /* optional endpoint address */ void *context; /* user-defined context */ uint64_t data; /* optional message data */ }; ``` For 0-byte operations, msg\_iov, desc (including FI\_MR\_LOCAL) and iov\_count may be ignored. ### fi\_inject The send inject call is an optimized version of fi\_send with the following characteristics. The data buffer is available for reuse immediately on return from the call, and no CQ entry will be written if the transfer completes successfully. If the transfer fails, an error entry will be written to the CQ. Conceptually, this means that the fi\_inject function behaves as if the FI\_INJECT transfer flag were set, selective completions are enabled, and the FI\_COMPLETION flag is not specified. Note that the CQ entry will be suppressed even if the default behavior of the endpoint is to write CQ entries for all successful completions. See the flags discussion below for more details. The requested message size that can be used with fi\_inject is limited by inject\_size. If FI\_HMEM is enabled and the provider requires the FI\_MR\_HMEM mr\_mode, fi\_inject can only accept buffers with iface equal to FI\_HMEM\_SYSTEM. This limitation does not affect how inject\_size is reported. ### fi\_senddata The send data call is similar to fi\_send, but allows for the sending of remote CQ data (see FI\_REMOTE\_CQ\_DATA flag) as part of the transfer. ### fi\_injectdata The inject data call is similar to fi\_inject, but allows for the sending of remote CQ data (see FI\_REMOTE\_CQ\_DATA flag) as part of the transfer. ### fi\_recv The fi\_recv call posts a data buffer to the receive queue of the corresponding endpoint. Posted receives are searched in the order in which they were posted in order to match sends. Message boundaries are maintained. The order in which the receives complete is dependent on the endpoint type and protocol. For connectionless endpoints, the src\_addr parameter can be used to indicate that a buffer should be posted to receive incoming data from a specific remote endpoint. ### fi\_recvv The fi\_recvv call adds support for a scatter-gather list to fi\_recv. The fi\_recvv posts the set of data buffers referenced by the iov parameter to a receive incoming data. ### fi\_recvmsg The fi\_recvmsg call supports posting buffers over both connected and connectionless endpoints, with the ability to control the receive operation per call through the use of flags. The fi\_recvmsg function takes a struct fi\_msg as input. ### fi\_xpu\_send / fi\_xpu\_recv The fi\_xpu\_send and fi\_xpu\_recv calls are device-side equivalents of the host message operations, callable from XPU kernel code. They operate on an exported endpoint handle obtained from fi\_ep\_export\_xpu. The desc parameter is a raw descriptor from fi\_mr\_get\_xpu\_desc with FI\_XPU, and dest\_addr/src\_addr are raw addresses from fi\_av\_lookup2 with FI\_XPU. The scope parameter specifies the cooperative threading scope (FI\_XPU\_WORK\_ITEM, FI\_XPU\_SUBGROUP, FI\_XPU\_WORK\_GROUP, or FI\_XPU\_DEVICE). See [fi\_xpu(3)](/fi_xpu/3.md) for the overall XPU programming model. ## FLAGS The fi\_recvmsg and fi\_sendmsg calls allow the user to specify flags which can change the default message handling of the endpoint. Flags specified with fi\_recvmsg / fi\_sendmsg override most flags previously configured with the endpoint, except where noted (see fi\_endpoint.3). The following list of flags are usable with fi\_recvmsg and/or fi\_sendmsg. - *FI\_REMOTE\_CQ\_DATA* Applies to fi\_sendmsg. Indicates that remote CQ data is available and should be sent as part of the request. See fi\_getinfo for additional details on FI\_REMOTE\_CQ\_DATA. This flag is implicitly set for fi\_senddata and fi\_injectdata. - *FI\_COMPLETION* Indicates that a completion entry should be generated for the specified operation. The endpoint must be bound to a completion queue with FI\_SELECTIVE\_COMPLETION that corresponds to the specified operation, or this flag is ignored. - *FI\_MORE* Indicates that the user has additional requests that will immediately be posted after the current call returns. Use of this flag may improve performance by enabling the provider to optimize its access to the fabric hardware. Providers that utilize delayed start optimizations for communication calls with FI\_MORE flag set must ensure that all previously delayed calls be flushed when an error is returned from a new call. - *FI\_INJECT* Applies to fi\_sendmsg. Indicates that the outbound data buffer should be returned to user immediately after the send call returns, even if the operation is handled asynchronously. This may require that the underlying provider implementation copy the data into a local buffer and transfer out of that buffer. This flag can only be used with messages smaller than inject\_size. If FI\_HMEM is enabled and the provider requires the FI\_MR\_HMEM mr\_mode, the FI\_INJECT flag can only be used with buffers whose iface is FI\_HMEM\_SYSTEM. - *FI\_MULTI\_RECV* Applies to posted receive operations. This flag allows the user to post a single buffer that will receive multiple incoming messages. Received messages will be packed into the receive buffer until the buffer has been consumed. Use of this flag may cause a single posted receive operation to generate multiple events as messages are placed into the buffer. The placement of received data into the buffer may be subjected to provider specific alignment restrictions. The buffer will be released by the provider when the available buffer space falls below the specified minimum (see FI\_OPT\_MIN\_MULTI\_RECV). Note that an entry to the associated receive completion queue will always be generated when the buffer has been consumed, even if other receive completions have been suppressed (i.e. the Rx context has been configured for FI\_SELECTIVE\_COMPLETION). See the FI\_MULTI\_RECV completion flag [fi\_cq(3)](/fi_cq/3.md). - *FI\_INJECT\_COMPLETE* Applies to fi\_sendmsg. Indicates that a completion should be generated when the source buffer(s) may be reused. - *FI\_TRANSMIT\_COMPLETE* Applies to fi\_sendmsg and fi\_recvmsg. For sends, indicates that a completion should not be generated until the operation has been successfully transmitted and is no longer being tracked by the provider. For receive operations, indicates that a completion may be generated as soon as the message has been processed by the local provider, even if the message data may not be visible to all processing elements. See [fi\_cq(3)](/fi_cq/3.md) for target side completion semantics. - *FI\_DELIVERY\_COMPLETE* Applies to fi\_sendmsg. Indicates that a completion should be generated when the operation has been processed by the destination. - *FI\_FENCE* Applies to transmits. Indicates that the requested operation, also known as the fenced operation, and any operation posted after the fenced operation will be deferred until all previous operations targeting the same peer endpoint have completed. Operations posted after the fencing will see and/or replace the results of any operations initiated prior to the fenced operation. The ordering of operations starting at the posting of the fenced operation (inclusive) to the posting of a subsequent fenced operation (exclusive) is controlled by the endpoint’s ordering semantics. - *FI\_MULTICAST* Applies to transmits. This flag indicates that the address specified as the data transfer destination is a multicast address. This flag must be used in all multicast transfers, in conjunction with a multicast fi\_addr\_t. - *FI\_AUTH\_KEY* Only valid with domains configured with FI\_AV\_AUTH\_KEY and connectionless endpoints configured with FI\_DIRECTED\_RECV. When used with fi\_recvmsg, this flag denotes that the src\_addr is an authorization key fi\_addr\_t instead of an endpoint fi\_addr\_t. ## NOTES If an endpoint has been configured with FI\_MSG\_PREFIX, the application must include buffer space of size msg\_prefix\_size, as specified by the endpoint attributes. The prefix buffer must occur at the start of the data referenced by the buf parameter, or be referenced by the first IO vector. Message prefix space cannot be split between multiple IO vectors. The size of the prefix buffer should be included as part of the total buffer length. ## RETURN VALUE Returns 0 on success. On error, a negative value corresponding to fabric errno is returned. Fabric errno values are defined in rdma/fi\_errno.h. See the discussion below for details handling FI\_EAGAIN. ## ERRORS - *-FI\_EAGAIN* Indicates that the underlying provider currently lacks the resources needed to initiate the requested operation. The reasons for a provider returning FI\_EAGAIN are varied. However, common reasons include insufficient internal buffering or full processing queues. Insufficient internal buffering is often associated with operations that use FI\_INJECT. In such cases, additional buffering may become available as posted operations complete. Full processing queues may be a temporary state related to local processing (for example, a large message is being transferred), or may be the result of flow control. In the latter case, the queues may remain blocked until additional resources are made available at the remote side of the transfer. In all cases, the operation may be retried after additional resources become available. When using FI\_PROGRESS\_MANUAL, the application must check for transmit and receive completions after receiving FI\_EAGAIN as a return value, independent of the operation which failed. This is also strongly recommended when using FI\_PROGRESS\_AUTO, as acknowledgements or flow control messages may need to be processed in order to resume execution. ## SEE ALSO [fi\_getinfo(3)](/fi_getinfo/3.md), [fi\_endpoint(3)](/fi_endpoint/3.md), [fi\_domain(3)](/fi_domain/3.md), [fi\_cq(3)](/fi_cq/3.md) ## AUTHORS OpenFabrics.