--- name: "fi_tinjectdata" section: "3" platform: "linux" package: "libfabric" version: "2.7.0-1" release: "Arch Linux 2026-11-01" platforms: ["linux"] url: "https://new.carta.tech/fi_tagged" --- # fi\_tinjectdata(3) Tagged data transfer operations Part of [fi\_tagged(3)](/fi_tagged/3.md). ## NAME fi\_tagged - Tagged data transfer operations - fi\_trecv / fi\_trecvv / fi\_trecvmsg Post a buffer to receive an incoming message - fi\_tsend / fi\_tsendv / fi\_tsendmsg / fi\_tinject / fi\_tsenddata Initiate an operation to send a message - fi\_xpu\_tsend Initiate a tagged send operation from an XPU. - fi\_xpu\_trecv Post a tagged receive buffer from an XPU. ## SYNOPSIS - ``` #include ssize_t fi_trecv(struct fid_ep *ep, void *buf, size_t len, void *desc, fi_addr_t src_addr, uint64_t tag, uint64_t ignore, void *context); ssize_t fi_trecvv(struct fid_ep *ep, const struct iovec *iov, void **desc, size_t count, fi_addr_t src_addr, uint64_t tag, uint64_t ignore, void *context); ssize_t fi_trecvmsg(struct fid_ep *ep, const struct fi_msg_tagged *msg, uint64_t flags); ssize_t fi_tsend(struct fid_ep *ep, const void *buf, size_t len, void *desc, fi_addr_t dest_addr, uint64_t tag, void *context); ssize_t fi_tsendv(struct fid_ep *ep, const struct iovec *iov, void **desc, size_t count, fi_addr_t dest_addr, uint64_t tag, void *context); ssize_t fi_tsendmsg(struct fid_ep *ep, const struct fi_msg_tagged *msg, uint64_t flags); ssize_t fi_tinject(struct fid_ep *ep, const void *buf, size_t len, fi_addr_t dest_addr, uint64_t tag); ssize_t fi_tsenddata(struct fid_ep *ep, const void *buf, size_t len, void *desc, uint64_t data, fi_addr_t dest_addr, uint64_t tag, void *context); ssize_t fi_tinjectdata(struct fid_ep *ep, const void *buf, size_t len, uint64_t data, fi_addr_t dest_addr, uint64_t tag); #include int fi_xpu_tsend(struct fid_xpu_ep *ep, const void *buf, size_t len, void *desc, uint64_t data, void *dest_addr, uint64_t tag, void *context, uint64_t flags, int scope); int fi_xpu_trecv(struct fid_xpu_ep *ep, void *buf, size_t len, void *desc, void *src_addr, uint64_t tag, uint64_t ignore, void *context, uint64_t flags, int scope); ``` ## ARGUMENTS - *fid* Fabric endpoint on which to initiate tagged communication operation. - *buf* Data buffer to send or receive. For 0-byte operations, buf may be ignored. - *len* Length of data buffer to send or receive, specified in bytes. Valid transfers are from 0 bytes up to the endpoint’s max\_msg\_size. - *iov* Vectored data buffer. For 0-byte operations, iov may be ignored. - *count* Count of vectored data entries. For 0-byte operations, count may be 0. - *tag* Tag associated with the message. - *ignore* Mask of bits to ignore applied to the tag for receive operations. - *desc* Memory descriptor associated with the data buffer. For 0-byte operations, desc may be ignored even for FI\_MR\_LOCAL. See [fi\_mr(3)](/fi_mr/3.md). - *data* Remote CQ data to transfer with the sent data. - *dest\_addr* Destination address for connectionless transfers. Ignored for connected endpoints. - *src\_addr* Applies only to connectionless endpoints configured with the FI\_DIRECTED\_RECV. For all other endpoint configurations, src\_addr is ignored. src\_addr defines the source address to receive from. By default, the src\_addr is treated as a source endpoint address (i.e. fi\_addr\_t returned from fi\_av\_insert / fi\_av\_insertsvc / fi\_av\_remove). If the FI\_AUTH\_KEY flag is specified with fi\_trecvmsg, src\_addr is treated as a source authorization key (i.e. fi\_addr\_t returned from fi\_av\_insert\_auth\_key). If set to FI\_ADDR\_UNSPEC, any source address may match. - *msg* Message descriptor for send and receive operations. - *flags* Additional flags to apply for the send or receive operation. - *context* User specified pointer to associate with the operation. This parameter is ignored if the operation will not generate a successful completion, unless an op flag specifies the context parameter be used for required input. - *scope* Cooperative threading scope for device-side operations. Specifies the set of threads issuing the same operation collectively. Only used by fi\_xpu\_tsend and fi\_xpu\_trecv. See [fi\_xpu(3)](/fi_xpu/3.md) for details. ## DESCRIPTION Tagged messages are data transfers which carry a key or tag with the message buffer. The tag is used at the receiving endpoint to match the incoming message with a corresponding receive buffer. Message tags match when the receive buffer tag is the same as the send buffer tag with the ignored bits masked out. This can be stated as: - ``` send_tag & ~ignore == recv_tag & ~ignore or send_tag | ignore == recv_tag | ignore ``` In general, message tags are checked against receive buffers in the order in which messages have been posted to the endpoint. See the ordering discussion below for more details. The send functions – fi\_tsend, fi\_tsendv, fi\_tsendmsg, fi\_tinject, and fi\_tsenddata – are used to transmit a tagged message from one endpoint to another endpoint. The main difference between send functions are the number and type of parameters that they accept as input. Otherwise, they perform the same general function. The receive functions – fi\_trecv, fi\_trecvv, fi\_recvmsg – post a data buffer to an endpoint to receive inbound tagged messages. Similar to the send operations, receive operations operate asynchronously. Users should not touch the posted data buffer(s) until the receive operation has completed. Posted receive buffers are matched with inbound send messages based on the tags associated with the send and receive buffers. An endpoint must be enabled before an application can post send or receive operations to it. For connected endpoints, receive buffers may be posted prior to connect or accept being called on the endpoint. This ensures that buffers are available to receive incoming data immediately after the connection has been established. Completed message operations are reported to the user through one or more event collectors associated with the endpoint. Users provide context which are associated with each operation, and is returned to the user as part of the event completion. See fi\_cq for completion event details. ### fi\_tsend The call fi\_tsend transfers the data contained in the user-specified data buffer to a remote endpoint, with message boundaries being maintained. The local endpoint must be connected to a remote endpoint or destination before fi\_tsend is called. Unless the endpoint has been configured differently, the data buffer passed into fi\_tsend must not be touched by the application until the fi\_tsend call completes asynchronously. ### fi\_tsendv The fi\_tsendv call adds support for a scatter-gather list to fi\_tsend. The fi\_sendv transfers the set of data buffers referenced by the iov parameter to a remote endpoint as a single message. ### fi\_tsendmsg The fi\_tsendmsg call supports data transfers over both connected and connectionless endpoints, with the ability to control the send operation per call through the use of flags. The fi\_tsendmsg function takes a struct fi\_msg\_tagged as input. - ``` struct fi_msg_tagged { const struct iovec *msg_iov; /* scatter-gather array */ void *desc; /* data descriptor */ size_t iov_count;/* # elements in msg_iov */ fi_addr_t addr; /* optional endpoint address */ uint64_t tag; /* tag associated with message */ uint64_t ignore; /* mask applied to tag for receives */ void *context; /* user-defined context */ uint64_t data; /* optional immediate data */ }; ``` For 0-byte operations, msg\_iov, desc (including FI\_MR\_LOCAL) and iov\_count may be ignored. ### fi\_tinject The tagged inject call is an optimized version of fi\_tsend. It provides similar completion semantics as fi\_inject [fi\_msg(3)](/fi_msg/3.md). If FI\_HMEM is enabled and the provider requires the FI\_MR\_HMEM mr\_mode, the fi\_tinject call can only accept buffers with iface equal to FI\_HMEM\_SYSTEM. This limitation does not affect how inject\_size is reported. ### fi\_tsenddata The tagged send data call is similar to fi\_tsend, but allows for the sending of remote CQ data (see FI\_REMOTE\_CQ\_DATA flag) as part of the transfer. ### fi\_tinjectdata The tagged inject data call is similar to fi\_tinject, but allows for the sending of remote CQ data (see FI\_REMOTE\_CQ\_DATA flag) as part of the transfer. ### fi\_trecv The fi\_trecv call posts a data buffer to the receive queue of the corresponding endpoint. Posted receives are searched in the order in which they were posted in order to match sends. Message boundaries are maintained. The order in which the receives complete is dependent on the endpoint type and protocol. ### fi\_trecvv The fi\_trecvv call adds support for a scatter-gather list to fi\_trecv. The fi\_trecvv posts the set of data buffers referenced by the iov parameter to a receive incoming data. ### fi\_trecvmsg The fi\_trecvmsg call supports posting buffers over both connected and connectionless endpoints, with the ability to control the receive operation per call through the use of flags. The fi\_trecvmsg function takes a struct fi\_msg\_tagged as input. ### fi\_xpu\_tsend / fi\_xpu\_trecv The fi\_xpu\_tsend and fi\_xpu\_trecv calls are device-side equivalents of the host tagged message operations. They operate on an exported endpoint handle and include tag matching parameters. The tag and ignore parameters function identically to their host-side counterparts. See [fi\_xpu(3)](/fi_xpu/3.md) for the overall XPU programming model. ## FLAGS The fi\_trecvmsg and fi\_tsendmsg calls allow the user to specify flags which can change the default message handling of the endpoint. Flags specified with fi\_trecvmsg / fi\_tsendmsg override most flags previously configured with the endpoint, except where noted (see fi\_endpoint). The following list of flags are usable with fi\_trecvmsg and/or fi\_tsendmsg. - *FI\_REMOTE\_CQ\_DATA* Applies to fi\_tsendmsg. Indicates that remote CQ data is available and should be sent as part of the request. See fi\_getinfo for additional details on FI\_REMOTE\_CQ\_DATA. This flag is implicitly set for fi\_tsenddata and fi\_tinjectdata. - *FI\_COMPLETION* Indicates that a completion entry should be generated for the specified operation. The endpoint must be bound to a completion queue with FI\_SELECTIVE\_COMPLETION that corresponds to the specified operation, or this flag is ignored. - *FI\_MORE* Indicates that the user has additional requests that will immediately be posted after the current call returns. Use of this flag may improve performance by enabling the provider to optimize its access to the fabric hardware. Providers that utilize delayed start optimizations for communication calls with FI\_MORE flag set must ensure that all previously delayed calls be flushed when an error is returned from a new call. - *FI\_INJECT* Applies to fi\_tsendmsg. Indicates that the outbound data buffer should be returned to user immediately after the send call returns, even if the operation is handled asynchronously. This may require that the underlying provider implementation copy the data into a local buffer and transfer out of that buffer. This flag can only be used with messages smaller than inject\_size. If FI\_HMEM is enabled and the provider requires the FI\_MR\_HMEM mr\_mode, the FI\_INJECT flag can only be used with buffers whose iface is FI\_HMEM\_SYSTEM. - *FI\_MULTI\_RECV* Applies to posted tagged receive operations when the FI\_TAGGED\_MULTI\_RECV capability is enabled. This flag allows the user to post a single tagged receive buffer that will receive multiple incoming messages. Received messages will be packed into the receive buffer until the buffer has been consumed. Use of this flag may cause a single posted receive operation to generate multiple events as messages are placed into the buffer. The placement of received data into the buffer may be subjected to provider specific alignment restrictions. The buffer will be released by the provider when the available buffer space falls below the specified minimum (see FI\_OPT\_MIN\_MULTI\_RECV). Note that an entry to the associated receive completion queue will always be generated when the buffer has been consumed, even if other receive completions have been suppressed (i.e. the Rx context has been configured for FI\_SELECTIVE\_COMPLETION). See the FI\_MULTI\_RECV completion flag [fi\_cq(3)](/fi_cq/3.md). - *FI\_INJECT\_COMPLETE* Applies to fi\_tsendmsg. Indicates that a completion should be generated when the source buffer(s) may be reused. - *FI\_TRANSMIT\_COMPLETE* Applies to fi\_tsendmsg. Indicates that a completion should not be generated until the operation has been successfully transmitted and is no longer being tracked by the provider. - *FI\_MATCH\_COMPLETE* Applies to fi\_tsendmsg. Indicates that a completion should be generated only after the message has either been matched with a tagged buffer or was discarded by the target application. - *FI\_FENCE* Applies to transmits. Indicates that the requested operation, also known as the fenced operation, and any operation posted after the fenced operation will be deferred until all previous operations targeting the same peer endpoint have completed. Operations posted after the fencing will see and/or replace the results of any operations initiated prior to the fenced operation. The ordering of operations starting at the posting of the fenced operation (inclusive) to the posting of a subsequent fenced operation (exclusive) is controlled by the endpoint’s ordering semantics. - *FI\_AUTH\_KEY* Only valid with domains configured with FI\_AV\_AUTH\_KEY and connectionless endpoints configured with FI\_DIRECTED\_RECV or FI\_TAGGED\_DIRECTED\_RECV. When used with fi\_trecvmsg, this flag denotes that the src\_addr is an authorization key fi\_addr\_t instead of an endpoint fi\_addr\_t. The following flags may be used with fi\_trecvmsg. - *FI\_PEEK* The peek flag may be used to see if a specified message has arrived. A peek request is often useful on endpoints that have provider allocated buffering enabled. Unlike standard receive operations, a receive operation with the FI\_PEEK flag set does not remain queued with the provider after the peek completes successfully. The peek operation operates asynchronously, and the results of the peek operation are available in the completion queue associated with the endpoint. If no message is found matching the tags specified in the peek request, then a completion queue error entry with err field set to FI\_ENOMSG will be available. If a peek request locates a matching message, the operation will complete successfully. The returned completion data will indicate the meta-data associated with the message, such as the message length, completion flags, available CQ data, tag, and source address. The data available is subject to the completion entry format (e.g. struct fi\_cq\_tagged\_entry). - *FI\_CLAIM* If this flag is used in conjunction with FI\_PEEK, it indicates if the peek request completes successfully – indicating that a matching message was located – the message is claimed by caller. Claimed messages can only be retrieved using a subsequent, paired receive operation with the FI\_CLAIM flag set. A receive operation with the FI\_CLAIM flag set, but FI\_PEEK not set is used to retrieve a previously claimed message. In order to use the FI\_CLAIM flag, an application must supply a struct fi\_context structure as the context for the receive operation. The same fi\_context structure used for an FI\_PEEK + FI\_CLAIM operation must be used by the paired FI\_CLAIM request. - *FI\_DISCARD* This flag may be used in conjunction with either FI\_PEEK or FI\_CLAIM. If this flag is used in conjunction with FI\_PEEK, it indicates if the peek request completes successfully – indicating that a matching message was located – the message is discarded by the provider, as the data is not needed by the application. This flag may also be used in conjunction with FI\_CLAIM in order to discard a message previously claimed using an FI\_PEEK + FI\_CLAIM request. If this flag is set, the input buffer(s) and length parameters are ignored. ## RETURN VALUE The tagged send and receive calls return 0 on success. On error, a negative value corresponding to fabric *errno* is returned. Fabric errno values are defined in fi\_errno.h. ## ERRORS - *-FI\_EAGAIN* See [fi\_msg(3)](/fi_msg/3.md) for a detailed description of handling FI\_EAGAIN. - *-FI\_EINVAL* Indicates that an invalid argument was supplied by the user. - *-FI\_EOTHER* Indicates that an unspecified error occurred. ## SEE ALSO [fi\_getinfo(3)](/fi_getinfo/3.md), [fi\_endpoint(3)](/fi_endpoint/3.md), [fi\_domain(3)](/fi_domain/3.md), [fi\_cq(3)](/fi_cq/3.md) ## AUTHORS OpenFabrics.