--- name: "fi_rxm" section: "7" platform: "linux" package: "libfabric" version: "2.7.0-1" release: "Arch Linux 2026-11-01" platforms: ["linux"] url: "https://new.carta.tech/fi_rxm" --- # fi\_rxm(7) The RxM (RDM over MSG) Utility Provider ## NAME fi\_rxm - The RxM (RDM over MSG) Utility Provider ## OVERVIEW The RxM provider (ofi\_rxm) is an utility provider that supports FI\_EP\_RDM type endpoint emulated over FI\_EP\_MSG type endpoint(s) of an underlying core provider. FI\_EP\_RDM endpoints have a reliable datagram interface and RxM emulates this by hiding the connection management of underlying FI\_EP\_MSG endpoints from the user. Additionally, RxM can hide memory registration requirement from a core provider like verbs if the apps don’t support it. ## REQUIREMENTS ### Requirements for core provider RxM provider requires the core provider to support the following features: - • MSG endpoints (FI\_EP\_MSG) - • RMA read/write (FI\_RMA) - Used for implementing rendezvous protocol for large messages. - • FI\_OPT\_CM\_DATA\_SIZE of at least 24 bytes. ### Requirements for applications Since RxM emulates RDM endpoints by hiding connection management and connections are established only on-demand (when app tries to send data), the first several data transfer calls would return EAGAIN. Applications should be aware of this and retry until the operation succeeds. If an application has chosen manual progress for data progress, it should also read the CQ so that the connection establishment progresses. Not doing so would result in a stall. See also the ERRORS section in [fi\_msg(3)](/fi_msg/3.md). ## SUPPORTED FEATURES The RxM provider currently supports *FI\_MSG*, *FI\_TAGGED*, *FI\_RMA* and *FI\_ATOMIC* capabilities. - *Endpoint types* The provider supports only *FI\_EP\_RDM*. - *Endpoint capabilities* The following data transfer interface is supported: *FI\_MSG*, *FI\_TAGGED*, *FI\_RMA*, *FI\_ATOMIC*. - *Progress* The RxM provider supports both *FI\_PROGRESS\_MANUAL* and *FI\_PROGRESS\_AUTO*. Manual progress in general has better connection scale-up and lower CPU utilization since there’s no separate auto-progress thread. - *Addressing Formats* FI\_SOCKADDR, FI\_SOCKADDR\_IN - *Memory Region* FI\_MR\_VIRT\_ADDR, FI\_MR\_ALLOCATED, FI\_MR\_PROV\_KEY MR mode bits would be required from the app in case the core provider requires it. ## LIMITATIONS When using RxM provider, some limitations from the underlying MSG provider could also show up. Please refer to the corresponding MSG provider man pages to find about those limitations. ### Unsupported features RxM provider does not support the following features: - • op\_flags: FI\_FENCE. - • Scalable endpoints - • Shared contexts - • FABRIC\_DIRECT - • FI\_MR\_SCALABLE - • Authorization keys - • Application error data buffers - • Multicast - • FI\_SYNC\_ERR - • Reporting unknown source addr data as part of completions - • Triggered operations ### Progress limitations When sending large messages, an app doing an sread or waiting on the CQ file descriptor may not get a completion when reading the CQ after being woken up from the wait. The app has to do sread or wait on the file descriptor again. This is needed because RxM uses a rendezvous protocol for large message sends. An app would get woken up from waiting on CQ fd when rendezvous protocol request completes but it would have to wait again to get an ACK from the receiver indicating completion of large message transfer by remote RMA read. ### FI\_ATOMIC limitations The FI\_ATOMIC capability will only be listed in the fi\_info if the fi\_info hints parameter specifies FI\_ATOMIC. If FI\_ATOMIC is requested, message order FI\_ORDER\_RAR, FI\_ORDER\_RAW, FI\_ORDER\_WAR, FI\_ORDER\_WAW, FI\_ORDER\_SAR, and FI\_ORDER\_SAW can not be supported. ### Message ordering limitations with multiple MSG endpoints When FI\_OFI\_RXM\_NUM\_MSG\_EPS is greater than 1, a peer connection spans several MSG endpoints, and hence several core provider connections that are unordered with respect to each other. Message, tagged and atomic transfers are issued on the first endpoint, so they remain ordered relative to each other, but RMA and rendezvous payload are spread across the remaining endpoints. Only FI\_ORDER\_SAS, FI\_ORDER\_ATOMIC\_RAR, FI\_ORDER\_ATOMIC\_RAW, FI\_ORDER\_ATOMIC\_WAR and FI\_ORDER\_ATOMIC\_WAW are reported, and max\_order\_raw\_size, max\_order\_war\_size and max\_order\_waw\_size are reported as 0. fi\_getinfo returns -FI\_ENODATA if the hints request ordering beyond this. Applications that need an RMA operation to be ordered against a later transfer must wait for its completion instead. ### Miscellaneous limitations - • RxM protocol peers should have same endian-ness otherwise connections won’t successfully complete. This enables better performance at run-time as byte order translations are avoided. ## RUNTIME PARAMETERS The ofi\_rxm provider checks for the following environment variables. - *FI\_OFI\_RXM\_BUFFER\_SIZE* Defines the transmit buffer size / inject size. Messages of size less than or equal to this would be transmitted via an eager protocol and messages greater in size would be transmitted via a rendezvous or SAR (Segmentation And Reassembly) protocol. Transmit data would be copied up to this size (default: ~16k). - *FI\_OFI\_RXM\_COMP\_PER\_PROGRESS* Defines the maximum number of MSG provider CQ entries (default: 1) that would be read per progress (RxM CQ read). - *FI\_OFI\_RXM\_ENABLE\_DYN\_RBUF* Enables support for dynamic receive buffering, if available by the message endpoint provider. This feature allows direct placement of received message data into application buffers, bypassing RxM bounce buffers. This feature targets providers that provide internal network buffering, such as the tcp provider. (default: false) - *FI\_OFI\_RXM\_SAR\_LIMIT* Set this environment variable to control the RxM SAR (Segmentation And Reassembly) protocol. Messages of size greater than this (default: 128 Kb) would be transmitted via rendezvous protocol. - *FI\_OFI\_RXM\_USE\_SRX* Set this to 1 to use shared receive context from MSG provider, or 0 to disable using shared receive context. Shared receive contexts reduce overall memory usage, but may increase in message latency. If not set, verbs will not use shared receive contexts by default, but the tcp provider will. - *FI\_OFI\_RXM\_TX\_SIZE* Defines default TX context size (default: 1024) - *FI\_OFI\_RXM\_RX\_SIZE* Defines default RX context size (default: 1024) - *FI\_OFI\_RXM\_MSG\_TX\_SIZE* Defines FI\_EP\_MSG TX size that would be requested (default: 128). - *FI\_OFI\_RXM\_MSG\_RX\_SIZE* Defines FI\_EP\_MSG RX size that would be requested (default: 128). - *FI\_UNIVERSE\_SIZE* Defines the expected number of ranks / peers an endpoint would communicate with (default: 256). - *FI\_OFI\_RXM\_CM\_PROGRESS\_INTERVAL* Defines the duration of time in microseconds between calls to RxM CM progression functions when using manual progress. Higher values may provide less noise for calls to fi\_cq read functions, but may increase connection setup time (default: 10000) - *FI\_OFI\_RXM\_CQ\_EQ\_FAIRNESS* Defines the maximum number of message provider CQ entries that can be consecutively read across progress calls without checking to see if the CM progress interval has been reached (default: 128) - *FI\_OFI\_RXM\_DETECT\_HMEM\_IFACE* Set this to 1 to allow automatic detection of HMEM iface of user buffers when such information is not supplied. This feature allows such buffers be copied or registered (e.g. in Rendezvous) internally by RxM. Note that no extra memory registration is performed with this option. (default: false) - *FI\_OFI\_RXM\_NUM\_MSG\_EPS* Number of MSG endpoints (and underlying connections/QPs) to open per RxM peer connection. Values greater than 1 spread traffic across multiple QPs in a round-robin fashion, which can improve throughput on hardware that benefits from multiple QPs per peer. Values greater than 1 also reduce the message ordering the provider guarantees, see the message ordering limitations above. The value is clamped to the range \[1, 255\]. (default: 1) ## Tuning ### Bandwidth To optimize for bandwidth, ensure you use higher values than default for FI\_OFI\_RXM\_TX\_SIZE, FI\_OFI\_RXM\_RX\_SIZE, FI\_OFI\_RXM\_MSG\_TX\_SIZE, FI\_OFI\_RXM\_MSG\_RX\_SIZE subject to memory limits of the system and the tx and rx sizes supported by the MSG provider. FI\_OFI\_RXM\_SAR\_LIMIT is another knob that can be experimented with to optimze for bandwidth. ### Memory To conserve memory, ensure FI\_UNIVERSE\_SIZE set to what is required. Similarly check that FI\_OFI\_RXM\_TX\_SIZE, FI\_OFI\_RXM\_RX\_SIZE, FI\_OFI\_RXM\_MSG\_TX\_SIZE and FI\_OFI\_RXM\_MSG\_RX\_SIZE env variables are set to only required values. ## NOTES The data transfer API may return -FI\_EAGAIN during on-demand connection setup of the core provider FI\_MSG\_EP. See [fi\_msg(3)](/fi_msg/3.md) for a detailed description of handling FI\_EAGAIN. ## Troubleshooting / Known issues If an RxM endpoint is expected to communicate with more peers than the default value of FI\_UNIVERSE\_SIZE (256) CQ overruns can happen. To avoid this set a higher value for FI\_UNIVERSE\_SIZE. CQ overrun can make a MSG endpoint unusable. At higher # of ranks, there may be connection errors due to a node running out of memory. The workaround is to use shared receive contexts for the MSG provider (FI\_OFI\_RXM\_USE\_SRX=1) or reduce eager message size (FI\_OFI\_RXM\_BUFFER\_SIZE) and MSG provider TX/RX queue sizes (FI\_OFI\_RXM\_MSG\_TX\_SIZE / FI\_OFI\_RXM\_MSG\_RX\_SIZE). ## SEE ALSO [fabric(7)](/fabric/7.md), [fi\_provider(7)](/fi_provider/7.md), [fi\_getinfo(3)](/fi_getinfo/3.md) ## AUTHORS OpenFabrics.