--- name: "rdma_cm" section: "7" platform: "linux" package: "rdma-core" version: "65.0-1" release: "Arch Linux 2026-11-01" platforms: ["linux"] url: "https://new.carta.tech/rdma_cm" --- # rdma\_cm(7) RDMA communication manager. ## NAME rdma\_cm - RDMA communication manager. ## SYNOPSIS **#include \** ## DESCRIPTION Used to establish communication over RDMA transports. ## NOTES The RDMA CM is a communication manager used to setup reliable, connected and unreliable datagram data transfers. It provides an RDMA transport neutral interface for establishing connections. The API concepts are based on sockets, but adapted for queue pair (QP) based semantics: communication must be over a specific RDMA device, and data transfers are message based. The RDMA CM can control both the QP and communication management (connection setup / teardown) portions of an RDMA API, or only the communication management piece. It works in conjunction with the verbs API defined by the libibverbs library. The libibverbs library provides the underlying interfaces needed to send and receive data. The RDMA CM can operate asynchronously or synchronously. The mode of operation is controlled by the user through the use of the rdma\_cm event channel parameter in specific calls. If an event channel is provided, an rdma\_cm identifier will report its event data (results of connecting, for example), on that channel. If a channel is not provided, then all rdma\_cm operations for the selected rdma\_cm identifier will block until they complete. The RDMA CM gives an option to different libibverbs providers to advertise and use various specific to that provider QP configuration options. This functionality is called ECE (enhanced connection establishment). ## RDMA VERBS The rdma\_cm supports the full range of verbs available through the libibverbs library and interfaces. However, it also provides wrapper functions for some of the more commonly used verbs funcationality. The full set of abstracted verb calls are: rdma\_reg\_msgs - register an array of buffers for sending and receiving rdma\_reg\_read - registers a buffer for RDMA read operations rdma\_reg\_write - registers a buffer for RDMA write operations rdma\_dereg\_mr - deregisters a memory region rdma\_post\_recv - post a buffer to receive a message rdma\_post\_send - post a buffer to send a message rdma\_post\_read - post an RDMA to read data into a buffer rdma\_post\_write - post an RDMA to send data from a buffer rdma\_post\_recvv - post a vector of buffers to receive a message rdma\_post\_sendv - post a vector of buffers to send a message rdma\_post\_readv - post a vector of buffers to receive an RDMA read rdma\_post\_writev - post a vector of buffers to send an RDMA write rdma\_post\_ud\_send - post a buffer to send a message on a UD QP rdma\_get\_send\_comp - get completion status for a send or RDMA operation rdma\_get\_recv\_comp - get information about a completed receive ## CLIENT OPERATION This section provides a general overview of the basic operation for the active, or client, side of communication. This flow assume asynchronous operation with low level call details shown. For synchronous operation, calls to rdma\_create\_event\_channel, rdma\_get\_cm\_event, rdma\_ack\_cm\_event, and rdma\_destroy\_event\_channel would be eliminated. Abstracted calls, such as rdma\_create\_ep encapsulate several of these calls under a single API. Users may also refer to the example applications for code samples. A general connection flow would be: - rdma\_getaddrinfo retrieve address information of the destination - rdma\_create\_event\_channel create channel to receive events - rdma\_create\_id allocate an rdma\_cm\_id, this is conceptually similar to a socket - rdma\_resolve\_addr obtain a local RDMA device to reach the remote address - rdma\_get\_cm\_event wait for RDMA\_CM\_EVENT\_ADDR\_RESOLVED event - rdma\_ack\_cm\_event ack event - rdma\_create\_qp allocate a QP for the communication - rdma\_resolve\_route determine the route to the remote address - rdma\_get\_cm\_event wait for RDMA\_CM\_EVENT\_ROUTE\_RESOLVED event - rdma\_ack\_cm\_event ack event - rdma\_connect connect to the remote server - rdma\_get\_cm\_event wait for RDMA\_CM\_EVENT\_ESTABLISHED event - rdma\_ack\_cm\_event ack event Perform data transfers over connection - rdma\_disconnect tear-down connection - rdma\_get\_cm\_event wait for RDMA\_CM\_EVENT\_DISCONNECTED event - rdma\_ack\_cm\_event ack event - rdma\_destroy\_qp destroy the QP - rdma\_destroy\_id release the rdma\_cm\_id - rdma\_destroy\_event\_channel release the event channel - rdma\_freeaddrinfo release the list of rdma\_addrinfo structures - rdma\_set\_local\_ece set desired ECE options An almost identical process is used to setup unreliable datagram (UD) communication between nodes. No actual connection is formed between QPs however, so disconnection is not needed. Although this example shows the client initiating the disconnect, either side of a connection may initiate the disconnect. ## SERVER OPERATION This section provides a general overview of the basic operation for the passive, or server, side of communication. A general connection flow would be: - rdma\_create\_event\_channel create channel to receive events - rdma\_create\_id allocate an rdma\_cm\_id, this is conceptually similar to a socket - rdma\_bind\_addr set the local port number to listen on - rdma\_listen begin listening for connection requests - rdma\_get\_cm\_event wait for RDMA\_CM\_EVENT\_CONNECT\_REQUEST event with a new rdma\_cm\_id - rdma\_create\_qp allocate a QP for the communication on the new rdma\_cm\_id - rdma\_accept accept the connection request - rdma\_ack\_cm\_event ack event - rdma\_get\_cm\_event wait for RDMA\_CM\_EVENT\_ESTABLISHED event - rdma\_ack\_cm\_event ack event Perform data transfers over connection - rdma\_get\_cm\_event wait for RDMA\_CM\_EVENT\_DISCONNECTED event - rdma\_ack\_cm\_event ack event - rdma\_disconnect tear-down connection - rdma\_destroy\_qp destroy the QP - rdma\_destroy\_id release the connected rdma\_cm\_id - rdma\_destroy\_id release the listening rdma\_cm\_id - rdma\_destroy\_event\_channel release the event channel - rdma\_get\_remote\_ece get ECe options sent by the client - rdma\_set\_local\_ece set desired ECE options ## RETURN CODES - \= 0 success - \= -1 error - see errno for more details Most librdmacm functions return 0 to indicate success, and a -1 return value to indicate failure. If a function operates asynchronously, a return value of 0 means that the operation was successfully started. The operation could still complete in error; users should check the status of the related event. If the return value is -1, then errno will contain additional information regarding the reason for the failure. Prior versions of the library would return -errno and not set errno for some cases related to ENOMEM, ENODEV, ENODATA, EINVAL, and EADDRNOTAVAIL codes. Applications that want to check these codes and have compatibility with prior library versions must manually set errno to the negative of the return code if it is \< -1. ## SEE ALSO [rdma\_accept(3)](/rdma_accept/3.md), [rdma\_ack\_cm\_event(3)](/rdma_ack_cm_event/3.md), [rdma\_bind\_addr(3)](/rdma_bind_addr/3.md), [rdma\_connect(3)](/rdma_connect/3.md), [rdma\_create\_ep(3)](/rdma_create_ep/3.md), [rdma\_create\_event\_channel(3)](/rdma_create_event_channel/3.md), [rdma\_create\_id(3)](/rdma_create_id/3.md), [rdma\_create\_qp(3)](/rdma_create_qp/3.md), [rdma\_dereg\_mr(3)](/rdma_dereg_mr/3.md), [rdma\_destroy\_ep(3)](/rdma_destroy_ep/3.md), [rdma\_destroy\_event\_channel(3)](/rdma_destroy_event_channel/3.md), [rdma\_destroy\_id(3)](/rdma_destroy_id/3.md), [rdma\_destroy\_qp(3)](/rdma_destroy_qp/3.md), [rdma\_disconnect(3)](/rdma_disconnect/3.md), [rdma\_event\_str(3)](/rdma_event_str/3.md), [rdma\_free\_devices(3)](/rdma_free_devices/3.md), [rdma\_freeaddrinfo(3)](/rdma_freeaddrinfo/3.md), [rdma\_getaddrinfo(3)](/rdma_getaddrinfo/3.md), [rdma\_get\_cm\_event(3)](/rdma_get_cm_event/3.md), [rdma\_get\_devices(3)](/rdma_get_devices/3.md), [rdma\_get\_dst\_port(3)](/rdma_get_dst_port/3.md), [rdma\_get\_local\_addr(3)](/rdma_get_local_addr/3.md), [rdma\_get\_peer\_addr(3)](/rdma_get_peer_addr/3.md), [rdma\_get\_recv\_comp(3)](/rdma_get_recv_comp/3.md), [rdma\_get\_remote\_ece(3)](/rdma_get_remote_ece/3.md), [rdma\_get\_request(3)](/rdma_get_request/3.md), [rdma\_get\_send\_comp(3)](/rdma_get_send_comp/3.md), [rdma\_get\_src\_port(3)](/rdma_get_src_port/3.md), [rdma\_join\_multicast(3)](/rdma_join_multicast/3.md), [rdma\_leave\_multicast(3)](/rdma_leave_multicast/3.md), [rdma\_listen(3)](/rdma_listen/3.md), [rdma\_migrate\_id(3)](/rdma_migrate_id/3.md), [rdma\_notify(3)](/rdma_notify/3.md), [rdma\_post\_read(3)](/rdma_post_read/3.md) [rdma\_post\_readv(3)](/rdma_post_readv/3.md), [rdma\_post\_recv(3)](/rdma_post_recv/3.md), [rdma\_post\_recvv(3)](/rdma_post_recvv/3.md), [rdma\_post\_send(3)](/rdma_post_send/3.md), [rdma\_post\_sendv(3)](/rdma_post_sendv/3.md), [rdma\_post\_ud\_send(3)](/rdma_post_ud_send/3.md), [rdma\_post\_write(3)](/rdma_post_write/3.md), [rdma\_post\_writev(3)](/rdma_post_writev/3.md), [rdma\_query\_addrinfo(3)](/rdma_query_addrinfo/3.md), [rdma\_reg\_msgs(3)](/rdma_reg_msgs/3.md), [rdma\_reg\_read(3)](/rdma_reg_read/3.md), [rdma\_reg\_write(3)](/rdma_reg_write/3.md), [rdma\_reject(3)](/rdma_reject/3.md), [rdma\_resolve\_addr(3)](/rdma_resolve_addr/3.md), [rdma\_resolve\_addrinfo(3)](/rdma_resolve_addrinfo/3.md), [rdma\_resolve\_route(3)](/rdma_resolve_route/3.md), [rdma\_get\_remote\_ece(3)](/rdma_get_remote_ece/3.md), [rdma\_set\_option(3)](/rdma_set_option/3.md), [rdma\_write\_cm\_event(3)](/rdma_write_cm_event/3.md), [mckey(1)](/mckey/1.md), [rdma\_client(1)](/rdma_client/1.md), [rdma\_server(1)](/rdma_server/1.md), [rping(1)](/rping/1.md), [ucmatose(1)](/ucmatose/1.md), [udaddy(1)](/udaddy/1.md)