--- name: "ib_send_lat" section: "1" platform: "linux" package: "perftest" version: "25.01.0+0.80-1" release: "Debian 13 (Trixie)" platforms: ["linux"] url: "https://new.carta.tech/ib_send_lat" --- # ib\_send\_lat(1) benchmarks for various types of infinabnd performance ## NAME ib\_write\_bw, ib\_read\_bw, ib\_send\_bw, ib\_atomic\_bw, ib\_write\_lat, ib\_read\_lat, ib\_send\_lat, ib\_atomic\_lat, raw\_ethernet\_bw, raw\_ethernet\_lat, raw\_ethernet\_burst\_lat, raw\_ethernet\_fs\_rate - benchmarks for various types of infinabnd performance ## DESCRIPTION - Perftest is a package that includes various benchmarks that measures different metrics & verbs performance which include many different options and modes. ### RUNNING TESTS - **Server:** ./\ \ - **Client:** ./\ \ \ - **Examples:** 1- Running bidirectional bandwidth test using Write verb for 5 seconds with 8388608 as a message size and 3 qps:\ Server: ./ib\_write\_bw -s 8388608 -b -D 5 -q 3\ Client: ./ib\_write\_bw -s 8388608 -b -D 5 -q 3 1.1.1.2 2- Running latency test using Read verb for 5000 iterations with 32 as a message size:\ Server: ./ib\_read\_lat -s 32 -n 5000\ Client: ./ib\_read\_lat -s 32 -n 5000 192.168.0.1 ### IMPORTANT NOTES - 1- The options that specific to modes in perftest must be the same for both server and client. 2- Perftest applications may need to be ran with sudo when running from non root.\ 3- Perftest applications usually installed to the /usr/bin/.\ 4- Perftest may print some failures with syndroms to the stderr, perftest get those errors from rdma-core. ## OPTIONS - **-h, --help** Lists the available options to the screen. - **-a, --all** Run sizes from 2 till 2^23.\ Not relevant for Atomic and RawEth. - **-A, --atomic\_type=\** Type of atomic operation from {CMP\_AND\_SWAP,FETCH\_AND\_ADD} (default FETCH\_AND\_ADD).\ Relevant only for Atomic. - **-b, --bidirectional** Measure bidirectional bandwidth (default unidirectional).\ Relevant only for BW. - **-c, --connection=\** Connection type RC/XRC/UC/UD/DC/SRD (default RC).\ UD relevant only for Send verb.\ SRD relevant only for Read, Write and Send verbs.\ UC relevant only for Write and Send verbs.\ Not relevant for RawEth. - **--log\_dci\_streams=\ (default 0)** Run DC initiator as DCS instead of DCI with \.\ Not relevant for RawEth.\ System support required. - **--log\_active\_dci\_streams=\ (default log\_num\_dci\_stream\_channels)** Not relevant for RawEth.\ System support required. - **--aes\_xts** Runs traffic with AES\_XTS feature (encryption).\ Not relevant for RawEth and Write latency.\ System support required. - **--encrypt\_on\_tx** Runs traffic with encryption on tx (default decryption on tx).\ Not relevant for RawEth and Write latency.\ System support required. - **--sig\_before** Puts signature on data before encrypting it (default after).\ Not relevant for RawEth and Write latency.\ System support required. - **--aes\_block\_size=\<512,520,4048,4096,4160\> (default 512)** Not relevant for RawEth and Write latency.\ System support required. - **--data\_enc\_keys\_number=\ (default 1)** Not relevant for RawEth and Write latency.\ System support required. - **--kek\_path \** Not relevant for RawEth and Write latency.\ System support required. - **--credentials\_path \** Not relevant for RawEth and Write latency.\ System support required. - **--data\_enc\_key\_app\_path \** Not relevant for RawEth and Write latency.\ System support required. - **-C, --report-cycles** Report times in cpu cycle units (default microseconds).\ Relevant only for latency. - **-d, --ib-dev=\** Use IB device \ (default first device found). - **-D, --duration** Run test for a customized period of seconds. - **-e, --events** Sleep on CQ events (default poll).\ Not relevant for Write and RawEth. - **-X, --vector=\** Set \ used for events.\ Not relevant for Write and RawEth. - **-f, --margin** measure results within margins. (default=2sec). - **-F, --CPU-freq** Do not show a warning even if cpufreq\_ondemand module is loaded, and cpu-freq is not on max. - **-g, --mcg** Send messages to multicast group with 1 QP attached to it.\ When there is no multicast gid specified, a default IPv6 typed gid '255:1:0:0:0:2:201:133:0:0:0:0:0:0:0:0' will be used.\ Relevant only for send non fsRate. - **-H, --report-histogram** Print out all results (default print summary only).\ Relevant only for latency and raw\_ethernet\_fs\_rate. - **-i, --ib-port=\** Use port \ of IB device (default 1). - **-I, --inline\_size=\** Max size of message to be sent in inline.\ Not relevant for Read and Atomic. - **-l, --post\_list=\** Post list of send WQEs of \ size (instead of single post).\ Relevant only for BW and raw\_ethernet\_burst\_lat. - **--recv\_post\_list=\** Post list of receive WQEs of \ size (instead of single post).\ Relevant only for BW and raw\_ethernet\_burst\_lat. - **-L, --hop\_limit=\** Set hop limit value (ttl for IPv4 RawEth QP). Values 0-255 (default 64).\ Relevant only for RawEth\ Not relevant for raw\_ethernet\_fs\_rate. - **-m, --mtu=\** MTU size : 64 - 9600 (default port mtu) for RawEth else 256 - 4096.\ Not relevant for raw\_ethernet\_fs\_rate. - **-M, --MGID=\** In multicast, uses \ as the group MGID.\ \ can be either decimal or hexadecimal, e.g. regarding the IPv4 224.0.0.30 :\ Decimal: 0:0:0:0:0:0:0:0:0:0:255:255:224:0:0:30 , Hexadecimal: 0:0:0:0:0:0:0:0:0:0:0xff:0xff:0xe0:0:0:0x1e\ Relevant only for send non fsRate. - **-n, --iters=\** Number of exchanges (at least 5, default for write 5000 else 1000 ). - **-N, --noPeak** Cancel peak-bw calculation (default with peak up to iters=20000).\ Relevant only for bandwidth. - **-o, --outs=\** Relevant only for Read and Atomic. - **-O, --dualport** Run test in dual-port mode.\ Not relevant for RawEth.\ Relevant only for bandwidth.\ System support required. - **-p, --port=\** Listen on/connect to port \ (default 18515). - **-q, --qp=\** Num of qp's(default 1).\ Relevant only for bandwidth. - **-Q, --cq-mod** Generate Cqe only after \<--cq-mod\> completion.\ Relevant only for bandwidth. - **-r, --rx-depth=\** Rx queue size (default 512), if using srq, rx-depth controls max-wr size of the srq.\ Relevant only for send non fsRate. - **-R, --rdma\_cm** Connect QPs with rdma\_cm and run test on those QPs.\ Not relevant for RawEth. - **-s, --size=\** Size of message to exchange (default 65536 for bw, for lat 2).\ Not relevant for Atomic. - **-S, --sl=\** SL (default 0).\ Not relevant for raw\_ethernet\_fs\_rate. - **-t, --tx-depth=\** Size of tx queue (default 128 for bw else 1).\ Relevant only for bw and raw\_ethernet\_burst\_lat. - **-T, --tos=\** Set \ to RDMA-CM QPs. available only with -R flag. values 0-256 (default off).\ Not relevant for RawEth - **-u, --qp-timeout=\** QP timeout, timeout value is 4 usec \* 2 ^(timeout), default 14. - **-U, --report-unsorted** (implies -H) print out unsorted results (default sorted).\ Relevant only for latency and raw\_ethernet\_burst\_lat and raw\_ethernet\_fs\_rate. - **-V, --version** Display perftest version number. - **-W, --report-counters=\** Report performance counter change (example: counters/port\_xmit\_data,hw\_counters/out\_of\_buffer). - **-x, --gid-index=\** Test uses GID with GID index.\ Not relevant for RawEth. - **-z, --comm\_rdma\_cm** Communicate with rdma\_cm module to exchange data - use regular QPs.\ Not relevant for RawEth. - **--out\_json** Save the report in a json file. - **--out\_json\_file=\** Name of the report json file. (Default: "perftest\_out.json" in the working directory). - **--cpu\_util** Show CPU Utilization in report, valid only in Duration mode. - **--dlid** Set a Destination LID instead of getting it from the other side.\ Not relevant for raw\_ethernet\_fs\_rate. - **--dont\_xchg\_versions** Do not exchange versions and MTU with other side.\ Not relevant for RawEth. - **--force-link=\** Force the link(s) to a specific type: IB or Ethernet.\ Not relevant for raw\_ethernet\_fs\_rate. - **--use-srq** Use a Shared Receive Queue. --rx-depth controls max-wr size of the SRQ.\ Relevant only for Send. - **--ipv6** Use IPv6 GID. Default is IPv4.\ Not relevant for RawEth. - **--ipv6-addr=\** Use IPv6 address for parameters negotiation. Default is IPv4.\ Not relevant for RawEth. - **--bind\_source\_ip** Source IP of the interface used for connection establishment. By default taken from routing table.\ Not relevant for RawEth. - **--latency\_gap=\** delay time between each post send.\ Relevant only for latency. - **--mmap=file** Use an mmap'd file as the buffer for testing P2P transfers.\ Not relevant for RawEth. - **--mmap-offset=\** The mmap offset.\ Not relevant for RawEth. - **--mr\_per\_qp** Create memory region for each qp.\ Relevant only for bandwidth. - **--odp** Use On Demand Paging instead of Memory Registration.\ System support required. - **--output=\** Set verbosity output level: bandwidth , message\_rate, latency.\ Latency measurement is Average calculation.\ bw (bandwidth / message\_rate), latency (latency). - **--payload\_file\_path=\** Set the payload by passing a txt file containing a pattern in the next form(little endian): '0xaaaaaaaa, 0xbbbbbbbb, ...\ Not relevant for RawEth and Write latency. - **--use\_old\_post\_send** Use old post send flow (ibv\_post\_send). - **--perform\_warm\_up** Perform some iterations before start measuring in order to warming-up memory cache.\ Not relevant for raw\_ethernet\_fs\_rate. - **--pkey\_index=\** PKey index to use for QP.\ Not relevant for raw\_ethernet\_fs\_rate. - **--report-both** Report RX & TX results separately on Bidirectional BW tests.\ Relevant only for bidirectional bandwidth. - **--report\_gbits** Report Max/Average BW of test in Gbit/sec (instead of MiB/sec).\ Relevant only for bandwidth. - **--report-per-port** Report BW data on both ports when running Dualport and Duration mode.\ Not relevant for RawEth.\ System support required. - **--reversed** Reverse traffic direction - Server send to client. - **--run\_infinitely** Run test forever, print results every \ seconds. - **--retry\_count=\** Set retry count value in rdma\_cm mode.\ Relevant only for rdma\_cm mode.\ Not relevant for RawEth. - **--tclass=\** Set the Traffic Class in GRH (if GRH is in use).\ Not relevant for raw\_ethernet\_fs\_rate. - **--use-null-mr** Allocate a null memory region for the client with [**ibv\_alloc\_null\_mr**(3)](/ibv_alloc_null_mr/3.md) - **--use\_cuda=\** Use CUDA specific device for GPUDirect RDMA testing.\ Not relevant for raw\_ethernet\_fs\_rate.\ System support required. - **--use\_cuda\_bus\_id=\** Use CUDA specific device, based on its full PCIe address, for GPUDirect RDMA testing.\ Not relevant for raw\_ethernet\_fs\_rate.\ System support required. - **--use\_cuda\_dmabuf** Use CUDA DMA-BUF for GPUDirect RDMA testing.\ Not relevant for raw\_ethernet\_fs\_rate.\ System support required. - **--use\_hl=\** Use HabanaLabs specific device for HW accelerator direct RDMA testing.\ System support required. - **--use\_neuron=\** Use Neuron specific device for HW accelerator direct RDMA testing.\ System support required. - **--use\_neuron\_dmabuf** Use Neuron DMA-BUF for HW accelerator direct RDMA testing.\ System support required. - **--use\_rocm=\** Use selected ROCm device for GPUDirect RDMA testing.\ Not relevant for raw\_ethernet\_fs\_rate.\ System support required. - **--use\_hugepages** Use Hugepages instead of contig, memalign allocations.\ Not relevant for raw\_ethernet\_fs\_rate. - **--wait\_destroy=\** Wait \ before destroying allocated resources (QP/CQ/PD/MR..).\ Relevant only for bandwidth and raw\_ethernet\_burst\_lat. - **--disable\_pcie\_relaxed** Disable PCIe relaxed ordering.\ Relevant only for bandwidth and raw\_ethernet\_burst\_lat.\ System support required. - **--burst\_size=\** Set the amount of messages to send in a burst when using rate limiter.\ Relevant only for bandwidth and raw\_ethernet\_burst\_lat. - **--typical\_pkt\_size=\** Set the size of packet to send in a burst. Only supports PP rate limiter.\ Relevant only for bandwidth and raw\_ethernet\_burst\_lat. - **--rate\_limit=\** Set the maximum rate of sent packages. default unit is \[Gbps\]. use --rate\_units to change that.\ Relevant only for bandwidth and raw\_ethernet\_burst\_lat. - **--rate\_units=\** \[Mgp\] Set the units for rate limit to MiBps (M), Gbps (g) or pps (p). default is Gbps (g).\ Relevant only for bandwidth and raw\_ethernet\_burst\_lat. - **--rate\_limit\_type=\** \[HW/SW/PP\] Limit the QP's by HW, PP or by SW. Disabled by default. When rate\_limit is not specified HW limit is Default.\ Relevant only for bandwidth and raw\_ethernet\_burst\_lat. - **--use\_ooo** Use out of order data placement.\ System support required. - **--write\_with\_imm** Use write-with-immediate verb instead of write.\ Write tests only. ### RawEth only options - **-B, --source\_mac** Source MAC address by this format XX:XX:XX:XX:XX:XX \*\*MUST\*\* be entered. - **-E, --dest\_mac** Destination MAC address by this format XX:XX:XX:XX:XX:XX \*\*MUST\*\* be entered. - **-G, --use\_rss** Use RSS on server side. need to open 2^x qps (using -q flag. default is -q 2). open 2^x clients that transmit to this server. - **-J, --dest\_ip** Destination ip address by this format X.X.X.X for IPv4 or X:X:X:X:X:X for IPv6 (using to send packets with IP header).\ System support required for IPv6. - **-j, --source\_ip** Source ip address by this format X.X.X.X for IPv4 or X:X:X:X:X:X for IPv6 (using to send packets with IP header).\ System support required for IPv6. - **-K, --dest\_port** Destination port number (using to send packets with UDP header as default, or you can use --tcp flag to send TCP Header). - **-k, --source\_port** Source port number (using to send packets with UDP header as default, or you can use --tcp flag to send TCP Header). - **-Y, --ethertype** Ethertype value in the ethernet frame by this format 0xXXXX. - **-Z, --server** Choose server side for the current machine (--server/--client must be selected ). - **--vlan\_en** Insert vlan tag in ethernet header. - **--vlan\_pcp** Specify vlan\_pcp value for vlan tag, 0~7. 8 means different vlan\_pcp for each packet. - **-P, --client** Choose client side for the current machine (--server/--client must be selected).\ Not relevant for raw\_ethernet\_fs\_rate. - **-v, --mac\_fwd** Run mac forwarding test.\ Not relevant for raw\_ethernet\_fs\_rate. - **--flows** Set number of TCP/UDP flows, starting from \.\ Not relevant for raw\_ethernet\_fs\_rate. - **--flows\_burst** Set number of burst size per TCP/UDP flow.\ Not relevant for raw\_ethernet\_fs\_rate. - **--promiscuous** Run promiscuous mode.\ Not relevant for raw\_ethernet\_fs\_rate. - **--reply\_every** In latency test, receiver pong after number of received pings.\ Not relevant for raw\_ethernet\_fs\_rate. - **--sniffer** Run sniffer mode.\ Not relevant for raw\_ethernet\_fs\_rate.\ System support required. - **--flow\_label=\** IPv6 flow label.\ Not relevant for raw\_ethernet\_fs\_rate. - **--tcp** Send TCP Packets. must include IP and Ports information. - **--raw\_ipv6** Send IPv6 Packets.\ System support required. - **--raw\_mcast.** Relevant only for bandwidth. ## AUTHORS - **Hassan Khadour \** - **Talat Batheesh \**