--- name: "nfstest_rdma" section: "1" platform: "linux" package: "nfstest" version: "3.2-3" release: "Debian 13 (Trixie)" platforms: ["linux"] url: "https://new.carta.tech/nfstest_rdma" --- # nfstest\_rdma(1) NFS-over-RDMA functional tests ## NAME nfstest\_rdma - NFS-over-RDMA functional tests ## SYNOPSIS nfstest\_rdma --server \ \[options\] ## DESCRIPTION Verify correct functionality of NFS-over-RDMA Remote Direct Memory Access (RDMA) provides fast data transfers between servers and storage. NFS over RDMA is best used when a large amount of data needs to be transferred with higher performance than regular NFS. NFS over RDMA is usually used over InfiniBand which provides higher performance and a lower latency. Although NFS over RDMA is mostly used over InfiniBand, Ethernet could be used as the link protocol as well. RDMA over Converged Ethernet (RoCE) which allows RDMA over Ethernet by encapsulating the InfiniBand transport packet over Ethernet. RoCE provides a couple of variants: RoCEv1 and RoCEv2. One is RoCEv1 which is an Ethernet link layer protocol and provides RDMA functionality between two hosts in the same Ethernet broadcast domain. While the second is RoCEv2 or RRoCE (Remote RoCE) which is an internet layer protocol so these packets can be routed. RoCEv2 runs over UDP/IPv4 or UDP/IPv6. There is also another variant called iWARP which runs over the TCP protocol. Testing is currently supported for all of these variants except for iWARP. NFS over RDMA has a couple of extra layers in the packet: InfiniBand layer and RPC-over-RDMA or RPCoRDMA layer. The InfiniBand layer contains the OpCode which specifies the type of RDMA operation to perform and the PSN which is the packet sequence number. The RPCoRDMA layer contains the XID and the RDMA chunk lists. The RDMA read chunk list is used to transfer DDP (Direct Data Placement) data from the NFS client to the server, e.g., NFS write call. On the other hand, the RDMA write chunk list is used to transfer DDP data from the NFS server back to the client, e.g., NFS read reply. Only certain NFS operations can be transferred using DDP and only the opaque part of the operation is transferred using either RDMA reads or writes while the rest of the NFS packet is transferred via the receive buffer using the RDMA SEND operation. Finally, the RDMA reply chunk is used to transfer replies having a variable length reply which could be larger than the receive buffer and could not be transferred using the write chunk list because it does not contain a single large opaque item. Tests are divided into three groups: basic, read and write. The basic tests deal mostly with verifying NFS packets using the reply chunk and some other basic RDMA functionality. The read tests deal with verifying NFS read which in turn verify the RDMA write functionality. Finally, the write tests deal with verifying NFS write which in turn verify the RDMA read functionality. Also, if the NFS read or write is small enough the client could not use the RDMA write or read functionality, but instead could use the receive buffer and transfer the data using the RDMA SEND operations. Tests verify the RPCoRDMA layer is sent when necessary and that the RDMA chunk lists are sent with the correct information which includes the number of chunks, number of segments in each chunk and the correct information for each segment. Tests verify each segment information is correct and their corresponding RDMA read or write information which includes correct handle, virtual offset, DMA length and the XDR position for the case of RDMA reads. In addition, the correct number of RDMA I/O fragments is also verified and their corresponding lengths and packet sequence numbers. ## OPTIONS - \--version show program's version number and exit - \-h, --help show this help message and exit - \-f FILE, --file=FILE File where options are specified besides the system wide file /etc/nfstest, user wide file $HOME/.nfstest or in the current directory .nfstest file ### NFS specific options: - \-s SERVER, --server=SERVER Server name or IP address - \-e EXPORT, --export=EXPORT Exported file system to mount \[default: '/'\] - \--nfsversion=NFSVERSION NFS version, e.g., 3, 4, 4.1, etc. \[default: 4.1\] - \-m MTPOINT, --mtpoint=MTPOINT Mount point \[default: '/mnt/t'\] - \-p PORT, --port=PORT NFS server port \[default: 20049\] - \--proto=PROTO NFS protocol name \[default: 'rdma'\] - \--sec=SEC Security flavor \[default: 'sys'\] - \--nconnect=NCONNECT Multiple TCP connections option \[default: '1'\] - \-o MTOPTS, --mtopts=MTOPTS Mount options \[default: 'hard'\] - \--datadir=DATADIR Data directory where files are created, directory is created on the mount point \[default: ''\] ### Logging options: - \-v VERBOSE, --verbose=VERBOSE Verbose level for debug messages \[default: 'opts\|info\|dbg1\|dbg2\|dbg3'\] - \--tverbose=TVERBOSE Verbose level for test messages \[default: '1'\] - \--createlog Create log file - \--rexeclog Create rexec log files - \--warnings Display warnings - \--tag=TAG Informational tag, it is displayed as an INFO message \[default: ''\] - \--notty Do not use terminal colors on output - \--isatty Use terminal colors on output -- useful when running with nohup ### Packet trace options: - \--createtraces Create a packet trace for each test - \--tbsize=TBSIZE Capture buffer size for tcpdump \[default: 192k\] - \--trcdelay=TRCDELAY Seconds to delay before stopping packet trace \[default: 2.0\] - \--keeptraces Do not remove any trace files \[default: remove trace files if no errors\] - \--rmtraces Remove trace files \[default: remove trace files if no errors\] - \-i INTERFACE, --interface=INTERFACE Device interface \[default: automatically selected\] ### File options: - \--nfiles=NFILES Number of files to create \[default: 0\] - \--filesize=FILESIZE File size to use for test files \[default: 64k\] - \--rsize=RSIZE Read size to use when reading files \[default: 4k\] - \--wsize=WSIZE Write size to use when writing files \[default: 4k\] - \--iodelay=IODELAY Seconds to delay I/O operations \[default: 0.1\] - \--offset-delta=OFFSET\_DELTA Read/Write offset delta \[default: 4k\] ### Path options: - \--sudo=SUDO Full path of binary for sudo \[default: '/usr/bin/sudo'\] - \--kill=KILL Full path of binary for kill \[default: '/usr/bin/kill'\] - \--nfsstat=NFSSTAT Full path of binary for nfsstat \[default: '/usr/sbin/nfsstat'\] - \--tcpdump=TCPDUMP Full path of binary for tcpdump \[default: '/usr/sbin/tcpdump'\] - \--iptables=IPTABLES Full path of binary for iptables \[default: '/usr/sbin/iptables'\] - \--messages=MESSAGES Full path of log messages file \[default: '/var/log/messages'\] - \--trcevents=TRCEVENTS Full path of tracing events directory \[default: '/sys/kernel/debug/tracing/events'\] - \--trcpipe=TRCPIPE Full path of trace pipe file \[default: '/sys/kernel/debug/tracing/trace\_pipe'\] - \--tmpdir=TMPDIR Temporary directory \[default: '/tmp'\] ### Debug options: - \--nocleanup Do not cleanup created files - \--notimestamps Do not display timestamps in debug messages - \--bugmsgs=BUGMSGS File containing test messages to mark as bugs if they failed - \--nomount Do not mount server and run the tests on local disk space - \--basename=BASENAME Base name for all files and logs \[default: automatically generated\] - \--nfsdebug=NFSDEBUG Set NFS kernel debug flags and save log messages \[default: ''\] - \--rpcdebug=RPCDEBUG Set RPC kernel debug flags and save log messages \[default: ''\] - \--tracepoints=TRACEPOINTS List of trace points modules to enable \[default: ''\] - \--nfsstats Get NFS stats \[default: 'False'\] - \--pktdisp Display main packets related to the given test - \--nfserrors Fail every NFS error found in the packet trace - \--client-ipaddr=CLIENT\_IPADDR IP address of localhost ### Reporting options: - \--xunit-report Generate xUnit compatible test report - \--xunit-report-file=XUNIT\_REPORT\_FILE Path to xout report file ### Test options: - \--runtest=RUNTEST Comma separated list of tests to run, if list starts with a '^' then all tests are run except the ones listed \[default: 'all'\] - \--small-filesize=SMALL\_FILESIZE File size to use for small files \[default: 4k\] - \--large-filesize=LARGE\_FILESIZE File size to use for large files \[default: 1m\] - \--strict Mark warnings for missing fragments as failures \[default: False\] ## TESTS ### basic ``` Run all NFS-over-RDMA basic functionality tests: basic01, basic02, basic03, basic04, basic05 ``` ### basic01 ``` Verify basic NFS-over-RDMA functionality ``` ### basic02 ``` Verify NFS-over-RDMA reply chunk on EXCHANGE_ID/SETCLIENTID ``` ### basic03 ``` Verify NFS-over-RDMA reply chunk on READDIR ``` ### basic04 ``` Verify NFS-over-RDMA reply chunk on READLINK ``` ### basic05 ``` Verify NFS-over-RDMA reply chunk on GETATTR(FATTR4_ACL) ``` ### read ``` Run all NFS-over-RDMA functionality tests where file is opened for reading: read01, read02, read03, read04 ``` ### read01 ``` Verify NFS-over-RDMA functionality on a file opened for reading (very small file) ``` ### read02 ``` Verify NFS-over-RDMA functionality on a file opened for reading (small file) ``` ### read03 ``` Verify NFS-over-RDMA functionality on a file opened for reading (medium file) ``` ### read04 ``` Verify NFS-over-RDMA functionality on a file opened for reading (large file) ``` ### write ``` Run all NFS-over-RDMA functionality tests where file is opened for writing: write01, write02, write03, write04 ``` ### write01 ``` Verify NFS-over-RDMA functionality on a file opened for writing (very small file) ``` ### write02 ``` Verify NFS-over-RDMA functionality on a file opened for writing (small file) ``` ### write03 ``` Verify NFS-over-RDMA functionality on a file opened for writing (medium file) ``` ### write04 ``` Verify NFS-over-RDMA functionality on a file opened for writing (large file) ``` ### all ``` Run all tests: basic01, basic02, basic03, basic04, basic05, read01, read02, read03, read04, write01, write02, write03, write04 ``` ## EXAMPLES The only required option is --server\ $ nfstest\_rdma --server 192.168.0.11 ## NOTES The user id in the local host must have access to run commands as root using the 'sudo' command without the need for a password. ## SEE ALSO **[formatstr(3)](/formatstr/3.md),** **[nfstest.test\_util(3)](/nfstest.test_util/3.md),** **[nfstest\_alloc(1)](/nfstest_alloc/1.md),** **[nfstest\_cache(1)](/nfstest_cache/1.md),** **[nfstest\_delegation(1)](/nfstest_delegation/1.md),** **[nfstest\_dio(1)](/nfstest_dio/1.md),** **[nfstest\_fcmp(1)](/nfstest_fcmp/1.md),** **[nfstest\_file(1)](/nfstest_file/1.md),** **[nfstest\_interop(1)](/nfstest_interop/1.md),** **[nfstest\_io(1)](/nfstest_io/1.md),** **[nfstest\_lock(1)](/nfstest_lock/1.md),** **[nfstest\_pkt(1)](/nfstest_pkt/1.md),** **[nfstest\_pnfs(1)](/nfstest_pnfs/1.md),** **[nfstest\_posix(1)](/nfstest_posix/1.md),** **[nfstest\_sparse(1)](/nfstest_sparse/1.md),** **[nfstest\_ssc(1)](/nfstest_ssc/1.md),** **[nfstest\_xattr(1)](/nfstest_xattr/1.md),** **[nfstest\_xid(1)](/nfstest_xid/1.md),** **[packet.application.rpc\_const(3)](/packet.application.rpc_const/3.md),** **[packet.application.rpcordma\_const(3)](/packet.application.rpcordma_const/3.md),** **[packet.nfs.nfs3\_const(3)](/packet.nfs.nfs3_const/3.md),** **[packet.nfs.nfs4\_const(3)](/packet.nfs.nfs4_const/3.md),** **[packet.transport.ib(3)](/packet.transport.ib/3.md),** **[packet.transport.rdmap(3)](/packet.transport.rdmap/3.md)** ## BUGS No known bugs. ## AUTHOR Jorge Mora (mora@netapp.com)