--- name: "nvme" section: "4" platform: "freebsd" package: "base" version: "16.0-CURRENT-20261027-61c70dc16a79" release: "FreeBSD 16.0-CURRENT (2026-10-27)" platforms: ["freebsd", "openbsd"] url: "https://new.carta.tech/nvme/4" --- # nvme(4) NVM Express core driver ## NAME **nvme** – NVM Express core driver ## SYNOPSIS **device** **nvme** **options** **NVME\_USE\_NVD=0** **options** **NVME\_2X\_RESET** In [loader.conf(5)](/loader.conf/5.md): **hw.nvme.per\_cpu\_io\_queues=***0* **hw.nvme.min\_cpus\_per\_ioq=**⟨*X*⟩ **hw.nvme.force\_intx=***1* **hw.nvme.hmb\_max** **hw.nvme.apst\_enable=***1* **hw.nvme.apst\_data** **hw.nvme.use\_nvd=***0* **nvme\_load="YES"** In [loader.conf(5)](/loader.conf/5.md) or [sysctl.conf(5)](/sysctl.conf/5.md): **dev.nvme.0.num\_cpus\_per\_ioq** **dev.nvme.0.int\_coal\_time** **dev.nvme.0.int\_coal\_threshold** **dev.nvme.0.ioq0.num\_entries** **dev.nvme.0.ioq0.num\_tr** **dev.nvme.0.ioq0.num\_prp\_list** **dev.nvme.0.ioq0.sq\_head** **dev.nvme.0.ioq0.sq\_tail** **dev.nvme.0.ioq0.cq\_head** **dev.nvme.0.ioq0.num\_cmds** **dev.nvme.0.ioq0.dump\_debug** ## DESCRIPTION The **nvme** driver provides support for NVM Express (NVMe) controllers, such as: - Hardware initialization - Per-CPU IO queue pairs - API for registering NVMe namespace consumers such as [nvd(4)](/nvd/4.md) or [nda(4)](/nda/4.md) - API for submitting NVM commands to namespaces - Ioctls for controller and namespace configuration and management The **nvme** driver creates controller device nodes in the format */dev/nvmeX* and namespace device nodes in the format */dev/nvmeXnsY*. Note that the NVM Express specification starts numbering namespaces at 1, not 0, and this driver follows that convention. Most users will also want to enable [nvd(4)](/nvd/4.md) or [nda(4)](/nda/4.md) to expose NVM Express namespaces as disk devices which can be partitioned. Note that in NVM Express terms, a namespace is roughly equivalent to a SCSI LUN. ## KERNEL CONFIGURATION The following settings can be configured at kernel compile time, see [build(7)](/build/7.md): - **options** **NVME\_2X\_RESET** Reset controller twice upon boot. This should not be necessary. - **options** **NVME\_USE\_NVD=0** Use the [nda(4)](/nda/4.md) driver to provide a disk device instead of the default [nvd(4)](/nvd/4.md). The [nvd(4)](/nvd/4.md) driver performs better with smaller transactions and few TRIM commands. It sends all commands directly to the drive immediately. The [nda(4)](/nda/4.md) driver performs better with larger transactions and also collapses TRIM commands giving better performance. It can queue commands to the drive; combine BIO\_DELETE commands into a single trip; and use the CAM I/O scheduler to bias one type of operation over another. This value can also be set at boot time with the *hw.nvme.use\_nvd=0* tunable. ## LOADER TUNABLES The following settings can be configured in [loader.conf(5)](/loader.conf/5.md) or at the boot [loader(8)](/loader/8.md) prompt during runtime: - *hw.nvme.per\_cpu\_io\_queues=0* Force a single I/O queue pair shared by all CPUs. By default, **nvme** will create an I/O queue pair for each CPU, provided enough MSI-X vectors and NVMe queue pairs can be allocated. If not enough vectors or queue pairs are available, [nvme(4)](/nvme/4.md) will use a smaller number of queue pairs and assign multiple CPUs per queue pair. - *hw.nvme.min\_cpus\_per\_ioq=X* Assign more than one CPU per I/O queue pair, thereby reducing the number of MSI-X vectors consumed by the device. - *hw.nvme.force\_intx=1* Force legacy interrupts for all **nvme** driver instances. Note that use of INTx implies disabling of per-CPU I/O queue pairs. - *hw.nvme.hmb\_max* Control maximum amount of system RAM in bytes to use as Host Memory Buffer for capable devices. The default value is 5% of physical memory size per device. - *hw.nvme.apst\_enable=1* Enable Autonomous Power State Transition (APST). - *hw.nvme.apst\_data* Override default APST settings. If unset, the default vendor-provided settings, if any, will be applied. The string must contain up to 32 encoded integers, e.g. “0x6418 0 0 0x3e820”. Each value corresponds to a specific available power state starting from the lowest, and defines the target state (bits 3..7) to transition to, as well as the idle time in milliseconds (bits 8..31) to wait before that transition. Bits 0..2 must be zero. - *hw.nvme.use\_nvd=0* The [nvd(4)](/nvd/4.md) driver is used to provide a disk driver to the system by default. The [nda(4)](/nda/4.md) driver can also be used instead. This value may also be set in the kernel config file with **options** **NVME\_USE\_NVD=0**. - *hw.nvme.verbose\_cmd\_dump=1* Enable verbose error messages. When there is an error, **nvme** prints only the most relevant information about the command by default. ## SYSCTL VARIABLES The following controller-level sysctls are currently implemented: - *dev.nvme.0.num\_cpus\_per\_ioq* (R) Number of CPUs associated with each I/O queue pair. - *dev.nvme.0.int\_coal\_time* (R/W) Interrupt coalescing timer period in microseconds. Set to 0 to disable. - *dev.nvme.0.int\_coal\_threshold* (R/W) Interrupt coalescing threshold in number of command completions. Set to 0 to disable. The following queue pair-level sysctls are currently implemented. Admin queue sysctls take the format of dev.nvme.0.adminq and I/O queue sysctls take the format of dev.nvme.0.ioq0. - *dev.nvme.0.ioq0.num\_entries* (R) Number of entries in this queue pair's command and completion queue. - *dev.nvme.0.ioq0.num\_tr* (R) Number of nvme\_tracker structures currently allocated for this queue pair. - *dev.nvme.0.ioq0.num\_prp\_list* (R) Number of nvme\_prp\_list structures currently allocated for this queue pair. - *dev.nvme.0.ioq0.sq\_head* (R) Current location of the submission queue head pointer as observed by the driver. The head pointer is incremented by the controller as it takes commands off of the submission queue. - *dev.nvme.0.ioq0.sq\_tail* (R) Current location of the submission queue tail pointer as observed by the driver. The driver increments the tail pointer after writing a command into the submission queue to signal that a new command is ready to be processed. - *dev.nvme.0.ioq0.cq\_head* (R) Current location of the completion queue head pointer as observed by the driver. The driver increments the head pointer after finishing with a completion entry that was posted by the controller. - *dev.nvme.0.ioq0.num\_cmds* (R) Number of commands that have been submitted on this queue pair. - *dev.nvme.0.ioq0.dump\_debug* (W) Writing 1 to this sysctl will dump the full contents of the submission and completion queues to the console. In addition to the typical pci attachment, the **nvme** driver supports attaching to a [ahci(4)](/ahci/4.md) device. Intel's Rapid Storage Technology (RST) hides the nvme device behind the AHCI device due to limitations in Windows. However, this effectively hides it from the FreeBSD kernel. To work around this limitation, FreeBSD detects that the AHCI device supports RST and when it is enabled. See [ahci(4)](/ahci/4.md) for more details. ## DIAGNOSTICS - nvme%d: System interrupt issues? The driver found a timed-out transaction had a pending completion record, indicating an interrupt had not been delivered. The system is either not configuring interrupts properly, or the system drops them under load. This message will appear at most once per boot per controller. ## SEE ALSO [nda(4)](/nda/4.md), [nvd(4)](/nvd/4.md), [pci(4)](/pci/4.md), [nvmecontrol(8)](/nvmecontrol/8.md), [disk(9)](/disk/9.md) ## HISTORY The **nvme** driver first appeared in FreeBSD 9.2. ## AUTHORS The **nvme** driver was developed by Intel and originally written by Jim Harris ⟨*jimharris@FreeBSD.org*⟩, with contributions from Joe Golio at EMC. This man page was written by Jim Harris ⟨*jimharris@FreeBSD.org*⟩.