--- name: "hwpmc" section: "4" platform: "linux" package: "freebsd-manpages" version: "6.0-5" release: "Debian 4.0 (Etch)" platforms: ["linux", "freebsd"] url: "https://new.carta.tech/hwpmc" --- # hwpmc(4) Hardware Performance Monitoring Counter support ## NAME **hwpmc** – Hardware Performance Monitoring Counter support ## SYNOPSIS **options HWPMC\_HOOKS** **device hwpmc** Additionally, for i386 systems: **device** **apic** ## DESCRIPTION The **hwpmc** driver virtualizes the hardware performance monitoring facilities in modern CPUs and provides support for using these facilities from user level processes. The driver supports multi-processor systems. PMCs are allocated using the PMC\_OP\_PMCALLOCATE request. A successful PMC\_OP\_PMCALLOCATE request will return an integer handle (typically a small integer) to the requesting process. Subsequent operations on the allocated PMC use this handle to denote the specific PMC. A process that has successfully allocated a PMC is termed an “owner process”. PMCs may be allocated to operate in process-private or in system-wide modes. - *Process-private* In process-private mode, a PMC is active only when a thread belonging to a process it is attached to is scheduled on a CPU. - *System-wide* In system-wide mode, a PMC operates independently of processes and measures hardware events for the system as a whole. The **hwpmc** driver supports the use of hardware PMCs for counting or for sampling: - *Counting* In counting modes, the PMCs count hardware events. These counts are retrievable using the PMC\_OP\_PMCREAD system call on all architectures, though some architectures like the i386 and amd64 offer faster methods of reading these counts. - *Sampling* In sampling modes, where PMCs are configured to sample the CPU instruction pointer after a configurable number of hardware events have been observed. These instruction pointer samples are directed to a log file for subsequent analysis. These modes of operation are orthogonal; a PMC may be configured to operate in one of four modes: - Process-private, counting These PMCs count hardware events whenever a thread in their attached process is scheduled on a CPU. These PMCs normally count from zero, but the initial count may be set using the PMC\_OP\_SETCOUNT operation. Applications can read the value of the PMC anytime using the PMC\_OP\_PMCRW operation. - Process-private, sampling These PMCs sample the target processes instruction pointer after they have seen the configured number of hardware events. The PMCs only count events when a thread belonging to their attached process is active. The desired frequency of sampling is set using the PMC\_OP\_SETCOUNT operation prior to starting the PMC. Log files are configured using the PMC\_OP\_CONFIGURELOG operation. - System-wide, counting These PMCs count hardware events seen by them independent of the processes that are executing. The current count on these PMCs can be read using the PMC\_OP\_PMCRW request. These PMCs normally count from zero, but the initial count may be set using the PMC\_OP\_SETCOUNT operation. - System-wide, sampling These PMCs will periodically sample the instruction pointer of the CPU they are allocated on, and will write the sample to a log for further processing. The desired frequency of sampling is set using the PMC\_OP\_SETCOUNT operation prior to starting the PMC. Log files are configured using the PMC\_OP\_CONFIGURELOG operation. System-wide statistical sampling can only be enabled by a process with super-user privileges. Processes are allowed to allocate as many PMCs are the hardware and current operating conditions permit. Processes may mix allocations of system-wide and process-private PMCs. Multiple processes are allowed to be concurrently using the facilities of the **hwpmc** driver. Allocated PMCs are started using the PMC\_OP\_PMCSTART operation, and stopped using the PMC\_OP\_PMCSTOP operation. Stopping and starting a PMC is permitted at any time the owner process has a valid handle to the PMC. Process-private PMCs need to be attached to a target process before they can be used. Attaching a process to a PMC is done using the PMC\_OP\_PMCATTACH operation. An already attached PMC may be detached from its target process using the converse PMC\_OP\_PMCDETACH operation. Issuing a PMC\_OP\_PMCSTART operation on an as yet unattached PMC will cause it to be attached to its owner process. The following rules determine whether a given process may attach a PMC to another target process: - A non-jailed process with super-user privileges is allowed to attach to any other process in the system. - Other processes are only allowed to attach to targets that they would be able to attach to for debugging (as determined by [p\_candebug(9)](/p_candebug/9.md)). PMCs are released using PMC\_OP\_PMCRELEASE. After a successful PMC\_OP\_PMCRELEASE operation the handle to the PMC will become invalid. ### Modifier Flags The PMC\_OP\_PMCALLOCATE operation supports the following flags that modify the behavior of an allocated PMC: - PMC\_F\_DESCENDANTS This modifier is valid only for a PMC being allocated in process-private mode. It signifies that the PMC will track hardware events for its target process and the target's current and future descendants. - PMC\_F\_KGMON This modifier is valid only for a PMC being allocated in system-wide sampling mode. It signifies that the PMC's sampling interrupt is to be used to drive kernel profiling via [kgmon(8)](/kgmon/8.md). - PMC\_F\_LOG\_PROCCSW This modifier is valid only for a PMC being allocated in process-private mode. When this modifier is present, at every context switch, **hwpmc** will log a record containing the number of hardware events seen by the target process when it was scheduled on the CPU. - PMC\_F\_LOG\_PROCEXIT This modifier is valid only for a PMC being allocated in process-private mode. With this modifier present, **hwpmc** will maintain per-process counts for each target process attached to a PMC. At process exit time, a record containing the target process' PID and the accumulated per-process count for that process will be written to the configured log file. Modifiers PMC\_F\_LOG\_PROCEXIT and PMC\_F\_LOG\_PROCCSW may be used in combination with modifier PMC\_F\_DESCENDANTS to track the behaviour of complex pipelines of processes. PMCs with modifiers PMC\_F\_LOG\_PROCEXIT and PMC\_F\_LOG\_PROCCSW cannot be started until their owner process has configured a log file. ### Signals The **hwpmc** driver may deliver signals to processes that have allocated PMCs: - SIGIO A PMC\_OP\_PMCRW operation was attempted on a process-private PMC that does not have attached target processes. - SIGBUS The **hwpmc** driver is being unloaded from the kernel. ## PROGRAMMING API The recommended way for application programs to use the facilities of the **hwpmc** driver is using the API provided by the [pmc(3)](/pmc/3.md) library. The **hwpmc** driver operates using a system call number that is dynamically allotted to it when it is loaded into the kernel. The **hwpmc** driver supports the following operations: - PMC\_OP\_CONFIGURELOG Configure a log file for sampling mode PMCs. - PMC\_OP\_FLUSHLOG Transfer buffered log data inside **hwpmc** to a configured output file. This operation returns to the caller after the write operation has returned. - PMC\_OP\_GETCPUINFO Retrieve information about the number of CPUs on the system and the number of hardware performance monitoring counters available per-CPU. - PMC\_OP\_GETDRIVERSTATS Retrieve module statistics (for analyzing the behavior of **hwpmc** itself). - PMC\_OP\_GETMODULEVERSION Retrieve the version number of API. - PMC\_OP\_GETPMCINFO Retrieve information about the current state of the PMCs on a given CPU. - PMC\_OP\_PMCADMIN Set the administrative state (i.e., whether enabled or disabled) for the hardware PMCs managed by the **hwpmc** driver. - PMC\_OP\_PMCALLOCATE Allocate and configure a PMC. On successful allocation, a handle to the PMC (a small integer) is returned. - PMC\_OP\_PMCATTACH Attach a process mode PMC to a target process. The PMC will be active whenever a thread in the target process is scheduled on a CPU. If the PMC\_F\_DESCENDANTS flag had been specified at PMC allocation time, then the PMC is attached to all current and future descendants of the target process. - PMC\_OP\_PMCDETACH Detach a PMC from its target process. - PMC\_OP\_PMCRELEASE Release a PMC. - PMC\_OP\_PMCRW Read and write a PMC. This operation is valid only for PMCs configured in counting modes. - PMC\_OP\_SETCOUNT Set the initial count (for counting mode PMCs) or the desired sampling rate (for sampling mode PMCs). - PMC\_OP\_PMCSTART Start a PMC. - PMC\_OP\_PMCSTOP Stop a PMC. - PMC\_OP\_WRITELOG Insert a timestamped user record into the log file. ### i386 Specific API Some i386 family CPUs support the RDPMC instruction which allows a user process to read a PMC value without needing to invoke a PMC\_OP\_PMCRW operation. On such CPUs, the machine address associated with an allocated PMC is retrievable using the PMC\_OP\_PMCX86GETMSR system call. - PMC\_OP\_PMCX86GETMSR Retrieve the MSR (machine specific register) number associated with the given PMC handle. The PMC needs to be in process-private mode and allocated without the PMC\_F\_DESCENDANTS modifier flag, and should be attached only to its owner process at the time of the call. ### amd64 Specific API AMD64 CPUs support the RDPMC instruction which allows a user process to read a PMC value without needing to invoke a PMC\_OP\_PMCRW operation. The machine address associated with an allocated PMC is retrievable using the PMC\_OP\_PMCX86GETMSR system call. - PMC\_OP\_PMCX86GETMSR Retrieve the MSR (machine specific register) number associated with the given PMC handle. The PMC needs to be in process-private mode and allocated without the PMC\_F\_DESCENDANTS modifier flag, and should be attached only to its owner process at the time of the call. ## SYSCTL VARIABLES AND LOADER TUNABLES The behavior of **hwpmc** is influenced by the following [sysctl(8)](/sysctl/8.md) and [loader(8)](/loader/8.md) tunables: - *kern.hwpmc.debugflags* (string, read-write) (Only available if the **hwpmc** driver was compiled with **-DDEBUG**.) Control the verbosity of debug messages from the **hwpmc** driver. - *kern.hwpmc.hashsize* (integer, read-only) The number of rows in the hash tables used to keep track of owner and target processes. The default is 16. - *kern.hwpmc.logbuffersize* (integer, read-only) The size in kilobytes of each log buffer used by **hwpmc**'s logging function. The default buffer size is 4KB. - *kern.hwpmc.mtxpoolsize* (integer, read-only) The size of the spin mutex pool used by the PMC driver. The default is 32. - *kern.hwpmc.nbuffers* (integer, read-only) The number of log buffers used by **hwpmc** for logging. The default is 16. - *kern.hwpmc.nsamples* (integer, read-only) The number of entries in the per-CPU ring buffer used during sampling. The default is 16. - *security.bsd.unprivileged\_syspmcs* (boolean, read-write) If set to non-zero, allow unprivileged processes to allocate system-wide PMCs. The default value is 0. - *security.bsd.unprivileged\_proc\_debug* (boolean, read-write) If set to 0, the **hwpmc** driver will only allow privileged processes to attach PMCs to other processes. These variables may be set in the kernel environment using [kenv(1)](/kenv/1.md) before **hwpmc** is loaded. ## SECURITY CONSIDERATIONS PMCs may be used to monitor the actual behaviour of the system on hardware. In situations where this constitutes an undesirable information leak, the following options are available: 1. Set the [sysctl(8)](/sysctl/8.md) tunable *security.bsd.unprivileged\_syspmcs* to 0. This ensures that unprivileged processes cannot allocate system-wide PMCs and thus cannot observe the hardware behavior of the system as a whole. This tunable may also be set at boot time using [loader(8)](/loader/8.md), or with [kenv(1)](/kenv/1.md) prior to loading the **hwpmc** driver into the kernel. 2. Set the [sysctl(8)](/sysctl/8.md) tunable *security.bsd.unprivileged\_proc\_debug* to 0. This will ensure that an unprivileged process cannot attach a PMC to any process other than itself and thus cannot observe the hardware behavior of other processes with the same credentials. System administrators should note that on IA-32 platforms FreeBSD makes the content of the IA-32 TSC counter available to all processes via the RDTSC instruction. ## IMPLEMENTATION NOTES ### SMP Symmetry The kernel driver requires all physical CPUs in an SMP system to have identical performance monitoring counter hardware. ### x86 TSC Handling Historically, on the x86 architecture, FreeBSD has permitted user processes running at a processor CPL of 3 to read the TSC using the RDTSC instruction. The **hwpmc** driver preserves this semantics. ### Intel P4/HTT Handling On CPUs with HTT support, Intel P4 PMCs are capable of qualifying only a subset of hardware events on a per-logical CPU basis. Consequently, if HTT is enabled on a system with Intel Pentium P4 PMCs, then the **hwpmc** driver will reject allocation requests for process-private PMCs that request counting of hardware events that cannot be counted separately for each logical CPU. ### Intel Pentium-Pro Handling Writing a value to the PMC MSRs found ing Intel Pentium-Pro style PMCs (found in Intel Pentium Pro, Pentium II, Pentium III, Pentium M and Celeron processors) will replicate bit 31 of the value being written into the upper 8 bits of the MSR, bringing down the usable width of these PMCs to 31 bits. For process-virtual PMCs, the **hwpmc** driver implements a workaround in software and makes the corrected 64 bit count available via the PMC\_OP\_RW operation. Processes that intend to use RDPMC instructions directly or that intend to write values larger than 2^31 into these PMCs with PMC\_OP\_RW need to be aware of this hardware limitation. ## DIAGNOSTICS - hwpmc: \[class/npmc/capabilities\]... Announce the presence of *npmc* PMCs of class *class*, with capabilities described by bit string *capabilities*. - hwpmc: kernel version (0x%x) does not match module version (0x%x). The module loading process failed because a version mismatch was detected between the currently executing kernel and the module being loaded. - hwpmc: this kernel has not been compiled with 'options HWPMC\_HOOKS'. The module loading process failed because the currently executing kernel was not configured with the required configuration option **HWPMC\_HOOKS**. - hwpmc: tunable hashsize=%d must be greater than zero. A negative value was supplied for tunable *kern.hwpmc.hashsize*. - hwpmc: tunable logbuffersize=%d must be greater than zero. A negative value was supplied for tunable *kern.hwpmc.logbuffersize*. - hwpmc: tunable nlogbuffers=%d must be greater than zero. A negative value was supplied for tunable *kern.hwpmc.nlogbuffers*. - hwpmc: tunable nsamples=%d out of range. The value for tunable *kern.hwpmc.nsamples* was negative or greater than 65535. ## COMPATIBILITY The **hwpmc** driver is currently under development. The API and ABI documented in this manual page may change in the future. The recommended method of accessing this driver is using the [pmc(3)](/pmc/3.md) API. ## ERRORS A command issued to the **hwpmc** driver may fail with the following errors: - \[EBUSY\] A PMC\_OP\_CONFIGURELOG operation was requested while an existing log was active. - \[EBUSY\] A DISABLE operation was requested using the PMC\_OP\_PMCADMIN request for a set of hardware resources currently in use for process-private PMCs. - \[EBUSY\] A PMC\_OP\_PMCADMIN operation was requested on an active system mode PMC. - \[EBUSY\] A PMC\_OP\_PMCATTACH operation was requested for a target process that already had another PMC using the same hardware resources attached to it. - \[EBUSY\] A PMC\_OP\_PMCRW request writing a new value was issued on a PMC that was active. - \[EBUSY\] A PMC\_OP\_PMCSETCOUNT request was issued on a PMC that was active. - \[EDOOFUS\] A PMC\_OP\_PMCSTART operation was requested without a log file being configured for a PMC allocated with PMC\_F\_LOG\_PROCCSW and PMC\_F\_LOG\_PROCEXIT modifiers. - \[EEXIST\] A PMC\_OP\_PMCATTACH request was reissued for a target process that already is the target of this PMC. - \[EFAULT\] A bad address was passed in to the driver. - \[EINVAL\] A process specified an invalid PMC handle. - \[EINVAL\] An invalid CPU number was passed in for a PMC\_OP\_GETPMCINFO operation. - \[EINVAL\] An invalid CPU number was passed in for a PMC\_OP\_PMCADMIN operation. - \[EINVAL\] An invalid operation request was passed in for a PMC\_OP\_PMCADMIN operation. - \[EINVAL\] An invalid PMC ID was passed in for a PMC\_OP\_PMCADMIN operation. - \[EINVAL\] A suitable PMC matching the parameters passed in to a PMC\_OP\_PMCALLOCATE request could not be allocated. - \[EINVAL\] An invalid PMC mode was requested during a PMC\_OP\_PMCALLOCATE request. - \[EINVAL\] An invalid CPU number was specified during a PMC\_OP\_PMCALLOCATE request. - \[EINVAL\] A CPU other than PMC\_CPU\_ANY was specified in a PMC\_OP\_ALLOCATE request for a process-private PMC. - \[EINVAL\] A CPU number of PMC\_CPU\_ANY was specified in a PMC\_OP\_ALLOCATE request for a system-wide PMC. - \[EINVAL\] The *pm\_flags* argument to an PMC\_OP\_PMCALLOCATE request contained unknown flags. - \[EINVAL\] A PMC allocated for system-wide operation was specified with a PMC\_OP\_PMCATTACH request. - \[EINVAL\] The *pm\_pid* argument to a PMC\_OP\_PMCATTACH request specified an illegal process ID. - \[EINVAL\] A PMC\_OP\_PMCDETACH request was issued for a PMC not attached to the target process. - \[EINVAL\] Argument *pm\_flags* to a PMC\_OP\_PMCRW request contained illegal flags. - \[EINVAL\] A PMC\_OP\_PMCX86GETMSR operation was requested for a PMC not in process-virtual mode, or for a PMC that is not solely attached to its owner process, or for a PMC that was allocated with flag PMC\_F\_DESCENDANTS. - \[EINVAL\] (On Intel Pentium 4 CPUs with HTT support) An allocation request for a process-private PMC was issued for an event that does not support counting on a per-logical CPU basis. - \[ENOMEM\] The system was not able to allocate kernel memory. - \[ENOSYS\] (i386 architectures) A PMC\_OP\_PMCX86GETMSR operation was requested for hardware that does not support reading PMCs directly with the RDPMC instruction. - \[ENXIO\] A PMC\_OP\_GETPMCINFO operation was requested for a disabled CPU. - \[ENXIO\] A system-wide PMC on a disabled CPU was requested to be allocated with PMC\_OP\_PMCALLOCATE. - \[ENXIO\] A PMC\_OP\_PMCSTART or PMC\_OP\_PMCSTOP request was issued for a system-wide PMC that was allocated on a currently disabled CPU. - \[EOPNOTSUPP\] A PMC\_OP\_PMCALLOCATE request was issued for PMC capabilities not supported by the specified PMC class. - \[EPERM\] A PMC\_OP\_PMCADMIN request was issued by a process without super-user privilege or by a jailed super-user process. - \[EPERM\] A PMC\_OP\_PMCATTACH operation was issued for a target process that the current process does not have permission to attach to. - \[EPERM\] (i386 and amd64 architectures) A PMC\_OP\_PMCATTACH operation was issued on a PMC whose MSR has been retrieved using PMC\_OP\_PMCX86GETMSR. - \[ESRCH\] A process issued a PMC operation request without having allocated any PMCs. - \[ESRCH\] A process issued a PMC operation request after the PMC was detached from all of its target processes. - \[ESRCH\] A PMC\_OP\_PMCATTACH request specified a non-existent process ID. - \[ESRCH\] The target process for a PMC\_OP\_PMCDETACH operation is not being monitored by the **hwpmc** driver. ## SEE ALSO [kenv(1)](/kenv/1.md), [pmc(3)](/pmc/3.md), [pmclog(3)](/pmclog/3.md), [kgmon(8)](/kgmon/8.md), [kldload(8)](/kldload/8.md), [pmccontrol(8)](/pmccontrol/8.md), [pmcstat(8)](/pmcstat/8.md), [sysctl(8)](/sysctl/8.md), [p\_candebug(9)](/p_candebug/9.md) ## HISTORY The **hwpmc** driver first appeared in FreeBSD 6.0. ## BUGS The driver samples the state of the kernel's logical processor support at the time of initialization (i.e., at module load time). On CPUs supporting logical processors, the driver could misbehave if logical processors are subsequently enabled or disabled while the driver is active. On the i386 architecture, the driver requires that the local APIC on the CPU be enabled for sampling mode to be supported. Many single-processor motherboards keep the APIC disabled in BIOS; on such systems **hwpmc** will not support sampling PMCs.