--- name: "fi_cntr_add" section: "3" platform: "linux" package: "libfabric" version: "2.7.0-1" release: "Arch Linux 2026-11-01" platforms: ["linux"] url: "https://new.carta.tech/fi_cntr" --- # fi\_cntr\_add(3) Completion and event counter operations Part of [fi\_cntr(3)](/fi_cntr/3.md). ## NAME fi\_cntr - Completion and event counter operations - fi\_cntr\_open / fi\_close Allocate/free a counter - fi\_cntr\_read Read the current value of a counter - fi\_cntr\_readerr Reads the number of operations which have completed in error. - fi\_cntr\_add Increment a counter by a specified value - fi\_cntr\_set Set a counter to a specified value - fi\_cntr\_wait Wait for a counter to be greater or equal to a threshold value - fi\_cntr\_export\_xpu Export a counter for XPU device-side access. ## SYNOPSIS - ``` #include int fi_cntr_open(struct fid_domain *domain, struct fi_cntr_attr *attr, struct fid_cntr **cntr, void *context); int fi_close(struct fid *cntr); uint64_t fi_cntr_read(struct fid_cntr *cntr); uint64_t fi_cntr_readerr(struct fid_cntr *cntr); int fi_cntr_add(struct fid_cntr *cntr, uint64_t value); int fi_cntr_adderr(struct fid_cntr *cntr, uint64_t value); int fi_cntr_set(struct fid_cntr *cntr, uint64_t value); int fi_cntr_seterr(struct fid_cntr *cntr, uint64_t value); int fi_cntr_wait(struct fid_cntr *cntr, uint64_t threshold, int timeout); int fi_cntr_export_xpu(struct fid_cntr *cntr, uint64_t flags, struct fid_xpu_cntr *xpu_cntr); ``` ## ARGUMENTS - *domain* Fabric domain - *cntr* Fabric counter - *attr* Counter attributes - *context* User specified context associated with the counter - *value* Value to increment or set counter - *threshold* Value to compare counter against - *timeout* Time in milliseconds to wait. A negative value indicates infinite timeout. ## DESCRIPTION Counters record the number of requested operations that have completed. Counters can provide a light-weight completion mechanism by allowing the suppression of CQ completion entries. They are useful for applications that only need to know the number of requests that have completed, and not details about each request. For example, counters may be useful for implementing credit based flow control or tracking the number of remote processes that have responded to a request. Counters typically only count successful completions. However, if an operation completes in error, it may increment an associated error value. That is, a counter actually stores two distinct values, with error completions updating an error specific value. Counters are updated following the completion event semantics defined in [fi\_cq(3)](/fi_cq/3.md). The timing of the update is based on the type of transfer and any specified operation flags. ### fi\_cntr\_open fi\_cntr\_open allocates a new fabric counter. The properties and behavior of the counter are defined by struct fi\_cntr\_attr. - ``` struct fi_cntr_attr { enum fi_cntr_events events; /* type of events to count */ enum fi_wait_obj wait_obj; /* requested wait object */ struct fid_wait *wait_set; /* optional wait set, deprecated */ uint64_t flags; /* operation flags */ struct fid_xpu_ctx *xpu_ctx; /* optional XPU context */ }; ``` - *events* A counter captures different types of events. The specific type which is to counted are one of the following: - \- *FI\_CNTR\_EVENTS\_COMP* The counter increments for every successful completion that occurs on an associated bound endpoint. The type of completions – sends and/or receives – which are counted may be restricted using control flags when binding the counter and the endpoint. Counters increment on all successful completions, separately from whether the operation generates an entry in an event queue. - \- *FI\_CNTR\_EVENTS\_BYTES* The counter is incremented by the number of user bytes, excluding any CQ data, transferred in a transport message upon reaching the specified completion semantic. For initiator side counters, the count reflects the size of the requested transfer and is updated after the message reaches the desired completion level (FI\_INJECT\_COMPLETE, FI\_TRANSMIT\_COMPLETE, etc.). For send and write operations, the count reflects the number of bytes transferred to the peer. For read operations, the count reflects the number of bytes returned in a read response. Operations which may both write and read data, such as atomics, behave as read operations at the initiator, but writes at the target. For target side counters, the count reflects the size of received user data and is incremented subject to target side completion semantics. In most cases, this indicates FI\_DELIVERY\_COMPLETE, but may differ when accessing device memory (HMEM). On error, the tranfer size is not applied to the error field, that field is increment by 1. The FI\_COLLECTIVE transfer type is not supported. - *wait\_obj* Counters may be associated with a specific wait object. Wait objects allow applications to block until the wait object is signaled, indicating that a counter has reached a specific threshold. Users may use fi\_control to retrieve the underlying wait object associated with a counter, in order to use it in other system calls. The following values may be used to specify the type of wait object associated with a counter: FI\_WAIT\_NONE, FI\_WAIT\_UNSPEC, FI\_WAIT\_SET, FI\_WAIT\_FD, FI\_WAIT\_MUTEX\_COND (deprecated), and FI\_WAIT\_YIELD. The default is FI\_WAIT\_NONE. - \- *FI\_WAIT\_NONE* Used to indicate that the user will not block (wait) for events on the counter. - \- *FI\_WAIT\_UNSPEC* Specifies that the user will only wait on the counter using fabric interface calls, such as fi\_cntr\_wait. In this case, the underlying provider may select the most appropriate or highest performing wait object available, including custom wait mechanisms. Applications that select FI\_WAIT\_UNSPEC are not guaranteed to retrieve the underlying wait object. - \- *FI\_WAIT\_SET* (deprecated) Indicates that the event counter should use a wait set object to wait for events. If specified, the wait\_set field must reference an existing wait set object. - \- *FI\_WAIT\_FD* Indicates that the counter should use a file descriptor as its wait mechanism. A file descriptor wait object must be usable in select, poll, and epoll routines. However, a provider may signal an FD wait object by marking it as readable, writable, or with an error. - \- *FI\_WAIT\_MUTEX\_COND* (deprecated) Specifies that the counter should use a pthread mutex and cond variable as a wait object. - \- *FI\_WAIT\_YIELD* Indicates that the counter will wait without a wait object but instead yield on every wait. Allows usage of fi\_cntr\_wait through a spin. - *wait\_set* (deprecated) If wait\_obj is FI\_WAIT\_SET, this field references a wait object to which the event counter should attach. When an event is added to the event counter, the corresponding wait set will be signaled if all necessary conditions are met. The use of a wait\_set enables an optimized method of waiting for events across multiple event counters. This field is ignored if wait\_obj is not FI\_WAIT\_SET. - *flags* Flags that apply to the counter. Set FI\_XPU to create a counter for XPU device-side access. Must be set to 0 if XPU is not used. - *xpu\_ctx* Optional pointer to an XPU context created via fi\_xpu\_ctx. When set together with FI\_XPU in the flags field, the counter is created for XPU device-side access. Counter values on such a counter can only be read or waited on using device-side functions. Host-side read operations are not available. The counter is exported via fi\_cntr\_export\_xpu for device-side use. See [fi\_xpu(3)](/fi_xpu/3.md) for details. This field must be NULL if the counter is not created with FI\_XPU. ### fi\_close The fi\_close call releases all resources associated with a counter. When closing the counter, there must be no opened endpoints, transmit contexts, receive contexts or memory regions associated with the counter. If resources are still associated with the counter when attempting to close, the call will return -FI\_EBUSY. ### fi\_cntr\_control The fi\_cntr\_control call is used to access provider or implementation specific details of the counter. Access to the counter should be serialized across all calls when fi\_cntr\_control is invoked, as it may redirect the implementation of counter operations. The following control commands are usable with a counter: - *FI\_GETOPSFLAG (uint64\_t \*)* Returns the current default operational flags associated with the counter. - *FI\_SETOPSFLAG (uint64\_t \*)* Modifies the current default operational flags associated with the counter. - *FI\_GETWAIT (void \*\*)* This command allows the user to retrieve the low-level wait object associated with the counter. The format of the wait-object is specified during counter creation, through the counter attributes. See fi\_eq.3 for addition details using control with FI\_GETWAIT. ### fi\_cntr\_read The fi\_cntr\_read call returns the current value of the counter. Applications must manage counter values that approach the limit defined by the domain attribute max\_cntr\_value. This value, which defaults to UINT64\_MAX but may be lower due to provider limitations, defines the maximum value returned by fi\_cntr\_read(). When a counter reaches its maximum value and additional operations complete, the counter behavior will be undefined. Applications that need to track the total number of operations beyond the provider’s counter limits are responsible for detecting and handling this condition by resetting counters with fi\_cntr\_set before they reach the maximum, or it could result in incorrect values. ### fi\_cntr\_readerr The read error call returns the number of operations that completed in error and were unable to update the counter. Applications must manage error counter values that approach the limit defined by the domain attribute max\_err\_cntr\_value. This value, which defaults to UINT64\_MAX but may be lower due to provider limitations, defines the maximum value returned by fi\_cntr\_readerr(). When an error counter reaches its maximum value and additional error operations complete, the counter behavior will be undefined. Applications that need to track the total number of error operations beyond the provider’s counter limits are responsible for detecting and handling this condition by resetting the error counters with fi\_cntr\_seterr before they reach the maximum, or it could result in incorrect values. ### fi\_cntr\_add This adds the user-specified value to the counter. ### fi\_cntr\_adderr This adds the user-specified value to the error value of the counter. ### fi\_cntr\_set This sets the counter to the specified value. ### fi\_cntr\_seterr This sets the error value of the counter to the specified value. ### fi\_cntr\_wait This call may be used to wait until the counter reaches the specified threshold, or until an error or timeout occurs. Upon successful return from this call, the counter will be greater than or equal to the input threshold value. If an operation associated with the counter encounters an error, it will increment the error value associated with the counter. Any change in a counter’s error value will unblock any thread inside fi\_cntr\_wait. If the call returns due to timeout, -FI\_ETIMEDOUT will be returned. The error value associated with the counter remains unchanged. It is invalid for applications to call this function if the counter has been configured with a wait object of FI\_WAIT\_NONE or FI\_WAIT\_SET. ### fi\_cntr\_export\_xpu The fi\_cntr\_export\_xpu call exports a counter for XPU access. The counter must have been created with an XPU context set in fi\_cntr\_attr. The caller provides a struct fid\_xpu\_cntr which the provider fills with fclass, prov\_id, and prov\_ctx (see [fi\_xpu(3)](/fi_xpu/3.md) for provider dispatch details). The application must copy the structure to device-accessible memory before passing it to device-side counter functions (fi\_xpu\_cntr\_read, fi\_xpu\_cntr\_wait). ## RETURN VALUES Returns 0 on success. On error, a negative value corresponding to fabric errno is returned. - fi\_cntr\_read / fi\_cntr\_readerr Returns the current value of the counter. - fi\_cntr\_export\_xpu Returns 0 on success. On error, returns a negative fabric errno. Returns -FI\_ENOSYS if the provider does not support exporting the counter for XPU access. Returns -FI\_EINVAL if the counter was not created with an XPU context. Fabric errno values are defined in rdma/fi\_errno.h. ## NOTES In order to support a variety of counter implementations, updates made to counter values (e.g. fi\_cntr\_set or fi\_cntr\_add) may not be immediately visible to counter read operations (i.e. fi\_cntr\_read or fi\_cntr\_readerr). A small, but undefined, delay may occur between the counter changing and the reported value being updated. However, a final updated value will eventually be reflected in the read counter value. Additionally, applications should ensure that the value of a counter is stable and not subject to change prior to calling fi\_cntr\_set or fi\_cntr\_seterr. Otherwise, the resulting value of the counter after fi\_cntr\_set / fi\_cntr\_seterr is undefined, as updates to the counter may be lost. A counter value is considered stable if all previous updates using fi\_cntr\_set / fi\_cntr\_seterr and results of related operations are reflected in the observed value of the counter. ## SEE ALSO [fi\_getinfo(3)](/fi_getinfo/3.md), [fi\_endpoint(3)](/fi_endpoint/3.md), [fi\_domain(3)](/fi_domain/3.md), [fi\_eq(3)](/fi_eq/3.md), [fi\_poll(3)](/fi_poll/3.md) ## AUTHORS OpenFabrics.