--- name: "fi_av" section: "3" platform: "linux" package: "libfabric" version: "2.7.0-1" release: "Arch Linux 2026-11-01" platforms: ["linux"] url: "https://new.carta.tech/fi_av" --- # fi\_av(3) Address vector operations ## NAME fi\_av - Address vector operations - fi\_av\_open / fi\_close Open or close an address vector - fi\_av\_bind Associate an address vector with an event queue. This function is deprecated and should not be used. - fi\_av\_insert / fi\_av\_insertsvc / fi\_av\_remove Insert/remove an address into/from the address vector. - fi\_av\_lookup Retrieve an address stored in the address vector. - fi\_av\_straddr Convert an address into a printable string. - fi\_av\_insert\_auth\_key Insert an authorization key into the address vector. - fi\_av\_lookup\_auth\_key Retrieve an authorization key stored in the address vector. - fi\_av\_set\_user\_id Set the user-defined fi\_addr\_t for an inserted fi\_addr\_t. - fi\_av\_lookup2 Look up an address vector entry with extended options. ## SYNOPSIS - ``` #include int fi_av_open(struct fid_domain *domain, struct fi_av_attr *attr, struct fid_av **av, void *context); int fi_close(struct fid *av); int fi_av_bind(struct fid_av *av, struct fid *eq, uint64_t flags); int fi_av_insert(struct fid_av *av, void *addr, size_t count, fi_addr_t *fi_addr, uint64_t flags, void *context); int fi_av_insertsvc(struct fid_av *av, const char *node, const char *service, fi_addr_t *fi_addr, uint64_t flags, void *context); int fi_av_insertsym(struct fid_av *av, const char *node, size_t nodecnt, const char *service, size_t svccnt, fi_addr_t *fi_addr, uint64_t flags, void *context); int fi_av_remove(struct fid_av *av, fi_addr_t *fi_addr, size_t count, uint64_t flags); int fi_av_lookup(struct fid_av *av, fi_addr_t fi_addr, void *addr, size_t *addrlen); fi_addr_t fi_rx_addr(fi_addr_t fi_addr, int rx_index, int rx_ctx_bits); fi_addr_t fi_group_addr(fi_addr_t fi_addr, uint32_t group_id); const char * fi_av_straddr(struct fid_av *av, const void *addr, char *buf, size_t *len); int fi_av_insert_auth_key(struct fid_av *av, const void *auth_key, size_t auth_key_size, fi_addr_t *fi_addr, uint64_t flags); int fi_av_lookup_auth_key(struct fid_av *av, fi_addr_t addr, void *auth_key, size_t *auth_key_size); int fi_av_set_user_id(struct fid_av *av, fi_addr_t fi_addr, fi_addr_t user_id, uint64_t flags); int fi_av_lookup2(struct fid_av *av, fi_addr_t fi_addr, void *buf, size_t *len, uint64_t flags, struct fid_xpu_ctx *ctx); ``` ## ARGUMENTS - *domain* Resource domain - *av* Address vector - *eq* Event queue - *attr* Address vector attributes - *context* User specified context associated with the address vector or insert operation. - *addr* Buffer containing one or more addresses to insert into address vector. - *addrlen* On input, specifies size of addr buffer. On output, stores number of bytes written to addr buffer. - *fi\_addr* For insert, a reference to an array where returned fabric addresses will be written. For remove, one or more fabric addresses to remove. If FI\_AV\_USER\_ID is requested, also used as input into insert calls to assign the user ID with the added address. - *count* Number of addresses to insert/remove from an AV. - *flags* Additional flags to apply to the operation. - *auth\_key* Buffer containing authorization key to be inserted into the address vector. - *auth\_key\_size* On input, specifies size of auth\_key buffer. On output, stores number of bytes written to auth\_key buffer. - *user\_id* For address vectors configured with FI\_AV\_USER\_ID, this defines the user ID to be associated with a specific fi\_addr\_t. ## DESCRIPTION Address vectors are used to map higher-level addresses, which may be more natural for an application to use, into fabric specific addresses. For example, an endpoint may be associated with a struct sockaddr\_in address, indicating the endpoint is reachable using a TCP port number over an IPv4 address. This may hold even if the endpoint communicates using a proprietary network protocol. The purpose of the AV is to associate a higher-level address with a simpler, more efficient value that can be used by the libfabric API in a fabric agnostic way. The mapped address is of type fi\_addr\_t and is returned through an AV insertion call. The process of mapping an address is fabric and provider specific, but may involve lengthy address resolution and fabric management protocols. AV operations are synchronous by default (the asynchrouous option has been deprecated, see below). See the NOTES section for AV restrictions on duplicate addresses. **Deprecated**: AV operations may be set to operate asynchronously by specifying the FI\_EVENT flag to fi\_av\_open. When requesting asynchronous operation, the application must first bind an event queue to the AV before inserting addresses. ### fi\_av\_open fi\_av\_open allocates or opens an address vector. The properties and behavior of the address vector are defined by struct fi\_av\_attr. - ``` struct fi_av_attr { enum fi_av_type type; /* type of AV */ int rx_ctx_bits; /* address bits to identify rx ctx */ size_t count; /* # entries for AV */ size_t ep_per_node; /* # endpoints per fabric address */ const char *name; /* system name of AV */ void *map_addr; /* base mmap address */ uint64_t flags; /* operation flags */ }; ``` - *type* An AV type corresponds to a conceptual implementation of an address vector. The type specifies how an application views data stored in the AV, including how it may be accessed. Valid values are: - \- *FI\_AV\_MAP* (deprecated) Addresses which are inserted into an AV are mapped to a native fabric address for use by the application. The use of FI\_AV\_MAP requires that an application store the returned fi\_addr\_t value that is associated with each inserted address. The advantage of using FI\_AV\_MAP is that the returned fi\_addr\_t value may contain encoded address data, which is immediately available when processing data transfer requests. This can eliminate or reduce the number of memory lookups needed when initiating a transfer. The disadvantage of FI\_AV\_MAP is the increase in memory usage needed to store the returned addresses. Addresses are stored in the AV using a provider specific mechanism, including, but not limited to a tree, hash table, or maintained on the heap. This option is deprecated, and providers are encouraged to align the behavior of FI\_AV\_MAP with FI\_AV\_TABLE. - \- *FI\_AV\_TABLE* Addresses which are inserted into an AV of type FI\_AV\_TABLE are accessible using a simple index. Conceptually, the AV may be treated as an array of addresses, though the provider may implement the AV using a variety of mechanisms. When FI\_AV\_TABLE is used, the returned fi\_addr\_t is an index, with the index for an inserted address the same as its insertion order into the table. The index of the first address inserted into an FI\_AV\_TABLE will be 0, and successive insertions will be given sequential indices. Sequential indices will be assigned across insertion calls on the same AV. Because the fi\_addr\_t values returned from an insertion call are deterministic, applications may not need to provide the fi\_addr\_t output parameters to insertion calls. The exception is when the fi\_addr\_t parameters are also used as input for supplying authentication keys or user defined IDs. - \- *FI\_AV\_UNSPEC* Provider will choose its preferred AV type. The AV type used will be returned through the type field in fi\_av\_attr. - *Receive Context Bits (rx\_ctx\_bits)* The receive context bits field is only for use with scalable endpoints. It indicates the number of bits reserved in a returned fi\_addr\_t, which will be used to identify a specific target receive context. See fi\_rx\_addr() and [fi\_endpoint(3)](/fi_endpoint/3.md) for additional details on receive contexts. The requested number of bits should be selected such that 2 ^ rx\_ctx\_bits \>= rx\_ctx\_cnt for the endpoint. - *count* Indicates the expected number of addresses that will be inserted into the AV. The provider uses this to optimize resource allocations. - *ep\_per\_node* This field indicates the number of endpoints that will be associated with a specific fabric, or network, address. If the number of endpoints per node is unknown, this value should be set to 0. The provider uses this value to optimize resource allocations. For example, distributed, parallel applications may set this to the number of processes allocated per node, times the number of endpoints each process will open. - *name* An optional system name associated with the address vector to create or open. Address vectors may be shared across multiple processes which access the same named domain on the same node. The name field allows the underlying provider to identify a shared AV. If the name field is non-NULL and the AV is not opened for read-only access, a named AV will be created, if it does not already exist. - *map\_addr* The map\_addr determines the base fi\_addr\_t address that a provider should use when sharing an AV of type FI\_AV\_MAP between processes. Processes that provide the same value for map\_addr to a shared AV may use the same fi\_addr\_t values returned from an fi\_av\_insert call. The map\_addr may be used by the provider to mmap memory allocated for a shared AV between processes; however, the provider is not required to use the map\_addr in this fashion. The only requirement is that an fi\_addr\_t returned as part of an fi\_av\_insert call on one process is usable on another process which opens an AV of the same name at the same map\_addr value. The relationship between the map\_addr and any returned fi\_addr\_t is not defined. If name is non-NULL and map\_addr is 0, then the map\_addr used by the provider will be returned through the attribute structure. The map\_addr field is ignored if name is NULL. - *flags* The following flags may be used when opening an AV. - \- *FI\_EVENT* (deprecated) When the flag FI\_EVENT is specified, all insert operations on this AV will occur asynchronously. There will be one EQ error entry generated for each failed address insertion, followed by one non-error event indicating that the insertion operation has completed. There will always be one non-error completion event for each insert operation, even if all addresses fail. The context field in all completions will be the context specified to the insert call, and the data field in the final completion entry will report the number of addresses successfully inserted. If an error occurs during the asynchronous insertion, an error completion entry is returned (see [fi\_eq(3)](/fi_eq/3.md) for a discussion of the fi\_eq\_err\_entry error completion struct). The context field of the error completion will be the context that was specified in the insert call; the data field will contain the index of the failed address. There will be one error completion returned for each address that fails to insert into the AV. If an AV is opened with FI\_EVENT, any insertions attempted before an EQ is bound to the AV will fail with -FI\_ENOEQ. Error completions for failed insertions will contain the index of the failed address in the index field of the error completion entry. Note that the order of delivery of insert completions may not match the order in which the calls to fi\_av\_insert were made. The only guarantee is that all error completions for a given call to fi\_av\_insert will precede the single associated non-error completion. • .RS 2 - *FI\_READ* Opens an AV for read-only access. An AV opened for read-only access must be named (name attribute specified), and the AV must exist. • .RS 2 - *FI\_SYMMETRIC* Indicates that each node will be associated with the same number of endpoints, the same transport addresses will be allocated on each node, and the transport addresses will be sequential. This feature targets distributed applications on large fabrics and allows for highly-optimized storage of remote endpoint addressing. • .RS 2 - *FI\_AV\_USER\_ID* Indicates that the user will be associating user-defined IDs with a address vector via fi\_av\_set\_user\_id. If the domain has been configured with FI\_AV\_AUTH\_KEY or the user requires FI\_AV\_USER\_ID support, using the FI\_AV\_USER\_ID flag per fi\_av\_insert / fi\_av\_insertsvc / fi\_av\_remove is not supported. fi\_av\_set\_user\_id must be used. ### fi\_close The fi\_close call is used to release all resources associated with an address vector. Note that any events queued on an event queue referencing the AV are left untouched. It is recommended that callers retrieve all events associated with the AV before closing it. When closing the address vector, there must be no opened endpoints associated with the AV. If resources are still associated with the AV when attempting to close, the call will return -FI\_EBUSY. ### fi\_av\_bind (deprecated) Associates an event queue with the AV. If an AV has been opened with FI\_EVENT, then an event queue must be bound to the AV before any insertion calls are attempted. Any calls to insert addresses before an event queue has been bound will fail with -FI\_ENOEQ. Flags are reserved for future use and must be 0. ### fi\_av\_insert The fi\_av\_insert call inserts zero or more addresses into an AV. The number of addresses is specified through the count parameter. The addr parameter references an array of addresses to insert into the AV. Addresses inserted into an address vector must be in the same format as specified in the addr\_format field of the fi\_info struct provided when opening the corresponding domain. When using the FI\_ADDR\_STR format, the addr parameter should reference an array of strings (char \*\*). **Deprecated**: For AV’s of type FI\_AV\_MAP, once inserted addresses have been mapped, the mapped values are written into the buffer referenced by fi\_addr. The fi\_addr buffer must remain valid until the AV insertion has completed and an event has been generated to an associated event queue. The value of the returned fi\_addr should be considered opaque by the application for AVs of type FI\_AV\_MAP. The returned value may point to an internal structure or a provider specific encoding of low-level addressing data, for example. In the latter case, use of FI\_AV\_MAP may be able to avoid memory references during data transfer operations. For AV’s of type FI\_AV\_TABLE, addresses are placed into the table in order. An address is inserted at the lowest index that corresponds to an unused table location, with indices starting at 0. That is, the first address inserted may be referenced at index 0, the second at index 1, and so forth. When addresses are inserted into an AV table, the assigned fi\_addr values will be simple indices corresponding to the entry into the table where the address was inserted. Index values accumulate across successive insert calls in the order the calls are made, not necessarily in the order the insertions complete. Because insertions occur at a pre-determined index, the fi\_addr parameter may be NULL. If fi\_addr is non-NULL, it must reference an array of fi\_addr\_t, and the buffer must remain valid until the insertion operation completes. Note that if fi\_addr is NULL and synchronous operation is requested without using FI\_SYNC\_ERR flag, individual insertion failures cannot be reported and the application must use other calls, such as fi\_av\_lookup to learn which specific addresses failed to insert. If the address vector is configured with authorization keys, the fi\_addr parameter may be used as input to define the authorization keys associated with the endpoint addresses being inserted. This is done by setting the fi\_addr to an authorization key fi\_addr\_t generated from fi\_av\_insert\_auth\_key and setting the FI\_AUTH\_KEY flag. If the FI\_AUTH\_KEY flag is not set, addresses being inserted will not be associated with any authorization keys. Whether or not an address can be disassociated with an authorization key is provider specific. If a provider cannot support this disassociation, an error will be returned. Upon successful insert with FI\_AUTH\_KEY flag, the returned fi\_addr\_t’s will map to endpoint address against the specified authorization keys. These fi\_addr\_t’s can be used as the target for local data transfer operations. If the endpoint supports FI\_DIRECTED\_RECV or FI\_TAGGED\_DIRECTED\_RECV, these fi\_addr\_t’s can be used to restrict receive buffers to a specific endpoint address and authorization key. For address vectors configured with FI\_AV\_USER\_ID, all subsequent target events corresponding to the address being inserted will return FI\_ADDR\_NOTAVAIL until the user defines a user ID for this fi\_addr\_t. This is done by using fi\_av\_set\_user\_id. - *flags* The following flag may be passed to AV insertion calls: fi\_av\_insert, fi\_av\_insertsvc, or fi\_av\_insertsym. - \- *FI\_MORE* In order to allow optimized address insertion, the application may specify the FI\_MORE flag to the insert call to give a hint to the provider that more insertion requests will follow, allowing the provider to aggregate insertion requests if desired. An application may make any number of insertion calls with FI\_MORE set, provided that they are followed by an insertion call without FI\_MORE. This signifies to the provider that the insertion list is complete. Providers are free to ignore FI\_MORE. - \- *FI\_SYNC\_ERR* This flag applies to synchronous insertions only, and is used to retrieve error details of failed insertions. If set, the context parameter of insertion calls references an array of integers, with context set to address of the first element of the array. The resulting status of attempting to insert each address will be written to the corresponding array location. Successful insertions will be updated to 0. Failures will contain a fabric errno code. - \- *FI\_AV\_USER\_ID* For address vectors configured without FI\_AV\_USER\_ID specified, this flag associates a user-assigned identifier with each AV entry that is returned with any completion entry in place of the AV’s address. If a provider does not support FI\_AV\_USER\_ID with insert, requesting this flag during insert will result runtime failure. Using the FI\_AV\_USER\_ID flag per insert is invalid if the AV was opened with the FI\_AV\_USER\_ID or if the corresponding domain was configured with FI\_AV\_AUTH\_KEY. With libfabric 1.20, users are encouraged to specify the FI\_AV\_USER\_ID when opening an AV and use fi\_av\_set\_user\_id. • .RS 2 - *FI\_AUTH\_KEY* Denotes that the address being inserted should be associated with the passed in authorization key fi\_addr\_t. See the user ID section below. • .RS 2 - *FI\_FIREWALL\_ADDR* This flag indicates that the address is behind a firewall and outgoing connections are not allowed. If there is not an existing connection and the provider is unable to circumvent the firewall, an FI\_EFIREWALLADDR error should be expected. If multiple addresses are being inserted simultaneously, the flag applies to all of them. Additionally, it is possible that a connection is available at insertion time, but is later torn down. Future reconnects triggered by operations on the ep (fi\_send, for example) may also fail with the same error. ### fi\_av\_insertsvc The fi\_av\_insertsvc call behaves similar to fi\_av\_insert, but allows the application to specify the node and service names, similar to the fi\_getinfo inputs, rather than an encoded address. The node and service parameters are defined the same as [fi\_getinfo(3)](/fi_getinfo/3.md). Node should be a string that corresponds to a hostname or network address. The service string corresponds to a textual representation of a transport address. Applications may also pass in an FI\_ADDR\_STR formatted address as the node parameter. In such cases, the service parameter must be NULL. See fi\_getinfo.3 for details on using FI\_ADDR\_STR. Supported flags are the same as for fi\_av\_insert. ### fi\_av\_insertsym fi\_av\_insertsym performs a symmetric insert that inserts a sequential range of nodes and/or service addresses into an AV. The svccnt parameter indicates the number of transport (endpoint) addresses to insert into the AV for each node address, with the service parameter specifying the starting transport address. Inserted transport addresses will be of the range {service, service + svccnt - 1}, inclusive. All service addresses for a node will be inserted before the next node is inserted. The nodecnt parameter indicates the number of node (network) addresses to insert into the AV, with the node parameter specifying the starting node address. Inserted node addresses will be of the range {node, node + nodecnt - 1}, inclusive. If node is a non-numeric string, such as a hostname, it must contain a numeric suffix if nodecnt \> 1. As an example, if node = “10.1.1.1”, nodecnt = 2, service = “5000”, and svccnt = 2, the following addresses will be inserted into the AV in the order shown: 10.1.1.1:5000, 10.1.1.1:5001, 10.1.1.2:5000, 10.1.1.2:5001. If node were replaced by the hostname “host10”, the addresses would be: host10:5000, host10:5001, host11:5000, host11:5001. The total number of inserted addresses will be nodecnt x svccnt. Supported flags are the same as for fi\_av\_insert. ### fi\_av\_remove fi\_av\_remove removes a set of addresses from an address vector. The corresponding fi\_addr\_t values are invalidated and may not be used in data transfer calls. The behavior of operations in progress that reference the removed addresses is undefined. Requesting removal of a fi\_addr that is unset or outside of the provider’s address vector bounds will result in -FI\_EINVAL being returned. Note that removing an address may not disable receiving data from the peer endpoint. fi\_av\_close will automatically cleanup any associated resource. If the address being removed came from fi\_av\_insert\_auth\_key, the address will only be removed if all endpoints, which have been enabled against the corresponding authorization key, have been closed. If all endpoints are not closed, -FI\_EBUSY will be returned. In addition, the FI\_AUTH\_KEY flag must be set when removing an authorization key fi\_addr\_t. - *flags* The following flags may be used when removing an AV entry. - \- *FI\_AUTH\_KEY* Denotes that the fi\_addr\_t being removed is an authorization key fi\_addr\_t. ### fi\_av\_lookup This call returns the address stored in the address vector that corresponds to the given fi\_addr. The returned address is the same format as those stored by the AV. Requesting lookup of a fi\_addr that is unset or is outside of the provider’s address vector bounds will result in -FI\_EINVAL being returned. On input, the addrlen parameter should indicate the size of the addr buffer. If the actual address is larger than what can fit into the buffer, it will be truncated. On output, addrlen is set to the size of the buffer needed to store the address, which may be larger than the input value. ### fi\_av\_lookup2 The fi\_av\_lookup2 call is an extended version of fi\_av\_lookup that accepts flags and an optional XPU context. When called with flags set to FI\_XPU and a valid fid\_xpu\_ctx, the call retrieves a provider-specific raw address usable by the XPU when posting work requests. The size of the returned data is av\_addr\_size from fi\_xpu\_ctx\_query. When called without FI\_XPU (flags = 0, ctx = NULL), the call behaves identically to fi\_av\_lookup. The caller allocates buf and sets *len to the buffer size on input;* *on output* len is set to the number of bytes written. The AV is not bound to an XPU context — it remains a host-only, domain-level resource. The same AV can be queried with different XPU contexts to get device-specific representations. See [fi\_xpu(3)](/fi_xpu/3.md) for details. ### fi\_rx\_addr This function is used to convert an endpoint address, returned by fi\_av\_insert, into an address that specifies a target receive context. The value for rx\_ctx\_bits must match that specified in the AV attributes for the given address. Connected endpoints that support multiple receive contexts, but are not associated with address vectors should specify FI\_ADDR\_NOTAVAIL for the fi\_addr parameter. ### fi\_av\_straddr The fi\_av\_straddr function converts the provided address into a printable string. The specified address must be of the same format as those stored by the AV, though the address itself is not required to have been inserted. On input, the len parameter should specify the size of the buffer referenced by buf. On output, addrlen is set to the size of the buffer needed to store the address. This size may be larger than the input len. If the provided buffer is too small, the results will be truncated. fi\_av\_straddr returns a pointer to buf. ### fi\_av\_insert\_auth\_key This function associates authorization keys with an address vector. This requires the domain to be opened with FI\_AV\_AUTH\_KEY. FI\_AV\_AUTH\_KEY enables endpoints and memory regions to be associated with authorization keys from the address vector. This behavior enables a single endpoint or memory region to be associated with multiple authorization keys. When endpoints or memory regions are enabled, they are configured with address vector authorization keys at that point in time. Later authorization key insertions will not propagate to already enabled endpoints and memory regions. The auth\_key and auth\_key\_size parameters are used to input the authorization key into the address vector. The structure of the authorization key is provider specific. If the auth\_key\_size does not align with provider specific structure, -FI\_EINVAL will be returned. The output of fi\_av\_insert\_auth\_key is an authorization key fi\_addr\_t handle representing all endpoint addresses against this specific authorization key. For all operations, including address vector, memory registration, and data transfers, which may accept an authorization key fi\_addr\_t as input, the FI\_AUTH\_KEY flag must be specified. Otherwise, the fi\_addr\_t will be treated as an fi\_addr\_t returned from the fi\_av\_insert and related functions. For endpoints enabled with FI\_DIRECTED\_RECV, authorization key fi\_addr\_t’s can be used to restrict incoming messages to only endpoint addresses within the authorization key. This will require passing in the FI\_AUTH\_KEY flag to fi\_recvmsg and fi\_trecvmsg. For domains enabled with FI\_DIRECTED\_RECV, authorization key fi\_addr\_t’s can be used to restrict memory region access to only endpoint addresses within the authorization key. This will require passing in the FI\_AUTH\_KEY flag to fi\_mr\_regattr. These authorization key fi\_addr\_t’s can later be used an input for endpoint address insertion functions to generate an fi\_addr\_t for a specific endpoint address and authorization key. This will require passing in the FI\_AUTH\_KEY flag to fi\_av\_insert and related functions. For address vectors configured with FI\_AV\_USER\_ID and endpoints with FI\_SOURCE\_ERR, all subsequent FI\_EADDRNOTAVAIL error events will return FI\_ADDR\_NOTAVAIL until the user defines a user ID for this authorization key fi\_addr\_t. This is done by using fi\_av\_set\_user\_id. For address vectors configured without FI\_AV\_USER\_ID and endpoints with FI\_SOURCE\_ERR, all subsequent FI\_EADDRNOTAVAIL error events will return the authorization key fi\_addr\_t handle. Flags are reserved for future use and must be 0. ### fi\_av\_lookup\_auth\_key This functions returns the authorization key associated with a fi\_addr\_t. Acceptable fi\_addr\_t’s input are the output of fi\_av\_insert\_auth\_key and AV address insertion functions. The returned authorization key is in a provider specific format. On input, the auth\_key\_size parameter should indicate the size of the auth\_key buffer. If the actual authorization key is larger than what can fit into the buffer, it will be truncated. On output, auth\_key\_size is set to the size of the buffer needed to store the authorization key, which may be larger than the input value. ### fi\_av\_set\_user\_id If the address vector has been opened with FI\_AV\_USER\_ID, this function defines the user ID for a specific fi\_addr\_t. By default, all fi\_addr\_t’s will be assigned the user ID FI\_ADDR\_NOTAVAIL. - *flags* The following flag may be passed to AV set user id. - \- *FI\_AUTH\_KEY* Denotes that the fi\_addr fi\_addr\_t, for which the user ID is being set for, is an authorization key fi\_addr\_t. ## NOTES An AV should only store a single instance of an address. Attempting to insert a duplicate copy of the same address into an AV may result in undefined behavior, depending on the provider implementation. Providers are not required to check for duplicates, as doing so could incur significant overhead to the insertion process. For portability, applications may need to track which peer addresses have been inserted into a given AV in order to avoid duplicate entries. However, providers are required to support the removal, followed by the re-insertion of an address. Only duplicate insertions are restricted. ## USER IDENTIFIERS FOR ADDRESSES As described above, endpoint addresses authorization keys that are inserted into an AV are mapped to an fi\_addr\_t value. The endpoint address fi\_addr\_t is used in data transfer APIs to specify the destination of an outbound transfer, in receive APIs to indicate the source for an inbound transfer, and also in completion events to report the source address of inbound transfers. The authorization key fi\_addr\_t are used in receive and MR APIs to resource incoming operations to a specific authorization key, and also in completion error events if the endpoint is configured with FI\_SOURCE\_ERR. The FI\_AV\_USER\_ID capability bit and flag provide a mechanism by which the fi\_addr\_t value reported by a completion success or error event is replaced with a user-specified value instead. This is useful for applications that need to map the source address to their own data structure. Support for FI\_AV\_USER\_ID is provider specific, as it may not be feasible for a provider to implement this support without significant overhead. For example, some providers may need to add a reverse lookup mechanism. This feature may be unavailable if shared AVs are requested, or negatively impact the per process memory footprint if implemented. For providers that do not support FI\_AV\_USER\_ID, users may be able to trade off lookup processing with protocol overhead, by carrying source identification within a message header. For address vectors opened without FI\_AV\_USER\_ID, user-specified fi\_addr\_t values are provided as part of address insertion (e.g. fi\_av\_insert) through the fi\_addr parameter. The fi\_addr parameter acts as input/output in this case. When the FI\_AV\_USER\_ID flag is passed to any of the insert calls, the caller must specify an fi\_addr\_t identifier value to associate with each address. The provider will record that identifier and use it where required as part of any completion event. Note that the output from the AV insertion call is unchanged. The provider will return an fi\_addr\_t value that maps to each address, and that value must be used for all data transfer operations. For address vectors opened with FI\_AV\_USER\_ID, fi\_av\_set\_user\_id is used to defined the user-specified fi\_addr\_t. ## PEER GROUPS Peer groups provide a direct mapping to HPC and AI communicator constructs. The addresses in an AV represent the full set of peers that a local process may communicate with. A peer group conceptually represents a subset of those peers. A peer group may be used to identify peers working on a common task, which need their communication logically separated from other traffic. Peer groups are not a security mechanism, but instead help separate data. A given peer may belong to 0 or more peer groups, with no limit placed on how many peers can belong to a single peer group. Peer groups are identified using an integer value, known as a group id. Group id’s are selected by the user and conveyed as part of an fi\_addr\_t value. The management of a group id and it’s relationship to addresses inserted into an AV is directly controlled by the user. When enabled, sent messages are marked as belonging to a specific peer group, and posted receive buffers must have a matching group id to receive the data. Users are responsible for selecting a valid peer group id, subject to the limitation negotiated using the domain attribute max\_group\_id. The group id of an fi\_addr\_t may be set using the fi\_group\_addr() function. ### fi\_group\_addr This function is used to set the group ID portion of an fi\_addr\_t. ## RETURN VALUES Insertion calls, excluding fi\_av\_insert\_auth\_key, for an AV opened for synchronous operation will return the number of addresses that were successfully inserted. In the case of failure, the return value will be less than the number of addresses that was specified. **Deprecated**: Insertion calls, excluding fi\_av\_insert\_auth\_key, for an AV opened for asynchronous operation (with FI\_EVENT flag specified) will return FI\_SUCCESS if the operation was successfully initiated. In the case of failure, a negative fabric errno will be returned. Providers are allowed to abort insertion operations in the case of an error. Addresses that are not inserted because they were aborted will fail with an error code of FI\_ECANCELED. In both the synchronous and asynchronous modes of operation, the fi\_addr buffer associated with a failed or aborted insertion will be set to FI\_ADDR\_NOTAVAIL. All other calls return FI\_SUCCESS on success, or a negative value corresponding to fabric errno on error. Fabric errno values are defined in rdma/fi\_errno.h. fi\_av\_lookup2 returns -FI\_ETOOSMALL if the provided buffer is too small; \*len is updated to the required size. Returns -FI\_EINVAL if an invalid fi\_addr or XPU context is provided. ## SEE ALSO [fi\_getinfo(3)](/fi_getinfo/3.md), [fi\_endpoint(3)](/fi_endpoint/3.md), [fi\_domain(3)](/fi_domain/3.md), [fi\_eq(3)](/fi_eq/3.md) ## AUTHORS OpenFabrics.