---
name: "memkind"
section: "3"
platform: "linux"
package: "libmemkind-dev"
version: "1.14.0-2"
release: "Debian 12 (Bookworm)"
platforms: ["linux"]
url: "https://new.carta.tech/memkind"
---

# memkind(3)

Heap manager that enables allocations to memory with different properties. This header expose STANDARD and EXPERIMENTAL API. API Standards are described below in this man page.

## NAME

memkind - Heap manager that enables allocations to memory with different properties.\
 This header expose STANDARD and EXPERIMENTAL API. API Standards are described below in this man page.

## SYNOPSIS

```
#include <memkind.h>

Link with -lmemkind
```

### EXPERIMENTAL API:

**HEAP MANAGEMENT:**\
 **int memkind\_posix\_memalign(memkind\_t** *kind***, void** *\*\*memptr***, size\_t** *alignment***, size\_t** *size***);**

**KIND MANAGEMENT:**\
 **int memkind\_create\_kind(memkind\_memtype\_t** *memtype\_flags***, memkind\_policy\_t** *policy***, memkind\_bits\_t** *flags***, memkind\_t** *\*kind***);**

### STANDARD API:

**ERROR HANDLING:**\
 **void memkind\_error\_message(int** *err***, char** *\*msg***, size\_t** *size***);**

**LIBRARY VERSION:**\
 **int memkind\_get\_version();**

**HEAP MANAGEMENT:**\
 **void \*memkind\_malloc(memkind\_t** *kind***, size\_t** *size***);**\
 **void \*memkind\_calloc(memkind\_t** *kind***, size\_t** *num***, size\_t** *size***);**\
 **void \*memkind\_realloc(memkind\_t** *kind***, void** *\*ptr***, size\_t** *size***);**\
 **void memkind\_free(memkind\_t** *kind***, void** *\*ptr***);**\
 **size\_t memkind\_malloc\_usable\_size(memkind\_t** *kind***, void** *\*ptr***);**\
 **void \*memkind\_defrag\_reallocate(memkind\_t** *kind***, void** *\*ptr***);**\
 **memkind\_t memkind\_detect\_kind(void** *\*ptr***);**

**KIND CONFIGURATION MANAGEMENT:**\
 **struct memkind\_config \*memkind\_config\_new();**\
 **void memkind\_config\_delete(struct memkind\_config** *\*cfg***);**\
 **void memkind\_config\_set\_path(struct memkind\_config** *\*cfg***, const char** *\*pmem\_dir***);**\
 **void memkind\_config\_set\_size(struct memkind\_config** *\*cfg***, size\_t** *pmem\_size***);**\
 **void memkind\_config\_set\_memory\_usage\_policy(struct memkind\_config** *\*cfg***, memkind\_mem\_usage\_policy** *policy***);**

**KIND MANAGEMENT:**\
 **int memkind\_create\_fixed(void** *\*addr***, size\_t** *size***, memkind\_t** *\*kind***);**\
 **int memkind\_create\_pmem(const char** *\*dir***, size\_t** *max\_size***, memkind\_t** *\*kind***);**\
 **int memkind\_create\_pmem\_with\_config(struct memkind\_config** *\*cfg***, memkind\_t** *\*kind***);**\
 **int memkind\_destroy\_kind(memkind\_t** *kind***);**\
 **int memkind\_check\_available(memkind\_t** *kind***);**\
 **ssize\_t memkind\_get\_capacity(memkind\_t** *kind***);**\
 **int memkind\_check\_dax\_path(const char** *\*pmem\_dir***);**

**STATISTICS:**\
 **int memkind\_update\_cached\_stats(void);**\
 **int memkind\_get\_stat(memkind\_t** *kind***, memkind\_stat** *stat***, size\_t** *\*value***);**\
 **int memkind\_stats\_print(void** *(\*write\_cb) (void \*, const char \*)***, void** *\*cbopaque***, memkind\_stat\_print\_opt** *opts***);**

**DECORATORS:**\
 **void memkind\_malloc\_pre(memkind\_t** *\*kind***, size\_t** *\*size***);**\
 **void memkind\_malloc\_post(memkind\_t** *kind***, size\_t** *size***, void** *\*\*result***);**\
 **void memkind\_calloc\_pre(memkind\_t** *\*kind***, size\_t** *\*nmemb***, size\_t** *\*size***);**\
 **void memkind\_calloc\_post(memkind\_t** *kind***, size\_t** *nmemb***, size\_t** *size***, void** *\*\*result***);**\
 **void memkind\_posix\_memalign\_pre(memkind\_t** *\*kind***, void** *\*\*memptr***, size\_t** *\*alignment***, size\_t** *\*size***);**\
 **void memkind\_posix\_memalign\_post(memkind\_t** *kind***, void** *\*\*memptr***, size\_t** *alignment***, size\_t** *size***, int** *\*err***);**\
 **void memkind\_realloc\_pre(memkind\_t** *\*kind***, void** *\*\*ptr***, size\_t** *\*size***);**\
 **void memkind\_realloc\_post(memkind\_t** *\*kind***, void** *\*ptr***, size\_t** *size***, void** *\*\*result***);**\
 **void memkind\_free\_pre(memkind\_t** *\*kind***, void** *\*\*ptr***);**\
 **void memkind\_free\_post(memkind\_t** *kind***, void** *\*ptr***);**

**ENHANCEMENTS:**\
 **int memkind\_set\_bg\_threads(bool** *state***);**

## DESCRIPTION

**memkind\_error\_message**() converts an error number *err* returned by a member of the memkind interface to an error message *msg* where the maximum size of the message is passed by the *size* parameter.

**HEAP MANAGEMENT:**\
 The functions described in this section define a heap manager with an interface modeled on the ISO C standard API's, except that the user must specify the *kind* of memory with the first argument to each function. See the **KINDS** section below for a full description of the implemented kinds. For file-backed kind of memory see **memkind\_create\_pmem**() or **memkind\_create\_pmem\_with\_config**(). For memory kind created on user-specified area, please check **memkind\_create\_fixed**().

**memkind\_malloc**() allocates *size* bytes of uninitialized memory of the specified *kind*. The allocated space is suitably aligned (after possible pointer coercion) for storage of any type of object. If *size* is 0, then **memkind\_malloc**() returns *NULL*.

**memkind\_calloc**() allocates space for *num* objects each *size* bytes in length in memory of the specified *kind*. The result is identical to calling **memkind\_malloc**() with an argument of *num* \* *size*, with the exception that the allocated memory is explicitly initialized to zero bytes. If *num* or *size* is 0, then **memkind\_calloc**() returns *NULL*.

**memkind\_realloc**() changes the size of the previously allocated memory referenced by *ptr* to *size* bytes of the specified *kind*. The contents of the memory remain unchanged up to the lesser of the new and old sizes. If the new size is larger, the contents of the newly allocated portion of the memory are undefined. Upon success, the memory referenced by *ptr* is freed and a pointer to the newly allocated memory is returned.

**Note:** **memkind\_realloc**() may move the memory allocation, resulting in a different return value than *ptr*.

If *ptr* is *NULL*, the **memkind\_realloc**() function behaves identically to **memkind\_malloc**() for the specified size. If *size* is equal to zero, and *ptr* is not *NULL*, then the call is equivalent to *memkind\_free(kind, ptr)* and *NULL* is returned. The address *ptr*, if not *NULL*, must have been returned by a previous call to **memkind\_malloc**(), **memkind\_calloc**(), **memkind\_realloc**(), **memkind\_defrag\_reallocate**() or **memkind\_posix\_memalign**() with the same *kind* as specified to the call to **memkind\_realloc**(). Otherwise, if *memkind\_free(kind, ptr)* was called before, undefined behavior occurs. In cases where the kind is unknown in the context of the call to **memkind\_realloc**() *NULL* can be given as the *kind* specified to **memkind\_realloc**(), but this will require an internal look up for correct kind. **Note:** The look up for *kind* could result in serious performance penalty, which can be avoided by specifying a correct *kind*. If *kind* is *NULL* and *ptr* is *NULL*, then **memkind\_realloc**() returns *NULL* and sets *errno* to **EINVAL**.

**memkind\_posix\_memalign**() allocates *size* bytes of memory of a specified *kind* such that the allocation's base address is an even multiple of *alignment*, and returns the allocation in the value pointed to by *memptr*. The requested *alignment* must be a power of 2 at least as large as *sizeof(void\*)*. If *size* is 0, then **memkind\_posix\_memalign**() returns 0, with a *NULL* returned in *memptr*.

**memkind\_malloc\_usable\_size**() function provides the same semantics as [**malloc\_usable\_size**(3)](/malloc_usable_size/3.md), but operates on specified *kind*.\
 **Note:** In cases where the kind is unknown in the context of the call to **memkind\_malloc\_usable\_size**() *NULL* can be given as the *kind* specified to **memkind\_malloc\_usable\_size**(), but this could require a internal look up for correct kind. **memkind\_malloc\_usable\_size**() is supported by TBB heap manager described in **ENVIRONMENT** section since Intel TBB 2019 Update 4.

**memkind\_defrag\_reallocate**() reallocates the object conditionally inside specific *kind*. Function determines if it's worthwhile to move allocation to reduce degree of external fragmentation of the heap. In case of failure function returns *NULL*, otherwise function returns a pointer to reallocated memory and memory referenced by *ptr* was released and should not be accessed. If *ptr* is *NULL*, then **memkind\_defrag\_reallocate**() returns *NULL*. In cases where the kind is unknown in the context of the call to **memkind\_defrag\_reallocate**() *NULL* can be given as the *kind* specified to **memkind\_defrag\_reallocate**(), but this will require an internal look up for correct kind. **Note:** The look up for *kind* could result in serious performance penalty, which can be avoided by specifying a correct *kind*.

**memkind\_detect\_kind**() returns the kind associated with allocated memory referenced by *ptr*. This pointer must have been returned by a previous call to **memkind\_malloc**(), **memkind\_calloc**(), **memkind\_realloc**(), **memkind\_defrag\_reallocate**() or **memkind\_posix\_memalign**(). If *ptr* is *NULL*, then **memkind\_detect\_kind**() returns *NULL*. **Note:** This function has non-trivial performance overhead.

**memkind\_free**() causes the allocated memory referenced by *ptr* to be made available for future allocations. This pointer must have been returned by a previous call to **memkind\_malloc**(), **memkind\_calloc**(), **memkind\_realloc**(), **memkind\_defrag\_reallocate**() or **memkind\_posix\_memalign**(). Otherwise, if *memkind\_free(kind, ptr)* was already called before, undefined behavior occurs. If *ptr* is *NULL*, no operation is performed. In cases where the kind is unknown in the context of the call to **memkind\_free**() *NULL* can be given as the *kind* specified to **memkind\_free**(), but this will require an internal look up for correct kind. **Note:** The look up for *kind* could result in serious performance penalty, which can be avoided by specifying a correct *kind*.

**KIND CONFIGURATION MANAGEMENT:**\
 The functions described in this section define a way to create, delete and update kind specific configuration. Except of **memkind\_config\_new**(), user must specify the memkind configuration with the first argument to each function. API described here is most useful with file-backed kind of memory, e.g. **memkind\_create\_pmem\_with\_config**() method.

**memkind\_config\_new**() creates the memkind configuration.

**memkind\_config\_delete**() deletes previously created memkind configuration, which must have been returned by a previous call to **memkind\_config\_new**().

**memkind\_config\_set\_path**() updates the memkind *pmem\_dir* configuration parameter, which specifies directory path, where file-backed kind of memory will be created. **Note:** This function does not validate that *pmem\_dir* specifies a valid path.

**memkind\_config\_set\_size**() updates the memkind *pmem\_size* configuration parameter, which allows to limit the file-backed kind memory partition. **Note:** This function does not validate that *pmem\_size* is in valid range.

**memkind\_config\_set\_memory\_usage\_policy**() updates the memkind *policy* configuration parameter, which allows to tune up memory utilization. The user should set the value based on the characteristics of application that is using the library (e.g. prioritize memory usage, CPU utilization), for more details about *policy* see the **MEMORY USAGE POLICY** section below. **Note:** This function does not validate that *policy* is in valid range.

**KIND MANAGEMENT:**\
 There are built-in kinds that are always available and these are enumerated in the **KINDS** section. The user can also create their own kinds of memory. This section describes the API's that enable the tracking of the different kinds of memory and determining their properties.

**memkind\_create\_fixed**() is a function used to create a kind under user-specified area of memory. The memory can be allocated in any possible way, e.g. it might be a static array or an mmapped area. User can specify any properties using functions such as mbind. User is also responsible for de-allocation of memory after the kind destruction. The memory area must remain valid until fixed\_kind is destroyed. The area starts at address *addr* and has size *size.* When heap manager runs out of memory (located under user-specified area), a call to **memkind\_malloc**() returns *NULL* and **errno** is set to **ENOMEM.**

**memkind\_create\_pmem**() is a convenient function used to create a file-backed kind of memory. It allocates a temporary file in the given directory *dir*. The file is created in a fashion similar to [**tmpfile**(3)](/tmpfile/3.md), so that the file name does not appear when the directory is listed and the space is automatically freed when the program terminates. The file is truncated to a size of *max\_size* bytes and the resulting space is memory-mapped.\
 Note that the actual file system space is not allocated immediately, but only on a call to **memkind\_pmem\_mmap**() (see [**memkind\_pmem**(3)](/memkind_pmem/3.md)). This allows to create a pmem memkind of a pretty large size without the need to reserve in advance the corresponding file system space for the entire heap. If the value of *max\_size* equals 0, pmem memkind is only limited by the capacity of the file system mounted under *dir* argument. The minimum *max\_size* value which allows to limit the size of kind by the library is defined as **MEMKIND\_PMEM\_MIN\_SIZE**. Calling **memkind\_create\_pmem**() with a size smaller than that and different than 0 will return an error. The maximum allowed size is not limited by **memkind**, but by the file system specified by the *dir* argument. The *max\_size* passed in is the raw size of the memory pool and **jemalloc** will use some of that space for its own metadata. Returns zero if the pmem memkind is created successfully or an error code from the **ERRORS** section if not.

**memkind\_create\_pmem\_with\_config**() is a second function used to create a file-backed kind of memory. Function behaves simillar to **memkind\_create\_pmem**() but instead of passing *dir* and *max\_size* arguments, it uses *config* param to specify characteristics of created file-backed kind of memory (see **KIND CONFIGURATION MANAGEMENT** section).

**memkind\_create\_kind**() creates kind that allocates memory with specific memory type, memory binding policy and flags (see **MEMORY FLAGS** section). The *memtype\_flags* (see **MEMORY TYPES** section) determine memory types to allocate, *policy* argument is policy for specifying page binding to memory types selected by *memtype\_flags*. Returns zero if the specified kind is created successfully or an error code from the **ERRORS** section if not.

**memkind\_destroy\_kind**() destroys previously created kind object, which must have been returned by a previous call to **memkind\_create\_pmem**(), **memkind\_create\_pmem\_with\_config**() or **memkind\_create\_kind**(). Otherwise, or if *memkind\_destroy\_kind(kind)* was already called before, undefined behavior occurs. Note that, when the kind was returned by **memkind\_create\_kind**() all allocated memory must be freed before kind is destroyed, otherwise this will cause memory leak. When the kind was returned by **memkind\_create\_pmem**() or **memkind\_create\_pmem\_with\_config**() all allocated memory will be freed after kind will be destroyed.

**memkind\_check\_available**() returns zero if the specified *kind* is available or an error code from the **ERRORS** section if it is not.

**memkind\_get\_capacity**() returns memory capacity of nodes available to a given *kind* (file size or filesystem capacity in case of a file-backed PMEM kind; total area size in the case of fixed-kind) or -1 in case of an error. Supported kinds are: **MEMKIND\_DEFAULT, MEMKIND\_HIGHEST\_CAPACITY, MEMKIND\_HIGHEST\_CAPACITY\_LOCAL, MEMKIND\_LOWEST\_LATENCY\_LOCAL,** **MEMKIND\_HIGHEST\_BANDWIDTH\_LOCAL, MEMKIND\_HUGETLB, MEMKIND\_INTERLEAVE, MEMKIND\_HBW,** **MEMKIND\_HBW\_ALL, MEMKIND\_HBW\_INTERLEAVE, MEMKIND\_DAX\_KMEM, MEMKIND\_DAX\_KMEM\_ALL,** **MEMKIND\_DAX\_KMEM\_INTERLEAVE, MEMKIND\_REGULAR** , file-backed PMEM and fixed-kind. *kind.* For **MEMKIND\_HUGETLB** only pages with a default size of 2MB are supported.

**memkind\_check\_dax\_path**() returns zero if file-backed kind memory is in the specified directory path *pmem\_dir* can be created with the DAX attribute or an error code from the **ERRORS** section if it is not.

**MEMKIND\_PMEM\_MIN\_SIZE** The minimum size which allows to limit the file-backed memory partition.

**STATISTICS:**\
 The functions described in this section define a way to get specific memory allocation statistics.

**memkind\_update\_cached\_stats**() is used to force an update of cached dynamic allocator statistics. Statistics are not updated real-time by memkind library and this method allows to force its update.

**memkind\_get\_stat**() retrieves statistic of the specified type and returns it in *value*. For more details about *stat* see the **MEMORY STATISTICS TYPE** section below. Measured statistic applies to specific *kind*, when *NULL* is given as *kind* then statistic applies to memory used by the whole memkind library. **Note:** You need to call **memkind\_update\_cached\_stats**() before calling **memkind\_get\_stat**() because statistics are cached by memkind library.

**memkind\_stats\_print**() prints summary statistics. This function wraps jemalloc's function **je\_malloc\_stats\_print**(). Uses *write\_cb* function to print the output. While providing custom writer function, use **[syscall(2)](/syscall/2.md)** rather than **[write(2)](/write/2.md).** Pass *NULL* in order to use the default *write\_cb* function which prints the output to the stderr. Use *cbopaque* parameter in order to pass some data to your *write\_cb* function. Pass additional options using *opts*. For more details on *opts* see the **MEMORY STATISTICS PRINT OPTIONS** section below. Returns MEMKIND\_ERROR\_INVALID when failed to parse options string, MEMKIND\_SUCCESS on success.

**DECORATORS:**\
 The memkind library enables the user to define decorator functions that can be called before and after each memkind heap management API. The decorators that are called at the beginning of the function end are named after that function with *\_pre* appended to the name and those that are called at the end of the function are named after that function with *\_post* appended to the name. These are weak symbols and if they are not present at link time they are not called. The memkind library does not define these symbols which are reserved for user definition. These decorators can be used to track calls to the heap management interface or to modify parameters. The decorators that are called at the beginning of the allocator pass all inputs by reference and the decorators that are called at the end of the allocator pass the output by reference. This enables the modification of the input and output of each heap management function by the decorators.

**ENHANCEMENTS:**\
 **memkind\_set\_bg\_threads()** enables/disables internal background worker threads in jemalloc.

**LIBRARY VERSION:**\
 The memkind library version scheme consist major, minor and patch numbers separated by dot. Combining those numbers, we got the following representation:\
 major.minor.patch, where:\
\
 	\-major number is incremented whenever API is changed (loss of backward compatibility),\
\
 	\-minor number is incremented whenever additional extensions are introduced or behavior has been changed,\
\
 	\-patch number is incremented whenever small bug fixes are added.

memkind library provide numeric representation of the version by exposing the following API:

**memkind\_get\_version**() returns version number represented by a single integer number, obtained from the formula:\
 major \* 1000000 + minor \* 1000 + patch

**Note:** major \< 1 means unstable API.

API standards:\
 \-STANDARD API, API is considered as stable\
 \-NON-STANDARD API, API is considered as stable, however this is not a standard way to use memkind\
 \-EXPERIMENTAL API, API is considered as unstable and the subject to change\

## RETURN VALUE

**memkind\_calloc**(), **memkind\_malloc**(), **memkind\_realloc**() and **memkind\_defrag\_reallocate**() returns the pointer to the allocated memory or *NULL* if the request fails. **memkind\_malloc\_usable\_size**() returns the number of usable bytes in the block of allocated memory pointed to by *ptr*, a pointer to a block of memory allocated by **memkind\_malloc**() or a related function. If *ptr* is *NULL*, 0 is returned. **memkind\_free**() and **memkind\_error\_message**() do not have return values. All other memkind API's return 0 upon success and an error code defined in the **ERRORS** section upon failure. The memkind library avoids setting *errno* directly, but calls to underlying libraries and system calls may set *errno* (e.g. **memkind\_create\_pmem**()).

## KINDS

The available kinds of memory:

- **MEMKIND\_DEFAULT**

  Default allocation using standard memory and default page size.

- **MEMKIND\_HIGHEST\_CAPACITY**

  Allocate from a NUMA node(s) that has the highest capacity among all nodes in the system.

- **MEMKIND\_HIGHEST\_CAPACITY\_PREFERRED**

  Same as **MEMKIND\_HIGHEST\_CAPACITY** except that if there is not enough memory in the NUMA node that has the highest capacity in the local domain to satisfy the request, the allocation will fall back on other memory NUMA nodes. **Note:** For this kind, the allocation will not succeed if there are two or more NUMA nodes that have the highest capacity.

- **MEMKIND\_HIGHEST\_CAPACITY\_LOCAL**

  Allocate from a NUMA node that has the highest capacity among all NUMA Nodes from the local domain. NUMA Nodes have the same local domain for a set of CPUs associated with them, e.g. socket or sub-NUMA cluster. **Note:** If there are multiple NUMA nodes in the same local domain that have the highest capacity - allocation will be done from NUMA node with worse latency attribute. This kind requires locality information described in **SYSTEM CONFIGURATION** section.

- **MEMKIND\_HIGHEST\_CAPACITY\_LOCAL\_PREFERRED**

  Same as **MEMKIND\_HIGHEST\_CAPACITY\_LOCAL** except that if there is not enough memory in the NUMA node that has the highest capacity to satisfy the request, the allocation will fall back on other memory NUMA nodes.

- **MEMKIND\_LOWEST\_LATENCY\_LOCAL**

  Allocate from a NUMA node that has the lowest latency among all NUMA Nodes from the local domain. NUMA Nodes have the same local domain for a set of CPUs associated with them, e.g. socket or sub-NUMA cluster. **Note:** If there are multiple NUMA nodes in the same local domain that have the lowest latency - allocation will be done from NUMA node with smaller memory capacity. This kind requires locality and memory performance characteristics information described in **SYSTEM CONFIGURATION** section.

- **MEMKIND\_LOWEST\_LATENCY\_LOCAL\_PREFERRED**

  Same as **MEMKIND\_LOWEST\_LATENCY\_LOCAL** except that if there is not enough memory in the NUMA node that has the lowest latency to satisfy the request, the allocation will fall back on other memory NUMA nodes.

- **MEMKIND\_HIGHEST\_BANDWIDTH\_LOCAL**

  Allocate from a NUMA node that has the highest bandwidth among all NUMA Nodes from the local domain. NUMA Nodes have the same local domain for a set of CPUs associated with them, e.g. socket or sub-NUMA cluster. **Note:** If there are multiple NUMA nodes in the same local domain that have the highest bandwidth - allocation will be done from NUMA node with smaller memory capacity. This kind requires locality and memory performance characteristics information described in **SYSTEM CONFIGURATION** section.

- **MEMKIND\_HIGHEST\_BANDWIDTH\_LOCAL\_PREFERRED**

  Same as **MEMKIND\_HIGHEST\_BANDWIDTH\_LOCAL** except that if there is not enough memory in the NUMA node that has the highest bandwidth to satisfy the request, the allocation will fall back on other memory NUMA nodes.

- **MEMKIND\_HUGETLB**

  Allocate from standard memory using huge pages. **Note:** This kind requires huge pages configuration described in **SYSTEM CONFIGURATION** section.

- **MEMKIND\_GBTLB (DEPRECATED)**

  Allocate from standard memory using 1GB chunks backed by huge pages. **Note:** This kind requires huge pages configuration described in **SYSTEM CONFIGURATION** section.

- **MEMKIND\_INTERLEAVE**

  Allocate pages interleaved across all NUMA nodes with transparent huge pages disabled.

- **MEMKIND\_HBW**

  Allocate from the closest high bandwidth memory NUMA node(s) at the time of allocation. If there is not enough high bandwidth memory to satisfy the request *errno* is set to **ENOMEM** and the allocated pointer is set to *NULL*. **Note:** This kind requires memory performance characteristics information described in **SYSTEM CONFIGURATION** section.

- **MEMKIND\_HBW\_ALL**

  Same as **MEMKIND\_HBW** except decision regarding closest NUMA node(s) is postponed until the time of first write.

- **MEMKIND\_HBW\_HUGETLB**

  Same as **MEMKIND\_HBW** except the allocation is backed by huge pages. **Note:** This kind requires huge pages configuration described in **SYSTEM CONFIGURATION** section.

- **MEMKIND\_HBW\_ALL\_HUGETLB**

  Combination of **MEMKIND\_HBW\_ALL** and **MEMKIND\_HBW\_HUGETLB** properties. **Note:** This kind requires huge pages configuration described in **SYSTEM CONFIGURATION** section.

- **MEMKIND\_HBW\_PREFERRED**

  Same as **MEMKIND\_HBW** except that if there is not enough high bandwidth memory to satisfy the request, the allocation will fall back on standard memory. **Note:** For this kind, the allocation will not succeed if two or more high bandwidth memory NUMA nodes are in the same shortest distance to the same CPU on which process is eligible to run. Check on that eligibility is done upon starting the application.

- **MEMKIND\_HBW\_PREFERRED\_HUGETLB**

  Same as **MEMKIND\_HBW\_PREFERRED** except the allocation is backed by huge pages. **Note:** This kind requires huge pages configuration described in **SYSTEM CONFIGURATION** section.

- **MEMKIND\_HBW\_GBTLB (DEPRECATED)**

  Same as **MEMKIND\_HBW** except the allocation is backed by 1GB chunks of huge pages. Note that *size* can take on any value, but full gigabyte pages will allocated for each request, so remainder of the last page will be wasted. This kind requires huge pages configuration described in **SYSTEM CONFIGURATION** section.

- **MEMKIND\_HBW\_PREFERRED\_GBTLB (DEPRECATED)**

  Same as **MEMKIND\_HBW\_GBTLB** except that if there is not enough high bandwidth memory to satisfy the request, the allocation will fall back on standard memory. **Note:** This kind requires huge pages configuration described in **SYSTEM CONFIGURATION** section. For this kind, the allocation will not succeed if two or more high bandwidth memory NUMA nodes are in the same shortest distance to the same CPU on which process is eligible to run. Check on that eligibility is done upon starting the application.

- **MEMKIND\_HBW\_INTERLEAVE**

  Same as **MEMKIND\_HBW** except that the pages that support the allocation are interleaved across all high bandwidth nodes and transparent huge pages are disabled.

- **MEMKIND\_DAX\_KMEM**

  Allocate from the closest persistent memory NUMA node at the time of allocation. If there is not enough memory in the closest persistent memory NUMA node to satisfy the request *errno* is set to **ENOMEM** and the allocated pointer is set to *NULL*.

- **MEMKIND\_DAX\_KMEM\_ALL**

  Allocate from the closest persistent memory NUMA node available at the time of allocation. If there is not enough memory on any of persistent memory NUMA nodes to satisfy the request *errno* is set to **ENOMEM** and the allocated pointer is set to *NULL*.

- **MEMKIND\_DAX\_KMEM\_PREFERRED**

  Same as **MEMKIND\_DAX\_KMEM** except that if there is not enough memory in the closest persistent memory NUMA node to satisfy the request, the allocation will fall back on other memory NUMA nodes. **Note:** For this kind, the allocation will not succeed if two or more persistent memory NUMA nodes are in the same shortest distance to the same CPU on which process is eligible to run. Check on that eligibility is done upon starting the application.

- **MEMKIND\_DAX\_KMEM\_INTERLEAVE**

  Same as **MEMKIND\_DAX\_KMEM** except that the pages that support the allocation are interleaved across all persistent memory NUMA nodes.

- **MEMKIND\_REGULAR**

  Allocate from regular memory using the default page size. Regular means general purpose memory from the NUMA nodes containing CPUs.

## MEMORY TYPES

The available types of memory:

- **MEMKIND\_MEMTYPE\_DEFAULT**

  Standard memory, the same as process uses.

- **MEMKIND\_MEMTYPE\_HIGH\_BANDWIDTH**

  High bandwidth memory (HBM). There must be at least two memory types with different bandwidth to determine which is the HBM.

## MEMORY BINDING POLICY

The available types of memory binding policy:

- **MEMKIND\_POLICY\_BIND\_LOCAL**

  Allocate local memory. If there is not enough memory to satisfy the request *errno* is set to **ENOMEM** and the allocated pointer is set to *NULL*.

- **MEMKIND\_POLICY\_BIND\_ALL**

  Memory locality is ignored. If there is not enough memory to satisfy the request *errno* is set to **ENOMEM** and the allocated pointer is set to *NULL*.

- **MEMKIND\_POLICY\_PREFERRED\_LOCAL**

  Allocate preferred memory that is local. If there is not enough preferred memory to satisfy the request or preferred memory is not available, the allocation will fall back on any other memory.

- **MEMKIND\_POLICY\_INTERLEAVE\_LOCAL**

  Interleave allocation across local memory. For n memory types the allocation will be interleaved across all of them.

- **MEMKIND\_POLICY\_INTERLEAVE\_ALL**

  Interleave allocation. Locality is ignored. For n memory types the allocation will be interleaved across all of them.

- **MEMKIND\_POLICY\_MAX\_VALUE**

  Max policy value.

## MEMORY FLAGS

The available types of memory flags:

- **MEMKIND\_MASK\_PAGE\_SIZE\_2MB**

  Allocation backed by 2MB page size.

## MEMORY USAGE POLICY

The available types of memory usage policy:

- **MEMKIND\_MEM\_USAGE\_POLICY\_DEFAULT**

  Default memory usage policy.

- **MEMKIND\_MEM\_USAGE\_POLICY\_CONSERVATIVE**

  Conservative memory usage policy - prioritize memory usage at cost of performance. **Note:** Memory usage policies have no effect for TBB heap manager described in **ENVIRONMENT** section.

## MEMORY STATISTICS TYPE

The available types of memory statistics:

- **MEMKIND\_STAT\_TYPE\_RESIDENT**

  Maximum number of bytes in physically resident data pages mapped.

- **MEMKIND\_STAT\_TYPE\_ACTIVE**

  Total number of bytes in active pages.

- **MEMKIND\_STAT\_TYPE\_ALLOCATED**

  Total number of allocated bytes.

## MEMORY STATISTICS PRINT OPTIONS

The available options for printing statistics:

- **MEMKIND\_STAT\_PRINT\_ALL**

  Print all statistics.

- **MEMKIND\_STAT\_PRINT\_JSON\_FORMAT**

  Print statistics in JSON format.

- **MEMKIND\_STAT\_PRINT\_OMIT\_GENERAL**

  Omit general information that never changes during execution.

- **MEMKIND\_STAT\_PRINT\_OMIT\_MERGED\_ARENA**

  Omit merged arena statistics.

- **MEMKIND\_STAT\_PRINT\_OMIT\_DESTROYED\_MERGED\_ARENA**

  Omit destroyed merged arena statistics.

- **MEMKIND\_STAT\_PRINT\_OMIT\_PER\_ARENA**

  Omit per arena statistics.

- **MEMKIND\_STAT\_PRINT\_OMIT\_PER\_SIZE\_CLASS\_BINS**

  Omit per size class statistics for bins.

- **MEMKIND\_STAT\_PRINT\_OMIT\_PER\_SIZE\_CLASS\_LARGE**

  Omit per size class statistics for large objects.

- **MEMKIND\_STAT\_PRINT\_OMIT\_MUTEX**

  Omit all mutex statistics.

- **MEMKIND\_STAT\_PRINT\_OMIT\_EXTENT**

  Omit extent statistics.

## ERRORS

- **memkind\_posix\_memalign**()

  returns the one of the POSIX standard error codes **EINVAL** or **ENOMEM** as defined in *\<errno.h\>* if an error occurs (these have positive values). If the *alignment* parameter is not a power of two or is not a multiple of *sizeof(void\*)*, then **EINVAL** is returned. If there is insufficient memory to satisfy the request then **ENOMEM** is returned.

All functions other than **memkind\_posix\_memalign**() which have an integer return type return one of the negative error codes as defined in *\<memkind.h\>* and described below.

- **MEMKIND\_ERROR\_UNAVAILABLE**

  Requested memory kind is not available

- **MEMKIND\_ERROR\_MBIND**

  Call to [**mbind**(2)](/mbind/2.md) failed

- **MEMKIND\_ERROR\_MMAP**

  Call to [**mmap**(2)](/mmap/2.md) failed

- **MEMKIND\_ERROR\_MALLOC**

  Call to jemalloc's **malloc**() failed

- **MEMKIND\_ERROR\_ENVIRON**

  Error parsing environment variable *MEMKIND\_\**

- **MEMKIND\_ERROR\_INVALID**

  Invalid input arguments to memkind routine

- **MEMKIND\_ERROR\_TOOMANY**

  Error trying to initialize more than maximum **MEMKIND\_MAX\_KIND** number of kinds

- **MEMKIND\_ERROR\_BADOPS**

  Error memkind operation structure is missing or invalid

- **MEMKIND\_ERROR\_HUGETLB**

  Unable to allocate huge pages

- **MEMKIND\_ERROR\_MEMTYPE\_NOT\_AVAILABLE**

  Error requested memory type is not available

- **MEMKIND\_ERROR\_OPERATION\_FAILED**

  Error memkind operation failed

- **MEMKIND\_ERROR\_ARENAS\_CREATE**

  Call to jemalloc's **arenas.create**() failed

- **MEMKIND\_ERROR\_RUNTIME**

  Unspecified run-time error

## FILES

- */usr/bin/memkind-hbw-nodes*

  Prints a comma-separated list of high bandwidth nodes.

- */usr/bin/memkind-auto-dax-kmem-nodes*

  Prints a comma-separated list of persistent memory NUMA nodes, which are automatically detected.

## ENVIRONMENT

- **MEMKIND\_HBW\_NODES**

  This environment variable is a comma-separated list of NUMA nodes that are treated as high bandwidth. Uses the *libnuma* routine **numa\_parse\_nodestring**() for parsing, so the syntax described in the [**numa**(3)](/numa/3.md) man page for this routine applies: e.g. 1-3,5 is a valid setting.

- **MEMKIND\_HBW\_THRESHOLD**

  This environment variable is bandwidth in MB/s that is the threshold for identifying high bandwidth memory. The default threshold is 204800 (200 GB/s), which is used if this variable is not set. When set, it must be greater than or equal to 0.

- **MEMKIND\_DAX\_KMEM\_NODES**

  This environment variable is a comma-separated list of NUMA nodes that are treated as PMEM memory. Uses the *libnuma* routine **numa\_parse\_nodestring**() for parsing, so the syntax described in the [**numa**(3)](/numa/3.md) man page for this routine applies: e.g. 1-3,5 is a valid setting.

- **MEMKIND\_ARENA\_NUM\_PER\_KIND**

  This environment variable allows leveraging internal mechanism of the library for setting number of arenas per kind. Value should be a positive integer (not greater than **INT\_MAX** defined in *\<limits.h\>*). The user should set the value based on the characteristics of application that is using the library. Higher value can provide better performance in extremely multithreaded applications at the cost of memory overhead. See section **IMPLEMENTATION NOTES** of [**jemalloc**(3)](/jemalloc/3.md) for more details about arenas.

- **MEMKIND\_HOG\_MEMORY**

  Controls behavior of memkind with regards to returning memory to underlying OS. Setting **MEMKIND\_HOG\_MEMORY** to 1 causes memkind to not release memory to OS in anticipation of memory reuse soon. This will improve latency of 'free' operations but increase memory usage. **Note:** For file-backed kind memory will be released to OS only after calling **memkind\_destroy\_kind(),** not after 'free' operations. In context of **MEMKIND\_MEM\_USAGE\_POLICY\_CONSERVATIVE** memory usage policy - it will also impact memory coalescing and results that blocks pages will be often reused (better memory usage at cost of performance).

- **MEMKIND\_DEBUG**

  Controls logging mechanism in memkind. Setting **MEMKIND\_DEBUG** to 1 enables printing messages like errors and general information about environment to *stderr*.

- **MEMKIND\_BACKGROUND\_THREAD\_LIMIT**

  Enable background worker threads. Value should be from range 0 to maximum number of cpus. Setting **MEMKIND\_BACKGROUND\_THREAD\_LIMIT** to specific value will limit maximum number of background worker threads to this value. 0 means maximum number of background worker threads will be limited to maximum number of cpus.

- **MEMKIND\_HEAP\_MANAGER**

  Controls heap management behavior in memkind library by switching to one of the available heap managers.\
   Values:\
  \
       JEMALLOC - sets the jemalloc heap manager\
  \
       TBB - sets the Intel Threading Building Blocks heap manager. This option requires installed\
       Intel Threading Building Blocks library.

If the **MEMKIND\_HEAP\_MANAGER** is not set then the jemalloc heap manager will be used by default.

## SYSTEM CONFIGURATION

Interfaces for obtaining 2MB (HUGETLB) memory need allocated huge pages in the kernel's huge page pool.

- **HUGETLB (huge pages)**

  Current number of "persistent" huge pages can be read from */proc/sys/vm/nr\_hugepages* file. Proposed way of setting hugepages is: **sudo sysctl vm.nr\_hugepages=\<number\_of\_hugepages\>**. More information can be found here: ⟨https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt⟩

Interfaces for obtaining locality information are provided by *libhwloc* dependency. Functionality based on locality requires that memkind library is configured and built with the support of *libhwloc* (./configure --enable-hwloc).

Interfaces for obtaining memory performance characteristics information are based on *HMAT* (Heterogeneous Memory Attribute Table) ⟨https://uefi.org/sites/default/files/resources/ACPI\_6\_3\_final\_Jan30.pdf⟩ Functionality based on memory performance characteristics requires that platform configuration fully supports HMAT and memkind library is configured and built with the support of *libhwloc* (./configure --enable-hwloc).

**Note:** For a given target NUMA Node, the OS exposes only the performance characteristics of the best performing NUMA node.

*libhwloc* can be reached on: ⟨https://www.open-mpi.org/projects/hwloc⟩

## STATIC LINKING

When linking statically against memkind, *libmemkind.a* should be used together with its dependencies *libnuma* and pthread. Pthread can be linked by adding */usr/lib64/libpthread.a* as a dependency (exact path may vary). Typically *libnuma* will need to be compiled from sources to use it as a static dependency. *libnuma* can be reached on GitHub: ⟨https://github.com/numactl/numactl⟩

## KNOWN ISSUES

- **HUGETLB (huge pages)**

  There might be some overhead in huge pages consumption caused by heap management. If your allocation fails because of OOM, please try to allocate extra huge pages (e.g. 8 huge pages).

## COPYRIGHT

Copyright (C) 2014 - 2021 Intel Corporation. All rights reserved.

## SEE ALSO

[**malloc**(3)](/malloc/3.md), [**malloc\_usable\_size**(3)](/malloc_usable_size/3.md), [**numa**(3)](/numa/3.md), **hwloc**(3), [**numactl**(8)](/numactl/8.md), [**mbind**(2)](/mbind/2.md), [**mmap**(2)](/mmap/2.md), [**move\_pages**(2)](/move_pages/2.md), [**jemalloc**(3)](/jemalloc/3.md), [**memkind\_dax\_kmem**(3)](/memkind_dax_kmem/3.md), [**memkind\_default**(3)](/memkind_default/3.md), [**memkind\_arena**(3)](/memkind_arena/3.md), [**memkind\_fixed**(3)](/memkind_fixed/3.md), [**memkind\_hbw**(3)](/memkind_hbw/3.md), [**memkind\_hugetlb**(3)](/memkind_hugetlb/3.md), [**memkind\_pmem**(3)](/memkind_pmem/3.md), **[syscall(2)](/syscall/2.md),** **[write(2)](/write/2.md)**