--- name: "DbMpool" section: "3db" platform: "linux" package: "libdb2++-dev" version: "2.4.14-1" release: "Debian 2.1 (Slink)" platforms: ["linux"] url: "https://new.carta.tech/DbMpool" --- # DbMpool(3db) shared memory buffer pool ## NAME DbMpool - shared memory buffer pool ## SYNOPSIS ``` #include static int DbMpool::open(char *dir, u_int32_t flags, int mode, DbEnv *dbenv, DbMpool **regionp); int DbMpool::close(); static int DbMpool::unlink(const char *dir, int force, DbEnv *); int DbMpool::db_register(int ftype, int (*pgin)(db_pgno_t pgno, void *pgaddr, Dbt *pgcookie), int (*pgout)(db_pgno_t pgno, void *pgaddr, Dbt *pgcookie)); int DbMpool::trickle(int pct, int *nwrotep); int DbMpool::sync(LSN *lsn); int DbMpool::stat(DB_MPOOL_STAT **gsp, DB_MPOOL_FSTAT *(*fsp)[], void *(*db_malloc)(size_t)); ``` ## DESCRIPTION The DB library is a family of classes that provides a modular programming interface to transactions and record-oriented file access. The library includes support for transactions, locking, logging and file page caching, as well as various indexed access methods. Many of the classes (e.g., the file page caching class) are useful independent of the other DB classes, although some classes are explicitly based on other classes (e.g., transactions and logging). For a general description of the DB package, see *db\_intro*(3). This manual page describes the specific details of the memory pool interface. The *DbMpool*(3) and *DbMpoolFile*(3) classes are the library interface intended to provide general-purpose, page-oriented buffer management of one or more files. While designed to work with the other Db classes, this class is also useful for more general purposes. The memory pools (DbMpool's) are referred to in this document as simply \`\`pools''. Pools may be shared between processes. Pools are usually filled by pages from one or more files (DbMpoolFile's). Pages in the pool are replaced in LRU (least-recently-used) order, with each new page replacing the page that has been unused the longest. Pages retrieved from the pool using *DbMpoolFile::get* are \`\`pinned'' in the pool, by default, until they are returned to the pool's control using the *DbMpoolFile::put* method. *DbMpool::open* The *DbMpool::open* method copies a pointer, to the memory pool identified by the **directory** *dir*, into the memory location referenced by *regionp*. If the *dbenv* argument to *DbMpool::open* was initialized using *DbEnv::appinit*, *dir* is interpreted as described by *DbEnv*(3). Otherwise, if *dir* is not NULL, it is interpreted relative to the current working directory of the process. If *dir* is NULL, the following environment variables are checked in order: \`\`TMPDIR'', \`\`TEMP'', and \`\`TMP''. If one of them is set, memory pool files are created relative to the directory it specifies. If none of them are set, the first possible one of the following directories is used: */var/tmp*, */usr/tmp*, */temp*, */tmp*, *C:/temp* and *C:/tmp*. All files associated with the memory pool are created in this directory. This directory must already exist when DbMpool::open is called. If the memory pool already exists, the process must have permission to read and write the existing files. If the memory pool does not already exist, it is optionally created and initialized. The *flags* and *mode* arguments specify how files will be opened and/or created when they don't already exist. The flags value is specified by **or**'ing together one or more of the following values: - DB\_CREATE Create any underlying files, as necessary. If the files do not already exist and the DB\_CREATE flag is not specified, the call will fail. - DB\_MPOOL\_PRIVATE Create a private MPOOL that is not shared with any other process (although it may be shared with other threads). - DB\_NOMMAP Always copy files in this memory pool into the local cache instead of mapping them into process memory (see the description of the *mp\_mmapsize* field of the DbEnv object for further information). - - DB\_THREAD Cause the DbMpool handle returned by the *DbMpool::open* method to be useable by multiple threads within a single address space, i.e., to be \`\`free-threaded''. All files created by the memory pool subsystem (other than files created by the *memp\_fopen* method, which are separately specified) are created with mode *mode* (as described in [*chmod*(2)](/chmod/2.md)) and modified by the process' umask value at the time of creation (see [*umask*(2)](/umask/2.md)). The group ownership of created files is based on the system and directory defaults, and is not further specified by DB. The memory pool subsystem is configured based on which set methods have been used. It is expected that applications will use a single DbEnv object as the argument to all of the subsystems in the DB package. The fields of the DbEnv object used by *DbMpool::open* are described below. As references to the DbEnv object may be maintained by *DbMpool::open*, it is necessary that the DbEnv object and memory it references be valid until the object is destroyed. Any of the DbEnv fields that are not explicitly set will default to appropriate values. The following fields in the DbEnv object may be initialized, using the appropriate set method, before calling *DbMpool::open*: - - void \*(\*db\_errcall)(char \*db\_errpfx, char \*buffer); - FILE \*db\_errfile; - const char \*db\_errpfx; - class ostream \*db\_error\_stream; - int db\_verbose; The error fields of the DbEnv behave as described for *DbEnv*(3). - size\_t mp\_mmapsize; Files that are opened read-only in the pool (and that satisfy a few other criteria) are, by default, mapped into the process address space instead of being copied into the local cache. This can result in better-than-usual performance, as available virtual memory is normally much larger than the local cache, and page faults are faster than page copying on many systems. However, in the presence of limited virtual memory it can cause resource starvation, and in the presence of large databases, it can result in immense process sizes. If *mp\_mmapsize* is non-zero, it specifies the maximum file size, in bytes, for a file to be mapped into the process address space. By default, it is set to 10Mb. - size\_t mp\_size; The suggested size of the pool, in bytes. This should be the size of the normal working data set of the application, with some small amount of additional memory for unusual situations. (Note, the working set is not the same as the number of simultaneously referenced pages, and should be quite a bit larger!) The default cache size is 128K bytes (16 8K byte pages), and may not be less than 20K bytes. The *DbMpool::open* method throws a *DbException*(3) or returns the value of *errno* on failure and 0 on success. *DbMpool::close* The *DbMpool::close* method closes the pool indicated by the DbMpool object, as returned by *DbMpool::open*. This method does not imply a call to *DbMpoolFile::sync*, but does imply a call to *DbMpoolFile::close* for any remaining open DbMpoolFile objects returned to this process by calls to *DbMpoolFile::open*. In addition, if the *dir* argument to *DbMpool::open* was NULL and *dbenv* was not initialized using *DbEnv::appinit*, all files created for this shared region will be removed, as if *DbMpool::unlink* were called. When multiple threads are using the DbMpool handle concurrently, only a single thread may call the *DbMpool::close* method. The *DbMpool::close* method throws a *DbException*(3) or returns the value of *errno* on failure and 0 on success. *DbMpool::unlink* The *DbMpool::unlink* method destroys the memory pool identified by the directory *dir*, removing all files used to implement the memory pool. (The directory *dir* is not removed.) If there are processes that have called *DbMpool::open* without calling *DbMpool::close* (i.e., there are processes currently using the memory pool), *DbMpool::unlink* will fail without further action, unless the force flag is set, in which case *DbMpool::unlink* will attempt to remove the memory pool files regardless of any processes still using the memory pool. The result of attempting to forcibly destroy the region when a process has the region open is unspecified. Processes using a shared memory region maintain an open file descriptor for it. On UNIX systems, the region removal should succeed and processes that have already joined the region should continue to run in the region without change, however processes attempting to join the memory pool will either fail or attempt to create a new region. On other systems, e.g., WNT, where the [*unlink*(2)](/unlink/2.md) system call will fail if any process has an open file descriptor for the file, the region removal will fail. In the case of catastrophic or system failure, database recovery must be performed (see [*db\_recover*(1)](/db_recover/1.md) or the DB\_RECOVER and DB\_RECOVER\_FATAL flags to *DbEnv::appinit*(3)). Alternatively, if recovery is not required because no database state is maintained across failures, it is possible to clean up a memory pool by removing all of the files in the directory specified to the *DbMpool::open* method, as memory pool files are never created in any directory other than the one specified to *DbMpool::open*. Note, however, that this has the potential to remove files created by the other DB subsystems in this database environment. The *DbMpool::unlink* method throws a *DbException*(3) or returns the value of *errno* on failure and 0 on success. *DbMpool::db\_register* The *DbMpool::db\_register* method registers page-in and page-out functions for files of type *ftype* in the specified pool. If the *pgin* function is non-NULL, it is called each time a page is read into the memory pool from a file of type *ftype*, or a page is created for a file of type *ftype* (see the DB\_MPOOL\_CREATE flag for the *DbMpoolFile::get* method). If the *pgout* function is non-NULL, it is called each time a page is written to a file of type *ftype*. Both the *pgin* and *pgout* functions are called with the page number, a pointer to the page being read or written, and any argument *pgcookie* that was specified to the *DbMpoolFile::open* method when the file was opened. The *pgin* and *pgout* functions should return 0 on success, and an applicable non-zero *errno* value on failure, in which case the *DbMpool* method calling it will also fail, returning that *errno* value. The purpose of the *DbMpool::db\_register* method is to support processing when pages are entered into, or flushed from, the pool. A file type must be specified to make it possible for unrelated threads or processes, that are sharing a pool, to evict each other's pages from the pool. Applications should call *DbMpool::db\_register*, during initialization, for each type of file requiring input or output processing that will be sharing the underlying pool. (No registry is necessary for the standard access method types, btree, hash and recno, as *Db::open*(3) registers them separately.) If a thread or process does not call *DbMpool::db\_register* for a file type, it is impossible for it to evict pages for any file requiring input or output processing from the pool. For this reason, *DbMpool::db\_register* should always be called by each application sharing a pool for each type of file included in the pool, regardless of whether or not the application itself uses files of that type. There are no standard values for *ftype*, *pgin*, *pgout* and *pgcookie*, except that the *ftype* value for a file must be a non-zero positive number, as negative numbers are reserved for internal use by the DB library. For this reason, applications sharing a pool must coordinate their values amongst themselves. The *DbMpool::db\_register* method throws a *DbException*(3) or returns the value of *errno* on failure and 0 on success. *DbMpool::trickle* The *DbMpool::trickle* method ensures that at least *pct* percent of the pages in the shared memory pool are clean by writing dirty pages to their backing files. If the *nwrotep* argument is non-NULL, the number of pages that were written to reach the correct percentage is returned in the memory location it references. The purpose of the *DbMpool::trickle* method is to enable a memory pool manager to ensure that a page is always available for reading in new information without having to wait for a write. The *DbMpool::trickle* method throws a *DbException*(3) or returns the value of *errno* on failure and 0 on success. *DbMpool::sync* The *DbMpool::sync* method ensures that all the modified pages in the pool with log sequence numbers (LSNs) less than the *lsn* argument are written to disk. The *DbMpool::sync* method throws a *DbException*(3) or returns the value of *errno* on failure, 0 on success, and DB\_INCOMPLETE if there were pages which need to be written but which *DbMpool::sync* was unable to write immediately. In addition, if *DbMpool::sync* returns success, the value of *lsn* will be overwritten with the largest LSN from any page which was written by *DbMpool::sync* to satisfy this request. The purpose of the *DbMpool::sync* method is to enable a transaction manager to ensure, as part of a checkpoint, that all pages modified by a certain time have been written to disk. Pages in the pool which cannot be written back to disk immediately (e.g., are currently pinned) are written to disk as soon as it is possible to do so. The expected behavior of the transaction manager is to call the *DbMpool::sync* method and then, if the return indicates that some pages could not be written immediately, to wait briefly and retry again with the same LSN until the *DbMpool::sync* method returns that all pages have been written. To support the *DbMpool::sync* functionality, it is necessary that the pool methods know the location of the LSN on the page for each file type. This location should be specified when the file is opened using the *DbMpoolFile::open* method. (Note, it is not required that the LSN be aligned on the page in any way.) *DbMpool::stat* The *DbMpool::stat* method creates statistical structures and copies pointers to them into user-specified memory locations. The statistics include the number of files participating in the pool, the active pages in the pool, and information as to how effective the cache has been. Statistical structures are created in allocated memory. If *db\_malloc* is non-NULL, it is called to allocate the memory, otherwise, the library function [*malloc*(3)](/malloc/3.md) is used. The function *db\_malloc* must match the calling conventions of the [*malloc*(3)](/malloc/3.md) library routine. Regardless, the caller is responsible for deallocating the returned memory. To deallocate the returned memory, free each returned memory pointer; pointers inside the memory do not need to be individually freed. If *gsp* is non-NULL, the global statistics for the memory pool object are copied into the memory location it references. The global statistics are stored in a structure of type DB\_MPOOL\_STAT (typedef'd in \). The following DB\_MPOOL\_STAT fields will be filled in: - u\_int32\_t st\_refcnt; The number of references to the region.\ - u\_int32\_t st\_regsize; The size of the region.\ - size\_t st\_cachesize; Cache size in bytes.\ - u\_int32\_t st\_cache\_hit; Requested pages found in the cache.\ - u\_int32\_t st\_cache\_miss; Requested pages not found in the cache.\ - u\_int32\_t st\_map; Requested pages mapped into the process' address space (there is no available information as to whether or not this request caused disk I/O, although examining the application page fault rate may be helpful).\ - u\_int32\_t st\_page\_create; Pages created in the cache.\ - u\_int32\_t st\_page\_in; Pages read into the cache.\ - u\_int32\_t st\_page\_out; Pages written from the cache to the backing file.\ - u\_int32\_t st\_ro\_evict; Clean pages forced from the cache.\ - u\_int32\_t st\_rw\_evict; Dirty pages forced from the cache.\ - u\_int32\_t st\_hash\_buckets; Number of hash buckets in buffer hash table.\ - u\_int32\_t st\_hash\_searches; Total number of buffer hash table lookups.\ - u\_int32\_t st\_hash\_longest; The longest chain ever encountered in buffer hash table lookups.\ - u\_int32\_t st\_hash\_examined; Total number of hash elements traversed during hash table lookups.\ - u\_int32\_t st\_page\_clean; Clean pages currently in the cache.\ - u\_int32\_t st\_page\_dirty; Dirty pages currently in the cache.\ - u\_int32\_t st\_page\_trickle; Dirty pages written using the *memp\_trickle* interface. If *fsp* is non-NULL, a pointer to a NULL-terminated variable length array of statistics for individual files in the memory pool is copied into the memory location it references. If no individual files currently exist in the memory pool, *fsp* will be set to NULL. The per-file statistics are stored in structures of type DB\_MPOOL\_FSTAT (typedef'd in \). The following DB\_MPOOL\_FSTAT fields will be filled in for each file in the pool, i.e., each element of the array: - char \*file\_name; The name of the file.\ - size\_t st\_pagesize; Page size in bytes.\ - u\_int32\_t st\_cache\_hit; Requested pages found in the cache.\ - u\_int32\_t st\_cache\_miss; Requested pages not found in the cache.\ - u\_int32\_t st\_map; Requested pages mapped into the process' address space.\ - u\_int32\_t st\_page\_create; Pages created in the cache.\ - u\_int32\_t st\_page\_in; Pages read into the cache.\ - u\_int32\_t st\_page\_out; Pages written from the cache to the backing file. The *DbMpool::stat* method throws a *DbException*(3) or returns the value of *errno* on failure and 0 on success. ## ENVIRONMENT VARIABLES The following environment variables affect the behavior of *DbMpoolFile*: - DB\_HOME If the *dbenv* argument to *DbMpool::open* was initialized using *db\_appinit*, the environment variable DB\_HOME may be used as the path of the database home for the interpretation of the *dir* argument to *DbMpool::open*, as described in *db\_appinit*(3). - TMPDIR If the *dbenv* argument to *DbMpool::open* was NULL or not initialized using *db\_appinit*, the environment variable TMPDIR may be used as the directory in which to create the memory pool, as described in the *DbMpool::open* section above. ## ERRORS Methods marked as returning *errno* will, by default, throw an exception that encapsulates the error information. The default error behavior can be changed, see *DbException*(3). The *DbMpool::open* method may fail and throw a *DbException*(3) or return *errno* for any of the errors specified for the following DB and library functions: DBmemp-\>pgin(3), DBmemp-\>pgout(3), DbLog::compare(3), DbLog::flush(3), DbMpool::close(3), DbMpool::unlink(3), [close(2)](/close/2.md), db\_version(3), [fcntl(2)](/fcntl/2.md), [fflush(3)](/fflush/3.md), [fsync(2)](/fsync/2.md), [lseek(2)](/lseek/2.md), [malloc(3)](/malloc/3.md), [memcmp(3)](/memcmp/3.md), [memcpy(3)](/memcpy/3.md), [memset(3)](/memset/3.md), [mmap(2)](/mmap/2.md), [munmap(2)](/munmap/2.md), [open(2)](/open/2.md), [sigfillset(3)](/sigfillset/3.md), [sigprocmask(2)](/sigprocmask/2.md), [stat(2)](/stat/2.md), [strcpy(3)](/strcpy/3.md), [strdup(3)](/strdup/3.md), [strerror(3)](/strerror/3.md), [strlen(3)](/strlen/3.md), [time(3)](/time/3.md), [unlink(2)](/unlink/2.md), and [write(2)](/write/2.md). In addition, the *DbMpool::open* method may fail and throw a *DbException*(3) or return *errno* for the following conditions: - \[EAGAIN\] The shared memory region was locked and (repeatedly) unavailable. - \[EINVAL\] An invalid flag value or parameter was specified. The DB\_THREAD flag was specified and spinlocks are not implemented for this architecture. A NULL pathname was specified without the DB\_MPOOL\_PRIVATE flag. The specified cache size was impossibly small. The *DbMpool::close* method may fail and throw a *DbException*(3) or return *errno* for any of the errors specified for the following DB and library functions: DbMpoolFile::close(3), [close(2)](/close/2.md), [fcntl(2)](/fcntl/2.md), [fflush(3)](/fflush/3.md), [munmap(2)](/munmap/2.md), and [strerror(3)](/strerror/3.md). The *DbMpool::unlink* method may fail and throw a *DbException*(3) or return *errno* for any of the errors specified for the following DB and library functions: [close(2)](/close/2.md), [fcntl(2)](/fcntl/2.md), [fflush(3)](/fflush/3.md), [malloc(3)](/malloc/3.md), [memcpy(3)](/memcpy/3.md), [memset(3)](/memset/3.md), [mmap(2)](/mmap/2.md), [munmap(2)](/munmap/2.md), [open(2)](/open/2.md), [sigfillset(3)](/sigfillset/3.md), [sigprocmask(2)](/sigprocmask/2.md), [stat(2)](/stat/2.md), [strcpy(3)](/strcpy/3.md), [strdup(3)](/strdup/3.md), [strerror(3)](/strerror/3.md), [strlen(3)](/strlen/3.md), and [unlink(2)](/unlink/2.md). In addition, the *DbMpool::unlink* method may fail and throw a *DbException*(3) or return *errno* for the following conditions: - \[EBUSY\] The shared memory region was in use and the force flag was not set. The *DbMpool::db\_register* method may fail and throw a *DbException*(3) or return *errno* for any of the errors specified for the following DB and library functions: [fcntl(2)](/fcntl/2.md), and [malloc(3)](/malloc/3.md). The *DbMpool::trickle* method may fail and throw a *DbException*(3) or return *errno* for any of the errors specified for the following DB and library functions: DBmemp-\>pgin(3), DBmemp-\>pgout(3), DbLog::compare(3), DbLog::flush(3), [close(2)](/close/2.md), [fcntl(2)](/fcntl/2.md), [fflush(3)](/fflush/3.md), [fsync(2)](/fsync/2.md), [lseek(2)](/lseek/2.md), [malloc(3)](/malloc/3.md), [memcmp(3)](/memcmp/3.md), [memcpy(3)](/memcpy/3.md), [memset(3)](/memset/3.md), [mmap(2)](/mmap/2.md), [open(2)](/open/2.md), [sigfillset(3)](/sigfillset/3.md), [sigprocmask(2)](/sigprocmask/2.md), [stat(2)](/stat/2.md), [strcpy(3)](/strcpy/3.md), [strdup(3)](/strdup/3.md), [strerror(3)](/strerror/3.md), [strlen(3)](/strlen/3.md), [time(3)](/time/3.md), [unlink(2)](/unlink/2.md), and [write(2)](/write/2.md). In addition, the *DbMpool::trickle* method may fail and throw a *DbException*(3) or return *errno* for the following conditions: - \[EINVAL\] An invalid flag value or parameter was specified. The *DbMpool::sync* method may fail and throw a *DbException*(3) or return *errno* for any of the errors specified for the following DB and library functions: DBmemp-\>pgin(3), DBmemp-\>pgout(3), DbLog::compare(3), DbLog::flush(3), [close(2)](/close/2.md), [fcntl(2)](/fcntl/2.md), [fflush(3)](/fflush/3.md), [fsync(2)](/fsync/2.md), [lseek(2)](/lseek/2.md), [malloc(3)](/malloc/3.md), [memcmp(3)](/memcmp/3.md), [memcpy(3)](/memcpy/3.md), [memset(3)](/memset/3.md), [mmap(2)](/mmap/2.md), [open(2)](/open/2.md), [qsort(3)](/qsort/3.md), [realloc(3)](/realloc/3.md), [sigfillset(3)](/sigfillset/3.md), [sigprocmask(2)](/sigprocmask/2.md), [stat(2)](/stat/2.md), [strcpy(3)](/strcpy/3.md), [strdup(3)](/strdup/3.md), [strerror(3)](/strerror/3.md), [strlen(3)](/strlen/3.md), [time(3)](/time/3.md), [unlink(2)](/unlink/2.md), and [write(2)](/write/2.md). In addition, the *DbMpool::sync* method may fail and throw a *DbException*(3) or return *errno* for the following conditions: - \[EINVAL\] An invalid flag value or parameter was specified. The *DbMpool::sync* method was called without logging having been initialized in the environment. The *DbMpool::stat* method may fail and throw a *DbException*(3) or return *errno* for any of the errors specified for the following DB and library functions: [fcntl(2)](/fcntl/2.md), [malloc(3)](/malloc/3.md), [memcpy(3)](/memcpy/3.md), and [strlen(3)](/strlen/3.md). ## SEE ALSO [*db\_archive*(1)](/db_archive/1.md), [*db\_checkpoint*(1)](/db_checkpoint/1.md), [*db\_deadlock*(1)](/db_deadlock/1.md), [*db\_dump*(1)](/db_dump/1.md), [*db\_load*(1)](/db_load/1.md), [*db\_recover*(1)](/db_recover/1.md), [*db\_stat*(1)](/db_stat/1.md), *db\_intro*(3), *db\_internal*(3), *db\_thread*(3), *Db*(3), *Dbc*(3), *DbEnv*(3), *DbException*(3), *DbInfo*(3), *DbLock*(3), *DbLockTab*(3), *DbLog*(3), *DbLsn*(3), *DbMpool*(3), *DbMpoolFile*(3), *Dbt*(3), *DbTxn*(3), *DbTxnMgr*(3)