--- name: "pmdaprometheus" section: "1" platform: "linux" package: "pcp" version: "4.3.2+really4.3.1-0.1" release: "Debian 10 (Buster)" platforms: ["linux"] url: "https://new.carta.tech/pmdaprometheus" --- # pmdaprometheus(1) Prometheus PMDA ## NAME **pmdaprometheus** - Prometheus PMDA ## SYNOPSIS **$PCP\_PMDAS\_DIR/prometheus/pmdaprometheus** \[**-D**\] \[**-n**\] \[**-c** *config*\] \[**-d** *domain*\] \[**-l** *logfile*\] \[**-r** *root*\] \[**-t** *timeout*\] \[**-u** *user*\] ## DESCRIPTION **pmdaprometheus** is a Performance Metrics Domain Agent (PMDA) which creates PCP metrics from Prometheus endpoints, which provide HTTP based access to application metrics. The default *config* directory is **$PCP\_PMDAS\_DIR/prometheus/config.d/**, see \`\`CONFIGURATION SOURCES'' below. The default URL fetch *timeout* is **2** seconds. The default *user*, if not specified with the **-u** option, is the current user. If the **-n** option is given, the list of configuration files will not be sorted prior to processing. This list is sorted by default but that can be expensive if there are a large number of configuration files (URLs and/or scripts). If the **-D** option is given, additional diagnostic messages will be written to the PMDA log file, which is **$PCP\_LOG\_DIR/pmcd/prometheus.log** by default (see also **-l**below). In addition, the metric **prometheus.control.debug** controls the same debug flag and can be set with the following command:\ **pmstore prometheus.control.debug** *value*\ where *value* is either **1** (to enable verbose log messages) or **0** (to disable verbose log messages). This is particularly useful for examining the http headers passed to each fetch request, filter settings and other processing details that are logged when the debugging flag is enabled. The **-d** option may be used to override the default performance metrics *domain* number, which defaults to **144.** It is strongly recommended not to change this. The *domain* number should be different for every PMDA on the one host, and the same *domain* number should be used for **pmdaprometheus** PMDA on all hosts. See also the **-r** option, which allows the root of the dynamic namespace to be changed from the default **prometheus**. The **-l** option may be used to specify *logfile* as the destination for PMDA messages instead of the default, **$PCP\_LOG\_DIR/pmcd/prometheus.log**. As a special case, *logfile* may be **"-"** to send messages to the **stderr** stream instead, e.g. **-l-**. This would normally be the **stderr** stream for the parent process, [**pmcd**(1)](/pmcd/1.md), which may itself have redirected **stderr**. This redirection is normally most useful in a containerized environment, or when using [**dbpmda**(1)](/dbpmda/1.md). The **-r** option allows the root of the dynamic namespace to be changed to *root* from the default, **prometheus**. In conjunction with other command line options, this allows **pmdaprometheus** to be deployed as a different PMDA with distinct metrics namespace and metrics domain on the same host system. Note that all PMDAs require a unique domain number so the **-d** option must also be specified. Use of the **-r** option may also change the defaults for some other command line options, e.g. the default log file name and the default configuration directory. ## CONFIGURATION SOURCES As it runs, **pmdaprometheus** periodically recursively scans the **$PCP\_PMDAS\_DIR/prometheus/config.d** directory (or the directory specified with the **-c** option), looking for source URL files (*\*.url*) and executable scripts or binaries. Any files that do not have the **.url** suffix or are not executable, are ignored - this allows documentation files such as "README" and non-executable "common" script function definitions to be present without being considered as config files. A remote server does not have to be up or stay running - the PMDA tolerates remote URLs that may come and go over time. The PMDA will relay data and metadata when/if they are available, and will return errors when/if they are down. PCP metric IDs, internal and external instance domain identifiers are persisted and will be restored when individual metric sources become available and/or when the PMDA is restarted. In addition, the PMDA checks directory modification times and will rescan for new or changed configuration files dynamically. It is *not* necessary to restart the PMDA when adding, removing or changing configuration files. ## URL SOURCES Each file with the *.url* suffix found in the config directory or sub-directory contains one complete HTTP or HTTPS URL at which **pmdaprometheus** can reach a Prometheus endpoint. Local file access is also supported with a conventional *file://somepath/somefile* URL, in which case *somepath/somefile* should contain prometheus formatted metric data. The first line of a *.url* config file should be the URL, as described above. Subsequent lines, if any, are prefixed with a keyword that can be used to alter the http GET request. A keyword must end with **':'** (colon) and the text extends to the end of the line. Comment lines that start with **#** and blank lines are ignored. The only currently supported keywords are **HEADER:** and **FILTER:**. **HEADER:** *headername**:** value ... to end of line*\ Adds *headername* and its value to the headers passed in the http GET request for the configured URL. An example configuration file that provides 3 commonly used headers and an authentication token might be : ``` http://somehost/path/endpoint.html # this is a comment HEADER: Accept: text/html HEADER: Keep-Alive: 300 HEADER: Connection: keep-alive HEADER: Authorization: token ABCDEF1234567890 ``` As mentioned above, header values extend to the end of the line. They may contain any valid characters, including colons. Multiple spaces will be collapsed to a single space, and leading and trailing spaces are trimmed. A common use for headers is to configure a proxy agent and the assorted parameters it may require. **FILTER: INCLUDE METRIC** *regex*\ or\ **FILTER: EXCLUDE METRIC** *regex*\ Dynamically created metric names that match *regex* will be either included or excluded in the name space, as specified. The simple rule is that the *first matching* filter regex for a particular metric name is the rule that prevails. If no filter regex matches (or there are no filters), then the metric is included by default, i.e. the default filter if none are specified is **FILTER: INCLUDE METRIC .\*** This is backward compatible with older versions of the configuration file that did not support filters. Multiple **FILTER:** lines would normally be used, e.g. to include some metrics but exclude all others, use **FILTER: EXCLUDE METRIC .\*** as the last of several filters that include the desired metrics. Conversely, to exclude some metrics but include all others, use **FILTER: EXCLUDE METRIC** *regex*. In this case it's not necessary (though doesn't hurt) to specify the final **FILTER: INCLUDE METRIC .\*** because, as stated above, any metric that does not match any filter regex will be included by default. Label filtering uses similar **FILTER:** syntax and semantics. **FILTER: EXCLUDE LABEL** *regex* will delete all labels matching *regex* from all metrics defined in the configuration file. The same rules as for metrics apply for labels too - an implicit rule: **FILTER: INCLUDE LABEL .\*** applies to all labels that do not match any earlier filter rule. Caution is needed with label filtering because by default, all labels are used to construct the PCP instance name. By excluding some labels, the instance names will change. Excluding all labels for a particular metric changes that metric to be singular, i.e. have no instance domain. In addition, by excluding some labels, different instances of the same metric may become duplicates. When such duplicates occur, the last duplicate instance returned by the end-point URL prevails over any earlier instances. For these reasons, it is recommended that label filtering rules be configured when the configuration file is first defined, and not changed thereafter. If a label filtering change is required, the configuration file should be renamed, which effectively defines a new metric, with the new (or changed) instance naming. Unrecognized keywords in configuration files are reported in the PMDA log file but otherwise ignored. ## SCRIPTED SOURCES Executable scripts present in the *$PCP\_PMDAS\_DIR/prometheus/config.d* directory or sub-directories will be executed and the **stdout** stream containing prometheus formatted metric data will be parsed as though it had come from a URL or file. The **stderr** stream from a script will be sent to the PMDA log file, which by default can be found in **$(PCP\_LOG\_DIR)/pmcd/prometheus.log**. Note that scripted sources do not support label or metric filtering (as described above for URL sources) - they can simply do their own filtering in the script itself with [**sed**(1)](/sed/1.md), [**awk**(1)](/awk/1.md), or whatever tool is desired. A simple example of a scripted config entry follows: ``` #! /bin/sh awk '{ print("# HELP loadavg local load average") print("# Type loadavg gauge") printf("loadavg {interval=\"1-minute\"} %.2f\n", $1) printf("loadavg {interval=\"5-minute\"} %.2f\n", $2) printf("loadavg {interval=\"15-minute\"} %.2f\n", $3) }' /proc/loadavg ``` This script produces the following Prometheus-formatted metric data when run: ``` # HELP loadavg local load average # Type loadavg gauge loadavg {interval="1-minute"} 0.12 loadavg {interval="5-minute"} 0.27 loadavg {interval="15-minute"} 0.54 ``` If the above script was saved and made executable in a file named *$PCP\_PMDAS\_DIR/prometheus/config.d/local/system.sh* then this would result in a new PCP metric named **prometheus.local.system.loadavg** which would have three instances for the current load average values: **1-minute**, **5-minute** and **15-minute**. Scripted config entries may produce more than one PCP leaf metric name. For example, the above "system.sh" script could also export other metrics such as CPU statistics, by reading */proc/stat* on the local system. Such additional metrics would appear as peer metrics in the same PCP metric subtree. In the case of CPU counters, the metric type definition should be **counter**, not **gauge**. For full details of the prometheus exposition formats, see *https://prometheus.io/docs/instrumenting/exposition\_formats*. ## METRIC NAMING All metrics from a file named *JOB*.\* will be exported as PCP metrics with the *prometheus.JOB* metric name prefix. Therefore, the JOB name must be a valid non-leaf name for PCP PMNS metric names. If the *JOB* name has multiple dot-separated components, the resulting PMNS names will include those components and care is needed to ensure there are no overlapping definitions, e.g. metrics returned by **JOB.response** may overlap or conflict with metrics returned by **JOB.response.time**. Config file entries (URLs or scripts) found in subdirectories of the config directory will also result in hierarchical metric names. For example, a config file named **$PCP\_PMDAS\_DIR/prometheus/config.d/mysource/latency/get.url** will result in metrics being created (by fetching that source URL) below **prometheus.mysource.latency.get** in the PCP namespace. Scripts found in subdirectories of the config directory similarly result in hierarchical PCP metric names. ## DYNAMIC METRIC NAMES As described above, changes and new additions can be made to files in the configuration directory without having to restart the PMDA. These changes are detected automatically and the PCP metric names below **prometheus** in the PMNS will be updated accordingly, i.e. new metrics will be dynamically added and/or existing metrics removed. In addition, **pmdaprometheus** honors the PMCD\_NAMES\_CHANGE [**pmFetch**(3)](/pmFetch/3.md) protocol that was introduced in PCP version 4.0. In particular, if **prometheus** metrics are being logged by a PCP version 4.0 or later [**pmlogger**(1)](/pmlogger/1.md), new metrics that appear as a result of changes in the PMDA configuration directory will automatically start to be logged, provided the root of the **prometheus** PMDA namespace is configured for logging in the **pmlogger** configuration file. See [**pmlogger**(1)](/pmlogger/1.md) for details. An example of such a **pmlogger** configuration file is : ``` log mandatory on 2 second { # log all metrics below the root of the prometheus namespace prometheus } ``` ## CONTROL METRICS The PMDA maintains special control metrics, as described below. Apart from **prometheus.control.debug**, each of these metrics is a counter and has one instance for each configured metric source. The instance domain is adjusted dynamically as new sources are discovered. If there are no sources configured, the metric names are still defined but the instance domain will be empty and a fetch will return no values. - **prometheus.control.calls** total number of times each configured metric source has been fetched (if it's a URL) or executed (if it's a script), since the PMDA started. - **prometheus.control.fetch\_time** Total time in milliseconds that each configured metric source has taken to return a document, excluding the time to parse the document. - **prometheus.control.parse\_time** Total time in milliseconds that each configured metric source has taken to parse each document, excluding the time to fetch the document. When converted to a rate, the **calls** metric represents the average fetch rate of each source over the sampling interval (time delta between samples). The **fetch\_time** and **parse\_time** counters, when converted to a rate, represent the average fetch and parsing latency (respectfully), during the sampling interval. The **prometheus.control.debug** metric has a singular value, defaulting to **0**. If a non-zero value is stored into this metric using [**pmstore**(1)](/pmstore/1.md), additional debug messages will be written to the PMDA log file. ## LIMITATIONS **pmdaprometheus** and **libpcp** internals impose some numerical constraints about the number of sources (4095), metrics (1024) within each source, and instances for each metric (4194304). ## INSTALLATION Install the Prometheus PMDA by using the Install script as root: \# cd $PCP\_PMDAS\_DIR/prometheus\ \ \# ./Install To uninstall, do the following as root: \# cd $PCP\_PMDAS\_DIR/prometheus\ \ \# ./Remove **pmdaprometheus** is launched by [**pmcd**(1)](/pmcd/1.md) and should never be executed directly. The Install and Remove scripts notify **pmcd** when the agent is installed or removed. When scripts and *.url* files are added, removed or changed in the configuration directory, it is usually not necessary to restart the PMDA - the changes will be detected and managed on subsequent requests to the PMDA. ## FILES - **$PCP\_PMDAS\_DIR/prometheus/Install** installation script for the **pmdaprometheus** agent - **$PCP\_PMDAS\_DIR/prometheus/Remove** undo installation script for the **pmdaprometheus** agent - **$PCP\_PMDAS\_DIR/prometheus/config.d/** contains URLs and scripts used by the **pmdaprometheus** agent as sources of prometheus metric data. - **$PCP\_LOG\_DIR/pmcd/prometheus.log** default log file for error messages from **pmdaprometheus** - **$PCP\_VAR\_DIR/config/144.\*** files containing internal tables for metric and instance ID number persistence (domain 144). ## PCP ENVIRONMENT Environment variables with the prefix **PCP\_** are used to parameterize the file and directory names used by **PCP**. On each installation, the file */etc/pcp.conf* contains the local values for these variables. The **$PCP\_CONF** variable may be used to specify an alternative configuration file, as described in [*pcp.conf*(5)](/pcp.conf/5.md). ## SEE ALSO [**pmcd**(1)](/pmcd/1.md), [**pminfo**(1)](/pminfo/1.md), [**pmlogger**(1)](/pmlogger/1.md), [**pmstore**(1)](/pmstore/1.md), [**PMWEBAPI**(3)](/PMWEBAPI/3.md), [**pmFetch**(3)](/pmFetch/3.md) and *https://prometheus.io/docs/instrumenting/exposition\_formats*.