--- name: "pmfault" section: "3" platform: "linux" package: "pcp" version: "7.2.2-2" release: "Arch Linux 2026-11-01" platforms: ["linux"] url: "https://new.carta.tech/pmfault" --- # pmfault(3) Fault Injection Infrastructure for QA ## NAME **\_\_pmFaultInject**, **\_\_pmFaultSummary**, **PM\_FAULT\_POINT**, **PM\_FAULT\_RETURN**, **PM\_FAULT\_CHECK**, **PM\_FAULT\_CLEAR** - Fault Injection Infrastructure for QA ## C SYNOPSIS **#include \**\ **#include \** **void \_\_pmFaultInject(const char \****ident***, int** *class***);**\ **void \_\_pmFaultSummary(FILE \****f***);** **PM\_FAULT\_POINT(***ident***,** *class***);**\ **PM\_FAULT\_RETURN(***retvalue***);**\ **PM\_FAULT\_CHECK;**\ **PM\_FAULT\_CLEAR;** **cc -DPM\_FAULT\_INJECTION=1 ... -lpcp\_fault** ## DESCRIPTION As part of the coverage-driven changes to QA in PCP 3.6, it became apparent that we needed someway to exercise the \`\`uncommon'' code paths associated with error detection and recovery. The facilities described below provide a basic fault injection infrastructure (for *libpcp* only at this stage, although the mechanism is far more general and could easily be extended). A special build is required to create *libpcp\_fault* and the associated *\* header file. Once this has been done, new QA applications may be built with **-DPM\_FAULT\_INJECTION=1** and/or existing applications can be exercised in presence of fault injection by forcing *libpcp\_fault* to be used in preference to *libpcp* as described below. In the code to be tested, **\_\_pmFaultInject** defines a fault point at which a fault of type *class* may be injected. *ident* is a string to uniquely identify the fault point across all of the PCP source code, so something like "libpcp/" \_\_FILE\_\_ ":\" works just fine. The *ident* string also determines if a fault will be injected at run-time or not - refer to the **RUN-TIME CONTROL** section below. *class* selects a failure type, using one of the following defined values (this list may well grow over time): - **PM\_FAULT\_ALLOC** Will cause the **next** call to [**malloc**(3)](/malloc/3.md), [**realloc**(3)](/realloc/3.md) or [**strdup**(3)](/strdup/3.md) to fail, returning NULL and setting *errno* to **ENOMEM**. We could extend the coverage to all of the malloc-related routines, but these three are sufficient to cover the vast majority of the uses within *libpcp*. - **PM\_FAULT\_CALL** Will cause the **next** call to an instrumented routine to fail by returning an error code (possibly the new **PM\_ERR\_FAULT** code). The actual error code is defined in the **PM\_FAULT\_RETURN** macro at the head of an instrumented routine. Initially, only **\_\_pmRegisterAnon**(3) (returns **PM\_ERR\_FAULT**), **\_\_pmGetPDU**(3) (returns **PM\_ERR\_TIMEOUT**) and **\_\_pmAllocResult**(3) (returns **NULL**) were instrumented as a proof of concept for this part of the facility, however other routines may have this fault injection capability added over time. - **PM\_FAULT\_MISC** The \`\`other'' class, currently used with **PM\_FAULT\_CHECK** as described below. To allow fault injection to co-exist within the production source code, **PM\_FAULT\_POINT** is a macro that emits no code by default, but when **PM\_FAULT\_INJECTION** is defined this becomes a call to **\_\_pmFaultInject**. Throughout *libpcp* we use **PM\_FAULT\_POINT** and **not** **\_\_pmFaultInject** so that both *libpcp* and *libpcp\_fault* can be built from the same source code. Similarly, the macro **PM\_FAULT\_RETURN** emits no code unless **PM\_FAULT\_INJECTION** is defined, in which case if a fault of type **PM\_FAULT\_CALL** has been armed with **\_\_pmFaultInject** then, the enclosing routine return with the function value *retvalue*. The **PM\_FAULT\_CHECK** macro returns a value that may be 0 or 1. If **PM\_FAULT\_INJECTION** is defined then if a fault of type **PM\_FAULT\_MISC** has been armed with **\_\_pmFaultInject** then the value is 1 else it is 0. **PM\_FAULT\_CHECK** is most often used in concert with the **PM\_FAULT\_POINT** macro with the **PM\_FAULT\_MISC** class to potentially arm a trigger, then test **PM\_FAULT\_CHECK** and if this has the value 1, then the **PM\_FAULT\_CLEAR** macro is used to clear any armed faults, and the fault injection code is executed. This is illustrated in the example below from *src/libpcp/src/exec.c*: ``` pid = fork(); /* begin fault-injection block */ PM_FAULT_POINT("libpcp/" __FILE__ ":4", PM_FAULT_MISC); if (PM_FAULT_CHECK) { PM_FAULT_CLEAR; if (pid > (pid_t)0) kill(pid, SIGKILL); setoserror(EAGAIN); pid = -1; } /* end fault-injection block */ ``` A summary of fault points seen and faults injected is produced on stdio stream *f* by **\_\_pmFaultSummary**. Additional tracing (via **-Dfault** or **pmDebugOptions.fault**) and a new PMAPI error code (**PM\_ERR\_FAULT**) are also defined, although these will only ever be seen or used in *libpcp\_fault*. If **pmDebugOptions.fault** is set the first time **\_\_pmFaultInject** is called, then **\_\_pmFaultSummary** will be called automatically to report on *stderr* when the application exits (via [**atexit**(3)](/atexit/3.md)). Fault injection cannot be nested. Each call to **\_\_pmFaultInject** clears any previous fault injection that has been armed, but not yet executed. The fault injection infrastructure is **not** thread-safe and should only be used with applications that are known to be single-threaded. ## RUN-TIME CONTROL By default, no fault injection is enabled at run-time, even when **\_\_pmFaultInject** is called. Faults are selectively enabled using a control file, identified by the environment variable **$PM\_FAULT\_CONTROL**; if this is not set, no faults are enabled. The control file (if it exists) is read the first time **\_\_pmFaultInject** is called, and contains lines of the form: *ident* *op* *number*\ that define fault injection guards. *ident* is a fault point string (as defined by a call to **\_\_pmFaultInject**, or more usually the **PM\_FAULT\_POINT** macro). So one needs access to the *libpcp* source code to determine the available *ident* strings and their semantics. *op* is one of the C-style operators **\>=**, **\>**, **==**, **\<**, **\<=**, **!=** or **%** and *number* is an unsigned integer. *op* *number* is optional and the default is **\>0** The semantics of the fault injection guards are that each time **\_\_pmFaultInject** is called for a particular *ident*, a trip count is incremented (the first trip is 1); if the C-style expression *tripcount* *op* *number* has the value 1 (so **true** for most *op*s, or the remainder equals 1 for the **%** *op*), then a fault of the *class* defined for the fault point associated with *ident* will be armed, and executed as soon as possible. Within the control file, blank lines are ignored and lines beginning with # are treated as comments. For an existing application linked with *libpcp* fault injection may still be used by forcing *libpcp\_fault* to be used in the place of *libpcp*. The following example shows how this might be done. ``` $ PM_FAULT_CONTROL=/tmp/control; export PM_FAULT_CONTROL $ cat $PM_FAULT_CONTROL # ok for 2 trips, then inject errors libpcp/events.c:1 >2 $ LD_PRELOAD=/usr/lib/libpcp_fault.so; export LD_PRELOAD $ pmevent -Dfault -s 3 sample.event.records host: localhost samples: 3 interval: 1.00 sec sample.event.records[fungus]: 0 event records __pmFaultInject(libpcp/events.c:1) ntrip=1 SKIP sample.event.records[bogus]: 2 event records 10:46:12.413 --- event record [0] flags 0x1 (point) --- sample.event.param_string "fetch #0" 10:46:12.413 --- event record [1] flags 0x1 (point) --- sample.event.param_string "bingo!" __pmFaultInject(libpcp/events.c:1) ntrip=2 SKIP sample.event.records[fungus]: 1 event records 10:46:03.416 --- event record [0] flags 0x1 (point) --- __pmFaultInject(libpcp/events.c:1) ntrip=3 INJECT sample.event.records[bogus]: pmUnpackEventRecords: Cannot allocate memory __pmFaultInject(libpcp/events.c:1) ntrip=4 INJECT sample.event.records[fungus]: pmUnpackEventRecords: Cannot allocate memory __pmFaultInject(libpcp/events.c:1) ntrip=5 INJECT sample.event.records[bogus]: pmUnpackEventRecords: Cannot allocate memory === Fault Injection Summary Report === libpcp/events.c:1: guard trip>2, 5 trips, 3 faults ``` ## EXAMPLES Refer to the PCP and PCP QA source code. The macro definitions are in *src/include/pcp/fault.h*. *src/libpcp/src/fault.c* contains all of the the underlying implementation. *src/libpcp\_fault* and *src/libpcp\_fault/src* contains the recipe and Makefiles for creating and installing *libpcp\_fault.so* and *\*. **PM\_FAULT\_RETURN** was initiallly used in the following *libpcp* source files: *derive\_parser.y.in*, *pdu.c* and *result.c*. **PM\_FAULT\_POINT**. was initiallly used in the following *libpcp* source files: *derive\_parser.y.in*, *desc.c*, *e\_indom.c*, *e\_labels.c*, *err.c*, *events.c*, *exec.c*, *fetch.c*, *help.c*, *instance.c*, *interp.c*, *labels.c*, *logmeta.c*, *pmns.c*, *p\_profile.c* and *store.c*. The \`\`fault'' group of QA tests show examples of control file use. To see which tests are involved ``` $ cd qa $ check -n -g fault ``` ## DIAGNOSTICS Some non-recoverable errors are reported on *stderr*. ## ENVIRONMENT - **PM\_FAULT\_CONTROL** Full path to the fault injection control file. - **LD\_PRELOAD** Force *libpcp\_fault* to be used in preference to *libpcp*. ## SEE ALSO [**PMAPI**(3)](/PMAPI/3.md)