--- name: "crm" section: "8" platform: "linux" package: "crmsh" version: "5.0.0~rc1-1" release: "Debian 13 (Trixie)" platforms: ["linux"] url: "https://new.carta.tech/crm/8" --- # crm(8) Pacemaker command line interface for configuration and management ## NAME crm - Pacemaker command line interface for configuration and management ## SYNOPSIS **crm** \[OPTIONS\] \[SUBCOMMAND ARGS...\] ## DESCRIPTION The crm shell is a command-line based cluster configuration and management tool. Its goal is to assist as much as possible with the configuration and maintenance of Pacemaker-based High Availability clusters. For more information on Pacemaker itself, see http://clusterlabs.org/. crm works both as a command-line tool to be called directly from the system shell, and as an interactive shell with extensive tab completion and help. The primary focus of the crm shell is to provide a simplified and consistent interface to Pacemaker, but it also provides tools for managing the creation and configuration of High Availability clusters from scratch. To learn more about this aspect of crm, see the cluster section below. The crm shell can be used to manage every aspect of configuring and maintaining a cluster. It provides a simplified line-based syntax on top of the XML configuration format used by Pacemaker, commands for starting and stopping resources, tools for exploring the history of a cluster including log scraping and a set of cluster scripts useful for automating the setup and installation of services on the cluster nodes. The crm shell is line oriented: every command must start and finish on the same line. It is possible to use a continuation character (\\) to write one command in two or more lines. The continuation character is commonly used when displaying configurations. ## OPTIONS - **-f, --file**=*FILE* Load commands from the given file. If a dash - is used in place of a file name, crm will read commands from the shell standard input (stdin). - **-c, --cib**=*CIB* Start the session using the given shadow CIB file. Equivalent to cib use \. - **-D, --display=***OUTPUT\_TYPE* Choose one of the output options: plain, color-always, color, or uppercase. The default is color if the terminal emulation supports colors. Otherwise, plain is used. - **-F, --force** Make crm proceed with applying changes where it would normally ask the user to confirm before proceeding. This option is mainly useful in scripts, and should be used with care. - **-w, --wait** Make crm wait for the cluster transition to finish (for the changes to take effect) after each processed line. - **-H, --history**=*DIR\|FILE\|SESSION* A directory or file containing a cluster report to load into the history commands, or the name of a previously saved history session. - **-h, --help** Print help page. - **--version** Print crmsh version and build information (Mercurial Hg changeset hash). - **-d, --debug** Print verbose debugging information. - **-R, --regression-tests** Enables extra verbose trace logging used by the regression tests. Logs all external calls made by crmsh. - **--scriptdir**=*DIR* Extra directory where crm looks for cluster scripts, or a list of directories separated by semi-colons (e.g. /dir1;/dir2;etc.). - **-o, --opt**=*OPTION=VALUE* Set crmsh option temporarily. If the options are saved using options save then the value passed here will also be saved. Multiple options can be set by using -o multiple times. ## INTRODUCTION This section of the user guide covers general topics about the user interface and describes some of the features of crmsh in detail. ### User interface The main purpose of crmsh is to provide a simple yet powerful interface to the cluster stack. There are two main modes of operation with the user interface of crmsh: • Command line (single-shot) use - Use crm as a regular UNIX command from your usual shell. crm has full bash completion built in, so using it in this manner should be as comfortable and familiar as using any other command-line tool. • Interactive mode - By calling crm without arguments, or by calling it with only a sublevel as argument, crm enters the interactive mode. In this mode, it acts as its own command shell, which remembers which sublevel you are currently in and allows for rapid and convenient execution of multiple commands within the same sublevel. This mode also has full tab completion, as well as built-in interactive help and syntax highlighting. Here are a few examples of using crm both as a command-line tool and as an interactive shell: **Command line (one-shot) use:**. ``` # crm resource stop www_app ``` **Interactive use:**. ``` # crm crm(live)# resource crm(live)resource# unmanage tetris_1 crm(live)resource# up crm(live)# node standby node4 ``` **Cluster configuration:**. ``` # crm configure<, or, if at the top level of the shell, simply typing help will provide an overview of all available levels and commands. The (live) string in the crm prompt signifies that the current CIB in use is the cluster live configuration. It is also possible to work with so-called shadow CIBs. These are separate, inactive configurations stored in files, that can be applied and thereby replace the live configuration at any time. ### Tab completion The crm makes extensive use of tab completion. The completion is both static (i.e. for crm commands) and dynamic. The latter takes into account the current status of the cluster or information from installed resource agents. Sometimes, completion may also be used to get short help on resource parameters. Here are a few examples: ``` crm(live)resource# ban demote maintenance param scores trace cd failcount manage promote secret unmanage cleanup help meta quit start untrace clear locate move refresh status up constraints ls operations restart stop utilization crm(live)configure# primitive fence-1 ocf: stonith: systemd: crm(live)configure# primitive fence-1 stonith: stonith:apcmaster stonith:fence_compute stonith:fence_ipmilanplus stonith:apcmastersnmp stonith:fence_docker stonith:fence_ironic stonith:apcsmart stonith:fence_drac5 stonith:fence_kdump ... crm(live)configure# primitive fence-1 stonith:fence_ipmilan params action= ip= passwd= pcmk_host_list= auth= ipaddr= passwd_script= pcmk_host_map= ... crm(live)configure# primitive fence-1 stonith:fence_ipmilan params auth= auth (select): IPMI Lan Auth type. Allowed values: md5, password, none ``` crmsh also comes with bash completion usable directly from the system shell. This should be installed automatically with the command itself. ### Shorthand syntax When using the crm shell to manage clusters, you will end up typing a lot of commands many times over. Clear command names like configure help in understanding and learning to use the cluster shell, but is easy to misspell and is tedious to type repeatedly. The interactive mode and tab completion both help with this, but the crm shell also has the ability to understand a variety of shorthand aliases for all of the commands. For example, instead of typing crm status, you can type crm st or crm stat. Instead of crm configure you can type crm cfg or even crm cf. crm resource can be shorted as crm rsc, and so on. The exact list of accepted aliases is too long to print in full, but experimentation and typos should help in discovering more of them. ## FEATURES The feature set of crmsh covers a wide range of functionality, and understanding how and when to use the various features of the shell can be difficult. This section of the guide describes some of the features and use cases of crmsh in more depth. The intention is to provide a deeper understanding of these features, but also to serve as a guide to using them. ### Shadow CIB usage A Shadow CIB is a normal cluster configuration stored in a file. They may be manipulated in much the same way as the *live* CIB, with the key difference that changes to a shadow CIB have no effect on the actual cluster resources. An administrator may choose to apply any of them to the cluster, thus replacing the running configuration with the one found in the shadow CIB. The crm prompt always contains the name of the configuration which is currently in use, or the string *live* if using the live cluster configuration. When editing the configuration in the configure level, no changes are actually applied until the commit command is executed. It is possible to start editing a configuration as usual, but instead of committing the changes to the active CIB, save them to a shadow CIB. The following example configure session demonstrates how this can be done: ``` crm(live)configure# cib new test-2 INFO: test-2 shadow CIB created crm(test-2)configure# commit ``` ### Configuration semantic checks Resource definitions may be checked against the meta-data provided with the resource agents. These checks are currently carried out: • are required parameters set • existence of defined parameters • timeout values for operations The parameter checks are obvious and need no further explanation. Failures in these checks are treated as configuration errors. The timeouts for operations should be at least as long as those recommended in the meta-data. Too short timeout values are a common mistake in cluster configurations and, even worse, they often slip through if cluster testing was not thorough. Though operation timeouts issues are treated as warnings, make sure that the timeouts are usable in your environment. Note also that the values given are just *advisory minimum*---your resources may require longer timeouts. User may tune the frequency of checks and the treatment of errors by the check-frequency and check-mode preferences. Note that if the check-frequency is set to always and the check-mode to strict, errors are not tolerated and such configuration cannot be saved. ### Configuration templates **Deprecation note** Configuration templates have been deprecated in favor of the more capable cluster scripts. To learn how to use cluster scripts, see the dedicated documentation on the crmsh website at http://crmsh.github.io/, or in the Script section. Configuration templates are ready made configurations created by cluster experts. They are designed in such a way so that users may generate valid cluster configurations with minimum effort. If you are new to Pacemaker, templates may be the best way to start. We will show here how to create a simple yet functional Apache configuration: ``` # crm configure crm(live)configure# template crm(live)configure template# list templates apache filesystem virtual-ip crm(live)configure template# new web apache filesystem virtual-ip crm(live)configure template# new web apache INFO: pulling in template apache INFO: pulling in template virtual-ip crm(live)configure template# list web2-d web2 vip2 web3 vip web ``` We enter the template level from configure. Use the list command to show templates available on the system. The new command creates a configuration from the apache template. You can use tab completion to pick templates. Note that the apache template depends on a virtual IP address which is automatically pulled along. The list command shows the just created web configuration, among other configurations (I hope that you, unlike me, will use more sensible and descriptive names). The show command, which displays the resulting configuration, may be used to get an idea about the minimum required changes which have to be done. All ERROR messages show the line numbers in which the respective parameters are to be defined: ``` crm(live)configure template# show ERROR: 23: required parameter ip not set ERROR: 61: required parameter id not set ERROR: 65: required parameter configfile not set crm(live)configure template# edit ``` The edit command invokes the preferred text editor with the web configuration. At the top of the file, the user is advised how to make changes. A good template should require from the user to specify only parameters. For example, the web configuration we created above has the following required and optional parameters (all parameter lines start with %%): ``` $ grep -n ^%% ~/.crmconf/web 23:%% ip 31:%% netmask 35:%% lvs_support 61:%% id 65:%% configfile 71:%% options 76:%% envfiles ``` These lines are the only ones that should be modified. Simply append the parameter value at the end of the line. For instance, after editing this template, the result could look like this (we used tabs instead of spaces to make the values stand out): ``` $ grep -n ^%% ~/.crmconf/web 23:%% ip 192.168.1.101 31:%% netmask 35:%% lvs_support 61:%% id websvc 65:%% configfile /etc/apache2/httpd.conf 71:%% options 76:%% envfiles ``` As you can see, the parameter line format is very simple: ``` %% ``` After editing the file, use show again to display the configuration: ``` crm(live)configure template# show primitive virtual-ip IPaddr \ params ip=192.168.1.101 primitive apache apache \ params configfile="/etc/apache2/httpd.conf" monitor apache 120s:60s group websvc \ apache virtual-ip ``` The target resource of the apache template is a group which we named websvc in this sample session. This configuration looks exactly as you could type it at the configure level. The point of templates is to save you some typing. It is important, however, to understand the configuration produced. Finally, the configuration may be applied to the current crm configuration (note how the configuration changed slightly, though it is still equivalent, after being digested at the configure level): ``` crm(live)configure template# apply crm(live)configure template# cd .. crm(live)configure# show node xen-b node xen-c primitive apache apache \ params configfile="/etc/apache2/httpd.conf" \ op monitor interval=120s timeout=60s primitive virtual-ip IPaddr \ params ip=192.168.1.101 group websvc apache virtual-ip ``` Note that this still does not commit the configuration to the CIB which is used in the shell, either the running one (live) or some shadow CIB. For that you still need to execute the commit command. To complete our example, we should also define the preferred node to run the service: ``` crm(live)configure# location websvc-pref websvc 100: xen-b ``` If you are not happy with some resource names which are provided by default, you can rename them now: ``` crm(live)configure# rename virtual-ip intranet-ip crm(live)configure# show node xen-b node xen-c primitive apache apache \ params configfile="/etc/apache2/httpd.conf" \ op monitor interval=120s timeout=60s primitive intranet-ip IPaddr \ params ip=192.168.1.101 group websvc apache intranet-ip location websvc-pref websvc 100: xen-b ``` To summarize, working with templates typically consists of the following steps: • new: create a new configuration from templates • edit: define parameters, at least the required ones • show: see if the configuration is valid • apply: apply the configuration to the configure level ### Resource testing The amount of detail in a cluster makes all configurations prone to errors. By far the largest number of issues in a cluster is due to bad resource configuration. The shell can help quickly diagnose such problems. And considerably reduce your keyboard wear. Let’s say that we entered the following configuration: ``` node xen-b node xen-c node xen-d primitive fencer stonith:external/libvirt \ params hypervisor_uri="qemu+tcp://10.2.13.1/system" \ hostlist="xen-b xen-c xen-d" \ op monitor interval=2h primitive svc Xinetd \ params service=systat \ op monitor interval=30s primitive intranet-ip IPaddr2 \ params ip=10.2.13.100 \ op monitor interval=30s primitive apache apache \ params configfile="/etc/apache2/httpd.conf" \ op monitor interval=120s timeout=60s group websvc apache intranet-ip location websvc-pref websvc 100: xen-b ``` Before typing commit to submit the configuration to the cib we can make sure that all resources are usable on all nodes: ``` crm(live)configure# rsctest websvc svc fencer ``` It is important that resources being tested are not running on any nodes. Otherwise, the rsctest command will refuse to do anything. Of course, if the current configuration resides in a CIB shadow, then a commit is irrelevant. The point being that resources are not running on any node. **Note on stopping all resources** Alternatively to not committing a configuration, it is also possible to tell Pacemaker not to start any resources: ``` crm(live)configure# property stop-all-resources=yes ``` Almost none---resources of class stonith are still started. But shell is not as strict when it comes to stonith resources. Order of resources is significant insofar that a resource depends on all resources to its left. In most configurations, it’s probably practical to test resources in several runs, based on their dependencies. Apart from groups, crm does not interpret constraints and therefore knows nothing about resource dependencies. It also doesn’t know if a resource can run on a node at all in case of an asymmetric cluster. It is up to the user to specify a list of eligible nodes if a resource is not meant to run on every node. ### Access Control Lists (ACL) **Note on ACLs in Pacemaker 1.1.12** The support for ACLs has been revised in Pacemaker version 1.1.12 and up. Depending on which version you are using, the information in this section may no longer be accurate. Look for the acl\_target configuration element for more details on the new syntax. By default, the users from the haclient group have full access to the cluster (or, more precisely, to the CIB). Access control lists allow for finer access control to the cluster. Access control lists consist of an ordered set of access rules. Each rule allows read or write access or denies access completely. Rules are typically combined to produce a specific role. Then, users may be assigned a role. For instance, this is a role which defines a set of rules allowing management of a single resource: ``` role bigdb_admin \ write meta:bigdb:target-role \ write meta:bigdb:is-managed \ write location:bigdb \ read ref:bigdb ``` The first two rules allow modifying the target-role and is-managed meta attributes which effectively enables users in this role to stop/start and manage/unmanage the resource. The constraints write access rule allows moving the resource around. Finally, the user is granted read access to the resource definition. For proper operation of all Pacemaker programs, it is advisable to add the following role to all users: ``` role read_all \ read cib ``` For finer grained read access try with the rules listed in the following role: ``` role basic_read \ read node attribute:uname \ read node attribute:type \ read property \ read status ``` It is however possible that some Pacemaker programs (e.g. ptest) may not function correctly if the whole CIB is not readable. Some of the ACL rules in the examples above are expanded by the shell to XPath specifications. For instance, meta:bigdb:target-role expands to: ``` //primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role'] ``` You can see the expansion by showing XML: ``` crm(live) configure# show xml bigdb_admin ... ``` Many different XPath expressions can have equal meaning. For instance, the following two are equal, but only the first one is going to be recognized as shortcut: ``` //primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role'] //resources/primitive[@id='bigdb']/meta_attributes/nvpair[@name='target-role'] ``` XPath is a powerful language, but you should try to keep your ACL xpaths simple and the builtin shortcuts should be used whenever possible. ### Syntax: Resource sets Using resource sets can be a bit confusing unless one knows the details of the implementation in Pacemaker as well as how to interpret the syntax provided by crmsh. Three different types of resource sets are provided by crmsh, and each one implies different values for the two resource set attributes, sequential and require-all. sequential If false, the resources in the set do not depend on each other internally. Setting sequential to true implies a strict order of dependency within the set. require-all If false, only one resource in the set is required to fulfil the requirements of the set. The set of A, B and C with require-all set to false is be read as "A OR B OR C" when its dependencies are resolved. The three types of resource sets modify the attributes in the following way: 1\. Implicit sets (no brackets). sequential=true, require-all=true 2\. Parenthesis set (( ... )). sequential=false, require-all=true 3\. Bracket set (\[ ... \]). sequential=false, require-all=false To create a set with the properties sequential=true and require-all=false, explicitly set sequential in a bracketed set, \[ A B C sequential=true \]. To create multiple sets with both sequential and require-all set to true, explicitly set sequential in a parenthesis set: A B ( C D sequential=true ). ### Syntax: Attribute list references Attribute lists are used to set attributes and parameters for resources, constraints and property definitions. For example, to set the virtual IP used by an IPAddr2 resource the attribute ip can be set in an attribute list for that resource. Attribute lists can have identifiers that name them, and other resources can reuse the same attribute list by referring to that name using an $id-ref. For example, the following statement defines a simple dummy resource with an attribute list which sets the parameter state to the value 1 and sets the identifier for the attribute list to on-state: ``` primitive dummy-1 Dummy params $id=on-state state=1 ``` To refer to this attribute list from a different resource, refer to the on-state name using an id-ref: ``` primitive dummy-2 Dummy params $id-ref=on-state ``` The resource dummy-2 will now also have the parameter state set to the value 1. ### Syntax: Attribute references In some cases, referencing complete attribute lists is too coarse-grained, for example if two different parameters with different names should have the same value set. Instead of having to copy the value in multiple places, it is possible to create references to individual attributes in attribute lists. To name an attribute in order to be able to refer to it later, prefix the attribute name with a $ character (as seen above with the special names $id and $id-ref: ``` primitive dummy-1 Dummy params $state=1 ``` The identifier state can now be used to refer to this attribute from other primitives, using the @\ syntax: ``` primitive dummy-2 Dummy params @state ``` In some cases, using the attribute name as the identifier doesn’t work due to name clashes. In this case, the syntax $\:\=\ can be used to give the attribute a different identifier: ``` primitive dummy-1 params $dummy-state-on:state=1 primitive dummy-2 params @dummy-state-on ``` ### Syntax: Rule expressions Many of the configuration commands in crmsh now support the use of *rule expressions*, which can influence what attributes apply to a resource or under which conditions a constraint is applied, depending on changing conditions like date, time, the value of attributes and more. Here is an example of a simple rule expression used to apply a a different resource parameter on the node named node1: ``` primitive my_resource Special \ params 2: rule #uname eq node1 interface=eth1 \ params 1: interface=eth0 ``` This primitive resource has two lists of parameters with descending priority. The parameter list with the highest priority is applied first, but only if the rule expressions for that parameter list all apply. In this case, the rule #uname eq node1 limits the parameter list so that it is only applied on node1. Note that rule expressions are not terminated and are immediately followed by the data to which the rule is applied. In this case, the name-value pair interface=eth1. Rule expressions can contain multiple expressions connected using the boolean operator or and and. The full syntax for rule expressions is listed below. Note: pacemaker has deprecated support for multiple top-level rules within a location constraint since version 2.1.8. ``` rules :: rule [id_spec] [$role=] : [rule [id_spec] [$role=] : ...] id_spec :: $id= | $id-ref= score :: | | [-]inf expression :: [ ...] bool_op :: or | and simple_exp :: [type:] | | date type :: | | binary_op :: lt | gt | lte | gte | eq | ne unary_op :: defined | not_defined date_expr :: lt | gt | in start= end= | in start= | spec duration|date_spec :: hours= | monthdays= | weekdays= | yearsdays= | months= | weeks= | years= | weekyears= ``` ## LIFETIME PARAMETER FORMAT Lifetimes can be specified in the ISO 8601 time format or the ISO 8601 duration format. To distinguish between months and minutes, use the PT prefix before specifying minutes. The duration format is one of PnYnMnDTnHnMnS, PnW, P\T\. P = duration. Y = year. M = month. W = week. D = day. T = time. H = hour. M = minute. S = second. Examples: ``` PT5M = 5 minutes later. 3D = 3 days later. PT1H = 1 hour later. ``` The cluster checks lifetimes at an interval defined by the cluster-recheck-interval property (default 15 minutes). ## COMMAND REFERENCE The commands are structured to be compatible with the shell command line. Sometimes, the underlying Pacemaker grammar uses characters that have special meaning in bash, that will need to be quoted. This includes the hash or pound sign (#), single and double quotes, and any significant whitespace. Whitespace is also significant when assigning values, meaning that key=value is different from key = value. Commands can be referenced using short-hand as long as the short-hand is unique. This can be either a prefix of the command name or a prefix string of characters found in the name. For example, status can be abbreviated as st or su, and configure as conf or cfg. The syntax for the commands is given below in an informal, BNF-like grammar. • \ denotes a string. • \[value\] means that the construct is optional. • The ellipsis (...) signifies that the previous construct may be repeated. • first\|second means either first or second. • The rest are literals (strings, :, =). ### status Show cluster status. The status is displayed by crm\_mon. Supply additional arguments for more information or different format. See [crm\_mon(8)](/crm_mon/8.md) for more details. Example: ``` status status simple status full ``` Usage: ``` status [