{"id":104,"date":"2008-01-20T00:00:00","date_gmt":"2008-01-20T00:00:00","guid":{"rendered":"http:\/\/netexpertise\/en\/?p=104"},"modified":"2021-10-15T20:51:02","modified_gmt":"2021-10-15T18:51:02","slug":"heartbeat-2-howto","status":"publish","type":"post","link":"http:\/\/www.netexpertise.eu\/en\/systems\/linux\/heartbeat-2-howto.html","title":{"rendered":"Heartbeat 2 Howto"},"content":{"rendered":"<div style=\"float: left; margin-right: 15px; margin-top: 0px;\"><script async=\"\" src=\"\/\/pagead2.googlesyndication.com\/pagead\/js\/adsbygoogle.js\"><\/script><br \/>\n<!-- 200x200, Netexpertise --><br \/>\n<ins class=\"adsbygoogle\" style=\"display: inline-block; width: 200px; height: 200px;\" data-ad-client=\"ca-pub-6495750100906580\" data-ad-slot=\"1946825373\"><\/ins><br \/>\n<script><br \/>\n(adsbygoogle = window.adsbygoogle || []).push({});<br \/>\n<\/script><\/div>\n<p>Important note:<br \/>\n<a href=\"http:\/\/www.linux-ha.org\/wiki\/Site_news\">Heartbeat<\/a> is now obsolete and has moved to a new stack available on <a href=\"https:\/\/clusterlabs.org\">Clusterlabs<\/a>. For simple high availability project using a virtual IP, try out keepalived that does monitoring and failover with just a simple configuration file.<\/p>\n<p>Since version 2, Heartbeat is able to manage more than 2 nodes, and doesn&#8217;t need &#8220;<a href=\"http:\/\/www.kernel.org\/software\/mon\/\">mon<\/a>&#8221; utility to monitor services.<br \/>\nThis fonctionnality is now implemented within Heartbeat.<br \/>\nAs a consequence, this flexibility and new features may make it harder to configure.<br \/>\nIt&#8217;s intresting to know version 1 configuration files are still supported.<\/p>\n<h3>Installation<\/h3>\n<p>Heartbeat source files are available from the official site <a href=\"http:\/\/www.linux-ha.org\">http:\/\/www.linux-ha.org<\/a>. Redhat Enterprise compatible rpms can be downloaded from the Centos website (link given from Heartbeat website). Packages are built pretty quickly as source version is 2.1.0 when rpm is 2.0.8-2 at the time this article was made. 3 rpms are needed:<\/p>\n<p>heartbeat-pils<br \/>\nheartbeat-stonith<\/p>\n<p>heartbeat<\/p>\n<p>We are going to monitor Apache but the configuration remains valid for any other service: mail, database, DNS, DHCP, file server, etc&#8230;<\/p>\n<h3>Diagrams<\/h3>\n<p><strong>Failover or load-balancing<\/strong><\/p>\n<p>Heartbeat supports Active-Passive for the failover mode and Active-Active for the load-balancing.<br \/>\nMany other options are possible adding servers and\/or services; There are many many possibilities.<\/p>\n<p>We will focus on load-balancing, only a few lines need to be removed for failover.<\/p>\n<div style=\"text-align: center;\">\n<table style=\"width: 100%;\">\n<tbody>\n<tr>\n<td style=\"width: 20%;\"><\/td>\n<td><img decoding=\"async\" src=\"\/images\/Heartbeat-Failover.png\" alt=\"Failover\" \/><\/td>\n<td style=\"width: 20%;\"><\/td>\n<td><img decoding=\"async\" src=\"\/images\/Heartbeat-Loadbalancing.png\" alt=\"Load-balancing\" \/><\/td>\n<td style=\"width: 20%;\"><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h3>Configuration<\/h3>\n<p>In this setup, the 2 servers are interconnected through their eth0 interface.<br \/>\nApplicative flows come on eth1, that is configured as in the diagram below, with addresses 192.168.0.4 and .5<br \/>\nAddresses on eth0 have to be in a sub-network dedicated to Heartbeat.<br \/>\nThese addresses must appear in \/etc\/hosts on every node.<\/p>\n<p>3 files must be configured for Heartbeat 2. Again, they are identical on each node.<\/p>\n<p>In \/etc\/ha.d\/<\/p>\n<ul>\n<li>ha.cf<\/li>\n<li>authkeys<\/li>\n<\/ul>\n<p>In \/var\/lib\/heartbeat\/crm\/<\/p>\n<ul>\n<li>cib.xml<\/li>\n<\/ul>\n<p><strong>ha.cf<\/strong><\/p>\n<p>ha.cf contains the main settings like cluster nodes or the communication topology.<\/p>\n<pre>use_logd on\n<em style=\"color: #ff0000;\"># Heartbeat packets sending frequency<\/em>\nkeepalive 500ms <em style=\"color: #ff0000;\"># seconds - Specify ms for times shorter than 1 second<\/em>\n\n<em style=\"color: #ff0000;\"># Period of time after which a node is declared \"dead\"<\/em>\ndeadtime 2\n<em style=\"color: #ff0000;\"># Send a warning message<\/em>\n<em style=\"color: #ff0000;\"># Important to adjust the deadtime value<\/em>\nwarntime 1\n<em style=\"color: #ff0000;\"># Identical to deadtime but when initializing<\/em>\ninitdead 8\nupdport 694\n\n<em style=\"color: #ff0000;\"># Host to be tested to check the node is still online<\/em>\n<em style=\"color: #ff0000;\"># The default gateway most likely<\/em>\nping 192.168.0.1\n<em style=\"color: #ff0000;\"># Interface used for Heartbeat packets<\/em>\n<em style=\"color: #ff0000;\"># Use the serial port,<\/em>\n\n<em style=\"color: #ff0000;\"># \"mcast\" and \"ucast\" for multicast and unicast<\/em>\nbcast eth0\n<em style=\"color: #ff0000;\"># Resource switches back on the primary node when it's available<\/em>\nauto_failback on\n<em style=\"color: #ff0000;\"># Cluster Nodes List<\/em>\nnode n1.domain.com\nnode n2.domain.com\n<em style=\"color: #ff0000;\"># Activate Heartbeat 2 Configuration<\/em>\n\ncrm yes\n<em style=\"color: #ff0000;\"># Allow to add dynamically a new node to the cluster<\/em>\nautojoin any\n<\/pre>\n<p>Other options are available such as compression or bandwidth for communication on a serial cable. Check <a href=\"http:\/\/linux-ha.org\/ha.cf\">http:\/\/linux-ha.org\/ha.cf<\/a><\/p>\n<p><strong>authkeys<\/strong><\/p>\n<p>Authkeys defines authentication keys.<br \/>\nSeveral types are available: crc, md5 and sha1. crc is to be used on a secured sub-network (vlan isolation or cross-over cable.<br \/>\nsha1 offers a higher level of secuirity but can consume a lot CPU resources.<br \/>\nmd5 sits in between. It&#8217;s not a bad choice.<\/p>\n<pre>auth 1\n1 md5 secret\n<\/pre>\n<p>Password is stored in clear text, it is important to change the file permissions to 600.<\/p>\n<p><strong>cib.xml<\/strong><\/p>\n<pre>&lt;cib&gt;\n&lt;configuration&gt;\n\n  &lt;crm_config\/&gt;\n  &lt;nodes\/&gt;\n  &lt;resources&gt;\n    &lt;group id=\"server1\"&gt;\n\n      &lt;primitive class=\"ocf\" id=\"IP1\" provider=\"heartbeat\" type=\"IPaddr\"&gt;\n        &lt;operations&gt;\n          &lt;op id=\"IP1_mon\" interval=\"10s\" name=\"monitor\" timeout=\"5s\"\/&gt;\n        &lt;\/operations&gt;\n        &lt;instance_attributes id=\"IP1_inst_attr\"&gt;\n          &lt;attributes&gt;\n            &lt;nvpair id=\"IP1_attr_0\" name=\"ip\" value=\"192.168.0.2\"\/&gt;\n            &lt;nvpair id=\"IP1_attr_1\" name=\"netmask\" value=\"24\"\/&gt;\n            &lt;nvpair id=\"IP1_attr_2\" name=\"nic\" value=\"eth1\"\/&gt;\n          &lt;\/attributes&gt;\n        &lt;\/instance_attributes&gt;\n      &lt;\/primitive&gt;\n\n      &lt;primitive class=\"lsb\" id=\"apache1\" provider=\"heartbeat\" type=\"apache\"&gt;\n        &lt;operations&gt;\n          &lt;op id=\"jboss1_mon\" interval=\"30s\" name=\"monitor\" timeout=\"20s\"\/&gt;\n        &lt;\/operations&gt;\n      &lt;\/primitive&gt;\n    &lt;\/group&gt;\n\n    &lt;group id=\"server2\"&gt;\n\n      &lt;primitive class=\"ocf\" id=\"IP2\" provider=\"heartbeat\" type=\"IPaddr\"&gt;\n        &lt;operations&gt;\n          &lt;op id=\"IP2_mon\" interval=\"10s\" name=\"monitor\" timeout=\"5s\"\/&gt;\n        &lt;\/operations&gt;\n        &lt;instance_attributes id=\"IP2_inst_attr\"&gt;\n          &lt;attributes&gt;\n            &lt;nvpair id=\"IP2_attr_0\" name=\"ip\" value=\"192.168.0.3\"\/&gt;\n            &lt;nvpair id=\"IP2_attr_1\" name=\"netmask\" value=\"24\"\/&gt;\n            &lt;nvpair id=\"IP2_attr_2\" name=\"nic\" value=\"eth1\"\/&gt;\n          &lt;\/attributes&gt;\n        &lt;\/instance_attributes&gt;\n      &lt;\/primitive&gt;\n\n      &lt;primitive class=\"lsb\" id=\"apache2\" provider=\"heartbeat\" type=\"apache\"&gt;\n        &lt;operations&gt;\n          &lt;op id=\"jboss2_mon\" interval=\"30s\" name=\"monitor\" timeout=\"20s\"\/&gt;\n        &lt;\/operations&gt;\n      &lt;\/primitive&gt;\n    &lt;\/group&gt;\n  &lt;\/resources&gt;\n\n  &lt;constraints&gt;\n    &lt;rsc_location id=\"location_server1\" rsc=\"server1\"&gt;\n      &lt;rule id=\"best_location_server1\" score=\"100\"&gt;\n        &lt;expression_attribute=\"#uname\" id=\"best_location_server1_expr\" operation=\"eq\"\n        value=\"n1.domain.com\"\/&gt;\n      &lt;\/rule&gt;\n    &lt;\/rsc_location&gt;\n\n    &lt;rsc_location id=\"location_server2\" rsc=\"server2\"&gt;\n      &lt;rule id=\"best_location_server2\" score=\"100\"&gt;\n        &lt;expression_attribute=\"#uname\" id=\"best_location_server2_expr\" operation=\"eq\"\n        value=\"n2.domain.com\"\/&gt;\n      &lt;\/rule&gt;\n    &lt;\/rsc_location&gt;\n\n    &lt;rsc_location id=\"server1_connected\" rsc=\"server1\"&gt;\n      &lt;rule id=\"server1_connected_rule\" score=\"-INFINITY\" boolean_op=\"or\"&gt;\n        &lt;expression id=\"server1_connected_undefined\" attribute=\"pingd\"\n        operation=\"not_defined\"\/&gt;\n        &lt;expression id=\"server1_connected_zero\" attribute=\"pingd\" operation=\"lte\"\n        value=\"0\"\/&gt;\n      &lt;\/rule&gt;\n    &lt;\/rsc_location&gt;\n\n    &lt;rsc_location id=\"server2_connected\" rsc=\"server2\"&gt;\n      &lt;rule id=\"server2_connected_rule\" score=\"-INFINITY\" boolean_op=\"or\"&gt;\n        &lt;expression id=\"server2_connected_undefined\" attribute=\"pingd\"\n        operation=\"not_defined\"\/&gt;\n        &lt;expression id=\"server2_connected_zero\" attribute=\"pingd\" operation=\"lte\"\n        value=\"0\"\/&gt;\n      &lt;\/rule&gt;\n    &lt;\/rsc_location&gt;\n  &lt;\/constraints&gt;\n\n&lt;\/configuration&gt;\n&lt;\/cib&gt;\n<\/pre>\n<p>It&#8217;s possible to generate the file from a Heartbeat 1 configuration file, haresources located in \/etc\/ha.d\/, with the following command:<br \/>\npython \/usr\/lib\/heartbeat\/haresources2cib.py &gt; \/var\/lib\/heartbeat\/crm\/cib.xml<\/p>\n<p>The file can be split in 2 parts: resources and constraints.<\/p>\n<p><strong>Resources<\/strong><\/p>\n<p>Resources are organized in groups (server1 &amp; 2) putting together a virtual IP address and a service: Apache.<br \/>\nResources are declared with the &lt;primitive&gt; syntax within the group.<br \/>\nGroups are useful to gather several resources under the same constraints.<\/p>\n<p>The IP1 primitive checks virtual IP 1 is reachable.<br \/>\nIt executes OCF type IPaddr script.<br \/>\nOCF scripts are provided with Heartbeat in the rpm packages.<br \/>\nIt&#8217;s also possible to specify the virtual address, the network mask as well as the interface.<br \/>\nApache resource type is LSB, meaning it makes a call to a startup script located in the usual \/etc\/init.d.<br \/>\nThe script&#8217;s name in the variable type: type=&#8221;name&#8221;.<br \/>\nIn order to run with Heartbeat, the script must be LSB compliant. LSB compliant means the script must:<\/p>\n<ul>\n<li>return its status with &#8220;script status&#8221;<\/li>\n<li>not fail &#8220;script start&#8221; on a service that is already running<\/li>\n<li>not fail stopping a service already stopped<\/li>\n<\/ul>\n<p>All LSB specifications can be checked at <a href=\"http:\/\/www.linux-ha.org\/LSBResourceAgent\">http:\/\/www.linux- ha.org\/LSBResourceAgent<\/a><\/p>\n<p>The following time values can be defined:<\/p>\n<ul>\n<li>interval: defines how often the resource&#8217;s status is checked<\/li>\n<li>timeout: defines the time period before considering a start, stop or status action failed<\/li>\n<\/ul>\n<p><strong>Constraints<\/strong><\/p>\n<p>2 constraints apply to each group of resources:<br \/>\nThe favourite location where resources &#8220;should&#8221; run.<br \/>\nWe give a score of 100 to n1.domain.com for the 1st group.<br \/>\nHence, if n1.domain.com is active, and option auto_failback is set to &#8220;on&#8221;, resources in this group will always come back there.<\/p>\n<p>Action depending on ping result. If none of the gateways answer ping packets, resources move on another server, and the node goes to standby status.<\/p>\n<p>Score -INFINITY means the the node will never ever accept these resources if the gateways are unreachable.<\/p>\n<h3>Important Notes<\/h3>\n<p><strong>Files Rights<\/strong><\/p>\n<p>\/etc\/ha.cf directory contains sensible data. Rights have to be changed to be accessed by the owner only &#8211; or the application will not launch.<\/p>\n<p>chmod 600 \/etc\/ha.d\/ha.cf<\/p>\n<p>The \/var\/lib\/heartbeat\/crm\/cib.crm file has to belong to hacluster and group haclient. It must not be accessible by other users.<br \/>\nchown hacluster:haclient \/var\/lib\/heartbeat\/crm\/cib.crm<br \/>\nchmod 660 \/var\/lib\/heartbeat\/crm\/cib.crm<\/p>\n<p>cib.xml is accessed in write mode by Heartbeat. Some files are created at the same time.<br \/>\nIf you need to edit it manually, stop Heartbeat on all nodes, remove cib.xml.sig in the same directory, edit cib.xml on all nodes, and restart Heartbeat.<\/p>\n<p>It&#8217;s advised to use crm_resource to bring modifications (see section below).<\/p>\n<p><strong>hosts files<\/strong><\/p>\n<p>The \/etc\/hosts file must contain all the cluster nodes hostnames. These names should be identical to the result of &#8216;uname -n&#8217; command.<\/p>\n<p><strong>Services Startup<\/strong><\/p>\n<p>Heartbeat starts Apache when the service is inactive. It is better to disable Apache automatic startup with chkconfig for instance, as it may take a while to boot, Heartbeat could start it a second time.<\/p>\n<p>chkconfig httpd off<\/p>\n<p>There is a startup script in \/etc\/init.d\/. To launch Heartbeat, run &#8220;\/etc\/init.d\/hearbeat start&#8221;. Yu can launch Heartbeat automatically at boot time:<br \/>\nchkconfig heartbeat on<\/p>\n<h3>Behaviour and Tests<\/h3>\n<p>The following actions simulate incidents that may occur at some stage, and alter the cluster&#8217;s health. We study Heartbeat&#8217;s behaviour in each case.<\/p>\n<p><strong>Apache shutdown<\/strong><\/p>\n<p>Local Heartbeat will restart Apache. If the reboot fails, a warning is sent to the logs. Heartbeat won&#8217;t launch the service anymore then. The virtual IP address remains on the node. However, the address can be moved manually with Heartbeat tools.<\/p>\n<p><strong>Server&#8217;s Shutdown or Crash<\/strong><\/p>\n<p>Virtual addresses move to other servers.<\/p>\n<p><strong>Heartbeat&#8217;s Manual Shutdown on one Node<\/strong><\/p>\n<p>Heartbeat stops Heartbeat. Virtual addresses are moved to other nodes. The normal procedure is to run crm_standby to turn the node into standby mide and migrate resources accross.<\/p>\n<p><strong>Node to node cable disconnection<\/strong><\/p>\n<p>Each machine thinks it&#8217;s on its own and takes the 2 virtual IPs. However, it is not really a problem as the gateway keeps on sending packets to the last entry in the ARP table. A ping on the broadcast address sends duplicates.<\/p>\n<p><strong>Gateway disconnection<\/strong><\/p>\n<p>This could happen in 2 cases:<br \/>\n&#8211; The gateway is unreachable. In this case, the 2 nodes remove their virtual IP address and stop Apache.<br \/>\n&#8211; The connection to one of the nodes is down. Addresses move to the other server which can normally reach the gateway.<\/p>\n<p>All simulations allow to maintain the cluster up except if the gateway is gone. But this is not a cluster problem&#8230;<\/p>\n<h3>Tools<\/h3>\n<p>Heartbeat&#8217;s health can be checked via different ways.<\/p>\n<p><strong>Logs<\/strong><\/p>\n<p>Heartbeat sends messages to the logd daemon that stores them in the \/var\/log\/messages system file.<\/p>\n<p><strong>Unix Tools<\/strong><\/p>\n<p>Usual Unix commands can be used. Heartbeat creates sub-interfaces that you can check with &#8220;ifconfig&#8221; or &#8220;ip address show&#8221;. Processes status can be displayed with the startup scripts or with the &#8220;ps&#8221; command for instance.<\/p>\n<p><strong>Heartbeat Commands<\/strong><\/p>\n<p>Heartbeat is provided with a set of commands. Here are the main ones:<\/p>\n<ul>\n<li>crmadmin: Controls nodes managers on each machine of the cluster.<\/li>\n<li>crm_mon: Quick and useful; Displays nodes status and resources.<\/li>\n<li>crm_resource: Makes requests and modifies resources\/services related data. Ability to list, migrate, disable or delete resources.<\/li>\n<li>crm_verify: Reports the cluster&#8217;s warnings and errors<\/li>\n<li>crm_standby: Migrate all node&#8217;s resources. Useful when upgrading.<\/li>\n<\/ul>\n<h3>Maintenance<\/h3>\n<p><strong>Service Shutdown<\/strong><\/p>\n<p>When upgrading or shutting Apache down, you may proceed as follow:<\/p>\n<ul>\n<li>Stop Apache on node 1 and migrate the virtual address on node 2:<br \/>\ncrm_standby -U n1.domain.com -v true<\/li>\n<li>Start upgrading n1 and restart the node:<br \/>\ncrm_standby -U n1.domain.com -v false<\/li>\n<li>Resources automatically fail back to n1 (if cib.xml is properly configured)<\/li>\n<\/ul>\n<p>Proceed the same way on n2<\/p>\n<p>Rem: The previous commands can be run from any node of the cluster.<\/p>\n<p>It is possible to switch the 2 nodes on standby at the same time; Apache will be stopped on the 2 machines and the virtual addresses deleted until it comes back to running mode.<\/p>\n<p><strong>Machine Reboot<\/strong><\/p>\n<p>It is not required to switch the node to standby mode before rebooting. However, it is better practice as resources migrate faster, detection time being inexistant.<\/p>\n<p><strong>Resource Failure<\/strong><\/p>\n<p>Example: Apache crashed and doesn&#8217;t restart anymore, all group resources are moved to the 2nd node.<\/p>\n<p>When the issue is resolved and the resource is up again, the cluster&#8217;s health can be checked with:<br \/>\ncrm_resource &#8211;reprobe (-P)<\/p>\n<p>and the resource restarted:<br \/>\ncrm_resource &#8211;cleanup &#8211;resource apache1 (ou crm_resource -C -r apache1).<br \/>\nIt will move automatically back to the original server.<\/p>\n<h3>Adding a New Node to the cluster<\/h3>\n<p><strong>In Standby<\/strong><\/p>\n<p>If the cluster contains 2 nodes connected with a cross-over cable, you then will need a switch for the heartbeats network interfaces.<\/p>\n<p>You first need to add new node&#8217;s informations. Edit the \/etc\/hosts files on the current nodes and add the node&#8217;s hostname. \/etc\/hosts contents must be copied accross to the new node. Configure Heartbeat in the same way as other nodes.<br \/>\nha.cf files must contain the &#8220;autojoin any&#8221; setting to accept new nodes on the fly.<\/p>\n<p>On the new host, start Heartbeat; The should join the cluster automatically.<\/p>\n<p>If not, run the following on one of the 1st nodes:<br \/>\n\/usr\/lib\/heartbeat\/hb_addnode n3.domain.com<br \/>\nThe new node only acts as a failover, no service is associated. If n1.domain.com goes in standby, resources will move to n3. They will come back to the original server as soon as it comes back up (n1 being the favourite server).<\/p>\n<p><strong>With a New Service<\/strong><\/p>\n<p>To add a 3rd IP (along with a 3rd Apache), you need to follow the above procedure and then:<br \/>\n&#8211; Either stop Heartbeat on the 3 servers and edit cib.xml files<br \/>\n&#8211; Or build similar files to cib.xml, containing new resources and constraints to be added, and add them to the cluster in live. This is the preferred method. Create the following files on one of the nodes:<\/p>\n<p>newGroup.xml<\/p>\n<pre>&lt;group id=\"server3\"&gt;\n  &lt;primitive class=\"ocf\" provider=\"heartbeat\" type=\"IPaddr\" id=\"IP3\"&gt;\n    &lt;operations&gt;\n      &lt;op id=\"IP3_mon\" interval=\"5s\" name=\"monitor\" timeout=\"2s\"\/&gt;\n    &lt;\/operations&gt;\n    &lt;instance_attributes id=\"IP3_attr\"&gt;\n      &lt;attributes&gt;\n        &lt;nvpair id=\"IP3_attr_0\" name=\"ip\" value=\"192.168.0.28\"\/&gt;\n        &lt;nvpair id=\"IP3_attr_1\" name=\"netmask\" value=\"29\"\/&gt;\n        &lt;nvpair id=\"IP3_attr_2\" name=\"nic\" value=\"eth1\"\/&gt;\n      &lt;\/attributes&gt;\n    &lt;\/instance_attributes&gt;\n  &lt;\/primitive&gt;\n\n  &lt;primitive class=\"lsb\" provider=\"heartbeat\" type=\"apache\" id=\"apache3\"&gt;\n    &lt;operations&gt;\n      &lt;op id=\"apache3_mon\" interval=\"5s\" name=\"monitor\" timeout=\"5s\"\/&gt;\n    &lt;\/operations&gt;\n  &lt;\/primitive&gt;\n&lt;\/group&gt;\n<\/pre>\n<p>newLocationConstraint.xml<\/p>\n<pre>&lt;rsc_location id=\"location_server3\" rsc=\"server3\"&gt;\n  &lt;rule id=\"best_location_server3\" score=\"100\"&gt;\n    &lt;expression attribute=\"#uname\" id=\"best_location_server3_expr\" operation=\"eq\"\n    value=\"n3.domain.com\"\/&gt;\n  &lt;\/rule&gt;\n&lt;\/rsc_location&gt;\n<\/pre>\n<p>newPingConstraint.xml<\/p>\n<pre>&lt;rsc_location id=\"server3_connected\" rsc=\"server3\"&gt;\n  &lt;rule id=\"server3_connected_rule\" score=\"-INFINITY\" boolean_op=\"or\"&gt;\n    &lt;expression id=\"server3_connected_undefined\" attribute=\"pingd\" \n    operation=\"not_defined\"\/&gt;\n    &lt;expression id=\"server3_connected_zero\" attribute=\"pingd\" operation=\"lte\" \n    value=\"0\"\/&gt;\n  &lt;\/rule&gt;\n&lt;\/rsc_location&gt;\n<\/pre>\n<p>Add n3 constraints<br \/>\ncibadmin -C -o constraints -x newLocationConstraint.xml<br \/>\ncibadmin -C -o constraints -x newPingConstraint.xml<\/p>\n<p>Add n3 resources<\/p>\n<p>cibadmin -C -o resources -x newGroup.xml<\/p>\n<p>n3 resources should start right away.<\/p>\n<p>Note: Adding constraints can be done one at a time. If you try to add 2 constraints from within the same file, only the first will be set.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Important note: Heartbeat is now obsolete and has moved to a new stack available on Clusterlabs. For simple high availability project using a virtual IP, try out keepalived that does monitoring and failover with just a simple configuration file. Since version 2, Heartbeat is able to manage more than 2 nodes, and doesn&#8217;t need &#8220;mon&#8221; [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0},"categories":[11],"tags":[51,52,386],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v21.8.1 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Netexpertise - Heartbeat 2 Howto<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"http:\/\/www.netexpertise.eu\/en\/systems\/linux\/heartbeat-2-howto.html\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Netexpertise - Heartbeat 2 Howto\" \/>\n<meta property=\"og:description\" content=\"Important note: Heartbeat is now obsolete and has moved to a new stack available on Clusterlabs. For simple high availability project using a virtual IP, try out keepalived that does monitoring and failover with just a simple configuration file. Since version 2, Heartbeat is able to manage more than 2 nodes, and doesn&#8217;t need &#8220;mon&#8221; [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"http:\/\/www.netexpertise.eu\/en\/systems\/linux\/heartbeat-2-howto.html\" \/>\n<meta property=\"og:site_name\" content=\"Netexpertise\" \/>\n<meta property=\"article:published_time\" content=\"2008-01-20T00:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-10-15T18:51:02+00:00\" \/>\n<meta name=\"author\" content=\"dave\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@netexpertise\" \/>\n<meta name=\"twitter:site\" content=\"@netexpertise\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"http:\/\/www.netexpertise.eu\/en\/systems\/linux\/heartbeat-2-howto.html\",\"url\":\"http:\/\/www.netexpertise.eu\/en\/systems\/linux\/heartbeat-2-howto.html\",\"name\":\"Netexpertise - Heartbeat 2 Howto\",\"isPartOf\":{\"@id\":\"http:\/\/www.netexpertise.eu\/en\/#website\"},\"datePublished\":\"2008-01-20T00:00:00+00:00\",\"dateModified\":\"2021-10-15T18:51:02+00:00\",\"author\":{\"@id\":\"http:\/\/www.netexpertise.eu\/en\/#\/schema\/person\/cb4cd666549d22e9070ec1cfc1a496fa\"},\"breadcrumb\":{\"@id\":\"http:\/\/www.netexpertise.eu\/en\/systems\/linux\/heartbeat-2-howto.html#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"http:\/\/www.netexpertise.eu\/en\/systems\/linux\/heartbeat-2-howto.html\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"http:\/\/www.netexpertise.eu\/en\/systems\/linux\/heartbeat-2-howto.html#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"http:\/\/www.netexpertise.eu\/en\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Heartbeat 2 Howto\"}]},{\"@type\":\"WebSite\",\"@id\":\"http:\/\/www.netexpertise.eu\/en\/#website\",\"url\":\"http:\/\/www.netexpertise.eu\/en\/\",\"name\":\"Netexpertise\",\"description\":\"Systems \/ Networks \/ DevOps\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"http:\/\/www.netexpertise.eu\/en\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"http:\/\/www.netexpertise.eu\/en\/#\/schema\/person\/cb4cd666549d22e9070ec1cfc1a496fa\",\"name\":\"dave\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"http:\/\/www.netexpertise.eu\/en\/#\/schema\/person\/image\/\",\"url\":\"http:\/\/1.gravatar.com\/avatar\/1129916e1f4955bd632f27f836f64e55?s=96&d=mm&r=g\",\"contentUrl\":\"http:\/\/1.gravatar.com\/avatar\/1129916e1f4955bd632f27f836f64e55?s=96&d=mm&r=g\",\"caption\":\"dave\"},\"sameAs\":[\"http:\/\/www.netexpertise.eu\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Netexpertise - Heartbeat 2 Howto","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"http:\/\/www.netexpertise.eu\/en\/systems\/linux\/heartbeat-2-howto.html","og_locale":"en_US","og_type":"article","og_title":"Netexpertise - Heartbeat 2 Howto","og_description":"Important note: Heartbeat is now obsolete and has moved to a new stack available on Clusterlabs. For simple high availability project using a virtual IP, try out keepalived that does monitoring and failover with just a simple configuration file. Since version 2, Heartbeat is able to manage more than 2 nodes, and doesn&#8217;t need &#8220;mon&#8221; [&hellip;]","og_url":"http:\/\/www.netexpertise.eu\/en\/systems\/linux\/heartbeat-2-howto.html","og_site_name":"Netexpertise","article_published_time":"2008-01-20T00:00:00+00:00","article_modified_time":"2021-10-15T18:51:02+00:00","author":"dave","twitter_card":"summary_large_image","twitter_creator":"@netexpertise","twitter_site":"@netexpertise","schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"http:\/\/www.netexpertise.eu\/en\/systems\/linux\/heartbeat-2-howto.html","url":"http:\/\/www.netexpertise.eu\/en\/systems\/linux\/heartbeat-2-howto.html","name":"Netexpertise - Heartbeat 2 Howto","isPartOf":{"@id":"http:\/\/www.netexpertise.eu\/en\/#website"},"datePublished":"2008-01-20T00:00:00+00:00","dateModified":"2021-10-15T18:51:02+00:00","author":{"@id":"http:\/\/www.netexpertise.eu\/en\/#\/schema\/person\/cb4cd666549d22e9070ec1cfc1a496fa"},"breadcrumb":{"@id":"http:\/\/www.netexpertise.eu\/en\/systems\/linux\/heartbeat-2-howto.html#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["http:\/\/www.netexpertise.eu\/en\/systems\/linux\/heartbeat-2-howto.html"]}]},{"@type":"BreadcrumbList","@id":"http:\/\/www.netexpertise.eu\/en\/systems\/linux\/heartbeat-2-howto.html#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"http:\/\/www.netexpertise.eu\/en"},{"@type":"ListItem","position":2,"name":"Heartbeat 2 Howto"}]},{"@type":"WebSite","@id":"http:\/\/www.netexpertise.eu\/en\/#website","url":"http:\/\/www.netexpertise.eu\/en\/","name":"Netexpertise","description":"Systems \/ Networks \/ DevOps","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"http:\/\/www.netexpertise.eu\/en\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"http:\/\/www.netexpertise.eu\/en\/#\/schema\/person\/cb4cd666549d22e9070ec1cfc1a496fa","name":"dave","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"http:\/\/www.netexpertise.eu\/en\/#\/schema\/person\/image\/","url":"http:\/\/1.gravatar.com\/avatar\/1129916e1f4955bd632f27f836f64e55?s=96&d=mm&r=g","contentUrl":"http:\/\/1.gravatar.com\/avatar\/1129916e1f4955bd632f27f836f64e55?s=96&d=mm&r=g","caption":"dave"},"sameAs":["http:\/\/www.netexpertise.eu"]}]}},"_links":{"self":[{"href":"http:\/\/www.netexpertise.eu\/en\/wp-json\/wp\/v2\/posts\/104"}],"collection":[{"href":"http:\/\/www.netexpertise.eu\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.netexpertise.eu\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.netexpertise.eu\/en\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/www.netexpertise.eu\/en\/wp-json\/wp\/v2\/comments?post=104"}],"version-history":[{"count":0,"href":"http:\/\/www.netexpertise.eu\/en\/wp-json\/wp\/v2\/posts\/104\/revisions"}],"wp:attachment":[{"href":"http:\/\/www.netexpertise.eu\/en\/wp-json\/wp\/v2\/media?parent=104"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.netexpertise.eu\/en\/wp-json\/wp\/v2\/categories?post=104"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.netexpertise.eu\/en\/wp-json\/wp\/v2\/tags?post=104"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}