Red Hat Cluster Part 8: Convert HA Cluster from Active/Passive to Active/Active

Recently, we had to convert an HA Cluster from Active/Passive  to Active/Active mode. So I thought of making a blog post for future reference. The scenario is as follows:

• Operating System: CentOS 7.7
• Cluster Mode: 2 Node Active/Passive
• Cluster Manager: Pacemaker/Corosync
• Shared Storage LVM: HA-LVM

Since the shared storage was configured as LVM resource using HA-LVM configuration, we had to reconfigure LVM to be managed by DLM, CLVMD and formatted as GFS2.

Remove HA-LVM Configuration

The first step in this process was to remove the HA-LVM configuration done previously. The steps used to configure HA-LVM are as follows.

While configuring HA-LVM on both nodes we ensure that the locking_type is set to 1 and use_lvmetad is set to 0 in the file /etc/lvm/lvm.conf  by issuing the following command disabling lvmetad processes.

 # lvmconf --enable-halvm --services --startstopservices  

Then in the same file in the volume_list we list all the Volume Group names that are not controlled by the Cluster but by the OS as given below on both nodes.

 volume_list = [ "centos_root", "centos_home" ]  

We rebuilt the initramfs boot image and rebooted both the nodes.

Now, we undo the above settings on both nodes.

1. Set use_lvmetad to 0 and locking_type to 3 in the file /etc/lvm/lvm.conf and stop lvm2-lvmetad service.

[root@node01 ~]# cat /etc/lvm/lvm.conf | grep use_lvmetad  
      use_lvmetad = 0  

[root@node01 ~]# cat /etc/lvm/lvm.conf | grep locking_type  
      locking_type = 3 

[root@node01 ~]# systemctl stop lvm2-lvmetad.service  

[root@node02 ~] # cat /etc/lvm/lvm.conf | grep use_lvmetad  
      use_lvmetad = 0  

[root@node02 ~]# cat /etc/lvm/lvm.conf | grep locking_type  
      locking_type = 3 

[root@node02 ~] # systemctl stop lvm2-lvmetad.service  

2. Comment the volume_list parameter in the file /etc/lvm/lvm.conf.

[root@node01 ~] # cat /etc/lvm/lvm.conf | grep volume_list  
     # volume_list = [ "centos_root", "centos_home" ]  

[root@node02 ~] # cat /etc/lvm/lvm.conf | grep volume_list  
     # volume_list = [ "centos_root", "centos_home" ]  

3. Rebuild the initramfs boot image and reboot both the nodes.

Before rebuilding the initramfs boot image we take a backup of the initramfs boot image.

[root@node01 ~] # cp -p /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak  

[root@node02 ~] # cp -p /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak  

Now we rebuild the initramfs boot image and reboot the nodes.

[root@node01 ~]# dracut -H -f /boot/initramfs-$(uname -r).img $(uname -r)

[root@node02 ~]# dracut -H -f /boot/initramfs-$(uname -r).img $(uname -r)




[root@node01 ~]# reboot

[root@node02 ~]# reboot




Configure CLVMD for Active/Active Mode

After removing HA-LVM configuration, we removed the previous Logical Volumes, Volume Groups and Physical Volumes that were on the shared storage. Since there was no data, we proceeded with the above step for ease. 

Then we re-created the PV, VG and LV for use in cluster mode and managed by CLVMD.

1. Install required packages.

We install lvm2-cluster and gfs2-utils packages.

 [root@node01 ~]# yum install lvm2-cluster gfs2-utils  

 [root@node02 ~]# yum install lvm2-cluster gfs2-utils  

2. Set no_quorum_policy to freeze in order to prevent fencing of all nodes when quorum is lost.

 [root@node01 ~]# pcs property set no-quorum-policy=freeze  

3. Setup DLM and CLVMD cluster resource.

Before proceeding with this step it is important to note that Cluster Fencing (STONITH) needs to be preconfigured. Otherwise the DLM and CLVMD resources do not activate the LVM and the shared storage will not be accessible. We had already configured iLO5 fencing as shown below.

[root@node01 ~]# pcs stonith create HPEiLO_N01_FENCE fence_ilo5_ssh \
ipaddr="192.168.3.5"login="fenceuser1" secure="true" passwd="password" \
method="onoff" delay=10 pcmk_reboot_action="reboot" pcmk_host_list="node01"

[root@node01 ~]# pcs stonith create HPEiLO_N02_FENCE fence_ilo5_ssh \
ipaddr="192.168.3.6" login="fenceuser2" secure="true" passwd="password" \
method="onoff" pcmk_reboot_action="reboot" pcmk_host_list="node02"

 [root@node01 ~]# pcs status
.........
HPEiLO_N01_FENCE      (stonith:fence_ilo5_ssh):      Started node01  
HPEiLO_N02_FENCE      (stonith:fence_ilo5_ssh):      Started node02  
.........

Setting up DLM resource.

[root@node01 ~]# pcs resource create dlm ocf:pacemaker:controld op \
monitor interval=30s on-fail=fence clone interleave=true ordered=true  

Setting up CLVMD resource.

[root@node01 ~]# pcs resource create clvmd ocf:heartbeat:clvm op \
monitor interval=30s on-fail=fence clone interleave=true ordered=true  

Setting DLM and CLVMD start up order and dependency. First the DLM resource must start then the CLVMD resource. Both must run on the same node.

[root@node01 ~]# pcs constraint order start dlm-clone then clvmd-clone
Adding dlm-clone clvmd-clone (kind: Mandatory) (Options: first-action=start then-action=start)
[root@node01 ~]# pcs constraint colocation add clvmd-clone with dlm-clone

We can verify if the above resources have been created by checking the status of the cluster.

[root@node01 ~]# pcs status
Clone Set: dlm-clone [dlm]
     Started: [ node01 node02 ]
Clone Set: clvmd-clone [clvmd]
     Started: [ node01 node02 ]

Configure GFS2 File System

Since we have deleted the previous LVM of the shared storage, we need to recreate the LVM and format as GFS2 Filesystem.

1. Create Clustered LVM

[root@node01 ~]# pvcreate /dev/mapper/shared01
[root@node01 ~]# vgcreate -Ay -cy vg_shared01 /dev/mapper/shared01
[root@node01 ~]# lvcreate -l 100%FREE -n lv_shared01 vg_shared01

The first command creates a Physical Volume of the shared storage mapped as device file /dev/mapper/shared01.

The second command creates a Volume Group vg_shared01 with the attributes Clustered (-cy) and Autobackup (-Ay) enabled.

The third command creates a Logical Volume lv_shared01 utilizing all the free space available in the Volume Group.

2. Create GFS2 file system

[root@node01 ~]# mkfs.gfs2 -p lock_dlm -j 2 -t dev_cluster:dev01 \
/dev/vg_shared01/lv_shared01

The above command formats the LVM device created in step 1 as GFS2 file system. It is recommended to only use GFS2 as the file system for Active/Active shared storage. For Active/Passive cluster EXT4 or XFS file system can be used as well.

The option -p stands for LockProtoName. We used lock_dlm. 

The option -j represents the number of journals. It is important to note that this number should be equal to the number of nodes in the cluster. So we have set this option to 2 journals. It creates one journal per system that mounts the file system.

The -t option stands for LockTableName. It's format is clustername:fsname. Clustername name must the same as the name of the cluster and fsname is a unique name of the GFS2 file system to differentiate from other file system names.

All the details are described here.

3. Create Filesystem resource to manage and mount the file system.

The /shared01 directory must be created on both the nodes before the following command is run since the command mounts the device to the directory /shared01 on both the nodes simultaneously.

[root@node01 ~]# pcs resource create FS_shared01 ocf:heartbeat:Filesystem \
device="/dev/vg_shared01/lv_shared01" directory="/shared01" fstype="gfs2" --clone

4. Set CLVMD and Filesystem resource startup order and dependency.

[root@node01 ~]# pcs constraint order start clvmd-clone then FS_shared01-clone
Adding dlm-clone clvmd-clone (kind: Mandatory) (Options: first-action=start then-action=start)
[root@node01 ~]# pcs constraint colocation add FS_shared01-clone with clvmd-clone

Configure Virtual IP Resource 

First we create the Virtual IP resource with the IP and subnet details.

[root@node01 ~]# pcs resource create VirtualIP IPaddr2 ip=198.168.90.17 cidr_netmask=24

Then we clone the Virtual IP resource. This step also depends on the application usage. If the application in the cluster requires load-balancing and failover, then this configuration is useful.

If only shared storage needs to be active on both the nodes and Virtual IP only connects to the application on one node at a time, then we do not need to clone the Virtual IP resource. If both shared storage and VIP only need to be active on a single node then an Active/Passive cluster mode is sufficient.

[root@node01 ~]# pcs resource clone VirtualIP \
globally-unique=true clone-max=2 clone-node-max=2
[root@node01 ~]# pcs resource update VirtualIP clusterip_hash=sourceip

The clone options are:

1. Globally-unique=true

When set true,  each copy of the clone performs different functions.

2. Clone-max=2

The number of resources to start, in this case 2.

3. Clone-node-max=2

The number of copies of resource to be started on a single node, in this case 2.

The clusterip_hash decides which hosts receive and process which requests. The parameter we have chosen is sourceip. The source IP address of incoming packets are hashed and the nodes each will receive certain hashes to process the requests.

Then we set the startup order and dependency of FS_shared01-clone and Virtual IP resource.

[root@node01 ~]# pcs constraint order start FS_shared01-clone then VirtualIP-clone
Adding dlm-clone clvmd-clone (kind: Mandatory) (Options: first-action=start then-action=start)
[root@node01 ~]# pcs constraint colocation add VirtualIP-clone with FS-shared01-clone

Similarly, we can setup other resources as per our requirement like Web Servers, Network File Shares or Database Services.

Finally checking the status of the cluster, we get the below output.

[root@node01 ~]# pcs status
Cluster name: dev_cluster
Stack: corosync

2 nodes configured
10 resources configured

Online: [ node01 node02 ]

Full list of resources:

 HPEiLO_N01_FENCE      (stonith:fence_ilo5_ssh):      Started node01  
 HPEiLO_N02_FENCE      (stonith:fence_ilo5_ssh):      Started node02  
 Clone Set: dlm-clone [dlm]
     Started: [ node01 node02 ]
 Clone Set: clvmd-clone [clvmd]
     Started: [ node01 node02 ]
 Clone Set: FS_shared01-clone [FS_shared01]
     Started: [ node01 node02 ]
 Clone Set: VirtualIP-clone [VirtualIP] (unique)
     VirtualIP:0        (ocf::heartbeat:IPaddr2) Started
     VirtualIP:1        (ocf::heartbeat:IPaddr2) Started