Red Hat Cluster Part 5: Extend Shared Storage

Often times we have the requirement to increase the storage size in the servers. In a clustered environment, the process is similar to that of non-clustered environments. However, since we have setup virtualized nodes for cluster setup in VMware vCenter, we have to perform some extra steps. We will discuss these steps in detail in this blog.

Increase the Volume size from Storage

The first step is to increase the size of the volumes that have been configured as shared RDM disks among the nodes. I have already increased the size of the zvol from 10 GB to 15 GB in free NAS and restarted the iSCSI service in the free NAS.

Check shared RDM disk compatibility settings

There are two settings for the disk compatibility in a RDM disk.

• Physical - The SCSI commands are not operated by the hypervisor, allows for SAN management of the LUN and useful for low level operations.
• Virtual - The SCSI commands are operated by the hypervisor, so snapshots are supported. Lots of VMware hypervisor features are supported in this mode.

Detailed explanation can be found here and here.

The RDM disks that I have added before have the compatibility mode set as "Virtual".  

If the compatibility mode was set as "Physical", after re-scanning the iSCSI adapter, the shared disk would need to be re-scanned in the operating system and the shared block device size would be increased.

However, since the RDM disk compatibility mode is set to "Virtual", we need to remove the RDM disk from the virtual machine settings, delete files from the datastore and then re-add it for the changes to the disk to be seen. 

Remove the RDM disk

Before removing the RDM disk we need to note the "Virtual Device Node" controller and the SCSI(X:Y) settings. Here the disk is using "SCSI controller 2" with "SCSI(2:0)" settings.

But first we need to bring the cluster services down and power off both the nodes.

Then we delete the disk and select "Delete files from datastore" option to completely remove the previous disk size settings.

Re-scan the iSCSI Adapter in the Host

The next step is to re-scan the iSCSI software adapter in the ESXi host. We can see in the image below that the changes are imminent after the re-scan.

Re-add the RDM disk

Now, we can re-add the RDM disk and verify the increased size of the disk. We can see that the new capacity of the LUN is 15 GB.

Then we change the new hard disk settings where the compatibility mode is "Virtual", the sharing mode is "Multi-writer" and the virtual device node is set to the same settings as we previously noted which is "SCSI controller 2" and "SCSI(2:0)".

We remove and re-add the RDM disk on both the nodes.

Extend the LVM

We can then power on the virtual machines and extend the disks from the OS. First we need to check that the disk block device size has increased from the terminal by issuing the command "lsblk". Although on boot the disk size should have increased, if the disk size has not increased we can issue the following command to re-scan the device:

 [root@N2 ~]# echo "1" > /sys/class/block/sdc/device/rescan  

We can now check and verify that the block device size has increased to 15 GB.

 [root@N2 ~]# lsblk  
 NAME             MAJ:MIN RM SIZE RO TYPE MOUNTPOINT  
 sr0              11:0  1 1024M 0 rom  
 sda              8:0  0  50G 0 disk  
 ├─sda1             8:1  0 500M 0 part /boot  
 └─sda2             8:2  0 49.5G 0 part  
  ├─vg_redhat-lv_root (dm-0) 253:0  0 45.6G 0 lvm /  
  └─vg_redhat-lv_swap (dm-1) 253:1  0 3.9G 0 lvm [SWAP]  
 sdb              8:16  0  1G 0 disk  
 sdc              8:32  0  15G 0 disk /Test  

The logical volume can be extended by issuing the following commands. These commands should be run on only one of the nodes as it is a shared disk:

 [root@N2 ~]# pvresize /dev/sdc  
 [root@N2 ~]# vgextend shared_vg /dev/sdc  
 [root@N2 ~]# lvresize -l +100% -n ha_lv shared_vg  

Finally we resize the ext4 file system.

 [root@N2 ~]# resize2fs /dev/shared_vg/ha_lv  

 [root@N2 ~]# df -h  
 Filesystem                     Size Used Avail Use% Mounted on  
 /dev/mapper/vg_redhat-lv_root  45G  11G  33G   25%  /  
 tmpfs                          1.9G 26M  1.9G  2%   /dev/shm  
 /dev/sda1                      485M 39M  421M  9%   /boot  
 /dev/mapper/shared_vg-ha_lv    15G  172M 14G   1%   /Test  

Finally, we have increased and verified the size of the shared storage in a clustered environment.