vSphere: Unable to add Datastore error

This is a short blog post regarding an issue I faced in VMware ESXi 6.0. I installed the ESXi after creating array from the array controller settings but local datastore was not present. When trying to add the datastore, an error was thrown: 

Call "HostDatastoreSystem.QueryVmfsDatastoreCreateOptions" for object "ha-datastore ...."

After quickly googling the issue, I found out the issue was caused due to existing information on the disk that had been previously used. Even after creating new array this information was not deleted and the vSphere was unable to understand it. So, the datastore was not allowed to be created. 

So to delete the partition and extra data present, I could connect through SSH to the host and use partedUtil utility for this purpose. So, first I enable SSH from the host Configuration tab and Security Profile settings under Software.  

We login to the host and go to the following location and list the partitions.

 [root@localhost:~] cd /vmfs/devices/disks  
 [root@localhost:/dev/disks] pwd  
 /vmfs/devices/disks  

 [root@localhost:/dev/disks] ls  
 naa.600508b1001c17376aeda14dd3174f90  
 naa.600508b1001c17376aeda14dd3174f90:1  
 naa.600508b1001c17376aeda14dd3174f90:2  
 naa.600508b1001c17376aeda14dd3174f90:3  
 vml.0200010000600508b1001c17376aeda14dd3174f904c4f47494341  
 vml.0200010000600508b1001c17376aeda14dd3174f904c4f47494341:1  
 vml.0200010000600508b1001c17376aeda14dd3174f904c4f47494341:2  
 vml.0200010000600508b1001c17376aeda14dd3174f904c4f47494341:3  

We need to change label as msdos.

 [root@localhost:/dev/disks] partedUtil mklabel "/vmfs/devices/disks/naa.600508b1001c17376aeda14dd3174f90" msdos  

Now, we can add the datastore normally and no errors will be thrown. And do not forget to disable SSH after completing your job !

Red Hat Cluster Part 7: High Availability Cluster using Pacemaker

So, we are going to setup a High Availability Active/Passive Cluster running NFS server. Although we are setting this environment in VMware vCenter, this is not a VMware cluster, rather an OS level cluster. We can see Node-N1 and Node-N2 virtual machines in VMware in the below image. I have setup RDM (Raw Device Mapping) as shared storage on both the nodes provided from FreeNAS.

We will be using CentOS 7.2 for the cluster test.

[root@node02 ~]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)

The IP and Hostname details of the nodes:

[root@node01 ~]# cat /etc/hosts
127.0.0.1   cluster-node1.domain.com localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.200.124 node01  node01.test.com
192.168.200.125 node02  node02.test.com

[root@node02 ~]# cat /etc/hosts
127.0.0.1   cluster-node2.domain.com localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.200.124 node01  node01.test.com
192.168.200.125 node02  node02.test.com

Install Pacemaker Configuration Tools

We will install the required packages on both the nodes, however, we will only install the vmware-soap fencing agent as the nodes are Virtual Machines in a VMware environment.

[root@node01 ~]# yum -y install pcs pacemaker fence-agents-vmware-soap

[root@node02 ~]# yum -y install pcs pacemaker fence-agents-vmware-soap

Add Exception rule in Firewall

[root@node01 ~]# firewall-cmd --permanent --add-service=high-availability
success
[root@node01 ~]# firewall-cmd --add-service=high-availability
success

[root@node02 ~]# firewall-cmd --permanent --add-service=high-availability
success
[root@node02 ~]# firewall-cmd --add-service=high-availability
success

Setup password for hacluster user

[root@node01 ~]# passwd hacluster
Changing password for user hacluster.
New password:

[root@node02 ~]# passwd hacluster
Changing password for user hacluster.
New password:

Start the service

[root@node01 ~]# systemctl start pcsd.service
[root@node01 ~]# systemctl enable pcsd.service
Created symlink from /etc/systemd/system/multi-user.target.wants/pcsd.service to /usr/lib/systemd/system/pcsd.service.

[root@node02 ~]# systemctl start pcsd.service
[root@node02 ~]# systemctl enable pcsd.service
Created symlink from /etc/systemd/system/multi-user.target.wants/pcsd.service to /usr/lib/systemd/system/pcsd.service.

Authorize the nodes in the cluster

[root@node01 ~]# pcs cluster auth node01.test.com node02.test.com
Username: hacluster
Password:
node02.test.com: Authorized
node01.test.com: Authorized

Create Cluster

[root@node01 ~]# pcs cluster setup --start --name cluster_pr node01.test.com node02.test.com
Destroying cluster on nodes: node01.test.com, node02.test.com...
node01.test.com: Stopping Cluster (pacemaker)...
node02.test.com: Stopping Cluster (pacemaker)...
node01.test.com: Successfully destroyed cluster
node02.test.com: Successfully destroyed cluster

Sending 'pacemaker_remote authkey' to 'node01.test.com', 'node02.test.com'
node01.test.com: successful distribution of the file 'pacemaker_remote authkey'
node02.test.com: successful distribution of the file 'pacemaker_remote authkey'
Sending cluster config files to the nodes...
node01.test.com: Succeeded
node02.test.com: Succeeded

Starting cluster on nodes: node01.test.com, node02.test.com...
node01.test.com: Starting Cluster...
node02.test.com: Starting Cluster...

Synchronizing pcsd certificates on nodes node01.test.com, node02.test.com...
node02.test.com: Success
node01.test.com: Success
Restarting pcsd on the nodes in order to reload the certificates...
node02.test.com: Success
node01.test.com: Success

Enable Cluster

[root@node01 ~]# pcs cluster enable --all
node01.test.com: Cluster Enabled
node02.test.com: Cluster Enabled

We can see below that the cluster is enabled and running.

[root@node01 ~]# pcs cluster status
Cluster Status:
 Stack: corosync
 Current DC: node01.test.com (version 1.1.16-12.el7_4.8-94ff4df) - partition with quorum
 Last updated: Thu May 10 02:58:50 2018
 Last change: Thu May 10 02:55:42 2018 by hacluster via crmd on node01.test.com
 2 nodes configured
 0 resources configured

PCSD Status:
  node01.test.com: Online
  node02.test.com: Online

Fencing Agent Configuration

For fencing, we are going to use VMware fencing agent, so we need to create a vCenter user with privileges to Power on and off the VMs.

Test the Fencing Agent

[root@node01 ~]# fence_vmware_soap -z --ssl-insecure -a 192.168.200.212 -l 'fenceuser@vsphere.local' -p '1CLUP@ss3rd' -o list | grep Node
Node-N1,421fd583-88b7-24ce-401d-5f59222d07ab
Node-N2,421fd09e-754b-bc16-aa46-5398b59261a9

Configure Fencing Agent

[root@node01 ~]# pcs stonith create vm_fence fence_vmware_soap ipaddr=192.168.200.212 ipport=443 ssl=1 ssl_insecure=1 login='fenceuser@vsphere.local' passwd='1CLUP@ss3rd' pcmk_off_action=reboot pcmk_host_map="node01.test.com:Node-N1;node02.test.com:Node-N2" delay=10
[root@node01 ~]# echo $?
0

We can see that the fencing agent has been created successfully and is running.

[root@node01 ~]# pcs status
Cluster name: cluster_pr
Stack: corosync
Current DC: node01.test.com (version 1.1.16-12.el7_4.8-94ff4df) - partition with quorum
Last updated: Thu May 10 04:18:14 2018
Last change: Thu May 10 04:16:44 2018 by root via cibadmin on node01.test.com

2 nodes configured
1 resource configured

Online: [ node01.test.com node02.test.com ]

Full list of resources:

 vm_fence       (stonith:fence_vmware_soap):    Started node01.test.com

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

Ignore Quorum

We ignore quorum configuration as it is not supported in 7.2.

 [root@node01 ~]# pcs property set no-quorum-policy=ignore  

Configure LVM on shared storage

We can see in the below images, the shared storage (sdb) is present on both the nodes.

[root@node01 ~]# lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
fd0               2:0    1    4K  0 disk
sda               8:0    0   20G  0 disk
├─sda1            8:1    0    1G  0 part /boot
└─sda2            8:2    0   19G  0 part
  ├─centos-root 253:0    0   17G  0 lvm  /
  └─centos-swap 253:1    0    2G  0 lvm  [SWAP]
sdb               8:16   0   90G  0 disk
sr0              11:0    1 1024M  0 rom

[root@node02 ~]# lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
fd0               2:0    1    4K  0 disk
sda               8:0    0   20G  0 disk
├─sda1            8:1    0    1G  0 part /boot
└─sda2            8:2    0   19G  0 part
  ├─centos-root 253:0    0   17G  0 lvm  /
  └─centos-swap 253:1    0    2G  0 lvm  [SWAP]
sdb               8:16   0   90G  0 disk
sr0              11:0    1 1024M  0 rom

The following steps are to be performed on one node only. First create Physical Volume.

[root@node01 ~]# pvcreate /dev/sdb
  Physical volume "/dev/sdb" successfully created.

Then Volume Group named "vg_cluster".

[root@node01 ~]# vgcreate vg_cluster /dev/sdb
  Volume group "vg_cluster" successfully created

Finally Logical Volume named "lv_cluster".

[root@node01 ~]# lvcreate -l 23039 -n lv_cluster vg_cluster
  Logical volume "lv_cluster" created.
[root@node01 ~]# lvs
  LV         VG         Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root       centos     -wi-ao---- <17.00g
  swap       centos     -wi-ao----   2.00g
  lv_cluster vg_cluster -wi-a----- <90.00g

Now, formatting the Logical Volume to 'xfs'.

[root@node01 ~]# mkfs.xfs /dev/vg_cluster/lv_cluster
[root@node01 ~]# blkid |grep cluster
/dev/mapper/vg_cluster-lv_cluster: UUID="886d99c3-a1d1-46f0-bfa7-00c81677e5c1" TYPE="xfs"

[root@node01 ~]# lsblk
NAME                    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
fd0                       2:0    1    4K  0 disk
sda                       8:0    0   20G  0 disk
├─sda1                    8:1    0    1G  0 part /boot
└─sda2                    8:2    0   19G  0 part
  ├─centos-root         253:0    0   17G  0 lvm  /
  └─centos-swap         253:1    0    2G  0 lvm  [SWAP]
sdb                       8:16   0   90G  0 disk
└─vg_cluster-lv_cluster 253:2    0   90G  0 lvm
sr0                      11:0    1 1024M  0 rom

We can see above that the LVM partition has been created on sdb.

Create the NFS Share

We will create the Network File System Share that is highly available which is the main objective of this cluster.

[root@node01 ~]# mkdir /Nfs
[root@node01 ~]# mount /dev/vg_cluster/lv_cluster /Nfs/

[root@node01 ~]# mkdir /Nfs/BACKUPDIR{1..5}
[root@node01 ~]# ls /Nfs/
BACKUPDIR1  BACKUPDIR2  BACKUPDIR3  BACKUPDIR4  BACKUPDIR5

[root@node01 ~]# touch /Nfs/BACKUPDIR1/hello_world{1..5}
[root@node01 ~]# ls /Nfs/BACKUPDIR1/
hello_world1  hello_world2  hello_world3  hello_world4  hello_world5

[root@node01 ~]# umount /Nfs
[root@node01 ~]# vgchange -an vg_cluster
  0 logical volume(s) in volume group "vg_cluster" now active

Activate Volume Group in a Cluster

The following steps need to be performed on both the nodes. We need to make some changes in the LVM configuration file so that the OS does not activate the LVM partition we created above as it will be managed by the Cluster services.

[root@node01 ~]# lvmconf --enable-halvm --services --startstopservices
Warning: Stopping lvm2-lvmetad.service, but it can still be activated by:
  lvm2-lvmetad.socket
Removed symlink /etc/systemd/system/sysinit.target.wants/lvm2-lvmetad.socket.  

[root@node02 ~]# lvmconf --enable-halvm --services --startstopservices
Warning: Stopping lvm2-lvmetad.service, but it can still be activated by:
  lvm2-lvmetad.socket
Removed symlink /etc/systemd/system/sysinit.target.wants/lvm2-lvmetad.socket.

We list all the Volume Groups present in the OS.

[root@node01 ~]# vgs --noheadings -o vg_name
  centos
  vg_cluster

[root@node02 ~]# vgs --noheadings -o vg_name
  centos
  vg_cluster

Then, we edit the LVM configuration file as shown below. We have added the "centos" Volume Group and not "vg_cluster". The "centos" Volume Group will be activated and managed by the OS.

[root@node01 ~]# vim /etc/lvm/lvm.conf

[root@node02 ~]# vim /etc/lvm/lvm.conf

[root@node01 ~]# vim /etc/lvm/lvm.conf
[root@node01 ~]# cat /etc/lvm/lvm.conf |grep volume_list
        # it is auto-activated. The auto_activation_volume_list setting
        # Configuration option activation/volume_list.
        #     or VG. See tags/hosttags. If any host tags exist but volume_list
        # volume_list = [ "vg1", "vg2/lvol1", "@tag1", "@*" ]
        volume_list = [ "centos" ]

[root@node02 ~]# vim /etc/lvm/lvm.conf
[root@node02 ~]# cat /etc/lvm/lvm.conf |grep volume_list
        # it is auto-activated. The auto_activation_volume_list setting
        # Configuration option activation/volume_list.
        #     or VG. See tags/hosttags. If any host tags exist but volume_list
        # volume_list = [ "vg1", "vg2/lvol1", "@tag1", "@*" ]
        volume_list = [ "centos" ]

Take a backup of the "initramfs-$(uname -r).img" file.

[root@node01 ~]# cp -p /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak

[root@node02 ~]# cp -p /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak

Now, we are going to rebuild the initramfs boot image for the changes to be permanent.

[root@node01 ~]# dracut -H -f /boot/initramfs-$(uname -r).img $(uname -r)

[root@node02 ~]# dracut -H -f /boot/initramfs-$(uname -r).img $(uname -r)

And finally we need to reboot both the nodes.

[root@node01 ~]# reboot

[root@node02 ~]# reboot

After rebooting we need to make sure that the cluster services are running and the status is good.

[root@node01 ~]# pcs status
Cluster name: cluster_pr
Stack: corosync
Current DC: node01.test.com (version 1.1.16-12.el7_4.8-94ff4df) - partition with quorum
Last updated: Thu May 10 05:29:30 2018
Last change: Thu May 10 05:27:12 2018 by hacluster via crmd on node01.test.com

2 nodes configured
1 resource configured

Online: [ node01.test.com node02.test.com ]

Full list of resources:

 vm_fence       (stonith:fence_vmware_soap):    Started node01.test.com

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

Create Resource

We need to create resources for services to be run and be managed by the cluster. 

First we create the LVM resource provided by 'OCF' (Open Cluster Framework).

[root@node01 ~]# pcs resource create lvm_res LVM volgrpname=vg_cluster exclusive=true --group group_Nfs
Assumed agent name 'ocf:heartbeat:LVM' (deduced from 'LVM')
[root@node01 ~]# echo $?
0

Second we create the Filesystem resource.

[root@node01 ~]# pcs resource create nfsshare Filesystem device=/dev/vg_cluster/lv_cluster directory=/Nfs fstype=xfs --group group_Nfs
Assumed agent name 'ocf:heartbeat:Filesystem' (deduced from 'Filesystem')
[root@node01 ~]# echo $?
0

Third we create the nfs server and nfs-export resource

[root@node01 ~]# pcs resource create nfs-daemon nfsserver nfs_shared_infodir=/Nfs/nfsinfo nfs_no_notify=true --group group_Nfs
Assumed agent name 'ocf:heartbeat:nfsserver' (deduced from 'nfsserver')
[root@node01 ~]# echo $?
0

[root@node01 ~]# pcs resource create nfs-root exportfs clientspec=192.168.200.0/255.255.255.0 options=rw,sync,no_root_squash directory=/Nfs/BACKUPDIR1 fsid=0 --group group_Nfs
Assumed agent name 'ocf:heartbeat:exportfs' (deduced from 'exportfs')

We also need the VIP (Virtual IP) for the shared resources.

[root@node01 ~]# pcs resource create nfs_ip IPaddr2 ip=192.168.200.123 cidr_netmask=24 --group group_Nfs
Assumed agent name 'ocf:heartbeat:IPaddr2' (deduced from 'IPaddr2')

Finally, the nfs-notify resource is created.

[root@node01 ~]# pcs resource create nfs-notify nfsnotify source_host=192.168.200.123 --group group_Nfs
Assumed agent name 'ocf:heartbeat:nfsnotify' (deduced from 'nfsnotify')

The resources have been created and are running.

[root@node02 ~]# pcs status
Cluster name: cluster_pr
Stack: corosync
Current DC: node01.test.com (version 1.1.16-12.el7_4.8-94ff4df) - partition with quorum
Last updated: Thu May 10 12:54:42 2018
Last change: Thu May 10 12:50:14 2018 by root via cibadmin on node01.test.com

2 nodes configured
7 resources configured

Online: [ node01.test.com node02.test.com ]

Full list of resources:

 vm_fence       (stonith:fence_vmware_soap):    Started node01.test.com
 Resource Group: group_Nfs
     lvm_res    (ocf::heartbeat:LVM):   Started node02.test.com
     nfsshare   (ocf::heartbeat:Filesystem):    Started node02.test.com
     nfs-daemon (ocf::heartbeat:nfsserver):     Started node02.test.com
     nfs-root   (ocf::heartbeat:exportfs):      Started node02.test.com
     nfs_ip     (ocf::heartbeat:IPaddr2):       Started node02.test.com
     nfs-notify (ocf::heartbeat:nfsnotify):     Started node02.test.com

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

The NFS share is running on the second node.

[root@node02 ~]# df -h
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/centos-root             17G  1.3G   16G   8% /
devtmpfs                           1.9G     0  1.9G   0% /dev
tmpfs                              1.9G   39M  1.9G   3% /dev/shm
tmpfs                              1.9G  8.6M  1.9G   1% /run
tmpfs                              1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/sda1                         1014M  164M  851M  17% /boot
tmpfs                              378M     0  378M   0% /run/user/0
/dev/mapper/vg_cluster-lv_cluster   90G   33M   90G   1% /Nfs

The first node is in passive mode, so the NFS share is not present.

[root@node01 ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   17G  1.3G   16G   8% /
devtmpfs                 1.9G     0  1.9G   0% /dev
tmpfs                    1.9G   54M  1.8G   3% /dev/shm
tmpfs                    1.9G  8.5M  1.9G   1% /run
tmpfs                    1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/sda1               1014M  164M  851M  17% /boot
tmpfs                    378M     0  378M   0% /run/user/0

Check the exports from client machine.

[root@srv ~]# showmount -e 192.168.200.123
clnt_create: RPC: Port mapper failure - Unable to receive: errno 113 (No route to host)

There was an error. So, we need to add firewall exceptions.

[root@node02 ~]# firewall-cmd --permanent --add-service=rpc-bind
success
[root@node02 ~]# firewall-cmd --permanent --add-service=mountd
success
[root@node02 ~]# firewall-cmd --permanent --add-port=2049/tcp
success
[root@node02 ~]# firewall-cmd --permanent --add-port=2049/udp
success
[root@node02 ~]# firewall-cmd --reload
success

Now, the command will be run successfully.

[root@srv ~]# showmount -e 192.168.200.123
Export list for 192.168.200.123:
/Nfs/BACKUPDIR1 192.168.200.0/255.255.255.0

The firewall exceptions need to be added on the first node as well.

[root@node01 ~]# firewall-cmd --permanent --add-service=rpc-bind
success
[root@node01 ~]# firewall-cmd --permanent --add-service=mountd
success
[root@node01 ~]# firewall-cmd --permanent --add-port=2049/tcp
success
[root@node01 ~]# firewall-cmd --permanent --add-port=2049/udp
success
[root@node01 ~]# firewall-cmd --reload
success

Testing NFS share

[root@srv ~]# mount  192.168.200.123:/Nfs/BACKUPDIR1 nfsshare
[root@srv ~]# df -h
Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/rhel-root             50G  8.0G   42G  16% /
devtmpfs                         2.9G     0  2.9G   0% /dev
tmpfs                            2.9G   80K  2.9G   1% /dev/shm
tmpfs                            2.9G   17M  2.9G   1% /run
tmpfs                            2.9G     0  2.9G   0% /sys/fs/cgroup
/dev/mapper/rhel-home            1.3T  253G 1019G  20% /home
/dev/sda1                        497M  119M  379M  24% /boot
192.168.200.123:/Nfs/BACKUPDIR1   90G   33M   90G   1% /root/nfsshare

[root@srv ~]# ls /root/nfsshare/
BACKUPDIR2  BACKUPDIR4  hello_world1  hello_world3  hello_world5
BACKUPDIR3  BACKUPDIR5  hello_world2  hello_world4

Changing the node02 to standby mode.

[root@node01 ~]# pcs cluster standby node02.test.com

We can still access the NFS share and list the contents.

[root@srv ~]# ls /root/nfsshare/
BACKUPDIR2  BACKUPDIR4  hello_world1  hello_world3  hello_world5
BACKUPDIR3  BACKUPDIR5  hello_world2  hello_world4

So, we have successfully tested the Cluster.

vSphere: Process Lock Issue

Recently, we faced an issue in vSphere 6.5, after cancelling an OVF export mid-process. In this blog we will go through the troubleshooting process in detail and the resolution. 

The Issue

One of our customers exported a Virtual Machine from vCenter and cancelled the process before completion. Then the VM could not be powered ON. The error received was:

 Error Stack:   
   An error was received from the ESX host while powering on VM TEST-VM.  
  Failed to start the virtual machine.  
  Module 'Disk' power on failed.   
  Cannot open the disk '/vmfs/volumes/5753f99f-d235ca6c-a280-d89d6779a96c/TEST-VM/TEST-VM_1.vmdk' or one of the snapshot disks it depends on.   
  Failed to lock the file  
  File system specific implementation of LookupAndOpen[file] failed  
  File system specific implementation of LookupAndOpen[file] failed  
  File system specific implementation of LookupAndOpen[file] failed  
  File system specific implementation of OpenFile[file] failed  

On checking the vmkernel log file we find the below issue.

 [root@test:/var/log] tail -f vmkernel.log   
 num 1 gblnum 0 gblgen 0 gblbrk 0] alloc owner 3833856  
 2019-11-12T11:38:08.515Z cpu18:72666)DLX: 4312: vol 'RAID5_DATASTORE2', lock at 94683136: [Req mode 1] Checking liveness:  
 2019-11-12T11:38:08.515Z cpu18:72666)[type 10c00001 offset 3763970 v 55, hb offset 3686970  
 gen 745, mode 2, owner 00000000-00000000-0000-000000000000 mtime 6167 num 1 gblnum 0 gblgen 0 gblbrk 0]  

On running the following command we can see that there is a Lock of type 10c00001 and mode 0. However the owner is 00000000-00000000-0000-000000000000.

 [root@test:/vmfs/volumes/5753f99f-d235ca6c-a280-d89d6779a96c/TEST-VM] vmkfstools -D /vmfs/volumes/5753f99f-d235ca6c-a280-d89d6779a96c/TEST-VM/TEST-VM.vmx   

 Lock [type 10c00001 offset 60387328 v 456, hb offset 3686970 gen 3, mode 0, owner 00000000-00000000-0000-000000000000 mtime 12601 num 0 gblnum 0 gblgen 0 gblbrk 0]   
 Addr <4, 121, 70>, gen 142, links 1, type reg, flags 0, uid 0, gid 0, mode 100755 len 3486, nb 1 tbz 0, cow 0, newSinceEpoch 1, zla 2, bs 8192  

The different mode types described on VMware site are as following:

• mode 0 = no lock
• mode 1 = is an exclusive lock (vmx file of a powered on VM, the currently used disk (flat ir delta), *vswp, etc)
• mode 2 = is a read-only lock (e.g. on the ..-flat.vmdk of a running VM with snapshots)
• mode 3 = is a multi-writer lock. (e.g. used for MSCS clusters disks or FT VMs)

We can determine the owner (host) where the process is locked by running the following command with the offset value from previous output (offset 3686970).

 [root@test: /vmfs/volumes/5753f99f-d235ca6c-a280-d89d6779a96c/TEST-VM] hexdump -C /vmfs/volumes/5753f99f-d235ca6c-a280-d89d6779a96c/.vh.sf -n 512 -s 3686970  

 003ba000 00 40 5d 7e 00 00 00 00 00 00 00 00 89 00 00 00 |……;………|  
 003ba010 00 00 00 00 c9 dd 7f a1 5c 01 00 00 89 e8 ba 55 |….._../…….|  
 003ba020 e1 ff 30 43 c1 dd 7f 8e 3d 3a 21 1a a4 a6 8a 99 |..m....S...….|  
 003ba030 e0 00 00 00 9d 04 00 00 00 00 00 00 00 00 00 00 |…......……..|  
 003ba040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |…………….|  

By checking the MAC address of the hosts, we can verify the host where the process is locked. In this case it was the same host where the VM was located. The MAC address of one the host's NIC included (7f 8e 3d 3a), as indicated in the above output.

Troubleshooting Steps

First, we removed the VM (TEST-VM) from inventory. Then registered the VM to the same host. The problem was still persistent.

Second, removed the VM from inventory and created a new VM with the same virtual drives from the problematic VM (TEST-VM). While trying to power ON the new VM the problem still persisted.

Third, we restarted the ESXi host daemon 'hostd' and vCenter Agent 'vpxa' services on the host to no avail.

 /etc/init.d/hostd restart  
 /etc/init.d/vpxa restart  

Fourth, we restarted all the management services on the host. However, the problem was not resolved.

 services.sh restart  

Fifth, we rebooted the host and tried to power ON the VM. The issue was still there.

The Resolution

Since the OVF export process was run from vCenter, there was still a process that was locking the VM in vCenter and we were unable to power on the VM. We had already rebooted the host, so we could conclude that the host was not locking the VM.

We rebooted the vCenter and finally we could power ON the VM without any issues.

References

Unable to perform operations on a virtual machine with a locked disk (1003397)

How to find virtual machine file locks on ESXi | Ginka's World

Releasing Locked Files in a VMFS Volume - VMadmin.co.uk

Identify Virtual Machine Locks & Find ESXi Hosts by its MAC Address | Virten.net

virtualguido: VMware ESXi locking and how to kill a frozen VM

ESXi – Knowledge is POWER…!!!!!

VM will not start - VMDK Locked but no owner : vmware

vSphere: ESXi Space Reclaim

In a thin provisioned Datastore before ESXi 6.5 in format VMFS 5, we need to reclaim unused storage blocks manually. To confirm that unmap/reclaim is supported, we need to run the following command from a SSH session to the host. 

 # esxcli storage core device vaai status get -d naa  

If the naa details show Delete Status: supported, then we can proceed with unused space reclamation.

The following command executes the unmap procedure and needs to be run from each host mapped with the thin provisioned LUN.

 # esxcli storage vmfs unmap --volume-label=volume_label|--volume-uuid=volume_uuid --reclaim-unit=number  

The command can be run with the option of either volume label -l or volume UUID -u. The option of reclaim-unit is not necessary unless there is a need to change the default value of 200. It is advised to consult with storage provider while using a different reclaim-unit number.

 # esxcli storage vmfs unmap -l DatastoreLabel 

Or

 # esxcli storage vmfs unmap -u 90af8df1-709bc101-8ff9-a90154e32ddff24f

After running the above command from each host, the reclaim command needs to be run from the storage. Then the unused and reclaimed space will be shown and can be used.