lunedì 14 gennaio 2013
vSphere: Unable to access file
since it is locked
Consolidate virtual machine disk files - Unable to access file <unspecified filename> since it is locked
As you probably know fixing file lock errors on VMware it's quite a mess since there could be 1000s of causes for a file lock and getting the origin of file lock usually it's not so immediate.
So, following KB:
Investigating virtual machine file locks on ESXi/ESX
I logged in to ESXi host on which my VM was running.
Got a list of all VM running on host just to make sure the host on which troubled VM was running on was the correct one:
# vim-cmd vmsvc/getallvms
Have a detailed look at VM logs. Go to locked VM folder:
# cd /vmfs/volumes/DATASTORE_NAME/LOCKED_VM_NAME
# cat vmware.log | grep lock
One line reported this specific error:
VigorSnapshotManagerConsolidateCallback: snapshotErr = Failed to lock the file (5:4008)
After a brief search I found a KB:
Unable to remove locked snapshots
Which stated that the error above it's related to a failed backup operation.
VMware Data Recovery, as you know better than me, basically backups a running VM taking a snapshot of it, mounting that snapshot to VMware Data Recovery virtual appliance and backupping it (using data deduplication). This procedure guarantees system consistency of the backupped VM and allows to free guest VM CPU's from the burden of backup since all CPU operations for backup are performed on Data Recovery virtual appliance.
The same KB reported this solution:
1)Power off backup server (in my case VMware Data Recovery Appliance)
2)Remove the snapshots
3)If still see File Locked error reboot ESXi host on which VM was running (also a services.sh restart could be feasible)
1) I powered off VMware Data Recovery VM
2) No snapshot available on VM Snapshot Manager for that guest VM. Orphaned snapshots existed since browsing datastore resulted in 11!!! vmdks.
3) Foreseeing the inevitability of step 3 (ESXi host reboot) I decided to live migrate VM to an empty host.
4) After migration I took a new snapshot of VM, opened Snapshot Manager and clicked Delete All
Removing all snapshot consolidated my just taken snapshot and all previously created (orphaned) snapshots allowing to solve the issue.
To summarize...just powering off VMware Data Recovery and taking a new snapshot and deleting it did the trick and allowed me to consolidate VM status without powering off neither the ESXi host neither the VM itself.
Got all working properly! As you can see in the image below the virtual hard disk now points to VM.vmdk and not to a snapshot file!
NOTE: When powering back on VMware Data Recovery I got:
Power On virtual machine VMware Data Recovery Cannot open the disk '/vmfs/ volumes/....' or one of the snapshot disks it depends on.
Edit VMware Data Recovery settings (right clicking on VMware Data Recovery VM) and delete the second hard disk device which points to a previos snapshot and in particular the original snapshot that failed and created all this issue being locked...