vSphere: VM Stuck during Power down at 95%

Occasionally I’ve run into a VM that gets stuck at 95% while powering down (or during a vMotion).  I know the issue isn’t unheard of, but I didn’t run into it until working with a few ESXi 4.0.0 208167 servers. So – if you have a virtual machine hangs while shutting down – and you’re certain that you’re just not waiting for it to finish powering down, and you’ve already tried to “power off” from the client- but the power-off command is stuck at 95%, you may have to manually kill the hung VM.

1. Login to the host with the hung machine via SSH (enable SSH if you haven’t already)
2. do a /sbin/services.sh restart (or services vmware-mgmt restart on ESX)… which is the same thing as doing this from the ESXi console
3. This command will restart the agents that are installed in /etc/init.d/ … including hostd, ntpd, sfcbd, sfcbd-watchdog, slpd and wsmand (and HA if you have it)
4. When you do this, the VI/vsphere client will loose connectivity as those services restart, but VM’s that are running will not be affected
5. After the services have restarted, you can re-connect via the VI client.
6. Via SSH, go to the right datastore (such as, /vmfs/volumes/DatastoreName/VMname), and delete (rm -r) the *.vswp file (the swap file).
7. If you can’t delete it, and you’re getting an error message to the effect… can not remove VM: device or resource busy… go find the processes associated with the VM.
8. “ps auxfww|grep “vmname”
9. “kill -9 ProcessIDNumber”
10. After doing so, remove the orphaned VM from inventory… just right-click the “unknown” VM, and select “remove from inventory”, being careful to not delete it.
11. Then delete the *.log, and *.0*. If you don’t, re-adding the VM may cause the interface to hang, and you’ll have to go through some of this all over again.
12. Add the VM back to the inventory, and you should be able to start the VM.

I have run into a situation once, where a host reboot was the only way to solve the problem. But other than that, this seems to be quite effective.  The short version – see steps 8 & 9.