When a vSAN “cache” disk fails and gets replaced you need to rebuild the entire disk group that the cache disk belongs to. The normal way of doing this is to put the host in maintenance mode and then click the three dots next to the disk group and select “remove” (see picture below) – but unfortunately most of the time you get this error when trying to remove the disk group:

General vSAN error. vSAN disk data evacuation resource check has failed for disk or disk-group vsan: <UUID>

Luckily there is an easy way of removing the disk group from ESXi CLI that does seem to work every time (at least for me!). Enable SSH on the host containing the defective disk group. Locate the diskgroup UUID and run the following command (remember to replace <UUID>):

esxcli vsan storage remove --uuid <UUID>

If you are unable to find the UUID for the diskgroup then you can run this command:

esxcli vsan storage list

Which will give you an output like this:

Important! – if your host has more than one disk group find the correct one by searching for an belonging disk (naa.xxxxxxx – first red arrrow). When a disk in the group that you need to delete is found, note the “vSAN Disk Group UUID” (second arrow) – this will be the UUID you need to delete with “remove” command mentioned earlier!

If you are uncertain what is going on here – then call VMware support. Deleting the wrong diskgroup can have fatal consequences!

For more information see VMware KB2150567 – link

If you see the above error when your “Veeam Agent Backup” jobs try to run, then you need to make a change to your Veeam database (on the Veeam server).

This is perfectly mentioned in this KB4421 – but if you are using Microsoft SQL (embedded with Veeam) and have no SQL management studio installed – here is a quick guide on how to fix the above problem – just using OSQL.exe (included with SQL embedded)

Locate the OSQL.EXE executable – I found it here: “C:\Program Files\Microsoft SQL Server\110\Tools\Binn”

Using command prompt (cmd.exe) run the executable with:

osql.exe -E -S <SERVERNAME>\<DATABASE>

Replace SERVERNAME and the database name (ex. “osql.exe -E -SVEEAMBACKUP\VEEAMSQL2012”). When OSQL prompt is ready for input type the following:

use VeeamBackup;
go
update dbo.[Backup.Model.EpHosts] set os_version='0.0' where os_version=''
go
exit

That should be it – try to run your “Veeam Agent Backups” again!

You are not very likely to bump into Windows 2003 physical servers anymore – but nevertheless that just happened to me a week ago. The task was clear – this server needs to be virtualized into a vSphere 7 environment, running vSAN.
The problem with this task is that to convert (P2V) a 2003 server you need to install vCenter Converter 6.2 on it, since the latest release 6.3 simply doesn’t work on 2003 servers (It won’t install).
Next problem is that vCenter Converter 6.2 doesn’t work with vSAN 7 – only “traditional storage” can be used as target – but in this case there were no other storage than vSAN that could be used as target.
What to do? – read on…

Continue reading

update installation failed, vCenter Server is non-operational
Problem: update installation failed, vCenter Server is non-operational

I recently ran in to this error upgrading my homelab vCenter from 7.0.0.10400 to 7.0.10600:

vCenter: update installation failed, vCenter Server is non-operational

Luckily, the fix was easy – all I needed to do was to delete the file “/etc/applmgmt/appliance/software_update_state.conf”

So you just need to SSH to your vCenter and execute this command:

rm /etc/applmgmt/appliance/software_update_state.conf
The fix: rm /etc/applmgmt/appliance/software_update_state.conf

A few days ago, I decided to update my vCenter server to version 6.7 U2c – normally this is an easy task with the update section in the VAMI interface. But this time I just encountered this error message when I tried to search for the update:

Error in method invocation ({‘default_message’: ‘Manifest verification failed’, ‘id’: ‘com.vmware.appliance.update.manifest_verification_failed’, ‘args’: []}, ‘Verification Failure\n’, ”)

Continue reading