I had a problem last night which was not necessary the type of problem you want on a friday night….

I got  report that the one of the volumes of a cluster array had failed.

So i did the usual checking, logged on to the san checked the volume was on line and it was… so the basic sanity check… 🙂 and it was registering used space…. another basic check…

Second i logged onto each hyper-v node and checked it’s iscsi initiator was running and connecting to the IQN for the lun that was missing

it wasnt so did the simple task of connecting the iscsi initiator to the appropriate IQN

But the lun could come back online in the cluster manager.

We are now into dangerous territory… and have to be very careful…

covered the basics and disk still not online….

Ok so on the event log….. check it at it said the expect disk signature was different to the one it was expecting so it could mount the disk…

I logged on to each hyper-v node ran disk part, selected the disk i wanted to check its disk signature on and ran detail disk (which gives you the disk signature)… and VERY STRANGE the disk signature was correct to what it was expect on all four nodes…

so the event log was a bit of a red herring…..

I decided it would be best to remove the disk from cluster shared volumes and as a cluster resource… disconnect the iscsi initiator on all but 1 node and try to force the volume online (i had a backup of the volume so i wasnt too worried but it can be a risky step)

and another strange message… it said the volume was right protected… RIGHT PROTECTED! i taught how the heck did it get that… but apparently it is by design ….

so how to get right protected off….

again using diskpart and selecting the disk you want to use

type detail disk and look for the read only setting to ensure it is actually set (current read-only state = yes)

image

once you confirm it type attribute disk clear readonly

and hey presto! the volume is online  and the data looks good…

all you got to do is add it back in to the cluster and mount the volume and bring the VM’s online…

Advertisements