Symptoms :
=========
Attempting to perform Test Failover fails with the error:
Error - Failed to recover datastore 'XX_HDXX_LUNXX_T4'. VMFS volume residing on recovered devices "5431", "5432", "5433" and expected to be auto-mounted during HBA rescan cannot be found
Started the replication via SRM the lun got successfully mounted.
Checked and found that the problem was with EMC replication.
The dr-vmware.log file showed the error
Failed to recover datastore 'VNX-MP-LUN07-TIER3'. VMFS volume residing on recovered devices '"60:06:01:60:07:E0:22:00:20:DD:9E:42:CF:0F:E1:11"' and expected to be auto-mounted during HBA rescan cannot be found
Logs :
=======
2012-01-06T23:06:54.263Z cpu14:114970)WARNING: HBX: 1889: Failed to initialize VMFS3 distributed locking on volume 4ef4e35b-1442b43c-038c-0025b5000c19: No 2012-01-06T23:06:54.271Z cpu14:114970)FSS: 4333: No FS driver claimed device 'snap-50d5bf4f-4ef4e359-709ff89d-a083-0025b5000c19': Not supported 2012-01-06T23:06:54.274Z cpu14:114970)VC: 1449: Device rescan time 38 msec (total number of devices 12) 2012-01-06T23:06:54.274Z cpu14:114970)VC: 1452: Filesystem probe time 64 msec (devices probed 9 of 12) 2012-01-06T23:06:54.292Z cpu11:2059)ScsiDeviceIO: 2316: Cmd(0x4124415c3540) 0x9e, CmdSN 0xcbc2 to dev "naa.50060160bce010a750060160bce010a7" failed H:0x0 2012-01-06T23:06:54.292Z cpu11:2059)ScsiDeviceIO: 2316: Cmd(0x4124415c3540) 0x25, CmdSN 0xcbc3 to dev "naa.50060160bce010a750060160bce010a7" failed H:0x0 2012-01-06T23:06:54.294Z cpu11:2059)ScsiDeviceIO: 2316: Cmd(0x4124415c3540) 0x28, CmdSN 0xcbcb to dev "naa.6006016007e02200608991b2ad2de111" failed H:0x0 2012-01-06T23:06:54.294Z cpu14:114970)Partition: 484: Read of GPT header failed on "naa.6006016007e02200608991b2ad2de111": I/O error 2012-01-06T23:06:54.294Z cpu11:2059)ScsiDeviceIO: 2316: Cmd(0x4124415c3540) 0x28, CmdSN 0xcbcc to dev "naa.6006016007e02200608991b2ad2de111" failed H:0x0 2012-01-06T23:06:54.294Z cpu14:114970)WARNING: Partition: 944: Partition table read from device naa.6006016007e02200608991b2ad2de111 failed: I/O error 2012-01-06T23:06:54.345Z cpu14:114970)HBX: 676: Setting pulse [HB state abcdef02 offset 3764224 gen 1 stampUS 177206183150 uuid 4f04ca84-1f05f6a1-c310-d8d 2012-01-06T23:06:54.345Z cpu14:114970)WARNING: FSAts: 1263: Denying reservation access on an ATS-only vol 'VNX-HP-LUN01-TIER2' 2012-01-06T23:06:54.345Z cpu14:114970)WARNING: HBX: 1889: Failed to initialize VMFS3 distributed locking on volume 4ef4e35b-1442b43c-038c-0025b5000c19: No 2012-01-06T23:06:54.352Z cpu14:114970)FSS: 4333: No FS driver claimed device 'snap-50d5bf4f-4ef4e359-709ff89d-a083-0025b5000c19': Not supported 2012-01-06T23:06:54.376Z cpu14:114970)Vol3: 647: Couldn't read volume header from control: Invalid handle 2012-01-06T23:06:54.376Z cpu14:114970)FSS: 4333: No FS driver claimed device 'control': Not supported 2012-01-06T23:06:54.396Z cpu14:114970)VC: 1449: Device rescan time 41 msec (total number of devices 12) Resolution :
============ Our initial workaround was to disable the single VAAI Primitive VMFS3.Hardware.AcceleratedLocking on the hosts. From EMC Had to change the affected hosts from Failover Mode 1 to Failover Mode 4 (within the Unisphere Storage System Connectivity Status menu). Once the hosts were changed to Failover Mode 4, we re-enabled the VAAI Primitive and no longer encountered the issue.
Logs :
=======
2012-01-06T23:06:54.263Z cpu14:114970)WARNING: HBX: 1889: Failed to initialize VMFS3 distributed locking on volume 4ef4e35b-1442b43c-038c-0025b5000c19: No 2012-01-06T23:06:54.271Z cpu14:114970)FSS: 4333: No FS driver claimed device 'snap-50d5bf4f-4ef4e359-709ff89d-a083-0025b5000c19': Not supported 2012-01-06T23:06:54.274Z cpu14:114970)VC: 1449: Device rescan time 38 msec (total number of devices 12) 2012-01-06T23:06:54.274Z cpu14:114970)VC: 1452: Filesystem probe time 64 msec (devices probed 9 of 12) 2012-01-06T23:06:54.292Z cpu11:2059)ScsiDeviceIO: 2316: Cmd(0x4124415c3540) 0x9e, CmdSN 0xcbc2 to dev "naa.50060160bce010a750060160bce010a7" failed H:0x0 2012-01-06T23:06:54.292Z cpu11:2059)ScsiDeviceIO: 2316: Cmd(0x4124415c3540) 0x25, CmdSN 0xcbc3 to dev "naa.50060160bce010a750060160bce010a7" failed H:0x0 2012-01-06T23:06:54.294Z cpu11:2059)ScsiDeviceIO: 2316: Cmd(0x4124415c3540) 0x28, CmdSN 0xcbcb to dev "naa.6006016007e02200608991b2ad2de111" failed H:0x0 2012-01-06T23:06:54.294Z cpu14:114970)Partition: 484: Read of GPT header failed on "naa.6006016007e02200608991b2ad2de111": I/O error 2012-01-06T23:06:54.294Z cpu11:2059)ScsiDeviceIO: 2316: Cmd(0x4124415c3540) 0x28, CmdSN 0xcbcc to dev "naa.6006016007e02200608991b2ad2de111" failed H:0x0 2012-01-06T23:06:54.294Z cpu14:114970)WARNING: Partition: 944: Partition table read from device naa.6006016007e02200608991b2ad2de111 failed: I/O error 2012-01-06T23:06:54.345Z cpu14:114970)HBX: 676: Setting pulse [HB state abcdef02 offset 3764224 gen 1 stampUS 177206183150 uuid 4f04ca84-1f05f6a1-c310-d8d 2012-01-06T23:06:54.345Z cpu14:114970)WARNING: FSAts: 1263: Denying reservation access on an ATS-only vol 'VNX-HP-LUN01-TIER2' 2012-01-06T23:06:54.345Z cpu14:114970)WARNING: HBX: 1889: Failed to initialize VMFS3 distributed locking on volume 4ef4e35b-1442b43c-038c-0025b5000c19: No 2012-01-06T23:06:54.352Z cpu14:114970)FSS: 4333: No FS driver claimed device 'snap-50d5bf4f-4ef4e359-709ff89d-a083-0025b5000c19': Not supported 2012-01-06T23:06:54.376Z cpu14:114970)Vol3: 647: Couldn't read volume header from control: Invalid handle 2012-01-06T23:06:54.376Z cpu14:114970)FSS: 4333: No FS driver claimed device 'control': Not supported 2012-01-06T23:06:54.396Z cpu14:114970)VC: 1449: Device rescan time 41 msec (total number of devices 12) Resolution :
============ Our initial workaround was to disable the single VAAI Primitive VMFS3.Hardware.AcceleratedLocking on the hosts. From EMC Had to change the affected hosts from Failover Mode 1 to Failover Mode 4 (within the Unisphere Storage System Connectivity Status menu). Once the hosts were changed to Failover Mode 4, we re-enabled the VAAI Primitive and no longer encountered the issue.
No comments:
Post a Comment