Lost Connection to physcal HBAs on R610

After power down Dell PS6000XV   firmware V4.3.4(R112816) and 2 dell R610 boxes,  I have been getting this warning messages from both ESX4.0 servers update 1 messages logs.

PS6000XV and ESX boxes ran good for almost 2 years.

both ESX 4 servers have physcal HBAs connected to an 1Gb switch, the PS 6k also connect to the same switch. The switch ports interface look clean.

 

looking at my dell PS600 san  box, I noticed the iscsi connections time are a few hours, which normally would have been connected for days.

 

looking at the event log of the VM, running windows 2008R2, no errors were reported. looking at the PS6000 event log, no errors are reported.

 

 

What does this mean and shall I be concerned ? is this a bad HBA, bad cable? driver ? I have not patch, or upgraded, update anything on the ESX servers or on the PS6000. I just simply powered it off, and changed the IP address on the ESX console. I also had replaced the existing ethernet cables with brand new cat6 ethernet cables.

 

any feedback is appreciated!

 

JOe

 

 

Nov 15 18:24:26 x2008esx1 vmkernel: 2:01:29:24.773 cpu1:4228)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe: NMP device "naa             .6090a06830c3887d25bad411392d6980" state in doubt; requested fast path state update...
Nov 15 18:24:26 x2008esx1 vmkernel: 2:01:29:24.773 cpu1:4228)ScsiDeviceIO: 747: Command 0x2a to device "naa.6090a06830c3887d             25bad411392d6980" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
Nov 15 18:24:26 x2008esx1 vmkernel: 2:01:29:24.773 cpu1:4228)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x410008183340)              to NMP device "naa.6090a06830c3f8fd17baf40f392d9922" failed on physical path "vmhba1:C0:T2:L0" H:0x5 D:0x0 P:0x0 Possible se             nse data: 0x0 0x0 0x0.
Nov 15 18:24:26 x2008esx1 vmkernel: 2:01:29:24.773 cpu1:4228)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe: NMP device "naa             .6090a06830c3f8fd17baf40f392d9922" state in doubt; requested fast path state update...
Nov 15 18:24:26 x2008esx1 vmkernel: 2:01:29:24.773 cpu1:4228)ScsiDeviceIO: 747: Command 0x2a to device "naa.6090a06830c3f8fd             17baf40f392d9922" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.