vSphere 4 U1 - NMP generating pathing errors

Hi guys,

We're seeing some NMP errors on our vSphere 4 hosts and I was wondering if anyone else had seen similar errors. Every few days we see this group of three errors reported in the ESX logs:

May 30 02:50:00 nzchchlmpsesx5 vmkernel: 1:15:34:16.470 cpu6:4255)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x41000201c880) to NMP device "naa.60060480000290102936533030314433" failed on physical path "vmhba1:C0:T0:L12" H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

May 30 02:50:00 nzchchlmpsesx5 vmkernel: 1:15:34:16.470 cpu6:4255)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe: NMP device "naa.60060480000290102936533030314433" state in doubt; requested fast path state update...

May 30 02:50:00 nzchchlmpsesx5 vmkernel: 1:15:34:16.470 cpu6:4255)ScsiDeviceIO: 747: Command 0x28 to device "naa.60060480000290102936533030314433" failed H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

From my (limited) understanding vSphere is attempting a read down a path which is failing with an HBA internal error (H:0x7). I've got calls open with VMware, HP and EMC and haven't had a lot of success getting a resolution of this issue.

The environment consists of:

OS: vSphere 4 Update 1 plus patches (using local boot)

Host: BL460 G1 with LP-1105 HBAs on firmware 2.82A4

C7000 Chassis with Brocade 4/24 SAN switch (AE372A) on FOS 6.1.1b

Main Fabric is Brocade 4900 switches running FOS 6.2.0c

Symmetrix DMX4 storage array on 5773 microcode

We've got this set up in two separate DataCenters and are seeing the issue on hosts in both environments, which to my mind reduces the chance of this strictly being a hardware fault. Also we're currently migrating from ESX 3.5 to vSphere 4 so we have a mixed environment of both types of hosts at the moment.

Any suggestions would be most welcome.

Cheers,

Duncan