Problem Creating Datastores on VNX5600 / ESXi 6.5

Hi All,

Basic situation is this. We have added a new cluster of ESXi hosts to our environment. We are in turn trying to create new some datastores for these hosts. The hosts are properly masked on the storage side. The ESXi hosts can properly see the devices and the paths when I perform a scan on the HBA. However when I go to create the datastore, it is very very slow and often time times out. I have tried mapping different hosts individually to different LUNs and creating a datastore that way, to rule out signature issues from other hosts (I cant even reliably create the datastores though). The create datastore wizard just hangs at "loading" at the partition information screen and finish the creation process. When I walk through datastore creation process on the working hosts, the datastore creation process is immediate and smooth as butter.

ESXi = 6.5.0, 7388607
vCentre = 6.7.0.1150 also tried on a different vCentre version, 6.5.0 13000 build 8024368

Poweredge R730's - All firmware and drivers are identical between working and non working hosts
VASA API version is 1.5 on both sets of hosts

Now here is where I am completely confused. We have another older cluster of several hosts, and every single host in that cluster works perfectly fine. I can create new LUNs on the storage side and create datastores on those LUNs without issue. These hosts are using the same Qlogic QLE2560 HBA's, same driver version, same firmware version. As far as I can tell they are identical. At this point both sets of hosts are also running the same version of ESXi (6.5.0, 7388607) custom Dell/EMC ISO.

Now that said, here is the only difference I can immediately spot. The hosts that work, were upgraded from ESXi 5.5. They worked prior to the upgrade and they continue to work just fine now that they are upgraded to the  6.5.0, 7388607 build. So I am not sure if there was a previous driver/integration component on these working hosts that was retained during the upgrade and that is what is allowing these hosts to continue to work fine.

What I have verified and compared:

- QLE 2560 firmware and driver version is the same between both sets of hosts
- Same ESXi builds
- Same sets of FE ports are being used on the storage side
- LUN masking is the same on the VNX between working and non-working hosts
- Switch zoning is the same on both sets of hosts

What I have tried.

- Create a new storage group with a single host
- Create a new LUN and mask it to just the single host storage group
- Rescan HBA on host, host see's the device and proper pathing
- Update the host information on the VNX (not sure if this is necessary but I wanted to force the SAN to see all up-to-date host information. Have tried this in reverse order as well)
- Walk through the datastore creation process:

- Create datastore, choose VMFS, choose device,  choose VMFS 6, and then I get the "loading..." progress bar and it eventually times out with a back-end service provider timed out error message: The query execution times out because of a back-end property provider 'com.vmware.vsphere.clients.storage.impl.DatastorePropertyProvider' which took more than 120 seconds. Why do I not have this problem on the other hosts? I have also tried to create the datastore local on the servers (by-passing vcentre) and same result.

I am have never seen experienced this issue before. Although the past several years have been mainly iSCSI, I do not remember experiencing issues like this with FC either. So obviously, somewhere in this configuration something is amiss. I am going to be trying to reinstall a host today with VMware's 6.7 release. I doubt it will make a difference but I am hoping it may be a dell/emc custom ISO issue though I doubt it, because as mentioned, the hosts that are currently working were upgraded from a native VMware 5.5 build to the 6.5.0, 7388607 Dell/EMC iso.

Any suggestions, pointers or feedback please!