Performance Issues with VMware Environment

Hi Quincy,

Thank for your response. Could you please tell me where exactly this setting is as I checked with VMware Team and we not able to find the setting option.

Thanks

Are the datastores on the VMAX using FAST VP? Are these LUNs perhaps still sitting on the lowest tier, like SATA? It takes a while for data to settle on the right tier.

Since it seems you have already ruled out the backend, I would look at the front-end.  I would have said to look there first even if you had not ruled out the backend.

You may need to add FA CPUs and/or Symm devices (striped meta volumes).

Also if you are using the native path manager, there is a parameter which sets the number of IOs per path before a switch.  It is set to 1000 by default.  It should be set to something much lower on Symm, like 1-8.

Could you make use of more smaller host LUNs?  A 40 way meta is probably too wide.  16 way is plenty for performance.  Maybe 4 1TB LUNs rather than 1 4TB LUN?

Also you should try and keep your IO sizes smaller than 256K on the Symm.  I would make your block size 64k or 128k.

I am not sure where the round robin policy setting is, but I can try and find out.

We have 8MB block size on CX disk and 1 MB block size on the VMax Disk. Does this make any diference.

Here is what we have for the configurations:

ESX UCS blades are with 64GB RAM,  5 UCS Blades. Each UCS Blade has 12CPU X2.925Ghz Intel xion Processor

VM thin Pool is of 898 Striped Datadevices made out of 236 Fiber channel Disks. Cores per socket is 6.

ESXi5.

Each USC blade is mapped 4 FA ports on Vmax.

4 TB MetaLun made out of 40X10GB  thin Devices.

We dont have tiering on this pool.

No Fast Polciy attached to the Devices or the Storage groups on the VM side.

We have migrated the TDEV from one pool to another just verify it as the Pool is not issue here.  Even then we have issues. The VM is moved to Datastore back on CX and the Performance just improved.

oops, totally missed that. I thought you only migrated it back to CX

Check this document.

http://www.emc.com/collateral/hardware/white-papers/h8119-tuning-vmware-symmetrix-wp.pdf

What was the size of your datastores and how many VM's were on a datastore on the CX compared to the 4TB new datastore on the VMAX?

This may not be the case but i'm thinking you had smaller datastores with less VM's so you queue depth from the VMware host was sufficient for the smaller datastores.  Now that you have more VM's and more I/O the performance is degraded. 

Are you using SIOC?  If so what is your response time threshold and if you look in the storage performance is VMware throttling your queue depth on that datastore?

is there any FA  flag setting special to vmware

cx4960-5TB with extents

vmax 4TB WITH 40 100gb tdv

same number of io.

You can check the elab settings for vmware, but I doubt those are making any difference in performance if you are getting IO, other than the speed setting for the port.

I believe your problem is being able to drive queue depth.  With extents you have an aggregate of the queue depth from all extents/LUNs in the datastore (32 x # of extents/luns).  Since you only have one large datastore that you migrated to you only have the queue depth for that single lun in the datastore which is by default 32.

What you could do is try bumping the queue depth from the default 32 to 64 on the new datastore to see if that makes a difference.  I believe adjusting the queue depth manually from 32 goes away from best practice so you may only want to use this for testing and then possibly re-architect based on your findings.

Hi Garret,

Thanks for your email. In my earlier email I stated that the extents are on the Clariion Disks which are performing very better(Raid 10). While the Vmax Disks(Raid 5) not part of FAST Tier are having issues. We should be testing Queue depth Values so that we can validate the Issues at some point.

We do have call with EMC on this issue and we will see if we get some revised tuning suggestion on this and I would share them.

Thanks again.

were you able to resolve this?  I'm curious to what it was as we will be adding capacity to our VMware environment from our VMAX at some point in the future.