Iscsi spikes normal?

Question

I'm noticing a lot of spikes on our Equallogic iscsi storage, when I go to the advanced>disk/Virtual disk monitor, we seem to be getting a lot of spikes at ranging from 22-160 ms. The average numbers look good, 2ms and such. But in the course of a few minutes we'll have a number of these (usually) read latencies on a number of vm's on different hosts. I fired up esxtop and looking at disk view under the DAVG i'm seeing a few of my luns hitting again anywhere from 14 all the way up to 166. It's not constant but I can usually find a few with in a few minute period. This seems to be on a few different SAN's. But before i go opening support cases I just wanted to see if this is normal behavior and I'm making a mountain out of a mole hill or if this warrants further investigation.

* Our sans are all a few firmware version behind except one. And it's having similar issues, though not as extreme. We'll be updating the controllers over the weekend.

* We are using the EQL MEM, SAN HQ is showing that everything is fine and not generating any alarms and our IOP counts are low/medium.

* I've been over the EQL best practices guide a few times and verified that we do have everything configured correctly for that document.

* our NIC fimrware and drivers all appear to be up to date on the hosts

* we do notice a performance problem on the servers a lot of times, they can be very sluggish when you are RDP'd into them.

* I'm having our network guy check the switches this traffic passes through, but they are saying all their metrics look good. Our entire network is 10 gig with the iscsi on it's own dedicated NIC's/Switches

* Weirdly though, the average rate for the iscsi data is only 10-15 mb's. At least what ops manager is reporting on the dvs.

admlshake · Answer

Thanks, I will probably do so when we get the firmware updated as that is tyicpally supports goto answer for everything, even if the notes don't mention the specific issue we are having at the time.

Yeah, thats the guide I used to check our settings. We are seeing this on the newer and older volumes, and after I last updated the MEM I went back and verified that all the older datastores had the proper configuration. However I will try removing the targets again on a few hosts and see if we notice any difference after adding it back in.

Most of our servers have the C:\ on one scsi controller and other disks on a different one or two depending on the server. Thank you for your reply.

admlshake · Answer

The IO's seem to be fairly consistent. Even when I see an IO spike I don't always see a latency increase, or at least not much of one in either vmware or SANHQ.

So for the old iscsi settings in the DB, even if I changed those in the vsphere client are you saying they still might persist on the hosts?

admlshake · Answer

Like I said, they are about normal. The IOP's on all our sans except one are "low" by SANHQ's measurements, and usually only a fraction of what the estimated overall total IOP count can max out at. The other ones has only gotten to about 50-60% of it's IOP max.

I guess I don't understand why it would still say those settings are configured properly after a reboot if it was populating that data from a database that had the old settings in it. I'm 99% sure we used a script after the last MEM upgrade to make sure all our older Datastores were given the new settings. But it won't cost us much to try it so I'll give it a go. And I believe that the MEM auto configures those settings for any equallogics no?

EqualLogic

Iscsi spikes normal?

Was this post helpful?