Best way to deal with mixed speed network (10Gb & 1Gb)

Hi ,

Let me check if I am able to get anything on this

Thanks

Vanitha

Few questions:

1. Why do you have the two 10Gb interfaces in a LACP channel?  I certainly don't think you need the bandwidth aggregation.  Hopefully it's for port redundancy.
2. I'm assuming you have the Neterion XBlade 65 (Tempest) 10Gb optical NIC (in the Celerra config, it's listed as a "fxg" port).  What code version are you running?
3. Celerra is not likely to be sending the pause frames, as link negotiation is disabled by default, so someone would have had to deliberately set it to enabled.  That would be pointless if the switch was not similarly set.  You would look at the following:

$ server_sysconfig server_X  -pci

     to see what the setting is for link negotiation.

   4.  What kind of client is it?  What kind of mount is it?

Remember that for TCP window sizing, it is the client that advertises its receive window size, and the Data Mover will match the max transmit window to the max receive window of the client.  If it becomes impossible, or proves to be ineffective, trying to determine a good receive window on the client, the Data Mover can be forced to a specific send window size regardless of what the client advertises.

A Professional Services network analysis and tuning is probably your best bet to maximize performance.

were you able to fix this issue? I am running into a similar issue. The Network Team says that there is nothing what they can change on the Switch side and I need to validate the host/VNX Storage for this issue.

Bill's comment above is still valid - a Professional Services engagement with network tuning may be likely.  Are you using 10GbE interfaces?  Can you provide background on your configuration, issue and troubleshooting so far?

in my opinion the network team always says the network is fine - until you prove otherwise

you need to look at the detailed statistics and logs from both the switch side and the VNX

we have a VNX5200 Data Movers which has the 10Gbe interface connecting to a Nexus 5k. The nexus 5K is configured globally for Jumbo frames but the VNX5200 Data movers are configured for 1500MTU becuase the host is 1Gbps and they are doing 1500MTU. Host is Cent OS 6.6 and its mounting a NFS export from my VNX5200.

While performing read operations I see a lot of re-transmission on the VNX5200 stats command. EMC Unified Performance team reviewed the stats and tcpdump I collected form the VNX5200 and they are pointing towards the network.

Notes from EMC support

The customer should investigate the cause of the frame delivery issues. The customer should check if any network device is unexpectedly discarding frames due to buffer discards, errors, QoS, etc. or if the frame delivery issues could be caused by saturation of the WAN bandwidth.

Network Team reviewed the Switches and they did not find any changes which can be done to improve the situation. Currently we are working on getting a 1Gb Modules for Data movers.

Anand Baghel.

Yes I have tried to setup MTU of 9000 for Host and the VNX5200 Data Movers. Did not see any improvement.

Have you tried configuring simply setting the datamover interfaces to use Jumbo Frames?  It seems as though you're going to force the switch to fragment anyway, since it's MTU is 9000, but devices on both sides (CentOS host and datamover) are doing 1500byte MTUs anyway....

We have been fighting the exact same problem with the 5k's connected to the VNXs at 10Gb, and clients on the 2k's at 1Gb for years now. MTU has no effect on the problem. We did however identify part of the problem with the 2k's with the smaller buffers as I think someone else noted, the EFEX's do help, but it doesn't eliminate the problem entirely. I am going to try setting the send window as well and see if things improve. BTW, I noticed this very recently between 2 VNXs doing IP replication, each VNX at 10Gb, crossing some 1Gb links in between sites, literally 10's of thousands of retrains and dup acks during the replication, pathetically slow transfers of like 20MB/sec when the distance between these sites is a few miles and not a congestion problem, FWIW. Will report back findings if we get this ironed out. Thanks a lot ebrundick2 for the insight, I'm sure it's a match to what we are seeing.

Adding more to the discussion- As it turns out our network team was able to enact a change from the Cisco Nexus core switches, something like "no hardware queue" which was configured in reference to the FEX switch uplinks, which disables the FEX switches from performing any hardware queueing, relying on the Nexus core entirely for queueing/buffering.

Once that parameter was enabled, speeds with sndcwnd=0 went up above 110MB/sec for hosts on the server network (attached to FEX switches which link back to the Nexus).  So this represented some sort of queueing issue on the part of the network equipment, and I think it's a foregone conclusion that you will encounter these sorts of "buffer" or "queue" limitations in your network any time you go from 10GbE to 1Gbps ethernet.

Thanks

Great feedback

Our network team got some suggestions from Cisco support that implied the risk of port starvation if the HW queuing was disabled outweighed the semi-limited problem we are fighting, so, they will not disable it on anything. Looking to limit the sndcwnd this weekend on one our VNXs and run some tests.

Update: Set the tcp sndcwnd to 65535 and have observed "0" duplicate acks or retransmissions of any kind between these sites when IP Replicator is running. The 1Gb link is saturated now for at least 50% of the session which resulted in the overall duration being cut by ~60% (9 hours). With the nature of the files, this is far better than it was before. No client-side issues noted yet. Thanks

This worked!

I've been working with our VNX 5200s for over a year now trying to figure out why performance was so poor.  Even our 7 year old Win2k3 servers were outperforming it.

Our environment is similar - we also have Nexus 5548 core switches with 2k FEX. 

I've had countless tickets open with EMC techs.  No one could point to a resolution.  We could see that retransmissions plagued us - between 1-4%, but no one knew what to do about it.  EMC technicians would write it off as a misconfigured network.  Cisco couldn't find anything wrong.  I finally brought in a wireshark expert to help pinpoint where the problem was - but we couldn't see any errors other than retransmissions on the link. 

I made the change like ebrundick2 suggested "server_param server_2 -facility tcp -modify sndcwnd -value 65535" and rebooted the datamovers.  Retranmissions went to almost zero, and CPU went down into the 20% range.

When I told the tech I was going to try this suggestion to modify the sndcwnd value, he was skeptical and said that this "enters into the realm of fine tuning".  Fine tuning it may be - but it fixed our environment. 

Thank you for finding this.