VLT split-brain after OS10 upgrade on one S5248F node?

Hi,

I'm not that good at networking, but I'll try to help out. Ultimately, if we're unable to locate the issue, or it may be the firmware issue, you will need to contact the support line to log a case up. 

I looked up the user guide on 10.5.5, it mentioned to make sure both peer devices are running on same firmware (page 1819: https://dell.to/3NJ4g5z). Can you check on that. 

Thanks but both firmware are on latest -26 before attempting updating NOS.
I cannot open support ticket because both nodes are out of support (much longer story).
Currently trying with latest 10.5.3 because suspect this might be related with silly decision of Dell to change default MTU for OS10 in 10.5.4.4.

Kind of expected, updating one node to 10.5.3.9 went without issues, VLT converged just fine.
Next would be latest 10.5.4 but before I probably plan to modify MTU for mgmt 1/1/1 (HB) to jumbo...

Little follow-up if anybody reading has an idea, Google basically only gives me this forum topic now ;)

MTU doesn't look like culprit here:
Updated both to v10.5.3.9 and all good as mentioned.
If I try to update to 10.5.4.10 on one node, I get same problem as with 10.5.5.7.

This time managed to get little bit more logging out of the attempt:
 - Some time after boot VLT backup link goes up and ping from reloaded switch works. Interesting thing is that mgmt1/1/1 stays on 1532 MTU (possibly by design although default is changed to 9216 on reloaded switch).
- VLTi interfaces stay down on both sides (VLT failure)
- VLT is in split-brain with both nodes set as primary.
- log has nothing relevant (to me that is)
- MTU of VLTi is and was from start automatically on 9216

As mentioned have latest switch firmware deployed as first step and officially out of clues.

One thing that I was able to find now is that after 10.5.1 in the user manual there seems to be a recommendation to:
"

While configuring a VLT MAC address, if the 8th bit of the MAC address is a 1, then the MAC address is considered to be a multicast MAC address. There are locally defined MAC addresses. For these addresses, the second least significant bit in the first byte must be a 1, which signifies a locally defined address.

The correct MAC addresses must have xxxxxx10 bits set in the first octet, such as x2, x6, xA, xE, and so on.

While manually configuring MAC addresses for VLT, make the 7th bit a 1 - to signify a locally assigned address - and the 8th bit a 0 - to signify a unicast address - which essentially means that you must use one of the following formulas
"

We use vlt-mac "00:00:00:00:00:02" but still not sure how this could break VLT on upgrade of OS10 on one node from 10.5.3 -> 10.5.4? There is nothing in logs or anywhere that 10.5.4 would start penalize "wrong" MAC format here?
Not sure if changing mac on-the-fly now is something that would not cause network outage?

Hello, 

Did you ever get a resolution to this?

I have a couple of Dell S4128F switches in a VLT configuration that are needing upgraded but after reading this I'm now cautious about going to 10.5.4.4 or later. 

In the terms of the VLT Mac Address we never manually set this during the initial VLT configuration, the system appears to have set this automatically.  

We haven't changed the default MTU either, so ours are still at 1532. 

I do remember reading somewhere within one of the OS10 release notes which stated a loop could occur if a static LAG was used. I know the VLTi port-channel interface between both switches use a static LAG by default but as far as I know this was always recommended by Dell?

Hi Greig

No still puzzled even more when google starts to spit this topic as one of only few results on any query related to "Dell OS10, vlt, split brain" ;)
Not an invitation to just do it but, knowing my luck lately, I'm kind of sure you will not have any issues ;)
Automatic MAC can be kept and is one of the nodes system MAC (don't remember higher or lower one on top of my head) which is AFAIR not recommended to keep, because it slows down primary election on e.g. cluster boot.
In the meantime got new pair of S5232f but already with 10.5.5.5 so wasn't keen on downgrading and possibly testing when it was blocking some other stuff to be installed.

Hi Nenard, 

Thanks for the update. 

Yeah, for automatic VLT Mac Address it uses the primary nodes Mac. I don’t know what the implications of manually changing this would do but given that it’s working and we haven’t experienced any issues I’ll probably leave it as is. 

What to you have in terms of Spanning Tree and VLT priorities? 

For us our S4128F pair acts as our Core/Distribution and we are using RSTP with the primary node set to 4096 and the secondary node set to 8192.

We have also used the same values for our VLT priorities as well. 

Hi @Greig_Mitchell​ 

True, automatic VLT MAC is AFAIK fully supported.
STP is almost the same as on your side, RSTP with primary on 0, sec on 4096, all other switches in network higher than that.
VLT primary prio. is 1 and 65535

We never had any issues with VLT except when I try to upgrade to 10.5.4.

Some updates:

I'm unable to reproduce the problem on freshly bought (second facility) pair of S5248f with identical configuration as used on my pair but of course isolated from rest of network (just connected to each other) ;)

Not sure why to assume here, since we are only left with Dell OS6 and Dell Os10 network devices.

Now also Dell OS6 is not compatible with Dell OS10?

Only difference that I can imagine is that my switches have HW rev 02 and new ones have it on 04.