Unsolved
This post is more than 5 years old
1 Message
0
1843
December 22nd, 2015 16:00
Failure of Primary MDM and tiebreaker + GFS2
In my very basic example setup, I have three MDM nodes (primary [mdm1], secondary [mdm2], tie-breaker [mdm3]), together with SDS nodes. I have two separate machines running SDC (call them sdc1 and sdc2).
Furthermore, the SDC nodes are in a linux cluster which use the exposed drive /dev/scinia in a clustered logical volume, which is then mounted using GFS2.
In the event of primary MDM failure, the system rebalances correctly and continues working normally.
However, in the event that the tie-breaker MDM also fails (leaving mdm2 as the only alive node), GFS2 nodes withdraw from the cluster requiring fsck on the filesystem to repair it. Once the failed MDMs are operational again, GFS2 will not start up at all, causing long delays during reboots, the inability to unmount the file system, etc.
My question is, can ScaleIO function with only one MDM node in the event the other two MDMs fail? Is there a configuration item that can be set to enable it? The non-disruptive-upgrade guide only mentions one MDM node being inactive at a time.
As a bonus, is there any documentation about GFS2 working together with ScaleIO?
Thanks.
tanr1
9 Posts
1
December 23rd, 2015 06:00
I don't think the issue is whether ScaleIO can function with only one MDM node. It may have to do with you have failed more than half of your SDS. I'm new to ScaleIO and I don't have your answer but for your test, you should first add more SDS nodes.
alexkh
60 Posts
0
December 23rd, 2015 07:00
ScaleIO can function with one MDM in case it is in single mode, however this is highly not recommended.
When MDM was configured in a cluster mode, ScaleIO needs at least 2 out of three nodes (voters) to function, otherwise you might be having a split brain scenario.
In case an MDM fails, we have a script which can be configured on more nodes and use them as stanby MDMs. When an MDM fails, one of the standby MDMs will take his place in the cluster.
In our next major version, we are going to introduce 5 nodes cluster which can sustain 5 MDMs and a TB failure.
daverush
51 Posts
1
December 30th, 2015 14:00
mkiric,
The short answer is that yes, ScaleIo can function fully with only one MDM up. Only one is active at a time anyway.
This is in no way recommended, however, as there is no redundancy/failover in case anything happens to that one MDM process.
The long answer is (as tanr mentioned above) that you didn't just remove an MDM or TB from the cluster by killing nodes 1 and 3, you also took down their SDS instances. ScaleIO keeps two copies of the data, which (as a single SDS has at most one copy of the data) means that you must have two SDSes alive in your small SDS/MDM cluster.
Going below 2 SDSes will always result in DU, as going below the required number of SDSes to provide enough storage will result in DU/DL.
I'm not surprised that even after booting all 3 nodes again it's giving you trouble, as ScaleIO isn't meant to have to recover from shutting down without first inactivating the Protection Domain.
In order to do so, you should first restore enough SDSes to complete a rebuild, and make sure no ScaleIO components are in an errored state.
At this point you should be OK to inactivate the PD with "scli --inactivate_protection_domain --protection_domain_name ". This syntax is found in the ScaleIO User Guide.