Why Redhat replace quorum partition/lock lun with new fencing mechanisms?

[jOe]

Hello all,

Sorry if this is a stupid question.

I deploy both HP MC/SG linux edition and RHCS for our customers. I just
wondered why the latest RHCS remove quorum partition/lock lun with the new
fencing mechanisms(powerswitch,iLO/DRAC, SAN switch....)?
Lots of our customers choosed HP's sophisticated MC/SG linux edition for
their mission critical system in Two Node Cluster Configuration. From our
monthly health check service and customers' feedback, i do think HP SGLX is
reliable and stable,
even under heavy I/O traffic, the lock lun(quorum disk) works pretty good.
And the whole cluster architecture is simple and clean, at same time means
less issue and problem .
I do think Redhat's product team is strong and obviously have their solid
reasons to choose new mechanisms in RHCS v4. I've investigated and i can
understand that quorum disk/lock lun in two node cluster configuration
"Might Bring" more latency and impact the cluster but according to my
previous words, i'm sure that it is pretty stable to use lock lun/quorum
partition of HP SG/LX even under heavy I/O loads.

I have no intention to start a comparison between HP SGLX and RedHat RHCS,
All i want to get clear is quorum disk/lock lun Vs RHCS's new fencing
mechanisms.

Regards,

Jun

[Kevin Anderson]

Post by jOe
Hello all,
Sorry if this is a stupid question.
I deploy both HP MC/SG linux edition and RHCS for our customers. I
just wondered why the latest RHCS remove quorum partition/lock lun
with the new fencing mechanisms(powerswitch,iLO/DRAC, SAN
switch....)?

First off, I don't think it is completely fair to compare quorum
partitions to fencing. They really serve different purposes. Quorum
partition gives you the ability to maintain the cluster through flakey
network spikes. It will keep you from prematurely removing nodes from
the cluster. Fencing is really used to provide data integrity of your
shared storage devices. You really want to make sure that a node is
gone before recovering their data. Just because a node isn't updating
the quorum partition, doesn't mean it isn't still scrogging your file
systems. However, a combination of the two provides a pretty solid
cluster in small configurations. And a quorum disk has another nice
feature that is useful.

That said, a little history before I get to the punch line. Two
clustering technologies were merged together for RHCS 4.x releases and
the resulting software used the core cluster infrastructure that was
part of the GFS product for both RHCS and RHGFS. GFS didn't have a
quorum partition as an option primarily due to scalability reasons. The
quorum disk works fine for a limited number of nodes, but the core
cluster infrastructure needed to be able to scale to large numbers. The
fencing mechanisms provide the ability to ensure data integrity in that
type of configuration. So, the quorum disk wasn't carried into the new
cluster infrastructure at that time.

Good news is we realized the deficiency and have added quorum disk
support and it will be part of the RHCS4.4 update release which should
be hitting the RHN beta sites within a few days. This doesn't replace
the need to have a solid fencing infrastructure in place. When a node
fails, you still need to ensure that it is gone and won't corrupt the
filesystem. Quorum disk will still have scalability issues and is
really targeted at small clusters, ie <16 nodes. This is primarily due
to having multiple machines pounding on the same storage device. It
also provides an additional feature, the ability to represent a
configurable number of votes. If you set the quorum device to have the
same number of votes as nodes in the cluster. You can maintain cluster
sanity down to a single active compute node in the cluster. We can get
rid of our funky special two node configuration option. You will then
be able to grow a two node cluster without having to reset.

Sorry I rambled a bit..

Thanks
Kevin

[jOe]

Post by Kevin Anderson

Post by jOe
Hello all,
Sorry if this is a stupid question.
I deploy both HP MC/SG linux edition and RHCS for our customers. I
just wondered why the latest RHCS remove quorum partition/lock lun
with the new fencing mechanisms(powerswitch,iLO/DRAC, SAN
switch....)?

First off, I don't think it is completely fair to compare quorum
partitions to fencing. They really serve different purposes. Quorum
partition gives you the ability to maintain the cluster through flakey
network spikes. It will keep you from prematurely removing nodes from
the cluster. Fencing is really used to provide data integrity of your
shared storage devices. You really want to make sure that a node is
gone before recovering their data. Just because a node isn't updating
the quorum partition, doesn't mean it isn't still scrogging your file
systems. However, a combination of the two provides a pretty solid
cluster in small configurations. And a quorum disk has another nice
feature that is useful.
That said, a little history before I get to the punch line. Two
clustering technologies were merged together for RHCS 4.x releases and
the resulting software used the core cluster infrastructure that was
part of the GFS product for both RHCS and RHGFS. GFS didn't have a
quorum partition as an option primarily due to scalability reasons. The
quorum disk works fine for a limited number of nodes, but the core
cluster infrastructure needed to be able to scale to large numbers. The
fencing mechanisms provide the ability to ensure data integrity in that
type of configuration. So, the quorum disk wasn't carried into the new
cluster infrastructure at that time.
Good news is we realized the deficiency and have added quorum disk
support and it will be part of the RHCS4.4 update release which should
be hitting the RHN beta sites within a few days. This doesn't replace
the need to have a solid fencing infrastructure in place. When a node
fails, you still need to ensure that it is gone and won't corrupt the
filesystem. Quorum disk will still have scalability issues and is
really targeted at small clusters, ie <16 nodes. This is primarily due
to having multiple machines pounding on the same storage device. It
also provides an additional feature, the ability to represent a
configurable number of votes. If you set the quorum device to have the
same number of votes as nodes in the cluster. You can maintain cluster
sanity down to a single active compute node in the cluster. We can get
rid of our funky special two node configuration option. You will then
be able to grow a two node cluster without having to reset.
Sorry I rambled a bit..
Thanks
Kevin
--
Linux-cluster mailing list
https://www.redhat.com/mailman/listinfo/linux-cluster

Thank you very much Kevin, your information is very useful to us and i've
shared it to our engineer team.
Here are two questions still left:
Q1: In a two node cluster config, how does RHCS(v4) handle the heartbeat
failed ? (suppose the bonded heartbeat path still failed by some bad
situations).
When using quorum disk/lock lun, the quorum will act as a tier breaker and
solve the brain-split if heartbeat failed. Currently the GFS will do this ?
or other part of RHCS?

Q2: As you mentioned the quorum disk support is added into RHCS v4.4 update
release, so in a two-nodes-cluster config "quorum disk+bonding
heartbeat+fencing(powerswitch or iLO/DRAC) (no GFS)" is the recommended
config from RedHat? Almost 80% cluster requests from our customers are
around two-nodes-cluster(10% is RAC and the left is hpc cluster), We really
want to provide our customers a simple and solid cluster config in their
production environment, Most customer configure their HA cluster as
Active/passive so GFS is not necessary to them and they even don't want GFS
exists in their two-nodes-cluster system.

I do think more and more customers will choose RHCS as their cluster
solution and we'll push this after completely understand RHCS's technical
benefits and advanced mechanisms.

Thanks a lot,

Jun

[Kevin Anderson]

Post by jOe
Thank you very much Kevin, your information is very useful to us and
i've shared it to our engineer team.
Q1: In a two node cluster config, how does RHCS(v4) handle the
heartbeat failed ? (suppose the bonded heartbeat path still failed by
some bad situations).

Current configuration requires using power fencing when running the
special case two node cluster. If you lose heartbeat between the two
machines, both nodes will attempt to fence the other node. The node
that wins the fencing race gets to stay up, the other node is reset and
won't be able to re-establish quorum until connectivity is restored.

Post by jOe
When using quorum disk/lock lun, the quorum will act as a tier breaker
and solve the brain-split if heartbeat failed. Currently the GFS will
do this ? or other part of RHCS?

Quorum support is integrated in the core cluster infrastructure so is
usable with just RHCS. You do not need GFS to use a quorum disk.

Post by jOe
Q2: As you mentioned the quorum disk support is added into RHCS v4.4
update release, so in a two-nodes-cluster config "quorum disk+bonding
heartbeat+fencing(powerswitch or iLO/DRAC) (no GFS)" is the
recommended config from RedHat? Almost 80% cluster requests from our
customers are around two-nodes-cluster(10% is RAC and the left is hpc
cluster), We really want to provide our customers a simple and solid
cluster config in their production environment, Most customer
configure their HA cluster as Active/passive so GFS is not necessary
to them and they even don't want GFS exists in their two-nodes-cluster
system.

If you have access to shared storage, then a two node cluster with
quorum disk/fencing would be a better configuration and could be the
recommended configuration. However, there are still cases where you
could have a two node cluster with no shared storage. Depends on how
the application is sharing state or accessing data. But for an
active/passive two node failover cluster, I can see where the quorum
disk will be very popular.

Kevin

[jOe]

Post by Kevin Anderson

Post by jOe
Thank you very much Kevin, your information is very useful to us and
i've shared it to our engineer team.
Q1: In a two node cluster config, how does RHCS(v4) handle the
heartbeat failed ? (suppose the bonded heartbeat path still failed by
some bad situations).

Current configuration requires using power fencing when running the
special case two node cluster. If you lose heartbeat between the two
machines, both nodes will attempt to fence the other node. The node
that wins the fencing race gets to stay up, the other node is reset and
won't be able to re-establish quorum until connectivity is restored.

Post by jOe
When using quorum disk/lock lun, the quorum will act as a tier breaker
and solve the brain-split if heartbeat failed. Currently the GFS will
do this ? or other part of RHCS?

Quorum support is integrated in the core cluster infrastructure so is
usable with just RHCS. You do not need GFS to use a quorum disk.

Post by jOe
Q2: As you mentioned the quorum disk support is added into RHCS v4.4
update release, so in a two-nodes-cluster config "quorum disk+bonding
heartbeat+fencing(powerswitch or iLO/DRAC) (no GFS)" is the
recommended config from RedHat? Almost 80% cluster requests from our
customers are around two-nodes-cluster(10% is RAC and the left is hpc
cluster), We really want to provide our customers a simple and solid
cluster config in their production environment, Most customer
configure their HA cluster as Active/passive so GFS is not necessary
to them and they even don't want GFS exists in their two-nodes-cluster
system.

If you have access to shared storage, then a two node cluster with
quorum disk/fencing would be a better configuration and could be the
recommended configuration. However, there are still cases where you
could have a two node cluster with no shared storage. Depends on how
the application is sharing state or accessing data. But for an
active/passive two node failover cluster, I can see where the quorum
disk will be very popular.
Kevin
--
Linux-cluster mailing list
https://www.redhat.com/mailman/listinfo/linux-cluster

Thank you very much.

Jun

[Herta Van den Eynde]

Re: [Linux-cluster] Why Redhat replace quorum partition/lock lun with
new fencing mechanisms?

Post by Kevin Anderson
If you have access to shared storage, then a two node cluster with
quorum disk/fencing would be a better configuration and could be the
recommended configuration. However, there are still cases where you
could have a two node cluster with no shared storage. Depends on how
the application is sharing state or accessing data. But for an
active/passive two node failover cluster, I can see where the quorum
disk will be very popular.
Kevin

When configuring the cluquorumd for a two node cluster (active-active
nfs server), the GUI recommends using a network tiebreaker ip address.
Why is that?

Under heavier network load, we occasionally see one of the members
(usually the highest priority member) reporting that the connection to
the tiebreaker is offline. It subsequently gets fenced by the other
node, and simply reboots.
(FWIIW, we checked the network cards, cables, and switch and
swithc-ports between the two nodes. The system that holds the TB
address is currently waiting to be re-installed, so it's pretty much idle.)

I thought the network tiebreaker was meant to avoid a split-brain
cluster, but if it isn't, needless to say, we'd be happy to get rid of it.

Kind regards,

Herta

Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm

[Troels Arvin]

Hello,

Good news is we realized the deficiency and have added quorum disk support
and it will be part of the RHCS4.4 update

Has RHCS 4.4 been released yet?

I'm asking because I'm unsure if Update 4 of RHEL 4 means that RHCS 4 is
now also at Update 4.

[Riaan van Niekerk]

Post by Troels Arvin
Hello,

Good news is we realized the deficiency and have added quorum disk support
and it will be part of the RHCS4.4 update

Has RHCS 4.4 been released yet?
I'm asking because I'm unsure if Update 4 of RHEL 4 means that RHCS 4 is
now also at Update 4.

The RPMs (via up2date/RHN) have been released, even though the ISOs for
RHCS 4.4 and GFS 6.1 u4 have not been released yet. Red Hat Support told
me they should be out within a week or so.

[Troels Arvin]

Post by Riaan van Niekerk

Post by Troels Arvin
Has RHCS 4.4 been released yet?

The RPMs (via up2date/RHN) have been released, even though the ISOs for
RHCS 4.4 and GFS 6.1 u4 have not been released yet. Red Hat Support told
me they should be out within a week or so.

OK. So if I run an "update -u" today on my servers running RHCS, I'll be
at RHEL 4.4 and RHCS 4.4.

I'm curious to read about the new quorum disk feature. Is it documented
somewhere yet? - The docs for RHCS don't seem to have been updated yet:
http://www.redhat.com/docs/manuals/csgfs/