7017617: Simulating a Cluster Network Failure

To simulate a cluster communication failure use iptables and drop the packages on the IP address that is configured for the communication.

Assuming the setup would be to use

Node A uses local IP 192.168.20.193 for cluster communication

Node B uses local IP 192.168.20.228 for cluster communication

The idea is to block the communication of the nodes. This can be done by implementing a Firewall Rule on one node, to

not send to the other ip

and

not receive from the other ip

Coming from the above Example with Node A and Node B, one can implement this by setting on Node B

iptables -A INPUT -s 192.168.20.193 -j DROP; iptables -A OUTPUT -d 192.168.20.193 -j DROP

which means that all Traffic coming from source 192.168.20.193 , which is Node A, and all Traffic going to 192.168.20.193, which is Node A, will be dropped by the Kernel on Node B.

This breaks the Cluster Communication apart without removing or influencing any relevant local Network settings and without System Notification to any Service, Socket or Application.

For the cluster stack this appears to be a split brain.

You can at any time with

iptables -F

flush the iptables rules to remove this.

Which might be especially useful as a split brain might lead to the node with the iptables rules being the survivor. But this means that the other node reboots into a split brain and might reboot the formerly surviving node because of startup fencing.

Keep in mind that -F removes all these Rules, so using the iptables / Firewall for something else might have an affect on other Areas.

Please also keep in mind that if the IP’s used for cluster communication are also used for Applications then there might not only be a Cluster Split Brain, but also a Resource Failure.

Related:

7017617: Simulating a Cluster Network Failure (COROSYNC)

To simulate a cluster communication failure use iptables and drop the packages on the IP address that is configured for the communication.

Assuming the setup would be to use

Node A uses local IP 192.168.20.193 for cluster communication

Node B uses local IP 192.168.20.228 for cluster communication

The idea is to block the communication of the nodes. This can be done by implementing a Firewall Rule on one node, to

not send to the other ip

and

not receive from the other ip

Coming from the above Example with Node A and Node B, one can implement this by setting on Node B

iptables -A INPUT -s 192.168.20.193 -j DROP; iptables -A OUTPUT -d 192.168.20.193 -j DROP

which means that all Traffic coming from source 192.168.20.193 , which is Node A, and all Traffic going to 192.168.20.193, which is Node A, will be dropped by the Kernel on Node B.

This breaks the Cluster Communication apart without removing or influencing any relevant local Network settings and without System Notification to any Service, Socket or Application.

For the cluster stack this appears to be a split brain.

You can at any time with

iptables -F

flush the iptables rules to remove this.

Which might be especially useful as a split brain might lead to the node with the iptables rules being the survivor. But this means that the other node reboots into a split brain and might reboot the formerly surviving node because of startup fencing.

Keep in mind that -F removes all these Rules, so using the iptables / Firewall for something else might have an affect on other Areas.

Please also keep in mind that if the IP’s used for cluster communication are also used for Applications then there might not only be a Cluster Split Brain, but also a Resource Failure.

Related:

Gre Tunnel Cisco Linux traffic forwarding

I setup a gre tunnel a cisco router and a Linux machine, the tunnel interface in the Linux box named pic.
Well i have to forward traffic coming from cisco through the Linux box.
the rules i’ve set in the Linux box is described as follow:


echo "1" > /proc/sys/net/ipv4/ip_forward
iptables  -A INPUT -p 47 -j ACCEPT
iptables  -A FORWARD -i ppp0 -j ACCEPT
iptables  -A FORWARD -i pic  -o ppp0 -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables  -A FORWARD -i ppp0 -o pic -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables  -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -t nat -A POSTROUTING -o ppp0 -j MASQUERADE


I see the traffic coming from tunnel and forwarded to internet but no reply from sent packet.

May i miss something like a routing rule.

Related: