7017617: Simulating a Cluster Network Failure

To simulate a cluster communication failure use iptables and drop the packages on the IP address that is configured for the communication.

Assuming the setup would be to use

Node A uses local IP 192.168.20.193 for cluster communication

Node B uses local IP 192.168.20.228 for cluster communication

The idea is to block the communication of the nodes. This can be done by implementing a Firewall Rule on one node, to

not send to the other ip

and

not receive from the other ip

Coming from the above Example with Node A and Node B, one can implement this by setting on Node B

iptables -A INPUT -s 192.168.20.193 -j DROP; iptables -A OUTPUT -d 192.168.20.193 -j DROP

which means that all Traffic coming from source 192.168.20.193 , which is Node A, and all Traffic going to 192.168.20.193, which is Node A, will be dropped by the Kernel on Node B.

This breaks the Cluster Communication apart without removing or influencing any relevant local Network settings and without System Notification to any Service, Socket or Application.

For the cluster stack this appears to be a split brain.

You can at any time with

iptables -F

flush the iptables rules to remove this.

Which might be especially useful as a split brain might lead to the node with the iptables rules being the survivor. But this means that the other node reboots into a split brain and might reboot the formerly surviving node because of startup fencing.

Keep in mind that -F removes all these Rules, so using the iptables / Firewall for something else might have an affect on other Areas.

Please also keep in mind that if the IP’s used for cluster communication are also used for Applications then there might not only be a Cluster Split Brain, but also a Resource Failure.

Related:

7017617: Simulating a Cluster Network Failure (COROSYNC)

To simulate a cluster communication failure use iptables and drop the packages on the IP address that is configured for the communication.

Assuming the setup would be to use

Node A uses local IP 192.168.20.193 for cluster communication

Node B uses local IP 192.168.20.228 for cluster communication

The idea is to block the communication of the nodes. This can be done by implementing a Firewall Rule on one node, to

not send to the other ip

and

not receive from the other ip

Coming from the above Example with Node A and Node B, one can implement this by setting on Node B

iptables -A INPUT -s 192.168.20.193 -j DROP; iptables -A OUTPUT -d 192.168.20.193 -j DROP

which means that all Traffic coming from source 192.168.20.193 , which is Node A, and all Traffic going to 192.168.20.193, which is Node A, will be dropped by the Kernel on Node B.

This breaks the Cluster Communication apart without removing or influencing any relevant local Network settings and without System Notification to any Service, Socket or Application.

For the cluster stack this appears to be a split brain.

You can at any time with

iptables -F

flush the iptables rules to remove this.

Which might be especially useful as a split brain might lead to the node with the iptables rules being the survivor. But this means that the other node reboots into a split brain and might reboot the formerly surviving node because of startup fencing.

Keep in mind that -F removes all these Rules, so using the iptables / Firewall for something else might have an affect on other Areas.

Please also keep in mind that if the IP’s used for cluster communication are also used for Applications then there might not only be a Cluster Split Brain, but also a Resource Failure.

Related:

7022442: Why do I get a “Another Firewall Active” message when using Docker with SuSEfirewall2 ?

This document (7022442) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 12 Service Pack 2 (SLES 12 SP2)

SUSE Linux Enterprise Server 12 Service Pack 3 (SLES 12 SP3)

Situation

While attempting to start SuSEfirewall2 in YaST after starting Docker an error is received: Another Firewall Is Active

Resolution

Reboot and then restart the service one more time so that the Docker service is no longer trying to work with iptables natively and starts working through it with SuSEfirewall2.

Cause

Docker on SLES 12 SP2 and SP3 modifies iptables in order to allow networking into and out of containers. If SuSEfirewall2 is started from the beginning, Docker will modify iptables with SuSEfirewall2. If it is not started, Docker will modify iptables directly.

If SuSEfirewall2 is started after Docker, then an error is received in YaST that, another firewall (iptables) is active. If SuSEfirewall2 is started anyway, the Docker containers may lose inbound and outbound connectivity as those rules are no longer available.

Disclaimer

This Support Knowledgebase provides a valuable tool for NetIQ/Novell/SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented “AS IS” WITHOUT WARRANTY OF ANY KIND.

Related:

WinCollect Agent Unable to Communicate with QRadar Appliance

Hey,

I have a WinCollect Agent, version 7.2.2-2, on a Windows Server 2008 (build 6002 sp2). We’re trying to have our WinCollect agent send logs to QRadar, but we’re running into some connectivity issues; we’ve tried a few things.

We checked the Events created by our WinCollect agent on QRadar and we see heartbeat messages sent from QRadar to the WinCollect agent as indicated by the payload

> msg=ApplicationHeartbeat

So we know our QRadar appliance is reaching out to the WinCollect agent fine, but we get a reply from the WinCollect agent, indicated by the payload:

> msg=The configuration server registration failed with response code 0x80000007 (The socket connection to the server failed); will try again later.

So we tried checking our iptables on the QRadar Appliance and they’re fine as well; we’re allowing tcp communication over port 8413 as needed and we haven’t made any changes to the QChain:

> [iptables here][1]

We then tried to rename the .PEM file and try to get QRadar to send a new one over to our WinCollect Agent, but we get a new event in our log, an error:

> msg=Failed to retrieve the certificate from the configuration server due to connection problems.

Again, pointing to our connection problems from our first occurrence, the socket error.

Any ideas on how to solve this issue?

[1]: /answers/storage/temp/18387-iptables-sanitized.txt

Related:

Re: vision simulator fails to install

2017-06-30 07:07:03: Start /opt/vmware/etc/isv/firstboot

2017-06-30 07:07:03: Checking OVF environment…

2017-06-30 07:07:18: Unable to find the ovf environment.

2017-06-30 07:07:18: Checking system FQDN…

2017-06-30 07:07:18: Checking system IP…

2017-06-30 07:07:18: Updating /etc/hosts for ‘vce-vision.labbuildr.local 192.168

.2.19’…

2017-06-30 07:07:18: Configuring iptables firewall…

iptables: Saving firewall rules to /etc/sysconfig/iptables:[ OK

]

2017-06-30 07:07:18: Installing and configuring VCE applications…

Preparing… ##################################################

mod_ssl ##################################################

Preparing… ##################################################

nodejs ##################################################

Preparing… ##################################################

package httpd-2.2.15-59.el6.centos.x86_64 is already installed

2017-06-30 07:08:51: Failed!

2017-06-30 07:10:32: Start /opt/vmware/etc/isv/firstboot

2017-06-30 07:10:32: Checking OVF environment…

2017-06-30 07:10:47: Unable to find the ovf environment.

2017-06-30 07:10:47: Checking system FQDN…

2017-06-30 07:10:47: Checking system IP…

2017-06-30 07:10:47: Updating /etc/hosts for ‘vce-vision.labbuildr.local 192.168

.2.19’…

2017-06-30 07:10:47: Configuring iptables firewall…

iptables: Saving firewall rules to /etc/sysconfig/iptables:[ OK

]

2017-06-30 07:10:48: Installing and configuring VCE applications…

error: File not found by glob: /opt/vce/packages/mod_ssl*.rpm

Related:

vision simulator fails to install

2017-06-30 07:07:03: Start /opt/vmware/etc/isv/firstboot

2017-06-30 07:07:03: Checking OVF environment…

2017-06-30 07:07:18: Unable to find the ovf environment.

2017-06-30 07:07:18: Checking system FQDN…

2017-06-30 07:07:18: Checking system IP…

2017-06-30 07:07:18: Updating /etc/hosts for ‘vce-vision.labbuildr.local 192.168

.2.19’…

2017-06-30 07:07:18: Configuring iptables firewall…

iptables: Saving firewall rules to /etc/sysconfig/iptables:[ OK

]

2017-06-30 07:07:18: Installing and configuring VCE applications…

Preparing… ##################################################

mod_ssl ##################################################

Preparing… ##################################################

nodejs ##################################################

Preparing… ##################################################

package httpd-2.2.15-59.el6.centos.x86_64 is already installed

2017-06-30 07:08:51: Failed!

2017-06-30 07:10:32: Start /opt/vmware/etc/isv/firstboot

2017-06-30 07:10:32: Checking OVF environment…

2017-06-30 07:10:47: Unable to find the ovf environment.

2017-06-30 07:10:47: Checking system FQDN…

2017-06-30 07:10:47: Checking system IP…

2017-06-30 07:10:47: Updating /etc/hosts for ‘vce-vision.labbuildr.local 192.168

.2.19’…

2017-06-30 07:10:47: Configuring iptables firewall…

iptables: Saving firewall rules to /etc/sysconfig/iptables:[ OK

]

2017-06-30 07:10:48: Installing and configuring VCE applications…

error: File not found by glob: /opt/vce/packages/mod_ssl*.rpm

Related:

Is "iptables-save" will reload iptable configuration?

I am little confused. I know that iptables-save command will help to take backup of iptables with > to a file. But will this command alone will reload iptable configuration. Sorry, I don’t have a server to check this and I was unable to see any online reference.

Related: