ECS – xDoctor: “One or more network interfaces are down or missing”

Article Number: 503814 Article Version: 5 Article Type: Break Fix



Elastic Cloud Storage,ECS Appliance,ECS Appliance Hardware



xDoctor is reporting the below warning:

admin@ecs1:~> sudo -i xdoctor --report --archive=2017-09-01_064438 -CEWDisplaying xDoctor Report (2017-09-01_064438) Filter:['CRITICAL', 'ERROR', 'WARNING'] ...Timestamp = 2017-09-01_064438 Category = platform Source = ip show Severity = WARNING Node = 169.254.1.1 Message = One or more network interfaces are down or missing Extra = {'169.254.1.4': ['slave-0']} 

Connect to the node 169.254.1.4 in question and see in this case connection to rabbit switch is down:

admin@ecs4:~> sudo lldpcli show neighbor-------------------------------------------------------------------------------LLDP neighbors:-------------------------------------------------------------------------------Interface: slave-1, via: LLDP, RID: 1, Time: 28 days, 16:42:58 Chassis: ChassisID: mac 44:4c:a8:f5:63:ad SysName: hare SysDescr: Arista Networks EOS version 4.16.6M running on an Arista Networks DCS-7050SX-64 MgmtIP: 192.168.219.253 Capability: Bridge, on Capability: Router, off Port: PortID: ifname Ethernet12 PortDescr: MLAG group 4-------------------------------------------------------------------------------Interface: private, via: LLDP, RID: 2, Time: 28 days, 16:42:44 Chassis: ChassisID: mac 44:4c:a8:d1:77:b9 SysName: turtle SysDescr: Arista Networks EOS version 4.16.6M running on an Arista Networks DCS-7010T-48 MgmtIP: 192.168.219.251 Capability: Bridge, on Capability: Router, off Port: PortID: ifname Ethernet4 PortDescr: Nile Node04 (Data)-------------------------------------------------------------------------------admin@ecs4:~> 

Check public interface config:

admin@ecs4:~> sudo cat /etc/sysconfig/network/ifcfg-publicBONDING_MASTER=yesBONDING_MODULE_OPTS="miimon=100 mode=4 xmit_hash_policy=layer3+4"BONDING_SLAVE0=slave-0BONDING_SLAVE1=slave-1BOOTPROTO=staticIPADDR=10.x.x.x/22MTU=1500STARTMODE=autoadmin@ecs4:~> 
admin@ecs4:~> viprexec -i "grep Mode /proc/net/bonding/public"Output from host : 192.168.219.1Bonding Mode: IEEE 802.3ad Dynamic link aggregationOutput from host : 192.168.219.2Bonding Mode: IEEE 802.3ad Dynamic link aggregationOutput from host : 192.168.219.3Bonding Mode: IEEE 802.3ad Dynamic link aggregationOutput from host : 192.168.219.4Bonding Mode: IEEE 802.3ad Dynamic link aggregationadmin@ecs4:~> 

Check interface link status:

admin@ecs4:~> viprexec -i 'ip link show | egrep "slave-|public"'Output from host : 192.168.219.1bash: public: command not found3: slave-0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master public state UP mode DEFAULT group default qlen 10005: slave-1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master public state UP mode DEFAULT group default qlen 100010: public: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group defaultOutput from host : 192.168.219.2bash: public: command not found3: slave-0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master public state UP mode DEFAULT group default qlen 10005: slave-1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master public state UP mode DEFAULT group default qlen 100010: public: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group defaultOutput from host : 192.168.219.3bash: public: command not found4: slave-0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master public state UP mode DEFAULT group default qlen 10005: slave-1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master public state UP mode DEFAULT group default qlen 100010: public: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group defaultOutput from host : 192.168.219.4bash: public: command not found2: slave-0: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master public state DOWN mode DEFAULT group default qlen 10005: slave-1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master public state UP mode DEFAULT group default qlen 100010: public: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group defaultadmin@ecs4:~> 
admin@ecs4:~> sudo ethtool slave-0Settings for slave-0: Supported ports: [ FIBRE ] Supported link modes: 10000baseT/Full Supported pause frame use: No Supports auto-negotiation: No Advertised link modes: 10000baseT/Full Advertised pause frame use: No Advertised auto-negotiation: No Speed: Unknown! Duplex: Unknown! (255) Port: Other PHYAD: 0 Transceiver: external Auto-negotiation: off Supports Wake-on: d Wake-on: d Current message level: 0x00000007 (7) drv probe link Link detected: no

Refer to ECS Hardware Guide for details of specific port on the switch.

The ECS Hardware Guide is available in SolVe as well as at support.emc.com:

https://support.emc.com/docu62946_ECS-3.1-Hardware-Guide.pdf?language=en_US

Port 12 on rabbit switch is connected to slave-0 interface of node 4.

Connect to rabbit with admin credentials from a different node and check interface status:

admin@ecs1:~> ssh rabbitPassword:Last login: Tue Sep 5 11:13:30 2017 from 192.168.219.1rabbit>show interfaces Ethernet12Ethernet12 is down, line protocol is notpresent (notconnect) Hardware is Ethernet, address is 444c.a8de.8f83 (bia 444c.a8de.8f83) Description: MLAG group 4 Member of Port-Channel4 Ethernet MTU 9214 bytes , BW 10000000 kbit Full-duplex, 10Gb/s, auto negotiation: off, uni-link: n/a Loopback Mode : None 0 link status changes since last clear Last clearing of "show interface" counters never 5 minutes input rate 0 bps (0.0% with framing overhead), 0 packets/sec 5 minutes output rate 0 bps (0.0% with framing overhead), 0 packets/sec 0 packets input, 0 bytes Received 0 broadcasts, 0 multicast 0 runts, 0 giants 0 input errors, 0 CRC, 0 alignment, 0 symbol, 0 input discards 0 PAUSE input 0 packets output, 0 bytes Sent 0 broadcasts, 0 multicast 0 output errors, 0 collisions 0 late collision, 0 deferred, 0 output discards 0 PAUSE outputrabbit> 

The above interface status shows link also down and there has been never any I/O traffic on this interface.



SFP was not properly seated during install phase.



Customer was able to re-seat the SFP interface. After that the link automatically was detected and came online.

Otherwise CE need to go onsite for a physical inspection of SFP module, cable etc. connecting to slave-x interface on node.

Extract from ECS Hardware Guide:

Network cabling

The network cabling diagrams apply to U-Series, D-Series, or C-Series ECS Appliance in an Dell EMC or customer provided rack.

To distinguish between the three switches, each switch has a nickname:
  • Hare: 10 GbE public switch is at the top of the rack in a U- or D-Series or the top switch in a C-Series segment.
  • Rabbit: 10 GbE public switch is located just below the hare in the top of the rack in a U- or D-Series or below the hare switch in a C-Series segment.
  • Turtle: 1 GbE private switch that is located below rabbit in the top of the rack in a U-Series or below the hare switch in a C-Series segment.
U- and D-Series network cabling

The following figure shows a simplified network cabling diagram for an eight-node configuration for a U- or D-Series ECS Appliance as configured by Dell EMC or a customer in a supplied rack. Following this figure, other detailed figures and tables provide port, label, and cable color information.

Switches cabling

The ECS Hardware and cabling guide is showing rabbit and hare switches labeled as switch 1 and switch 2 what can cause confusion when cabling want to be verified.

See below table for matching switches and ports as well as the picture for also showing the appropriate switch port numbers.

Switch 1 = Rabbit = Bottom switch

Switch 2 = Hare = Top switch

Node ports:

Slave-0 = P01 = right port – connects to Switch 1 / Rabbit / Bottom switch

Slave-1 = P02 = left port – connects to Switch 2 / Hare / Top switch

User-added image

Related:

can linuxPE join multicast session

I need a solution

A quick questin, this forum has been invaluable for me.

Can a pc pxe booted to the linuxPE image (using the ghost executable in the ghost folder on the shared drive) access a multicast session on a ghostcast server, has anybody used that or got it to work?

Everything I try just comes back with the error that it cannot access multicast session <session name>

The reason I am asking is if multicasting is not supported then why is it saying that it can’t access the session and not something like “this feature is not available”

I have tried the following;

  • Windows PE works fine, connects to session and starts deploying the image, so firewalls and ports etc… should be good
  • linux can ping the ghost server and can mount browse the drive etc…so linux networking looks good to me.
  • I have 30 machine all the same type, all the same problem
  • I have other machines with the same problem
  • they are all using the intel e1000e network module under linux the machine has onboard intel gigabit card.
  • wireshark shows two udp connection atempts to port 6666 but then no further activity
  • wireshark under windows does alot more and then starts multicast.
  • image of the error is attached
  • I use the following command ghost -clone,mode=restore,src=mc”g”,dst=1 -sure -blind -jaddr=xxx.xxx.xxx.xxx
  • to kick of the session
  • i have tried starting the ghostcast server in unicast mode, no change
  • i have tried to start the client in unicast but cannot find a switch to do that
  • the ghosterr.txt file does not have anything in it about what the issue is.

does anybody have any insight? I have raised a ticket with symantec support and while they have been very helpful they do not have many people using linux and do not have much experience with it. My reason for using it is that the boot image file is 18MB as opposed to the smallest windows PE i can get with is about 120MB

thank you all.

Alan

0

Related:

Ghost 3.1 Console freezes when restoring an Image

I need a solution

When Distributing an Image to 12 computers using Ghost Cast server with multicasting the restoring process continues to freeze without any errors given. I reach about 12 to 15 % before it freezes?? So far I have not had an issue with per to per multicasting however my imaging job takes much longer to complete. hours for a 12gb image to compete …? 

Any Idea on what I should try next?  Does 3.1 have an error log I can look at or enable? 

0

Related:

multicasting speeds on HPe network

I need a solution

we are on a HPe network with 1G to every desktop.  when our end user group images a lab with rdeploy multicasting to 20 machines the speeds flucuate from 100 to 0 MB/s.  the size of their images range for 20G – 100+G.

our switch environment is setup on vlans and ip igmp is enabled.  When we unicast we get speeds of 800 and up…

we are wondering if anyone on a HPe switching environment uses Altiris and is getting good speeds.  we are also curious about realistics expectations for multicasting speeds out to 20+ machines. 

thanks everyone

0

Related:

Ghost 3.1 multicast issue

I need a solution

Hi

We have ghost 3.1 that deploy an image to multiple computers in multicast, and it works fast and doing it job.

When we try to deploy another image at the same time to multiple diferent computers from the same ghost machine, it fail to work in multicast, and works slow in unicast. it does not matter which image is the first and which one is second, the first one works in multicast and the second in unicast.

So, the question is basicly, is it an option to deploy two different images at the same time in a single ghost 3.1 server and both of them will work in multicast?

Thanks in advance

Ran

0

Related:

Utilisation of 2 GUP

I need a solution

Hello,

Can you tell me please if we can configure 2 GUPs in order to:

– The GUP number 2 download the updates from the GUP number 1.

– Knowing that the 2 GUPs are in Two diffrents networks separated by a firewall (Risk of non support broadcask/multicast ? )

Thanks in advance for your help

Weresafe

0

Related:

Event ID 2160 — Message Queuing Connectivity

Event ID 2160 — Message Queuing Connectivity

Updated: January 31, 2008

Applies To: Windows Server 2008

Message Queuing relies on network connectivity for much of its functionality. If network connectivity is lost or compromised, messages may not be sent or delivered.

Event Details

Product: Windows Operating System
ID: 2160
Source: MSMQ
Version: 6.0
Symbolic Name: MULTICAST_BIND_ERROR
Message: The queue cannot listen/bind to the multicast address %3. Error %1: %2

Resolve
Change the multicast address

The multicast address that is specified in the event is not valid. Enter a valid multicast address for the queue that is using this address. IP multicast addresses must consist of four bytes, and they must lie in the class D range from 224.0.0.0 to 239.255.255.255. However, only certain ranges of addresses in this range are available for sending multicast messages. For more information, see Internet Multicast Addresses (http://go.microsoft.com/fwlink/?LinkID=94475).

For more information about setting up multicasting queues, see Specify or View Queue Multicast Addresses and Ports (http://go.microsoft.com/fwlink/?LinkId=102330).

Verify

Confirm that the MSMQ Service is installed and running.

To perform this procedure, you must have membership in Administrators, or you must have been delegated the appropriate authority.

To confirm that the MSMQ Service is installed and running:

  1. Open the Services snap-in. To open Services, click Start. In the search box, type services.msc, and then press ENTER.
  2. Locate the Message Queuing service, and then confirm that the value in the Status column is Started.

Verify that the computer has connectivity to the network.

Related Management Information

Message Queuing Connectivity

Message Queuing

Related: