In this final article of the series, we’ll take a look at some of the important aspects of large cluster front end networking and connection balancing.
OneFS supports a variety of Ethernet speeds, cable and connector styles, and network interface counts, depending on the node type selected. Unlike the back-end network, Isilon does not specify particular front-end switch models.
The recommendation is that each node have at least one front-end interface configured, preferably in at least one static SmartConnect zone. Additionally, the following cluster services actually require that every node has an address that can reach the external servers:
- ICAP antivirus scanning.
- Monitoring services such as SNMP & ESRS.
- Authentication services such as Microsoft Active Directory (AD), LDAP, and NIS.
ESRS support is not a hard requirement because other nodes that are online can proxy out ESRS dial-home events. Ensure that the ESRS service can reach external servers so that every node can properly register with the ESRS gateway servers.
Sometimes a cluster needs to be expanded to increase storage capacity or CPU, not because more front-end I/O is required. With Gen5 and earlier platforms, diskless accelerator nodes with large amounts of L1 cache can be added to handle streaming and concurrent client connections, thereby freeing up resources on the storage nodes. With Gen6 nodes, the storage to compute ratio has changed in favor of the latter, so this is less frequently an issue.
Although a cluster can be run in a ‘not all nodes on the network’ (NANON) configuration, the recommendation is to connect all nodes to the front-end network(s). In contrast with scale-up NAS platforms that use separate network interfaces for out-of-band management and configuration, Isilon traditionally performs all cluster network management in-band. Bear in mind that the Gen6 Isilon nodes all contain a 1Gb Ethernet port that can be configured for use as a dedicated management network and/or ICMP, simplifying administration of a large cluster.
OneFS also supports using the nodes’ DB9 serial ports as RS232 out-of-band management interfaces, and this practice is highly recommended for large clusters. Serial ports can provide reliable BIOS-level command line access for on-site or remote service staff to perform maintenance, troubleshooting and installation operations.
For most workflows on large clusters, the recommendation is to configure at least one front-end 40Gb or 10 Gb Ethernet connection per node to support the high levels of network utilization that often take place. Assigning each workload or data store to a unique IP address enables Isilon SmartConnect to move each workload to one of the other interfaces, minimizing the additional work that a remaining node in the SmartConnect pool must absorb and ensuring that the workload is evenly distributed across all the other nodes in the pool.
For a SmartConnect pool with four-node interfaces, using the N * (N – 1) model will result in three unique IP addresses being allocated to each node. A failure on one node interface will cause each of that interface’s three IP addresses to fail over to a different node in the pool. This ensuring that each of the three active interfaces remaining in the pool receives one IP address from the failed node interface. If client connections to that node were evenly balanced across its three IP addresses, SmartConnect distributes the workloads to the remaining pool members evenly.
For large clusters, the highest IP allocation per cluster that Isilon recommends is a /23 subnet, or 510 usable addresses. There are very few cases that would require such a large IP allocation. From a load-balancing perspective, for dynamic pools, it is ideal (though optional) that all the interfaces have the same number of IP addresses, whenever possible.
More information on IP allocation sizing is available in the Isilon Advanced Networking Fundamentals guide.
Enabling jumbo frames (Maximum Transmission Unit set to 9000 bytes) yields slightly better throughput performance with slightly less CPU usage than standard frames, where the MTU is set to 1500 bytes. For example, with 10 Gb Ethernet connections, jumbo frames provide about 5 percent better throughput and about 1 percent less CPU usage.
OneFS provides the ability to optimize storage performance by designating zones to support specific workloads or subsets of clients. Different network traffic types can be segregated on separate subnets using SmartConnect pools.
For large clusters, partitioning the cluster’s networking resources and allocate bandwidth to each workload minimizes the likelihood that heavy traffic from one workload will affect network throughput for another. This is particularly true for SyncIQ replication and NDMP backup traffic, which can definitely benefit from its own set of interfaces, separate from user and client IO load.
Many customers as a best practice create separate SmartConnect subnets for the following traffic segregation:
- Workflow separation.
- SyncIQ Replication.
- NDMP backup on target cluster.
- Service Subnet for cluster administration and management traffic.
- Different node types and performance profiles.
OneFS 8.0 and later include the ‘groupnet’ networking object as part of the support for multi-tenancy. Groupnets sit above subnets and pools and allow separate Access Zones to contain distinct DNS settings. In the example below, each node’s four front end NICs are configured for SyncIQ replication traffic, Isilon management traffic, and two client access data networks.
The management and data networks can then be incorporated into different Access Zones, each with their own DNS, directory access services, and routing, as appropriate.
|
Network Interface
|
Network
|
Access Zone
|
|
igb0
|
Management traffic
|
Systems zone
|
|
igb1
|
SyncIQ replication traffic
|
Systems zone
|
|
bnxe0
|
Data network 1
|
Data 1 zone
|
|
bnxe1
|
Data network 2
|
Data 2 zone
|
By default, OneFS SmartConnect balances connections among nodes by using a round-robin policy and a separate IP pool for each subnet. A SmartConnect license adds advanced balancing policies to evenly distribute CPU usage, client connections, or throughput. It also lets you define IP address pools to support multiple DNS zones in a subnet.
|
Load-balancing Policy
|
General or Other
|
Few Clients with Extensive Usage
|
Large Number of Persistent NFS & SMB Connections
|
Large Number of Transitory Connections (HTTP, FTP)
|
NFS Automount of UNC Paths are Used
|
|
Round Robin
|
P
|
P
|
P
|
P
|
P
|
|
Connection Count
|
P
|
P
|
|
P
|
P
|
|
CPU Usage
|
|
|
|
|
|
|
Network Throughput
|
|
|
|
|
|
A ‘round robin’ load balancing strategy is the recommendation for both client connection balancing and IP failover. During a cluster split or merge group change the SmartConnect service will not respond to DNS inquiries. This is seldom as group changes typically take around 30 seconds. However, the time taken for a group change to complete can vary due to the load on the cluster at the time of the change. Any time a node is added, removed, or rebooted in a cluster there will be two group changes that cause SmartConnect to be impacted, one from down/split and one from up/merge.
For large clusters, if group changes are adversely impacting SmartConnect’s load-balancing performance, the core site DNS servers can be configured to use a Round Robin configuration instead of redirecting DNS requests to SmartConnect. SmartConnect supports IP failover to provide continuous access to data when hardware or a network path fails. Dynamic failover is recommended for high availability workloads on SmartConnect subnets that handle traffic from NFS clients.
For optimal network performance, observe the following SmartConnect best practices:
- Do not mix interface types (40Gb / 10Gb / 1Gb) in the same SmartConnect Pool
- Do not mix node types with different performance profiles (for example, H600 and A200 interfaces).
- Use the ‘round-robin’ SmartConnect Client Connection Balancing and IP-failover policies.
To evenly distribute connections and optimize performance, the recommendation is to size SmartConnect for the expected number of connections and for the anticipated overall throughput likely to be generated. The sizing factors for a pool include:
- The total number of active client connections expected to use the pool’s bandwidth at any time.
- Expected aggregate throughput that the pool needs to deliver.
- The minimum performance and throughput requirements in case an interface fails.
Since OneFS is a single volume, fully distributed file system, a client can access all the files and associated metadata that are stored on the cluster, regardless of the type of node a client connects to or the node pool on which the data resides. For example, data stored for performance reasons on a pool of performance nodes (eg. H600) can be mounted and accessed by connecting to an archive node (eg. A200) in the same cluster. Note that the different types of Isilon nodes will deliver different levels of performance.
To avoid unnecessary network latency under most circumstances, the recommendation is to configure SmartConnect subnets such that client connections are to the same physical pool of nodes on which the data resides. In other words, if a workload’s data lives on a pool of F800 nodes for performance reasons, the clients that work with that data should mount the cluster through a pool that includes the same F800 nodes that host the data.
Keep in mind the following networking and name server considerations:
- Minimize disruption by suspending nodes in preparation for planned maintenance and resuming them after maintenance is complete
- If running OneFS 8.0 or later, leverage the groupnet feature to enhance multi-tenancy and DNS delegation, where desirable.
- Ensure traffic flows through the right interface by tracing routes. Leverage OneFS Source-Based Routing (SBR) feature to keep traffic on desired paths.
- If you have firewalls, ensure that the appropriate ports are open. For example, open both UDP port 53 and TCP port 53 for the DNS service.
- The client never sends a DNS request directly to the cluster. Instead, the site name-servers handle DNS requests from clients and route the requests appropriately.
- In order to successfully distribute IP addresses, the OneFS SmartConnect DNS delegation server answers DNS queries with a time-to-live (TTL) of 0 so that the answer is not cached. Certain DNS servers (particularly Windows DNS Servers) will fix the value to one second. If you have many clients requesting an address within the same second, this will cause all of them to receive the same address. If you encounter this problem, you may need to use a different DNS server, such as BIND.
- Certain clients perform DNS caching and might not connect to the node with the lowest load if they make multiple connections within the lifetime of the cached address. Recommend turning off client DNS caching, where possible. To handle client requests properly, SmartConnect requires that clients use the latest DNS entries
- The site DNS servers must be able to communicate with the node that is currently hosting the SmartConnect service. This is the node with the lowest logical node number (LNN) with an active interface in the subnet that contains the SSIP address. This behavior cannot be modified.
- Connection policies other than round robin are sampled every 10 seconds. The CPU policy is sampled every 5 seconds. If multiple requests are received during the same sampling interval, SmartConnect will attempt to balance these connections by estimating or measuring the additional load.
