In this next article in the series, we’ll take a look at some of the significant aspects of large cluster physical design and hardware installation.
Most Isilon nodes utilize a 35 inch depth chassis and will fit in a standard depth data center cabinet. However, high capacity models such as the HD400 and A2000 have 40 inch depth chassis and require extended depth cabinets such as the APC 3350 or Dell EMC Titan-HD rack.
Additional room must be provided for opening the FRU service trays at the rear of the nodes and, in Gen6 hardware, the disk sleds at the front of the chassis. Isilon nodes are either 2RU or 4RU in height (with the exception of the 1RU diskless accelerator and backup accelerator nodes).
Note that the Isilon A2000 nodes can also be purchased as a 7.2PB turnkey pre-racked solution.
Weight is another critical factor to keep in mind. Individual 4RU chassis can weigh up to around 300lbs each, and the floor tile capacity for each individual cabinet or rack must be kept in mind. For the large archive nodes styles (HD400 and A2000), the considerable node weight may prevent racks from being fully populated with Isilon equipment. If the cluster uses a variety of node types, installing the larger, heavier nodes at the bottom of each rack and the lighter chassis at the top can help distribute weight evenly across the cluster racks’ floor tiles.
There are no lift handles on a Gen6 chassis. However, the drive sleds can be removed to provide handling points if no lift is available. With all the drive sleds removed, but leaving the rear compute modules inserted, the chassis weight drops to a more manageable 115lbs. It is strongly recommended to use a lift for installation of Gen6 chassis and the 4RU earlier generation nodes.
Ensure that smaller Ethernet switches are drawing cool air from the front of the rack, not from inside the cabinet, as they are shorter than the IB switches. This can be achieved either with switch placement or by using rack shelving.Cluster backend switches ship with the appropriate rails (or tray) for proper installation of the switch in the rack. These rail kits are adjustable to fit NEMA front rail to rear rail spacing ranging from 22 in to 34 in.
Note that the Celestica Ethernet switch rails are designed to overhang the rear NEMA rails to align the switch with the Generation 6 chassis at the rear of the rack. These require a minimum clearance of 36 in from the front NEMA rail to the rear of the rack, in order to ensure that the rack door can be closed.
Consider the following large cluster topology, for example:
This contiguous eleven rack architecture is designed to scale up to ninety six 4RU nodes as the environment grows, while keeping cable management simple and taking the considerable weight of the Infiniband cables off the connectors as much as possible.
Best practices include:
- Pre-allocate and reserve adjacent racks in the same isle to fully accommodate the anticipated future cluster expansion
- Reserve an empty 4RU ‘mailbox’ slot above the center of each rack for pass-through cable management.
- Dedicate the central rack in the group for the back-end and front-end switches – in this case rack F (image below).
Below, the two top Ethernet switches are for front-end connectivity and the lower two Infiniband switches handle the cluster’s redundant back-end connections.
Image showing cluster Front and Back-end Switches (Rack F Above)
The 4RU “mailbox” space is utilized for cable pass-through between node racks and the central switch rack. This allows cabling runs to be kept as short and straight as possible.
Rear of Rack View Showing Mailbox Space and Backend Network Cabling (Rack E Above)
Excess cabling can be neatly stored in 12” service coils on a cable tray above the rack, if available, or at the side of the rack as illustrated below.
Rack Side View Detailing Excess Cable Coils (Rack E Above)
Successful large cluster infrastructures depend heavily on the proficiency of the installer and their optimizations for maintenance and future expansion.
Note that for Hadoop workloads, Isilon is compatible with the rack awareness feature of HDFS to provide balancing in the placement of data. Rack locality keeps the data flow internal to the rack.