Host Channel Adapter Research Articles

Accelerating communication with multi‐HCA aware collectives in MPI

SummaryTo accelerate the communication between nodes, supercomputers are now equipped with multiple network adapters per node, also referred to as HCAs (Host Channel Adapters), resulting in a “multi‐rail”/“multi‐HCA” network. For example, the ThetaGPU system at Argonne National Laboratory (ANL) has eight adapters per node; with this many networking resources available, utilizing all of them becomes non‐trivial. The Message Passing Interface (MPI) is a dominant model for high‐performance computing clusters. Not all MPI collectives utilize all resources, and this becomes more apparent with advances in bandwidth and adapter count in a given cluster. In this work, we provide a thorough performance analysis of existing multirail solutions and their implications on collectives and present the necessity for further enhancement. Specifically, we propose novel designs for hierarchical, multi‐HCA‐aware Allgather. The proposed designs fully utilize all the available network adapters within a node and provide high overlap between inter‐node and intra‐node communication. At the micro‐benchmark level, we see large inter‐node improvements up to 62% and 61% better than HPC‐X and MVAPICH2‐X for 1024 processes. Because Allgather is used in Ring‐Allreduce, our designs also improve its performance by 56% and 44% compared to HPC‐X and MVAPICH2‐X, respectively. At the application level, our enhanced Allgather shows and improvement in a matrix‐vector multiplication kernel when compared to HPC‐X and MVAPICH2‐X, and Allreduce performs up to 7.83% better in deep learning training against MVAPICH2‐X.

Assessment of the ALICE O2 readout servers

The ALICE experiment at CERN LHC (Large Hadron Collider) is undertaking a major upgrade during LHC Long Shutdown 2 in 2019-2020. The raw data input from the detector will then increase a hundredfold, up to 3.4 TB/s. In order to cope with such a large throughput, a new Online-Offline com-9 puting system, called O2, will be deployed. The FLP servers (First Layer Pro-10 cessor) are the readout nodes hosting the CRU (Common Readout Unit) cards in charge of transferring the data from the detector links to the computer mem-12 ory. The data then flow through a chain of software components until they are shipped over network to the processing nodes. In order to select a suitable plat-14 form for the FLP (First Level Processor), it is essential that the hardware and the software are tested together. Each candidate server is therefore equipped with multiple readout cards (CRU), one InfiniBand 100G Host Channel Adapter, and the O2 readout software suite. A series of tests are then run to ensure the readout system is stable and fulfils the data throughput requirement of 42Gbps (highest data rate in output of the FLP equipped with three CRUs). This paper presents the software and firmware features developed to evaluate and validate different candidates for the FLP servers. In particular we describe the data flow from the CRU firmware generating data, up to the network card where the buffers are sent over the network using RDMA. We also discuss the testing procedure and the results collected on different servers.

HCA aware Parallel Communication Library: A feasibility study for offloading MPI requirements

Message Passing Interface (MPI) is a standardized message passing system, independent of underlying network, and the most widely used parallel programming paradigm. The communication library should make full use of the Host Channel Adapter (HCA) characteristics to maximize performance of the HPC cluster. The communication library may not able to take full advantage of the underlying network adapter, if the library is made generalized. This can have a significant impact on the performance. Our primary goal is to develop a network dependent message passing library called a Parallel Communication Library (PCL) that will exploit C-DAC's proprietary PARAMNet HCA features for efficient message transfer. Using PCL, we intend to observe the feasibility of the network and performance enhancement for additional features. The objective is to carry out different trials by implementing additional features and analyze the implications which would give us more insight towards suitability of transport offload/onload mechanism. This experimentation would give us feedbacks for designing the next generation architecture.

Comparison of UDP Transmission Performance Between IP-Over-InfiniBand and 10-Gigabit Ethernet

Amongst link technologies, InfiniBand has gained wide acceptance in the framework of High Performance Computing (HPC), due to its high bandwidth and in particular to its low latency. Since InfiniBand is very flexible, supporting several kinds of messages, it is suitable, in principle, not only for HPC, but also for the data acquisition systems of High Energy Physics (HEP) Experiments. In order to check the InfiniBand capabilities in the framework of on-line systems of HEP Experiments, we performed measurements with point-to-point UDP data transfers over a 4-lane Double Data Rate InfiniBand connection, by means of the IPoIB (IP over InfiniBand) protocol stack, using the Host Channel Adapter cards mounted on a 8-lane PCI-Express bus of commodity PCs both as transmitters and receivers, thus measuring not only the capacity of the link itself, but also the effort required by the host CPUs, buses and Operating Systems. Using either the "Unreliable Datagram" or the "Reliable Connected" InfiniBand transfer modes, we measured the maximum achievable UDP data transfer throughput, the frame rate and the CPU loads of the sender/receiver processes and of the interrupt handlers as a function of the datagram size. Performance of InfiniBand in UDP point-to-point data transfer are then compared with that obtained with analogous tests per formed between the same PCs, using a 10-Gigabit Ethernet link.

Integration of admission, congestion, and peak power control in QoS-aware clusters

Admission, congestion, and peak power control mechanisms are essential parts of a cluster network design for supporting integrated traffic. While an admission control algorithm helps in delivering the assured performance, a congestion control algorithm regulates traffic injection to avoid network saturation. Peak power control forces to meet pre-specified power constraints while maintaining the service quality by regulating the injection of packets. In this paper, we propose these control algorithms for clusters, which are increasingly being used in a diverse set of applications that require QoS guarantees. The uniqueness of our approach is that we develop these algorithms for wormhole-switched networks, which have been used in designing clusters. We use QoS-capable wormhole routers and QoS-capable network interface cards (NICs), referred to as Host Channel Adapters (HCAs) in InfiniBand™ Architecture (IBA), to evaluate the effectiveness of these algorithms. The admission control is applied at the HCAs and the routers, while the congestion control and the peak power control are deployed only at the HCAs. A mixed workload consisting of best-effort, real-time, and control traffic is used to investigate the effectiveness of the proposed schemes. Simulation results with a single router (8-port) cluster and a 2-D mesh network cluster indicate that the admission, congestion, and peak power control algorithms are quite effective in delivering the assured performance. The proposed credit-based congestion control algorithm is simple and practical in that it relies on hardware already available in the HCA/NIC to regulate traffic injection.

Design and verification of the IBM System z10 I/O subsystem chips

In this paper, we discuss the microarchitecture, design, and S. Junghans verification of two IBM System z10™ I/O (input/output) chips: the z10™ hub chip, an InfiniBand™ host channel adapter with IBMproprietary enhancements, and the InfiniBand memory bus adapter (MBA) chip, an InfiniBand-to-self-timed-interface fanout chip for attaching legacy I/O. Designing and verifying these chips presented many challenges. We describe our transaction- and packet-tracking concepts and the use of communication groups that emulate the behavior of logical partitions and their role in handling error and recovery cases. A novel technique has been employed to ensure that design implementation and architectural register definitions are consistent in a fully automated approach. Finally, we describe our approach to improving self-test coverage, which is based on an automated process of test-point insertion.

A High-Speed Link Layer Architecture for Low Latency and Memory Cost Reduction

This paper presents the design and implementation of the InfiniBand link layer with special efforts made for packet latency reduction and buffer space optimization. The link layer is designed to avoid any architectural conflict while its components are executed in parallel as far as possible. For high-speed packet processing with the various quality of service supports required by InfiniBand, three candidates for packet receiving architecture are investigated. The maximum and minimum delays from an input to an output of a switch adopting each of the three candidates is estimated by mathematically modeling the switch delays. Then, the candidate architecture with the best performance is chosen, and a novel first-in first-out (FIFO) is designed to efficiently implement the chosen architecture. Simulation results show that the chosen architecture achieves the least packet latency and uses the least memory space among the three candidates. The link layer core is implemented in an InfiniBand host channel adapter system-on-chip called KINCA.

Evaluating InfiniBand performance with PCI Express

The InfiniBand architecture is an industry standard that offers low latency and high bandwidth as well as advanced features such as remote direct memory access (RDMA), atomic operations, multicast, and quality of service. InfiniBand products can achieve a latency of several microseconds for small messages and a bandwidth of 700 to 900 Mbytes/s. As a result, it is becoming increasingly popular as a high-speed interconnect technology for building high-performance clusters. The Peripheral Component Interconnect (PCI) has been the standard local-I/O-bus technology for the last 10 years. However, more applications require lower latency and higher bandwidth than what a PCI bus can provide. As an extension, PCI-X offers higher peak performance and efficiency. InfiniBand host channel adapters (HCAs) with PCI Express achieve 20 to 30 percent lower latency for small messages compared with HCAs using 64-bit, 133-MHz PCI-X interfaces. PCI Express also improves performance at the MPI level, achieving a latency of 4.1μs for small messages. It can also improve MPI collective communication and bandwidth-bound MPI application performance.

고성능 클러스터 시스템을 위한 인피니밴드 시스템 연결망의 설계 및 구현

InfiniBand technology is being accepted as the future system interconnect to serve as the high-end enterprise fabric for cluster computing. This paper presents the design and implementation of the InfiniBand system interconnect, focusing on an InfiniBand host channel adapter (HCA) based on dual ARM9 processor cores The HCA is an SoC tailed KinCA which connects a host node onto the InfiniBand network both in hardware and in software. Since the ARM9 processor core does not provide necessary features for multiprocessor configuration, novel inter-processor communication and interrupt mechanisms between the two processors were designed and embedded within the KinCA chip. Kinch was fabricated as a 564-pin enhanced BGA (Bail Grid Array) device using 0.18 CMOS technology Mounted on host nodes, it provides 10 Gbps outbound and inbound channels for transmit and receive, respectively, resulting in a high-performance cluster system.