Efficient Collective Communication Research Articles

Accelerating parallel data processing using optically tightly coupled FPGAs

A cutting-edge field programmable gate array (FPGA) card can be equipped with high-bandwidth inputs and outputs by high-density optical integration, e.g., onboard Si-photonics transceivers or co-packaged optics. It is possible to implement highly parallel data processing by using tightly coupled FPGAs. We previously proposed a lightweight, fully connected inter-FPGA network called OPTWEB for efficient collective communications. OPTWEB discards the traditional packet communication mechanism and introduces dedicated memory processing to minimize communication overhead. In this paper, we provide the design guidelines for a parallel data application on FPGAs connected by OPTWEB. We then present a generalized diameter-two hypercube network topology for connecting multiple FPGAs, which enables larger applications. In the guidelines, data to be exchanged should be directly processed by the dedicated memory on each FPGA, and parallelism should be implemented to match the collective communications provided by OPTWEB. We illustrate two case studies, one on parallel counting sort and the other on sharing averaged parameters, on eight custom Stratix10 MX2100 FPGA cards with 800 Gbps network bandwidth. The parallel counting sort operates up to 5.3 times faster than a comparable CPU cluster with the conventional InfiniBand network, and with sharing averaged parameters, up to 250 times faster operation is achieved. The two applications utilize a total of 80.1% and 60.6% of the adaptive logic modules of the FPGA, respectively.

ACCL: Architecting Highly Scalable Distributed Training Systems With Highly Efficient Collective Communication Library

Distributed systems have been widely adopted for deep neural networks model training. However, the scalability of distributed training systems is largely bounded by the communication cost. We design a highly efficient collective communication library, namely Alibaba Collective Communication Library (ACCL), to build distributed training systems with linear scalability. ACCL provides optimized algorithms to fully make use of heterogeneous interconnects simultaneously. And the experimental results show significant performance improvement.

A parallel strategy for density functional theory computations on accelerated nodes

Using the Löwdin orthonormalization of tall-skinny matrices as a proxy-app for wavefunction-based Density Functional Theory solvers, we investigate a distributed memory parallel strategy focusing on Graphics Processing Unit (GPU)-accelerated nodes as available on some of the top ranked supercomputers at the present time. We present numerical results in the strong limit regime, as it is particularly relevant for First-Principles Molecular Dynamics. We also examine how matrix product-based iterative solvers provide a competitive alternative to dense eigensolvers on GPUs, allowing to push the strong scaling limit of these computations to a larger number of distributed tasks. Our strategy, which relies on replicated Gram matrices and efficient collective communications using the NCCL library, leads to a time-to-solution under 0.5 s for the Löwdin orthonormalization of a tall-skinny matrix of 3000 columns on Summit at Oak Ridge Leadership Facility (OLCF). Given the similarity in computational operations between one iteration of a DFT solver and this proxy-app, this shows the possibility of solving accurately the DFT equations well under a minute for 3000 electronic wave functions, and thus perform First-Principles molecular dynamics of physical systems much larger than traditionally solved on CPU systems.

A distributed approach for graph mining in massive networks

We propose a novel distributed algorithm for mining frequent subgraphs from a single, very large, labeled network. Our approach is the first distributed method to mine a massive input graph that is too large to fit in the memory of any individual compute node. The input graph thus has to be partitioned among the nodes, which can lead to potential false negatives. Furthermore, for scalable performance it is crucial to minimize the communication among the compute nodes. Our algorithm, DistGraph, ensures that there are no false negatives, and uses a set of optimizations and efficient collective communication operations to minimize information exchange. To our knowledge DistGraph is the first approach demonstrated to scale to graphs with over a billion vertices and edges. Scalability results on up to 2048 IBM Blue Gene/Q compute nodes, with 16 cores each, show very good speedup.

Power consumption management in fat-tree interconnection networks

Novel mechanism for reducing power consumption in fat-trees.Flexible mechanism that can be tuned in terms of aggressiveness and sensitivity.Simple implementation: same routing algorithm regardless of the power saving level.Ability of the mechanism to provide very significant power savings.Light impact in latency while providing always favorable energy balance. As higher communication bandwidth is required in current designs of high performance parallel computers, the amount of power consumed by the interconnection network also increases. Fat-tree is one of the most popular topologies in high performance interconnection networks aiming at low latency, efficient collective communication and scalability. We present a new methodology for managing power consumption of fat-tree interconnection networks. Our proposal is based on dynamically adjusting available network bandwidth according to traffic requirements. To meet this goal, we define a mechanism for managing the operating status of network links as a function of network load. Our main contributions include a complete definition of the mechanism and a tuning methodology based on its sensitivity and aggressiveness in terms of potential power savings. Results show that our proposal can provide significant power savings (up to 67% in a 4-ary 4-tree) with no changes in the underlying routing algorithm, with minimal impact on network performance. Experiments conducted on a 16-ary 3-tree topology provide up to 36% energy savings with performance degradation below 1.10%.

On Directed Edge-Disjoint Spanning Trees in Product Networks, An Algorithmic Approach

In (Ku et al. 2003), the authors have proposed a construction of edge-disjoint spanning trees EDSTs in undirected product networks. Their construction method focuses more on showing the existence of a maximum number (n1+n2-1) of EDSTs in product network of two graphs, where factor graphs have respectively n1 and n2 EDSTs. In this paper, we propose a new systematic and algorithmic approach to construct (n1+n2) directed routed EDST in the product networks. The direction of an edge is added to support bidirectional links in interconnection networks. Our EDSTs can be used straightforward to develop efficient collective communication algorithms for both models store-and-forward and wormhole.

Benefits of Adding Hardware Support for Broadcast and Reduce Operations in MPSoC Applications

MPI has been used as a parallel programming model for supercomputers and clusters and recently in MultiProcessor Systems-on-Chip (MPSoC). One component of MPI is collective communication and its performance is key for certain parallel applications to achieve good speedups. Previous work showed that, with synthetic communication-only benchmarks, communication improvements of up to 11.4-fold and 22-fold for broadcast and reduce operations, respectively, can be achieved by providing hardware support at the network level in a Network-on-Chip (NoC). However, these numbers do not provide a good estimation of the advantage for actual applications, as there are other factors that affect performance besides communications, such as computation. To this end, we extend our previous work by evaluating the impact of hardware support over a set of five parallel application kernels of varying computation-to-communication ratios. By introducing some useful computation to the performance evaluation, we obtain more representative results of the benefits of adding hardware support for broadcast and reduce operations. The experiments show that applications with lower computation-to-communication ratios benefit the most from hardware support as they highly depend on efficient collective communications to achieve better scalability. We also extend our work by doing more analysis on clock frequency, resource usage, power, and energy. The results show reasonable scalability for resource utilization and power in the network interfaces as the number of channels increases and that, even though more power is dissipated in the network interfaces due to the added hardware, the total energy used can still be less if the actual speedup is sufficient. The application kernels are executed in a 24-embedded-processor system distributed across four FPGAs.

Design and Implementation of Hybrid and Native Communication Devices for Java HPC

MPJ Express is a messaging system that allows computational scientists to write and execute parallel Java applications on High Performance Computing (HPC) hardware. The software is capable of executing in two modes namely cluster and multicore modes. In the cluster mode, parallel applications execute in a typical cluster environment where multiple processing elements communicate with one another using a fast interconnect like Gigabit Ethernet or other proprietary networks like Myrinet and Infiniband. In this context, the MPJ Express library provides communication devices for Ethernet and Myrinet. In the multicore mode, the parallel Java application executes on a single system comprising of shared memory or multicore processors. In this paper, we extend the MPJ Express software to provide two new communication devices namely the native and hybrid device. The goal of the native communication device is to interface the MPJ Express software with native—typically written in C—MPI libraries. In this setting the bulk of messaging logic is offloaded to the underlying MPI library. This is attractive because MPJ Express can exploit latest features, like support for new interconnects and efficient collective communication algorithms of the native MPI library. The second device, called the hybrid device, is developed to allow efficient execution of parallel Java applications on clusters of shared memory or multicore processors. In this setting the MPJ Express runtime system runs a single multithreaded process on each node of the cluster—the number of threads in each process is equivalent to processing elements within a node. Our performance evaluation reveals that the native device allows MPJ Express to achieve comparable performance to native MPI libraries—for latency and bandwidth of point-to-point and collective communications—which is a significant gain in performance compared to existing communication devices. The hybrid communication device—without any modifications at application level—also helps parallel applications achieve better speedups and scalability. We witnessed comparative performance for various benchmarks—including NAS Parallel Benchmarks—with hybrid device as compared to

Multi-hop all-to-all optical routings in Cartesian product networks

This paper considers the wavelength assignment problem for Cartesian product networks with multi-hops. An upper bound of the (uniform) wavelength index for Cartesian product networks with single-hop is established. This result leads to a consequence for the nth power of an arbitrary network with k-hops. In particular, if k = 1 , this bound partially generalizes the results of Pankaj [R.K. Pankaj, Architectures for linear lightwave networks, PhD thesis, Dept. of Electrical Engineering and Computer Science, MIT, Cambridge, MA, 1992], Bermond et al. [J.-C. Bermond, L. Gargano, S. Perennes, A.A. Rescigno, U. Vaccaro, Efficient collective communication in optical networks, Theoret. Comput. Sci. 233 (2000) 165–189] and Beauquier [B. Beauquier, All-to-all communication for some wavelength-routed all-optical networks, Networks 33 (1999) 179–187] for hypercubes and Hamming graphs. As an application, a tight upper bound for Hamming graph with k hops is established and a corresponding open problem is also proposed.

An Approximation Algorithm and Dynamic Programming for Reduction in Heterogeneous Environments

Network of workstation (NOW) is a cost-effective alternative to massively parallel supercomputers. As commercially available off-the-shelf processors become cheaper and faster, it is now possible to build a cluster that provides high computing power within a limited budget. However, a cluster may consist of different types of processors and this heterogeneity complicates the design of efficient collective communication protocols. For example, it is a very hard combinatorial problem to find an optimal reduction schedule for such heterogeneous clusters. Nevertheless, we show that a simple technique called slowest-node-first (SNF) is very effective in designing efficient reduction protocols for heterogeneous clusters. First, we show that SNF is actually a 2-approximation algorithm, which means that an SNF schedule length is always within twice of the optimal schedule length, no matter what kind of cluster is given. In addition, we show that SNF does give the optimal reduction time when the cluster consists of two types of processors, when the ratio of communication speed between them is at least two. When the communication speed ratio is less than two, we develop a dynamic programming technique to find the optimal schedule. Our dynamic programming utilizes the monotone property of the objective function, and can significantly reduce the amount of computation time. Finally, combined with an approximation algorithm for broadcast 2004, we propose an all-reduction algorithm which sends the reduction answer to all processors, with approximation ratio 3.5. We conduct three groups of experiments. First, we show that SNF performs better than the built-in MPI_Reduce in a test cluster. Second, we observe a factor of 93 times saving in computation time to find the optimal schedule, when compared with a naive dynamic programming implementation. Thirdly, we apply the theoretical results to a branch-and-bound search and show that they can reduce the search time of the optimal reduction schedule by a factor of 500, when the cluster has three kinds of processors.

Bandwidth Efficient All-to-All Broadcast on Switched Clusters

Clusters of workstations employ flexible topologies: regular, irregular, and hierarchical topologies have been used in such systems. The flexibility poses challenges for developing efficient collective communication algorithms since the network topology can potentially have a strong impact on the communication performance. In this paper, we consider the all-to-all broadcast operation on clusters with cut-through and store-and-forward switches. We show that near-optimal all-to-all broadcast on a cluster with any topology can be achieved by only using the links in a spanning tree of the topology when the message size is sufficiently large. The result implies that increasing network connectivity beyond the minimum tree connectivity does not improve the performance of the all-to-all broadcast operation when the most efficient topology specific algorithm is used. All-to-all broadcast algorithms that achieve near-optimal performance are developed for clusters with cut-through and clusters with store-and-forward switches. We evaluate the algorithms through experiments and simulations. The empirical results confirm our theoretical finding.

Efficient Collective Communications in Dual-Cube

The hypercube, or n-cube, has been widely used as the interconnection network in parallel computers. However, the major drawback of the hypercube is the increase in the number of communication links for each node with the increase in the total number of nodes in the system. This paper introduces a new interconnection network, namely dual-cube, for large-scale parallel computers and describes the algorithms for efficient collective communications in dual-cube. The dual-cube network mitigates the problem of increasing number of links in the large-scale hypercube network while retains hypercube’s topological properties. Design of efficient routing algorithms for collective communications is the key issue for any interconnection network. In this paper, we show that the collective communications can be done in dual-cube with almost the same communication times as in hypercube.

Scheduling for broadcast operation in heterogeneous parallel computing environments

AbstractParallel computing utilizes heterogeneous parallel computing environments in which multiple processors of various kinds are connected by communication links. Efficient collective communication operations are needed to realize efficient parallel computing in parallel computing environments. Although methods for realizing efficient collective communication operations have previously been proposed for homogeneous parallel computing environments, these methods applied as they are to heterogeneous parallel computing environments cannot realize efficient collective communication operations. In this paper, we consider design of an efficient broadcast, which is one of the most important collective communication operations. We show that we can extend the known sufficient condition for Fastest‐Node‐First (FNF) to output an optimal schedule, where FNF is an efficient heuristic algorithm for the broadcast. We show that we can reduce the time cost of the branch‐and‐bound search for the optimal schedule. © 2004 Wiley Periodicals, Inc. Syst Comp Jpn, 35(5): 44–54, 2004; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/scj.10533

Efficient collective communication in distributed heterogeneous systems

With recent advances in high-speed networks, distributed heterogeneous computing has emerged as an attractive computational paradigm. Wide-area grid infrastructures will enable distributed applications—such as video conferencing and distributed interactive simulation—to seamlessly integrate collections of heterogeneous workstations, multiprocessors, and mobile nodes. The underlying network is typically a collection of several heterogeneous links, of different networking technologies. Such a heterogeneous network is also typical in local area workstation clusters, which are increasingly being used as alternatives to parallel computing systems. This paper introduces a framework for developing efficient collective communication schedules over such heterogeneous networks. We focus on application-level communication, between processes of a parallel program. Our framework consists of analytical models of the heterogeneous system, scheduling algorithms for the collective communication pattern, and performance evaluation mechanisms. We show that previous models, which considered node heterogeneity but ignored network heterogeneity, can lead to solutions which are worse than the optimal by an unbounded factor. We then introduce an enhanced communication model, and develop three heuristic algorithms for the broadcast and multicast patterns. The completion time of the schedule is chosen as the performance metric. The heuristic algorithms are fastest edge first (FEF), earliest completing edge first (ECEF), and ECEF with look-ahead. For small system sizes, we find the optimal solution using exhaustive search. Our simulation experiments indicate that the performance of our heuristic algorithms is close to optimal. For performance evaluation of larger systems, we have also developed a simple lower bound on the completion time. Our heuristic algorithms achieve significant performance improvements over previous approaches.

Uniform multi-hop all-to-all optical routings in rings

We consider all-to-all routing problem in an optical ring network that uses the wavelength-division multiplexing (WDM). Since 1-hop, all-to-all optical routing in a WDM optical ring of n nodes needs ⌈ 1 2 ⌊n 2/4⌋⌉ wavelengths (see Efficient Collective Communication in Optical Networks, Lecture Notes in Computer Science, Vol. 1099, Springer, Berlin, 1996, pp. 574–585), which can be too large even for moderate values of n, we consider in this paper j-hop implementations of all-to-all routing in a WDM optical ring, j⩾2. From among the possible routings we focus our attention on uniform routings, in which each node of the ring uses the same communication pattern and the communication load is distributed evenly among the nodes. We show that there exists a uniform 2-hop implementation of all-to-all routing that needs at most (n/4)( n 3 +3) wavelengths. This value is within multiplicative constants of a lower bound. We then give a uniform 3-hop, 4-hop implementation of all-to-all routing that needs at most (n/2)( n/16 7 +3) , (n/2)( n/2 15 +6) wavelengths, respectively.

Broadcast Scheduling Optimization for Heterogeneous Cluster Systems

Network of workstation (NOW) is a cost-effective alternative to massively parallel supercomputers. As commercially available off-the-shelf processors become cheaper and faster, it is now possible to build a PC or workstation cluster that provides high computing power within a limited budget. However, a cluster may consist of different types of processors, and this heterogeneity within a cluster complicates the design of efficient collective communication protocols. This paper shows that a simple heuristic called fastest-node-first (FNF) (1998, M. Banikazemi, V. Moorthy, and D. K. Panda, in “Proceedings of the International Parallel Processing Conference”) is very effective in reducing the broadcast time for heterogeneous cluster systems. Despite the fact that the FNF heuristic fails to give the optimal broadcast time for a general heterogeneous network of workstations, we prove that FNF always gives the optimal broadcast time in several special cases of clusters. Based on these special case results, we show that FNF is an approximation algorithm that guarantees a competitive ratio of 2. From these theoretical results we also derive techniques to speed up the branch-and-bound search for the optimal broadcast schedule in HNOW.

Efficient collective communication in optical networks

This paper studies the problems of One-to-All and All-to-All Communication in optical networks. In such networks the vast bandwidth available is utilized through wavelength division multiplexing: a single physical optical link can carry several logical signals, provided that they are transmitted on different wavelengths. In this paper we consider both single-hop and multi-hop optical networks. In single-hop networks the information, once transmitted as light, reaches its destination without being converted to electronic form in between, thus reaching high-speed communication. In multihop networks a packet may have to be routed through a few intermediate nodes before reaching its final destination. In both models we give efficient One-to-All and All-to-All Communication algorithms, in terms of time and number of wavelengths. We consider both networks with arbitrary topologies and particular networks of practical interest. Several of our algorithms exhibit optimal performances.

Adaptive Communication Algorithms for Distributed Heterogeneous Systems

Many grand challenge applications can benefit from metacomputing, i.e., the coordinated use of geographically distributed heterogeneous supercomputers. A salient feature of such systems is the heterogeneity in the network performance between different processor pairs. This paper considers the problem of efficient application-level communication in heterogeneous network-based systems. We present a uniform communication scheduling framework for developing adaptive communication schedules for various collective communication patterns. The framework enables schedules to be developed at runtime, based on network performance information obtained from a directory service. Based on this framework, we have developed communication schedules for the total exchange communication pattern. Our first algorithm develops a schedule by computing a series of matchings in a bipartite graph. We also present a heuristic algorithm based on the open shop scheduling problem. The completion time of the heuristic is guaranteed to be within twice the optimal. Simulation results show performance improvements by a factor of 5 over well-known homogeneous scheduling techniques. This paper is an early effort in formalizing and solving communication problems for metacomputing systems. We discuss several research issues that must be addressed to allow efficient collective communication in such environments.

Efficient broadcast and multicast on multistage interconnection networks using multiport encoding

This paper proposes anew approach for implementing fast multicast and broadcast in unidirectional and bidirectional multistage interconnection networks (MINs) with multiport encoded multidestination worms. For a MIN with n stages, such worms use n header flits each. One flit is used for each stage of the network and it indicates the output ports to which a multicast message needs to be replicated. A multiport encoded worm with (d/sub 1/, d/sub 2/..., d/sub n/, 1/spl les/d/sub i//spl les/k) degrees of replication for the respective stages is capable of covering (d/sub 1//spl times/d/sub x//spl times/.../spl times/d/sub n/) destinations with a single communication start-up. In this paper, a switch architecture is proposed for implementing multidestination worms without deadlock. Three grouping algorithms of varying complexity are presented to derive the associated multiport encoded worms for a multicast to an arbitrary set of destinations. Using these worms, a multinomial tree-based scheme is proposed to implement the multicast. This scheme significantly reduces broadcast/multicast latency compared to schemes using unicast messages. Simulation studies for both unidirectional and bidirectional MIN systems indicate that improvement in broadcast/multicast latency up to a factor of four is feasible using the new approach. Interestingly, this approach is able to implement multicast with reduced latency as the number of destinations increases beyond a certain number. Compared to implementing unicast messages, this approach requires little additional logic at the switches. Thus, the scheme demonstrates significant potential for implementing efficient collective communication operations on current and future MIN-based systems.

Optimal software multicast in wormhole-routed multistage networks

Multistage interconnection networks are a popular class of interconnection architecture for constructing scalable parallel computers (SPCs). The focus of this paper is on the multistage network system which supports wormhole routed turnaround routing. Existing machines characterized by such a system model include the IBM SP-1 and SP-2, TMC CM-5, and Meiko CS-2. Efficient collective communication among processor nodes is critical to the performance of SPCs. A system-level multicast service, in which the same message is delivered from a source node to an arbitrary number of destination nodes, is fundamental in supporting collective communication primitives including the application-level broadcast, reduction, and barrier synchronization. This paper addresses how to efficiently implement multicast services in wormhole-routed multistage networks, in the absence of hardware multicast support, by exploiting the properties of the turnaround switching technology. An optimal multicast algorithm is proposed. The results of implementations on a 64-node SP-1 show that the proposed algorithm significantly outperforms the application-level broadcast primitives provided by currently existing collective communication libraries including the public domain MPI.