Dragonfly Topology Research Articles

AMLR: An Adaptive Multi-Level Routing Algorithm for Dragonfly Network

High-radix hierarchical structures, such as the dragonfly, fat-tree, and torus, are cost-effective topologies for high-performance computer (HPC) networks. In these networks, dragonfly outperforms traditional topologies such as fat-tree and torus in cost and scalability. However, network congestion occurs due to the imbalanced traffic pattern, which can lead to degraded performance. The routing algorithm influences the performance of the dragonfly topology in many ways. Routing algorithm can be designed to avoid saturating global or local links, and to avoid deadlock in the network. In this letter, we introduce an adaptive multi-level routing (AMLR) for dragonfly networks. AMLR has three-level routes. By dividing these routes meticulously, all paths of the network can be used more effectively. Traffic between groups will be more balanced. In particular, we propose a congestion control scheme to cooperate with AMLR in the data transmission process. Furthermore, congestion detection and notification are leveraged to identify congested packet and inform the network. Evaluations show that the proposed adaptive multi-level routing and congestion control mechanism can relieve the imbalance between groups in the 100-node dragonfly topology. As a result, AMLR provides 26%, 98%, 78%, and 99% lower latencies, and 13%, 87%, 13%, and 128% higher throughputs compared to the shortest routing under uniform, adv+i, hotspot, and permutation traffic, respectively.

On the performance investigation of a fast optical switches based optical high performance computing infrastructure

Optical networks based on fast optical switches (FOSes) could potentially solve the latency, bandwidth, cost and power consumption challenges in current electrical switches (ESes) based high performance computing (HPC) networks. In this work, we present a novel HPC network which employs distributed FOS interconnecting by removing ES (Firefly). In Firefly, Dragonfly topology is adopted for the inter-group connection of blades, while the intra-group connection of blades is implemented by FOS with fast optical flow control. The Firefly exploits the wavelength, space, and time switching domain with nanoseconds reconfiguration time of the FOS to achieve efficient statistical multiplexing operation. We numerically investigate the Firefly performance with real HPC traffic traces collected by running multiple computing applications in MareNostrum 3 HPC infrastructure with Leaf-Spine architecture. Compared with Leaf Spine architecture, results show that Firefly performs 62.4%, 54%, 68.6%, and 71.8% less latency for the applications Conjugate Gradient (CG), Multi-Grid (MG), Multiple Instruction Lattice Computation (MILC), and Miniature Molecular Dynamics (MINI_MD), respectively. Moreover, Firefly can save 56.4% cost and 65.7% power consumption, respectively, with respect to Leaf-Spine when both support around 10,000 blades.

A scheduling policy to save 10% of communication time in parallel fast Fourier transform

SummaryThe fast Fourier transform (FFT) has applications in almost every frequency related study, for example, in image and signal processing, and radio astronomy. It is also used as a Poisson operator inversion kernel in partial differential equations in fluid flows, in density functional theory, many‐body theory, and others. The three‐dimensional FFT has large time complexity . Hence, parallelization is used to compute such FFTs. Popular libraries perform slab division or pencil decomposition of data. None of the existing libraries achieve perfect inverse scaling of time with cores because FFT requires all‐to‐all communication and clusters hitherto do not have physical all‐to‐all connections. Dragonfly, one of the popular topologies for the interconnect, supports hierarchical connections among the components. We show that if we align the all‐to‐all communication of FFT with the physical connections of Dragonfly topology we will achieve a better scaling and reduce communication time.

Leveraging InfiniBand controller to configure deadlock-free routing engines for Dragonflies

The Dragonfly topology is currently one of the most popular network topologies in high-performance parallel systems. The interconnection networks of many of these systems are built from components based on the InfiniBand specification. However, due to some constraints in this specification, the available versions of the InfiniBand network controller (OpenSM) do not include routing engines based on some popular deadlock-free routing algorithms proposed theoretically for Dragonflies, such as the one proposed by Kim and Dally based on Virtual-Channel shifting. In this paper we propose a straightforward method to integrate this routing algorithm in OpenSM as a routing engine, explaining in detail the configuration required to support it. We also provide experiment results, obtained both from a real InfiniBand-based cluster and from simulation, to validate the new routing engine and to compare its performance and requirements against other routing engines currently available in OpenSM.

Dragonfly addressing model for software defined networks based on datacenters

With the advancement of technology, virtualization has become very important for Information Technology (IT) experts. Network Functions Virtualization (NFV) means to address issues resulting from complex hardware-based appliances by developing standard IT virtualization technologies. Software Defined Networking (SDN) solidifies the advantages of datacenter virtualization, increases resource flexibility and utilization, and reduces infrastructure costs and overhead. Datacenter networks should have the ability to guarantee high throughput and resiliency. For such reasons, typical datacenter networks (e.g. Fat Tree) have been evolved to high-radix networks (e.g. Dragonfly). This work aims to investigate how SDN and NFV can improve the advantages of datacenter virtualization by utilizing datacenter topologies such as Dragonfly (DF) topology and Fat Tree (FT) topology in SDN, thus expanding resource flexibility and utilization and diminishing infrastructure costs and overhead. By using Dragonfly topology, the cost is reduced and better scalability is introduced compared to the folded clos networks such as Fat Tree. Here in, a novel addressing scheme is proposed for Dragonfly topology with simulation results included utilizing Mininet, which incorporates MiniEdit that is used to create and run network simulations.

Scalable Deadlock-Free Deterministic Minimal-Path Routing Engine for InfiniBand-Based Dragonfly Networks

Dragonfly topologies are gathering great interest nowadays as one of the most promising interconnect options for High-Performance Computing (HPC) systems. However, Dragonflies contain physical cycles that may lead to traffic deadlocks unless the routing algorithm prevents them properly. In general, existing deadlock-free routing algorithms, either deterministic or adaptive, proposed for Dragonflies, use Virtual Channels (VCs) to prevent cyclic dependencies. However, these topology-aware algorithms are difficult to implement, or even unfeasible, in systems based on the InfiniBand (IB) architecture, which is nowadays the most widely used network technology in HPC systems. This is due to some limitations in the IB specification, specifically regarding the way Virtual Lanes (VLs), which are considered as similar to VCs, can be assigned to traffic flows. Indeed, none of the routing engines currently available in the official releases of the IB control software has been specifically proposed for Dragonflies. In this paper, we present a new deterministic, minimal-path routing for Dragonfly that prevents deadlocks using VLs according to the IB specification, so that it can be straightforwardly implemented in IB-based networks. We have called this proposal D3R (Deterministic Deadlock-free Dragonfly Routing). Specifically, D3R maps each route to a single, specific VL depending on the destination group, and according to a specific order, so that cyclic dependencies (so deadlocks) are prevented. D3R is scalable as it requires only 2 VLs to prevent deadlocks regardless of network size, i.e., fewer VLs than the required by the deadlock-free routing engines available in IB that are suitable for Dragonflies. Alternatively, D3R achieves higher throughput if an additional VL is used to reduce internal contention in the Dragonfly groups. We have implemented D3R as a new routing engine in OpenSM, the control software including the subnet manager in IB. We have evaluated D3R by means of simulation and by experiments performed in a real IB-based cluster, the results showing that, in general, D3R outperforms other routing engines.

Providing differentiated services, congestion management, and deadlock freedom in dragonfly networks with adaptive routing

SummaryThe number of endnodes in high‐performance computing systems has grown significantly in the last years. Hence, the interconnection network has become an essential issue as it may end up being the system bottleneck if it is not properly designed. In that sense, the Dragonfly topology has become very popular for interconnecting high‐performance computing systems in the last years because it offers high performance at an affordable cost. However, when using deterministic minimal‐path routing, this topology is not able to offer a high performance under certain traffic conditions. This problem can be solved by using oblivious or adaptive routing. However, there are no congestion management techniques specially tailored to Dragonfly topologies using oblivious or adaptive routing. Note that in congestion situations, the Dragonfly performance may drop because of the head‐of‐line blocking effect. This effect could be even more dangerous in systems where several applications with different priorities coexist. In this work we propose several techniques especially designed for providing differentiated services and congestion management in Dragonfly networks using oblivious or adaptive routing. First, we propose the hierarchical 3‐level queuing queuing scheme, which configures several virtual channels distributed into 3 virtual networks to reduce the head‐of‐line blocking while deadlocks derived from the routing algorithm are prevented. Second, we extend hierarchical 3‐level queuing to provide differentiated services through 2 different solutions. Finally, some experiments are performed to show the benefits obtained by using the proposed techniques.

Topological Characterization of Hamming and Dragonfly Networks and Its Implications on Routing

Current High-Performance Computing (HPC) and data center networks rely on large-radix routers. Hamming graphs (Cartesian products of complete graphs) and dragonflies (two-level direct networks with nodes organized in groups) are some direct topologies proposed for such networks. The original definition of the dragonfly topology is very loose, with several degrees of freedom, such as the inter- and intragroup topology, the specific global connectivity, and the number of parallel links between groups (or trunking level). This work provides a comprehensive analysis of the topological properties of the dragonfly network, providing balancing conditions for network dimensioning, as well as introducing and classifying several alternatives for the global connectivity and trunking level. From a topological study of the network, it is noted that a Hamming graph can be seen as a canonical dragonfly topology with a high level of trunking. Based on this observation and by carefully selecting the global connectivity, the Dimension Order Routing (DOR) mechanism safely used in Hamming graphs is adapted to dragonfly networks with trunking. The resulting routing algorithms approximate the performance of minimal, nonminimal, and adaptive routings typically used in dragonflies but without requiring virtual channels to avoid packet deadlock, thus allowing for lower cost router implementations. This is obtained by properly selecting the link to route between groups based on a graph coloring of network routers. Evaluations show that the proposed mechanisms are competitive with traditional solutions when using the same number of virtual channels and enable for simpler implementations with lower cost. Finally, multilevel dragonflies are discussed, considering how the proposed mechanisms could be adapted to them.

Photonic Interconnects for Exascale and Datacenter Architectures

Exascale and datacenter systems require terabits per second of internode communication bandwidth to meet the performance demands of high-performance computing applications. High-radix routers combined with scalable dragonfly topology have been proposed to reduce execution time and improve power dissipation. Although the dragonfly network has low diameter for exascale networks, fewer global links reduce the bisection bandwidth and require adaptive routing to prevent hot spots due to congestion. Moreover, the number of ports in a high-radix router affects the router cost when implemented with alternate emerging technologies. In this article, the authors advocate multitier network topologies that combine scalable topologies for local (intracabinet) and global (intercabinet) interconnects such as the k-ary n-cube, the flattened butterfly, and the dragonfly, to lead to improved bisection, manageable radix, and reduced link costs, albeit at higher packet latency owing to increased diameter. Because the performance per watt delivered by metallic interconnects or coaxial cables significantly exceeds the available power budget, we envision an entire exascale network composed of photonic links for communication and CMOS routers for switching. Results indicate that multitier topologies are comparable to the single-level dragonfly topology in terms of power and latency while providing higher bisection and reduced area overhead.

Indirect adaptive routing on large scale interconnection networks

Recently proposed high-radix interconnection networks [10] require global adaptive routing to achieve optimum performance. Existing direct adaptive routing methods are slow to sense congestion remote from the source router and hence misroute many packets before such congestion is detected. This paper introduces indirect global adaptive routing (IAR) in which the adaptive routing decision uses information that is not directly available at the source router. We describe four IAR routing methods: credit round trip (CRT) [10], progressive adaptive routing (PAR), piggyback routing (PB), and reservation routing (RES). We evaluate each of these methods on the dragonfly topology under both steady-state and transient loads. Our results show that PB, PAR, and CRT all achieve good performance. PB provides the best absolute performance, with 2-7% lower latency on steady-state uniform random traffic at 70% load, while PAR provides the fastest response on transient loads. We also evaluate the implementation costs of the indirect adaptive routing methods and show that PB has the lowest implementation cost requiring <1% increase in the total storage of a typical high-radix router.

Cost-Efficient Dragonfly Topology for Large-Scale Systems

It is more efficient to use increasing pin bandwidth by creating high-radix routers with a large number of narrow ports instead of low-radix routers with fewer wide ports. building networks using high-radix routers lowers cost and improves performance, but also presents many challenges. the dragonfly topology minimizes network cost by reducing the number of global channels required.

Technology-Driven, Highly-Scalable Dragonfly Topology

Evolving technology and increasing pin-bandwidth motivate the use of high-radix routers to reduce the diameter, latency, and cost of interconnection networks. High-radix networks, however, require longer cables than their low-radix counterparts. Because cables dominate network cost, the number of cables, and particularly the number of long, global cables should be minimized to realize an efficient network. In this paper, we introduce the dragonfly topology which uses a group of high-radix routers as a virtual router to increase the effective radix of the network. With this organization, each minimally routed packet traverses at most one global channel. By reducing global channels, a dragonfly reduces cost by 20% compared to a flattened butterfly and by 52% compared to a folded Clos network in configurations with ≥ 16K nodes.We also introduce two new variants of global adaptive routing that enable load-balanced routing in the dragonfly. Each router in a dragonfly must make an adaptive routing decision based on the state of a global channel connected to a different router. Because of the indirect nature of this routing decision, conventional adaptive routing algorithms give degraded performance. We introduce the use of selective virtual-channel discrimination and the use of credit round-trip latency to both sense and signal channel congestion. The combination of these two methods gives throughput and latency that approaches that of an ideal adaptive routing algorithm.