Dragonfly Networks Research Articles

Fault-tolerant unicast paths constructive algorithms in a family of recursive networks

ABSTRACT In parallel and distributed systems, interconnection networks and data center networks are two crucial networks. They are also crucial parts of high-performance computing and cloud computing services. With the rapid growth of high-performance computing and cloud computing, both networks are growing in size. At the same time, vertex faults are unavoidable. In computer/communication networks, unicast refers to one-to-one communication between the source vertex and the destination vertex. Fault-tolerant routing of vertices in networks is an important problem that has been widely discussed. In this paper, for a class of recursive networks with less than g -restrict connectivity fault vertices and each fault-free vertex having at least g fault-free neighbors, we provide a fault-tolerant unicast path design algorithm in this study. The algorithm’s correctness is then demonstrated and its time complexity is analyzed. In addition, the results obtained can be used for unknown and known networks, including the dragonfly network, DCell, and generalized DCell.

Efficient implementation of multi-level Dragonfly networks with Hamming graph for future optical networks

Efficient implementation of multi-level Dragonfly networks with Hamming graph for future optical networks

Pancyclic And Hamiltonian Properties Of Dragonfly Networks

Abstract Dragonfly networks have significant advantages in data exchange due to the small network diameter, low cost and modularization. A graph $G$ is $vertex$-$pancyclic$ if for any vertex $u\in V(G)$, there exist cycles through $u$ of every length $\ell $ with $3\leq \ell \leq |V(G)|$. A graph $G$ is $Hamiltonian$-$connected$ if there exists a Hamiltonian path between any two distinct vertices $u,v\in V(G)$. In this paper, we mainly research the pancyclic and Hamiltonian properties of the dragonfly network $D(n,h)$, and find that it is Hamiltonian with $n\geq 1,\,\,h\geq 2$, pancyclic, vertex-pancyclic and Hamiltonian-connected with $n\geq 4,\,\,h\geq 2$.

Reducing the Impact of Interjob Interference in Dragonfly Networks Using Virtual Partitions

In High-Performance Computing systems, the interference among applications that share network resources is one of the major causes of performance degradation and increase variability among different executions of the same application. In this paper we propose a Virtual Partitioning scheme for Dragonfly networks that combines a job-allocation policy with a virtual channel assignment. Our experiments confirm that our proposal offers better performance than the established schemes, also reducing variability.

Connectivity and constructive algorithms of disjoint paths in dragonfly networks

Dragonfly networks have been widely used in the current High Performance Computing (HPC) computers due to lower global network diameter and other advantages of communication performance such as modularity and cost-effectiveness. The original definition of the dragonfly network was very loose on account of its uncertain and diversified global link arrangements. In this paper, we study the logical structure of the dragonfly network which can be treated as a compound graph of complete graphs. Firstly, we give the general definition of the dragonfly network, named DF(n,h,g), and the specific definition of the dragonfly network under the relative global link arrangement, named D(n,h). Then, we prove that the connectivity of D(n,h) is n−1+h. In the end, we propose an O(n) algorithm to give the disjoint path between any two distinct vertices in D(n,h) and analyze the maximum length of these disjoint paths which is no more than 7.

AMLR: An Adaptive Multi-Level Routing Algorithm for Dragonfly Network

High-radix hierarchical structures, such as the dragonfly, fat-tree, and torus, are cost-effective topologies for high-performance computer (HPC) networks. In these networks, dragonfly outperforms traditional topologies such as fat-tree and torus in cost and scalability. However, network congestion occurs due to the imbalanced traffic pattern, which can lead to degraded performance. The routing algorithm influences the performance of the dragonfly topology in many ways. Routing algorithm can be designed to avoid saturating global or local links, and to avoid deadlock in the network. In this letter, we introduce an adaptive multi-level routing (AMLR) for dragonfly networks. AMLR has three-level routes. By dividing these routes meticulously, all paths of the network can be used more effectively. Traffic between groups will be more balanced. In particular, we propose a congestion control scheme to cooperate with AMLR in the data transmission process. Furthermore, congestion detection and notification are leveraged to identify congested packet and inform the network. Evaluations show that the proposed adaptive multi-level routing and congestion control mechanism can relieve the imbalance between groups in the 100-node dragonfly topology. As a result, AMLR provides 26%, 98%, 78%, and 99% lower latencies, and 13%, 87%, 13%, and 128% higher throughputs compared to the shortest routing under uniform, adv+i, hotspot, and permutation traffic, respectively.

ACOR: Adaptive congestion-oblivious routing in dragonfly networks

Low-diameter network topologies require non-minimal routing to avoid network congestion, such as Valiant routing. This increases base latency but avoids congestion issues. Optimized restricted variants focus on reducing path length. However, these optimizations only reduce paths for local traffic, where source and destination of each packet belong to the same partition of the network. This paper introduces ACOR: Adaptive Congestion-Oblivious Routing. ACOR leverages the restricted and recomputation mechanisms to reduce path length for local and global traffic, and extends it when the network conditions are adverse. ACOR relies on a sequence of misrouting policies ordered by path length. A hysteresis mechanism improves performance and avoids variability in the results. The ACOR mechanism can be combined with other non-minimal routing mechanism such as Piggyback. Results show that ACOR improves base latency in all cases, up to 28% standalone and up to 25.5% when combined with Piggyback, while requiring a simple implementation.

Fault-Tolerant Adaptive Routing in Dragonfly Networks

Dragonfly networks have been widely used in the current high-performance computers or high-end servers. Fault-tolerant routing in dragonfly networks is essential. The rich interconnects provide good fault-tolerance ability for the network. A new deadlock-free adaptive fault-tolerant routing algorithm based on a new two-layer safety information model, is proposed by mapping routers in a group, and groups of the dragonfly network into two separate hypercubes. The new fault-tolerant routing algorithm tolerates static and dynamic faults. Our method can determine whether a packet can reach the destination at the source by using the new safety information model, which avoids dead-ends and aimless misrouting. Sufficient simulation results show that the proposed fault-tolerant routing algorithm even outperforms the previous minimal routing algorithm in fault-free networks in many cases.

TPR: Traffic Pattern-Based Adaptive Routing for Dragonfly Networks

The Cray Cascade architecture uses Dragonfly as its interconnect topology and employs a globally adaptive routing scheme called UGAL. UGAL directs traffic based on link loads but may make inappropriate adaptive routing decisions in various situations, which degrades its performance. In this work, we propose traffic pattern-based adaptive routing (TPR) for Dragonfly that improves UGAL by incorporating a traffic pattern-based adaptation mechanism. The idea is to explicitly use the link usage statistics that are collected in performance counters to infer the traffic pattern, and to take the inferred traffic pattern plus link loads into consideration when making adaptive routing decisions. Our performance evaluation results on a diverse set of traffic conditions indicate that by incorporating the traffic pattern-based adaptation mechanism, TPR is much more effective in making adaptive routing decisions and achieves significant lower latency under low load and higher throughput under high load than its underlying UGAL.

Software-defined networking control plane for seamless integration of multiple silicon photonic switches in Datacom networks.

Silicon photonics based switches offer an effective option for the delivery of dynamic bandwidth for future large-scale Datacom systems while maintaining scalable energy efficiency. The integration of a silicon photonics-based optical switching fabric within electronic Datacom architectures requires novel network topologies and arbitration strategies to effectively manage the active elements in the network. We present a scalable software-defined networking control plane to integrate silicon photonic based switches with conventional Ethernet or InfiniBand networks. Our software-defined control plane manages both electronic packet switches and multiple silicon photonic switches for simultaneous packet and circuit switching. We built an experimental Dragonfly network testbed with 16 electronic packet switches and 2 silicon photonic switches to evaluate our control plane. Observed latencies occupied by each step of the switching procedure demonstrate a total of 344 µs control plane latency for data-center and high performance computing platforms.

A visual analytics system for optimizing the performance of large-scale networks in supercomputing systems

The overall efficiency of an extreme-scale supercomputer largely relies on the performance of its network interconnects. Several of the state of the art supercomputers use networks based on the increasingly popular Dragonfly topology. It is crucial to study the behavior and performance of different parallel applications running on Dragonfly networks in order to make optimal system configurations and design choices, such as job scheduling and routing strategies. However, in order to study these temporal network behavior, we would need a tool to analyze and correlate numerous sets of multivariate time-series data collected from the Dragonfly’s multi-level hierarchies. This paper presents such a tool–a visual analytics system–that uses the Dragonfly network to investigate the temporal behavior and optimize the communication performance of a supercomputer. We coupled interactive visualization with time-series analysis methods to help reveal hidden patterns in the network behavior with respect to different parallel applications and system configurations. Our system also provides multiple coordinated views for connecting behaviors observed at different levels of the network hierarchies, which effectively helps visual analysis tasks. We demonstrate the effectiveness of the system with a set of case studies. Our system and findings can not only help improve the communication performance of supercomputing applications, but also the network performance of next-generation supercomputers.

Providing differentiated services, congestion management, and deadlock freedom in dragonfly networks with adaptive routing

Providing differentiated services, congestion management, and deadlock freedom in dragonfly networks with adaptive routing

A scalable synthetic traffic model of Graph500 for computer networks analysis

SummaryThe Graph500 benchmark attempts to steer the design of High‐Performance Computing systems to maximize the performance under memory‐constricted application workloads. A realistic simulation of such benchmarks for architectural research is challenging due to size and detail limitations. By contrast, synthetic traffic workloads constitute one of the least resource‐consuming methods to evaluate the performance. In this work, we provide a simulation tool for network architects that need to evaluate the suitability of their interconnect for BigData applications. Our development is a low computation‐ and memory‐demanding synthetic traffic model that emulates the behavior of the Graph500 communications and is publicly available in an open‐source network simulator. The characterization of network traffic is inferred from a profile of several executions of the benchmark with different input parameters. We verify the validity of the equations in our model against an execution of the benchmark with a different set of parameters. Furthermore, we identify the impact of the node computation capabilities and network characteristics in the execution time of the model in a Dragonfly network.

Providing differentiated services, congestion management, and deadlock freedom in dragonfly networks with adaptive routing

SummaryThe number of endnodes in high‐performance computing systems has grown significantly in the last years. Hence, the interconnection network has become an essential issue as it may end up being the system bottleneck if it is not properly designed. In that sense, the Dragonfly topology has become very popular for interconnecting high‐performance computing systems in the last years because it offers high performance at an affordable cost. However, when using deterministic minimal‐path routing, this topology is not able to offer a high performance under certain traffic conditions. This problem can be solved by using oblivious or adaptive routing. However, there are no congestion management techniques specially tailored to Dragonfly topologies using oblivious or adaptive routing. Note that in congestion situations, the Dragonfly performance may drop because of the head‐of‐line blocking effect. This effect could be even more dangerous in systems where several applications with different priorities coexist. In this work we propose several techniques especially designed for providing differentiated services and congestion management in Dragonfly networks using oblivious or adaptive routing. First, we propose the hierarchical 3‐level queuing queuing scheme, which configures several virtual channels distributed into 3 virtual networks to reduce the head‐of‐line blocking while deadlocks derived from the routing algorithm are prevented. Second, we extend hierarchical 3‐level queuing to provide differentiated services through 2 different solutions. Finally, some experiments are performed to show the benefits obtained by using the proposed techniques.

Deadlock-Free Broadcast Routing in Dragonfly Networks without Virtual Channels

A new deadlock-free unicast-based broadcast scheme is proposed based on a new routing scheme called minus-first routing. Minus-first routing is a partially adaptive routing scheme in dragonfly networks without any virtual channels. The main goals of the broadcast schemes are to minimize the total delivery time, and any router does not receive any message more than once. No channel competition is introduced. Two different broadcast schemes are proposed: (1) the group-first, and (2) the router-first. It is shown that unicast-based broadcast schemes are necessary to avoid deadlocks at the consumption channels. The group-first broadcast scheme delivers a message to all groups as early as possible; and the router-first scheme minimizes the number of unicast steps to traverse global links. To our knowledge, the method in this paper is the first collective communication work for dragonfly networks in the literature. Simulation results are presented to evaluate the proposed unicast-based broadcast schemes.

Straightforward solutions to reduce HoL blocking in different Dragonfly fully-connected interconnection patterns

The performance of interconnection networks is a challenging issue for High-Performance Computing (HPC) systems, which becomes even more important when the number of interconnected endnodes grows. In that sense, Dragonfly interconnection patterns are a very popular option to configure the network topology, especially for large systems, as they are able to achieve a high scalability relying on high-radix switches. This kind of hierarchical topologies has two levels of interconnection (i.e., connections within the element of a group and connections among groups) and each one can be interconnected using different patterns. However, regardless of the Dragonfly interconnection pattern, the Head-of-Line (HoL) blocking effect derived from congestion situations may jeopardize the Dragonfly performance. This paper analyzes the dynamics of congestion in different Dragonfly fully-connected interconnection patterns. Also, we describe a queuing scheme called Hierarchical Two-Level Queuing (H2LQ), designed specially to reduce HoL blocking in any fully-connected Dragonfly network that uses minimal-path routing. Finally, we present experiment results which show that this scheme significantly boost Dragonfly performance, regardless the interconnection pattern, especially when congestion arises, while requiring fewer network resources than other techniques oriented to deal with the effects of congestion.

Network unfairness in dragonfly topologies

Dragonfly networks arrange network routers in a two-level hierarchy, providing a competitive cost-performance solution for large systems. Non-minimal adaptive routing (adaptive misrouting) is employed to fully exploit the path diversity and increase the performance under adversarial traffic patterns. Network fairness issues arise in the dragonfly for several combinations of traffic pattern, global misrouting and traffic prioritization policy. Such unfairness prevents a balanced use of the resources across the network nodes and degrades severely the performance of any application running on an affected node. This paper reviews the main causes behind network unfairness in dragonflies, including a new adversarial traffic pattern which can easily occur in actual systems and congests all the global output links of a single router. A solution for the observed unfairness is evaluated using age-based arbitration. Results show that age-based arbitration mitigates fairness issues, especially when using in-transit adaptive routing. However, when using source adaptive routing, the saturation of the new traffic pattern interferes with the mechanisms employed to detect remote congestion, and the problem grows with the network size. This makes source adaptive routing in dragonflies based on remote notifications prone to reduced performance, even when using age-based arbitration.

On-the-fly adaptive routing for dragonfly interconnection networks

Adaptive deadlock-free routing mechanisms are required to handle variable traffic patterns in dragonfly networks. However, distance-based deadlock avoidance mechanisms typically employed in Dragonflies increase the router cost and complexity as a function of the maximum allowed path length. This paper presents on-the-fly adaptive routing (OFAR), a routing/flow-control scheme that decouples the routing and the deadlock avoidance mechanisms. OFAR allows for in-transit adaptive routing with local and global misrouting, without imposing dependencies between virtual channels, and relying on a deadlock-free escape subnetwork to avoid deadlock. This model lowers latency, increases throughput, and adapts faster to transient traffic than previously proposed mechanisms. The low capacity of the escape subnetwork makes it prone to congestion. A simple congestion management mechanism based on injection restriction is considered to avoid such issues. Finally, reliability is considered by introducing mechanisms to find multiple edge-disjoint Hamiltonian rings embedded on the dragonfly, allowing to use multiple escape subnetworks.

Topological Characterization of Hamming and Dragonfly Networks and Its Implications on Routing

Current High-Performance Computing (HPC) and data center networks rely on large-radix routers. Hamming graphs (Cartesian products of complete graphs) and dragonflies (two-level direct networks with nodes organized in groups) are some direct topologies proposed for such networks. The original definition of the dragonfly topology is very loose, with several degrees of freedom, such as the inter- and intragroup topology, the specific global connectivity, and the number of parallel links between groups (or trunking level). This work provides a comprehensive analysis of the topological properties of the dragonfly network, providing balancing conditions for network dimensioning, as well as introducing and classifying several alternatives for the global connectivity and trunking level. From a topological study of the network, it is noted that a Hamming graph can be seen as a canonical dragonfly topology with a high level of trunking. Based on this observation and by carefully selecting the global connectivity, the Dimension Order Routing (DOR) mechanism safely used in Hamming graphs is adapted to dragonfly networks with trunking. The resulting routing algorithms approximate the performance of minimal, nonminimal, and adaptive routings typically used in dragonflies but without requiring virtual channels to avoid packet deadlock, thus allowing for lower cost router implementations. This is obtained by properly selecting the link to route between groups based on a graph coloring of network routers. Evaluations show that the proposed mechanisms are competitive with traditional solutions when using the same number of virtual channels and enable for simpler implementations with lower cost. Finally, multilevel dragonflies are discussed, considering how the proposed mechanisms could be adapted to them.

Photonic Interconnects for Exascale and Datacenter Architectures

Exascale and datacenter systems require terabits per second of internode communication bandwidth to meet the performance demands of high-performance computing applications. High-radix routers combined with scalable dragonfly topology have been proposed to reduce execution time and improve power dissipation. Although the dragonfly network has low diameter for exascale networks, fewer global links reduce the bisection bandwidth and require adaptive routing to prevent hot spots due to congestion. Moreover, the number of ports in a high-radix router affects the router cost when implemented with alternate emerging technologies. In this article, the authors advocate multitier network topologies that combine scalable topologies for local (intracabinet) and global (intercabinet) interconnects such as the k-ary n-cube, the flattened butterfly, and the dragonfly, to lead to improved bisection, manageable radix, and reduced link costs, albeit at higher packet latency owing to increased diameter. Because the performance per watt delivered by metallic interconnects or coaxial cables significantly exceeds the available power budget, we envision an entire exascale network composed of photonic links for communication and CMOS routers for switching. Results indicate that multitier topologies are comparable to the single-level dragonfly topology in terms of power and latency while providing higher bisection and reduced area overhead.