Torus Networks Research Articles

Adaptive routing for n-Dimensional Twin torus

Torus topology is one of the most common topologies used in the current largest supercomputers due to its properties related to cost, implementation or scalability. N-dimensional twin torus (nDT) topology has been proposed to increase the number of dimensions of the torus networks when port-limited low cost expansion cards are available. These topologies have been characterized and evaluated considering only deterministic routing. Adaptive routing algorithms improve communication performance exploiting the path diversity of the torus networks. Due to the particular properties of the nDT torus, designing an adaptive routing algorithm presents a challenge. The peculiarities of the internal link, which interconnects the two communication cards of an nDT torus node, complicate the design of the adaptive routing. In this paper, we study these peculiarities and propose an adaptive routing for nDT tori. Moreover, we show that, by using cards with the same number of ports, we can improve the network performance by building an adaptive nDT torus instead of an adaptive nD torus.

Torus–Connected Cycles: A Simple and Scalable Topology for Interconnection Networks

Abstract Supercomputers are today made up of hundreds of thousands of nodes. The interconnection network is responsible for connecting all these nodes to each other. Different interconnection networks have been proposed; high performance topologies have been introduced as a replacement for the conventional topologies of recent decades. A high order, a low degree and a small diameter are the usual properties aimed for by such topologies. However, this is not sufficient to lead to actual hardware implementations. Network scalability and topology simplicity are two critical parameters, and they are two of the reasons why modern supercomputers are often based on torus interconnection networks (e.g., Fujitsu K, IBM Sequoia). In this paper we first describe a new topology, torus-connected cycles (TCCs), realizing a combination of a torus and a ring, thus retaining interesting properties of torus networks in addition to those of hierarchical interconnection networks (HINs). Then, we formally establish the diameter of a TCC, and deduce a point-to-point routing algorithm. Next, we propose routing algorithms solving the Hamiltonian cycle problem, and, in a two dimensional TCC, the Hamiltonian path one. Correctness and complexities are formally proved. The proposed algorithms are time-optimal.

Scheduling Divisible Loads in Gaussian, Mesh and Torus Network of Processors

In this paper, we propose a novel analysis method for divisible load scheduling in mesh, torus and Gaussian network , a new type of interconnection network that has the same node degree as the mesh and torus, but shorter network diameter and shorter average hop distances under equal network size. The divisible scheduling in these three networks are uniformly formulated as the Maximum Finish Time Minimization (MFTM) problem. It involves minimizing the makespan of the load distribution and processing. The MTFM problem, a relaxed MFTM problem, a linear programming problem version and a heuristic algorithm are described and solved. The first three of these problems have identical solutions. The heuristic algorithm is close in performance to the optimal solution, significantly outperforms the previously described dimensional algorithm, and has much wider application range than the previously proposed phase algorithm.

Low-cost and low-power unidirectional torus network-on-chip with corner buffer power-gating

ABSTRACTNetwork-on-chip (NoC) is one of critical communication architectures for the scaling of future many-core processors. The challenge for on-chip network is reducing design complexity to save both area and power while providing high performance such as low latency and high throughput. Especially, with increase of network size, both design complexity and power consumption have become the bottlenecks preventing proper network scaling. Moreover, as technology continuously scales down, leakage power takes up a larger fraction of total NoC power. It is increasingly important for a power-efficient NoC design to reduce the increasing leakage power. Power-gating, as a representative low-power technique, can be applied to an on-chip network for mitigating leakage power. In this paper, we propose a low-cost and low-power router architecture for the unidirectional torus network, and adopt an improved corner buffer structure for the inoffensive power-gating, which has minimal impact on network performance. Besides, an explicit starvation avoidance mechanism is introduced to guarantee injection fairness while decreasing its negative impact on network throughput. Simulation results with synthetic traffic show that our design can improve network throughput by 11.3% on average and achieve significant power-saving in low- and medium-load regions. In the SPLASH-2 workload simulation, our design can save on average 27.2% of total power compared to the baseline, and decrease 42.8% average latency compared to the baseline with power-gating.

Torus Data Center Network With Smart Flow Control Enabled by Hybrid Optoelectronic Routers [Invited

A torus-topology photonic data center network with smart flow control is presented, where optical packet switching (OPS), optical circuit switching (OCS), and virtual OCS (VOCS) schemes are all enabled on a unified hardware platform via the combination of hybrid optoelectronic routers (HOPR) and a centralized network controller. An upgraded HOPR is being developed with the goal of handling 100 Gbps optical packets with a high energy efficiency of 90 mW/Gbps and low latency of less than 100 ns. The architecture of HOPR and its enabling technologies are reviewed, including a label processor, optical switch, and optoelectronic shared buffer. We also explain the concept and operation mechanisms of both the data and control planes in the hybrid OPS/OCS/VOCS torus network. The performance of these three transmission modes is evaluated by numerical simulations for a network composed of 4096 HOPRs.

HTM: a new hierarchical interconnection network for future generation parallel computers

ABSTRACTIn this paper, we have discussed the architectural structure, static network performance, and dynamic communication performance of a new hierarchical interconnection network called hierarchical Tori connected mesh network (HTM). For the exploration of static network performance, we have evaluated degree, diameter, cost, average distance, arc-connectivity, bisection width, and wiring complexity. We have also evaluated the dynamic communication performance of HTM, its counter rival H3DM, and conventional mesh and torus networks using a deadlock-free dimension order routing using two virtual channels under uniform and non-uniform traffic patterns. The dynamic communication performance is evaluated using computer simulation. We discovered that the HTM has a number of lucrative properties. These include constant node degree, small diameter, low cost, small average distance, moderate (neither too low, nor too high) bisection width, and less wiring complexity. HTM also yields high throughput per link and very low zero load latency, which provide better dynamic communication performance than that of H3DM, mesh, and torus networks.

Paired 2-disjoint path covers of multidimensional torus networks with faulty edges

A paired k-disjoint path cover (paired k-DPC for short) of a graph is a set of k disjoint paths joining k distinct source-sink pairs that cover all vertices of the graph. Clearly, the paired k-DPC is stronger than Hamiltonian-connectivity. The n-dimensional torus T(k1,k2,…,kn) (including the k-ary n-cube Qnk) is one of the most popular interconnection networks. In this paper, we obtain the following results. (1) Assume even ki≥4 for i=1,2,…,n. Let T=T(k1,k2,…,kn) be a bipartite torus and F be a set of faulty edges with |F|≤2n−3. Given any four vertices s1,t1,s2 and t2, such that each partite set contains two vertices. Then the graph T−F has a paired 2-DPC consisting of s1−t1 path and s2−t2 path. And the upper bound 2n−3 of edge faults tolerated is optimal. The result is a generalization of the result of Park et al. concerning the case of n=2[17]. (2) Assume ki≥3 for i=1,2,…,n, with at most one ki being even. Let T=T(k1,k2,…,kn) be a torus and F be a set of faulty edges with |F|≤2n−4. Then the graph T−F has a paired 2-DPC. And the upper bound 2n−4 of edge faults tolerated is nearly optimal. The result is a generalization of the result of Park concerning the case of n=2[16]. Our brief proofs are based on a technique that is of interest and may find some applications.

Better traffic distribution one-to-all broadcast in higher dimensional Gaussian networks

A higher dimensional Gaussian network, $$G_\alpha ^n$$G?n, has been proposed in Shamaei et al. (IEEE 28th international parallel and distributed processing symposium workshops, pp 1438---1447, 2014) as a useful alternative of the classical multi-dimensional torus network. The one-to-all broadcast is a well-known algorithm that spreads a message from one node to all other nodes in the network. This paper presents an improved one-to-all broadcasting algorithm in $$G_\alpha ^n$$G?n that achieves a lower average number of steps to receiving the broadcasted message. Furthermore, the all-to-all broadcasting method is illustrated based on dividing the proposed one-to-all broadcasting algorithm into four phases. The analytical and simulation results show that the proposed algorithm achieves better traffic performance than the current known algorithm and has 4.1 % less total number of senders.

N-Dimensional Twin Torus Topology

Torus topology is one of the preferred topologies for the interconnection network in high-performance clusters and supercomputers. Cost and scalability are some of the properties that make torus suitable for systems with a large number of nodes. The 3D torus is the version more extended due to its excellent nearest neighbor. However, some of the last supercomputers have been built using a torus network with five or six dimensions. To obtain an $nD$ torus, $2n$ ports per node are needed, which can be offered by a single or several cards per node. In the second case, there are multiple ways of assigning the dimension and direction of the card ports. In previous work we defined and characterized the 3D Twin (3DT) torus which uses two four-port cards per node. In this paper we extend that previous work to define the $n$ -dimensional Twin ( $n$ DT) torus topology. In this case, we formally obtain the optimal port configuration when ( $n$ + 1)-port cards are used instead of $2n$ -port cards. Moreover, we explain how deadlock problem can appear and propose a simple solution. Finally, we include evaluation results which show performance increases when an $nDT$ torus is used instead of an $nD$ torus with fewer dimensions and with the same computational resources.

All-To-All Broadcast in Hexagonal Torus Networks On-Chip

Hexagonal torus networks are special family of Eisenstein-Jacobi (EJ) networks which have gained popularity as goodcandidates network On-Chip (NoC) for interconnecting Multiprocessor System-on-Chips (MPSoCs). They showed better topological properties compared to the 2D torus networks with the same number of nodes. All-to-all broadcast is a collective communicationalgorithm used frequently in some parallel applications. Recently, an off-chip all-to-all broadcast algorithm has been proposed forhexagonal torus networks assuming half-duplex links and all-ports communication. The proposed all-to-all broadcast algorithm does not achieve the minimum transmission time and requires 24 k extra buffers, where k is the network diameter. We first extend this work by proposing an efficient all-to-all broadcast on hexagonal torus networks under full-duplex links and all-ports communications assumptions which achieves the minimum transmission delay but requires 3 6 k extra buffers per router. In a second stage, we develop a new all-to-all broadcast more suitable for hexagonal torus network on-chip that achieves optimal transmission delay time without requiring any extra buffers per router. By reducing the amount of buffer space, the new all-to-all broadcast reduces the routers cost which is an important issue in NoCs architectures.

Lattice Graphs for High-Scale Interconnection Topologies

Torus networks of moderate degree have been widely used in the supercomputer industry. Tori are superb when used for executing applications that require near-neighbor communications. Nevertheless, they are not so good when dealing with global communications. Hence, typical 3D implementations have evolved to 5D networks, among other reasons, to reduce network distances. Most of these big systems are mixed-radix tori, which are not the best option for minimizing distances and efficiently using network resources. This paper is focused on improving the topological properties of this kind of networks. By using integral matrices to deal with Cayley graphs over Abelian groups, we have been able to propose and analyze a family of high-dimensional mesh-based interconnection networks. As they are built over $n$ -dimensional grids that induce a regular tiling of space, these topologies have been denoted lattice graphs . Higher dimensional networks can be composed over these graphs by means of a lift operation, which is also introduced in the paper. Easy network partitioning and minimal routing algorithm are also provided for these topologies based on this new network operation. Later we focus on cubic crystal lattices for modeling symmetric 3D networks and to show how lattice graphs can help in the design of twisted interconnection networks. In all cases, the networks obtained are better, in topological terms, than their standard tori counterparts. Finally, some practical issues such as implementability and preliminary performance evaluations have been addressed at the end of this work.

Convergence Analysis for Regular Wireless Consensus Networks

Average consensus algorithms can be implemented over wireless sensor networks (WSN), where global statistics can be computed using communications among sensor nodes locally. Simple execution, robustness to global topology changes due to frequent node failures and underlying distributed philosophy has made consensus algorithms more suitable to WSNs. Since these algorithms are iterative in nature, their performance is characterized by convergence speed. We study the convergence of the average consensus algorithms for WSNs using regular graphs. We obtained the analytical expressions for optimal consensus and convergence parameters which decides the convergence time for r-nearest neighbor cycle and torus networks. We have also derived the generalized expression for optimal consensus and convergence parameters for m-dimensional r-nearest neighbor torus networks. The obtained analytical results agree with the simulation results and shown the effect of network dimension, number of nodes and transmission radius on convergence time. This work provides the basic analytical tools for managing and controlling the performance of average consensus algorithm in the finite sized practical networks.

Novo‐G#: a multidimensional torus‐based reconfigurable cluster for molecular dynamics

SummaryMolecular dynamics (MD) is a large‐scale, communication‐intensive problem that has been the subject of high‐performance computing research and acceleration for years. Not surprisingly, the most success in accelerating MD comes from specialized systems such as the Anton machine. In this paper, we describe Novo‐G# (novo‐jee‐sharp), a multi‐node reconfigurable system designed for the acceleration of communication‐intensive scientific problems in general, and MD in particular. This system provides a high‐bandwidth, low‐latency 3D torus network to allow direct communication between kernels running on multiple field‐programmable gate arrays. We also present a performance model for Novo‐G# running the 3D Fast Fourier Transform (FFT) kernel that forms the core of MD simulations. We validate the model against published Anton performance data and through initial hardware experiments on Novo‐G#. Finally, through simulation studies, we show that this system at scale performs better than specialized systems like Anton and outperforms established CPU‐based clusters like Blue Gene/Q by an order of magnitude for the 3D FFT kernel, with greater flexibility and lower costs. Copyright © 2015 John Wiley & Sons, Ltd.

Large-scale, high-speed tsunami prediction for the Great Nankai Trough Earthquake on the K computer

We improved the tsunami simulation code JAGURS, which is a paralleled version of URSGA code for a large-scale, high-speed tsunami prediction in the Nankai trough, Japan. We optimized the loop kernel for velocity update and intergrid communication on a three-dimensional torus network. Linear scaling was achieved up to the full system capability of the K computer (82,944 nodes) in a strong scaling test that used 100 billion finite-difference grid points. The measured performance on the K computer was 1.2 petaflops (11.5% of peak speed). Intergrid communication was optimized for a three-nested-grid model consisting of 0.68 billion grid points. Grid spacing in the area with the finest grid (180 km × 120 km) was about 5 m. We successfully implemented a large-scale tsunami simulation using this model that ran in about 30% of real time. We believe that this is the fastest tsunami prediction achieved to date with such a large-scale model. Our code can provide high-resolution tsunami prediction for broad regions within a reasonable time to assist emergency rescue and relief operations during future devastating tsunamis comparable to the 2004 Sumatra, 2010 Chile, and 2011 Tohoku tsunamis.

Balancing virtual channel utilization for deadlock-free routing in torus networks

Torus networks have been widely used by modern commercial supercomputers due to low node degree and linear scalability cost. Meanwhile, the ring in each dimension of a torus creates cyclic channel dependencies, which pose challenges to the design of deadlock-free routing and virtual channel allocation schemes. Existing virtual channel allocation schemes such as dateline employ two virtual channels to avoid deadlock in a ring. While the unbalanced utilization of virtual channels caused by these deadlock avoidance schemes brings lots of performance pathologies. This paper focuses on improving deadlock-free routing algorithms in torus by balancing virtual channel utilization. First, we propose BVCU, a novel virtual channel allocation scheme that can balance the utilization of the two virtual channels used to prevent deadlock. The technique is deadlock free and can achieve almost perfect balanced utilization. Second, we propose a balanced dimension-order routing (BDOR) to provide deadlock-free deterministic routing for $$n$$n-D torus by combining BVCU with dimension-order routing. Also, a balanced fully adaptive routing (BFAR) is proposed. These routing algorithms yield higher performance than existing solutions as they achieve a proper balanced utilization of virtual channels. Finally, simulation results show that the proposed methods achieve up to 16 % throughput improvement.

On All-to-All Broadcast in Dense Gaussian Network On-Chip

Gaussian networks are gaining popularity as good candidates Network On-Chip (NoC) for interconnecting Multiprocessor System-on-Chips (MPSoCs). They showed better topological properties compared to the 2D torus networks with the same number of nodes $N$ and the same degree 4. All-to-all broadcast is a collective communication algorithm used frequently in many parallel applications. Recently, Z. Zhang et al. [1] have proposed an all-to-all broadcast algorithm for Gaussian on-chip networks that achieves the minimum delay time but requires 4 $k$ extra buffers per router, where $k$ is the network diameter. In this paper, we propose a new all-to-all broadcast algorithm for dense Gaussian on-chip networks that achieves the minimum delay time without requiring any extra buffers per router. In this paper, we propose a new all-to-all broadcast algorithm for dense Gaussian on-chip networks that achieves the minimum delay time without requiring any extra buffers per router. Along with low latency, reducing the amount of buffer space and power consumption are very important issues in NoCs architectures.

All-to-all broadcasting in torus Network on Chip

This paper proposes and evaluates the performance of an all-to-all broadcasting algorithm for a 2D torus Network on Chip (NoC). The proposed algorithm uses special spanning trees called NEWS spanning trees. These trees are link conflict free which implies that the communication steps of the all-to-all algorithm are contention free. The proposed all-to-all broadcasting algorithm is optimal in terms of transmission time and does not need any additional buffer memory like in the all-to-all algorithm for the 2D torus (IEEE Trans Comput 50:1029---1032, 2001). Reducing the amount of buffer space is a very important issue in NoC architectures. Our algorithm is therefore a more efficient solution for all-to-all broadcasting in 2D torus NoC multi-core systems compared to previously proposed algorithms.

Leaving One Slot Empty: Flit Bubble Flow Control for Torus Cache-Coherent NoCs

Short and long packets co-exist in cache-coherent NoCs. Existing designs for torus networks do not efficiently handle variable-size packets. For deadlock free operations, a design uses two VCs, which negatively affects the router frequency. Some optimizations use one VC. Yet, they regard all packets as maximum-length packets, inefficiently utilizing the precious buffers. We propose flit bubble flow control (FBFC), which maintains one free flit-size buffer slot to avoid deadlock. FBFC uses one VC, and does not treat short packets as long ones. It achieves both high frequency and efficient buffer utilization. FBFC performs 92.8 and 34.2 percent better than LBS and CBS for synthetic traffic in a <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$4 \times 4$</tex> <mathgraphic fileref="ma-ieq1-2295523.gif" graphicformat="GIF"/></formula> torus. The gains increase in larger networks; they are 107.2 and 40.1 percent in an <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$8 \times 8$</tex> <mathgraphic fileref="ma-ieq2-2295523.gif" graphicformat="GIF"/></formula> torus. FBFC achieves an average 13.0 percent speedup over LBS for PARSEC workloads. Our results also show that FBFC is more power efficient than LBS and CBS, and a torus with FBFC is more power efficient than a mesh.

Collective Algorithms for Multiported Torus Networks

Modern supercomputers with torus networks allow each node to simultaneously pass messages on all of its links. However, most collective algorithms are designed to only use one link at a time. In this work, we present novel multiported algorithms for the scatter, gather, all-gather, and reduce-scatter operations. Our algorithms can be combined to create multiported reduce, all-reduce, and broadcast algorithms. Several of these algorithms involve a new technique where we relax the MPI message-ordering constraints to achieve high performance and restore the correctordering using an additional stage of redundant communication. According to our models, on an n -dimensional torus, our algorithms should allow for nearly a 2 n -fold improvement in communication performance compared to known, single-ported torus algorithms. In practice, we have achieved nearly 6x better performance on a 32k-node 3-dimensional torus.

BTL - An Efficient Deadlock-Free Multicast Wormhole Algorithm to Optimize Traffic in 2D Torus Multicomputers

Multicast communication, in which the same message is sending from a source node to a set of destination nodes, is being increasingly demanded in multicomputer systems. It can be used to support several other collective communication operations. 2D torus network has many features. So, it has become increasingly important to multicomputer design. This paper presents an efficient multicast wormhole deadlock-free algorithm that Balance Traffic Load on 2D torus network; hence the name BTL algorithm. BTL algorithm handles multicast operation with a fixed number of message-passing steps irrespective of the network size. Also, it is designed such that can send messages to any number of destinations within two communication phases. Results from extensive comparative analysis show that BTL algorithm exhibit superior performance advantages over a well-known algorithm. General Terms Network, Multicomputer, and Algorithms.