Torus Networks Research Articles

Performance Analysis of a Scalable Algorithm for 3D Linear Transforms on Supercomputer with Intel Processors/Co-Processors

Abstract Practical realizations of 3D forward/inverse separable discrete transforms, such as Fourier transform, cosine/sine transform, etc. are frequently the principal limiters that prevent many practical applications from scaling to a large number of processors. Existing approaches, which are based primarily on 1D or 2D data decompositions, prevent the 3D transforms from effectively scaling to the maximum (possible/available) number of computer nodes. A highly scalable approach to realize forward/inverse 3D transforms has been proposed. It is based on a 3D decomposition of data and geared towards a torus network of computer nodes. The proposed algorithms requires compute-and-roll time-steps, where each step consists of an execution of multiple GEMM operations and concurrent movement of cubical data blocks between nearest neighbors. The aim of this paper is to present an experimental performance study of an implementation on high performance computer architecture.

TFBN: A Cost Effective High Performance Hierarchical Interconnection Network

In order to fulfill the increasing demand for computation power to process a boundless data concurrently within a very short time or real-time in many areas such as IoT, AI, machine learning, smart grid, and big data analytics, we need exa-scale or zetta-scale computation in the near future. Thus, to have this level of computation, we need a massively parallel computer (MPC) system that shall consist of millions of nodes; and, for the interconnection of these massive numbers of nodes, conventional topologies are infeasible. Thus, a hierarchical interconnection network (HIN) is a rational way to connect huge nodes. Through this article, we are proposing a new HIN, which is a tori-connected flattened butterfly network (TFBN) for the next generation MPC system. Numerous basic modules are hierarchically interconnected as a toroidal connection, whereby the basic modules are flattened butterfly networks. We have studied the network architecture, static network performance, and static cost-effectiveness of the proposed TFBN in detail; and compared static network and cost-effectiveness performance of the TFBN to those of TTN, torus, TESH, and mesh networks. It is depicted that TFBN possesses low diameter and average distance, high arc connectivity, and temperate bisection width. It also has better cost-effectiveness and cost-performance trade-off factor compared to those of TTN, torus, TESH, and mesh networks. The only shortcoming is that the complexity of wiring of the TFBN is higher than that of those networks; this is because the basic module necessitates some extra short length link to form the flattened butterfly network. Therefore, TFBN is a high performance and cost-effective HIN, and it will be a good option for the next generation MPC system.

Torus-like graphs and their paired many-to-many disjoint path covers

Given two disjoint vertex-sets, S={s1,…,sk} and T={t1,…,tk} in a graph, a paired many-to-manyk-disjoint path cover joining S and T is a set of pairwise vertex-disjoint paths {P1,…,Pk} that altogether cover every vertex of the graph, in which each path Pi runs from si to ti. In this paper, we propose a family of graphs, called torus-like graphs,that include torus networks, and reveal that a torus-like graph, if built from lower dimensional torus-like graphs that have good Hamiltonian and disjoint-path-cover properties, retain such good properties. As a result, every m-dimensional nonbipartite torus, m≥2, with at most f vertex and/or edge faults has a paired many-to-many k-disjoint path cover joining arbitrary disjoint sets S and T of size k each, subject to k≥2 and f+2k≤2m. The bound 2m on f+2k is nearly optimal.

Implementing a new Torus network with intra-chip encryption

In recent years, the ad hoc network has emerged as a solution to the challenges of designing high-performance complex systems at the nanoscale. Maintaining information security in complex systems is an important issue. Therefore, this paper presents a new topology based on the Torres network with encryption capability to build internal network interfaces on the chip with tens or even hundreds of processor units considering the need for information security. The main platform for inter-chip communication is the PRDT network (2.1) and the basic encryption algorithm, RC6. Designing a new node and modifying the source and switched source network PRDT (2,1) encrypts the algorithm based on the RC6 accelerated algorithm. Evaluations performed by the synthesis of this method on the FPGA shows that this provides a 21% overhead of approximately 6% increase in hardware resources that cannot be compromised.

Optimal Parallel Prefix Sum Computation on Three-Dimensional Torus Network

In this article, an optimal parallel algorithm is suggested for fast computation of a crucial yet fundamental numerical problem in science named ‘prefix sum’ on a popular three-dimensional torus network. The proposed parallel algorithm demands total ‘4n + 2’ communication moves for mapping prefix sum problem on odd network sizes of n × n×n three-dimensional torus network, whereas total ‘4n’ communication moves is required for parallel mapping of same on even network sizes. In fact, a unit increase in each dimension from odd to even and even to odd network sizes resulted in the constant increase of 2 and 6 units, respectively, in the required communication moves for mapping prefix sum. The algorithmic performance of the proposed parallel algorithm can be compared to counter-part parallel prefix sum approaches on grid-based networks.

Energy and performance improvements in stencil computations on multi-node HPC systems with different network and communication topologies

Energy and performance improvements in stencil computations are relevant for both application developers and data center administrators. They appear as the fundamental scheme in many large-scale scientific simulations and workloads. Many research efforts have focused on some estimation techniques of the energy usage of HPC systems based on specific characteristics of parallel applications. In case of stencils, we have previously concentrated on detailed estimations of energy consumption and the energy-aware distribution of stencil computations on heterogeneous processors. However, we have restricted our comprehensive studies to a single heterogeneous computing node only. In this paper, we show how scheduling and optimization techniques can be applied for energy and performance improvements of stencil computations on multi-node HPC systems using different network topologies. We formulate a scheduling model together with a new Tabu Search algorithm, called Task Movement (TM), taking into account the communication hierarchies, to minimize the overall energy usage and the execution time of stencil computations. Experimental studies show that this algorithm solves the considered problem more efficiently comparing to other, simpler heuristics. We present computational experiments for a reference 7 point stencil computation pattern on three commonly used low-diameter network topologies: Fat-tree, Dragonfly, and Torus. According to our studies, the most promising multi-node HPC architecture for stencil computations is based on the Torus network concept. Finally, we argue that the proposed scheduling model and TM algorithm can be easily adopted within existing high-level parallel execution environments for stencils automatic performance tuning.

Terminal and broadcast reliability analysis of direct 2-D symmetric torus network.

Reliability analysis is one of the crucial issues for any scalable optical interconnection network. Torus is a highly scalable optical interconnect for data centre networks. The traditional torus network has XY routing algorithm. We have proposed a novel optimised routing algorithm. This paper focuses on the time-dependent and time-independent analysis for both terminal and broadcast reliabilities of the torus network using XY and optimised routing algorithm under various network sizes (Ntimes N where N=8, 16, 32, 64). The results are evaluated and compared considering nodes failures in MATLAB.

An enhanced cost-aware mapping algorithm based on improved shuffled frog leaping in network on chips

Network on chip (NoC) has been of great interest in recent years. However, according to the recent studies, high communication cost has been raised as the one most important challenges in mapping process in NoC. In order to address these issues, this research takes the advantages of the improved shuffled frog leaping based on two-dimensional torus networks in figure of an enhanced algorithm. In other words, using the proposed mapping category features, increase in the number of nodes has very little effect on the provided performance by the proposed algorithm, because it first improves the write cost of each cluster with less instead of high communication cost. Moreover, nodes having greater relevance in the graph of application are often adjacent to each other. Therefore, while the communication cost is reduced, the search space mapping gets larger which leads to easier and less costly access to the optimal mapping. In fact, between two nodes with higher weight, it is less distance to write in the network substrate. This reduction reduces the distance between the two nodes in the end, reducing the cost of communication. Simulation results show that the communication cost is improved by 3.62, 0.90, 0.72 and 1.35% compared to PSMAP algorithm for the 263dec mp3dec graph, Lmap algorithm for the 263enc mp3dec graph, Lmap algorithm for the MWD graph and PSMAP algorithm for MPEG-4 graph, respectively.

Biswapped–Torus Network: A New Efficient Node-Symmetrical Optoelectronic Network

In this paper, a node-symmetrical optoelectronic grid-based variation of biswapped network framework named biswapped–torus network is introduced. It is a hybrid network that combines both biswapped network and torus. The proposed network claims to be a faster alternative as compared to biswapped–mesh network due to smaller network diameter. Also, node symmetry is an essential property of this network. Further, compared to efficient node-symmetrical optoelectronic biswapped framework named biswapped–hyper hexa-cell network, the presented network is cost-efficient and capable of delivering competitive performance, concerning network diameter for network size up to 1250.

Geometric Mapping of Tasks to Processors on Parallel Computers with Mesh or Torus Networks

We present a new method for reducing parallel applications’ communication time by mapping their MPI tasks to processors in a way that lowers the distance messages travel and the amount of congestion in the network. Assuming geometric proximity among the tasks is a good approximation of their communication interdependence, we use a geometric partitioning algorithm to order both the tasks and the processors, assigning task parts to the corresponding processor parts. In this way, interdependent tasks are assigned to “nearby” cores in the network. We also present a number of algorithmic optimizations that exploit specific features of the network or application to further improve the quality of the mapping. We specifically address the case of sparse node allocation, where the nodes assigned to a job are not necessarily located in a contiguous block nor within close proximity to each other in the network. However, our methods generalize to contiguous allocations as well, and results are shown for both contiguous and non-contiguous allocations. We show that, for the structured finite difference mini-application MiniGhost, our mapping methods reduced communication time up to 75 percent relative to MiniGhost's default mapping on 128K cores of a Cray XK7 with sparse allocation. For the atmospheric modeling code E3SM/HOMME, our methods reduced communication time up to 31% on 16K cores of an IBM BlueGene/Q with contiguous allocation.

Modeling Universal Globally Adaptive Load-Balanced Routing

Universal globally adaptive load-balanced (UGAL) routing has been proposed for various interconnection networks and has been deployed in a number of current-generation supercomputers. Although UGAL-based schemes have been extensively studied, most existing results are based on either simulation or measurement. Without a theoretical understanding of UGAL, multiple questions remain: For which traffic patterns is UGAL most suited? In addition, what determines the performance of the UGAL-based scheme on a particular network configuration? In this work, we develop a set of throughput models for UGALbased on linear programming. We show that the throughput models are valid across the torus, Dragonfly, and Slim Fly network topologies. Finally, we identify a robust model that can accurately and efficiently predict UGAL throughput for a set of representative traffic patterns across different topologies. Our models not only provide a mechanism to predict UGAL performance on large-scale interconnection networks but also reveal the inner working of UGAL and further our understanding of this type of routing.

Multi-axis decomposition of density functional program for strong scaling up to 82,944 nodes on the K computer: Compactly folded 3D-FFT communicators in the 6D torus network

Density functional calculations with a plane-wave basis set are widely used in materials science. Due to recent developments in high-performance computers, the number of nodes equipped in such computers greatly exceeds the number of atoms included in a typical simulation. Thus, it is becoming difficult to perform calculations efficiently even when only a portion of all nodes are used (e.g., 10%). We have developed a multi-axis decomposition scheme in which both G-vectors and band axes are decomposed and 3D-FFT communicators are folded compactly. This proposed scheme retains the inner-most do-loop lengths sufficiently long and restrains the increased MPI communication costs as the number of nodes increases. In an investigation of a wide-gap semiconductor material (SiC), our PHASE/0 DFT code exhibits efficient and strong scaling (up to 82,944 nodes) even for a relatively small system with 3848 atoms, and demonstrates maximum peak performance of 2.25 PFLOPS for a 25,200-atom system despite employing 3D-FFT.

HMMN: a cost-effective derivative of midimew-connected mesh network

A horizontal midimew-connected mesh network (HMMN) is constructed using multiple mesh-connected basic modules, whereby the basic modules are hierarchically interconnected by a horizontal midimew network for a higher level of hierarchy. In the HMMN, the vertical direction is toroidal connected and the horizontal direction is midimew links connected. This is why it is called HMMN. The architecture of the HMMN, the static network performance, and the dynamic communication performance were evaluated in our previous studies and were found to be good. However, the cost-effectiveness and cost-performance trade-off analysis of the HMMN have not been evaluated yet. Therefore, in this paper, we have evaluated packing density, message traffic density, cost-effective factor, time cost-effective factor, and static cost-performance trade-off factor of the proposed HMMN and compared these with those of mesh, torus, TESH, and MMN networks. Accordingly, the HMMN yields better results than the mesh, TESH, and MMN networks with respect to the above parameters. Even in regard to the extra cost of wrap-around links in all end-to-end nodes, the HMMN is more cost-effective than the torus network. However, this additional path between the end-to-end node using wrap-around links results in a low diameter, which in turn yields a slightly better density parameter and static cost-performance trade of factor for the torus network than for the HMMN.

The Study on Adaptive Routing Algorithm of 2-D Torus Network with Fault Tolerance

As an application of the turn model in a two-dimensional torus network, we propose a north-south first (NSF) algorithm. Until now, evaluations focusing on fault tolerance were inadequate because we focused on proposing a routing algorithm aimed at avoiding congestion. In this paper, we propose NFS-IP, which is a new routing algorithm with improved fault tolerance, along with an evaluation of the conventional NFS algorithm.

Energy efficient torus networks with on/off links

Future exascale computing systems will require energy and performance efficient interconnection networks to respond to the high data movement demands of new applications, such as those coming from big-data and artificial intelligence areas. The network structure plays a major role in the overall interconnect performance, for this reason torus is a common topology used in the current largest supercomputers. There are several proposals to improve energy efficiency of interconnection networks. However, few works combine both energy and performance, and sometimes they are treated as opposed issues. In this paper, we try to determine which torus network configuration offers the best performance/energy ratio when high-radix switches are used to build the interconnect system. The performance/energy evaluation has been performed by trace-driven simulation under realistic scenarios, where several mixes of scientific applications share a supercomputer system and are scheduled to be executed with the available resources at each moment.

Fault-Tolerant Adaptive Routing Algorithm for 2D Torus Network

A 2D torus network is one of the most popular networks for parallel processing. We have researched the North-South First (NSF) routing which is applicable to a 2D torus and combines the north-first (NF) and south-first (SF) methods. We focused on the proposal of a routing algorithm aimed at avoiding congestion of the crowded network. It was superior in congestion tolerance but not in fault tolerance. We have therefore been researching algorithms considering fault tolerance of the NSF method. In this paper we propose an NSF-FT method which is a new routing algorithm with improved fault tolerance. We evaluated the congestion resistance and fault tolerance of the proposed method by dynamic communication performance evaluation by simulation. The software simulation showed that the proposed algorithm has higher performance.

Angara interconnect makes GPU-based Desmos supercomputer an efficient tool for molecular dynamics calculations

In this article, we describe the Desmos supercomputer that consists of 32 hybrid nodes connected by a low-latency high-bandwidth Angara interconnect with torus topology. This supercomputer is aimed at cost-effective classical molecular dynamics calculations. Desmos serves as a test bed for the Angara interconnect that supports 3-D and 4-D torus network topologies and verifies its ability to unite massively parallel programming systems speeding-up effectively message-passing interface (MPI)-based applications. We describe the Angara interconnect presenting typical MPI benchmarks. Desmos benchmarks results for GROMACS, LAMMPS, VASP and CP2K are compared with the data for other high-performance computing (HPC) systems. Also, we consider the job scheduling statistics for several months of Desmos deployment.

Exploring Properties and Correlations of Fatal Events in a Large-Scale HPC System

In this paper, we explore potential correlations of fatal system events for one of the most powerful supercomputers—IBM Blue Gene/Q Mira, which is deployed at Argonne National Laboratory, based on its 5-year reliability, availability, and serviceability (RAS) log. Our contribution is two-fold. (1) We design an efficient log analysis tool, namely LogAider, with a novel filtering method to effectively extract fatal events from masses of system messages that are heavily duplicated in the log. LogAider exhibits a very precise detection of temporal-correlation with a high similarity (up to 95 percent) to the ground-truth (i.e., compared to the failure records reported by the administrators). The total number of fatal events can be reduced to about 1,255 compared with originally 2.6 million duplicated fatal messages. (2) We analyze the 5-year RAS log of the MIRA system using LogAider, and summarize six important “takeaways” which can help system vendors and administrators better understand an extreme-scale system's fatal events. Specifically, we find that the distribution or proportion of the fatal system events follow a Pareto-like principle in general. The temporal correlation among fatal events is much stronger than that of warn messages and info messages, and the correlated events tend to constitute a few clusters. The mean time between fatal events (MTBFE) of the Mira system is about 1.3 days from the perspective of the system, and the MTTI is 2–4 days from the perspective of users. The most error-prone item value with respect to any key attribute appears likely in the log every 2–10 days. Weibull, Gamma, and Pearson6 are the three best-fit distributions for the fatal event intervals. The overall correlation of fatal events on the 5D torus network is not prominent, whereas the small-region locality correlation (e.g., the fatal events inside racks) is relatively strong. We believe our work will be interesting to large-scale HPC system administrators and vendors and to fault tolerance researchers, enabling them to better understand fatal events and mitigate such events accordingly.

POWAR

In order to save energy in HPC interconnection networks, one usual proposal is to switch idle links into a low-power mode after a certain time without any transmission, as IEEE Energy Efficient Ethernet standard proposes. Extending the low-power mode mechanism, we propose POW er- A ware R outing ( POWAR ), a simple power-aware routing and selection function for fat-tree and torus networks. POWAR adapts the amount of network links that can be used, taking into account the network load, and obtaining great energy savings in the network (55%--65%) and the entire system (9%--10%) with negligible performance overhead.

Regular Connected Bipancyclic Spanning Subgraphs of Torus Networks

It is well known that an [Formula: see text]-dimensional torus [Formula: see text] is Hamiltonian. Then the torus [Formula: see text] contains a spanning subgraph which is 2-regular and 2-connected. In this paper, we explore a strong property of torus networks. We prove that for any even integer [Formula: see text] with [Formula: see text], the torus [Formula: see text] contains a spanning subgraph which is [Formula: see text]-regular, k-connected and bipancyclic; and if [Formula: see text] is odd, the result holds when some [Formula: see text] is even.