2D-torus Network Research Articles

An Extensive Power and Performance Analysis for High Dimensional Mesh and Torus Interconnection Networks

The next generation parallel computers are the keys to achieve exascale performance, whereas sequential computers have already been saturated. In order to achieve this mighty target of exascale computing, one of the main challenges is the reduction of power consumption along with achieving suitable performance. Energy efficiency is a key feature to ensure the trade-off between the performances over the required power usages. Hence, to focus on those issues, the target of this article is to analyze the performance versus power usage trade-off for the conventional networks like- Mesh and Torus. High degree networks show much better performance than the low degree of networks. However, high degree networks require higher power usage for their high degree of interconnected links. This article showed that with zero load latency, the 3D Torus could show about 57.07% better performance than a 2D Torus. On the other hand, a 2D Mesh network requires about 24.22% less router power usage than the 3D Mesh, & 5D Torus requires about 66.8% higher router power usage than a 3D Torus network.

3D-TTN: a power efficient cost effective high performance hierarchical interconnection network for next generation green supercomputer.

Green computing is an important factor to ensure the eco-friendly use of computers and their resources. Electric power used in a computer converts into heat and thus, the system takes fewer watts ensuring less cooling. This lower energy consumption allows to be less costly to run as well as reduces the environmental impact of powering the computer. One of the most challenging problems for the modern green supercomputers is the reduction of current power consumptions. Consequently, regular conventional interconnection networks also show poor cost performance. On the other hand, hierarchical interconnection networks (like-3D-TTN) can be a possible solution to those issues. The main focus for this paper is the estimation of power usage at the on-chip level for 3D-TTN with the various other networks along with the analysis of static network performance. In our analysis, 3D-TTN requires about 32.48% less router power usage at the on-chip level and can also achieve near about 21% better diameter performance as well as 12% better average distance performance than the 5D-Torus network. Similarly, it also requires only about 14.43% higher router power usage; however, can achieve 23.21% better diameter performance and 26.3% better average distance than recent hierarchical interconnection network- 3D-TESH. The most attractive feature of this paper is the static hop distance parameter and per watt analysis (power-performance). According to our power-performance results, 3D-TTN can also show better result than the 3D-Mesh, 2D-Mesh, 2D-Torus and 3D-TESH network even at the lowest network level. Moreover, this paper is also featured with the static effectiveness analysis, which ensures cost and time efficiency of 3D-TTN.

Lea-TN: leader election algorithm considering node and link failures in a torus network

Torus network topology offers many advantages such as higher speed, lower latency, better fairness, and lower energy consumption. For these kinds of benefits, nowadays, it is used to construct many parallel and distributed systems like IBM Blue Gene, IBM Sequoia, Mira, and Sugon TC8600. In parallel and distributed computing, multiple nodes act together to carry out large tasks fast. Hence, coordination is indispensable among these nodes to complete such tasks. A possible coordination method could be to elect a leader among the nodes. Along with coordination, the elected leader node also manages other activities such as task allocation, result aggregation, clock synchronization, proficient resource sharing management, and communication among the system’s nodes. In this paper, we propose a lower bound $$\varOmega (N\log _3 N)$$ of message complexity on a comparison-based leader election for a 2D torus network (where N is the number of nodes in the network). Next, we sketch a new leader election algorithm (Lea-TN) considering both the node and link failures for a 2D torus network. This Lea-TN is a deterministic and robust algorithm that elects a leader for a synchronous distributed system. The algorithm chooses a leader, even when there are some link or node failures in the system. We consider the number of non-faulty links and the subsisting nodes’ failure rate to elect a reliable leader. We introduce new patterns for sending messages that help reduce the number of exchanged messages and the execution time of the election process. The proposed algorithm (Lea-TN) enables a node to identify its link failures during the election also. Further, we simulate the Lea-TN algorithm and compare its performance with that of the well-known existing algorithms.

Fault-Tolerant Adaptive Routing Algorithm for 2D Torus Network

A 2D torus network is one of the most popular networks for parallel processing. We have researched the North-South First (NSF) routing which is applicable to a 2D torus and combines the north-first (NF) and south-first (SF) methods. We focused on the proposal of a routing algorithm aimed at avoiding congestion of the crowded network. It was superior in congestion tolerance but not in fault tolerance. We have therefore been researching algorithms considering fault tolerance of the NSF method. In this paper we propose an NSF-FT method which is a new routing algorithm with improved fault tolerance. We evaluated the congestion resistance and fault tolerance of the proposed method by dynamic communication performance evaluation by simulation. The software simulation showed that the proposed algorithm has higher performance.

Constructing virtual 5‐dimensional tori out of lower‐dimensional network cards

SummaryIn the Top500 and Graph500 lists of the last years, some of the most powerful systems implement a torus topology to interconnect the millions of computing nodes they include. Some of these torus networks are of five or six dimensions, which implies an additional difficulty as the node degree increases. In previous works, we proposed and evaluated the nD Twin (nDT) torus topology to virtually increase the dimensions a torus is able to implement. We showed that this new topology reduces the distances between nodes, increasing, therefore, global network performance. In this work, we present how to build a 5DT torus network using a specific commercial 6‐port network card (EXTOLL card) to interconnect those nodes. We show, using the same number of cards, that the performance of the 5DT torus network we are able to implement using our proposal is higher than the performance of the 3D torus network for the same number of compute nodes.

The Performance Evaluation of a 3D Torus Network Using Partial Link-Sharing Method in NoC Router Buffer

The high performance network-on-chip (NoC) router using minimal hardware resources to minimize the layout area is very essential for NoC design. In this paper, we have proposed a memory sharing method of a wormhole routed NoC architecture to alleviate the area overhead of a NoC router. In the proposed method, a memory is shared by multiple physical links by using a multi-port memory. In this paper, we have proposed a partial link-sharing method and evaluated the communication performance using the proposed method. It is revealed that the resulted communication performance by the proposed methods is higher than that of the conventional method, and the progress ratio of the 3D-torus network is higher than that of 2D-torus network. It is shown that the improvement of communication performance using partial link sharing method is achieved with slightly increase of hardware cost. Copyright © 2017 The Institute of Electronics, Information and Communication Engineers.

A new power efficient high performance interconnection network for many-core processors

Next generation high performance computing will most likely depend on the massively parallel computers. The overall performance of a massively parallel computer system is heavily affected by the interconnection network and its processing nodes. Continuing advances in VLSI technologies promise to deliver more power to individual nodes. However, the on-chip interconnection networks consume up to 50% of the total chip power and off-chip bandwidth is limited to the maximum number of possible out going physical links. In addition, the long wiring and low performance of communication network overwhelm the benefit of parallel computer system whereas it increases total cost. In this paper, we propose a new interconnection network that reduces the problems of high power consumption, long wiring length and low bandwidth issues. We have measured the static network performance and required power consumption of our proposed ‘3D-TESH’ interconnection network and compared the performance with other networks at different levels of hierarchy such as inter-chips, inter-nodes and inter-cabinets. 3D-TESH network has achieved about 52.08% better diameter and about 45.71% better average distance than the 3D-Torus network with 12.61% less router power usage at on-chip level. Furthermore, 3D-TESH requires about 41% less router power usage than 5D-Torus at the on-chip level.

All-To-All Broadcast in Hexagonal Torus Networks On-Chip

Hexagonal torus networks are special family of Eisenstein-Jacobi (EJ) networks which have gained popularity as goodcandidates network On-Chip (NoC) for interconnecting Multiprocessor System-on-Chips (MPSoCs). They showed better topological properties compared to the 2D torus networks with the same number of nodes. All-to-all broadcast is a collective communicationalgorithm used frequently in some parallel applications. Recently, an off-chip all-to-all broadcast algorithm has been proposed forhexagonal torus networks assuming half-duplex links and all-ports communication. The proposed all-to-all broadcast algorithm does not achieve the minimum transmission time and requires 24 k extra buffers, where k is the network diameter. We first extend this work by proposing an efficient all-to-all broadcast on hexagonal torus networks under full-duplex links and all-ports communications assumptions which achieves the minimum transmission delay but requires 3 6 k extra buffers per router. In a second stage, we develop a new all-to-all broadcast more suitable for hexagonal torus network on-chip that achieves optimal transmission delay time without requiring any extra buffers per router. By reducing the amount of buffer space, the new all-to-all broadcast reduces the routers cost which is an important issue in NoCs architectures.

Novo‐G#: a multidimensional torus‐based reconfigurable cluster for molecular dynamics

SummaryMolecular dynamics (MD) is a large‐scale, communication‐intensive problem that has been the subject of high‐performance computing research and acceleration for years. Not surprisingly, the most success in accelerating MD comes from specialized systems such as the Anton machine. In this paper, we describe Novo‐G# (novo‐jee‐sharp), a multi‐node reconfigurable system designed for the acceleration of communication‐intensive scientific problems in general, and MD in particular. This system provides a high‐bandwidth, low‐latency 3D torus network to allow direct communication between kernels running on multiple field‐programmable gate arrays. We also present a performance model for Novo‐G# running the 3D Fast Fourier Transform (FFT) kernel that forms the core of MD simulations. We validate the model against published Anton performance data and through initial hardware experiments on Novo‐G#. Finally, through simulation studies, we show that this system at scale performs better than specialized systems like Anton and outperforms established CPU‐based clusters like Blue Gene/Q by an order of magnitude for the 3D FFT kernel, with greater flexibility and lower costs. Copyright © 2015 John Wiley & Sons, Ltd.

On All-to-All Broadcast in Dense Gaussian Network On-Chip

Gaussian networks are gaining popularity as good candidates Network On-Chip (NoC) for interconnecting Multiprocessor System-on-Chips (MPSoCs). They showed better topological properties compared to the 2D torus networks with the same number of nodes $N$ and the same degree 4. All-to-all broadcast is a collective communication algorithm used frequently in many parallel applications. Recently, Z. Zhang et al. [1] have proposed an all-to-all broadcast algorithm for Gaussian on-chip networks that achieves the minimum delay time but requires 4 $k$ extra buffers per router, where $k$ is the network diameter. In this paper, we propose a new all-to-all broadcast algorithm for dense Gaussian on-chip networks that achieves the minimum delay time without requiring any extra buffers per router. In this paper, we propose a new all-to-all broadcast algorithm for dense Gaussian on-chip networks that achieves the minimum delay time without requiring any extra buffers per router. Along with low latency, reducing the amount of buffer space and power consumption are very important issues in NoCs architectures.

All-to-all broadcasting in torus Network on Chip

This paper proposes and evaluates the performance of an all-to-all broadcasting algorithm for a 2D torus Network on Chip (NoC). The proposed algorithm uses special spanning trees called NEWS spanning trees. These trees are link conflict free which implies that the communication steps of the all-to-all algorithm are contention free. The proposed all-to-all broadcasting algorithm is optimal in terms of transmission time and does not need any additional buffer memory like in the all-to-all algorithm for the 2D torus (IEEE Trans Comput 50:1029---1032, 2001). Reducing the amount of buffer space is a very important issue in NoC architectures. Our algorithm is therefore a more efficient solution for all-to-all broadcasting in 2D torus NoC multi-core systems compared to previously proposed algorithms.

BTL - An Efficient Deadlock-Free Multicast Wormhole Algorithm to Optimize Traffic in 2D Torus Multicomputers

Multicast communication, in which the same message is sending from a source node to a set of destination nodes, is being increasingly demanded in multicomputer systems. It can be used to support several other collective communication operations. 2D torus network has many features. So, it has become increasingly important to multicomputer design. This paper presents an efficient multicast wormhole deadlock-free algorithm that Balance Traffic Load on 2D torus network; hence the name BTL algorithm. BTL algorithm handles multicast operation with a fixed number of message-passing steps irrespective of the network size. Also, it is designed such that can send messages to any number of destinations within two communication phases. Results from extensive comparative analysis show that BTL algorithm exhibit superior performance advantages over a well-known algorithm. General Terms Network, Multicomputer, and Algorithms.

Architectural improvements and technological enhancements for theAPEnet+ interconnect system

The APEnet+ board delivers a point-to-point, low-latency, 3D torus network interface card. In this paper we describe the latest generation of APEnet NIC, APEnet v5, integrated in a PCIe Gen3 board based on a state-of-the-art, 28 nm Altera Stratix V FPGA. The NIC features a network architecture designed following the Remote DMA paradigm and tailored to tightly bind the computing power of modern GPUs to the communication fabric. For the APEnet v5 board we show characterizing figures as achieved bandwidth and BER obtained by exploiting new high performance ALTERA transceivers and PCIe Gen3 compliancy.

ASIP acceleration for virtual-to-physical address translation on RDMA-enabled FPGA-based network interfaces

We developed a point-to-point, low latency, 3D torus Network Controller integrated in an FPGA-based PCIe board which implements a Remote Direct Memory Access (RDMA) communication protocol. RDMA requires ability to directly access the remote node application memory with minimal OS or CPU intervention. To this purpose, a key element is the design of a direct memory writing mechanism to address the destination buffers; on Virtual Memory supporting OSes this corresponds to a number of page-segmented DMAs. To minimally affect overall performance, mechanisms with lowest possible latency are needed for either Virtual-to-Physical address translation and registered buffers list scanning. In a first implementation these tasks were set on a soft-core μC on the FPGA, leading to a 1.6 μs latency to process a single packet and limiting the peak bandwidth. As a second trial, we present an accelerated version for these time-critical network functions exploiting an application-specific processor (ASIP) designed using a retargetable ASIP development toolsuite that allows architectural exploration. Benchmark results for Buffer Search and Virtual-to-Physical tasks on the ASIP show improvements for latency with up to ten times lower cycles cost compared with the soft-core μC.

A tree-based algorithm for multicasting in 2D torus networks

Torus network has become increasingly important to multicomputer design because of its many desirable properties including low bandwidth and fixed degree of nodes. Also, torus networks can be partitioned into mesh networks. The multicast pattern, in which one source node sends the same message to multiple destination nodes, is the essential pattern in a wide variety of applications. This paper proposes a multicast routing scheme in 2D torus networks. The proposed scheme is a Tree-based Algorithm which Splits torus Networks into two Equally Meshes, hence it is called TASNEM. TASNEM algorithm is a tree-based technique, in which the router simultaneously sends incoming flits on more than one outgoing channel. It requires at most two start-up times, one for each mesh subnetwork. For each mesh subnetwork, the source node delivers a message to the destination nodes along one main path and different horizontal paths branched from the main path. TASNEM algorithm can achieve high degree of parallelism and low communication latency over a wide range of traffic loads. Performance results of a simulation study on torus networks are discussed to compare TASNEM algorithm with some previous algorithms.

Architectural improvements and 28 nm FPGA implementation of the APEnet+ 3D Torus network for hybrid HPC systems

Modern Graphics Processing Units (GPUs) are now considered accelerators for general purpose computation. A tight interaction between the GPU and the interconnection network is the strategy to express the full potential on capability computing of a multi-GPU system on large HPC clusters; that is the reason why an efficient and scalable interconnect is a key technology to finally deliver GPUs for scientific HPC. In this paper we show the latest architectural and performance improvement of the APEnet+ network fabric, a FPGA-based PCIe board with 6 fully bidirectional off-board links with 34 Gbps of raw bandwidth per direction, and X8 Gen2 bandwidth towards the host PC. The board implements a Remote Direct Memory Access (RDMA) protocol that leverages upon peer-to-peer (P2P) capabilities of Fermi- and Kepler-class NVIDIA GPUs to obtain real zero-copy, low-latency GPU-to-GPU transfers. Finally, we report on the development activities for 2013 focusing on the adoption of the latest generation 28 nm FPGAs and the preliminary tests performed on this new platform.

Analysis of performance improvements for host and GPU interface of the APENet+ 3D Torus network

APEnet+ is an INFN (Italian Institute for Nuclear Physics) project aiming to develop a custom 3-Dimensional torus interconnect network optimized for hybrid clusters CPU-GPU dedicated to High Performance scientific Computing. The APEnet+ interconnect fabric is built on a FPGA-based PCI-express board with 6 bi-directional off-board links showing 34 Gbps of raw bandwidth per direction, and leverages upon peer-to-peer capabilities of Fermi and Kepler-class NVIDIA GPUs to obtain real zero-copy, GPU-to-GPU low latency transfers. The minimization of APEnet+ transfer latency is achieved through the adoption of RDMA protocol implemented in FPGA with specialized hardware blocks tightly coupled with embedded microprocessor. This architecture provides a high performance low latency offload engine for both trasmit and receive side of data transactions: preliminary results are encouraging, showing 50% of bandwidth increase for large packet size transfers. In this paper we describe the APEnet+ architecture, detailing the hardware implementation and discuss the impact of such RDMA specialized hardware on host interface latency and bandwidth.

Eigenanalysis-based task mapping on parallel computers with cellular networks

Through eigenanalysis of communication matrices, we develop a new objective function formulation for mapping tasks to parallel computers with cellular networks. This new formulation significantly speeds up the solution process through consideration of the symmetries in the supply matrix of a network and a transformation of the demand matrix of any application. The extent of the speedup is not easily obtainable through analytical means for most production networks. However, numerical experiments of mapping wave equations on 2D mesh onto 3D torus networks by simulated annealing demonstrate a far superior convergence rate and quicker escape from local minima with our new formulation than with the standard graph theory-based one.

APEnet+ 34 Gbps data transmission system and custom transmission logic

APEnet+ is a point-to-point, low-latency, 3D-torus network controller integrated in a PCIe Gen2 board based on the Altera Stratix IV FPGA. We characterize the transmission system (embedded transceivers driving external QSFP+ modules) analyzing signal integrity, throughput, latency, BER and jitter at different data rates up to 34 Gbps. We estimate the efficiency of a custom logic able to sustain 2.6 GB/s per link with an FPGA on-chip memory footprint of 40 KB, providing deadlock-free routing and systemic awareness of faults. Finally, we show the preliminary results obtained with the embedded transceivers of a next-generation FPGA and outline some ideas to increase the performance with the same FPGA memory footprint.

Randomized Throughput-Optimal Oblivious Routing for Torus Networks

In this paper, we study the problem of optimal oblivious routing for 1D and 2D torus networks. We introduce a new closed-form oblivious routing algorithm called W2TURN that is worst-case throughput optimal for 2D torus networks. W2TURN is based on a weighted random selection of paths that contain at most two turns. Restricting the maximum number of turns in routing paths to just two enables a simple deadlock-free implementation of W2TURN. In terms of average hop count, W2TURN outperforms the best previously known closed-form worst-case throughput optimal routing algorithm called IVAL [CHECK END OF SENTENCE]. When the network radix is odd, W2TURN achieves the minimum average hop count that can be achieved with 2-turn paths while remaining worst-case throughput optimal. When the network radix is even, W2TURN comes very close to achieving the minimum average hop count while remaining worst-case throughput optimal, within just 0.72 percent on a 12\times 12 torus. We also describe another routing algorithm based on weighted random selection of paths with at most two turns called I2TURN and show that I2TURN is equivalent to IVAL. However, I2TURN eliminates the need for loop removal at runtime and provides a closed-form analytical expression for evaluating the average hop count. The latter enables us to demonstrate analytically that W2TURN strictly outperforms IVAL (and I2TURN) in average hop count. Finally, we present a new optimal weighted random routing algorithm for rings called Weighted Random Direction (WRD). WRD provides a closed-form expression for the optimal distribution of traffic along the minimal and nonminimal directions in a ring topology to achieve minimum average hop count while guaranteeing optimal worst-case throughput. Based on our evaluations, in addition to being worst-case throughput optimal, W2TURN and WRD also perform well in the average case, and outperform the best previously known worst-case throughput optimal routing algorithms with closed-form descriptions in latency and throughput over a wide range of traffic patterns.