Efficient Interconnection Network Research Articles

A smart and novel approach for managing incast and in-network congestion through adaptive routing

High-Performance Computing and Datacenter systems, with numerous endnodes, demand an efficient interconnection network to prevent performance bottlenecks. Fat-Tree topologies are preferred for their high bisection bandwidth and multiple shortest-path routes. While existing adaptive routing excels in light or in-network congestion, it struggles with incast congestion. This paper proposes a new technique, called Congestion-Aware Adaptive Routing (SCAR), which addresses both in-network and incast congestion. SCAR limits adaptivity for incast congestion, using deterministic routing, while employing adaptive routing for non-congesting flows. It also resolves in-network congestion by routing traffic flows through alternative routes. Simulation experiments on large Fat-Trees using synthetic and trace-based traffic patterns modeling realistic applications demonstrate SCAR’s immediate reaction on mitigating in-network congestion, and a reasonable delay during incast situations, while other state-of-the-art solutions are not able to cope with incast and in-network situations at the same time.

Scalability Limitations of Processing-in-Memory using Real System Evaluations

Processing-in-memory (PIM), where the compute is moved closer to the memory or the data, has been widely explored to accelerate emerging workloads. Recently, different PIM-based systems have been announced by memory vendors to minimize data movement and improve performance as well as energy efficiency. One critical component of PIM is the large amount of compute parallelism provided across many PIM "nodes'' or the compute units near the memory. In this work, we provide an extensive evaluation and analysis of real PIM systems based on UPMEM PIM. We show that while there are benefits of PIM, there are also scalability challenges and limitations as the number of PIM nodes increases. In particular, we show how collective communications that are commonly found in many kernels/workloads can be problematic for PIM systems. To evaluate the impact of collective communication in PIM architectures, we provide an in-depth analysis of two workloads on the UPMEM PIM system that utilize representative common collective communication patterns -- AllReduce and All-to-All communication. Specifically, we evaluate 1) embedding tables that are commonly used in recommendation systems that require AllReduce and 2) the Number Theoretic Transform (NTT) kernel which is a critical component of Fully Homomorphic Encryption (FHE) that requires All-to-All communication. We analyze the performance benefits of these workloads and show how they can be efficiently mapped to the PIM architecture through alternative data partitioning. However, since each PIM compute unit can only access its local memory, when communication is necessary between PIM nodes (or remote data is needed), communication between the compute units must be done through the host CPU, thereby severely hampering application performance. To increase the scalability (or applicability) of PIM to future workloads, we make the case for how future PIM architectures need efficient communication or interconnection networks between the PIM nodes that require both hardware and software support.

A Novel Hybrid Hexagonal Star Topology for On-Chip Interconnection Networks

Network-on-Chip (NoC) is an emerging and efficient on-chip interconnection network. NoC is expected to be the communication backbone of next-generation Multi-processor System-on-Chip (MPSoC) architectures. Topology is a crucial design aspect of NoC, as it affects the performance of the interconnection network. This paper proposes a novel, scalable, hybrid Hexagonal Star (HS) topology for on-chip interconnection networks. Properties of the proposed topology have been explored and compared with those of Mesh, Torus and Honeycomb Mesh topologies. The performance of the Hexagonal Star topology has been evaluated and compared with that of Mesh topology for different scenarios. The comparative studies of topological properties have indicated that the proposed topology can be a potential choice for on-chip interconnection networks. The simulation results have shown that the proposed topology outperforms Mesh topology in terms of latency for low traffic loads. For different traffic patterns and traffic loads, HS topology has registered a reduction of packet latency ranging from 15% to 50% and from 9% to 23% for 18 nodes and 32 nodes, respectively, compared to Mesh topology.

HFBN: An Energy Efficient High Performance Hierarchical Interconnection Network for Exascale Supercomputer

Supercomputers are trying to be eco-friendly using their main components like- interconnection networks, processors, and shared memory through high power efficiency (GFlops/watts). The reason for this is the exascale systems, are on the horizon, require a 1000 times performance improvement over the petascale computers and energy efficiency has attracted as the key factor for to achieve exascale system. The main contribution of this paper is to introduce a new hierarchical interconnection network; considering simulations at the million processing cores especially for the exascale system. Moreover, our designed network claims the supremacy of high network performance and low power consumption over the conventional networks and ensuring the utmost preferability for exaFLOPS system. On the other hand, the performance per watt metric, used for the TOP500 list, doesn’t reflect the overall performance of a given system. Hence, one of the possible solutions to reach the next generation exascale performance is to redesign the "Interconnection Network", which holds the main responsibility for the intercommunication between the CPUs and also the power consumption for the supercomputers. This paper focuses on a redesigned new energy efficient interconnection network that mitigates the problems of high power consumption, longer wiring length, and low bandwidth issues. Our designed network (HFBN) has been compared against the Tofu network and in the case of 1M cores, HFBN can obtain about 87.26% better energy efficiency at the zero load latency with uniform traffic, about 86.32% with perfect shuffle traffic, and about 92.98% with the bit-compliment traffic.

3D-TTN: a power efficient cost effective high performance hierarchical interconnection network for next generation green supercomputer.

Green computing is an important factor to ensure the eco-friendly use of computers and their resources. Electric power used in a computer converts into heat and thus, the system takes fewer watts ensuring less cooling. This lower energy consumption allows to be less costly to run as well as reduces the environmental impact of powering the computer. One of the most challenging problems for the modern green supercomputers is the reduction of current power consumptions. Consequently, regular conventional interconnection networks also show poor cost performance. On the other hand, hierarchical interconnection networks (like-3D-TTN) can be a possible solution to those issues. The main focus for this paper is the estimation of power usage at the on-chip level for 3D-TTN with the various other networks along with the analysis of static network performance. In our analysis, 3D-TTN requires about 32.48% less router power usage at the on-chip level and can also achieve near about 21% better diameter performance as well as 12% better average distance performance than the 5D-Torus network. Similarly, it also requires only about 14.43% higher router power usage; however, can achieve 23.21% better diameter performance and 26.3% better average distance than recent hierarchical interconnection network- 3D-TESH. The most attractive feature of this paper is the static hop distance parameter and per watt analysis (power-performance). According to our power-performance results, 3D-TTN can also show better result than the 3D-Mesh, 2D-Mesh, 2D-Torus and 3D-TESH network even at the lowest network level. Moreover, this paper is also featured with the static effectiveness analysis, which ensures cost and time efficiency of 3D-TTN.

BSN MOT: A Fast and Cost-efficient Optoelectronic Architecture

Recently, an efficient recursive and symmetrical optoelectronic interconnection network based on biswapped framework named “Biswapped-Hyper Hexa-Cell (BSN HHC)” is reported. In spite of containing a prime advantage concerning network diameter compared to its competitive Biswapped-Mesh (BSN Mesh) optoelectronic architecture; a major limitation of the network is its high cost. In a search for a competitive alternative variation of this network, a fast and a cost-efficient member of biswapped family named “Biswapped-Mesh of Trees (BSN MOT)” is proposed here. The suggested BSN MOT optoelectronic network claims to be a better alternative of counter-part OTIS-Mesh of Trees (OTIS MOT) optoelectronic network, claims advantages concerning diameter, minimum node degree, and network total and optical cost (total cost drops about 10–30% and optical cost drops about 50%). More specifically, compared to competitive Biswapped-Mesh and BSN HHC optoelectronic networks, the proposed network has a major advantage concerning network diameter. Moreover, BSN MOT is a cost-efficient architecture, where total network cost drops about 30–45% than that of BSN HHC. Besides, it is a better alternative for processing numerical problems that require network with “low static maximum node degree”. Thus, on the basis of above-mentioned properties, the proposed network claims to be the most economical and the fastest variation of biswapped framework.

Lower bounds for dilation, wirelength, and edge congestion of embedding graphs into hypercubes

Interconnection networks provide an effective mechanism for exchanging data between processors in a parallel computing system. One of the most efficient interconnection networks is the hypercube due to its structural regularity, potential for parallel computation of various algorithms, and the high degree of fault tolerance. Thus it becomes the first choice of topological structure of parallel processing and computing systems. In this paper, lower bounds for the dilation, wirelength, and edge congestion of an embedding of a graph into a hypercube are proved. Two of these bounds are expressed in terms of the bisection width. Applying these results, the dilation and wirelength of embedding of certain complete multipartite graphs, folded hypercubes, wheels, and specific Cartesian products are computed.

Upper and Lower Bounds for the Kirchhoff Index of the n-Dimensional Hypercube Network

The hypercube Qn is one of the most admirable and efficient interconnection network due to its excellent performance for some practical applications. The Kirchhoff index KfG is equal to the sum of resistance distances between any pairs of vertices in networks. In this paper, we deduce some bounds with respect to Kirchhoff index of hypercube network Qn.

Energy efficient torus networks with on/off links

Future exascale computing systems will require energy and performance efficient interconnection networks to respond to the high data movement demands of new applications, such as those coming from big-data and artificial intelligence areas. The network structure plays a major role in the overall interconnect performance, for this reason torus is a common topology used in the current largest supercomputers. There are several proposals to improve energy efficiency of interconnection networks. However, few works combine both energy and performance, and sometimes they are treated as opposed issues. In this paper, we try to determine which torus network configuration offers the best performance/energy ratio when high-radix switches are used to build the interconnect system. The performance/energy evaluation has been performed by trace-driven simulation under realistic scenarios, where several mixes of scientific applications share a supercomputer system and are scheduled to be executed with the available resources at each moment.

On Cyclic Codes of Odd Lengths from the Stable Variety of Regular Cayley Graphs

The use of the adjacency matrix of a graph as a generator matrix for some classes of binary codes had been reported and studied. This paper concerns the utilization of the stable variety of Cayley regular graphs of odd order for efficient interconnection networks as studied, in the area of Codes Generation and Analysis. The Use of some succession scheme in the construction of a stable variety of the Cayley regular graph had been considered. We shall enumerate the adjacency matrices of the regular Cayley graphs so constructed which are of odd order (2m+1), for m≥3 as in [1]. Next, we would show that the Matrices are cyclic and can be used in the generation of cyclic codes of odd lengths.

The Half Cleaner Lemma: Constructing Efficient Interconnection Networks from Sorting Networks

In general, efficient non-blocking interconnection networks can be derived from sorting networks, and to this end, one may either follow the merge-based or the radix-based sorting paradigm. Both paradigms require special modifications to handle partial permutations. In this article, we present a general lemma about half cleaner modules that were introduced as building blocks in Batcher’s bitonic sorting network. This lemma is the key to prove the correctness of many known optimizations of interconnection networks. In particular, we first show how to use any ternary sorter and a half cleaner for implementing an efficient split module as required for radix-based sorting networks for partial permutations. Second, our lemma formally proves the correctness of another known optimization of the Batcher-Banyan network.

Rearranging links: a cost-effective approach to improve the reliability of multistage interconnection networks

The use of multiprocessor systems is the main method for providing a high computational power. Multistage interconnection networks (MINs) are widely used to connect processors and memory modules in multiprocessor systems. Therefore, the design of an efficient MIN is an essential requirement for the development of multiprocessor systems. In addition, a critical parameter for any efficient interconnection network is reliability. However, the problem in the way of designing high-reliable interconnection networks is high hardware cost. To solve this problem, contribution of this paper is to propose a new approach to improve the reliability of the MINs, called the rearranging links. The proposed approach is implemented on two common MINs namely extra-stage shuffle-exchange network (SEN+) and augmented shuffle-exchange network (ASEN). Meticulous analysis of terminal reliability proves that the proposed approach is an efficient method to improve the reliability of MINs. In addition, performed cost analysis confirms that utilising it leads to emerge cost-effective MINs.

Hamiltonian properties on a class of circulant interconnection networks

Classes of circulant graphs play an important role in modeling interconnection networks in parallel and distributed computing. They also find applications in modeling quantum spin networks supporting the perfect state transfer. It has been noticed that unitary Cayley graphs as a class of circulant graphs possess many good properties such as small diameter, mirror symmetry, recursive structure, regularity, etc. and therefore can serve as a model for efficient interconnection networks. In this paper we go a step further and analyze some other characteristics of unitary Cayley graphs important for the modeling of a good interconnection network. We show that all unitary Cayley graphs are hamiltonian. More precisely, every unitary Cayley graph is hamiltonian-laceable (up to one exception for X6) if it is bipartite, and hamiltonianconnected if it is not. We prove this by presenting an explicit construction of hamiltonian paths on Xnm using the hamiltonian paths on Xn and Xm for gcd(n,m) = 1. Moreover, we also prove that every unitary Cayley graph is bipancyclic and every nonbipartite unitary Cayley graph is pancyclic.

A new power efficient high performance interconnection network for many-core processors

Next generation high performance computing will most likely depend on the massively parallel computers. The overall performance of a massively parallel computer system is heavily affected by the interconnection network and its processing nodes. Continuing advances in VLSI technologies promise to deliver more power to individual nodes. However, the on-chip interconnection networks consume up to 50% of the total chip power and off-chip bandwidth is limited to the maximum number of possible out going physical links. In addition, the long wiring and low performance of communication network overwhelm the benefit of parallel computer system whereas it increases total cost. In this paper, we propose a new interconnection network that reduces the problems of high power consumption, long wiring length and low bandwidth issues. We have measured the static network performance and required power consumption of our proposed ‘3D-TESH’ interconnection network and compared the performance with other networks at different levels of hierarchy such as inter-chips, inter-nodes and inter-cabinets. 3D-TESH network has achieved about 52.08% better diameter and about 45.71% better average distance than the 3D-Torus network with 12.61% less router power usage at on-chip level. Furthermore, 3D-TESH requires about 41% less router power usage than 5D-Torus at the on-chip level.

Performance evaluation of hybrid optical switch architecture for data center networks

In response to the need for high bandwidth and power efficient data center interconnection networks, different interconnects have been proposed based on the optical technology used: micro-electromechanical system (MEMS), optical cross connects (OXCs), arrayed waveguide grating routers (AWGRs) and semiconductor optical amplifier (SOAs). MEMS switches are based on mature technology, have low insertion loss and cross-talk, and are data rate independent. They are also the most scalable and the cheapest class of optical switches. However, the reconfiguration time of these switches is of the order of tens of milliseconds while fast optical switches have switching time in the range of a few nanoseconds. Fast optical switches can be based on AWGRs in conjunction with tunable wavelength converters or tunable lasers or they are based on SOAs in broadcast-and-select architecture. In this paper, we propose an optical interconnect architecture for the large scale data centers. The proposed interconnect: Hybrid Optical Switch Architecture (HOSA) is a hybrid design that features slow and fast optical switches. The hybrid design leverages strengths of both types of optical switches. To reduce complexity, we employ a single stage core topology that can be easily scaled up (in capacity) and scaled out (in the number of racks) without requiring major re-cabling and network reconfiguration. We investigate the scalability of the HOSA and show that by using a single stage core topology, it can be scaled to a hundreds of thousands of servers. We also investigate a trade-off between cost and power consumption of our design by comparing it with other well-known interconnects by using analytical modelling. We demonstrate power efficiency as compared to other conventional interconnects on account of upfront CAPEX but the additional CAPEX incurred in deploying our solution instead of traditional architecture is mitigated to some extent by reduced OPEX, due to its greater energy efficiency. We evaluate the performance of the system using network-level simulation by considering diverse workload communication patterns and system design parameters. Our results show low latency and high throughput with different workload communication patterns.

Properties and Performance of Cube-Based Multiprocessor Architectures

Parallel architectures provide the possibility of solving highly computational parallel applications in a variety of ways. Numerous interconnection topologies have been designed to achieve the desired performance. Nevertheless, the actual performance is far below the expectation of users when executing parallel applications on a particular multiprocessor network. This paper presents the performance study on a special class of parallel architectures known as cube based multiprocessor architectures. It describes the issues and challenges related to the design of cube-based architectures. The issues related to the design of highly parallel system such as scalability, complexity of the system and mapping of parallel application on to it are discussed. Furthermore, the problem of routing between nodes has been analyzed along with the topological properties of cube-based architectures. Simulation results are obtained by applying task scheduling algorithm on various multiprocessor networks. The comparative study implies the various aspects while designing an efficient multiprocessor interconnection network with optimal scheduling algorithm.

Fault-tolerant edge-bipancyclicity of faulty hypercubes under the conditional-fault model

It is well-known that the n-dimensional hypercube Qn is one of the most versatile and efficient interconnection network architecture yet discovered for building massively parallel or distributed systems. Let F be the faulty set of Qn and let fv, fe be the numbers of faulty vertices and faulty edges in F, respectively. An edge e=(x,y) is said to be free if e, x, y are not in F, and a cycle is said to be fault-free if there is no faulty vertex or faulty edge on the cycle. In this paper, we prove that each free edge (x, y) in Qn for n ≥ 3 lies on a fault-free cycle of any even length from 6 to 2n−2fv if fv+fe≤2n−5,fe≤n−2 and both x and y are incident to at least two free edges. This result confirms a conjecture reported in the literature.

A Soft Error Tolerant Network-on-Chip Router Pipeline for Multi-Core Systems

Network-on-Chip (NoC) paradigm is rapidly evolving into an efficient interconnection network to handle the strict communication requirements between the increasing number of cores on a single chip. Diminishing transistor size is making the NoC increasingly vulnerable to both hard faults and soft errors. This paper concentrates on soft errors in NoCs. A soft error in an NoC router results in significant consequences such as data corruption, packet retransmission and deadlock among others. To this end, we propose S oft Error T olerant N oC R outer (STNR) architecture, that is capable of detecting and recovering from soft errors occurring in different control stages of the routing pipeline. STNR exploits the use of idle cycles inherent in NoC packet routing pipeline to perform time redundant executions necessary for soft error tolerance. In doing so, STNR is able to detect and correct all single transient faults in the control stages of the pipeline. Simulation results using PARSEC and SPLASH-2 benchmarks show that STNR is able to accomplish such high level of soft error protection with a minimal impact on latency (an increase of 1.7 and 1.6 percent respectively). Additionally, STNR incurs an area overhead of 7 percent and power overhead of 13 percent as compared to the baseline unprotected router.

A HoL-blocking aware mechanism for selecting the upward path in fat-tree topologies

Large cluster-based machines require efficient high-performance interconnection networks. Routing is a key design issue of interconnection networks. Adaptive routing usually outperforms deterministic routing at the expense of introducing out-of-order packet delivery. Many of the commodity interconnects for clusters are based on fat-trees. The adaptive routing algorithm commonly used in fat-trees is composed of a fully adaptive upward subpath, followed by a deterministic downward subpath. As the latter is determined by the former, choosing the most adequate upward path for each packet is critical in fat-trees to achieve a good performance. In this paper, we present a mechanism for selecting the upward path in fat-trees, which enables optimum use of the available network resources to achieve a high network throughput. The proposed path selection is destination based, which allows reducing the head-of-line blocking effect. Indeed, the proposed mechanism can be used either as a selection function (the provided path is used as the preferred one), or as a deterministic routing algorithm (the path is the only possible one). The results show that the resulting selection function outperforms any other known one. Moreover, the proposed deterministic routing algorithm can achieve a similar, or even higher, level of performance than adaptive routing, while providing in-order packet delivery and a simpler switch implementation.

A High-Performance and Cost-Efficient Interconnection Network for High-Density Servers

The flourishing large-scale and high-throughput web applications have emphasized the importance of high-density servers for their distinct advantages, such as high computing density, low power and low space requirements. To achieve above advantages, an efficient intra-server interconnection network is necessary. Most state-of-the-art high-density servers adopt the fully-connected intra-server network to achieve high network performance. Unfortunately, this solution is very expensive due to the high degree of nodes. To address this problem, we exploit the theory optimized moore graph to interconnect the chips within a server. Considering the size of applications, the 50-size moore graph, namely the Hoffman-Singleton graph, is extensively discussed in this paper. The simulation results show that it could attain comparative performance as the fully-connected network with much lower cost. In practice, however, chips could be integrated onto multiple boards. Thus, the graph should be divided into self-connected sub-graphs with the same size. Unfortunately, state-of-the-art solutions do not consider the production problem and generate heterogeneous sub graphs. To address this problem, we propose two equivalent-partition solutions for Hoffman-Singleton graph depending on the density of boards. Finally, we propose and evaluate a deadlock-free routing algorithm for each partition scheme.