High-radix Switches Research Articles

Toward lower-diameter large-scale HPC and data center networks with co-packaged optics

We investigate the advantages of using co-packaged optics for building low-diameter, large-scale high-performance computing (HPC) and data center networks. The increased escape bandwidth offered by co-packaged optics can enable high-radix switch implementations of more than 150 switch ports, which can be combined with data rates of up to 400 Gb/s per port. From the network architecture perspective, the key benefits of using co-packaged optics in future fat-tree networks include (a) the ability to implement large-scale topologies of > --> 11 , 000 end points by eliminating the need for a third switching layer and (b) the ability to provide up to 4 × higher bisection bandwidth compared to existing solutions, reducing at the same time the number of required switch application-specific integrated circuits by > --> 80 % . From the network operation perspective, both reduced energy consumption and lower packet delays can be achieved since fewer hops are required; i.e., packets need to traverse fewer serializer/deserializer lanes and fewer switch buffers, which reduces the probability of contending with other packets and improves the tolerance of network congestion. The performance of the proposed architecture is evaluated via discrete-event simulations for a wide range of representative HPC synthetic-traffic cases that include both hotspot and non-hotspot scenarios. The simulation results suggest that co-packaged optics form a promising solution to keep up with bandwidth scaling in future networks, while the reduced number of switching layers can lead to significant mean packet delay improvements that start from 30% and reach up to 74% for high-load conditions.

Energy efficient torus networks with on/off links

Future exascale computing systems will require energy and performance efficient interconnection networks to respond to the high data movement demands of new applications, such as those coming from big-data and artificial intelligence areas. The network structure plays a major role in the overall interconnect performance, for this reason torus is a common topology used in the current largest supercomputers. There are several proposals to improve energy efficiency of interconnection networks. However, few works combine both energy and performance, and sometimes they are treated as opposed issues. In this paper, we try to determine which torus network configuration offers the best performance/energy ratio when high-radix switches are used to build the interconnect system. The performance/energy evaluation has been performed by trace-driven simulation under realistic scenarios, where several mixes of scientific applications share a supercomputer system and are scheduled to be executed with the available resources at each moment.

Optimize the Power Consumption and SNR of the 3D Photonic High-Radix Switch Architecture Based on Extra Channels and Redundant Rings

The demand from exascale computing has made the design of high-radix switch chips an attractive and challenging research field in EHPC (exascale high-performance computing). The static power, due to the thermal sensitivity and process variation of the microresonator rings, and the cross talk noise of the optical network become the main bottlenecks of the network’s scalability. This paper proposes the analyze model of the trimming power, process variation power, and signal-to-noise ratio (SNR) for the Graphein-based high-radix optical switch networks and uses the extra channels and the redundant rings to decrease the trimming power and the process variation power. The paper also explores the SNR under different configurations. The simulation result shows that when using 8 extra channels in the 64×64 crossbar optical network, the trimming power reduces almost 80% and the process variation power decreases 65% by adding 16 redundant rings in the 64×64 crossbar optical network. All of these schemes have little influence on the SNR. Meanwhile, the greater channel spacing has great advantages to decrease the static power and increase the SNR of the optical network.

Straightforward solutions to reduce HoL blocking in different Dragonfly fully-connected interconnection patterns

The performance of interconnection networks is a challenging issue for High-Performance Computing (HPC) systems, which becomes even more important when the number of interconnected endnodes grows. In that sense, Dragonfly interconnection patterns are a very popular option to configure the network topology, especially for large systems, as they are able to achieve a high scalability relying on high-radix switches. This kind of hierarchical topologies has two levels of interconnection (i.e., connections within the element of a group and connections among groups) and each one can be interconnected using different patterns. However, regardless of the Dragonfly interconnection pattern, the Head-of-Line (HoL) blocking effect derived from congestion situations may jeopardize the Dragonfly performance. This paper analyzes the dynamics of congestion in different Dragonfly fully-connected interconnection patterns. Also, we describe a queuing scheme called Hierarchical Two-Level Queuing (H2LQ), designed specially to reduce HoL blocking in any fully-connected Dragonfly network that uses minimal-path routing. Finally, we present experiment results which show that this scheme significantly boost Dragonfly performance, regardless the interconnection pattern, especially when congestion arises, while requiring fewer network resources than other techniques oriented to deal with the effects of congestion.

Blocking probability in optical interconnects in data center networks

Cloud computing and Web-based applications are creating a need for powerful data centers. Data centers have a great need for high bandwidth, low latency, low blocking probability, and low bit-error rate to sustain the interaction between different applications. Current data center networks (DCNs) suffer from several problems such as high-energy consumption, high latency, fixed throughput of links, and limited reconfigurability. Electronic switches are low radix and have high latency due to a large hop count since each hop employs a store-and-forward mechanism. Optical interconnects, on the other hand, offer several advantages such as low-energy consumption, high bandwidth, reconfigurability, malleability to changing traffic, high-radix switch design, fast switching transition times, and wavelength multiplexing. These benefits provide the incentive to shift from electrical interconnects to optical interconnects in DCNs. Despite several advantages over their electrical counterparts, the performance of optical interconnects can be further improved by considering some performance parameters of optical interconnects. One such important parameter for the performance of any communication network is the blocking probability. This paper makes a comprehensive investigation of the performance of optical interconnects in different DCN architectures on the basis of blocking probability and concludes by suggesting ways to reduce the blocking.

Optimizing the configuration of combined high-radix switches

Combined high-radix switches are an attractive option for building high-radix switches. The idea basically consists in combining several current smaller single-chip switches to obtain switches that have a greater number of ports. The performance of these kinds of switches varies depending on their internal configuration because the subnetwork interconnecting all the internal switches could become a bottleneck if an inappropriate internal configuration is established. In this paper, we show how to obtain the optimal internal switch configuration by applying a specific methodology. We highlight the impact of internal switch configuration on the network performance by means of case studies.

Challenges for Optically Enabled High-Radix Switches for Data Center Networks

We discuss challenges for optically enabled high-radix switches in data centers. We highlight the clear advantages of using high-radix switches and explain why coupling such switches with midboard optics can be a winning solution. We further analyze the trends in pluggable transceivers and data center switches and indicate that if current scaling trends are set to continue, very different hardware, software, and architectures should be adopted to support future data centers networks.

Formalization and configuration methodology for high-radix combined switches

In large switch-based interconnection networks, increasing the switch radix results in a decrease in the total number of network components, and consequently the overall cost of the network can be significantly reduced. Moreover, high-radix switches are an attractive option to improve the network performance in terms of latency since hop count is also reduced. However, there are some difficulties related to integration scale to design such switches. In this paper we present and formalize an interesting alternative for building high-radix switches going beyond the integration scale bounds. The idea basically consists in combining several current smaller switches to obtain switches having greater number of ports. This strategy will remain valid as the scale of integration keeps evolving. Although simple, this strategy raises key design challenges in order to these high-radix switches achieve the best performance. The resultant internal structure of these switches becomes an important design decision, and an arbitrary selection may produce a significant performance degradation. For this reason, we also propose a general methodology to configure in an optimal way the internal switch structure and apply it to a particular case in order to show how it works. The resultant switch configurations are evaluated in order to show the real potential of our proposal.

Obtaining the optimal configuration of high-radix Combined switches

High-radix switches reduce network cost and improve network performance, especially in large switch-based interconnection networks. However, there are some problems related to the integration scale to implement such switches in a single chip. An interesting alternative for building high-radix switches consists of combining several current smaller single-chip switches to obtain switches with a greater number of ports. A key design issue of this kind of high-radix switches is the internal switch configuration, specifically, the correspondence between the ports of these high-radix switches and the ports of their smaller internal single-chip switches. In this paper we use artificial intelligence and data mining techniques in order to obtain the optimal internal configuration of all the switches in the network of large supercomputers running parallel applications. Simulation results show that using the resultant switch configurations, it is possible to achieve similar performance as with single-chip switches with the same radix, which would be unfeasible with the current integration scale.

On the topological properties of HyperX

In modern world, all sciences especially engineering have insatiable demand for more power of processing. Although the use of modern micro-architectures has increased the performance of processors, this increment is only part of speeding up in responding such these demands. In fact, the need of some applications to parallel systems in large scales makes these systems more popular. Therefore, these systems are only the possible way of performing enormous computing power for applications with high performance computing. This paper comprehensively studies the topological properties of a class of n-D networks that are called HyperX from different aspects. In this paper we are going to provide a detailed description of HyperX topology in an algebraic framework with basic features (such as regularity, symmetry, etc.). The important parameters in this topology are evaluated parametrically and compared with other topologies. Having expressed this fact, we emphasize that our study is among the very few attempts reported in the literature to analyze the important parameters that can capture the performance behavior of HyperX topology. Since HyperX has many advantages of high radix switch components, it becomes a serious competitor against the other topologies and high radix networks. Hence, this study leads to finding an optimum topology for these kinds of networks.

A case for random shortcut topologies for HPC interconnects

As the scales of parallel applications and platforms increase the negative impact of communication latencies on performance becomes large. Fortunately, modern High Performance Computing (HPC) systems can exploit low-latency topologies of high-radix switches. In this context, we propose the use of random shortcut topologies, which are generated by augmenting classical topologies with random links. Using graph analysis we find that these topologies, when compared to non-random topologies of the same degree, lead to drastically reduced diameter and average shortest path length. The best results are obtained when adding random links to a ring topology, meaning that good random shortcut topologies can easily be generated for arbitrary numbers of switches. Using flit-level discrete event simulation we find that random shortcut topologies achieve throughput comparable to and latency lower than that of existing non-random topologies such as hypercubes and tori. Finally, we discuss and quantify practical challenges for random shortcut topologies, including routing scalability and larger physical cable lengths.

Optical high radix switch design

Networking consumes up to 33 percent of modern data center power. Network switches are the key source of inefficiency: a switch traversal costs an order of magnitude more than a link traversal. The authors propose a new high-radix switch architecture that uses emerging integrated optical interconnect technology to reduce switch power. They tailor every component of a switch to best exploit optical technology and improve switch scalability and energy efficiency.

The role of optics in future high radix switch design

For large-scale networks, high-radix switches reduce hop and switch count, which decreases latency and power. The ITRS projections for signal-pin count and per-pin bandwidth are nearly flat over the next decade, so increased radix in electronic switches will come at the cost of less per-port bandwidth. Silicon nanophotonic technology provides a long-term solution to this problem. We first compare the use of photonic I/O against an all-electrical, Cray YARC inspired baseline. We compare the power and performance of switches of radix 64, 100, and 144 in the 45, 32, and 22 nm technology steps. In addition with the greater off-chip bandwidth enabled by photonics, the high power of electrical components inside the switch becomes a problem beyond radix 64. We propose an optical switch architecture that exploits highspeed optical interconnects to build a flat crossbar with multiplewriter, single-reader links. Unlike YARC, which uses small buffers at various stages, the proposed design buffers only at input and output ports. This simplifies the design and enables large buffers, capable of handling ethernet-size packets. To mitigate head-of-line blocking and maximize switch throughput, we use an arbitration scheme that allows each port to make eight requests and use two grants. The bandwidth of the optical crossbar is also doubled to to provide a 2x internal speedup. Since optical interconnects have high static power, we show that it is critical to balance the use of optical and electrical components to get the best energy efficiency. Overall, the adoption of photonic I/O allows 100,000 port networks to be constructed with less than one third the power of equivalent all-electronic networks. A further 50% reduction in power can be achieved by using photonics within the switch components. Our best optical design performs similarly to YARC for small packets while consuming less than half the power, and handles 80% more load for large message traffic.

Routing of asynchronous Clos networks

Clos networks provide theoretically optimal solution to build high-radix switches. Dynamically reconfiguring a three-stage Clos network is more difficult in asynchronous circuits than in synchronous circuits. This study proposes a novel asynchronous dispatching (AD) algorithm for general three-stage Clos networks. It is compared with the classic synchronous concurrent round-robin dispatching (CRRD) algorithm in unbuffered Clos networks. The AD algorithm avoids the contention in central modules using a state feedback scheme and outperforms the throughput of CRRD in behavioural simulations. Two asynchronous Clos networks using the AD algorithm are implemented and compared with a synchronous Clos network using the CRRD algorithm. The asynchronous Clos scheduler is smaller than its synchronous counterpart. Synchronous Clos networks achieve higher throughput than asynchronous Clos networks because asynchronous Clos networks cannot hide the arbitration latency and their data paths are slow. The asynchronous Clos scheduler consumes significantly lower power than the synchronous scheduler and the asynchronous Clos network using bundled-data data switches shows the best power efficiency in all implementations.

Design and performance of speculative flow control for high-radix datacenter interconnect switches

High-radix switches are desirable building blocks for large computer interconnection networks, because they are more suitable to convert chip I/O bandwidth into low latency and low cost than low-radix switches [J. Kim, W.J. Dally, B. Towles, A.K. Gupta, Microarchitecture of a high-radix router, in: Proc. ISCA 2005, Madison, WI, 2005]. Unfortunately, most existing switch architectures do not scale well to a large number of ports, for example, the complexity of the buffered crossbar architecture scales quadratically with the number of ports. Compounded with support for long round-trip times and many virtual channels, the overall buffer requirements limit the feasibility of such switches to modest port counts. Compromising on the buffer sizing leads to a drastic increase in latency and reduction in throughput, as long as traditional credit flow control is employed at the link level. We propose a novel link-level flow control protocol that enables high-performance scalable switches that are based on the increasingly popular buffered crossbar architecture, to scale to higher port counts without sacrificing performance. By combining credited and speculative transmission, this scheme achieves reliable delivery, low latency, and high throughput, even with crosspoint buffers that are significantly smaller than the round-trip time. The proposed scheme substantially reduces message latency and improves throughput of partially buffered crossbar switches loaded with synthetic uniform and non-uniform bursty traffic. Moreover, simulations replaying traces of several typical MPI applications demonstrate communication speedup factors of 2 to 10 times.

Beyond Fat--tree: Unidirectional Load--Balanced Multistage Interconnection Network

The fat-tree is one of the most widely-used topologies by interconnection network manufacturers. Recently, it has been demonstrated that a deterministic routing algorithm that optimally balances the network traffic can not only achieve almost the same performance than an adaptive routing algorithm but also outperforms it. On the other hand, fat-trees require a high number of switches with a non-negligible wiring complexity. In this paper, we propose replacing the fat--tree by a unidirectional multistage interconnection network (UMIN) that uses a traffic balancing deterministic routing algorithm. As a consequence, switch hardware is almost reduced to the half, decreasing, in this way, the power consumption, the arbitration complexity, the switch size itself, and the network cost. Preliminary evaluation results show that the UMIN with the load balancing scheme obtains lower latency than fat--tree for low and medium traffic loads. Furthermore, in networks with a high number of stages or with high radix switches, it obtains the same, or even higher, throughput than fat-tree.