Supercomputer Interconnect Research Articles

An Integrated Tutorial on InfiniBand, Verbs, and MPI

This tutorial presents the details of the interconnection network utilized in many high performance computing (HPC) systems today. “InfiniBand” is the hardware interconnect utilized by over 35% of the top 500 supercomputers in the world as of June, 2017. “Verbs” is the term used for both the semantic description of the interface in the InfiniBand architecture specifications, and the name used for the functions defined in the widely used OpenFabrics alliance implementation of the software interface to InfiniBand. “Message passing interface” is the primary software library by which HPC applications portably pass messages between processes across a wide range of interconnects including InfiniBand. Our goal is to explain how these three components are designed and how they interact to provide a powerful, efficient interconnect for HPC applications. We provide a succinct look into the inner technical workings of each component that should be instructive to both novices to HPC applications as well as to those who may be familiar with one component, but not necessarily the others, in the design and functioning of the total interconnect. A supercomputer interconnect is not a monolithic structure, and this tutorial aims to give non-experts a “big-picture” overview of its substructure with an appreciation of how and why features in one component influence those in others. We believe this is one of the first tutorials to discuss these three major components as one integrated whole. In addition, we give detailed examples of practical experience and typical algorithms used within each component in order to give insights into what issues and trade-offs are important.

Low-power DAC-less PAM-4 transmitter using a cascaded microring modulator.

Future super-computer interconnect systems and data centers request ultrahigh data rate links at low cost and power consumption, for which transmitters with a high level of integration and spectral efficient formats are key components. We report 60 Gb/s pulse-amplitude modulation (PAM-4) of an optical signal using a dual-microring silicon photonics circuit, making a low-power, digital-to-analog converter (DAC)-less PAM modulator. The power consumption is evaluated below 100 fJ/bit, including thermal adjustments. To the best of our knowledge, these results feature the lowest reported power consumption for PAM signaling in a DAC-less scheme for data rate beyond 40 Gb/s.

Reducing Static Energy in Supercomputer Interconnection Networks Using Topology-Aware Partitioning

The key to reducing static energy in supercomputers is switching off their unused components. Routers are the major components of a supercomputer. Whether routers can be effectively switched off or not has become the key to static energy management for supercomputers. For many typical applications, the routers in a supercomputer exhibit low utilization. However, there is no effective method to switch the routers off when they are idle. By analyzing the router occupancy in time and space, for the first time, we present a routing-policy guided topology partitioning methodology to solve this problem. We propose topology partitioning methods for three kinds of commonly used topologies (mesh, torus and fat-tree) equipped with the three most popular routing policies (deterministic routing, directionally adaptive routing and fully adaptive routing). Based on the above methods, we propose the key techniques required in this topology partitioning based static energy management in supercomputer interconnection networks to switch off unused routers in both time and space dimensions. Three topology-aware resource allocation algorithms have been developed to handle effectively different job-mixes running on a supercomputer. We validate the effectiveness of our methodology by using Tianhe-2 and a simulator for the aforementioned topologies and routing policies. The energy savings achieved on a subsystem of Tianhe-2 range from 3.8 to 79.7 percent. This translates into a yearly energy cost reduction of up to half a million US dollars for Tianhe-2.

Creating interconnect topologies by algorithmic edge removal: MOD and SMOD graphs

We introduce a method of constructing classes of graphs by algorithmic removal of entire groups of edges. Our approach to creating new classes of graphs is to focus entirely on the structure and properties of an adjacency matrix. At an initialisation step of the algorithm we start with a complete fully connected graph. In Part I we present MOD and arrested MOD graphs resulting from removal of square blocks of edges at each iteration and substitution of removed blocks with a diagonal matrix with one extra pivotal element along the main diagonal. The MOD graphs possess unique and useful properties. All important graph measures are easily expressed in analytical form and are presented in the paper. Several important properties of MOD graphs are compared very favourably with graphs representing common interconnect topologies: hypercube, 3D and 5D tori, TOFU and dragonfly. This lead us to consider MOD and arrested MOD graphs as interesting candidates for effective supercomputer interconnects. In Part II, at each iterative step we successively remove triangular shapes from the adjacency matrix. This iterative process leads to the final matrix which has two Sierpinski gaskets aligned along the main diagonal. It will be shown below that this new class of graphs is not a Sierpinski graph, since it is the adjacency matrix which has a structure of a Sierpinski gasket, and not a graph described by this matrix. We call this new class of graphs Sierpinski-Michalewicz-Or lowski-Deng SMOD graphs. The most remarkable property of the SMOD class of graphs, is that irrespective of the graph order, the diameter is constant and equals 2. The size of the graph, or the total number of edges, is about 10% of the size of a complete graph of the same order. We analyse important graph theoretic characteristics related to the topology such as diameter as a function of graph order, size, mean path length, ratio of the graph size to the size of a complete graph of the same order, and some spectral properties.

TH Express-2 reaches new heights for supercomputer interconnects

Modern supercomputers are highly dependent on parallel processing to boost their performance, and thus are all effectively parallel computers. The basic architecture of a parallel computer is composed of two parts: a single processor architecture and a communication architecture [1]. The latter is mainly implemented and supported by the interconnection network. During the design of an interconnection network, the following aspects must be considered carefully: (i) The communication requirements of the system, and the type of communication abstract that will be delivered to the programmer. (ii) The performance requirements of the network, in terms of both pointto-point communication and collective communication performances, ensure that the overall system is balanced with respect to communication speed and computation speed. (iii) The type of interconnection topology required to reduce the average distance between nodes (and thus reduce communication latency) while maintaining a moderate number of links (enabling increased communication bandwidth without making the implementation overly complex). (iv) The type of communication protocol to be used, including the routing algorithm (how to deliver messages from one node to another) and the flow control mechanism (how to handle congestion when it occurs in network traffic). (v) Implementation of the network with appropriate building blocks, including host interfaces, switches, routers, and electrical or optical cables.

Focus on silicon photonics

Si photonics has attracted much interest over the last decade as it allows integration of photonic and electronic devices on a small Si chip, providing high-capacity, energy-efficient, and cost-effective optical interconnects. Active optical cables based on Si photonics are now commercially available for rack-to-rack interconnects in supercomputers and data centers. Shorter reach optical interconnects, i.e., inter- and intra-chip optical interconnects, have been studied as the next challenge for higher speed computing. Science and technology of advanced materials is essential for rapid development in this field, and therefore it is very appropriate to publish this focus issue in STAM.This focus issue introduces state-of-the-art material and device technologies in Si photonics. The topics cover not only technologies for the optical interconnects but also optical sensing as a novel application.We are grateful to the authors who contributed to this focus issue, and hope that it will help researchers understand the frontiers of Si photonics, stimulating novel concepts in material and device technologies, as well as novel developments in and applications of Si photonics.

Interview

Prof. Dieter Bimberg of Technische Universität Berlin in Germany talks about the work behind the Letter ‘Energy-efficient and temperature-stable oxide-confined 980-nm VCSELs operating error-free at 38 Gb/s at 85 °C’ on page 103. Prof. Dieter Bimberg My group's work was originally focussing on the growth and characterisation of low-dimensional semiconductor structures – quantum wells, wires and dots. Mastering growth, I thought more and more about using such structures for devices having properties superior to classical double/multiple heterostructure based ones, being of particular importance for optical communication systems, like Ethernet or Datacom. Important properties are, for example, increased bit rate at large signal operation or high temperature performance. VCSELs are the key enabling devices for optical interconnects in work station clusters and high performance supercomputers. They must simultaneously operate with little energy dissipated, at high bit rate and at high temperature. Generally there is a trade-off between these parameters. We show, for the first time, that high bit rate at high temperatures can be achieved with record energy efficiency. An array of 25 of our VCSELs can transmit a data stream of 1 Tbit/s at an operating temperature of 75°C, typical for the inside of a work station. The internet and its computers at its active end, together with supercomputers, consume an increasing amount of our net electricity production. In 2013 the consumption amounted, in the US, to 5% of production, equalling the photovoltaic power production. For 2014 the prediction is 6% and some predict a brake-even or cross-over in 2023, if the power consumption of the photonic devices is not radically reduced. At the same time the devices must become faster, with larger cut-off frequencies, thus increasing energy consumption. We show here that careful design of the devices significantly reduces the power consumption of VCSELS and allows increased cut-off frequency and resulting maximum bit rate, still at the high temperatures inside a computer. The rapidly increasing number of optical interconnects for short reach (around 300 m and more), very short reach (less than 1 m) and ultra-short reach (less than 1 mm) leads to rapidly increasing consumption of electrical power. We need radical reduction of power consumption to enable further growth of the internet and creation of novel supercomputer generations like Exa-Flop ones, predicted for 2020. Systematic further increase of maximum bit rate of VCSELs and their drivers at the lowest possible energy consumption and still larger operating temperatures. Present theoretical device concepts must be extended to full numerical 3D ones in order to predict the optimum combination of the multitude of parameters which presently influence the performance. Shift of the emission wavelength from 850 nm, 980 nm to longer ones between 1200 and 1300 nm, also enabling p-side down mounting will further increase temperature stability. Device lift-off with redisposition on Si substrates allows us to benefit from the advantageous heat conductivity of Si. The next generation of supercomputers are true optical computers, where optics is decisive and counts for about 50% of the cost and performance. The footprint. We need lasers still much smaller than the present VCSELS in order to deposit arrays of them on memories, logics, microprocessors, where space is limited. Combination of metals with semiconductors will allow shrinking of the feature size of surface emitters by one order of magnitude or more. Plasmonic effects might help. The high speed properties of metal-cavity nanolasers are, however, yet unexplored. In the same way quantum dots as active material might stabilise the active area, reduce losses and improve temperature stability. Putting these features together we can make it happen!

Mode group division multiplexing in graded-index multimode fibers

Data transmission over multimode fibers has the potential to considerably increase the capacity limit of single-mode fibers by exploiting the spatial dimension. For long-haul systems, mode division multiplexing using one mode per channel in a few-mode fiber requires coherent detection in conjunction with digital multiple input multiple output (MIMO) processing for signal separation. In the following we focus on spatial multiplexing using mode groups rather than spatial modes in standard graded-index multimode fibers. In combination with on-off keying (OOK) modulation and direct detection, these systems might be advantageous for access- and connection-related applications, e.g., for super computer interconnects or in the near metro region at link lengths shorter than about 10 km. In this paper we study the potential of mode groups for mode group division multiplexing (MGDM). Single-mode group transmission experiments at 10 Gb/s per mode group, experimental mode group launching by phase mask, and concepts for passive optical mode demultiplexers are discussed.

Advanced Modulation Techniques for High-Performance Computing Optical Interconnects

We experimentally assess the performance of a 64 × 64 optical switch fabric used for ns-speed optical cell switching in supercomputer optical interconnects. More specifically, we study four alternative modulation formats and detection schemes, namely, 10-Gb/s nonreturn-to-zero differential phase-shift keying with balanced direct detection, 10-Gb/s polarization division multiplexed (PDM) quadrature phase-shift keying, 40-Gb/s single-polarization 16-ary quadrature amplitude modulation (16QAM), and 80-Gb/s PDM-16QAM, with coherent intradyne detection, in conjunction with an optimized version of the optical shared memory supercomputer interconnect system switch fabric. In particular, we investigate the resilience of the aforementioned advanced modulation formats to the nonlinearities of semiconductor optical amplifiers, used as ON/OFF gates in the supercomputer optical switch fabric under study. In addition, we compare their performance using as a benchmark the performance of conventional 10-Gb/s intensity modulation direct detection (IM/DD). We show that the choice of the appropriate advanced modulation format can increase the capacity of the switch fabric, while, at the same time, it can mitigate the main nonlinear effect, i.e., cross-gain modulation that arises when using conventional IM/DD. Nonlinear phase distortion becomes the main limiting factor when advanced modulation formats are used.

Performance comparison of 850-nm and 1550-nm VCSELs exploiting OOK, OFDM, and 4-PAM over SMF/MMF links for low-cost optical interconnects

We experimentally compare the performance of two commercially available vertical-cavity surface-emitting laser diodes (VCSELs), a multi-mode 850-nm and a single-mode 1550-nm, exploiting on–off keying/direct detection (OOK/DD), and orthogonal frequency division multiplexed (OFDM) quadrature phase-shift keying (QPSK)/16-ary quadrature amplitude modulation (16QAM) with direct detection, over SMF (100m and 5km) and MMF (100m and 1km) short-range links, for their potential application in low-cost rack-to-rack optical interconnects. Moreover, we assess the performance of quaternary-pulse amplitude modulation (4-PAM), for the 1550-nm transmitter over SMF and MMF links and we compare it to the data-rate equivalent NRZ-OOK. The extensive performance comparison under various transmission scenarios shows the superiority of 1550-nm single-mode VCSEL compared to its multi-mode 850-nm counterpart. Moreover, OFDM/DD and 4-PAM in conjunction with low-cost, inexpensive VCSELs as transmitters prove to be an enabling technology for next-generation WDM, point-to-point, short-reach, SMF/MMF optical interconnects and potential candidates to substitute NRZ-OOK. Nevertheless, the sensitivity requirements are higher in that case, whereas these advanced, spectrally-efficient modulation formats become severely degraded when transmitted over MMF links, especially, when employing the inexpensive 850-nm VCSELs as transmitter. Finally, we compare the performance of the point-to-point links under investigation to the performance of a semiconductor optical amplifier (SOA)- based, scalable permutation switch fabric, the Optical Shared MemOry Supercomputer Interconnect System (OSMOSIS).

SC-FDE for MMF short reach optical interconnects using directly modulated 850 nm VCSELs

We propose the use of single-carrier frequency-domain equalization (SC-FDE) for the compensation of modal dispersion in short distance optical links using multimode fibers and 850 nm VCSELs. By post-processing of experimental data, we demonstrate, at 7.9% overhead, the error-free transmission (over a 4 Mbit sequence) of OOK-modulated 5 Gbps over 2443 meters of OM3 fiber (with a nominal 3300 MHz×km bandwidth). The proposed solution may be applied as a low cost alternative for data center and supercomputer interconnects.

Optimal wavelength-space crossbar switches for supercomputer optical interconnects

We propose a most economical design of the Optical Shared MemOry Supercomputer Interconnect System (OSMOSIS) all-optical, wavelength-space crossbar switch fabric. It is shown, by analysis and simulation, that the total number of on-off gates required for the proposed N × N switch fabric can scale asymptotically as N ln N if the number of input/output ports N can be factored into a product of small primes. This is of the same order of magnitude as Shannon's lower bound for switch complexity, according to which the minimum number of two-state switches required for the construction of a N × N permutation switch is log2 (N!).

Influence of Transmission Impairments on the OSMOSIS HPC Optical Interconnect Architecture

We examine the impact of transmission impairments on the performance of the optical supercomputer interconnect architecture, initially proposed in the context of the optical shared memory supercomputer interconnect system (OSMOSIS) project. We study two versions of the aforementioned optical interconnect that differ in terms of the number of semiconductor optical amplifiers (SOAs) used as ON-OFF gates. For practical reasons related to packet arbitration, the size of the crossbar switch of the optical interconnect in this study is limited to 64 ports. The switch is based on a broadcast-and-select architecture and employs DWDM in conjunction with 10 Gb/s intensity modulation/direct detection per wavelength channel. We show, both by experiment and by simulation, that the minimization of the number of SOAs in the optical switch by taking advantage of the cyclic routing capability of optical arrayed waveguide multiplexers/demultiplexers leads to negligible performance deterioration compared to conventional wavelength-space switches that are prohibitive slower and do not use any inherent gain properties like in OSMOSIS.

Designing a Crossbar Scheduler for HPC Applications

A crucial part of any high-performance computing (HPC) system is its interconnection network. Corning and IBM are jointly developing a demonstration interconnect based on optical cell switching with electronic control. The Corning-IBM joint optical shared memory supercomputer interconnect system (Osmosis) project explores the opportunity to advance the role of optical-switching technologies in such systems. Key innovations in the scheduler architecture directly address the main HPC requirements: low latency, high throughput, efficient multicast support, and high reliability

The Supercomputer Supernet testbed: a WDM-based supercomputer interconnect

Current fiber optic networks effectively provide local connectivity among end user computing devices, and can serve as backbone fabric between LAN subnets across campus and metropolitan areas. However, combining both stream service (in which ATM excels) and low latency datagram service (in which cluster networks like Myrinet and POLO excel) has been difficult to realize. This paper describes a new wavelength division multiplexed (WDM) fiber optic network that supports both stream and datagram service and extends reach and functionality of low-latency, high bandwidth workstation clusters to a campus and MAN setting. The novel concept is based on combining the rich interconnect structure of WDM fiber optics with the low-latency mesh of crossbar switches recently developed for workstation groups. This system, called the Supercomputer Supernet (SSN) achieves a high level of performance by replacing the point-to-point copper wire links with a parallel channel (WDM) fiber optic interconnect system. The novel scheme interconnects asynchronous wormhole routing switches used in parallel supercomputers via multi-channel WDM fiber optic links embedded in to an optical star (or tree) physical topology, WDM will be used to subdivide the very large fiber bandwidth into several channels, each of Gb/s bandwidth, WDM channels (supporting also time division multiplexing) will be established between modules, thus defining a dense virtual interconnection topology, which is dynamically reconfigurable and responds to changing traffic patterns. A pool of channels will be set aside for direct, end-to-end connections between crossbars, providing circuit-switched service for real-time traffic applications.

High-speed ATM/SONET infrastructure research in ATDNet

The Advanced Technology Demonstration Network (ATDNet) was established by DARPA to enable increased collaboration among the Department of Defense and other federal agencies. Its primary goal is to serve as an experimental testbed for high-speed network research and defense initiatives with emphasis on ATM an SONET technologies. This article presents an overview of the network infrastructure research that has been performed on ATDNet. Specific areas of focus include network management, signalling, traffic management, supercomputer interconnect, and network testing. The authors present their research activities and results in each of these areas.

The Supercomputer Supernet: A Scalable Distributed Terabit Network

Conventional supercomputer interconnection networks consist of crossbar modules, which are connected by point-to-point copper or fiber links to create distributed mesh topologies (e.g., CP*, Nectar). This type of “physical networking” topology creates cable layout problems, dealing with bundles of cables/fibers between various pairs of modules. It also introduces several routing hops, increasing the probability of interference between connections and making it difficult to guarantee quality of service to real time applications. We describe a new network called the Supercomputer Supernet (SSN) that attempts to overcome these problems by replacing the point-to-point links with a fiber optic interconnect system. The novel scheme employs asynchronous pipeline crossbar switches (APCS) used in parallel supercomputers to interconnect multi-channel wavelength division multiplexed (WDM) fiber optic links to an optical star (or tree) “physical” topology. WDM will be used to subdivide the very large fiber bandwidth into several channels each of Gb/s bandwidth. WDM channels (supporting also time division multiplexing) will be established between modules, thus defining a dense “virtual” interconnection topology, which is dynamically reconfigurable, responding to changing traffic patterns. A pool of channels will be set aside for direct, end-to-end connections between crossbars, providing circuit-switched service for real-time traffic applications.

The photonic ring: Algorithms for optimized node arrangements

Abstract New applications such as the future university and research environment, medical imaging, and supercomputer interconnects indicate that the future end-user will need to operate at close to peak electronic processing speed, viz. a few Gigabits per second. By exploiting the unique properties of emerging lightwave technology, it is now possible to embed different logical structures over a given physical network topology. Regular multihop structures, for example, bus, ring, toroid, are attractive candidates for logical topologies since they use simple routing schemes, that require less processing time.