Future Data Centers Research Articles

Special Issue: Euro‐Par 2012

This special issue of Concurrency and Computation: Practice and Experience contains revised and extended versions of selected papers presented at the conference Euro-Par 2012.

A Multiport Microsecond Optical Circuit Switch for Data Center Networking

We experimentally evaluate the network-level switching time of a functional 23-host prototype hybrid optical circuit-switched/electrical packet-switched network for datacenters called Mordia (Microsecond Optical Research Datacenter Interconnect Architecture). This hybrid network uses a standard electrical packet switch and an optical circuit-switched architecture based on a wavelength-selective switch that has a measured mean port-to-port network reconfiguration time of 11.5 <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$\mu{\rm s}$</tex> </formula> including the signal acquisition by the network interface card. Using multiple parallel rings, we show that this architecture can scale to support the large bisection bandwidth required for future datacenters.

An Energy-Efficient Optically Connected Memory Module for Hybrid Packet- and Circuit-Switched Optical Networks

Demands on future data centers and high-performance computing systems will require processor-memory interconnects with greater performance and flexibility than can be provided by existing electronic interconnects. The bandwidth density and bit-rate transparency offered by optical systems are uniquely suited to address these challenges facing memory interconnects. We, thus, investigate a hybrid packet- and circuit-switched optical interconnection network linking microprocessors with their associated main memory, which can simultaneously reduce memory access latency and improve energy efficiency performance. This novel hybrid approach allows low-bandwidth memory control data and small memory transactions to be efficiently transmitted as wavelength-striped optical packets, while long bursts of memory accesses are optically circuit switched. In this study, we experimentally demonstrate an optically connected memory system in which a microprocessor accesses multiple 80-Gb/s memory modules all-optically across a hybrid packet- and circuit- switched optical network. Error-free communication between the microprocessor and main memory is confirmed (bit-error rates less than 10 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-12</sup> ) with the optical network providing low-memory access latencies. The overall memory system reduces energy consumption by 28%.

A layer 2 multipath solution and its performance evaluation for Data Center Ethernets

SUMMARYData Center Ethernet is likely to be deployed as the communication infrastructure for future data centers, which carries multiple types of traffic with very different characteristics and handling requirements. Conventional Spanning Tree Protocol (STP) cannot meet the requirement of a Data Center Ethernet framework because of its poor bandwidth utilization and lack of multipathing capability. In this paper, we propose a layer 2 multipathing solution, namely optimized dynamic load‐balancing multipathing (ODLBMP), to be deployed in Data Center Ethernet. Our proposed method utilizes all available links and ports for frame delivery and can split traffic of a communication pair along multiple paths. In ODLBMP, the traffic loads of all paths are continuously monitored so that traffic assigned to each path can be dynamically adjusted to avoid path/link over‐utilization. Per‐flow forwarding is observed in ODLBMP to guarantee the in‐order delivery, which is important for most storage traffic. In addition, ODLBMP finely differentiates flows from application perspective so it has more flexibility in traffic splitting and route selection, and achieves better multipath load balancing. Computer simulations show that our proposed algorithm performs better than other compared algorithms, including STP, Transparent Interconnection of Lots of Links, and DLBMP, in all simulation scenarios in terms of frame delivery ratio and network throughput. Copyright © 2013 John Wiley &amp; Sons, Ltd.

Next-Generation Optically-Interconnected High-Performance Data Centers

The communication requirements imposed by the unabated growth in both the size and computational density of modern data centers will soon outpace the fundamental capabilities of conventional electronic interconnection networks. Optical interconnects represent a potentially disruptive technology that can simultaneously satisfy the throughput, latency, and energy demands of these next-generation systems. In this paper, we propose an end-to-end photonic networking platform for future optically-interconnected data center networks. A reconfigurable hybrid photonic network building block and a protocol-agnostic optical network interface comprise the key elements of our proposed system. A modular optical switch fabric prototyping platform is designed and implemented to support a wide variety of potential photonic technologies. We successfully demonstrate nanosecond-scale optical packet-switching at 1-Tb/s per port (40-Gb/s ×25λs) through our prototype, configured as a 4×4 switch, and confirm error-free transmission across all wavelengths. In order to flexibly allocate appropriate bandwidth for heterogeneous applications, we further implement and evaluate a novel wavelength-reconfigurable optical packet- and circuit-switch by utilizing our optical switch platform. Finally, we detail our implementation of an optical network interface card (O-NIC) designed to bridge the gap between the well-established protocol layers utilized in current networks and the physical layer of the proposed photonic networks and demonstrate its functionality by the streaming of high-definition video through the end-to-end optical network.

A single-stage optical load-balanced switch for data centers

Load balancing is an attractive technique to achieve maximum throughput and optimal resource utilization in large-scale switching systems. However current electronic load-balanced switches suffer from severe problems in implementation cost, power consumption and scaling. To overcome these problems, in this paper we propose a single-stage optical load-balanced switch architecture based on an arrayed waveguide grating router (AWGR) in conjunction with fast tunable lasers. By reuse of the fast tunable lasers, the switch achieves both functions of load balancing and switching through the AWGR. With this architecture, proof-of-concept experiments have been conducted to investigate the feasibility of the optical load-balanced switch and to examine its physical performance. Compared to three-stage load-balanced switches, the reported switch needs only half of optical devices such as tunable lasers and AWGRs, which can provide a cost-effective solution for future data centers.

Mode-filtered semiconductor lasers enable longer-reach optical interconnects

An integrated mode filter significantly decreases the spectral width of conventional short-wavelength vertical-cavity surface-emitting lasers, promising longer optical interconnects for future data centers.

Power consumption evaluation of all-optical data center networks

Cloud computing and web emerging applications have created the need for more powerful data centers. These data centers need high bandwidth interconnects that can sustain the high interaction between the web-, application- and database-servers. Data center networks based on electronic packet switches will have to consume excessive power in order to satisfy the required communication bandwidth of future data centers. Optical interconnects have gained attention recently as a promising energy efficient solution offering high throughput, low latency and reduced energy consumption compared to current networks based on commodity switches. This paper presents a comparison on the power consumption of several optical interconnection schemes based on AWGRs, Wavelength Selective Switches (WSS) or Semiconductor Optical Amplifiers (SOAs). Based on a thorough analysis of each architecture, it is shown that optical interconnects can achieve at least an order of magnitude higher energy efficiency compared to current data center networks based on electrical packet based switches and they could contribute to greener IT network infrastructures.

DPillar: Dual-port server interconnection network for large scale data centers

To meet the huge demands of computation power and storage space, a future data center may have to include up to millions of servers. The conventional hierarchical tree-based data center network architecture faces several challenges in scaling a data center to that size. Previous research effort has shown that a server-centric architecture, where servers are not only computation and storage workstations but also intermediate nodes relaying traffic for other servers, performs well in scaling a data center to a huge number of servers. This paper presents a server-centric data center network called DPillar, whose topology is inspired by the classic butterfly network. DPillar provides several nice properties and achieves the balance between topological scalability, network performance, and cost efficiency, which make it suitable for building large scale future data centers. Using only commodity hardware, a DPillar network can easily accommodate millions of servers. The structure of a DPillar network is symmetric so that any network bottleneck is eliminated at the architectural level. With each server having only two ports, DPillar is able to provide the bandwidth to support communication intensive distributed applications. This paper studies the interconnection features of DPillar, how to compute routes in DPillar, and how to forward packets in DPillar. Extensive simulation experiments have been performed to evaluate the performance of DPillar. The results show that DPillar performs well even in the presence of a large number of server and switch failures.

A Survey on Optical Interconnects for Data Centers

Data centers are experiencing an exponential increase in the amount of network traffic that they have to sustain due to cloud computing and several emerging web applications. To face this network load, large data centers are required with thousands of servers interconnected with high bandwidth switches. Current data center networks, based on electronic packet switches, consume excessive power to handle the increased communication bandwidth of emerging applications. Optical interconnects have gained attention recently as a promising solution offering high throughput, low latency and reduced energy consumption compared to current networks based on commodity switches. This paper presents a thorough survey on optical interconnects for next generation data center networks. Furthermore, the paper provides a qualitative categorization and comparison of the proposed schemes based on their main features such as connectivity and scalability. Finally, the paper discusses the cost and the power consumption of these schemes that are of primary importance in the future data center networks.

A hybrid optical packet and wavelength selective switching platform for high-performance data center networks

We propose and experimentally demonstrate for the first time a hybrid optical packet and wavelength selective switching platform for high-performance data center networks. This architecture based on cascaded silicon microrings and semiconductor optical amplifiers (SOAs) supports wavelength reconfigurable packet and circuit switching, and is highly scalable, energy efficient and potentially integratable. By combining the wavelength-selective behavior of the microring and the broadband behavior of the SOA switch, we are able to achieve fast switching transitions, high extinction ratios, and low driving voltages, which are all requirements for future optical high-performance data center networks. Routing correctness and error-free operation (<10(-12)) are verified for both 10-Gb/s and 40-Gb/s packets and streaming data with format transparency.

Building Data Centers With Optically Connected Memory

Future data centers will require novel, scalable memory architectures capable of sustaining high bandwidths while still achieving low memory access latencies. Electronic interconnects cannot meet the challenges presented by the need for multi-terabit off-chip memory data paths. In this work, the electronic bus between main memory and its host processor is replaced with a circuit-switched optical interconnection network. We investigate the impact of our optically connected memory system on large-scale architectures and experimentally validate the protocol using field-programmable gate array based processor nodes and a custom-designed memory controller. The processor communicates all-optically with multiple synchronous dynamic random access memory nodes using 4 × 2.5-Gb/s wavelength-striped payloads, operating error free with bit-error rates less than 10-12.

Cooling solutions in an operational data centre: A case study

The rapid growth in data centres – large computing infrastructures containing vast quantities of data processing and storage equipment – has resulted in their consumption of up to 100 times more energy per square metre than office accommodation. The decrease in processing server sizes and the more efficient use of space and server processing are challenging data centre facilities to provide more power and cooling, significantly increasing energy demands. Energy consumption of data centres can be severely and unnecessarily high due to inadequate localised cooling and densely packed server rack layouts. However, as heat dissipation in data centres rises by orders of magnitude, inefficiencies such as air recirculation causing hot spots and flow short-circuiting will have a significant impact on the thermal manageability and energy efficiency of the cooling infrastructure. Therefore, an efficient thermal management of high-powered electronic equipment is a significant challenge for cooling of data centres. To highlight the importance of some of these issues, in this project, an operational data centre has been studied. Field measurements of temperature have been performed. Numerical analysis of flow and temperature fields is conducted in order to evaluate the thermal behaviour of the data centre. A number of undesirable hot spots have been identified. To rectify the problem, a few practical design and remedial solutions to improve the cooling effectiveness have been proposed and examined to allow a reduced air-conditioning power requirement. The findings lead to a better understanding of the cooling issues and the respective proposed solutions allow an improved design for future data centres.

Energy accounting for shared virtualized environments under DVFS using PMC-based power models

Virtualized infrastructure providers demand new methods to increase the accuracy of the accounting models used to charge their customers. Future data centers will be composed of many-core systems that will host a large number of virtual machines (VMs) each. While resource utilization accounting can be achieved with existing system tools, energy accounting is a complex task when per-VM granularity is the goal. In this paper, we propose a methodology that brings new opportunities to energy accounting by adding an unprecedented degree of accuracy on the per-VM measurements. We present a system — which leverages CPU and memory power models based in performance monitoring counters (PMCs) — to perform energy accounting in virtualized systems. The contribution of this paper is threefold. First, we show that PMC-based power modeling methods are still valid on virtualized environments. Second, we show that the Dynamic Voltage and Frequency Scaling (DVFS) mechanism, which commonly is used by infrastructure providers to avoid power and thermal emergencies, does not affect the accuracy of the models. And third, we introduce a novel methodology for accounting of energy consumption in virtualized systems. Accounting is done on a per-VM basis, even in the case where multiple VMs are deployed on top of the same physical hardware, bypassing the limitations of per-server aggregated power metering. Overall, the results for an Intel ® Core™ 2 Duo show errors in energy estimations <5%. Such an approach brings flexibility to the chargeback models used by service and infrastructure providers. For instance, we are able to detect cases where VMs executed during the same amount of time, present more than 20% differences in energy consumption even only taking into account the consumption of the CPU and the memory.

Data center design and location: Consequences for electricity use and greenhouse-gas emissions

The rapidly increasing electricity demand for data center operation has motivated efforts to better understand current data center energy use and to identify strategies that reduce the environmental impact of these buildings. This paper builds on previous data center energy modeling efforts by characterizing local climate and mechanical equipment differences among data centers and then evaluating their consequences for building energy use. Cities in the United States with significant data center activity are identified. Representative climate conditions for these cities are applied to data center energy models for several different prototypical space types. Results indicate that widespread, effective economizer use in data centers could reduce energy demand for data centers by about 20–25%, equivalent to an energy efficiency resource in the US of ∼13–17 billion kWh per year. Almost half of the potential savings would result from better airflow management and proper control sequences. The total energy savings potential of economizers, although substantial, is constrained by their limited potential for use in server closets and server rooms, which together are estimated to account for about 30% of all data center energy demand. Incorporating economizer use into the mechanical systems of larger data centers would increase the variation in energy efficiency among geographic regions, indicating that as data center buildings become more energy efficient, their locations will have an increasing effect on overall energy demand. Differences among regions become even more important when accounting for greenhouse-gas emissions. Future data center development could consider site location, along with efficiency measures, to limit the environmental impact attributable to this increasingly prominent economic sector.

Symbiotic routing in future data centers

Building distributed applications that run in data centers is hard. The CamCube project explores the design of a shipping container sized data center with the goal of building an easier platform on which to build these applications. CamCube replaces the traditional switch-based network with a 3D torus topology, with each server directly connected to six other servers. As in other proposals, e.g. DCell and BCube, multi-hop routing in CamCube requires servers to participate in packet forwarding. To date, as in existing data centers, these approaches have all provided a single routing protocol for the applications. In this paper we explore if allowing applications to implement their own routing services is advantageous, and if we can support it efficiently. This is based on the observation that, due to the flexibility offered by the CamCube API, many applications implemented their own routing protocol in order to achieve specific application-level characteristics, such as trading off higher-latency for better path convergence. Using large-scale simulations we demonstrate the benefits and network-level impact of running multiple routing protocols. We demonstrate that applications are more efficient and do not generate additional control traffic overhead. This motivates us to design an extended routing service allowing easy implementation of application-specific routing protocols on CamCube. Finally, we demonstrate that the additional performance overhead incurred when using the extended routing service on a prototype CamCube is very low.

Modeling virtual machine performance

Data centers are increasingly employing virtualization and consolidation as a means to support a large number of disparate applications running simultaneously on server platforms. However, server platforms are still being designed and evaluated based on performance modeling of a single highly parallel application or a set of homogenous work-loads running simultaneously. Since most future datacenters are expected to employ server virtualization, this paper takes a look at the challenges of modeling virtual machine (VM) performance on a datacenter server. Based on vConsolidate (a server virtualization benchmark) and latest multi-core servers, we show that the VM modeling challenge requires addressing three key problems: (a) modeling the contention of visible resources (cores, memory capacity, I/O devices, etc), (b) modeling the contention of invisible resources (shared microarchitecture resources, shared cache, shared memory bandwidth, etc) and (c) modeling overheads of virtual machine monitor (or hypervisor) implementation. We take a first step to addressing this problem by describing a VM performance modeling approach and performing a detailed case study based on the vConsolidate benchmark. We conclude by outlining outstanding problems for future work.

PortLand

This paper considers the requirements for a scalable, easily manageable, fault-tolerant, and efficient data center network fabric. Trends in multi-core processors, end-host virtualization, and commodities of scale are pointing to future single-site data centers with millions of virtual end points. Existing layer 2 and layer 3 network protocols face some combination of limitations in such a setting: lack of scalability, difficult management, inflexible communication, or limited support for virtual machine migration. To some extent, these limitations may be inherent for Ethernet/IP style protocols when trying to support arbitrary topologies. We observe that data center networks are often managed as a single logical network fabric with a known baseline topology and growth model. We leverage this observation in the design and implementation of PortLand, a scalable, fault tolerant layer 2 routing and forwarding protocol for data center environments. Through our implementation and evaluation, we show that PortLand holds promise for supporting a ``plug-and-play" large-scale, data center network.

The Thermal Design of a Next Generation Data Center: A Conceptual Exposition

In the near future, electronic cabinets of data centers will house high performance chips with heat fluxes approaching 100 W/cm2 and associated high volumetric heat generation rates. With the power trends in the electronic cabinets indicating 60 kW cabinets in the near future, the current cooling systems of data centers will be insufficient and new solutions will need to be explored. Accordingly, the key issue that merits investigation is identifying and satisfying the needed specifications of the new thermal solutions, considering the design environment of the next generation data centers. Anchoring our work in the open engineering system paradigm, we identify the requirements of the future thermal solutions and explore various design specifications of an ideally open thermal solution for a next generation data center. To approach an open cooling system for the future data centers, the concept of a thermal solution centered on the multiscale (multilevel) nature of the data centers is discussed. The potential of this solution to be open, along with its theoretical advantages compared with the typical air-cooling solutions, is demonstrated through some scenarios. The realization problems and the future research needs are highlighted to achieve a practical open multiscale thermal solution in data centers. Such solution is believed to be both effective and efficient for the next generation data centers.