Server Racks Research Articles

Design and Verification of Large-Scale Optical Circuit Switch Using ULCF AWGs for Datacenter Application

The exponential increase in the volume of traffic being processed by datacenters demands massive electrical switches and complicated interconnections. The power consumption will soon become critical, and the problem is expected to be resolved by introducing large-port-count optical circuit switches that connect many server racks or pods. In this paper, we propose a novel optical-circuit-switch architecture that uses n × N uniformloss and cyclic-frequency (ULCF) arrayed-waveguide gratings (AWGs) as low-loss wavelength routers. To confirm the effectiveness of our proposal, a pair of 12 × 48 ULCF AWGs is fabricated using planar-lightwave-circuit technology. Part of a 1536 × 1536 optical circuit switch is constructed, and its good transmission performance is experimentally confirmed through 96-wavelength 100-Gbps signal transmission in the full C-band. The switch throughput reaches 153.6 Tbps.

Quasi-Deterministic Channel Model Parameters for a Data Center at 60 GHz

We describe a measurement campaign conducted in a data center, a unique environment in which transceivers are positioned above the server racks. A total of 80 channel acquisitions, including small-scale measurements, were recorded using our three-dimensional double-directional 60 GHz channel sounder with 0.5 ns delay resolution and 2° average angle error. For each acquisition, the channel multipath components were extracted in the delay-angle space and subsequently reduced to the parameters of the Quasi-Deterministic channel model: a stochastic map-based model that represents the scattering of incident waves off ambient reflectors as specular components, each surrounded by a cluster of diffuse components. The weaker diffuse components interfere with the stronger specular component, giving rise to small-scale fading. Only with super-resolution capabilities, such as that of our system, were we able to show how densely the diffuse components were packed, with 1/ns to 2/ns arrival rates and cluster angular spreads between 3° and 4°. Most importantly, we measured the diffuse power to be as high as 26% of the total cluster power, demonstrating that small-scale fading at millimeter-wave frequencies can be significant.

Venice

Consolidated server racks are quickly becoming the standard infrastructure for engineering, business, medicine, and science. Such servers are still designed much in the way when they were organized as individual, distributed systems. Given that many fields rely on big-data analytics substantially, its cost-effectiveness and performance should be improved, which can be achieved by flexibly allowing resources to be shared across nodes. Here we describe Venice, a family of data-center server architectures that includes a strong communication substrate as a first-class resource. Venice supports a diverse set of resource-joining mechanisms that enables applications to leverage non-local resources efficiently. We have constructed a hardware prototype to better understand the implications of design decisions about system support for resource sharing. We use it to measure the performance of at-scale applications and to explore performance, power, and resource-sharing transparency tradeoffs (i.e., how many programming changes are needed). We analyze these tradeoffs for sharing memory, accelerators, and NICs. We find that reducing/hiding latency is particularly important, the chosen communication channels should match the sharing access patterns of the applications, and of which we can improve performance by exploiting inter-channel collaboration.

Capacity Optimization for Resource Pooling in Virtualized Data Centers with Composable Systems

Recent research trends exhibit a growing imbalance between the demands of tenants’ software applications and the provisioning of hardware resources. Misalignment of demand and supply gradually hinders workloads from being efficiently mapped to fixed-sized server nodes in traditional data centers. The incurred resource holes not only lower infrastructure utilization but also cripple the capability of a data center for hosting large-sized workloads. This deficiency motivates the development of a new rack-wide architecture referred to as the composable system . The composable system transforms traditional server racks of static capacity into a dynamic compute platform. Specifically, this novel architecture aims to link up all compute components that are traditionally distributed on traditional server boards, such as central processing unit (CPU), random access memory (RAM), storage devices, and other application-specific processors. By doing so, a logically giant compute platform is created and this platform is more resistant against the variety of workload demands by breaking the resource boundaries among traditional server boards. In this paper, we introduce the concepts of this reconfigurable architecture and design a framework of the composable system for cloud data centers. We then develop mathematical models to describe the resource usage patterns on this platform and enumerate some types of workloads that commonly appear in data centers. From the simulations, we show that the composable system sustains nearly up to 1.6 times stronger workload intensity than that of traditional systems and it is insensitive to the distribution of workload demands. This demonstrates that this composable system is indeed an effective solution to support cloud data center services.

Time Synchronization Solution for FPGA-based Distributed Network Monitoring

Distributed network monitoring solutions face various challenges with the increase of line speed, the extending variety of protocols, and new services with complex KPIs. This paper addresses one part of the first challenge: faster line speed necessitates time-stamping with higher granularity and higher precision than ever. Proper, system-wide time-stamping is inevitable for network monitoring and traffic analysis point of view. It is hard to find feasible time synchronization solutions for those systems that have nation-wide, physically distributed probes. Current networking equipment reside in server rooms, and have many legacy nodes. Access to GPS signal is complicated in these places, and Precision Time Protocol (PTP) does not seem to be supported by all network nodes in the near future – so high precision time-stamping is indeed a current problem. This paper suggests a novel, practical solution to overcome the obstacles. The core idea is that in real-life, distributed network monitoring systems operate with a few, finite number of probeclusters, and their site should have a precise clock provided by PTP or GPS somewhere in the building. The distribution of time information within a site is still troublesome, even within a server rack. This paper presents a closed control loop solution implemented in an FPGA-based device in order to minimize the jitter, and compensate the calculated delay.

Experimental study on heat sink with porous copper as conductive material for CPU cooling

With the fast development of information science and technology, the increasing demand for computer data processing caused a rapid growth in central processing unit (CPU) of high power capacity. CPU Cooling plays an important role in guaranteeing reliable working conditions with proper temperature for IT equipment, rack servers and related devices. In this paper, a novel kind of heat sink filled with porous copper is proposed and experiment is conducted to investigate its cooling effect for CPU. The results show that cooling capacity of porous copper increases with increasing inlet and outlet pressure difference, while it almost remains the same with heat flux variations. Moreover, the CPU surface temperatures of two different heat sinks are compared. It indicates that the heat sink coefficient of porous copper increases by 28.5 % and the surface temperature decreases by more than 10 oC, compared to traditional CPU heat sink with micro channels. This work is important for CPU cooling system design and optimization in practical applications.

SOLIDpower SOFC units commissioned at Microsoft data centre

More details have emerged about the Microsoft fuel cell powered data centre announced last month [FCB, October 2017, p1], which uses solid oxide fuel cell generators manufactured by Italian-based SOLIDpower to directly power the server racks. Microsoft is hosting the demonstration in a data centre just a few kilometres from its corporate headquarters in Seattle, Washington.

Effectiveness of Rack-Level Fans—Part I: Energy Savings Through Consolidation

In general, smaller fans operate at lower efficiencies than larger fans of proportional linear dimensions. In this work, the applicability of replacing smaller, 60 mm baseline fans from within the chassis of web servers with an array of larger, geometrically proportional 80 mm and 120 mm fans consolidated to the back of a rack is experimentally tested. Initial characterization of the selected fans showed that the larger fans operate at double peak total efficiency of the smaller fans. A stack of four servers were used in a laboratory setting to represent a rack of servers. When all four servers were stressed at uniform computational loadings, the 80 mm fans resulted in 50.1–52.6% reduction in total rack fan power compared to the baseline fans. The 120 mm fans showed similar reduction in rack fan power of 47.6–54.0% over the baseline. Since actual data centers rarely operate at uniform computational loading across servers in a rack, a worst case scenario test was conceived in which the array of larger fans were controlled by a single server operating at peak computational workload while the other three in the rack remained idle. Despite significant overcooling in the three idle servers, the 80 mm and 120 mm fan configurations still showed 35% and 34% reduction in total rack fan power compared to the baseline fans. The findings strongly suggest that a rack-level fan scheme in which servers share airflow from an array of consolidated larger fans is superior to traditional chassis fans.

Energy efficiency comparison of hypervisors

Current cloud data centers are fully virtualized for service consolidation and power/energy reduction. Although virtualization could reduce the real-time power consumption and overall energy consumption, the energy characteristics of hypervisors hosting different workloads have not been well profiled or understood thus far. In this study, we investigate the power and energy characteristics of four mainstream hypervisors and a container engine, namely VMware ESXi, Microsoft Hyper-V, KVM, XenServer, and Docker, on six different platforms (three mainstream 2U rack servers, one emerging ARM64 server, one desktop server, and one laptop) with power measurements made over prolonged periods. We use computation-intensive, memory-intensive, and mixed Web server-database workloads to explore the power and energy characteristics of different hypervisors in order to emulate realistic multi-tenant cloud environments. The results of extensive experiments conducted with four workload levels (very light, light, fair, and very heavy) indicate that the hypervisors exhibit different power and energy characteristics. Our findings are as follows. (1) Hypervisors exhibit different power and energy consumptions on the same hardware running the same workload. (2) Although mainstream hypervisors have different energy efficiencies aligned with different workload types and workload levels, no single hypervisor outperforms the other hypervisors on all platforms in terms of power or energy consumption. (3) Although container virtualization is considered as lightweight virtualization in terms of implementation and maintenance, it is essentially not more power-efficient than conventional virtualization technology. (4) Although the ARM64 server has low power consumption, it completes computation tasks with a long execution time and, sometimes, high energy consumption. Further, ARM64 servers have medium energy consumption per database operation for mixed workloads. The results presented in this paper can provide system designers and data center operators with useful insights for power-aware workload placement and virtual machine scheduling.

On achieving intelligent traffic-aware consolidation of virtual machines in a data center using Learning Automata

Unlike the computational mechanisms of the past many decades, that involved individual (extremely powerful) computers or clusters of machines, cloud computing (CC) is becoming increasingly pertinent and popular. Computing resources such as CPU and storage are becoming cheaper, and the servers themselves are becoming more powerful. This enables clouds to host more virtual machines (VMs). A natural consequence of this is that many modern-day data centers experience very high internal traffic within the data centers themselves. This is, of course, due to the occurrence of servers that belong to the same tenant, communicating between themselves. The problem is accentuated when the VM deployment tools are not traffic-aware. In such cases, the VMs with high mutual traffic often end up being far apart in the data center network, forcing them to communicate over unnecessarily long distances. The consequent traffic bottlenecks negatively affect both the performance of the application and the network in its entirety, posing non-trivial challenges for the administrators of these cloud-based data centers.The problem, and consequently the solution, can, quite naturally, be compartmentalized into two phases which follow each other. In the first, the task is to consolidate VMs into clusters, where those that communicate with each other fall into the same cluster. The second phase assigns these clusters onto the available server racks. Both of these phases must be executed in a traffic-aware manner. This paper provides efficient intelligent solutions for both these phases. First of all, the VMs are consolidated with a VM clustering algorithm, and this is achieved by utilizing the toolbox involving Learning Automata (LA). By mapping the clustering problem onto the Graph Partitioning (GP) problem, our LA-based solution successfully reduces the total communication cost by amounts that range between 34% and 85%. Thereafter, the resulting clusters are assigned to the server racks using a cluster placement algorithm that involves a completely different intelligent strategy, i.e., one that invokes Simulated Annealing (SA). This phase further reduces the total cost of communication by amounts that range between 89% and 99%. The analysis and results for different models and topologies demonstrate that the optimization is done in a fast and computationally-efficient way.

Towards Rack Utilization in Internet Datacenters: An Approach Based on Dynamic Programming

In the datacenter rented to Internet Service Providers (ISPs), the low utilization of racks can seriously affect ISPs’ benefit because the ISPs are charged by “rack per year” rather than servers. Based on our analysis about the utilization data from production systems, we find that the over-provisioning of power results in low power utilization, which potentially decreases rack utilization as well as the ISPs’ benefit. To improve the rack utilization, maximizing the number of servers in racks is an effective way. In this paper, we propose a server placement algorithm to minimize the power fragment in a rack. The experimental results show that it can save more than 50% leasing cost (rack utilization improvement) in evaluation.

Experimental Study of Energy Saving Performances in Chip Cooling by using Heat Sink with Embedded Heat Pipe

With the rapid development of computer industry, the computer performance is seriously restricted by the energy consumption and heat dissipation of high heat flux chip. The increase in high density server racks makes the power supply and the heat dissipation more complex. Thus, it is necessary to optimize the computer chip cooling technology. In this study, heat pipe was used to enhance the cooling performance of CPU. Different forms of heat sink embedded with heat pipe were adopted, and the heat dissipation and energy saving performance were studied. Consequently, the cooling performance can be enhanced with the new cooling technology, and the energy consumption can also be reduced in order to make the CPU work at normal operation temperature, due to high efficiency heat dissipation technology.

Studying the fan-assisted cooling using the Taguchi approach in open and closed data centers

This research examined the application of fan-assisted cooling within open and contained raised-floor data centers with the intention of assessing the effectiveness of the system’s cooling method. A computational fluid dynamics (CFD) approach that incorporated both airflow and thermal analysis was employed to examine the detailed effectiveness of fan-assisted air-cooling in a data center that exhibited the behavior of under-cabinet leakage. The CFD technique was also used to assess the impact that three distinct aspects had on the performance of the fan-assisted air-cooling system: cold aisle containment, straightened air flow and fan-to-tile distance. In addition, the mixed-level Taguchi statistical approach was applied to understand how variations in these parameters directly and interactively modified the heat transfer in the surroundings of the inlets of the server racks. The results of the research reveal that fan-assisted perforations, taking the combination of the key factors (the main and interact effects) into account, would represent a viable means through which the cooling systems employed in both open and closed aisle data centers can be enhanced. The well-constructed CFD-based Taguchi methodology might yield a good level of validity for use in substantial and timely optimization procedures to further improve the cooling effectiveness and efficiencies of data centers.

Tandem Equipment Arranged Architecture with Exhaust Heat Reuse System for Software-Defined Data Center Infrastructure

In this paper, we propose a novel energy-efficient architecture for software-defined data center infrastructures. In our proposed data center architecture, we include an exhaust heat reuse system that utilizes high-temperature exhaust heat from servers in conditioning humidity and air temperature of office space near the data center. To obtain high-temperature exhaust heat, equipment such as server racks and air conditioners are deployed in tandem so that the aisles are divided into three types: cold, hot, and super-hot. In this paper, to investigate the fundamental characteristics of our proposed data center architecture, we consider various types of data center models and conduct numerical simulations that use results obtained by experiments at an actual data center. Through simulation, we show that the total power consumption by a data center with our proposed architecture is 27 percent lower than that by data center with a conventional architecture. In addition, it is also shown that the proposed tandem equipment arrangement is suitable for obtaining high-temperature exhaust heat and decreasing the total power consumption significantly under a wider range of conditions than in the conventional equipment arrangement.

Dynamic performance of an in-rack proton exchange membrane fuel cell battery system to power servers

To improve the reliability and the energy efficiency of data centers, as well as to reduce infrastructure costs and environmental impacts, we experimentally evaluated in-rack powering of servers with a hybrid 12 kW Proton Exchange Membrane Fuel Cell (PEMFC) and battery system. The steady state and the transient performance of the PEMFC and battery in response to dynamic AC loads and real server loads have been evaluated and characterized. The PEMFC system responds quickly and reproducibly to load changes directly from the server rack. Peak efficiency of 55.2% in a single server rack can be achieved. The effect of fuel cell coolant temperature on the hybrid system transient behavior is also captured and evaluated. The observed PEMFC transient responses obtained from the experiments were used to design the size of the energy storage component for the hybrid system. Simulations and analysis of various types of energy storage devices for the hybrid system were carried out. To provide power to meet the most significant transient demand, energy storage capacity greater than 0.3 kWh is required for all battery types, while only 0.053 kWh capacity is required for the ultracapacitor. During charging, the ultracapacitor uses the shortest amount of time to recover to the original SOC, while the charging duration for the lead acid battery is twice as long as that of the ultracapacitor.

A Wireless Interconnection Framework for Seamless Inter and Intra-Chip Communication in Multichip Systems

Computing modules in typical datacenter nodes or server racks consist of several multicore chips either on a board or in a System-in-Package (SiP) environment. State-of-the-art inter-chip communication over wireline channels require data signals to travel from internal nets to the peripheral I/O ports and then get routed over the inter-chip channels to the I/O port of the destination chip. Following this, the data is finally routed from the I/O to internal nets of the destination chip over a wireline interconnect fabric. This multihop communication increases energy consumption while decreasing data bandwidth in a multichip system. Also, traditional I/O does not scale well with technology generations due to limitations of pitch. Moreover, intra-chip and inter-chip communication protocol within such a multichip system is often decoupled to facilitate design flexibility. However, a seamless interconnection between on-chip and off-chip data transfer can improve the communication efficiency significantly. Here, we propose the design of a seamless hybrid wired and wireless interconnection network for multichip systems with dimensions spanning up to tens of centimeters with on-chip wireless transceivers. We demonstrate with cycle accurate simulations that such a design increases the bandwidth and reduces the energy consumption in comparison to state-of-the-art wireline I/O based multichip communication.

Experimental study of solving thermal heterogeneity problem of data center servers

Desirable thermal management of data center requires uniform temperature distribution along the servers. Hot air recirculation and cold air bypass in data center leads to non-homogeneous cold air distribution along the servers of the racks which may lead to heterogeneous temperatures distribution along the servers. The present work aims to experimentally study the possibility of controlling these heterogeneous temperature distributions by controlling the cold air flow rates along the servers. A physical scaled data center model was used to conduct this investigation. The effectiveness of thermal management of the servers racks of the data centers has been expressed in terms of intake, rare and surface temperature distributions along the rack servers and the supply and return heat indices (commonly symbolized as SHI and RHI; respectively). Excessive tests were firstly performed under uniform servers fans speed (uniform air flow rates through the different servers). Then the air flow rates distributions along the racks servers has been changed by regulating the server’s fans speeds using different schemes of fans speeds regulations at different data centers power densities. It is concluded that a uniform increase of server’s flow rate from the bottom to the top of servers rack cabinet provides (i) the lowest temperature at both cooling aisle (around 10%) and exhaust aisles (around 5%), (ii) the best uniform surface temperature of all rack servers (as the standard deviation is reduced from 10 to around 2), and (iii) the best values of thermal management metrics (SHI and RHI) typically SHI is reduced by around 20% while RHI is increased by around 3% to approach the targeted values; 0.1 and 0.9, respectively.

파티션 시스템 적용을 통한 기존 데이터센터 서버실의 냉방 에너지 절감 성능평가

본 연구에서는 기존 데이터센터 서버실에서 공기분배시스템으로 파티션 시스템의 적용성을 평가하기 위해 기존 시스템 그리고 파티션의 설치 높이와 위치를 변수로 하는 파티션 시스템과 컨테인먼트 시스템에 대해 총 21가지 경우의 컴퓨터시뮬레이션을 실시하고 다음과 같은 결론을 얻었다. 공기온도 및 기류 분포 해석결과를 이용하여 평가해 본 공기분배시스템별 냉방 에너지 절감 성능은 컨테인먼트 시스템, 파티션 시스템, 기존 시스템 순서로 우수한 것을 확인할 수 있었다. 기존 시스템과 파티션 설치높이 0.1~0.7m까지의 파티션 시스템은 재순환 공기에 의해 냉복도에서 1.0~1.2m 높이를 기점으로 공기 온도가 급격히 상승하면서 중간 높이의 서버와 상부 서버의 인입구 온도차는 <TEX>$11{\sim}15^{\circ}C$</TEX>로 큰 차이를 보여, 재순환 공기가 상부 서버의 과열의 원인을 제공하는 것으로 나타났다. 파티션 시스템에서 냉복도 상부에 파티션을 설치할 경우에 파티션의 적정 높이는 서버 랙 상부에서 천장까지 높이의 90%(0.9m)이상, 열복도 상부에 파티션을 설치하는 경우는 파티션 높이가 80%(0.8m)이상은 되어야 재순환 공기를 충분히 차단하여 서버 인입구 공기온도가 RCI 권장온도 범위를 만족하는 냉방 에너지 절감효과를 볼 수 있는 것으로 나타났다. In this study, a computer simulation of the three types of air distribution systems, open type system, aisle partition system and aisle containment system, to evaluate the applicability of the aisle partition system on a data center server room. The variables of the simulation were the height and location of the partition fixed on the top server rack. The energy efficiency of the air distribution systems were confirmed to be excellent in the order of the aisle containment system, aisle partition system, and open type system. In the cold aisle partition system, the height of the partition that can be effective in saving cooling energy by obstructing sufficient air recirculation was found to be more than 0.9m. In the hot aisle partition system, the height of the partition was found to be more than 0.8m.

Estimated aggregate cost of ownership of a data processing center

Lyubov A. Pirogova - Project manager LEAN, Lamoda CompanyAddress: 10, Letnikovskaya Street, Moscow, 115114, Russian FederationE-mail: lbaydalina@gmail.comVladimir I. Grekoul - Professor, Department of Information Systems and Digital Infrastructure Management, National Research University Higher School of EconomicsAddress: 20, Myasnitskaya Street, Moscow, 101000, Russian FederationE-mail: grekoul@hse.ru Boris Е. Poklonov - Associate Professor, Department of Information Systems and Digital Infrastructure Management, National Research University Higher School of EconomicsAddress: 20, Myasnitskaya Street, Moscow, 101000, Russian FederationE-mail: bpoklonov@hse.ru Under current conditions, we see growth in demand for IT outsourcing services. This implies the activation of design and construction processes for data processing centers (DPC). Since a DPC is a complicated and expensive system, there arises the issue of justifying selection of the future project based on the estimated costs of designing and operating data processing centers. This paper analyzes one of the possible complexes of measures to estimate costs for development and operation of data processing centers. The analysis identified main groups of capital cost in development of data processing centers which were not fully taken into account in assessments of the total volume of capital investments in previously proposed methods. The article proposes regression models to evaluate processing center construction projects based on two measures. We propose to estimate the capital cost as a function of the projected floor space of service platforms and projected number of server racks. On the basis of the models developed, analysis of the construction sites of processing data centers was conducted. This showed the model’s suitability to real data. The main groups of operating costs for DPC maintenance were established, and a regression model of their evaluation was proposed. Based on the regression equation, we propose to calculate the processing center’s power consumption depending on the area of the service platform or the number of server racks. The operating cost of the data processing center is determined by the power value. Analysis of information on the operating cost of various data processing centers is in fairly good agreement with the calculations obtained on the basis of the model developed. The proposed models make it possible to evaluate with reasonable accuracy the project characteristics of development and subsequent operation of a data processing center.

Power attack defense

Battery systems are crucial components for mission-critical data centers. Without secure energy backup, existing under-provisioned data centers are largely unguarded targets for cyber criminals. Particularly for today's scale-out servers, power oversubscription unavoidably taxes a data center's backup energy resources, leaving very little room for dealing with emergency. Besides, the emerging trend towards deploying distributed energy storage architecture causes the associated energy backup of each rack to shrink, making servers vulnerable to power anomalies. As a result, an attacker can generate power peaks to easily crash or disrupt a power-constrained system. This study aims at securing data centers from malicious loads that seek to drain their precious energy storage and overload server racks without prior detection. We term such load as Power Virus (PV) and demonstrate its basic two-phase attacking model and characterize its behaviors on real systems. The PV can learn the victim rack's battery characteristics by disguising as benign loads. Once gaining enough information, the PV can be mutated to generate hidden power spikes that have a high chance to overload the system. To defend against PV, we propose power attack defense (PAD), a novel energy management patch built on lightweight software and hardware mechanisms. PAD not only increases the attacking cost considerably by hiding vulnerable racks from visible spikes, it also strengthens the last line of defense against hidden spikes. Using Google cluster traces we show that PAD can effectively raise the bar of a successful power attack: compared to prior arts, it increases the data center survival time by 1.6~11X and provides better performance guarantee. It enables modern data centers to safely exploit the benefits that power oversubscription may provide, with the slightest cost overhead.