Server Board Research Articles

L2A Cdus Performance and Considerations for Server Rooms Upgrade with Conventional Air Conditioning

Abstract As web-based AI applications are growing rapidly, server rooms face escalating computational demands, prompting enterprises to either upgrade their facilities or outsource to co-located sites. This growth strains conventional HVAC systems, which struggle to handle the substantial thermal load, often resulting in hotspots. Liquid-to-Air (L2A) Coolant Distribution Units (CDUs) emerge as a solution, efficiently cooling servers by circulating liquid coolant through cooling loops mounted on each server board. In this study, the performance of a 24-kW L2A CDU is evaluated across various scenarios, emphasizing cooling effect, stability, and reliability. Experimental tests involve a rack with three thermal test vehicles (TTVs), monitoring both liquid coolant and air sides for analysis. Tests are conducted in a limited air-conditioned environment, resembling upgraded server rooms with conventional AC systems. The study also assesses the impact of high-power density cooling units on the server room environment, measuring noise, air velocity, and ambient temperature against ASHRAE standards for human comfort. Recommendations for optimal practices and potential system improvements are included in the research, addressing the growing need for efficient cooling solutions amidst escalating computational demands.

Modeling, Verification, and Signal Integrity Analysis of High-Speed Signaling Channel with Tabbed Routing in High Performance Computing Server Board

It is necessary to reduce the crosstalk noise in high-speed signaling channels. In the channel routing area, the tabbed routing pattern is used to mitigate far-end crosstalk (FEXT), and the electrical length is controlled with a time domain reflectometer (TDR) and time domain transmission (TDT). However, unlike traditional channels having uniform width and space, the width and space of tabbed routing changes by segment, and the capacitance and inductance values of tabbed routing also change. In this paper, we propose a tabbed routing equivalent circuit modeling method using the segmentation approach. The proposed model was verified using 3D EM simulation and measurement results in the frequency domain. Based on the calculated inductance and capacitance parameters, we analyzed the insertion loss, FEXT, and self-impedance in the frequency domain, and TDT and FEXT in the time domain, by comparing the values of these metrics with and without tabbed routing. Using the proposed tabbed routing model, we analyzed tabbed routing with variations of design parameters based on self- and mutual-capacitance and inductance.

Silicon circuits for chip‐to‐chip communications in multi‐socket server board interconnects

Multi-socket server boards (MSBs) exploit the interconnection of multiple processor chips towards forming powerful cache coherent systems, with the interconnect technology comprising a key element in boosting processing performance. Here, we present an overview of the current electrical interconnects for MSBs, outlining the main challenges currently faced. We propose the use of silicon photonics (SiPho) towards advancing interconnect throughput, socket connectivity and energy efficiency in MSB layouts, enabling a flat-topology wavelength division multiplexing (WDM)-based point-to-point (p2p) optical MSB interconnect scheme. We demonstrate WDM SiPho transceivers (TxRxs) co-assembled with their electronic circuits for up to 50 Gb/s line rate and 400 Gb/s aggregate data transmission and SiPho arrayed waveguide grating routers that can offer collision-less time of flight connectivity for up to 16 nodes. The capacity can scale to 2.8 Gb/s for an eight-socket MSB, when line rate scales to 50 Gb/s, yielding up to 69% energy reduction compared with the QuickPath Interconnect and highlighting the feasibility of single-hop p2p interconnects in MSB systems with >4 sockets.

400 Gb/s Silicon Photonic Transmitter and Routing WDM Technologies for Glueless 8-Socket Chip-to-Chip Interconnects

Arrayed Waveguide Grating Router (AWGR)-based interconnections for Multi-Socket Server Boards (MSBs) have been identified as a promising solution to replace the electrical interconnects in glueless MSBs towards boosting processing performance. In this article, we present an 8-socket glueless optical flat-topology Wavelength Division Multiplexing (WDM)-based point-to-point (P2P) interconnect pursued within the H2020 ICT project ICT-STREAMS and we report on our latest achievements in the deployment of the constituent silicon (Si)-photonic transmitter and routing building blocks, exploiting experimentally obtained performance metrics for analyzing the 8-socket chip-to-chip (C2C) connectivity in terms of throughput and energy efficiency. We demonstrate an 8-channel WDM Si-photonic microring-based transmitter (Tx) capable of providing 400 (8 × 50) Gb/s non-return-to-zero (NRZ) Tx capacity and an 8 × 8 Coarse-WDM (CWDM) Si-AWGR with verified cyclic data routing capability in O-band. Following an overview of our recently demonstrated crosstalk (XT)-aware wavelength allocation scheme, that enables fully-loaded AWGR-based interconnects even for typical sub-optimal XT values of silicon integrated CWDM AWGRs, we validate the performance of a full-scale 8-socket interconnect architecture through physical layer simulations exploiting experimentally-verified simulation models for the underlying Si-photonic Tx and routing circuits. This analysis reveals a total aggregate capacity of 1.4 Tb/s for an 8-socket interconnect when operating with 25 Gb/s line-rates, which can scale to 2.8 Tb/s at an energy efficiency of just 5.02 pJ/bit by exploiting the experimentally verified building block performance at 50 Gb/s line. This highlights the perspectives for up to 69% energy savings compared to the standard QuickPath Interconnect (QPI) typically employed in electronic glueless MSB interconnects, while scaling the single-hop flat connectivity from 4- to 8-socket interconnection systems.

An ARM cluster for running CMSSW jobs

The ARM platform extends from the mobile phone area to development board computers and servers. It could be that in the future the importance of the ARM platform will increase for High Performance Computing/High Throughput Computing (HPC/HTC) if new more powerful (server) boards are released. For this reason Compact Muon Solenoid Software (CMSSW) has previously been ported to ARM in earlier work. The CMSSW is deployed using the CERN Virtual Machine File System (CVMFS) and the jobs are run inside Singularity containers. Some ARM AArch64 CMSSW releases are available in CVMFS for testing and development. In this work CVMFS and Singularity have been compiled and installed on an ARM cluster and the AArch64 CMSSW releases in CVMFS have been used. We report on our experiences with this ARM cluster for CMSSW jobs. Commodity hardware designed around the 64-bit architecture has been the basis of current virtualization trends with the advantage to emulate diverse environments for a wide range of computational scenarios. However, in parallel the mobile revolution have given a rise to ARM SoCs with primary focus on power efficiency. While still in the experimental phase, the power efficiency and 64-bit heterogeneous computing already point to an alternative option for traditional x86_64 CPUs servers for datacenters.

A 40 Gb/s Chip-to-Chip Interconnect for 8-Socket Direct Connectivity Using Integrated Photonics

We present an O-band any-to-any chip-to-chip (C2C) interconnection at 40 Gb/s suitable for up to 8-socket direct connectivity in multi-socket server boards, utilizing integrated low-energy photonics for the transceiver and routing functions. The C2C interconnect exploits an Si-based ring modulator as its transmitter and a co-packaged photodiode/transimpedance amplifier enabled receiver interconnected over an 8 x 8 Si-based arrayed waveguide grating router, allowing for a single-hop flat-topology interconnection between eight nodes. A proof-of-concept demonstration of the C2C interconnect is presented at 25 and 40 Gb/s for eight possible routing scenarios, revealing clear eye diagrams at both data rates with extinction ratios of 4.8 +/- 0.3 and 4.38 +/- 0.31 dB, respectively, among the eight routed signals.

Capacity Optimization for Resource Pooling in Virtualized Data Centers with Composable Systems

Recent research trends exhibit a growing imbalance between the demands of tenants’ software applications and the provisioning of hardware resources. Misalignment of demand and supply gradually hinders workloads from being efficiently mapped to fixed-sized server nodes in traditional data centers. The incurred resource holes not only lower infrastructure utilization but also cripple the capability of a data center for hosting large-sized workloads. This deficiency motivates the development of a new rack-wide architecture referred to as the composable system . The composable system transforms traditional server racks of static capacity into a dynamic compute platform. Specifically, this novel architecture aims to link up all compute components that are traditionally distributed on traditional server boards, such as central processing unit (CPU), random access memory (RAM), storage devices, and other application-specific processors. By doing so, a logically giant compute platform is created and this platform is more resistant against the variety of workload demands by breaking the resource boundaries among traditional server boards. In this paper, we introduce the concepts of this reconfigurable architecture and design a framework of the composable system for cloud data centers. We then develop mathematical models to describe the resource usage patterns on this platform and enumerate some types of workloads that commonly appear in data centers. From the simulations, we show that the composable system sustains nearly up to 1.6 times stronger workload intensity than that of traditional systems and it is insensitive to the distribution of workload demands. This demonstrates that this composable system is indeed an effective solution to support cloud data center services.

A Feasible Architecture for ARM-Based Microserver Systems Considering Energy Efficiency

In this paper, we propose the architecture of the ARM-based server system and physically build the ARM-based server cluster board (SCB). Based on a real implementation, the efficiency of the microserver is evaluated to verify the benefits of the proposed scheme. In experiments, every SCB is equipped with four server-grade ARM quad-core processors for enterprise-class applications. The major difference between our work and other studies is that this paper tests the performance of the server-grade processors, whereas previous works use the cluster system formed by the embedded processors. The experiment results show that the SCB-based systems perform with lower power-consumption than the x86 systems. Since various types of cluster systems can be built with the SCB through either the PCI Express (PCIe) interface or gigabit Ethernet interface, the proposed design of the ARM server board and the systems can, therefore, be extensively applied. This physical implementation and measurement show that the proposed architecture can work well. The proposed energy-saving (i.e., green) server designs with low-power processors are suitable for relative applications in the coming era of Industry 4.0 and the Internet of Things.

On-chip two-phase cooling of datacenters: Cooling system and energy recovery evaluation

Cooling of datacenters is estimated to have an annual electricity cost of 1.4 billion dollars in the United States and 3.6 billion dollars worldwide. Currently, refrigerated air is the most widely used means of cooling datacenter’s servers, which typically represents 40-45% of the total energy consumed in a datacenter. Based on the above issues, thermal designers of datacenters and server manufacturers now seem to agree that there is an immediate need to improve the server cooling process. The goal of the present study is to propose and simulate the performance of a novel hybrid two-phase cooling cycle with micro-evaporator elements (multi-microchannel evaporators) for direct cooling of the chips and auxiliary electronics on blade server boards (savings in energy consumption of over 60% are expected). Different working fluids were considered, namely water, HFC134a and a new, more environmentally friendly, refrigerant HFO1234ze. The results so far demonstrated that the pumping power consumption is on the order of 5 times higher for the water-cooled cycle. Additionally, a case study considering the hybrid cooling cycle applied on a datacenter and exploring the application of energy recovered in the condenser on a feedwater heater of a coal power plant was also investigated (modern datacenters require the dissipation of 5-15 MW of heat). Aspects such as minimization of energy consumption and CO2 footprint and maximization of energy recovery (exergetic efficiency) and power plant efficiency are investigated.

Spray Cooling Thermal Management for Increased Device Reliability

A study has been conducted to quantify the ability of spray cooling to handle transient die power dissipation, as well as to quantify its effect on device reliability. The transient study was conducted with a bare die thermal test vehicle, while the device reliability studies were conducted with a dual Opteron Compact PCI single board computer. For the transient response studies, the high heat transfer coefficients associated with the spray cooling enabled the die to transition from one steady-state point to another in under 2 seconds, when cycling the power delivered to the die between 0 and 94 W. For the device reliability studies, an air-cooled version of the server board was compared to a spray cooled version, for the same clock frequency and supply voltages. For identical conditions, the spray cooled microprocessor processor diode temperature was 33.3/spl deg/C lower, and the die dissipated 35% less power. The paper will present an analysis that explains the manner in which modern microprocessors operate, and will explain the reason for the higher power dissipation associated with the hotter microprocessor.