At Scale Development of Thermal Test Vehicles for Data Center Liquid Cooling Developments

Abstract

Abstract High-performance computing with the latest generation of high-density chips and microprocessors in data centers requires advanced and efficient cooling technology to meet the challenging thermal requirements. Indirect liquid cooling with cold plates on top of the processor has become an attractive and efficient solution while transitioning from traditional air cooling inside the data center. Using a thermal test vehicle (TTV) can be a cost-effective and reliable methodology to commission liquid-cooled components successfully without risking actual IT equipment. TTVs can also be highly beneficial in testing and developing data center cooling technologies for future products with individual thermal requirements. In this paper, TTVs are created to replicate the thermal behavior of heat-generating components to characterize case-to-coolant thermal resistance, find hotspots, develop and test cooling technologies cost-effectively. Different design requirements influence the development and the intricacy involved in the construction, manufacturing, and telemetry of the TTV. In our study, TTVs for a 3U sled are discussed in detail, touching on different aspects such as proof of concept, design, verification, manufacturing, testing, and verification in a data center environment. The effects of choosing appropriate material properties, electrical requirements, form factors, stack-up method, type of heat-generating components, thermocouple placement, and nonuniform heat flux are also presented following analytical, numerical, and experimental results. The results and failure cases suggest some best practices in designing non-silicon TTVs while mimicking higher heat load in form factor of high heat density servers. Manufactured TTVs are placed in a chassis following the configuration and form factor of the 3U sled in high heat density servers and scaled up in a rack. In addition, a custom rack mount TTV server named “Olympus” is presented as an outcome of the study. Cold plates and cooling loops can be tested on the TTV server for thermal and hydraulic characterization with rack and row manifolds, Liquid-to-Liquid (L2L), and Liquid-to-Air (L2A) heat exchangers in a data center test environment. Later in the paper, the commissioning of liquid cooling technologies with TTVs is demonstrated in a data center environment. Real-time data for heater spreader temperature in TTVs are plotted with the help of a front-end application with boundary conditions on both primary and secondary sides. As a result, this paper provides complete guidelines on the development, verification, and application of TTV to determine critical performance parameters in commissioning (such as flow rate, temperature, pressure drop, cooling capacity, etc.) and simulate failure scenarios to ensure the reliability of components in the loop.