GPU Cluster Research Articles

Taming Flexible Job Packing in Deep Learning Training Clusters

Job packing is an effective technique to harvest the idle resources allocated to the deep learning (DL) training jobs but not fully utilized, especially when clusters may experience low utilization, and users may overestimate their resource needs. However, existing job packing techniques tend to be conservative due to the mismatch in scope and granularity between job packing and cluster scheduling. In particular, tapping the potential of job packing in the training cluster requires a local and fine-grained coordination mechanism. To this end, we propose a novel job-packing middleware named Gimbal , which operates between the cluster scheduler and the hardware resources. As middleware, Gimbal must not only facilitate coordination among the packed jobs but also support various scheduling objectives of different schedulers. Gimbal achieves dual functionality by introducing a set of worker calibration primitives designed to calibrate workers’ execution status in a fine-grained manner. The primitives obscure the complexity of the underlying job and resource management mechanisms, thus offering the generality and extensibility for crafting coordination policies tailored to various scheduling objectives. We implement Gimbal on a real-world GPU cluster and evaluate it with a set of representative DL training jobs. The results show that Gimbal improves different scheduling objectives up to 1.32 × compared with the state-of-the-art job packing techniques.

SLoB: Suboptimal Load Balancing Scheduling in Local Heterogeneous GPU Clusters for Large Language Model Inference

SLoB: Suboptimal Load Balancing Scheduling in Local Heterogeneous GPU Clusters for Large Language Model Inference

Advanced Techniques for High-Performance Fock Matrix Construction on GPU Clusters.

This Article presents two optimized multi-GPU algorithms for Fock matrix construction, building on the work of Ufimtsev and Martinez [ J. Chem. Theory Comput. 2009, 5, 1004-1015] and Barca et al. [ J. Chem. Theory Comput. 2021, 17, 7486-7503]. The novel algorithms, opt-UM and opt-Brc, introduce significant enhancements, including improved integral screening, exploitation of sparsity and symmetry, a linear scaling exchange matrix assembly algorithm, and extended capabilities for Hartree-Fock caculations up to f-type angular momentum functions. Opt-Brc excels for smaller systems and for highly contracted triple-ζ basis sets, while opt-UM is advantageous for large molecular systems. Performance benchmarks on NVIDIA A100 GPUs show that our algorithms in the EXtreme-scale Electronic Structure System (EXESS), when combined, outperform all current GPU and CPU Fock build implementations in TeraChem, QUICK, GPU4PySCF, LibIntX, ORCA, and Q-Chem. The implementations were benchmarked on linear and globular systems and average speed ups across three double-ζ basis sets of 1.4×, 8.4×, and 9.4× were observed compared to TeraChem, QUICK, and GPU4PySCF respectively. An increased average speedup of 2.1× over TeraChem is observed when using four A100 GPUs. Strong scaling analysis reveals over 91% parallel efficiency on four GPUs for opt-Brc, making it typically faster for multi-GPU execution. Single-compute-node comparisons with CPU-based software like ORCA and Q-Chem show speedups of up to 42× and 31×, respectively, enhancing power efficiency by up to 18×.

GPU cluster dynamics: insights from Alibaba’s 2023 trace release

GPU cluster dynamics: insights from Alibaba’s 2023 trace release

Multisegment Overlap–Save Method for Coherent Dedispersion

Dispersion occurs due to the interstellar medium, which functions as a prism and causes different time delays in radio waves of varying frequencies. The coherent dedispersion technique is often used in pulsar and fast radio burst observations to mitigate this phenomenon. The widely recognized Overlap–Save approach enables this dedispersion algorithm to efficiently perform long linear convolution. However, with the present implementation, the necessary filter length for dedispersion can reach 100 million points or more. The corresponding fast Fourier transform (FFT) points should be larger than this value and a GPU cluster can be used to tackle this demanding process. This study presents the Multisegment Overlap–Save Method (MS-OSM) to effectively address this problem. Our algorithm divides signal bands into separate short segments based on frequency component delays during dedispersion. By using the short segments shuffling technique with the Overlap-Save structure, MS-OSM can greatly reduce the FFT and inverse FFT points required down to fewer than 65,536 points. To evaluate the performance of MS-OSM, a synthetic pulsar signal is created and verified using standard software tools like DSPSR and PRESTO. The results show that MS-OSM maintains the same resolution while reducing execution time and resource usage. The validation of MS-OSM is taken by processing real pulsar observation data.

DISTRIBUTED HIGH-PERFORMANCE COMPUTING METHODS FOR ACCELERATING DEEP LEARNING TRAINING

This paper comprehensively analyzes distributed high-performance computing methods for accelerating deep learning training. We explore the evolution of distributed computing architectures, including data parallelism, model parallelism, and pipeline parallelism, and their hybrid implementations. The study delves into optimization techniques crucial for large-scale training, such as distributed optimization algorithms, gradient compression, and adaptive learning rate methods. We investigate communication-efficient algorithms, including Ring All Reduce variants and decentralized training approaches, which address the scalability challenges in distributed systems. The research examines hardware acceleration and specialized systems, focusing on GPU clusters, custom AI accelerators, high-performance interconnects, and distributed storage systems optimized for deep learning workloads. Finally, we discuss this field's challenges and future directions, including scalability-efficiency trade-offs, fault tolerance, energy efficiency in large-scale training, and emerging trends like federated learning and neuromorphic computing. Our findings highlight the synergy between advanced algorithms, specialized hardware, and optimized system designs in pushing the boundaries of large-scale deep learning, paving the way for future breakthroughs in artificial intelligence.

Design and implementation of a scalable correlator based on ROACH2 + GPU cluster for tianlai 96-dual-polarization antenna array

The digital correlator is one of the most crucial data processing components of a radio telescope array. With the scale of radio interferometeric array growing, many efforts have been devoted to developing a cost-effective and scalable correlator in the field of radio astronomy. In this paper, a 192-input digital correlator with six CASPER ROACH2 boards and seven GPU servers has been deployed as the digital signal processing system for Tianlai cylinder pathfinder located in Hongliuxia observatory. The correlator consists of 192 input signals (96 dual-polarization), 125-MHz bandwidth, and full-Stokes output. The correlator inherits the advantages of the CASPER system, for example, low cost, high performance, modular scalability, and a heterogeneous computing architecture. With a rapidly deployable ROACH2 digital sampling system, a commercially expandable 10 Gigabit switching network system, and a flexible upgradable GPU computing system, the correlator forms a low-cost and easily-upgradable system, poised to support scalable large-scale interferometeric array in the future.

UWLPIPE: Ultra-wide Bandwidth Low-frequency Pulsar Data Processing Pipeline

For real-time processing of ultra-wide bandwidth low-frequency pulsar baseband data, we designed and implemented an ultra-wide bandwidth low-frequency pulsar data processing pipeline (UWLPIPE) based on the shared ringbuffer and GPU parallel technology. UWLPIPE runs on the GPU cluster and can simultaneously receive multiple 128 MHz dual-polarization VDIF data packets preprocessed by the front-end FPGA. After aligning the dual-polarization data, multiple 128M subband data are packaged into PSRDADA baseband data or multi-channel coherent dispersion filterbank data, and multiple subband filterbank data can be spliced into wideband data after time alignment. We used the Nanshan 26 m radio telescope with the L-band receiver at 964 ∼ 1732 MHz to observe multiple pulsars. Finally, we processed the data using DSPSR software, and the results showed that each subband could correctly fold out the pulse profile, and the wideband pulse profile accumulated by multiple subbands could be correctly aligned.

PM2D: A parallel GPU-based code for the kinetic simulation of laser plasma instabilities at large scales

Laser plasma instabilities (LPIs) have significant influences on the laser energy deposition efficiency and therefore are important processes in inertial confined fusion (ICF). Numerical simulations play important roles in revealing the complex physics of LPIs. Since LPIs are typically a three wave coupling process, the precise simulations of LPIs with kinetic effects require to resolve the laser period (around one femtosecond) and laser wavelength (less than one micron). In the typical ICF experiments, however, LPIs are involved in a spatial scale of several millimeters and a temporal scale of several nanoseconds. Therefore, the precise kinetic simulations of LPIs in such scales require huge computational resources and are hard to be carried out by present kinetic codes like particle-in-cell (PIC) codes. In this paper, a full wave fluid model of LPIs is constructed and numerically solved by the particle-mesh method, where the plasma is described by macro particles that can move across the mesh grids freely. Based upon this model, a two-dimensional (2D) GPU code named PM2D is developed. The PM2D code can simulate the kinetic effects of LPIs self-consistently as normal PIC codes. Moreover, as the physical model adopted in the PM2D code is specifically constructed for LPIs, the required macro particles per grid in the simulations can be largely reduced and thus overall simulation cost is considerably reduced comparing with PIC codes. More importantly, the numerical noise in the PM2D code is much lower, which makes it more robust than PIC codes in the simulation of LPIs for the long-time scale above 10 picoseconds. After the distributed computing is realized, our PM2D code is able to run on GPU clusters with a total mesh grids up to several billions, which meets the requirements for the simulations of LPIs at ICF experimental scale with reasonable cost. Program summaryProgram Title: PM2DCPC Library link to program files:https://doi.org/10.17632/xscj6vnkkw.1Licensing provisions: GNU General Public License v3.0.Programming language: C++, CUDA.Nature of problem: Although the large scale simulations of laser plasma instabilities (LPIs) is of great significance for the inertial confinement fusion (ICF), there is still no suitable code to simulate these problems. PM2D code based on a GPU platform provides an effective method to simulate these large scale problems in ICF.Solution method: A fluid model for LPIs is established firstly, which contains wave equations that describe the laser propagating process, electron and ion fluid equations that describe the plasma motions, and a Poisson's equation that describes the electrostatic field induced by charge separation. The wave equation is solved on a rectangular region using absorption boundary conditions on all of four boundaries. The absorption boundary condition on the left boundary is further extended to allow the incidence of driven lasers and absorption of scattering lasers simultaneously. The fluid equations in the physical model are solved by the particle-mesh method, in which the macro particles are driven to move by fluid forces. Since macro particles can move freely within the fixed fluid grids, the PM2D code can capture the kinetic effects self-consistently. The Poisson's equation for the electrostatic field is solved by a Fourier decomposition method in the y direction, which helps to decrease the simulation cost greatly. The PM2D code is developed on a GPU platform base on CUDA toolkit, which largely increase the computational speed.

PerfTop: Towards performance prediction of distributed learning over general topology

Distributed learning with multiple GPUs has been widely adopted to accelerate the training process of large-scale deep neural networks. However, misconfiguration of the GPU clusters with various communication primitives and topologies could potentially diminish the gains in parallel computation and lead to significant degradation in training efficiency. Predicting the performance of distributed learning enables service providers to identify potential bottlenecks beforehand. In this work, we propose a Performance prediction framework over General Topologies, called PerfTop, for accurate estimation of per-iteration execution time. The main strategy is to integrate computation time prediction with an analytical model to map the nonlinearity in communication and fine-grained computation-communication patterns. This enables accurate prediction of a variety of neural network models over general topologies, such as tree, hierarchical, and exponential. Our extensive experiments show that PerfTop outperforms existing methods in estimating both computation and communication time, particularly for communication, surpassing the existing methods by over 45%. Meanwhile, it achieves an accuracy of above 85% in predicting the execution time over general topologies compared to simple topologies such as star and ring from the previous works.

Allok: a machine learning approach for efficient graph execution on CPU–GPU clusters

Allok: a machine learning approach for efficient graph execution on CPU–GPU clusters

PPS: Fair and efficient black-box scheduling for multi-tenant GPU clusters

Multi-tenant GPU clusters are common, where users purchase GPU quota to run their neural network training jobs. However, strict quota-based scheduling often leads to cluster under-utilization, while allowing quota groups to use excess GPUs improves utilization but results in fairness problems. We propose PPS, a probabilistic prediction based scheduler, which uses job history statistics to predict future cluster status for making good scheduling decisions. Different from existing schedulers that rely on deep learning frameworks to adjust bad scheduling decisions and/or require detailed job information, PPS treats jobs as black boxes in that PPS runs a job to completion without adjustment once scheduled and requires only aggregate job statistics. The black-box feature is favorable due to its good generality, compatibility and security, and made possible by the predictability of aggregate resource utilization statistics of large clusters. Extensive experiments show that PPS achieves high cluster utilization and good fairness simultaneously.

LBB: load-balanced batching for efficient distributed learning on heterogeneous GPU cluster

LBB: load-balanced batching for efficient distributed learning on heterogeneous GPU cluster

Large-Scale Molecular Dynamics Simulation Based on Heterogeneous Many-Core Architecture.

As the application of molecular dynamics (MD) simulations continues to evolve, the demand for accelerating large-scale simulation systems and handling of enormous simulation tasks is steadily increasing. We propose a parallel acceleration method for large-scale MD simulations based on Sunway heterogeneous many-core processors. This method integrates task scheduling, simulation calculations, and data storage, effectively tackling issues related to large-scale simulations and numerous simulation tasks. The task scheduling strategy flexibly handles tasks on various scales and enables parallel execution of multiple tasks. During the simulation calculations, we ported GROMACS to the Sunway architecture and accelerated the calculation of short-range forces through a heterogeneous processor. Our method achieves approximately 10-fold acceleration and 90% scalability when executing a single simulation task. When handling numerous simulation tasks, our method achieves parallel execution of all of the tasks with 90% scalability. By employing our method, we carried out 50 ns simulations on over 3000 distinct conotoxin structures individually within just 5 h. Additionally, we evaluated more than 200 protein-ligand complexes, and the simulation efficiency significantly exceeded that of midsized to small GPU clusters.

PSRDP: A Parallel Processing Method for Pulsar Baseband Data

To address the problem of real-time processing of ultra-wide bandwidth pulsar baseband data, we designed and implemented a pulsar baseband data processing algorithm (PSRDP) based on GPU parallel computing technology. PSRDP can perform operations such as baseband data unpacking, channel separation, coherent dedispersion, Stokes detection, phase and folding period prediction, and folding integration in GPU clusters. We tested the algorithm using the J0437-4715 pulsar baseband data generated by the CASPSR and Medusa backends of the Parkes, and the J0332+5434 pulsar baseband data generated by the self-developed backend of the NanShan Radio Telescope. We obtained the pulse profiles of each baseband data. Through experimental analysis, we have found that the pulse profiles generated by the PSRDP algorithm in this paper are essentially consistent with the processing results of Digital Signal Processing Software for Pulsar Astronomy (DSPSR), which verified the effectiveness of the PSRDP algorithm. Furthermore, using the same baseband data, we compared the processing speed of PSRDP with DSPSR, and the results showed that PSRDP was not slower than DSPSR in terms of speed. The theoretical and technical experience gained from the PSRDP algorithm research in this article lays a technical foundation for the real-time processing of QTT (Qi Tai radio Telescope) ultra-wide bandwidth pulsar baseband data.

Utilization-prediction-aware energy optimization approach for heterogeneous GPU clusters

Utilization-prediction-aware energy optimization approach for heterogeneous GPU clusters

MSHGN: Multi-scenario adaptive hierarchical spatial graph convolution network for GPU utilization prediction in heterogeneous GPU clusters

Accurately predicting GPU utilization is crucial for effectively managing heterogeneous GPU clusters, yet existing prediction methods are tailored to homogeneous clusters or ignore the unique characteristics of heterogeneous ones. To address this problem, we propose the Multi-Scenarios Adaptive Hierarchical Spatial Graph Convolution Network (MSHGN) model. This model leverages the hierarchical relationships among users, tasks, and machines to construct multiple scenarios' undirected graphs and uses Graph Convolution Neural (GCN) to capture the spatial dependency relationships between the computational resources consumed during execution. The MSHGN model accurately predicts the future GPU utilization rates of each machine in a heterogeneous GPU cluster, achieving an RMSE of only 0.0101, an MAE of 0.0072, and an MAPE of 0.0641. We evaluate the MSHGN model on a real-world Alibaba dataset collected from a heterogeneous GPU cluster and find that it achieves superior accuracy and robustness in predicting resource utilization, outperforming other baseline models.

OF-WFBP: A near-optimal communication mechanism for tensor fusion in distributed deep learning

The communication bottleneck has severely restricted the scalability of distributed deep learning. Tensor fusion improves the scalability of data parallelism by overlapping computation and communication tasks. However, existing tensor fusion schemes only result in suboptimal training performance. In this paper, we propose an efficient communication mechanism (OF-WFBP) to find the optimal tensor fusion scheme for synchronous data parallelism. We present the mathematical model of OF-WFBP and prove it is an NP-hard problem. We mathematically solve the mathematical model of OF-WFBP in two cases. We propose an improved sparrow search algorithm (GradSSA) to find the near-optimal tensor fusion scheme efficiently in other cases. Experimental results on two different GPU clusters show that OF-WFBP achieves up to 1.43x speedup compared to the state-of-the-art tensor fusion mechanisms.

Rise of Distributed Deep Learning Training in the Big Model Era: From a Software Engineering Perspective

Deep learning (DL) has become a key component of modern software. In the “ big model ” era, the rich features of DL-based software (i.e., DL software) substantially rely on powerful DL models, e.g., BERT, GPT-3, and the recently emerging GPT-4, which are trained on the powerful cloud with large datasets. Hence, training effective DL models has become a vital stage in the whole software lifecycle. When training deep learning models, especially those big models, developers need to parallelize and distribute the computation and memory resources amongst multiple devices (e.g., a cluster of GPUs) in the training process, which is known as distributed deep learning training , or distributed training for short. However, the unique challenges that developers encounter in distributed training process have not been studied in the software engineering community. Given the increasingly heavy dependence of current DL-based software on distributed training, this paper aims to fill in the knowledge gap and presents the first comprehensive study on developers’ issues in distributed training. To this end, we focus on popular DL frameworks that support distributed training (including TensorFlow, PyTorch, Keras, and Horovod) and analyze 1,131 real-world developers’ issues about using these frameworks reported on Stack Overflow and GitHub. We construct a fine-grained taxonomy consisting of 30 categories regarding the fault symptoms and summarize common fix patterns for different symptoms. We find that: (1) many distributed-specific faults and non-distributed-specific faults inherently share the same fault symptoms, making it challenging to debug; (2) most of the fault symptoms have frequent fix patterns; (3) about half of the faults are related to system-level configurations. Based on the results, we suggest actionable implications on research avenues that can potentially facilitate the distributed training to develop DL-based software, such as focusing on the frequent and common fix patterns when designing testing or debugging tools, developing efficient testing and debugging techniques for communication configuration along with the synthesis of network configuration analysis, designing new multi-device checkpoint-and-replay techniques to help reproduction, and designing serverless APIs for cloud platforms.

Interference-aware opportunistic job placement for shared distributed deep learning clusters

Distributed deep learning frameworks facilitate large deep learning workloads. These frameworks support sharing one GPU device among multiple jobs to improve resource utilization. Modern deep learning training jobs consume a large amount of GPU memory. Despite that, sharing GPU memory among jobs is still possible because a training job has iterative steps that its memory usage fluctuates over time. However, resource sharing also introduces the risk of job performance degradation. Co-located jobs sharing a GPU device may suffer from different levels of interference, mainly caused by memory oversharing. How to improve resource utilization while maintaining good job performance is a novel challenge for job placement strategies. This paper studies the job placement problem. We propose an opportunistic memory sharing model to describe the time-varying job memory requirements. Based on this model, we introduce an Opportunistic Job Placement Problem (OJPP) for shared GPU clusters that seek job placement configurations using a minimum number of GPU devices and guarantee user-defined performance requirements at the same time. We propose a greedy algorithm and a heuristic algorithm with computational complexities of O(nlog⁡n) and O(n2log⁡n), respectively, to solve the problem. We also propose an online adjustment algorithm with the computational complexity of O(nlog⁡n) to perform updates to job placement configurations in runtime. A machine-learning-based interference prediction method is used to prepare accurate interference estimations. Extensive experiments are conducted on a GPU cluster to verify the correctness and effectiveness of our algorithms. Compared with standalone training jobs on dedicated clusters, the proposed approach reduces resource consumption by 46% in a shared cluster, while guaranteeing over 92.97% of the job performance, in terms of average job completion time.