multi-GPU Environment Research Articles

Optimization of Large-Scale Sparse Matrix-Vector Multiplication on Multi-GPU Systems

Sparse matrix-vector multiplication (SpMV) is one of the important kernels of many iterative algorithms for solving sparse linear systems. The limited storage and computational resources of individual GPUs restrict both the scale and speed of SpMV computing in problem-solving. As real-world engineering problems continue to increase in complexity, the imperative for collaborative execution of iterative solving algorithms across multiple GPUs is increasingly apparent. Although the multi-GPU-based SpMV takes less kernel execution time, it also introduces additional data transmission overhead, which diminishes the performance gains derived from parallelization across multi-GPUs. Based on the non-zero elements distribution characteristics of sparse matrices and the trade-off between redundant computations and data transfer overhead, this paper introduces a series of SpMV optimization techniques tailored for multi-GPU environments and effectively enhances the execution efficiency of iterative algorithms on multiple GPUs. Firstly, we propose a two-level non-zero elements-based matrix partitioning method to increase the overlap of kernel execution and data transmission. Then, considering the irregular non-zero elements distribution in sparse matrices, a long-row-aware matrix partitioning method is proposed to hide more data transmissions. Finally, an optimization using redundant and inexpensive short-row execution to exchange costly data transmission is proposed. Our experimental evaluation demonstrates that, compared with the SpMV on a single GPU, the proposed method achieves an average speedup of 2.00x and 1.85x on platforms equipped with two RTX 3090 and two Tesla V100-SXM2, respectively. The average speedup of 2.65x is achieved on a platform equipped with four Tesla V100-SXM2.

URANOS-2.0: Improved performance, enhanced portability, and model extension towards exascale computing of high-speed engineering flows

We present URANOS-2.0, the second major release of our massively parallel, GPU-accelerated solver for compressible wall flow applications. This latest version represents a significant leap forward in our initial tool, which was launched in 2023 (De Vanna et al. [1]), and has been specifically optimized to take full advantage of the opportunities offered by the cutting-edge pre-exascale architectures available within the EuroHPC JU. In particular, URANOS-2.0 emphasizes portability and compatibility improvements with the two top-ranked supercomputing architectures in Europe: LUMI and Leonardo. These systems utilize different GPU architectures, AMD and NVIDIA, respectively, which necessitates extensive efforts to ensure seamless usability across their distinct structures. In pursuit of this objective, the current release adheres to the OpenACC standard. This choice not only facilitates efficient utilization of the full potential inherent in these extensive GPU-based architectures but also upholds the principles of vendor neutrality, a distinctive characteristic of URANOS solvers in the CFD solvers' panorama. However, the URANOS-2.0 version goes beyond the goals of improving usability and portability; it introduces performance enhancements and restructures the most demanding computational kernels. This translates into a 2× speedup over the same architecture. In addition to its enhanced single-GPU performance, the present solver release demonstrates very good scalability in multi-GPU environments. URANOS-2.0, in fact, achieves strong scaling efficiencies of over 80% across 64 compute nodes (256 GPUs) for both LUMI and Leonardo. Furthermore, its weak scaling efficiencies reach approximately 95% and 90% on LUMI and Leonardo, respectively, when up to 256 nodes (1024 GPUs) are considered. These significant performance advancements position URANOS-2.0 as a state-of-the-art supercomputing platform tailored for compressible wall turbulence applications, establishing the solver as an integrated tool for various aerospace and energy engineering applications, which can span from direct numerical simulations, wall-resolved large eddy simulations, up to most recent wall-modeled large eddy simulations. Program summaryProgram title: Unsteady Robust All-around Navier-StOkes Solver (URANOS)CPC Library link to program files:https://doi.org/10.17632/pw5hshn9k6.2Developer's repository link:https://github.com/uranos-gpu/uranos-gpu, https://github.com/uranos-gpu/uranos-gpu/tree/v2.0Licensing provisions: BSD License 2.0Programming language: Modern Fortran, OpenACC, MPINature of problem: Solving the compressible Navier-Stokes equations in a three-dimensional Cartesian framework.Solution method: Convective terms are treated with high-resolution shock-capturing schemes. The system dynamics is advanced in time with a three-stage Runge-Kutta method. Parallelization adopts MPI+OpenACC.

BGS: Accelerate GNN training on multiple GPUs

Emerging Graph Neural Networks (GNNs) have made significant progress in processing graph-structured data, yet existing GNN frameworks face scalability issues when training large-scale graph data using multiple GPUs. Frequent feature data transfers between CPUs and GPUs are a major bottleneck, and current caching schemes have not fully considered the characteristics of multi-GPU environments, leading to inefficient feature extraction. To address these challenges, we propose BGS, an auxiliary framework designed to accelerate GNN training from a data perspective in multi-GPU environments. Firstly, we introduce a novel training set partition algorithm, assigning independent training subsets to each GPU to enhance the spatial locality of node access, thus optimizing the efficiency of the feature caching strategy. Secondly, considering that GPUs can communicate at high speeds via NVLink connections, we designed a feature caching placement strategy suitable for multi-GPU environments. This strategy aims to improve the overall hit rate by setting reasonable redundant caches on each GPU. Evaluations on two representative GNN models, GCN and GraphSAGE, show that BGS significantly improves the hit rate of feature caching strategies in multi-GPU environments and substantially reduces the time overhead of data loading, achieving a performance improvement of 1.5 to 6.2 times compared to the baseline.

TLPGNN : A Lightweight Two-Level Parallelism Paradigm for Graph Neural Network Computation on Single and Multiple GPUs

Graph Neural Networks (GNNs) are an emerging class of deep learning models specifically designed for graph-structured data. They have been effectively employed in a variety of real-world applications, including recommendation systems, drug development, and analysis of social networks. The GNN computation includes regular neural network operations and general graph convolution operations, which take most of the total computation time. Though several recent works have been proposed to accelerate the computation for GNNs, they face the limitations of heavy pre-processing, low efficiency atomic operations, and unnecessary kernel launches. In this paper, we design TLPGNN , a lightweight two-level parallelism paradigm for GNN computation. First, we conduct a systematic analysis of the hardware resource usage of GNN workloads to understand the characteristics of GNN workloads deeply. With the insightful observations, we then divide the GNN computation into two levels, i.e., vertex parallelism for the first level and feature parallelism for the second. Next, we employ a novel hybrid dynamic workload assignment to address the imbalanced workload distribution. Furthermore, we fuse the kernels to reduce the number of kernel launches and cache the frequently accessed data into registers to avoid unnecessary memory traffic. To scale TLPGNN to multi-GPU environments, we propose an edge-aware row-wise 1-D partition method to ensure a balanced workload distribution across different GPU devices. Experimental results on various benchmark datasets demonstrate the superiority of our approach, achieving substantial performance improvement over state-of-the-art GNN computation systems, including Deep Graph Library (DGL), GNNAdvisor, and FeatGraph, with speedups of 6.1 ×, 7.7 ×, and 3.0 ×, respectively, on average. Evaluations of multiple-GPU TLPGNN also demonstrate that our solution achieves both linear scalability and a well-balanced workload distribution.

GPU acceleration of conjugate gradient method obtaining Green's function for transport-property calculation

We present a graphics processing unit (GPU) acceleration procedure for the transport-property calculation of the RSPACE code, which uses density functional theory and the real-space finite-difference method. Porting the RSPACE code to run on a GPU using Open Accelerator and libraries results in the improvement of the computational speed and efficiency compared with a central processing unit. Furthermore, we achieved a speed up of ∼ 4x compared with a single GPU and effective processing in a multi-GPU environment by parallelizing in a way that minimizes communication and synchronization. Our GPU-acceleration procedure is demonstrated by the calculation of electron-transport properties of (4,4) carbon nanotubes (CNTs) with a 5-8-5 defect for which the calculations are suitable for parallel processing on GPUs. It is found that the information loss is strongly affected by the defect geometry when electron waves propagating in CNTs are utilized as the information transfer signal.

PaScaL_TDMA 2.0: A multi-GPU-based library for solving massive tridiagonal systems

PaScaL_TDMA 2.0: A multi-GPU-based library for solving massive tridiagonal systems

Parallel graph coloring algorithms for distributed GPU environments

Graph coloring is often used in parallelizing scientific computations that run in distributed and multi-GPU environments; it identifies sets of independent data that can be updated in parallel. Many algorithms exist for graph coloring on a single GPU or in distributed memory, but to the best of our knowledge, hybrid MPI+GPU algorithms have been unexplored until this work. We present several MPI+GPU coloring approaches based on the distributed coloring algorithms of Gebremedhin et al. and the shared-memory algorithms of Deveci et al. The on-node parallel coloring uses implementations in KokkosKernels, which provide parallelization for both multicore CPUs and GPUs. We further extend our approaches to compute distance-2 and partial distance-2 colorings, giving the first known distributed, multi-GPU algorithm for these problems. In addition, we propose a novel heuristic to reduce communication for recoloring in distributed graph coloring. Our experiments show that our approaches operate efficiently on inputs too large to fit on a single GPU and scale up to graphs with 76.7 billion edges running on 128 GPUs.

An efficient GPU implementation and scaling for higher-order 3D stencils

Stencil computation patterns are the backbone of many scientific and engineering simulations. The stencil computation is known to be constrained by its high demand of memory bandwidth, which limits performance on accelerators such as GPUs. Prior GPU-based approaches concentrated on stencils with only axis-aligned grid points with 2D caching schemes. However, for stencils with non-axis grid points, prior approaches use 3D caching or multi-pass 2D caching schemes. These methods suffer from either large number of global memory accesses or required large size of the shared memory. In this work, we present an efficient GPU implementation scheme “Scatter Without Write Conflict” (SWiC) for large advanced 3D stencil patterns involving non-axis-aligned grid points. Unlike other 3D caching schemes, SWiC only needs 2D caching and a single pass over the stencil per iteration. SWiC achieves, significant reductions in the global memory accesses, without increasing the size of shared memory. SWiC can also be applied to simple axis-aligned stencils without any performance loss. Moreover, we propose a scalable implementation for the halo region exchange of 3D stencils on multi-GPU nodes. For evaluation, we test SWiC on three Nvidia GPU generations and show that our approach significantly outperforms existing state-of-the-art GPU implementations with a speedup ranging from 1.6× to 5.75×. We also provide a detailed scaling analysis in multi-node and multi-GPU environments.

MDScale: Scalable multi-GPU bonded and short-range molecular dynamics

GPUs have enabled a drastic change to computing environments, making massively parallel computing possible. Molecular dynamics is a perfect candidate problem for massively parallel computing, but to date it has not taken full advantage of multi-GPU environments due to the difficulty of partitioning molecular dynamics problems and exchanging problem data among compute nodes. These difficulties restrict the use of GPUs to only some of the computations in a full molecular dynamics problem, and hence prevent scalability beyond just a few GPUs. This work presents a scalable parallelization solution for the bonded and short-range forces present in a molecular dynamics problem. Together with existing solutions for long-range forces, it enables highly scalable, parallel molecular dynamics on multi-GPU computing environments. Specifically, the proposed solution divides the molecular volume into independent parts assigned to different GPUs, but it maintains a global bond structure that is efficiently exchanged when atoms move across GPUs. We demonstrate close-to-linear speedup of the proposed solution, simulating the dynamics of gigamolecules with 1 billion atoms on a computing environment with 96 GPUs, and obtaining superior performance to the well known molecular dynamics simulator NAMD.

NVIDIA IndeX accelerated computing for visualizing Cholla's galactic winds

Galactic winds – outflows of gas driven out of galaxies by the combined effects of thousands of supernovae – are a crucial feature of galaxy evolution. Despite their importance, a complete theoretical picture of these winds has been elusive. Simulating the complicated interaction between the hot, high pressure gas created by supernovae and the cooler, high density gas in the galaxy disk requires massive computational resources and sophisticated software. For this purpose, Computational Hydrodynamics On Parallel Architectures (Cholla) has been demonstrated to be a scalable and efficient tool that operates in large, distributed multi-GPU environments at high levels of performance. This additional performance results in unprecedented resolution for this type of simulation and massive amounts of generated data. This raises the requirements for analysis tools that can cope with scale and complexity of the simulated physical processes. To address those challenges, we utilize NVIDIA IndeX as a scalable framework to visualize the simulation output: NVIDIA IndeX features a streaming-based architecture to interactively explore simulation results in large-scale, multi-GPU environments. We utilize customized sampling programs for multi-volume and surface rendering to address analysis questions of galactic wind simulations. This combination of massively parallel simulation and analysis allows us to utilize recent supercomputer capabilities and to speed up the exploration of galactic wind simulations.

A Web-Based Tool for Simulating Molecular Dynamics in Cloud Environments

Molecular dynamics simulations take advantage of supercomputing environments, e.g., to solve molecular systems composed of millions of atoms. Supercomputers are increasing their computing and memory power while they are becoming more complex with the introduction of Multi-GPU environments. Despite these capabilities, the molecular dynamics simulation is not an easy process. It requires properly preparing the simulation data and configuring the entire operation, e.g., installing and managing specific software packages to take advantage of the potential of Multi-GPU supercomputers. We propose a web-based tool that facilitates the management of molecular dynamics workflows to be used in combination with a multi-GPU cloud environment. The tool allows users to perform data pipeline and run the simulation in a cloud environment, even for those who are not specialized in the development of molecular dynamics simulators or cloud management.

Efficient Join Algorithms For Large Database Tables in a Multi-GPU Environment.

Relational join processing is one of the core functionalities in database management systems. It has been demonstrated that GPUs as a general-purpose parallel computing platform is very promising in processing relational joins. However, join algorithms often need to handle very large input data, which is an issue that was not sufficiently addressed in existing work. Besides, as more and more desktop and workstation platforms support multi-GPU environment, the combined computing capability of multiple GPUs can easily achieve that of a computing cluster. It is worth exploring how join processing would benefit from the adaptation of multiple GPUs. We identify the low rate and complex patterns of data transfer among the CPU and GPUs as the main challenges in designing efficient algorithms for large table joins. To overcome such challenges, we propose three distinctive designs of multi-GPU join algorithms, namely, the nested loop, global sort-merge and hybrid joins for large table joins with different join conditions. Extensive experiments running on multiple databases and two different hardware configurations demonstrate high scalability of our algorithms over data size and significant performance boost brought by the use of multiple GPUs. Furthermore, our algorithms achieve much better performance as compared to existing join algorithms, with a speedup up to 25X and 2.8X over best known code developed for multi-core CPUs and GPUs respectively.

Explicit incompressible smoothed particle hydrodynamics in a multi-GPU environment for large-scale simulations

We present an explicit incompressible smoothed particle hydrodynamics formulation with stabilized pressure distribution and its implementation in a multiple graphics processing unit environment. The pressure Poisson equation is stabilized via both pressure invariance and divergence-free conditions, and its explicit formulation is derived using the first step of the Jacobi iterative solver. Also, we show how to adapt the fixed wall ghost particle for the boundary condition into our explicit approach. Verification and validation of the method include hydrostatic and dam break numerical tests. The computational performance in the multi-GPU environment was notably high with reasonable speedup values compared to our single-GPU implementation. In particular, our code allows simulations with very large number of particles reaching up to 200 million per GPU card. Finally, to illustrate the potential of our formulation in simulating natural disasters, we present a simulation of the famous Fukushima Dai-ichi Power Plant inundation by the tsunami from The Great East Japan Earthquake in 2011, in Japan.

G-PICS: A Framework for GPU-Based Spatial Indexing and Query Processing

Support for efficient spatial data storage and retrieval has become a vital component in almost all spatial database systems. While GPUs have become a mainstream platform for high-throughput data processing in recent years, exploiting the massively parallel processing power of GPUs is non-trivial. Current approaches that parallelize one query at a time have low work efficiency and cannot make good use of GPU resources. On the other hand, many spatial database systems could receive a large number of queries simultaneously. In this paper, we present a comprehensive framework named G-PICS for parallel processing of concurrent spatial queries on GPUs. G-PICS encapsulates efficient parallel algorithms for constructing a variety of spatial trees with different space partitioning methods. G-PICS also provides highly optimized programs for processing major spatial query types, and such programs can be accessed via an API that could be further extended to implement user-defined algorithms. While support for dynamic data inputs is missing in existing work, G-PICS implements efficient parallel algorithms for bulk updates of data. Furthermore, G-PICS is designed to work in a Multi-GPU environment to support datasets beyond the size of a single GPU’s global memory. Empirical evaluation of G-PICS shows significant performance improvement over the state-of-the-art GPU and parallel CPU-based spatial query processing systems. In particular, G-PICS achieves double-digit speedup over such systems in tree construction (up to 53X) and query processing (up to 80X).

Cytokit: a single-cell analysis toolkit for high dimensional fluorescent microscopy imaging

BackgroundMultiplexed in-situ fluorescent imaging offers several advantages over single-cell assays that do not preserve the spatial characteristics of biological samples. This spatial information, in addition to morphological properties and extensive intracellular or surface marker profiling, comprise promising avenues for rapid advancements in the understanding of disease progression and diagnosis. As protocols for conducting such imaging experiments continue to improve, it is the intent of this study to provide and validate software for processing the large quantity of associated data in kind.ResultsCytokit offers (i) an end-to-end, GPU-accelerated image processing pipeline; (ii) efficient input/output (I/O) strategies for operations specific to high dimensional microscopy; and (iii) an interactive user interface for cross filtering of spatial, graphical, expression, and morphological cell properties within the 100+ GB image datasets common to multiplexed immunofluorescence. Image processing operations supported in Cytokit are generally sourced from existing deep learning models or are at least in part adapted from open source packages to run in a single or multi-GPU environment. The efficacy of these operations is demonstrated through several imaging experiments that pair Cytokit results with those from an independent but comparable assay. A further validation also demonstrates that previously published results can be reproduced from a publicly available multiplexed image dataset.ConclusionCytokit is a collection of open source tools for quantifying and analyzing properties of individual cells in large fluorescent microscopy datasets that are often, but not necessarily, generated from multiplexed antibody labeling protocols over many fields of view or time periods. This project is best suited to bioinformaticians or other technical users that wish to analyze such data in a batch-oriented, high-throughput setting. All source code, documentation, and data generated for this article are available under the Apache License 2.0 at https://github.com/hammerlab/cytokit.

Comprehensive techniques of multi-GPU memory optimization for deep learning acceleration

This paper presents a comprehensive suite of techniques for optimized memory management in multi-GPU systems to accelerate deep learning application execution. We employ a hybrid utilization of GPU and CPU memories in a multi-GPU environment by effectively addressing contention issues in the shared interconnect (e.g., PCIe, NVLink). In addition, we designed and implemented an intelligent prefetching algorithm (from CPU memory to GPU) that achieves the highest processing throughput while sustaining a large mini-batch size. We successfully implemented our optimization techniques on TensorFlow, and performed extensive experiments in various multi-GPU environments including traditional PCIe and the latest high-bandwidth interconnect, NVLink. Evaluation results show that our proposed scheme actually improves computing performance by decreasing the I/O bottleneck, and effectively increasing the mini-batch size without sacrificing overall training throughput.

Efficient model of tumor dynamics simulated in multi-GPU environment

The application of computer simulation as a tool in predicting cancer dynamics (e.g. during anticancer therapy) requires tumor models, which are nontrivial and, simultaneously, not computationally demanding. To this end, both the level of details and computational efficiency of the model should be well balanced. The restrictions on computational time are forced by very demanding data assimilation process in the phase of parameters learning and their correction on the basis of incoming medical data. Herein we present a very efficient multi-GPU/CUDA implementation of three-dimensional (3-D) cancer model which allows for simulating both the growth and treatment phases of tumor dynamics. We demonstrate that the interaction between the tissue and the discrete network of blood vessels is a crucial component, which influences considerably the simulation time. Here we present a new solution which eliminates this flaw. We show also that the efficiency of our model does not depend on the complexity of tumor setup. As an example, we confront the growth of tumor in a simple and homogeneous environment with melanoma evolution, which proliferates in a complex environment of human skin. Consequently, the 3-D simulation of a tumor growth up to 1 cm in diameter requires an hour of computations on a midrange multi-GPU server.

Heterogeneous and unconventional cluster architectures and applications

Heterogeneous and unconventional cluster architectures and applications

Scalable non-negative matrix tri-factorization

BackgroundMatrix factorization is a well established pattern discovery tool that has seen numerous applications in biomedical data analytics, such as gene expression co-clustering, patient stratification, and gene-disease association mining. Matrix factorization learns a latent data model that takes a data matrix and transforms it into a latent feature space enabling generalization, noise removal and feature discovery. However, factorization algorithms are numerically intensive, and hence there is a pressing challenge to scale current algorithms to work with large datasets. Our focus in this paper is matrix tri-factorization, a popular method that is not limited by the assumption of standard matrix factorization about data residing in one latent space. Matrix tri-factorization solves this by inferring a separate latent space for each dimension in a data matrix, and a latent mapping of interactions between the inferred spaces, making the approach particularly suitable for biomedical data mining.ResultsWe developed a block-wise approach for latent factor learning in matrix tri-factorization. The approach partitions a data matrix into disjoint submatrices that are treated independently and fed into a parallel factorization system. An appealing property of the proposed approach is its mathematical equivalence with serial matrix tri-factorization. In a study on large biomedical datasets we show that our approach scales well on multi-processor and multi-GPU architectures. On a four-GPU system we demonstrate that our approach can be more than 100-times faster than its single-processor counterpart.ConclusionsA general approach for scaling non-negative matrix tri-factorization is proposed. The approach is especially useful parallel matrix factorization implemented in a multi-GPU environment. We expect the new approach will be useful in emerging procedures for latent factor analysis, notably for data integration, where many large data matrices need to be collectively factorized.

Simulating electron wave dynamics in graphene superlattices exploiting parallel processing advantages

This work introduces a parallel computing framework to characterize the propagation of electron waves in graphene-based nanostructures. The electron wave dynamics is modeled using both “microscopic” and effective medium formalisms and the numerical solution of the two-dimensional massless Dirac equation is determined using a Finite-Difference Time-Domain scheme. The propagation of electron waves in graphene superlattices with localized scattering centers is studied, and the role of the symmetry of the microscopic potential in the electron velocity is discussed. The computational methodologies target the parallel capabilities of heterogeneous multi-core CPU and multi-GPU environments and are built with the OpenCL parallel programming framework which provides a portable, vendor agnostic and high throughput-performance solution. The proposed heterogeneous multi-GPU implementation achieves speedup ratios up to 75x when compared to multi-thread and multi-core CPU execution, reducing simulation times from several hours to a couple of minutes. Program summaryProgram title: GslSim.Program Files doi:http://dx.doi.org/10.17632/prmfv63nj6.1Licensing provisions: GPLv3.Programming language: C, OpenCL and Matlab for results analysis.Nature of problem: Computing the time evolution of electron waves in graphene superlattices is a time consuming process due to the high number of necessary nodes to discretize the spatial and time domains.Solution method: We develop a simulator based on the C/OpenCL standards to study the time evolution of electron waves in graphene superlattices by exploiting hardware architectures such as graphics processing units (GPUs) to speedup the computation of the pseudospinor.