GPU-accelerated Simulation Research Articles

Particle-based modeling and GPU-accelerated simulation of cellular blood flow

Computational modeling and simulation of cellular blood flow is highly desired for understanding blood microcirculation and blood-related diseases such as thrombosis and tumor, but it remains a challenging task primarily because blood in microvessels should be described as a dense suspension of different types of deformable cells. The focus of the present work is on the development of a particle-based and GPU-accelerated numerical method that is able to quickly simulate the various behaviors of deformable cells in three-dimensional arbitrarily complex geometries. We employ a two-fluid model to describe blood flow, incorporating the deformation and aggregation of cells. A smoothed dissipative particle dynamics is used to solve the two-fluid model, and a discrete microstructure model is applied for the cell deformation, as well as a Morse potential model for the cell aggregation. The heterogeneous CPU-GPU environment is established, where each GPU thread is dedicated to a particle, and the CPU is mainly responsible for loading and exporting data. Five test cases are conducted against analytical theory, experimental data, and previous numerical results, for pure fluid, cell deformation, cell aggregation, cell suspension and the cellular flow in a complex network, respectively. It is shown that the methodology can accurately predict various behaviors of cells, and the GPU is well suited for particle-based modeling. Especially for cellular blood flow, where calculating cellular forces is a compute-intensive and time-consuming task, the GPU offers exceptional parallel capabilities, significantly enhancing the simulation efficiency. The speedup is about 3.5 times faster than the CPU parallelization with 96 cores for the pure fluid, and this acceleration nearly reaches 20 times when cells are included in the simulations. Particularly, the calculations for deformation and aggregation forces demonstrate a substantial speedup, achieving the improvements of up to 120 and 640 times, respectively, compared to their serial counterparts. The present methodology can effectively integrate various behaviors of cells, and has the potential in simulating very large microvascular networks at organ levels.

Energy- and Angle-Resolved Scattering of Ne from Dodecane Liquid Surfaces: Theory Corroborating Experiment.

Motivated by recent experimental work by the Neumark group, we present here an all-atom molecular dynamics study of Ne scattering from a dodecane liquid surface with the objective of elucidating the fundamental aspects of gas-liquid dynamics. Using a fine-tuned force field, the GPU-accelerated simulations reproduced semiquantitatively the energy- and angle-resolved experimental results. The branching ratio between the impulsive scattering (IS) and thermal desorption (TD) channels exhibits a clear correlation with the incidence energy (Ei) and angle. Ne atoms with lower Ei values are more likely to be trapped, yielding an increased TD ratio. For a given Ei, a large incidence angle led to a higher IS ratio. The energy transfer between Ne atoms and liquid dodecane was found to be more sensitive to the deflection angle than to the incidence or reflection angle. With an increasing deflection angle, the fractional energy loss increases, suggesting that more kinetic energy is transferred to the liquid.

Method for scalable and performant GPU-accelerated simulation of multiphase compressible flow

Multiphase compressible flows are often characterized by a broad range of space and time scales, entailing large grids and small time steps. Simulations of these flows on CPU-based clusters can thus take several wall-clock days. Offloading the compute kernels to GPUs appears attractive but is memory-bound for many finite-volume and -difference methods, damping speedups. Even when realized, GPU-based kernels lead to more intrusive communication and I/O times owing to lower computation costs. We present a strategy for GPU acceleration of multiphase compressible flow solvers that addresses these challenges and obtains large speedups at scale. We use OpenACC for directive-based offloading of all compute kernels while maintaining low-level control when needed. An established Fortran preprocessor and metaprogramming tool, Fypp, enables otherwise hidden compile-time optimizations. This strategy exposes compile-time optimizations and high memory reuse while retaining readable, maintainable, and compact code. Remote direct memory access realized via CUDA-aware MPI and GPUDirect reduces halo-exchange communication time. We implement this approach in the open-source solver MFC [1]. Metaprogramming results in an 8-times speedup of the most expensive kernels compared to a statically compiled program, reaching 46% of peak FLOPs on modern NVIDIA GPUs and high arithmetic intensity (about 10 FLOPs/byte). In representative simulations, a single NVIDIA A100 GPU is 7-times faster compared to an Intel Xeon Cascade Lake (6248) CPU die, or about 300-times faster compared to a single such CPU core. At the same time, near-ideal (97%) weak scaling is observed for at least 13824 GPUs on OLCF Summit. A strong scaling efficiency of 84% is retained for an 8-times increase in GPU count. Collective I/O, implemented via MPI3, helps ensure the negligible contribution of data transfers (<1% of the wall time for a typical, large simulation). Large many-GPU simulations of compressible (solid-)liquid-gas flows demonstrate the practical utility of this strategy.

Two-phase approach to modeling the grain-fluid flows with deposition and entrainment over rugged topography

We present a grain-fluid mixture for debris flows moving on a rugged (non-trivial) topography, where entrainment and deposition may take place. The model equations are derived with respect to a terrain-following coordinate system, which is constructed based on the topographic surface. The coordinates are fixed in space, and a “sub-topography” is added on the coordinate surface to account for the variation in the local topography when entrainment or deposition takes place. Numerical implementation is made based on a GPU-accelerated simulation tool, into which the entrainment-deposition mechanism is integrated accordingly. Two numerical examples are assigned to investigate the key features of the proposed model. One is on a horizontal plane, on which a finite mass of grain-fluid mixture is released from the state of rest. In this example, debris flow deposits significantly impact the post-event morphology and the associated flow behaviors. The other concerns a moving mass down an inclined chute merging into a horizontal deposition plane, where the levee formation is reproduced. At the end, the model is validated against grain-water experiments and applied to a large-scale historical event to evaluate its applicability.

A GPU-accelerated simulator for the DEM analysis of granular systems composed of clump-shaped elements

A GPU-accelerated simulator for the DEM analysis of granular systems composed of clump-shaped elements

A comparison of numerical schemes for the GPU-accelerated simulation of variably-saturated groundwater flow

The optimal strategy to numerically solve the Richards equation is problem-dependent, especially on GPUs because an efficient numerical scheme on CPU might not scale well on GPU. In this work, four numerical schemes are coded to investigate their parallel performance on CPU and GPU. The results indicate that the scaling of Richards solvers on GPU is affected by the numerical scheme, the linear system solver, the soil constitutive model, the code structure, the problem size and the adaptive time stepping strategies. Compared with CPU, parallel simulations on GPU exhibit stronger variance in the scaling of different code sections. The poorly-scaled components could deteriorate the overall scaling. Under all circumstances, using a GPU significantly enhances computational speed, especially for large problems. Clearly, GPU computing have significant potential in accelerating large-scale hydrological simulations, but care must be taken on the design and implementation of the model structure.

GPU Rasterization-Based 3D LiDAR Simulation for Deep Learning.

High-quality data are of utmost importance for any deep-learning application. However, acquiring such data and their annotation is challenging. This paper presents a GPU-accelerated simulator that enables the generation of high-quality, perfectly labelled data for any Time-of-Flight sensor, including LiDAR. Our approach optimally exploits the 3D graphics pipeline of the GPU, significantly decreasing data generation time while preserving compatibility with all real-time rendering engines. The presented algorithms are generic and allow users to perfectly mimic the unique sampling pattern of any such sensor. To validate our simulator, two neural networks are trained for denoising and semantic segmentation. To bridge the gap between reality and simulation, a novel loss function is introduced that requires only a small set of partially annotated real data. It enables the learning of classes for which no labels are provided in the real data, hence dramatically reducing annotation efforts. With this work, we hope to provide means for alleviating the data acquisition problem that is pertinent to deep-learning applications.

High-Precision GPU-Accelerated Simulation Algorithm for Targets under Non-Uniform Cluttered Backgrounds

This article presents a high-precision airborne video synthetic aperture radar (SAR) raw echo simulation method aimed at addressing the issue of simulation accuracy in video SAR image generation. The proposed method employs separate techniques for simulating targets and ground clutter, utilizing pre-existing SAR images for clutter simulation and employing the shooting and bouncing rays (SBR) approach to generate target echoes. Additionally, the method accounts for target-generated shadows to enhance the realism of the simulation results. The fast simulation algorithm is implemented using the C++ programming language and the Accelerated Massive Parallelism (AMP) framework, providing a fusion technique for integrating clutter and target simulations. By combining the two types of simulated data to form the final SAR image, the method achieves efficient and accurate simulation technology. Experimental results demonstrate that this method not only improves computational speed but also ensures the accuracy and stability of the simulation outcomes. This research holds significant implications for the development of algorithms pertaining to video SAR target detection and tracking, providing robust support for practical applications.

Proactive project scheduling using GPU-accelerated simulations under uncertainty environment

ABSTRACT Numerous scheduling generation and revision methods have been developed for project scheduling under uncertain environments in previous research. However, most of these methods have yet to address a practical project with more than a hundred activities involving multiple decisions in scheduling generation and execution phases. This study proposes a local search-based scheduling method that comprehensively evaluates a baseline schedule considering decision-making in both the planning and execution phases. The proposed method performs simulations to evaluate schedule robustness accurately using GPU to find a locally optimal solution for large problem instances in a reasonable time. A series of numerical experiments demonstrate that the proposed method can generate a robust schedule for large-scale project instances. These results conclude that the proposed method utilizing the simulation-based evaluation and the GPU acceleration is effective and provide insight into developing a scheduling method for practical projects.

Suppression of the inclination instability in the trans-Neptunian Solar system

ABSTRACT The trans-Neptunian scattered disc exhibits unexpected dynamical structure, ranging from an extended dispersion of perihelion distance to a clustered distribution in orbital angles. Self-gravitational modulation of the scattered disc has been suggested in the literature as an alternative mechanism to Planet nine for sculpting the orbital architecture of the trans-Neptunian region. The numerics of this hypothesis have hitherto been limited to N &amp;lt; O(103) superparticle simulations that omit direct gravitational perturbations from the giant planets and instead model them as an orbit-averaged (quadrupolar) potential, through an enhanced J2 moment of the central body. For sufficiently massive discs, such simulations reveal the onset of collective dynamical behaviour – termed the ‘inclination instability’ – wherein orbital circularisation occurs at the expense of coherent excitation of the inclination. Here, we report N = O(104) GPU-accelerated simulations of a self-gravitating scattered disc (across a range of disc masses spanning 5–40 M⊕) that self-consistently account for intraparticle interactions as well as Neptune’s perturbations. Our numerical experiments show that even under the most favourable conditions, the inclination instability never ensues. Instead, due to scattering, the disc depletes. While our calculations show that a transient lopsided structure can emerge within the first few hundreds of Myr, the terminal outcomes of these calculations systematically reveal a scattered disc that is free of any orbital clustering. We conclude thus that the inclination instability mechanism is an inadequate explanation of the observed architecture of the Solar system.

TORCWA: GPU-accelerated Fourier modal method and gradient-based optimization for metasurface design

TORCWA is an electromagnetic wave simulation and optimization tool utilizing rigorous coupled-wave analysis. One of the advantages of TORCWA is that it provides GPU-accelerated simulation. It shows a greatly accelerated simulation speed compared to when the same simulation is performed on a CPU-based. Although it has accelerated speed, the simulation results are almost identical to the commercialized electromagnetic wave simulations. The second advantage is that it provides GPU-accelerated gradient calculation for the simulation results with reverse-mode automatic differentiation of PyTorch version 1.10.1. In particular, the instability of gradient calculation of eigendecomposition is also improved. With this property, TORCWA can be utilized for the optimization of various nanophotonic devices. Here, we first introduce the formulation used in TORCWA, compare it with other commercial simulations, and show the computational performance in multiple environments. Then, the gradient calculation and optimization examples are shown. Thanks to accelerated computational performance and gradient calculation, TORCWA is a worthy program for designing and optimizing various nanophotonic devices. Program summaryProgram title: TORCWACPC Library link to program files:https://doi.org/10.17632/2dybvpk42g.1Developer's repository link:https://github.com/kch3782/torcwaLicensing provisions: LGPL v3Programming language: Python 3.8Nature of Problem: Time-harmonic electromagnetic fields.Solution method: Fourier modal methods and extended scattering matrix method.Additional comments including restrictions and unusual features: TORCWA utilizes PyTorch version 1.10.1 [1].

GPU-accelerated simulations of quantum annealing and the quantum approximate optimization algorithm

We study large-scale applications using a GPU-accelerated version of the massively parallel Jülich universal quantum computer simulator (JUQCS–G). First, we benchmark JUWELS Booster, a GPU cluster with 3744 NVIDIA A100 Tensor Core GPUs. Then, we use JUQCS–G to study the relation between quantum annealing (QA) and the quantum approximate optimization algorithm (QAOA). We find that a very coarsely discretized version of QA, termed approximate quantum annealing (AQA), performs surprisingly well in comparison to the QAOA. It can either be used to initialize the QAOA, or to avoid the costly optimization procedure altogether. Furthermore, we study the scaling of the success probability when using AQA for problems with 30 to 40 qubits. We find that the case with the largest discretization error scales most favorably, surpassing the best result obtained from the QAOA.

MetaFEM: A generic FEM solver by meta-expressions

Current multi-physics Finite Element Method (FEM) solvers are complex systems in terms of both their mathematical complexity and lines of code. This paper proposes a skeleton generic FEM solver, named MetaFEM, in total about 6,000 lines of Julia code, which translates generic input Partial Differential Equation (PDE) weak forms into corresponding GPU-accelerated simulations with a grammar similar to FEniCS or FreeFEM. Two novel approaches differentiate MetaFEM from the common solvers: (1) the FEM kernel is based on an original theory/algorithm which explicitly processes meta-expressions, as the name suggests, and (2) the symbolic engine is a rule-based Computer Algebra System (CAS), i.e., the equations are rewritten/derived according to a set of rewriting rules instead of going through completely fixed routines, supporting easy customization by developers. Example cases in thermal conduction, linear elasticity and incompressible flow are presented to demonstrate utility.

Accelerating agent-based demand-responsive transport simulations with GPUs

A novel GPU-accelerated simulation model of large-scale fleet deployment, which can run country-wide, multi-modal scenarios with millions of agents and fleets of tens of thousands of vehicles within a couple of minutes, is presented. Multiple scenarios of the deployment of fleets of automated vehicles in Switzerland’s largest city, Zurich, are assessed. The simulations include the whole population of Switzerland (3.5 million car owners and 1.7 million public transit users) with their detailed travel demand, the road network (1.1 million links and 0.5 million intersections), and public transit (30 000 stops and 20 000 routes). It is demonstrated that in Zurich one automated vehicle could replace 7–8 private cars with an average increase in the road travel time of 44% and with wait times in the range of 10–15 min, provided travel demand remains constant. Furthermore, for the same fleet size, this novel accelerated simulation model runs up to 9 times faster compared to existing state-of-the-art tools.

Single-framework simulations of acoustic-wave–bubble cloud–stone interactions

Understanding the interaction of acoustic waves and stones is of principal importance to the development and optimization of shock wave and ultrasound-based therapies, such as shock wave and burst wave lithotripsy (SWL and BWL). Simulating these interactions is useful in furthering our understanding, but the development of simulation tools is complex. These tools often use separate numerical frameworks for compressible fluids and solids. Introducing cavitating bubble clouds which can occur during these therapies further complicates the problem, introducing a broad range of relevant spatio-temporal scales. To tackle these challenges, we have developed a single-framework code capable of simulating all relevant physics. Cavitating bubble clouds are introduced via a sub-grid model, and a hypoelastic model simulates solid materials, all in the same Eulerian framework. These are implemented in our open-source Multi-component Flow Code (MFC) [Bryngelson et al., Comp. Phys. Commun. (2020)]. We demonstrate the solver's capabilities for 3D SWL and BWL simulations, where we can obtain the stress state in the stone for various configurations. GPU-accelerated simulations also elucidate the computational advantages of the single-framework approach. [This research was supported by the National Institutes of Health (NIH Grant No. 2P01-DK043881) and the Office of Naval Research (ONR Grant No. N0014-18-1-2625).]

Interaction of Large- and Small-scale Dynamos in Isotropic Turbulent Flows from GPU-accelerated Simulations

Abstract Magnetohydrodynamical (MHD) dynamos emerge in many different astrophysical situations where turbulence is present, but the interaction between large-scale dynamos (LSDs) and small-scale dynamos (SSDs) is not fully understood. We performed a systematic study of turbulent dynamos driven by isotropic forcing in isothermal MHD with magnetic Prandtl number of unity, focusing on the exponential growth stage. Both helical and nonhelical forcing was employed to separate the effects of LSD and SSD in a periodic domain. Reynolds numbers ( ) up to ≈250 were examined and multiple resolutions used for convergence checks. We ran our simulations with the Astaroth code, designed to accelerate 3D stencil computations on graphics processing units (GPUs) and to employ multiple GPUs with peer-to-peer communication. We observed a speedup of ≈35 in single-node performance compared to the widely used multi-CPU MHD solver Pencil Code. We estimated the growth rates from both the averaged magnetic fields and their power spectra. At low LSD growth dominates, but at high SSD appears to dominate in both helically and nonhelically forced cases. Pure SSD growth rates follow a logarithmic scaling as a function of . Probability density functions of the magnetic field from the growth stage exhibit SSD behavior in helically forced cases even at intermediate . We estimated mean field turbulence transport coefficients using closures like the second-order correlation approximation (SOCA). They yield growth rates similar to the directly measured ones and provide evidence of α quenching. Our results are consistent with the SSD inhibiting the growth of the LSD at moderate , while the dynamo growth is enhanced at higher .

Supercomputing Pipelines Search for Therapeutics Against COVID-19

The urgent search for drugs to combat SARS-CoV-2 has included the use of supercomputers. The use of general-purpose graphical processing units (GPUs), massive parallelism, and new software for high-performance computing (HPC) has allowed researchers to search the vast chemical space of potential drugs faster than ever before. We developed a new drug discovery pipeline using the Summit supercomputer at Oak Ridge National Laboratory to help pioneer this effort, with new platforms that incorporate GPU-accelerated simulation and allow for the virtual screening of billions of potential drug compounds in days compared to weeks or months for their ability to inhibit SARS-COV-2 proteins. This effort will accelerate the process of developing drugs to combat the current COVID-19 pandemic and other diseases.

GPU-accelerated smoothed particle finite element method for large deformation analysis in geomechanics

Particle finite element method (PFEM) is an effective numerical tool for solving large-deformation problems in geomechanics. By incorporating the node integration technique with strain smoothing into the PFEM, we proposed the smoothed particle finite element method (SPFEM). This paper extends the SPFEM to three-dimensional cases and presents a SPFEM executed on graphics processing units (GPUs) to boost the computational efficiency. The detailed parallel computing strategy on GPU is introduced. New computation formulations related to the strain smoothing technique are proposed to save memory space in the GPU parallel computing. Several benchmark problems are solved to validate the proposed approach and to evaluate the GPU acceleration performance. Numerical examples show that with the new formulations not only the memory space can be saved but also the computational efficiency is improved. The computational cost is reduced by ~70% for the double-precision GPU parallel computing with the new formulations. Compared with the sequential CPU simulation, the GPU-accelerated simulation results in a significant speedup. The overall speedup ranges from 8.21 to 11.17 for double-precision simulations. Furthermore, the capability of the GPU-accelerated SPFEM in solving large-scale complicated problems is demonstrated by modelling the progressive failure of a long slope with strain-softening soil.

Multiscale molecular simulations of the morphological evolution of small-molecule organic solar cells during the vacuum codeposition process

The mesoscale morphologies of organic small molecular films fabricated via vacuum deposition processes are critical to the performance of small-molecule solar cells and organic light-emitting diodes. In the present study, the morphological evolution of the active layer of DPDCPB:${\mathrm{C}}_{70}$ small-molecule solar cells during vacuum codeposition processes was revealed by a series of GPU-accelerated coarse-grained molecular-dynamics simulations. The ${\mathrm{C}}_{70}$ and DPDCPB molecules were coarsened into ellipsoids and bonded ellipsoids, respectively. The interactions between ellipsoids were described by the Gay-Berne formulation and were parametrized to reproduce potential energy surfaces from all-atom atomistic simulations using a genetic algorithm. Due to the significantly reduced overall degrees of freedom, this coarse-grained scheme allowed us to simulate the vacuum codeposition processes and monitor the morphological evolution of systems with system length scales compatible with those of the experiments (\ensuremath{\sim}30 nm). Our simulations indicate that the film morphologies are closely correlated with the DPDCPB:${\mathrm{C}}_{70}$ blending ratio. High ${\mathrm{C}}_{70}$ concentration leads to a rough film surface, which is in accordance with experimental observations and can be attributed to the strong self-aggregation behavior of ${\mathrm{C}}_{70}$ molecules. The morphological property analysis indicates that the rough film surface has an almost negligible impact on the DPDCPB/${\mathrm{C}}_{70}$ domain percolations, and the device with the optimal deposition ratio should give the most balanced hole/electron transfer in respective DPDCPB/${\mathrm{C}}_{70}$ domains. The present study demonstrates that by using the ellipsoid-based coarse-grained model, it is possible to study the morphological evolution of small-molecule organic thin film during vacuum deposition processes with molecular scale details, which can provide valuable insights for experimental teams to further optimize device fabrication protocols for the next generation of organic optoelectronic devices.

GPU-Accelerated Simulation of a Rotary Valve by the Discrete Element Method

The rotary valve is the most frequently used piece of equipment that is suitable for the controlled feeding or discharging of products in powdered or granular form. It is usually connected to silos, hoppers, pneumatic conveying systems, bag filters or cyclones. In this paper, a simulation study is presented on the discharge of solid particles from a silo through a rotary valve. The discrete element method (DEM), which accounts for collisions between particles and particle-wall collisions, was used to model and simulate the motion of individual particles. The diameter of the simulated silo was 0.2 m and a total of 245,000 particles were calculated. In the simulations, the effect of the geometric and operational parameters of the rotary valve on the mass outflow rate was investigated. The diameter of the rotary valve varied between 0.06 and 0.12 m and the rotational speed of the rotor was changed between 0.5 and 5 1 . The simulations showed that the mass outflow rate of the particles from the rotary valve changes periodically due to its rotary cell structure. Within the lower range of rotational speeds of the rotor, the mass outflow rate of particles changes linearly in correlation with the rotational speed. The identification of this linear section is important in terms of control as this would facilitate the implementation of control devices by applying well-established linear control algorithms. Adjacent to the linear section, the dependence of the average mass outflow rate on the rotational speed was found to be nonlinear. Within the upper range of examined rotational speeds for each diameter of the rotary valve, the mass outflow rate reaches a maximum then decreases. The simulations were performed using GPU hardware. The application of parallel programming was an essential aspect of the simulations and significantly decreased the calculation time of simulations. In the treatment of particle-wall contacts, a novel flat triangular-based geometric representation technique was used which allows the particle-wall contacts to be calculated more effectively and their treatment implemented more easily into the parallel programming code. Using the calculated particle positions, the particles were visualized to view the effect of the interactions between the particles and rotor blades on particle motion. The simulation results showed that the discrete element method is capable of determining the detailed flow patterns of particles through the rotary valve at various rotational speeds.