Simulation Speedup Research Articles

Load-Balancing Strategies in Discrete Element Method Simulations

In this research, we investigate the influence of a load-balancing strategy and parametrization on the speed-up of discrete element method simulations using Lethe-DEM. Lethe-DEM is an open-source DEM code which uses a cell-based load-balancing strategy. We compare the computational performance of different cell-weighing strategies based on the number of particles per cell (linear and quadratic). We observe two minimums for particle to cell weights (at 3, 40 for quadratic, and 15, 50 for linear) in both linear and quadratic strategies. The first and second minimums are attributed to the suitable distribution of cell-based and particle-based functions, respectively. We use four benchmark simulations (packing, rotating drum, silo, and V blender) to investigate the computational performances of different load-balancing schemes (namely, single-step, frequent and dynamic). These benchmarks are chosen to demonstrate different scenarios that may occur in a DEM simulation. In a large-scale rotating drum simulation, which shows the systems in which particles occupy a constant region after reaching steady-state, single-step load-balancing shows the best performance. In a silo and V blender, where particles move in one direction or have a reciprocating motion, frequent and dynamic schemes are preferred. We propose an automatic load-balancing scheme (dynamic) that finds the best load-balancing steps according to the imbalance of computational load between the processes. Furthermore, we show the high computational performance of Lethe-DEM in the simulation of the packing of 108 particles on 4800 processes. We show that simulations with optimum load-balancing need ≈40% less time compared to the simulations with no load-balancing.

POET (v0.1): speedup of many-core parallel reactive transport simulations with fast DHT lookups

Abstract. Coupled reactive transport simulations are extremely demanding in terms of required computational power, which hampers their application and leads to coarsened and oversimplified domains. The chemical sub-process represents the major bottleneck: its acceleration is an urgent challenge which gathers increasing interdisciplinary interest along with pressing requirements for subsurface utilization such as spent nuclear fuel storage, geothermal energy and CO2 storage. In this context we developed POET (POtsdam rEactive Transport), a research parallel reactive transport simulator integrating algorithmic improvements which decisively speed up coupled simulations. In particular, POET is designed with a master/worker architecture, which ensures computational efficiency in both multicore and cluster compute environments. POET does not rely on contiguous grid partitions for the parallelization of chemistry but forms work packages composed of grid cells distant from each other. Such scattering prevents particularly expensive geochemical simulations, usually concentrated in the vicinity of a reactive front, from generating load imbalance between the available CPUs (central processing units), as is often the case with classical partitions. Furthermore, POET leverages an original implementation of the distributed hash table (DHT) mechanism to cache the results of geochemical simulations for further reuse in subsequent time steps during the coupled simulation. The caching is hence particularly advantageous for initially chemically homogeneous simulations and for smooth reaction fronts. We tune the rounding employed in the DHT on a 2D benchmark to validate the caching approach, and we evaluate the performance gain of POET's master/worker architecture and the DHT speedup on a 3D benchmark comprising around 650 000 grid elements. The runtime for 200 coupling iterations, corresponding to 960 simulation days, reduced from about 24 h on 11 workers to 29 min on 719 workers. Activating the DHT reduces the runtime further to 2 h and 8 min respectively. Only with these kinds of reduced hardware requirements and computational costs is it possible to realistically perform the long-term complex reactive transport simulations, as well as perform the uncertainty analyses required by pressing societal challenges connected with subsurface utilization.

Qulacs: a fast and versatile quantum circuit simulator for research purpose

To explore the possibilities of a near-term intermediate-scale quantum algorithm and long-term fault-tolerant quantum computing, a fast and versatile quantum circuit simulator is needed. Here, we introduce Qulacs, a fast simulator for quantum circuits intended for research purpose. We show the main concepts of Qulacs, explain how to use its features via examples, describe numerical techniques to speed-up simulation, and demonstrate its performance with numerical benchmarks.

Multiscale modelling of diffusion and enzymatic reaction in porous electrodes in Direct Electron Transfer mode

This work is dedicated to a multi-scale modelling of coupled diffusion and reaction in a porous micro-electrode operating in the Direct Electron Transfer mode. The pore-scale physico-electrochemical unsteady model is developed considering the oxygen reduction, catalyzed by an enzyme coating the pores of the electrode, coupled to the diffusion of oxygen and mass balance of enzymes. This model is formally upscaled to obtain an original closed unsteady macroscopic model operating at the electrode scale, together with the associated closure providing the effective diffusivity tensor. A validation of this model is carried out from a comparison with the solution of the initial 3D pore-scale governing equations considering the bilirubin oxydase as the catalyst. The relevance and accuracy of the macroscale model are proved allowing a considerable simulation speedup. It is further employed to successfully predict experimental voltammetry results obtained with porous gold electrodes functionnalized with a bilirubin oxidase mutant (BOD S362C). This model represents a breakthrough by providing an operational simple way of understanding and further optimizing porous electrodes functioning in DET mode.

Parallel‐in‐time‐and‐space electromagnetic transient simulation of multi‐terminal DC grids with device‐level switch modelling

The electromagnetic transient (EMT) simulation of multi-terminal DC (MTDC) grids requires a detailed device-level modular multilevel converter (MMC) model, which can have thousands of state variables and complex internal structures. The fast device-level insulated gate bipolar transistor (IGBT) transient requires a very small time-step, making the computational overhead prohibitive. Based on the analysis of the parallel-in-time (PiT) implementation of detailed modelled MMCs, this paper proposes a task-based hybrid PiT algorithm to achieve high parallel efficiency and speed-up of MMC with device-level modelling. Moreover, a transmission line model(TLM)-based parallel-in-time-and-space (PiT+PiS) method is proposed to connect PiT grids to conventional or other PiT grids and exploit the maximum parallelism. Simulation results show greater than 30 × speed-up and 60% parallel efficiency on a 48 cores computer for the hybrid PiT method in a 201-level three-phase MMC test case, and 20 × speed-up in the transient simulation of CIGRÉ B4 DC grid test system for the PiT+PiS method.

High-performance symbolic-numerics via multiple dispatch

As mathematical computing becomes more democratized in high-level languages, high-performance symbolic-numeric systems are necessary for domain scientists and engineers to get the best performance out of their machine without deep knowledge of code optimization. Naturally, users need different term types either to have different algebraic properties for them, or to use efficient data structures. To this end, we developed Symbolics.jl, an extendable symbolic system which uses dynamic multiple dispatch to change behavior depending on the domain needs. In this work we detail an underlying abstract term interface which allows for speed without sacrificing generality. We show that by formalizing a generic API on actions independent of implementation, we can retroactively add optimized data structures to our system without changing the pre-existing term rewriters. We showcase how this can be used to optimize term construction and give a 113x acceleration on general symbolic transformations. Further, we show that such a generic API allows for complementary term-rewriting implementations. Exploiting this feature, we demonstrate the ability to swap between classical term-rewriting simplifiers and e-graph-based term-rewriting simplifiers. We illustrate how this symbolic system improves numerical computing tasks by showcasing an e-graph ruleset which minimizes the number of CPU cycles during expression evaluation, and demonstrate how it simplifies a real-world reaction-network simulation to halve the runtime. Additionally, we show a reaction-diffusion partial differential equation solver which is able to be automatically converted into symbolic expressions via multiple dispatch tracing, which is subsequently accelerated and parallelized to give a 157x simulation speedup. Together, this presents Symbolics.jl as a next-generation symbolic-numeric computing environment geared towards modeling and simulation.

Accelerating self-gravitating hydrodynamics simulations with adaptive force updates

ABSTRACT Many astrophysical hydrodynamics simulations must account for gravity, and evaluating the gravitational field at the positions of all resolution elements can incur significant cost. Typical algorithms update the gravitational field at the position of each resolution element every time the element is updated hydrodynamically, but the actual required update frequencies for hydrodynamics and gravity can be different in general. We show that the gravity calculation in hydrodynamics simulations can be optimized by only updating gravity on a time-scale dictated by the already determined maximum time-step for accurate gravity integration Δtgrav, while staying well within the typical error budget of hydro schemes and gravity solvers. Our implementation in the gizmo code uses the time-scale derived from the tidal tensor $t_{\rm tidal} = \Vert \mathbf {T}\Vert ^{-1/2}$ to determine Δtgrav and the force update frequency in turn, and uses the rate of change of acceleration evaluated by the gravity solver to construct a predictor of the acceleration for use between updates. We test the scheme on standard self-gravitating hydrodynamics test problems, finding solutions very close to the standard scheme while evaluating far fewer gravity forces, optimizing the simulations. We also demonstrate a $\sim 70{{\ \rm per\ cent}}$ speed-up in an example simulation of a giant molecular cloud. In general, this scheme introduces a new tunable parameter for obtaining an optimal compromise between accuracy and computational cost, in conjunction with, e.g. time-step tolerance, numerical resolution, and gravity solver tolerance.

DecTree v1.0 – chemistry speedup in reactive transport simulations: purely data-driven and physics-based surrogates

Abstract. The computational costs associated with coupled reactive transport simulations are mostly due to the chemical subsystem: replacing it with a pre-trained statistical surrogate is a promising strategy to achieve decisive speedups at the price of small accuracy losses and thus to extend the scale of problems which can be handled. We introduce a hierarchical coupling scheme in which “full-physics” equation-based geochemical simulations are partially replaced by surrogates. Errors in mass balance resulting from multivariate surrogate predictions effectively assess the accuracy of multivariate regressions at runtime: inaccurate surrogate predictions are rejected and the more expensive equation-based simulations are run instead. Gradient boosting regressors such as XGBoost, not requiring data standardization and being able to handle Tweedie distributions, proved to be a suitable emulator. Finally, we devise a surrogate approach based on geochemical knowledge, which overcomes the issue of robustness when encountering previously unseen data and which can serve as a basis for further development of hybrid physics–AI modelling.

Scale-Model Simulation

Computer architects extensively use simulation to steer future processor research and development. Simulating large-scale multicore processors is extremely time-consuming and is sometimes impossible because of simulation infrastructure limitations. This paper proposes scale-model simulation, a novel methodology to predict large-scale multicore system performance. Scale-model simulation first constructs and simulates a scale model of the target system with reduced core count and shared resources. Target system performance is then predicted through machine-learning (ML) based extrapolation. Scale-model simulation predicts 32-core target system performance based on a single-core scale model with an average error of 8.0% and 15.8% for homogeneous and heterogeneous workloads, respectively, while yielding a 28× simulation speedup.

Data-driven reduced bond graph for nonlinear multiphysics dynamic systems

We propose a data-driven approach aiming at reducing Bond Graphs by using simulation data. The output of the approach is a hybrid model that contains a Reduced Bond Graph coupled to a simple artificial neural network. In this paper, the neural network is obtained by a linear calibration procedure. We propose a method that contains two training steps. First, the method selects the components of the original Bond Graph that are kept in the reduced Bond Graph. Secondly, the method builds an artificial neural network that supplements the reduced Bond Graph. Because the output of the proposed approach is a hybrid model, not solely data, it becomes difficult to use a usual Backpropagation Through Time to calibrate the weights of the neural network. So, in a first attempt, a very simple neural network is proposed by following a model reduction approach. As an industrial application, we consider the modeling of the automotive cabins thermal behavior. The data used for the training step are obtained via solutions of differential algebraic equations by means of a design of experiment. Simple cooling simulations are run during the training step. We show a simulation speed-up when the reduced bond graph is used to simulate the driving cycle of the WLTP vehicles homologation procedure, while preserving accuracy on output variables. The variables of the original bond graph are split into a set of primary variables, a set of secondary variables and a set of tertiary variables. The reduced bond graph contains all the primary variables, but none of the tertiary variables. While secondary variables are coupled to primary ones via an artificial neural network. Besides, the mathematical modeling used to describe the proposed method allows generalizing it to any set of differential algebraic equations, and not only to bond graphs. Finally, we discuss the extension of the coupling approach to more complex artificial neural networks.

Towards a Bio-Inspired Real-Time Neuromorphic Cerebellum.

This work presents the first simulation of a large-scale, bio-physically constrained cerebellum model performed on neuromorphic hardware. A model containing 97,000 neurons and 4.2 million synapses is simulated on the SpiNNaker neuromorphic system. Results are validated against a baseline simulation of the same model executed with NEST, a popular spiking neural network simulator using generic computational resources and double precision floating point arithmetic. Individual cell and network-level spiking activity is validated in terms of average spike rates, relative lead or lag of spike times, and membrane potential dynamics of individual neurons, and SpiNNaker is shown to produce results in agreement with NEST. Once validated, the model is used to investigate how to accelerate the simulation speed of the network on the SpiNNaker system, with the future goal of creating a real-time neuromorphic cerebellum. Through detailed communication profiling, peak network activity is identified as one of the main challenges for simulation speed-up. Propagation of spiking activity through the network is measured, and will inform the future development of accelerated execution strategies for cerebellum models on neuromorphic hardware. The large ratio of granule cells to other cell types in the model results in high levels of activity converging onto few cells, with those cells having relatively larger time costs associated with the processing of communication. Organizing cells on SpiNNaker in accordance with their spatial position is shown to reduce the peak communication load by 41%. It is hoped that these insights, together with alternative parallelization strategies, will pave the way for real-time execution of large-scale, bio-physically constrained cerebellum models on SpiNNaker. This in turn will enable exploration of cerebellum-inspired controllers for neurorobotic applications, and execution of extended duration simulations over timescales that would currently be prohibitive using conventional computational platforms.

A higher-order parametric nonlinear reduced-order model for imperfect structures using Neumann expansion

We present an enhanced version of the parametric nonlinear reduced-order model for shape imperfections in structural dynamics we studied in a previous work. In this model, the total displacement is split between the one due to the presence of a shape defect and the one due to the motion of the structure. This allows to expand the two fields independently using different bases. The defected geometry is described by some user-defined displacement fields which can be embedded in the strain formulation. This way, a polynomial function of both the defect field and actual displacement field provides the nonlinear internal elastic forces. The latter can be thus expressed using tensors, and owning the reduction in size of the model given by a Galerkin projection, high simulation speedups can be achieved. We show that the adopted deformation framework, exploiting Neumann expansion in the definition of the strains, leads to better accuracy as compared to the previous work. Two numerical examples of a clamped beam and a MEMS gyroscope finally demonstrate the benefits of the method in terms of speed and increased accuracy.

OpenAWSEM with Open3SPN2: A fast, flexible, and accessible framework for large-scale coarse-grained biomolecular simulations.

We present OpenAWSEM and Open3SPN2, new cross-compatible implementations of coarse-grained models for protein (AWSEM) and DNA (3SPN2) molecular dynamics simulations within the OpenMM framework. These new implementations retain the chemical accuracy and intrinsic efficiency of the original models while adding GPU acceleration and the ease of forcefield modification provided by OpenMM's Custom Forces software framework. By utilizing GPUs, we achieve around a 30-fold speedup in protein and protein-DNA simulations over the existing LAMMPS-based implementations running on a single CPU core. We showcase the benefits of OpenMM's Custom Forces framework by devising and implementing two new potentials that allow us to address important aspects of protein folding and structure prediction and by testing the ability of the combined OpenAWSEM and Open3SPN2 to model protein-DNA binding. The first potential is used to describe the changes in effective interactions that occur as a protein becomes partially buried in a membrane. We also introduced an interaction to describe proteins with multiple disulfide bonds. Using simple pairwise disulfide bonding terms results in unphysical clustering of cysteine residues, posing a problem when simulating the folding of proteins with many cysteines. We now can computationally reproduce Anfinsen's early Nobel prize winning experiments by using OpenMM's Custom Forces framework to introduce a multi-body disulfide bonding term that prevents unphysical clustering. Our protein-DNA simulations show that the binding landscape is funneled towards structures that are quite similar to those found using experiments. In summary, this paper provides a simulation tool for the molecular biophysics community that is both easy to use and sufficiently efficient to simulate large proteins and large protein-DNA systems that are central to many cellular processes. These codes should facilitate the interplay between molecular simulations and cellular studies, which have been hampered by the large mismatch between the time and length scales accessible to molecular simulations and those relevant to cell biology.

Hierarchical Aggregation/Disaggregation for Adaptive Abstraction-Level Conversion in Digital Twin-Based Smart Semiconductor Manufacturing

In smart manufacturing, engineers typically analyze unexpected real-time problems using digitally cloned discrete-event (DE) models for wafer fabrication. To achieve a faster response to problems, it is essential to increase the speed of DE simulations because making optimal decisions for addressing the issues requires repeated simulations. This paper presents a hierarchical aggregation/disaggregation (A/D) method that substitutes complex event-driven operations with two-layered abstracted models—single-group mean-delay models (SMDMs) and multi-group MDMs (MMDMs)—to gain simulation speedup. The SMDM dynamically abstracts a DE machine group’s behaviors into observed mean-delay constants when the group converges into a steady state. The MMDM fast-forwards the input lots by bypassing the chained processing steps in multiple steady-state groups until it schedules the lots for delivery to subsequent unsteady groups after corresponding multi-step mean delays. The key component, the abstraction-level converter (ALC), has the roles of MMDM allocation, deallocation, extension, splitting, and controls the flow of each group’s input lot by deciding the destination DE model, SMDM, and MMDMs. To maximize the reuse of previously computed multi-step delays for the dynamically changing MMDMs, we propose an efficient method to manage the delays using two-level caches. Each steady-state group’s ALC performs statistical testing to detect the lot-arrival change to reactivate the DE model. However, fast-forwarding (FF) results in incorrect test results of the bypassed group’s ALCs due to the missed observations of the bypassed lots. Thus, we propose a method for test-sample reinitialization that considers the bypassing. Moreover, since a bypassed group’s unexpected divergence can change the multi-step delays of previously scheduled events, a method for examination of FF history is designed to trace the highly influenced events. This proposed method has been applied in various case studies, and it has achieved speedups of up to about 5.9 times, with 2.5 to 8.3% degradation in accuracy.

Integration of JUNO simulation framework with Opticks: GPU accelerated optical propagation via NVIDIA® OptiX™

Opticks is an open source project that accelerates optical photon simulation by integrating NVIDIA GPU ray tracing, accessed via NVIDIA OptiX, with Geant4 toolkit based simulations. A single NVIDIA Turing architecture GPU has been measured to provide optical photon simulation speedup factors exceeding 1500 times single threaded Geant4 with a full JUNO analytic GPU geometry automatically translated from the Geant4 geometry. Optical physics processes of scattering, absorption, scintillator reemission and boundary processes are implemented within CUDA OptiX programs based on the Geant4 implementations. Wavelength-dependent material and surface properties as well as inverse cumulative distribution functions for reemission are interleaved into GPU textures providing fast interpolated property lookup or wavelength generation. In this work we describe major recent developments to facilitate integration of Opticks with the JUNO simulation framework including on GPU collection effciency hit culling which substantially reduces both the CPU memory needed for photon hits and copying overheads. Also progress with the migration of Opticks to the all new NVIDIA OptiX 7 API is described.

UNRES-Dock-protein-protein and peptide-protein docking by coarse-grained replica-exchange MD simulations.

The majority of the proteins in living organisms occur as homo- or hetero-multimeric structures. Although there are many tools to predict the structures of single-chain proteins or protein complexes with small ligands, peptide-protein and protein-protein docking is more challenging. In this work, we utilized multiplexed replica-exchange molecular dynamics (MREMD) simulations with the physics-based heavily coarse-grained UNRES model, which provides more than a 1000-fold simulation speed-up compared with all-atom approaches to predict structures of protein complexes. We present a new protein-protein and peptide-protein docking functionality of the UNRES package, which includes a variable degree of conformational flexibility. UNRES-Dock protocol was tested on a set of 55 complexes with size from 43 to 587 amino-acid residues, showing that structures of the complexes can be predicted with good quality, if the sampling of the conformational space is sufficient, especially for flexible peptide-protein systems. The developed automatized protocol has been implemented in the standalone UNRES package and in the UNRES server. UNRES server: http://unres-server.chem.ug.edu.pl; UNRES package and data used in testing of UNRES-Dock: http://unres.pl. Supplementary data are available at Bioinformatics online.

Efficient discrete-event based particle tracking simulation for high energy physics

This work presents novel discrete event-based simulation algorithms based on the Quantized State System (QSS) numerical methods. QSS provides attractive features for particle transportation processes, in particular a very efficient handling of discontinuities in the simulation of continuous systems. We focus on High Energy Physics (HEP) particle tracking applications that typically rely on discrete time-based methods, and study the advantages of adopting a discrete event-based numerical approach that resolves efficiently the crossing of geometry boundaries by a traveling particle. For this purpose we follow two complementary strategies. First, a new co-simulation technique connects the Geant4 simulation toolkit with a standalone QSS solver. Second, a new native QSS numerical stepper is embedded into Geant4. We compare both approaches against the latest Geant4 default steppers in different HEP setups, including a complex realistic scenario (the CMS particle detector at CERN). Our techniques achieve relevant simulation speedups in a wide range of scenarios, particularly when the intensity of discrete-event handling dominates performance in the solving of the continuous laws of particle motion.

A novel rockbolt formulation for a GPU-accelerated, finite-discrete element method code and its application to underground excavations

Rockbolting is widely used in civil and mining engineering to increase the inherent strength of the rock mass and reduce deformations around underground excavations. Rockbolt design by means of computer-aided numerical simulations has been playing an increasingly important role in geotechnical engineering. The research work presented herein introduces a novel rockbolt logic for the finite-discrete element method (FDEM), a numerical modelling approach capable of explicitly capturing brittle fracturing in complex geological media. Adoption of a GPU-based implementation was required to achieve substantial simulation speed-ups compared to conventional CPU computing. Rockbolts are discretized by 1D structural elements which are coupled to the solid finite element mesh by dedicated spring-slider connectors representing the bolt anchoring system. The formulation is verified by comparing simulated stress distributions with closed-form solutions for the cases of (i) a pull-out test on a fully-grouted rockbolt and (ii) a circular tunnel excavated in a bolted elastic medium. The effectiveness of the method for practical design applications is demonstrated by simulating the behavior of a bolted underground excavation in a heavily jointed rock mass. The effect of bolt spacing on the fracturing and deformational response of the rock mass is analyzed together with the force distributions in the rockbolts.

A Bit-Time-Dependent Model of I/O Drivers for Overclocking Analysis

This article proposes a bit-time-dependent behavioral model of input–output (I/O) drivers for overclocking simulation. The driver behavior under overclocking conditions is investigated, and the corresponding weighting function response surface is demonstrated. In contrast to the previous approaches that use fixed timing signals extracted under normal operating conditions only, the proposed model addresses the modeling of overclocking behavior by using bit-time-dependent weighting functions (BTDWFs) along with a transition variable. The weighting coefficients can be generated appropriately for different overclocking scenarios. The corresponding model extraction flow is presented. Using the proposed BTDWFs, the driver input signal is processed during run-time by a finite state machine (FSM) algorithm which can be implemented in the widely supported hardware description languages such as Verilog-A. The proposed model is able to capture the driver’s behavior accurately under both normal operation and overclocking conditions. Its fidelity and simulation speedup are validated with modeling examples using commercial driver circuit.

A third-order multirate Runge–Kutta scheme for finite volume solution of 3D time-dependent Maxwell’s equations

A third-order multirate time-stepping based on an SSP Runge–Kutta method is applied to solve the three-dimensional Maxwell’s equations on unstructured tetrahedral meshes. This allows for an evolution of the solution on fine and coarse meshes with time steps satisfying a local stability condition to improve the computational efficiency of numerical simulations. Two multirate strategies with flexible time-step ratios are compared for accuracy and efficiency. Numerical experiments with a third-order finite volume discretization are presented to validate the theory. Our results of electromagnetic simulations demonstrate that 1D analysis is also valid for linear conservation laws in 3D. In one of the methods, significant speedup in 3D simulations is achieved without sacrificing third-order accuracy.