Parallel Simulation Framework Research Articles

Afivo: A framework for quadtree/octree AMR with shared-memory parallelization and geometric multigrid methods

Afivo is a framework for simulations with adaptive mesh refinement (AMR) on quadtree (2D) and octree (3D) grids. The framework comes with a geometric multigrid solver, shared-memory (OpenMP) parallelism and it supports output in Silo and VTK file formats. Afivo can be used to efficiently simulate AMR problems with up to about 108 unknowns on desktops, workstations or single compute nodes. For larger problems, existing distributed-memory frameworks are better suited. The framework has no built-in functionality for specific physics applications, so users have to implement their own numerical methods. The included multigrid solver can be used to efficiently solve elliptic partial differential equations such as Poisson’s equation. Afivo’s design was kept simple, which in combination with the shared-memory parallelism facilitates modification and experimentation with AMR algorithms. The framework was already used to perform 3D simulations of streamer discharges, which required tens of millions of cells. Program summaryProgram Title: AfivoProgram Files doi:http://dx.doi.org/10.17632/5y43rjdmxd.1Licensing provisions: GPLv3Programming language: Fortran 2011External routines/libraries: Silo (LLNL)Nature of problem: Performing multiscale simulations, especially those requiring a fast elliptic solver.Solution method: Provide a framework for parallel simulations on adaptively refined quadtree/octree grids, including a geometric multigrid solver.Unusual features: The framework uses shared-memory parallelism (OpenMP) instead of MPI.

Prophet: A Parallel Instruction-Oriented Many-Core Simulator

Most existing computer architecture simulators are cycle oriented, i.e., they are driven cycle by cycle. However, frequent switches among simulation contexts, excessive buffer accesses and tightly coupled manner often make such an architecture simulator slow, difficult to parallelize and hard to scale to large-scale many-core systems. In this paper, we propose Prophet, a parallel instruction-oriented simulation framework for many-cores. Prophet adopts a general instruction-oriented model to simulate processor cores, in which a simulator is built from the perspective of each simulated instruction impacting a small number of relevant processor components, as opposed to that of a large number of processor components executing many instructions in each cycle as in the cycle-oriented approach. Prophet determines the execution cycle of a simulated instruction based on the states of the relevant components impacted by the instruction, and update the components states after the execution of the instruction. Prophet also adopts a speculative model to decouple private resources from the shared resources (e.g., shared cache), which avoids unnecessary interactions between them and only pays a penalty upon a rare mis-speculation. We have designed and implemented a prototype of Prophet that supports both user-level and full-system simulation. Experimental results show Prophet can scale up to simulate thousands of simulated cores (4,096 cores in the current implementation) with good performance and small accuracy loss. It achieves average simulation speeds of about 98 and 235 MIPS (millions of simulated instructions per second) for full-system and user-level simulation, respectively, with only 3 percent IPC error rate and negligible deviation in cache simulation results. When run on a many-core platform (i.e., Intel Xeon Phi), it achieved an average simulation speed of about 413 MIPS.

Large-scale simulation of a self-organizing self-management cloud computing framework

A recently introduced cloud simulation framework is extended to support self-organizing and self-management local strategies in the cloud resource hierarchy. This dynamic hardware resource allocation system is evolving toward the goals defined by local strategies, which are determined as maximization of: energy efficiency of cloud infrastructures, task throughput, computational efficiency and resource management efficiency. Heterogeneous hardware resources are considered that are except from commodity CPU servers, hardware accelerators such as GPUs, MICs and FPGAs, thus forming a heterogeneous cloud infrastructure. Energy consumption and task execution models for the heterogeneous accelerators are also proposed, in order to demonstrate the energy efficiency of the proposed resource allocation system. Implementation details of the new functionalities on the parallel cloud simulation framework are discussed, while numerical results are given for the scalability and utilization of the cloud elements using the self-organization and self-management framework with two VM placement strategies.

Automatic Model Generation for Gate-Level Circuit PDES with Reverse Computation

Gate-level circuit simulation is an important step in the design and validation of complex circuits. This step of the process relies on existing libraries for gate specifications. We start with a generic gate model for Rensselaer’s Optimistic Simulation System, a parallel discrete-event simulation framework. This generic model encompasses all functionality needed by optimistic simulation using reverse computation. We then describe a parser system that uses a standardized gate library to create a specific model for simulation. The generated model is composed of several functions, including those needed for an accurate model of timing behavior. To quantify the improvements that an automatically generated model can have over a hand written model, we compare two gate library models: an automatically generated lsi -10 k library model and a previously investigated, handwritten, simplified gtech library model. We conclude that the automatically generated model is a more accurate model of actual hardware. In comparison to previous results, we find that the automatically generated model is able to achieve better optimistic simulation performance when measured against conservative simulation. To test the automatically generated model, we evaluate the performance of a simulation of a full-scale OpenSPARC T2 processor model. This model consists of nearly 6 million LPs. We achieve a peak performance of 1.63 million events per second during a conservative simulation. To understand the relatively weaker performance of optimistic simulation, we investigate hot spots of event activity and visually identify a workload imbalance.

A Novel GPU-Enabled Simulator for Large Scale Spiking Neural Networks

The understanding of the structural and dynamic complexity of neural networks is greatly facilitated by computer simulations. An ongoing challenge for simulating realistic models is, however, computational speed. In this paper a framework for modeling and parallel simulation of biological-inspired large scale spiking neural networks on high-performance graphics processors is described. This tool is implemented in the OpenCL programming technology. It enables simulation study with three models: Integrate-andfire, Hodgkin-Huxley and Izhikevich neuron model. The results of extensive simulations are provided to illustrate the operation and performance of the presented software framework. The particular attention is focused on the computational speed-up factor.

Energy-momentum correlations for Abelian Higgs cosmic strings

We report on the energy-momentum correlators obtained with recent numerical simulations of the Abelian Higgs model, essential for the computation of cosmic microwave background and matter perturbations of cosmic strings. Due to significant improvements both in raw computing power and in our parallel simulation framework, the dynamical range of the simulations has increased four-fold both in space and time, and for the first time we are able to simulate strings with a constant physical width in both the radiation and matter eras. The new simulations improve the accuracy of the measurements of the correlation functions at the horizon scale and confirm the shape around the peak. The normalization is slightly higher in the high wave-number tails, due to a small increase in the string density. We study for the first time the behaviour of the correlators across cosmological transitions, and discover that the correlation functions evolve adiabatically, ie the network adapts quickly to changes in the expansion rate. We propose a new method for constructing source functions for Einstein-Boltzmann integrators, comparing it with two other methods previously used. The new method is more consistent, easier to implement, and significantly more accurate.

Parallel GPU Architecture Simulation Framework Exploiting Architectural-Level Parallelism with Timing Error Prediction

The performance analysis and study of large-scale many-core processor architectures require fast and highly accurate simulation techniques in order to reduce time consumption. State-of-the-art graphics processing units (GPUs), which are used extensively as coprocessors in the high-performance-computing area, also require fast simulation techniques because they have massively complex microarchitectures with thousands of processing elements. At present, however, GPU simulators do not have sufficient simulation speed for advanced software and architecture studies. In this study, we propose a new parallel simulation framework and a new parallel simulation technique for improving the simulation speed of GPUs. The proposed framework facilitates multithreaded simulation by exploiting the architectural-level parallelism and execution model parallelism of GPUs. In addition, an error predictive synchronization scheme based on a timing error prediction mechanism is used to minimize the cycle errors and simulator slowdown during parallel simulations. The experimental results obtained using a simulator with the proposed framework showed that the proposed technique provided a speedup of up to 8.9 times compared with an existing single-thread-based GPU simulator on a 16-core machine.

A numerical study of the effect of particle properties on the radial distribution of suspensions in pipe flow

We employ a Lagrangian–Lagrangian (LL) numerical formalism to study two- and three-dimensional (2D, 3D) pipe flow of dilute suspensions of macroscopic neutrally buoyant rigid bodies at flow regimes with Reynolds numbers (Re) between 0.1 and 1400. A validation study of particle migration over a wide spectrum of Re and average volumetric concentrations demonstrates the good predictive attributes of the LL approach adopted herein. Using a scalable parallel implementation of the approach, 3D direct numerical simulation is used to show that (1) rigid body rotation affects the behavior of a particle laden flow; (2) an increase in neutrally buoyant particle size decreases radial migration; (3) a decrease in inter-particle distance slows down the migration and shifts the stable position further away from the channel axis; (4) rigid body shape influences the stable radial distribution of particles; (5) particle migration is influenced, both quantitatively and qualitatively, by the Reynolds number; and (6) the stable radial particle concentration distribution is affected by the initial concentration. The parallel LL simulation framework developed herein does not impose restrictions on the shape or size of the rigid bodies and was used to simulate 3D flows of dense, colloidal suspensions of up to 23,000 neutrally buoyant ellipsoids.

Parallel simulation of groundwater non-point source pollution using algebraic multigrid preconditioners

The simulation of non-point source pollution in agricultural basins is a computationally demanding process due to the large number of individual sources and potential pollution receptors (e.g., drinking water wells). In this study, we present an efficient computational framework for parallel simulation of diffuse pollution in such groundwater basins. To derive a highly detailed velocity field, we employed algebraic multigrid (AMG) preconditioners to solve the groundwater flow equation. We compare two variants of AMG implementations, the multilevel preconditioning provided by Trilinos and the BoomerAMG provided by HYPRE. We also perform a sensitivity analysis on the configuration of AMG methods to evaluate the application of these libraries to groundwater flow problems. For the transport simulation of diffuse contamination, we use the streamline approach, which decomposes the 3D transport problem into a large number of 1D problems that can be executed in parallel. The proposed framework is applied to a 2,600-km2 groundwater basin in California discretized into a grid with over 11 million degrees of freedom. Using a Monte Carlo approach with 200 nitrate loading realizations at the aquifer surface, we perform a stochastic analysis to quantify nitrate breakthrough prediction uncertainty at over 1,500 wells due to random, temporally distributed nitrate loading. The results show that there is a significant time lag between loading and aquifer response at production wells. Generally, typical production wells respond after 5–50 years depending on well depth and screen length, while the prediction uncertainty for nitrate in individual wells is very large—approximately twice the drinking water limit for nitrate.

Parallel Simulation of Large Linear Circuits With Nonlinear Terminations Using High-Order Stable Methods

To meet the growing demand for efficient circuit simulation tools, Obreshkov formula (ObF)-based high-order integration methods were recently proposed. However, one of the challenges in this method is the growing computational cost of the solution at a particular time point with the increasing orders of the ObF. To address this issue and target high-speed circuits with large linear blocks and nonlinear loads, a new parallel framework is proposed in this paper to minimize the CPU time associated with the solution at any time point. For this purpose, recently identified special properties of the resulting circuit matrices and the node-tearing principles are exploited, while developing an efficient parallel simulation framework. Numerical examples are presented to demonstrate the validity and efficiency of the proposed parallel method.

Real‐time physical deformation and cutting of heterogeneous objects via hybrid coupling of meshless approach and finite element method

ABSTRACTThis paper advocates a method for real‐time physical deformation and arbitrary cutting simulation of heterogeneous objects with multi‐material distribution, whose originality centers on the tight coupling of domain‐specific finite element method (FEM) and material distance‐aware meshless approach in a CUDA‐centric parallel simulation framework. We employ hierarchical hexahedron serving as basic building blocks for accurate material‐aware FEM simulation. Meanwhile, local meshless systems are designed to support cross‐FEM‐domain coupling and material‐sensitive propagation while respecting the regularity of finite elements. Directly benefiting from the structural regularity and uniformity of finite elements, our hybrid solution enables the local stiffness matrix pre‐computation and dynamic assembling, adaptive topological updating and precise cutting reconstruction. Moreover, our mathematically‐rigorous solver guarantees unconditional stableness. Experiments demonstrate the superiorities of our system. Copyright © 2014 John Wiley & Sons, Ltd.

Parallel kinetic Monte Carlo simulation framework incorporating accurate models of adsorbate lateral interactions

Ab initio kinetic Monte Carlo (KMC) simulations have been successfully applied for over two decades to elucidate the underlying physico-chemical phenomena on the surfaces of heterogeneous catalysts. These simulations necessitate detailed knowledge of the kinetics of elementary reactions constituting the reaction mechanism, and the energetics of the species participating in the chemistry. The information about the energetics is encoded in the formation energies of gas and surface-bound species, and the lateral interactions between adsorbates on the catalytic surface, which can be modeled at different levels of detail. The majority of previous works accounted for only pairwise-additive first nearest-neighbor interactions. More recently, cluster-expansion Hamiltonians incorporating long-range interactions and many-body terms have been used for detailed estimations of catalytic rate [C. Wu, D. J. Schmidt, C. Wolverton, and W. F. Schneider, J. Catal. 286, 88 (2012)]. In view of the increasing interest in accurate predictions of catalytic performance, there is a need for general-purpose KMC approaches incorporating detailed cluster expansion models for the adlayer energetics. We have addressed this need by building on the previously introduced graph-theoretical KMC framework, and we have developed Zacros, a FORTRAN2003 KMC package for simulating catalytic chemistries. To tackle the high computational cost in the presence of long-range interactions we introduce parallelization with OpenMP. We further benchmark our framework by simulating a KMC analogue of the NO oxidation system established by Schneider and co-workers [J. Catal. 286, 88 (2012)]. We show that taking into account only first nearest-neighbor interactions may lead to large errors in the prediction of the catalytic rate, whereas for accurate estimates thereof, one needs to include long-range terms in the cluster expansion.

Complex fluid simulations with the parallel tree-based Lattice Boltzmann solver Musubi

We present the open source Lattice Boltzmann solver Musubi. It is part of the parallel simulation framework APES, which utilizes octrees to represent sparse meshes and provides tools from automatic mesh generation to post-processing. The octree mesh representation enables the handling of arbitrarily complex simulation domains, even on massively parallel systems. Local grid refinement is implemented by several interpolation schemes in Musubi. Various kernels provide different physical models based on stream-collide algorithms. These models can be computed concurrently and can be coupled with each other. This paper explains our approach to provide a flexible yet scalable simulation environment and elaborates its design principles and implementation details. The efficiency of our approach is demonstrated with a performance evaluation on two supercomputers and a comparison to the widely used Lattice Boltzmann solver Palabos.

Multi-dimensional finite-volume scheme for hyperbolic conservation laws on three-dimensional solution-adaptive cubed-sphere grids

A scalable parallel and block-adaptive cubed-sphere grid simulation framework is described for solution of hyperbolic conservation laws in domains between two concentric spheres. In particular, the Euler and ideal magnetohydrodynamics (MHD) equations are considered. Compared to existing cubed-sphere grid algorithms, a novelty of the proposed approach involves the use of a fully multi-dimensional finite-volume method. This leads to important advantages when the treatment of boundaries and corners of the six sectors of the cubed-sphere grid is considered. Most existing finite-volume approaches use dimension-by-dimension differencing and require special interpolation or reconstruction procedures at ghost cells adjacent to sector boundaries in order to achieve an order of solution accuracy higher than unity. In contrast, in our multi-dimensional approach, solution blocks adjacent to sector boundaries can directly use physical cells from the adjacent sector as ghost cells while maintaining uniform second-order accuracy. This leads to important advantages in terms of simplicity of implementation for both parallelism and adaptivity at sector boundaries. Crucial elements of the proposed scheme are: unstructured connectivity of the six grid root blocks that correspond to the six sectors of the cubed-sphere grid, multi-dimensional k-exact reconstruction that automatically takes into account information from neighbouring cells isotropically and is able to automatically handle varying stencil size, and adaptive division of the solution blocks into smaller blocks of varying spatial resolution that are all treated exactly equally for inter-block communication, flux calculation, adaptivity and parallelization. The proposed approach is fully three-dimensional, whereas previous studies on cubed-sphere grids have been either restricted to two-dimensional geometries on the sphere or have grids and solution methods with limited capabilities in the third dimension in terms of adaptivity and parallelism. Numerical results for several problems, including systematic grid convergence studies, MHD bow-shock flows, and global modelling of solar wind flow are discussed to demonstrate the accuracy and efficiency of the proposed solution procedure, along with assessment of parallel computing scalability for up to thousands of computing cores.

Scalable parallel dynamic fracture simulation using an extrinsic cohesive zone model

In order to achieve realistic cohesive fracture simulation, a parallel computational framework is developed in conjunction with the parallel topology based data structure (ParTopS). Communications with remote partitions are performed by employing proxy nodes, proxy elements and ghost nodes, while synchronizations are identified on the basis of computational patterns (at-node, at-element, nodes-to-element, and elements-to-node). Several approaches to parallelize a serial code are discussed. An approach combining local computations and replicated computations with stable iterators is proposed, which is shown to be the most efficient one among the approaches discussed in this study. Furthermore, computational experiments demonstrate the scalability of the parallel dynamic fracture simulation framework for both 2D and 3D problems. The total execution time of a test problem remains nearly constant when the number of processors increases at the same rate as the number of elements.

Significantly reducing MPI intercommunication latency and power overhead in both embedded and HPC systems

Highly parallel systems are becoming mainstream in a wide range of sectors ranging from their traditional stronghold high-performance computing, to data centers and even embedded systems. However, despite the quantum leaps of improvements in cost and performance of individual components over the last decade (e.g., processor speeds, memory/interconnection bandwidth, etc.), system manufacturers are still struggling to deliver low-latency, highly scalable solutions. One of the main reasons is that the intercommunication latency grows significantly with the number of processor nodes. This article presents a novel way to reduce this intercommunication delay by implementing, in custom hardware, certain communication tasks. In particular, the proposed novel device implements the two most widely used procedures of the most popular communication protocol in parallel systems the Message Passing Interface (MPI). Our novel approach has initially been simulated within a pioneering parallel systems simulation framework and then synthesized directly from a high-level description language (i.e., SystemC) using a state-of-the-art synthesis tool. To the best of our knowledge, this is the first article presenting the complete hardware implementation of such a system. The proposed novel approach triggers a speedup from one to four orders of magnitude when compared with conventional software-based solutions and from one to three orders of magnitude when compared with a sophisticated software-based approach. Moreover, the performance of our system is from one to two orders of magnitude higher than the simulated performance of a similar but, relatively simpler hardware architecture; at the same time the power consumption of our device is about two orders of magnitude lower than that of a low-power CPU when executing the exact same intercommunication tasks.

Software design for a highly parallel molecular dynamics simulation framework in chemical engineering

Highlights ► Software structure of a nanofluidics simulation program in chemical engineering. ► Design supports especially testing and comparison of different methods. ► Focus on modules for parallelisation using MPI and allowing hybrid parallelisation. ► Supports different libraries (e.g. TBB, OpenMP) for memory-coupled parallelisation. Abstract The software structure of MarDyn, a molecular dynamics simulation program for nanofluidics in chemical engineering, is presented. Multi-component mixtures in heterogeneous states with huge numbers of particles put great challenges on the simulation of scenarios in this field, which cannot be tackled with the established molecular simulation programs. The need to develop a new software for such simulations with an interdisciplinary team opened the chance of using state-of-the-art methods on the modelling as well as on the simulation side. This entails the need to test and compare different methods in all parts of the program to be able to find the best method for each task. It is shown how the software design of MarDyn supports testing and comparing of various methods in all parts of the program. The focus lies on those parts concerning parallelisation, which is on the one hand a pure MPI parallelisation and on the other hand a hybrid approach using MPI in combination with a memory-coupled parallelisation. For the latter, MarDyn not only allows the use of different algorithms, but also supports the use of different libraries such as OpenMP and TBB.

A software framework for multimodal interactive simulations (SoFMIS).

The development of a multimodal interactive simulation is a very elaborate task due to the various complex software components involved, which run simultaneously at very high rates with maximum CPU load. In this work, we propose a multimodal parallel simulation framework called SoFMIS to create rapid interactive simulations such as surgical simulations. Our framework offers great flexibility and customization allowing simulation developers and researchers to concentrate on the simulation logic rather than component development.

A fast MPI-based parallel framework for cycle-accurate HDL multi-parametric simulations

The growing complexity of digital architectures strongly impacts on their verification phase, which becomes critical. In fact, without giving up cycle-accuracy, it seems there are not at the state-of-the-art fast frameworks allowing multi-parametric simulations at a fine granularity level for accurate verification and design space exploration purposes. In this paper, we propose a parallel, fast and cycle-accurate SystemC simulation framework: SysCgrid. SysCgrid is designed to provide automatic generation and parallel execution of multi-parametric simulations with minimum effort by hardware architects. This framework is conceived to run on a cluster/grid computing infrastructure, exploiting the message passing interface (MPI) to automatically distribute the multi-parametric simulation set over more than one node. The achieved results on a network-on-chip (NoC) architecture verification demonstrate the good SysCgrid scalability, both in case of shared and non-shared memory computing infrastructures, achieving impressive speed up with respect to traditional single-run simulations approaches.

HPABM: A Hierarchical Parallel Simulation Framework for Spatially‐explicit Agent‐based Models

AbstractA Hierarchical Parallel simulation framework for spatially‐explicit Agent‐Based Models (HPABM) is developed to enable computationally intensive agent‐based models for the investigation of large‐scale geospatial problems. HPABM allows for the utilization of high‐performance and parallel computing resources to address computational challenges in agent‐based models. Within HPABM, an agent‐based model is decomposed into a set of sub‐models that function as computational units for parallel computing. Each sub‐model is comprised of a sub‐set of agents and their spatially‐explicit environments. Sub‐models are aggregated into a group of super‐models that represent computing tasks. HPABM based on the design of super‐ and sub‐models leads to the loose coupling of agent‐based models and underlying parallel computing architectures. The utility of HPABM in enabling the development of parallel agent‐based models was examined in a case study. Results of computational experiments indicate that HPABM is scalable for developing large‐scale agent‐based models and, thus, demonstrates efficient support for enhancing the capability of agent‐based modeling for large‐scale geospatial simulation.