Parallel Mesh Research Articles

Developing a parallel density-based implicit solver with mesh deformation in OpenFOAM

Numerical simulations like the aeroelastic computation and aerodynamic shape optimization usually involve moving boundaries, and are always carried out with supersonic compressible flows. In this paper, based on the OpenFOAM platform, we present a parallel density-based implicit solver with mesh deformation to address this kind of problems, and fill the gap for the deficiencies of implicit solvers. The core implementation details of the solver based on OpenFOAM are introduced. The Godunov type schemes in ALE (Arbitrary Lagrangian Eulerian) formulation and the dual-time stepping LU-SGS (Lower-Upper Symmetric Gauss-Seidel) algorithm are applied to solve the Navier-Stokes equations with turbulence model. For dynamic meshes, the novel parallel mesh deformation approach based on the Support Vector Machine is implemented. Four typical cases are tested to demonstrate the validity and parallel efficiency of the proposed solver. The result shows that the scalability of the solver reaches its limit at around 128 MPI (Message Passing Interface) tasks, and the cost of MPI communication is the bottleneck for large-scale simulations. Overall, our solver is applicable for various dynamic mesh compressible problems and all types of grids.

Distributed Combinatorial Maps for Parallel Mesh Processing

We propose a new strategy for the parallelization of mesh processing algorithms. Our main contribution is the definition of distributed combinatorial maps (called n-dmaps), which allow us to represent the topology of big meshes by splitting them into independent parts. Our mathematical definition ensures the global consistency of the meshes at their interfaces. Thus, an n-dmap can be used to represent a mesh, to traverse it, or to modify it by using different mesh processing algorithms. Moreover, an nD mesh with a huge number of elements can be considered, which is not possible with a sequential approach and a regular data structure. We illustrate the interest of our solution by presenting a parallel adaptive subdivision method of a 3D hexahedral mesh, implemented in a distributed version. We report space and time performance results that show the interest of our approach for parallel processing of huge meshes.

Quantitative phase-field lattice-Boltzmann study of lamellar eutectic growth under natural convection.

The influence of natural convection on lamellar eutectic growth was determined by a comprehensive phase-field lattice-Boltzmann study for Al-Cu and CBr_{4}-C_{2}Cl_{6} eutectic alloys. The mass differences resulting from concentration differences led to the fluid flow and a robust parallel and adaptive mesh refinement algorithm was employed to improve the computational efficiency. By means of carefully designed "numerical experiments", the eutectic growth under natural convection was explored and a simple analytical model was proposed to predict the adjustment of the lamellar spacing. Furthermore, by alternating the solute expansion coefficient, initial lamellar spacing, and undercooling, the microstructure evolution was presented and compared with the classical eutectic growth theory. Results showed that both interfacial solute distribution and average curvature were affected by the natural convection, the effect of which could be further quantified by adding a constant into the growth rule proposed by Jackson and Hunt [Jackson and Hunt, Trans. Metall. Soc. AIME 236, 1129 (1966)].

In‐memory integration of existing software components for parallel adaptive unstructured mesh workflows

SummaryReliable mesh‐based simulations are needed to solve complex engineering problems. Mesh adaptivity can increase reliability by reducing discretization errors but requires multiple software components to exchange information. Often, components exchange information by reading and writing a common file format. This file‐based approach becomes a problem on massively parallel computers where filesystem bandwidth is a critical performance bottleneck. Our approach using data streams and component interfaces avoids the filesystem bottleneck. In this paper, we present these techniques and their use for coupling mesh adaptivity to the PHASTA computational fluid dynamics solver, the Albany multi‐physics framework, and the Omega3P linear accelerator frequency analysis applications. Performance results are reported on up to 16,384 cores of an Intel Knights Landing‐based system.

Efficient and Robust Parallel Mesh Motion Solver Using Radial Basis Functions

AbstractThis paper presents a parallel mesh deformation solver using radial basis function (RBF) interpolation. The solver computes the displacement of each internal point independently without usi...

Parallel adaptive mesh refinement method based on bubble-type local mesh generation

An efficient parallel adaptive mesh refinement method based on bubble-type local mesh generation (BLMG), employing ParMETIS-based dynamic domain decomposition method, is developed. The BLMG method is applied to generate the mesh with high quality, and the local mesh for each processor can be generated simultaneously without communication. The node-based distributed mesh structure is designed to reduce the communication amount spent in mesh generation and finite element calculation. To gain the load balance, a new load balancing algorithm for the new mesh structure is devised to make sure the whole algorithm is efficient. Several numerical examples are carried out to verify the high efficiency of the algorithm.

ADER-DG with a-posteriori finite-volume limiting to simulate tsunamis in a parallel adaptive mesh refinement framework

In this paper we present the application of the ADER Discontinuous Galerkin method with an a-posteriori Finite-Volume limiter on the simulation of tsunamis. Goal is to obtain a method of high-order convergence in deep water areas while being able to handle wetting & drying at coast lines. Several adjustments of the original ADER-DG method are presented to preserve characteristics like the well-balanced property. We evaluate and confirm developed concepts by a series of numerical tests and present them in the context of reconstructed tsunamis.

An ideal compressible magnetohydrodynamic solver with parallel block-structured adaptive mesh refinement

We present an adaptive parallel solver for the numerical simulation of ideal magnetohydrodynamics in two and three space dimensions. The discretisation uses a finite volume scheme based on a Cartesian mesh and an explicit compact Runge–Kutta scheme for time integration. Numerically, a generalized Lagrangian multiplier approach with a mixed hyperbolic-parabolic correction is used to guarantee a control on the incompressibility of the magnetic field. We implement the solver in the AMROC (Adaptive Mesh Refinement in Object-oriented C++) framework that uses a structured adaptive mesh refinement (SAMR) method discretisation-independent and is fully parallelised for distributed memory systems. Moreover, AMROC is a modular framework providing manageability, extensibility and efficiency. In this paper, we give an overview of the ideal magnetohydrodynamics solver developed in this framework and its capabilities. We also include an example of this solver’s verification with other codes and its numerical and computational performance.

Thread-parallel mesh improvement using face and edge swapping and vertex insertion

The purpose of this paper is three-fold. First, we devise a memory model for unstructured mesh data for efficient use of memory on parallel shared memory architectures, with the purpose of lowering the synchronization overhead between threads while avoiding race conditions. Second, we present a new thread-parallel edge and face swapping algorithm for two and three dimensional meshes using OpenMP for shared memory architectures. We show how removing the conflicts from the reconfiguration procedure by applying a vertex locking strategy can result in a near linear speed-up with parallel efficiency of close to one on two threads and 70% with 24 threads on shared-memory processors. Finally, we derive a parallel mesh generation and refinement module for shared memory architectures based on pre-existing serial modules by implementing Chernikov and Chrisochoides' parallel insertion algorithm along with the two above tools. Experiments show a worst case parallel efficiency of 50% for parallel refinement with 24 threads.

Parallel mesh partitioning based on space filling curves

Larger supercomputers allow the simulation of more complex phenomena with increased accuracy. Eventually this requires finer and thus also larger geometric discretizations. In this context, and extrapolating to the Exascale paradigm, meshing operations such as generation, deformation, adaptation/regeneration or partition/load balance, become a critical issue within the simulation workflow. In this paper we focus on mesh partitioning. In particular, we present a fast and scalable geometric partitioner based on Space Filling Curves (SFC), as an alternative to the standard graph partitioning approach. We have avoided any computing or memory bottleneck in the algorithm, while we have imposed that the solution achieved is independent (discounting rounding off errors) of the number of parallel processes used to compute it. The performance of the SFC-based partitioner presented has been demonstrated using up to 4096 CPU-cores in the Blue Waters supercomputer.

Hybrid parallel multigrid preconditioner based on automatic mesh coarsening for 3D metal forming simulations

SummaryA parallel multigrid (MG) method is developed to reduce the large computational costs involved by the finite element simulation of highly viscous fluid flows, especially those resulting from metal forming applications, which are characterized by using a mixed velocity/pressure implicit formulation, unstructured meshes of tetrahedra, and frequent remeshings. The developed MG method follows a hybrid approach where the different levels of nonnested meshes are geometrically constructed by mesh coarsening, while the linear systems of the intermediate levels result from the Galerkin algebraic approach. A linear O(N) convergence rate is expected (with N being the number of unknowns), while keeping software parallel efficiency. These objectives lead to selecting unusual MG smoothers (iterative solvers) for the upper grid levels and to developing parallel mesh coarsening algorithms along with parallel transfer operators between the different levels of partitioned meshes. Within the utilized PETSc library, the developed MG method is employed as a preconditioner for the usual conjugate residual algorithm because of the symmetric undefinite matrix of the system to solve. It shows a convergence rate close to optimal, an excellent parallel efficiency, and the ability to handle the complex forming problems encountered in 3‐dimensional hot forging, which involve large material deformations and frequent remeshings.

Parallel-In-Space-Time, Adaptive Finite Element Framework for Nonlinear Parabolic Equations

We present an adaptive methodology for the solution of (linear and) nonlinear time dependent problems that is especially tailored for massively parallel computations. The basic concept is to solve for large blocks of space-time unknowns instead of marching sequentially in time. The methodology is a combination of a computationally efficient implementation of a parallel-in-space-time finite element solver coupled with a posteriori space-time error estimates and a parallel mesh generator. While we focus on spatial adaptivity in this work, the methodology enables simultaneous adaptivity in both space and time domains. We explore this basic concept in the context of a variety of time steppers including $\Theta$-schemes and backward difference formulas. We specifically illustrate this framework with applications involving time dependent linear, quasi-linear, and semilinear diffusion equations. We focus on investigating how the coupled space-time refinement indicators for this class of problems affect spatial adaptivity. Finally, we show good scaling behavior up to 150,000 processors on the NCSA Blue Waters machine. This conceptually simple methodology enables scaling on next generation multicore machines by simultaneously solving for a large number of timesteps, and reducing computational overhead by locally refining spatial blocks that can track localized features. This methodology also opens up the possibility of efficiently incorporating adjoint equations for error estimators and inverse design problems, since blocks of space-time are simultaneously solved and stored in memory.

Enhancing speed and scalability of the ParFlow simulation code

Regional hydrology studies are often supported by high resolution simulations of subsurface flow that require expensive and extensive computations. Efficient usage of the latest high performance parallel computing systems becomes a necessity. The simulation software ParFlow has been demonstrated to meet this requirement and shown to have excellent solver scalability for up to 16,384 processes. In the present work we show that the code requires further enhancements in order to fully take advantage of current petascale machines. We identify ParFlow's way of parallelization of the computational mesh as a central bottleneck. We propose to reorganize this subsystem using fast mesh partition algorithms provided by the parallel adaptive mesh refinement library p4est. We realize this in a minimally invasive manner by modifying selected parts of the code to reinterpret the existing mesh data structures. We evaluate the scaling performance of the modified version of ParFlow, demonstrating good weak and strong scaling up to 458k cores of the Juqueen supercomputer, and test an example application at large scale.

Phase-field-lattice Boltzmann study for lamellar eutectic growth in a natural convection melt

In the present study, the influence of natural convection on the lamellar eutectic growth is determined by a phase-field-lattice Boltzmann study for Al-Cu eutectic alloy. The mass difference resulting from concentration difference led to the fluid flow, and a robust parallel and adaptive mesh refinement algorithm was employed to improve the computational efficiency without any compromising accuracy. Results show that the existence of natural convection would affect the growth undercooling and thus control the interface shape by adjusting the lamellar width. In particular, by alternating the magnitude of the solute expansion coefficient, the strength of the natural convection is changed. Corresponding microstructure patterns are discussed and compared with those under no-convection conditions.

Ternary Sparse Matrix Representation for Volumetric Mesh Subdivision and Processing on GPUs

AbstractIn this paper, we present a novel volumetric mesh representation suited for parallel computing on modern GPU architectures. The data structure is based on a compact, ternary sparse matrix storage of boundary operators. Boundary operators correspond to the first‐order top‐down relations of k‐faces to their (k − 1)‐face facets. The compact, ternary matrix storage format is based on compressed sparse row matrices with signed indices and allows for efficient parallel computation of indirect and bottom‐up relations. This representation is then used in the implementation of several parallel volumetric mesh algorithms including Laplacian smoothing and volumetric Catmull‐Clark subdivision. We compare these algorithms with their counterparts based on OpenVolumeMesh and achieve speedups from 3× to 531×, for sufficiently large meshes, while reducing memory consumption by up to 36%.

A parallel log barrier-based mesh warping algorithm for distributed memory machines

Parallel dynamic meshes are essential for computational simulations of large-scale scientific applications involving motion. To address this need, we propose parallel LBWARP, a parallel log barrier-based tetrahedral mesh warping algorithm for distributed memory machines. Our algorithm is a general-purpose, geometric mesh warping algorithm that parallelizes the sequential LBWARP algorithm proposed by Shontz and Vavasis. The first step of the algorithm involves computation of a set of local weights for each interior node which describe the relative distances of the node to each of its neighbors. The weight computation step is the most time consuming in the parallel algorithm. Based on our choice of the mesh partition and the corresponding distribution of data and assignment of tasks to processors, communication among processors is avoided in an embarrassingly parallel computation of the weights. Once this representation of the initial mesh is determined, a target deformation of the boundary is applied, also in an embarrassingly parallel manner. Finally, new coordinates of the interior nodes are obtained by solving a system of linear equations with multiple right-hand sides that is based on the weights and boundary deformation. This linear system can be solved using one of three parallel sparse linear solvers, i.e., the distributed block BiCG, block GMRES, or LU algorithm, all of which support the solution of linear systems with multiple right-hand side vectors. Our numerical results demonstrate good efficiency and strong scalability of parallel LBWARP on up to 64 processors, as the experiments show close to linear speedup in all cases. Weak scalability is also demonstrated. The performance of the parallel sparse linear solvers is dependent on factors such as the mesh size, the amount of available memory, and the number of processors. For example, the distributed LU algorithm gives better performance on small meshes, whereas the distributed block BiCG and distributed block GMRES algorithms yield better performance when the amount of available memory is limited. Finally, we demonstrate the parallel LBWARP performance for a sequence of mesh deformations which can significantly reduce the runtime of the overall algorithm. When applied to k deformations, parallel LBWARP reuses the weight matrix, that was computed during the first deformation, when the distributed LU linear solver is employed. This gives close to k-time performance for sufficiently many deformations.

Distributed poly-square mapping for large-scale semi-structured quad mesh generation

We develop a distributed poly-square mapping algorithm for large-scale 2D geometric regions, which is suitable for generating huge quadrilateral meshes in parallel using computer clusters. Our approach adopts a divide-and-conquer strategy based on domain decomposition. We first partition the data into solvable subregions, balancing their size, geometry, and communication cost; then, poly-square maps will be solved on subregions; these maps are finally merged and optimized globally through a multi-pass optimization algorithm. We demonstrate that our meshing framework can handle very big and complex geometric datasets using high performance clusters efficiently and generate high-quality semi-structured quad meshes.

Numerical investigation of fish exploiting vortices based on the Kármán gaiting model

To understand the mechanism of fish exploiting vortices, this paper, for the first time, employs the Kármán gaiting model to numerically investigate the hydrodynamics of fish swimming in the vortex street. Considering the time-consuming computations during our large-scale simulations, we implement a parallel mesh deformation method based on the radial basis function(RBF) interpolation. Based on simulations over a wide range of controlling parameters, we discuss in detail the efficiency of fish exploiting vortices, the force coefficient and the hydrodynamic interactions between fish and vortices. It is observed that as the phase difference between fish's lateral translation and the shedding vortex changes, the efficiency presents a variation trend opposite to the mean thrust on the fish body; the optimum swimming distance downstream from the cylinder is positively affected by the inlet velocity and the diameter of the cylinder, which is consistent with the experimental results. From the hydrodynamic characteristics in the flow fields, we qualitatively draw a conclusion that the maximum efficiency is obtained when the fish is undulating with the tail flapping towards the center of the vortex, and the optimum distance is gained where vortices are drifting with the slipping mode.

Domain decomposition approach for parallel improvement of tetrahedral meshes

Presently, a tetrahedral mesher based on the Delaunay triangulation approach may outperform a tetrahedral improver based on local smoothing and flip operations by nearly one order in terms of computing time. Parallelization is a feasible way to speed up the improver and enable it to handle large-scale meshes. In this study, a novel domain decomposition approach is proposed for parallel mesh improvement. It analyses the dual graph of the input mesh to build an inter-domain boundary that avoids small dihedral angles and poorly shaped faces. Consequently, the parallel improver can fit this boundary without compromising the mesh quality. Meanwhile, the new method does not involve any inter-processor communications and therefore runs very efficiently. A parallel pre-processing pipeline that combines the proposed improver and existing parallel surface and volume meshers can prepare a quality mesh containing hundreds of millions of elements in minutes. Experiments are presented to show that the developed system is robust and applicable to models of a complication level experienced in industry.

Water Injection for Oil Recovery by using Reservoir Simulation via CFD

Waterflooding process involves the water injection into the reservoir to increase the oil recovery by oil displacement and pressure maintenance mechanisms. This article is based on three-dimensional numerical simulation of a homogeneous reservoir saturated by oil, 22.1°API, with water injection using a non-structured mesh. In order to analyze the behavior of formation and prediction of the fluid production, it was used the ANSYS CFX 12.0 ® commercial software. The model consists in a reservoir with the dimensions, 270x180x15 m³, and two in parallel five-spot meshes, considering the internal injection, and in the top of the reservoir, and variations of water rate injected, during a total time of 72,000 hours of project. Results of recovery factor, water production, saturation advance front and pressure are presented and analyzed. Was observed that increasing the water rate injected caused an increment in the oil recovery, and consequently the water production, however better results were obtained with injection on the top. This study allowed identifying best practices to water injection optimization.