Parallel Direct Numerical Simulation Research Articles

A FFT-accelerated multi-block finite-difference solver for massively parallel simulations of incompressible flows

We present a multi-block finite-difference solver for massively parallel Direct Numerical Simulations (DNS) of incompressible flows. The algorithm combines the versatility of a multi-block solver with the method of eigenfunctions expansions, to speedup the solution of the pressure Poisson equation. This is achieved by employing FFT-based transforms along one homogeneous direction, which effectively reduce the problem complexity at a low cost. These FFT-based expansions are implemented in a framework that unifies all valid combinations of boundary conditions for this type of method. Subsequently, a geometric multigrid solver is employed to solve the reduced Poisson equation in a multi-block geometry. Particular care was taken here, to guarantee the parallel performance of the multigrid solver when solving the reduced linear systems equations. We have validated the overall numerical algorithm and assessed its performance in a massively parallel setting. The results show that 2- to 8-fold reductions in computational cost may be easily achieved when exploiting FFT-accelerated for the solution of the Poisson equation. The solver, SNaC, has been made freely available and open-source under the terms of an MIT license.

Parallel direct numerical simulation and analysis of turbulent Rayleigh–Bénard convection at moderate Rayleigh numbers using an efficient algorithm

Direct numerical simulation of turbulent Rayleigh-Bénard convection up to Rayleigh number 108 is performed using a fully-implicit, non-dissipative, discrete kinetic energy-conserving algorithm and a parallel flow solver based on it. The algorithm is especially suitable for simulating low-Mach number, variable density/viscosity, transitional and turbulent flows with or without heat transfer. Furthermore, since it does not rely on the Boussinesq assumption, large temperature differences and high Rayleigh numbers can be handled without loss of accuracy, unlike the pure incompressible ones. It is first shown that the algorithm is able to predict the evolution of thermally-driven instability to turbulent regime and all the characteristics of turbulent convection accurately, using low- and high-order turbulent statistics and various secondary diagnostics derived. Then, effects of increasing Rayleigh numbers on the development of the instability are analyzed in detail. Additionally, Nusselt-Rayleigh scaling properties are studied and a scaling relation is provided. Results show that Rayleigh-Bénard convection at relatively high Rayleigh numbers, corresponding to a boundary layer-dominated regime and little beyond it to a bulk-dominated regime, is characterized by weakening thermal fluctuations, thinning thermal boundary layers, increasing vertical velocity fluctuations and decreasing skewness. It is also observed that the turbulent heat flux dominates the heat transfer. Finally, the corresponding Nusselt-Rayleigh scaling relation is predicted as Nu=0.132Ra0.297.

Roughness-Induced Boundary-Layer Transition on a Hypersonic Capsule-Like Forebody Including Nonequilibrium

The present work investigates the influence of high-temperature gas effects on the laminar–turbulent transition induced by a patch of distributed roughness on a hemispherical capsule-like geometry. The freestream conditions correspond to a realistic reentry scenario at Mach 20. At these conditions, chemical reactions and nonequilibrium effects are present in the high-enthalpy boundary layer of the hemisphere. Parallel Direct Numerical Simulations are undertaken to analyze the instability mechanisms in the crossflow-type vortex developing in the wake of a skewed protuberance of the roughness patch. Both linear and nonlinear growth of unsteady disturbances forced in the roughness wake, including laminar–turbulent breakdown, are considered in the analysis. The primary focus of the study is how chemical and thermal nonequilibrium affect the location of the laminar–turbulent transition as well as the level of wall heating in the transitional boundary layer for the considered capsule configuration. The results highlight the necessity to include nonequilibrium effects in this problem of roughness-induced transition at high-altitude reentry conditions because the gas modeling turns out to have a notable influence on the development of instabilities, both in the linear and in the nonlinear ranges.

A scalable multiphysics algorithm for massively parallel direct numerical simulations of electrophoretic motion

We introduce a novel coupled algorithm for massively parallel direct numerical simulations of electrophoresis in microfluidic flows. This multiphysics algorithm employs an Eulerian description of fluid and ions, combined with a Lagrangian representation of moving charged particles. The fixed grid facilitates efficient solvers and the employed lattice Boltzmann method can efficiently handle complex geometries. Validation experiments with more than 70 000 time steps are presented, together with scaling experiments with over 4 × 106 particles and 1.96 × 1011 grid cells for both hydrodynamics and electric potential. We achieve excellent performance and scaling on up to 65 536 cores of a current supercomputer.

A solver for massively parallel direct numerical simulation of three-dimensional multiphase flows

We present a new solver for massively parallel simulations of fully three-dimensional multiphase flows. The solver runs on a variety of computer architectures from laptops to supercomputers and on 65536 threads or more (limited only by the availability to us of more threads). The code is wholly written by the authors in Fortran 2003 and uses a domain decomposition strategy for parallelization with MPI. The fluid interface solver is based on a parallel implementation of the LCRM hybrid Front Tracking/Level Set method designed to handle highly deforming interfaces with complex topology changes. We discuss the implementation of this interface method and its particular suitability to distributed processing where all operations are carried out locally on distributed subdomains. We have developed parallel GMRES and Multigrid iterative solvers suited to the linear systems arising from the implicit solution of the fluid velocities and pressure in the presence of strong density and viscosity discontinuities across fluid phases. Particular attention is drawn to the details and performance of the parallel Multigrid solver. The code includes modules for flow interaction with immersed solid objects, contact line dynamics, species and thermal transport with phase change. Here, however, we focus on the simulation of the canonical problem of drop splash onto a liquid film and report on the parallel performance of the code on varying numbers of threads. The 3D simulations were run on mesh resolutions up to $1024^3$ with results at the higher resolutions showing the fine details and features of droplet ejection, crown formation and rim instability observed under similar experimental conditions. Keywords:

Turbulent flow field analysis of a jet in cross flow by DNS

The turbulent flow field induced by a round jet in crossflow is calculated by parallel direct numerical simulation (DNS) on multi-GPU clusters. The DNS is based on the lattice- Boltzmannmethod.With currentGPUsettings, a grid systemof 1.5×108 is adopted and the largest jet Reynolds number reaches 3000. The jet is orthogonal to the mainstream flow direction. The validated code produces good agreements with theory and experiment. Steady and unsteady vortical structures are presented based on velocity fields and vorticity distributions. Profiles of Reynolds stress components are also displayed and analyzed. Hair-pin coherent structures are presented based on second invariant of velocity gradient. Transport of turbulent kinetic energy is represented by budget terms in x-, y- and z-direction planes and along the leading and trailing edges of jet trajectory.

High Performance Computation of a Jet in Crossflow by Lattice Boltzmann Based Parallel Direct Numerical Simulation

Direct numerical simulation (DNS) of a round jet in crossflow based on lattice Boltzmann method (LBM) is carried out on multi-GPU cluster. Data parallel SIMT (single instruction multiple thread) characteristic of GPU matches the parallelism of LBM well, which leads to the high efficiency of GPU on the LBM solver. With present GPU settings (6 Nvidia Tesla K20M), the present DNS simulation can be completed in several hours. A grid system of 1.5 × 108is adopted and largest jet Reynolds number reaches 3000. The jet-to-free-stream velocity ratio is set as 3.3. The jet is orthogonal to the mainstream flow direction. The validated code shows good agreement with experiments. Vortical structures of CRVP, shear-layer vortices and horseshoe vortices, are presented and analyzed based on velocity fields and vorticity distributions. Turbulent statistical quantities of Reynolds stress are also displayed. Coherent structures are revealed in a very fine resolution based on the second invariant of the velocity gradients.

Advancing predictive models for particulate formation in turbulent flames via massively parallel direct numerical simulations.

Combustion of fossil fuels is likely to continue for the near future due to the growing trends in energy consumption worldwide. The increase in efficiency and the reduction of pollutant emissions from combustion devices are pivotal to achieving meaningful levels of carbon abatement as part of the ongoing climate change efforts. Computational fluid dynamics featuring adequate combustion models will play an increasingly important role in the design of more efficient and cleaner industrial burners, internal combustion engines, and combustors for stationary power generation and aircraft propulsion. Today, turbulent combustion modelling is hindered severely by the lack of data that are accurate and sufficiently complete to assess and remedy model deficiencies effectively. In particular, the formation of pollutants is a complex, nonlinear and multi-scale process characterized by the interaction of molecular and turbulent mixing with a multitude of chemical reactions with disparate time scales. The use of direct numerical simulation (DNS) featuring a state of the art description of the underlying chemistry and physical processes has contributed greatly to combustion model development in recent years. In this paper, the analysis of the intricate evolution of soot formation in turbulent flames demonstrates how DNS databases are used to illuminate relevant physico-chemical mechanisms and to identify modelling needs.

Rising of 3D catalyst particles in a natural convection dominated flow by a parallel DNS method

Abstract We investigate the problem of particulate flows with heat transfer by parallel direct numerical simulation (DNS). Among other heat transfer problems, we examine in detail the case of a 3D spherical catalyst rising in an enclosure due to natural convection though it is heavier than the suspending fluid. Natural convection is created by heat transferred from the warmer particle to the fluid. Heat is assumed to be produced at a constant rate in the particle bulk. As expected, there exists a critical production rate that leads to the rising of the catalyst. Compared to the 2D circular cylinder counterpart, momentum and heat transfers are slower in 3D and the spherical catalyst rises for a lower production rate. At the numerical level, we employ a Distributed Lagrange Multiplier/Fictitious Domain formulation together with an operator-splitting algorithm to solve the coupled problem. Two families of Lagrange multiplier are introduced to relax the velocity and temperature constraints respectively. As suggested in Wachs (2009), particle collisions are handled by an efficient Discrete Element Method granular solver. As it is, the model is restricted to the case of homogeneous temperature over the particles. From a computational viewpoint, this work might be regarded as an extension of the method proposed in our previous contributions (Dan & Wachs, 2010; Yu, Shao, et al., 2006) to distributed computing with our new parallel code PeliGRIFF. 1 1 ▪ Parallel Efficient LIbrary for GRains In Fluid Flow, see Wachs (2011) for details. This opens up new possibilities to study a broad range of applications in 3D and to get more insight in the comprehension of particulate flows with heat transfer. In particular, we examine how a bed of spherical catalysts can be self-fluidized as a result of the heat produced in the particles bulk, mimicking an exothermic catalyst reaction in a chemical engineering reactor.

PeliGRIFF, a parallel DEM-DLM/FD direct numerical simulation tool for 3D particulate flows

The problem of particulate flows at moderate to high concentration and finite Reynolds number is addressed by parallel direct numerical simulation. The present contribution is an extension of the work published in Computers & Fluids 38:1608 (2009), where systems of moderate size in a 2D geometry were examined. At the numerical level, the suggested method is inspired by the framework established by Glowinski et al. (Int J Multiph Flow 25:755, 1999) in the sense that their Distributed Lagrange Multiplier/Fictitious Domain (DLM/FD) formulation and their operator-splitting idea are employed. In contrast, particle collisions are handled by an efficient Discrete Element Method (DEM) granular solver, which allows one to consider both smoothly (sphere) and non-smoothly (angular polyhedron) shaped particles. From a computational viewpoint, a basic though efficient strategy has been developed to implement the DLM/FD method in a domain decomposition/distributed fashion. To achieve this goal, the serial code, GRIFF (GRains In Fluid Flow; see Comput Fluids 38:1608–1628, 2009) is upgraded to fully MPI capabilities. The new code, PeliGRIFF (Parallel Efficient Library for GRains in Fluid Flow) is developed under the framework of the fully MPI open-source platform PELICANS. The parallel computing capabilities of PeliGRIFF offer new perspectives in the study of particulate flows and indeed increase the number of particles usually simulated in the literature. Solutions to address new issues raised by the parallelization of the DLM/FD method and assess the scalable properties of the code are proposed. Results on the 2D/3D sedimentation of a significant collection of isometric polygonal/polyhedral particles in a Newtonian fluid with collisions are presented as a validation test and an illustration of the class of particulate flows PeliGRIFF is able to investigate.

Swirling and Impinging Effects in an Annular Nonpremixed Jet Flame

The effects of swirl and downstream wall confinement on an annular nonpremixed flame were investigated using direct numerical simulation (DNS). Fully three-dimensional parallel DNS was performed employing high-order numerical methods and high-fidelity boundary conditions to solve governing equations for variable-density flow and finite-rate Arrhenius chemistry. Three swirl numbers have been examined: 0 (without swirl), 0.4 and 0.8, while the effects of downstream wall confinement have been examined for swirl numbers 0 and 0.4. Results have been presented in terms of instantaneous and time-averaged flow quantities, which have also been analysed using energy spectra and proper orthogonal decomposition (POD). Effects of swirl on the fluid dynamic behaviour of the annular nonpremixed flame were found to be significant. The fluid dynamic behaviour of the flame is greatly affected by the interaction between the geometrical recirculation zone (GRZ) near the jet nozzle exit due to the annular configuration, the central recirculation zone (CRZ) associated with swirl, the unsteady vortical structures in the jet column due to the shear instability, and the downstream wall confinement. Depending on the degree of swirl, the GRZ near the burner mouth and the CRZ may co-exist or one zone may be overwhelmed by another. At a moderate swirl number, the co-existence leads to a flame with strong reaction attached to the burner mouth; while at a high swirl number, the CRZ dominates over the GRZ. The precessing vortex core was observed to exist in the swirling flow fields. The Nusselt number distribution of the annular impinging flames differs from that of round impinging jets. The POD analysis revealed that wall effects on the flow field are mainly associated with the higher mode numbers.

A numerical study of the stabilitiy of helical vortices using vortex methods

We present large-scale parallel direct numerical simulations using particle vortex methods of the instability of the helical vortices. We study the instability of a single helical vortex and find good agreement with inviscid theory. We outline equilibrium configurations for three double helical vortices—similar to those produced by three blade wind turbines. The simulations confirm the stability of the inviscid model, but predict a breakdown of the vortical system due to viscosity.

Transient simulation of reacting radiating flows

Laminar methane–air diffusion flame was simulated by coupling a method of lines based parallel direct numerical simulation code with a radiation code based on method of lines solution of discrete ordinates method. The predictions of the code are validated against experimental data as well as numerical results of the same code without radiation model. Comparisons show that incorporation of radiation code to the computational fluid dynamics code results in a significant improvement in the predicted temperatures. Transient results exhibit the physically expected trends. The coupled code is a promising tool for the simulation of transient reacting radiating flows.

Scalability of parallel spatial direct numerical simulations on intel hypercube and IBM SP1 and SP2

The implementation and performance of a parallel spatial direct numerical simulation (PSDNS) approach on the Intel iPSC/860 hypercube and IBM SP1 and SP2 parallel computers is documented. Spatially evolving disturbances associated with laminar-to-turbulent transition in boundary-layer flows are computed with the PSDNS code. The feasibility of using the PSDNS to perform transition studies on these computers is examined. The results indicate that PSNDS approach can effectively be parallelized on a distributed memory parallel machine by remapping the distributed data structure during the course of the calculation. The performance of the PSDNS indicates that 32 nodes on the IBM SP1 or SP2 are required to match a comparable computation on the CRAY C-90 single processor; the hypercube was found to be two times slower than the CRAY Y/MP for this application.