Direct Numerical Simulation Code Research Articles

Performance evaluation of parallel direct numerical simulation code on supercomputer SX-Aurora TSUBASA

Although direct numerical simulations (DNSs) of incompressible turbulent flows have been performed since the late 1960s, simulations that reproduce the strongly nonlinear turbulent flows observed in the real world have not been realized. A DNS code is parallelized using pencil decomposition and optimized for the vector-type supercomputer SX-Aurora TSUBASA. A loop blocking technique is used to optimize the code. It is found that some loops of the three-dimensional fast Fourier transform have an optimal block size for loop blocking that enhances the DNS code performance. The optimized DNS code achieves a 3.3-fold speedup compared to the original code. In addition, a computation time estimation model for large-scale DNSs is constructed based on the measured throughput data for all-to-all communications on SX-Aurora TSUBASA. One step of the Runge–Kutta–Gill integration takes one minute to complete for a DNS with 18,4323 grid points on 16,384 vector engines of SX-Aurora TSUBASA.

Direct numerical simulation of natural convection between an enclosure and multiple circular cylinders: An influence of horizontal arrangement of cylinders

The work involves a three-dimensional numerical analysis of the buoyancy-driven heat transfer in an enclosure having four heated cylinders for Rayleigh number, Ra = 10 4 -10 6 , and Prandtl number, Pr = 0.7. An in-house direct numerical simulation code, finite volume-based immersed boundary method incorporates the inner cylinder's virtual wall boundary and computes the governing transport equations. Heat and flow characteristics at different Ra and spacing between cylinders are discussed by using three-dimensional vortices and isotherms. The effect of heat and flow on three-dimensionality and heat transfer features are captured and reported. Limitations of two-dimensional simulation for capturing the physics of the problem and the importance of the three-dimensional analysis for the present study are highlighted.

An unstructured adaptive mesh refinement approach for computational fluid dynamics of reacting flows

A new Direct Numerical Simulation (DNS) code HAMISH with Adaptive Mesh Refinement (AMR) has been developed to simulate compressible reacting flow in a computationally economical manner. The focus is on problems where high gradients of temperature, density and species mass fraction remain localised, for example within the interior structure of flames. The resolution requirements of such local high-gradient regions often determine the global mesh spacing when uniform meshes are used. The numerical framework in HAMISH is based on an unstructured finite-volume approach together with a Runge-Kutta algorithm for time-stepping. The unstructured Cartesian mesh used in HAMISH allows for adaptive refinement and de-refinement of the local mesh depending on the demands of the flow physics and numerical accuracy. The code utilises an octree data structure and a Morton space-filling curve, allowing efficient cell division as well as straightforward parallel domain decomposition. The spatial discretisation scheme uses fourth-order polynomial reconstruction to evaluate the inter-cell fluxes. A third-order three-step explicit Runge-Kutta time-marching algorithm is used together with adaptive time-stepping with proportional-integral-type error control using an embedded Runge-Kutta scheme. The code is fully parallelised through domain decomposition over an arbitrary number of processors and exhibits a high level of parallel efficiency. In this paper the key capabilities of HAMISH are demonstrated based on a number of test cases. These test cases include (a) simulation of 1-D planar laminar premixed flames to demonstrate the capability of AMR in capturing the sharp gradients within the flame; (b) fully developed laminar channel flow, which shows that the code is capable of refining the mesh in the boundary layer next to the wall; (c) 2-D expanding flame under quiescent laminar condition, which demonstrates that the code is capable of dynamic local mesh refinement based on the position of the flame front; (d) 3-D non-reacting Taylor-Green vortex which demonstrates HAMISH can deal with vortical motion typical of turbulent flows; (e) 3-D premixed turbulent flame propagation under isotropic homogeneous decaying turbulence to demonstrate that the code can be employed for production level DNS of turbulent reacting flows.

Effect of mass injection on secondary instability of hypersonic boundary layer over a blunt cone

In low environmental disturbances, secondary mechanisms play a crucial role in flow instability and transition. Over the years, researchers have shown the existence of secondary waves prior to nonlinear breakdown and turbulence. In high-speed flows, the vehicles are subjected to extreme thermal loads, and usually, an ablative heat shield is used for protection. The ablative materials eject gaseous products upon heating, significantly affecting flow stability. Mass injection through the surface in the boundary layer loosely replicates the ablation process. Its effect on stability and transition has been explored earlier using experiments, numerical simulation, and Linear Stability Theory (LST). However, the effect of the surface injection on secondary instability, the most viable path to transition, remains uncertain. The present work studies the secondary instability of a hypersonic boundary over a 7° half-angle blunt cone in the presence of mass injection through the surface of the boundary layer. Mean flow over the cone is solved using a high-order shock fitting direct numerical simulation code. Primary instability is studied using the LST, and secondary instability is studied using the secondary instability theory based on the Floquet model. Computations are carried out for different injection rates, and it is found that instability increases with mass injection rate. The mass injection has increased the primary and secondary growth rates. Fundamental modes are more dominant than subharmonic and detuned modes at higher primary wave amplitudes. The mass injection has increased the primary and secondary N-factor, and transitioning behavior is observed at the maximum injection rate.

Optimizing an acoustic liner by automatic differentiation of a compressible flow solver

The attenuation power of an acoustic liner is optimized by unsteady compressible flow simulations that model the presence of the liner with a time-domain impedance boundary condition (TDIBC). Two test cases are considered: a one-dimensional impedance tube and a two-dimensional grazing incidence tube. The impact of the liner on the pressure field is tuned through various TDIBC parameters imposed at the boundary where the liner surface is located. Automatic differentiation of the direct numerical simulation (DNS) code yields the gradients of the time-integrated pressure perturbations with respect to the TDIBC parameters. The gradients are subsequently used in an iterative process that minimizes the perturbations at a given location. In both test cases considered, TDIBC parameters are found which are more effective at attenuating perturbation frequencies other than those the initial TDIBC is designed to cancel. Finally, a sensitivity analysis reveals the importance of the choice of the initial TDIBC parameters for finding optimized settings. Our results pave the way to a wider application of the presented procedure in more complex problems with bulk flow motion and larger perturbation amplitudes.

High-Resolution Numerical Simulation of Microfiltration of Oil-in-Water Emulsion Permeating through a Realistic Membrane Microporous Structure Generated by Focused Ion Beam Scanning Electron Microscopy Images.

Owing to the limitations of visualization techniques in experimental studies and low-resolution numerical models based on computational fluid dynamics (CFD), the detailed behavior of oil droplets during microfiltration is not well understood. Hence, a high-resolution CFD model based on an in-house direct numerical simulation (DNS) code was constructed in this study to analyze the detailed dynamics of an oil-in-water (O/W) emulsion using a microfiltration membrane. The realistic microporous structure of commercial ceramic microfiltration membranes (mullite and α-alumina membranes) was obtained using an image processing technique based on focused ion beam scanning electron microscopy (FIB-SEM). Numerical simulations of microfiltration of O/W emulsions on the membrane microstructure obtained by FIB-SEM were performed, and the effects of different parameters, including contact angle, transmembrane pressure, and membrane microporous structure, on filtration performance were studied. Droplet deformation had a strong impact on filtration behavior because coalesced droplets with diameters larger than the pore diameter permeated the membrane pores. The permeability, oil hold-up fraction inside the pores, and rejection were considerably influenced by the contact angle, while the transmembrane pressure had a little impact on the permeability and oil hold-up fraction. The membrane structure, especially the pore size distribution, also had a significant effect on the microfiltration behavior and performance.

Simulation of three-dimensional forced compressible isotropic turbulence by a redesigned discrete unified gas kinetic scheme

In this paper, we implemented the Boltzmann-equation-based mesoscopic model, developed recently by Chen et al. [“Inverse design of mesoscopic models for compressible flow using the Chapman–Enskog analysis,” Adv. Aerodyn. 3, 5 (2021)], to simulate three-dimensional (3D) forced compressible isotropic turbulence. In this model, both the Prandtl number and the ratio of bulk to shear viscosity can be arbitrary prescribed. The statistically stationary turbulent flow is driven by a large-scale momentum forcing in the Fourier space, with the internal heating due to the viscous dissipation at small scales being removed by a thermal cooling function. Under the framework of discrete unified gas kinetic scheme (DUGKS), a 3D direct numerical simulation code has been developed, incorporating a generalized Strang-splitting scheme. The weighted essentially non-oscillatory (WENO) scheme is used to increase local spatial accuracy in the reconstruction of particle distribution functions at the cell interface. A 3D discrete particle velocity model with a ninth-order Gauss–Hermite quadrature accuracy is used to ensure accurate evaluation of viscous stress and heat flux in the continuum regime. We simulate forced compressible isotropic turbulence at both low and high turbulent Mach numbers. A direct comparison is performed with the results obtained from a hybrid compact finite difference-WENO scheme solving directly the Navier–Stokes–Fourier system. The comparison validates our DUGKS code and indicates that DUGKS is a reliable and promising tool for simulating forced compressible isotropic turbulence. The work represents a first study to directly simulate forced compressible turbulence by a mesoscopic method based on the Boltzmann equation.

A numerical study on the combustion of a resolved carbon particle

Combustion of a single, resolved carbon particle is studied using a novel numerical approach that makes use of an overset grid. The model is implemented into the framework of a compressible Direct Numerical Simulation (DNS) code. A method to artificially reduce the speed of sound is presented. For Mach numbers lower than ∼0.1 this method may dramatically improve numerical efficiency without affecting any physical aspects except for the acoustics. The ability of the model to simulate solid fuel combustion is demonstrated and all parts of the model are validated against experimental and numerical data. A sensitivity of the carbon conversion rate to selected parameters (diffusion coefficients and homogeneous and heterogeneous kinetics) is investigated. A strong dependence on the oxygen diffusivity is observed and explained.

Optimization of DNS code and visualization of entrainment and mixing phenomena at cloud edges

Entrainment and mixing processes occur during the entire life of a cloud. These processes change the droplet size distribution, which determines rain formation and radiative properties. Since it is a microphysical process, it cannot be resolved in large scale weather forecasting models. Small scale simulations such as Direct Numerical Simulations (DNS) are required to resolve the most minute scale of these processes. The DNS of cloud dynamics are performed by integrating two mathematical models, Eulerian and Lagrangian, in a coupled way. Running DNS is a tedious task as it requires a huge amount of computational resources. In this work, we provide a projection of the required resources for running DNS in different size domains.Visualizing these large simulations presents an added challenge, as they generate petabytes of data. Visualization plays a vital role in analyzing and understanding these huge data outputs. Here, we experimented with multiple tools to conduct a visual analysis of this data. Two of these tools are well established and tested technologies: ParaView and VAPOR. The others are emergent technologies in the development phase. This data simulation and visualization, in addition to exploring DNS as mentioned above, provided an opportunity to test and improve development of several tools and methods.

Direct numerical simulations of turbulent reacting flows with shock waves and stiff chemistry using many-core/GPU acceleration

Compressible reacting flows may display sharp spatial variation related to shocks, contact discontinuities or reactive zones embedded within relatively smooth regions. The presence of such phenomena emphasizes the relevance of shock-capturing schemes such as the weighted essentially non-oscillatory (WENO) scheme as an essential ingredient of the numerical solver. However, these schemes are complex and have more computational cost than the simple high-order compact or non-compact schemes. In this paper, we present the implementation of a seventh-order, minimally-dissipative mapped WENO (WENO7M) scheme in a newly developed direct numerical simulation (DNS) code called KAUST Adaptive Reactive Flows Solver (KARFS). In order to make efficient use of the computer resources and reduce the solution time, without compromising the resolution requirement, the WENO routines are accelerated via graphics processing unit (GPU) computation. The performance characteristics and scalability of the code are studied using different grid sizes and block decomposition. The performance portability of KARFS is demonstrated on a variety of architectures including NVIDIA Tesla P100 GPUs and NVIDIA Kepler K20X GPUs. In addition, the capability and potential of the newly implemented WENO7M scheme in KARFS to perform DNS of compressible flows is also demonstrated with model problems involving shocks, isotropic turbulence, detonations and flame propagation into a stratified mixture with complex chemical kinetics.

Direct Numerical Simulation of Water Droplets in Turbulent Flow

Details on the fall speeds of raindrops are essential in both applications and natural events, such as rain-rate retrieval and soil erosion. Here, we examine the influence of turbulence on the terminal velocity of two water drops of different sizes. For the first time, computations of droplets in turbulent surroundings are conducted with a direct numerical simulation code based on a volume of fluid method. Both the drop surface deformation and internal circulation are captured. The turbulence intensity at the inflow area, as well as the turbulence length scale are varied. In turbulent flow, we find a decline in the terminal velocities for both drops. Based on the study of the wake flow characteristics and drop surface deformation, the decrease in the terminal velocity is found to be directly linked to a shortening of the wake recirculation region resulting from an earlier and more drastic increase in the turbulence kinetic energy in the shear layer. The turbulent surroundings trigger substantial rises in the drop axis ratio amplitude and a slight increase in the drop oscillation frequency, but barely influence the time-averaged drop axis length.

Understanding Transmission Dynamics of COVID-19-Type Infections by Direct Numerical Simulations of Cough/Sneeze Flows.

The transmission dynamics of highly contagious respiratory diseases like COVID-19 (through coughing/sneezing) is an open problem in the epidemiological studies of such diseases (Bourouiba, JAMA. https://doi.org/10.1001/jama.2020.4756. 2020). The problem is basically the fluid dynamics of a transient turbulent jet/puff with buoyancy, laden with evaporating droplets carrying the pathogen. A turbulent flow of this nature does not lend itself to reliable estimates through modeling approaches such as RANS (Reynolds-Averaged Navier–Stokes equations) or other droplet-based models. However, direct numerical simulations (DNS) of what may be called “cough/sneeze flows” can play an important role in understanding the spread of the contagion. The objective of this work is to develop a DNS code for studying cough/sneeze flows by a suitable combination of the DNS codes available with the authors (developed to study cumulus cloud flows including thermodynamics of phase change and the dynamics of small water droplets) and to generate useful data on these flows. Recent results from the cumulus cloud simulations are included to highlight the effect of turbulent entrainment (which is one of the key processes in determining the spread of the expiratory flows) on the distribution of liquid water content in a moist plume. Furthermore, preliminary results on the temperature distribution in a “dry cough” (i.e., without inclusion of liquid droplets) are reported to illustrate the large spatial extent and time duration over which the cough flow can persist after the coughing has stopped. We believe that simulations of this kind can help to devise more accurate guidelines for separation distances between neighbors in a group, design better masks, and minimize the spread of respiratory diseases of the COVID-19 type.

Numerical simulation of a drop impact on a superhydrophobic surface with a wire

Superhydrophobic surfaces patterned with macroscale (≈1 mm) structures have gained increasing interest in the past years because of their potential in reducing the contact time between impacting liquid drops and the solid surface. The reduced wettability of these surfaces is of interest in numerous technical applications, as, for example, in anti-icing on airplane wings. Several experimental studies have been carried out on this topic in the literature; on the other hand, only very limited numerical investigations are available in the literature. In this paper, we present a numerical study based on a volume of fluid code for direct numerical simulation of incompressible multiphase flows. A necessary condition for the realization of this study was the implementation of arbitrary-shaped boundaries using a Cartesian grid system. Our implementation of embedded boundaries is based on a volume fraction representation of the boundaries and on a piecewise linear approximation of their surface. The discretized boundaries are then cut off from the computational domain, leading to an altered formulation of the discretized governing equations. To validate the method, we show simulation results for different impact velocities for the case of a droplet impacting on a wire, which has been investigated experimentally in the literature. The simulations show good agreement in terms of contact time and impact morphology, thus, showing the validity of the implementation. Moreover, an extensive analysis of the velocity field for this setup is presented, helping us to better understand the underlying physical phenomena.

CFD-DNS simulation of irregular-shaped particle dissolution

A coupled approach between the conventional computational fluid dynamics platform COMSOL and an in-house-developed direct numerical simulation code DigiDiss is presented to study the dissolution kinetics of an irregular-shaped particle in a stirred cuvette. The complex flow dynamics from COMSOL were imported into DigiDiss as an initial flow condition. A digitised 3D particle structure scanned and reconstructed by X-ray micro-tomography was used in the dissolution simulation. A quantitative assessment of the simulation results was made using as a reference experimental data and a theoretical calculation based on the shrinking spherical model with different flow velocity profiles near the particle. The comprehensive agreement demonstrates the coherence of the simulation method in reproducing the experimental behaviour and is seen as a step closer towards developing a computer software design aide to help with formulation development.

Near-wall dynamics of inertial particles in dilute turbulent channel flows

This investigation considers the effect of the Stokes number on the near-wall particle dynamics of two-phase (solid-fluid) turbulent channel flows. The spectral element method-based direct numerical simulation code Nek5000 is used to model the fluid phase at a shear Reynolds number, Reτ = 180. Dispersed particles are tracked using a Lagrangian approach with one-way coupling. Eulerian fluid and particle statistics are gathered and analyzed to determine the effect of the Stokes number, first on macroscopic statistics. Previous work of this nature indicates that mean streamwise particle velocities and root-mean-square velocity fluctuations are reduced in the bulk and increased very close to the wall, an effect which is stronger with increased particle Stokes number or inertial particles. This phenomenon has important consequences for mechanisms such as particle deposition and preferential concentration, and so for the first time, this work aims to elucidate the dynamics of this effect through rigorous analysis on various scales. An in-depth force analysis indicates the importance of the lift force, even at increased Stokes numbers, in predicting particle motion in the buffer layer and log-law regions. It is also observed that pressure gradient and virtual mass forces are significant close to the wall. Alongside bulk velocity and acceleration statistics, microscopic behavior is analyzed by considering region-based particle dynamics. Probability density functions are used to determine the effect of the Stokes number on particle motion in three near-wall regions, as well as within the bulk flow. It is observed that at higher Stokes numbers, the viscous sublayer contains particles with dynamic properties similar to those present in the buffer layer. This suggests rapid interlayer migration in the wall direction, causing increased particle turbulence intensities in near-wall regions. A local flow topology classification method is also used to correlate particle behavior with near-wall coherent turbulent structures, and a mechanism for particle sweep toward the wall is suggested. Finally, low-speed streak accumulation and interlayer particle fluxes are considered and the extent of mixing for low and high Stokes numbers is discussed.

Direct Numerical Simulation Code Validation for Compressible Shear Flows Using Linear Stability Theory

In order to simulate compressible shear flow stability and aeroacoustic problems, a numerical code must be able to capture how a baseflow behaves when submitted to small disturbances. If the disturbances are amplified, the flow is unstable. The linear stability theory (LST) provides a framework to obtain information about the growth rate in relation to the excitation frequency for a given baseflow. A linear direct numerical simulation (DNS) should capture the same growth rate as the LST, providing a severe test for the code. In the present study, DNS simulations of a two-dimensional compressible mixing layer and of a two-dimensional compressible plane jet are performed. Disturbances are introduced at the domain inflow and spatial growth rates obtained with a DNS code are compared with growth rates obtained from LST analyses, for each baseflow, in order to verify and validate the DNS code. The good comparison between DNS simulations and LST results indicates that the code is able to simulate compressible flow problems and it is possible to use it to perform numerical simulation of instability and aeroacoustic problems.

Direct numerical simulations of reacting flows with detailed chemistry using many-core/GPU acceleration

A new direct numerical simulation (DNS) code for multi-component gaseous reacting flows has been developed at KAUST, with the state-of-the-art programming model for next generation high performance computing platforms. The code, named KAUST Adaptive Reacting Flows Solver (KARFS), employs the MPI+X programming, and relies on Kokkos for “X” for performance portability to multi-core, many-core and GPUs, providing innovative software development while maintaining backward compatibility with established parallel models and legacy code. The capability and potential of KARFS to perform DNS of reacting flows with large, detailed reaction mechanisms is demonstrated with various model problems involving ignition and turbulent flame propagations with varying degrees of chemical complexities.

A dual communicator and dual grid-resolution algorithm for petascale simulations of turbulent mixing at high Schmidt number

A new dual-communicator algorithm with very favorable performance characteristics has been developed for direct numerical simulation (DNS) of turbulent mixing of a passive scalar governed by an advection–diffusion equation. We focus on the regime of high Schmidt number (Sc), where because of low molecular diffusivity the grid-resolution requirements for the scalar field are stricter than those for the velocity field by a factor Sc. Computational throughput is improved by simulating the velocity field on a coarse grid of Nv3 points with a Fourier pseudo-spectral (FPS) method, while the passive scalar is simulated on a fine grid of Nθ3 points with a combined compact finite difference (CCD) scheme which computes first and second derivatives at eighth-order accuracy. A static three-dimensional domain decomposition and a parallel solution algorithm for the CCD scheme are used to avoid the heavy communication cost of memory transposes. A kernel is used to evaluate several approaches to optimize the performance of the CCD routines, which account for 60% of the overall simulation cost. On the petascale supercomputer Blue Waters at the University of Illinois, Urbana–Champaign, scalability is improved substantially with a hybrid MPI-OpenMP approach in which a dedicated thread per NUMA domain overlaps communication calls with computational tasks performed by a separate team of threads spawned using OpenMP nested parallelism. At a target production problem size of 81923 (0.5 trillion) grid points on 262,144 cores, CCD timings are reduced by 34% compared to a pure-MPI implementation. Timings for 163843 (4 trillion) grid points on 524,288 cores encouragingly maintain scalability greater than 90%, although the wall clock time is too high for production runs at this size. Performance monitoring with CrayPat for problem sizes up to 40963 shows that the CCD routines can achieve nearly 6% of the peak flop rate. The new DNS code is built upon two existing FPS and CCD codes. With the grid ratio Nθ∕Nv=8, the disparity in the computational requirements for the velocity and scalar problems is addressed by splitting the global communicator MPI_COMM_WORLD into disjoint communicators for the velocity and scalar fields, respectively. Inter-communicator transfer of the velocity field from the velocity communicator to the scalar communicator is handled with discrete send and non-blocking receive calls, which are overlapped with other operations on the scalar communicator. For production simulations at Nθ=8192 and Nv=1024 on 262,144 cores for the scalar field, the DNS code achieves 94% strong scaling relative to 65,536 cores and 92% weak scaling relative to Nθ=1024 and Nv=128 on 512 cores.

Josephson Traveling-Wave Antennas

We propose a new approach to the problem of obtaining coherent radiation from systems with a great number of Josephson junctions, which is based on the concept of traveling-wave antennas. The traveling wave in a line ensures identity of the electrodynamic conditions, under which the junctions operate, whereas the energy leakage to radiation in the lateral direction prevents saturation of the nonlinearity of the individual junctions having a small dynamic range. Simple analytical models, which demonstrate feasibility of the traveling-wave regime, are considered. A code for direct numerical simulation of Josephson microchips including microantennas, lumped elements, and power supply circuits have been developed. Using the direct numerical simulation, a version of the Josephson antenna, which is similar to the simplest single-wire antenna, is studied and the possibility to realize the traveling-wave regime is demonstrated.

Statistical analysis of electroconvection near an ion-selective membrane in the highly chaotic regime

We investigate electroconvection and its impact on ion transport in a model system comprised of an ion-selective membrane, an aqueous electrolyte, and an external electric field applied normal to the membrane. We develop a direct numerical simulation code to solve the governing Poisson-Nernst-Planck and Navier-Stokes equations in three dimensions using a specialized parallel numerical algorithm and sufficient resolution to capture the high frequency and high wavenumber physics. We show a comprehensive statistical analysis of the transport phenomena in the highly chaotic regime. Qualitative and quantitative comparisons of two-dimensional (2D) and 3D simulations include prediction of the mean concentration fields as well as the spectra of concentration, charge density, and velocity signals. Our analyses reveal a significant quantitative difference between 2D and 3D electroconvection. Furthermore, we show that high-intensity yet short-lived current density hot spots appear randomly on the membrane surface, contributing significantly to the mean current density. By examining cross correlations between current density on the membrane and other field quantities we explore the physical mechanisms leading to current hot spots. We also present analysis of transport fluxes in the context of ensemble-averaged equations. Our analysis reveals that in the highly chaotic regime the mixing layer (ML), which spans the majority of the domain extent, is governed by advective fluctuations. Furthermore, we show that in the ML the mean electromigration fluxes cancel out for positive and negative ions, indicating that the mean transport of total salt content within the ML can be represented via the electroneutral approximation. Finally, we present an assessment of the importance of different length scales in enhancing transport by computing the cross covariance of concentration and velocity fluctuations in the wavenumber space. Our analysis indicates that in the majority of the domain the large scales contribute most significantly to transport, while the effects of small scales become more appreciable in regions very near the membrane.7 MoreReceived 14 June 2016DOI:https://doi.org/10.1103/PhysRevFluids.1.073601©2016 American Physical SocietyPhysics Subject Headings (PhySH)Research AreasElectrokinetic flowsNanofluidicsFluid Dynamics