Enabling mixed-precision in spectral element codes

Development of a parallel spectral element code using SPMD constructs

Controlling the vibrations induced by high-speed trains on environmental infrastructures, especially buildings situated along railway tracks, is an essential demand and a challenging task. In order to accurately predict the dynamic response of the ground and adjacent buildings to railway-induced vibrations, we conduct numerical validation based on an existing in-situ model test in Portugal. This validation is done using the spectral element numerical code SPEED, developed at Politecnico di Milano, which considers a fully coupled 3D model of both the ground and building. The mechanical parameters of the track structure, modeled by the beam on elastic foundation, are obtained by iteratively calibrating the analytical dynamic receptance curve according to the experimental data. We use a multi-objective optimization-based method to estimate the equivalent rectangular sections of building structural components. The recorded dynamic vertical responses of the nearby ground and building slab under the excitation of 219 km/h moving trains are compared with the numerical results. Specific attention is paid to the frequency range that dominates the dynamic response of the building to discuss the accuracy of the results.

Overlapping Schwarz Methods for Unstructured Spectral Elements

This article presents the development of a three-dimensional spectral element-based Piezo-Enabled Spectral Element Analysis code to simulate piezo-induced ultrasonic wave propagation in composite structures. Piezo-Enabled Spectral Element Analysis solves the coupled electromechanical governing equations for a given arbitrary voltage input to a piezoelectric actuator and outputs the voltage response of the piezoelectric sensors. In case of modeling laminated composites, one element per ply is computationally expensive, and smeared material properties may give inaccurate results; hence, a layered solid spectral element is introduced. Layered solid spectral element can model several plies per element and uses combined nodal quadrature and Simpson’s 1/3 integration rule for numerical integration of mechanical stiffness matrix. It is worth noting that Piezo-Enabled Spectral Element Analysis can interface with commercial finite element mesh generators such as Abaqus/CAE to model structures with complex configuration. Delaminations and debonds are modeled by separating the nodes in the damaged area to create a volume split. Experiments and simulations have been carried out to verify and validate Piezo-Enabled Spectral Element Analysis for composites. This article also presents simulations of ultrasonic wave propagation in a stiffened composite plate with delamination and debond.

Single well imaging using full-waveform sonic log data, a known technique for locating fractures and mapping bed boundaries, can also be used to locate boreholes from nearby observation wells. For this technique, generating accurate seismograms including not only reflected waves but also borehole-guided waves is of paramount importance for both the prejob planning phase and for testing and validating the processing sequence in the imaging phase. We implemented a 3D GPU-based spectral element code, which is well suited for this type of modelling in terms of accuracy and performance. With the spectral element method modelling, we were able to reproduce, qualitatively, the case of a vertical borehole acoustically sensed by a nearby horizontal well.

We develop hybrid RANS–LES strategies within the spectral element code Nek5000 based on the k − τ SST turbulence model. We chose airfoil sections with chord-based Reynolds number on the order of 10 5 − 10 6 , in both attached and stalled conditions, as our target problem to comprehensively test the solver accuracy and performance. Verification and validation of the k − τ SST model are performed for two reference cases: for the zero-pressure gradient boundary layer developing on a flat plate and for mild adverse-pressure gradient boundary layers developing on suction side of NACA0012. The k − τ SST model shows good grid convergence characteristics, at par or better in comparison to existing reference results. The results also show good corroboration with existing experimental and numerical datasets for low incoming flow angles. A small discrepancy appears at higher angle in comparison with the experiments, which is in line with our expectations from an RANS formulation. Building on this foundation, we construct a hybrid RANS–LES framework based on the Delayed Detached-Eddy Simulation (DDES) approach. DDES captures both the attached and separated flow dynamics well when compared with available numerical datasets. We demonstrate that for the hybrid approach a high-order spectral element discretization converges faster (i.e. with less resolution) and captures the flow dynamics more accurately than representative low-order approaches. We also revise some of the guidelines on sample size requirements for statistics convergence for massively separated flow within the current numerical framework. Finally, we analyse some of the observed discrepancies of our unconfined DDES at higher angles with the experiments by evaluating the ‘blocking’ effect of wind tunnel walls. We carry out additional simulations for confined domains and assess the observed differences as a function of Reynolds number.

High-Performance Spectral Element Algorithms and Implementations

Spectral element simulation of ultrafiltration

Near-fault effects are known to produce specific features of earthquake ground motion (such as long-period velocity pulses and directivity) that cannot be predicted by numerical approaches involving vertical plane wave propagation in one-dimensional (1D) soil models that are used as a standard in engineering applica- tions. Coupling near-fault conditions with site effects induced by complex geological structures (such as deep alluvial basins or steep topographic irregularities) further con- tributes to the complexity of earthquake ground motion and to the difficulty to provide reliable predictions without making use of large-size 3D numerical simulations. In this article, we present a parametric study of the seismic response of the Grenoble Valley, France (due to an Mw 6 seismic source at some 10 km epicentral distance from the urban area) that was carried out in the framework of an international benchmark for earthquake ground-motion prediction. The spectral element code GeoELSE for seismic-wave propagation analyses in 3D heterogeneous media, in the linear and non- linear range, was used for this purpose; full advantage was taken of its implementation on parallel computer architectures. After introducing GeoELSE and its parallel per- formance, and after introducing some of its validation benchmarks, the spatial varia- bility of the seismic response of the Grenoble Valley is quantitatively investigated taking into account two effects: (i) the hypocenter location and (ii) the nonlinear soil behavior through a nonlinear viscoelastic soil model. Finally, numerical results are compared with available data and attenuation relationships of peak values of ground motion in the near-fault region. Based on the results of this work, the unfavorable interaction between fault rupture, radiation mechanism, and complex geological con- ditions may give rise to large values of peak ground velocity (exceeding 1 m=sec) even in low-to-moderate seismicity areas; it may therefore considerably increase the level of seismic risk, especially in highly populated and industrially active regions, such as the Alpine valleys.

Direct numerical simulation of turbulent non-Newtonian flow using OpenFOAM

Current supercomputing systems have tens or hundreds of thousands of cores and are trending to GPU and co-compute platforms that deliver thousands of cores per node. Modern computational fluid dynamics codes must be designed to take advantage of these developments in order to further their use in the design cycle. Furthermore, these codes must be highly accurate, stable, and geometrically flexible. NEK5000 is a massively-parallel spectral element code that exhibits these characteristics but currently only for incompressible and low-Mach flows. Adding capabilities for NEK5000 to solve the fully compressible Navier-Stokes equations will extend its usefulness to aerospace applications. As a first step the following work extends NEK5000’s capabilities to solve the 2D compressible Euler equations. Using the conservative formulation, the equations are discretized using a non-staggered spectral element mesh, and the state variables are advanced using 1st order explicit Euler time stepping. A channel with a 10% bump is used as a test case for the modification. The modified NEK5000 code performs very well despite not being optimized for use in hyperbolic equations.

T-waves are underwater acoustic waves generated by earthquakes. Modeling of their generation and propagation is a challenging problem. Using a spectral element code-SPECFEM2D, this paper presents the first realistic simulations of T-waves taking into account major aspects of this phenomenon: The radiation pattern of the source, the propagation of seismic waves in the crust, the seismic to acoustic conversion on a non-planar seafloor, and the propagation of acoustic waves in the water column. The simulated signals are compared with data from the mid-Atlantic Ridge recorded by an array of hydrophones. The crust/water interface is defined by the seafloor bathymetry. Different combinations of water sound-speed profiles and sub-seafloor seismic velocities, and frequency content of the source are tested. The relative amplitudes, main arrival-times, and durations of simulated T-phases are in good agreement with the observed data; differences in the spectrograms and early arrivals are likely due to too simplistic source signals and environmental model. These examples demonstrate the abilities of the SPECFEM2D code for modeling earthquake generated T-waves.

Natural convection in differentially heated enclosures is a benchmark problem used to investigate the physics of buoyant flows and to validate numerical methods. Such configurations are also of interest in engineering applications such as cooling of electronic components and air flow around buildings. In this work a spectral element method is used to carry out direct numerical simulations of natural convection in a tall enclosure of aspect ratio 4 with isothermal vertical walls and adiabatic horizontal walls. Spectral element methods combine the flexibility of classical finite element methods with the high accuracy and efficiency of single-domain spectral methods. The flow is solved in a three-dimensional domain with periodic boundary conditions imposed in the third direction. The numerical results are compared with solutions available in the literature and with numerical results obtained using a commercial software that employs a low-order finite volume method. Good agreement with previous work is obtained for the value of the Rayleigh number investigated, Ra = 2E9, which is greater than the critical value of Ra where transition to an unsteady, chaotic state is known to occur. The results are presented in terms of the time-averaged flow structure, Reynolds stresses and modal energies. Although the time-averaged velocity and temperature fields obtained with a commercial finite volume code are in general good agreement with the results obtained with the spectral element code, it does not give accurate predictions of second-order statistics.

A still difficult, yet pressing task for blade manufacturers and turbine producers is the correct prediction of the effects of turbulent winds on the blade. Reynolds Averaged Numerical Simulations (RANS) are a limited tool for calculating the effects. For large eddy simulations (LES) boundary layer calculation are still difficult therefore the spectral element method seems to be an approach to improve numerical calculations of flow separation. The flow field around an fx79-w151a airfoil has been calculated by the spectral element code \U0001d4a9ϵκ\U0001d4afαrusing a direct numerical simulation (DNS) solver. In a first step a laminar inflow on the airfoil at angle of attack of α = 12° and a Reynolds number of Re= 33000 was simulated using the 2D Version of the code. The flow pattern was compared to measurements using holographic particle induced velocimetry (HPIV) in a wind tunnel.

Turbulent flows are most often wall-bounded, rendering the treatment of the wall essential. In this work, the near-wall layer is modelled in large eddy simulations which enables simulating high Reynolds number flows. An algebraic wall model has been implemented in the spectral element code Nek5000. It consists of an approximate boundary condition that relates the wall shear stress to the velocity measured close to the wall, on the upper edge of the first spectral element. The wall shear stress model approximates the law of the wall for hydraulically smooth cases. The model is applied to channel flow cases at Re_{tau }=1000 and at Re_{tau }=5200. The wmLES results obtained with the present implementation are seen to compare very well with those of reference direct numerical simulations in the resolved region. They also remain remarkably close to the reference results for a large part of the under-resolved region; which is not necessarily the case when using low order implementations and even other types of high order discretizations, as found in the literature. Various parameters are studied such as the time averaging, the height of the near wall under-resolved element, and the mesh requirements. The obtained results indicate that accurate results can be obtained with Nek5000 at a reduced cost thanks to this newly implemented wmLES model. This work provides the necessary guidelines for simple flows, and it will serve as a first basis for simulating more complex flows at high Reynolds numbers.