Weak Scale Research Articles

Hot and heavy dark matter from a weak scale phase transition

We point out that dark matter which is produced non-adiabatically in a phase transition (PT) with fast bubble walls receives a boost in velocity which leads to long free-streaming lengths. We find that this could be observed via the suppressed matter power spectrum for dark matter masses around \mathbf{ 10^8 - 10^9}108-109 GeV and energy scales of the PT around \mathbf{ 10^{2} - 10^3}102-103 GeV. The PT should take place at the border of the supercooled regime, i.e. approximately when the Universe becomes vacuum dominated. This work offers novel physics goals for galaxy surveys, Lyman-\alphaɑ, stellar stream, lensing, and 21-cm observations, and connects these to the gravitational waves from such phase transitions, and more speculatively to possible telescope signals of heavy dark matter decay.

Bio-ESMD: A Data Centric Implementation for Large-Scale Biological System Simulation on Sunway TaihuLight Supercomputer

Molecular dynamics (MD) simulations of biological systems are playing an increasingly important role in the research of pathogens and drugs. Most MD methods for biological simulations rely on the listed bonds which interact among specific groups of atoms identified by atom tags (unique atom tags regardless the storage location). However, efficient mapping of tags to atom locations is often challenging on modern many-core processors because data locality can not always be guaranteed for large-scale systems. In this paper, we present Bio-ESMD, a new MD implementation supporting listed bonds. Bio-ESMD is designed and developed based on our previously designed ESMD framework for many-core processors. In Bio-ESMD, we have introduced a data-centric approach for refactoring MD algorithms by reorganizing the cell list data structure to adopt bond lists with guaranteed data locality. Our implementation achieves speedups of over two compared to SW_GROMACS on Sunway TaihuLight. Furthermore, Bio-ESMD can simulate a system of 308.8 million atoms at 1.33 ns/day or 14.44 million atoms at 17.28 ns/day with linear weak scaling efficiency.

Exploring the parameter space of MagLIF implosions using similarity scaling. III. Rise-time scaling

Magnetized liner inertial fusion (MagLIF) is a z-pinch magneto-inertial-fusion concept studied at the Z Pulsed Power Facility of Sandia National Laboratories. Two important metrics characterizing current delivery to a z-pinch load are the peak current and the current-rise time, which is roughly the time interval to reach the peak current. It is known that, when driving a z-pinch load with a longer current-rise time, the performance of the z-pinch decreases. However, a theory to understand and quantify this effect is still lacking. In this paper, we utilize a framework based on similarity scaling to analytically investigate the variations in the performance of MagLIF loads when varying the current-rise time, or equivalently, the implosion timescale. To maintain similarity between the implosions, we provide scaling prescriptions of experimental input parameters defining a MagLIF load and derive the expected scaling laws for stagnation conditions and for various performance metrics. We compare predictions of the theory to 2D numerical simulations using the radiation, magneto-hydrodynamic code hydra. For several metrics, we find acceptable agreement between the theory and simulations. Our results show that the voltage φload near the MagLIF load follows a weak scaling law φload∝tφ−0.12 with respect to the characteristic timescale tφ of the voltage source, instead of the ideal φload∝tφ−1 scaling. This occurs because the imploding height of the MagLIF load must increase to preserve end losses. As a consequence of the longer imploding liners, the required total laser preheat energy and delivered electric energy increase. Overall, this study helps understand the trade-offs of the MagLIF design space when considering future pulsed-power generators with shorter and longer current-rise times.

A Right-handed neutrino portal to the hidden sector: active neutrinos and their twins in an F-theory model

We analyze the neutrino phenomenology in an SU(5) F-theory model with both a visible sector and a twin hidden sector. At low energies, the strong and weak scales of the two sectors may differ but the spectrum of states is described by the MSSM (MSSM′) in the visible (twin) sectors. What is special about the model is that there are right-handed neutrinos which couple to both sectors via Yukawa couplings. As a result, assuming 3 right-handed neutrinos with a large mass much greater than the weak scale, at tree-level the seesaw mechanism results in 3 massive Majorana neutrinos and 3 massless ones. The massless neutrinos acquire mass via radiative corrections. In our analysis, the massless neutrinos are predominantly active neutrinos, while the massive neutrinos are predominantly sterile neutrinos. We fit the active neutrino masses and mixing angles and discuss the phenomenology of the lightest sterile neutrino. Finally we consider some possible scenarios for cosmology.

Scalable matrix-free solver for 3D transfer of polarized radiation in stellar atmospheres

We present an efficient and massively parallel solution strategy for the transfer problem of polarized radiation, for a 3D stationary medium out of local thermodynamic equilibrium. Scattering processes are included accounting for partial frequency redistribution effects. Such a setting is one of the most challenging ones in radiative transfer modeling. The problem is formulated for a two-level atomic model, which allows linearization. The discrete ordinate method alongside an exponential integrator are used for discretization. Efficient solution is obtained with a Krylov method equipped with a tailored physics-based preconditioner. A matrix-free approach results in a lightweight implementation, suited for tackling large problems. Near-optimal strong and weak scalability are obtained with two complementary decompositions of the computational domain. The presented approach made it possible to perform simulations for the Ca i line at 4227Å with more than 109 degrees of freedom in less than half an hour on massively parallel machines, always converging in a few iterations for the proposed tests.

Spin-2 Kaluza-Klein scattering in a stabilized warped background

Scattering amplitudes involving massive spin-2 particles typically grow rapidly with energy. In this paper we demonstrate that the anomalous high-energy growth of the scattering amplitudes cancel for the massive spin-2 Kaluza-Klein modes arising from compactified five-dimensional gravity in a stabilized warped geometry. Generalizing previous work, we show that the two sum rules which enforce the cancellations between the contributions to the scattering amplitudes coming from the exchange of the (massive) radion and those from the exchange of the tower of Goldberger-Wise scalar states (admixtures of the original gravitational and scalar fields of the theory) still persist in the case of the warping which would be required to produce the hierarchy between the weak and Planck scales in a Randall-Sundrum model. We provide an analytic proof of one combination of these generalized scalar sum rules and show how the sum rule depends on the Einstein equations determining the background geometry and the mode-equations and normalization of the tower of physical scalar states. Finally, we provide a consistent and self-contained derivation of the equations governing the physical scalar modes, and we list, in appendixes, the full set of sum rules ensuring proper high-energy growth of all $2\ensuremath{\rightarrow}2$ massive spin-2 scattering amplitudes.

Scalable computational kernels for mortar finite element methods

Targeting simulations on parallel hardware architectures, this paper presents computational kernels for efficient computations in mortar finite element methods. Mortar methods enable a variationally consistent imposition of coupling conditions at high accuracy, but come with considerable numerical effort and cost for the evaluation of the mortar integrals to compute the coupling operators. In this paper, we identify bottlenecks in parallel data layout and domain decomposition that hinder an efficient evaluation of the mortar integrals. We then propose a set of computational strategies to restore optimal parallel communication and scalability for the core kernels devoted to the evaluation of mortar terms. We exemplarily study the proposed algorithmic components in the context of three-dimensional large-deformation contact mechanics, both for cases with fixed and dynamically varying interface topology, yet these concepts can naturally and easily be transferred to other mortar applications, e.g. classical meshtying problems. To restore parallel scalability, we employ overlapping domain decompositions of the interface discretization independent from the underlying volumes and then tackle parallel communication for the mortar evaluation by a geometrically motivated reduction of ghosting data. Using three-dimensional contact examples, we demonstrate strong and weak scalability of the proposed algorithms up to 480 parallel processes as well as study and discuss improvements in parallel communication related to mortar finite element methods. For the first time, dynamic load balancing is applied to mortar contact problems with evolving contact zones, such that the computational work is well balanced among all parallel processors independent of the current state of the simulation.

EPSILOD: efficient parallel skeleton for generic iterative stencil computations in distributed GPUs

Iterative stencil computations are widely used in numerical simulations. They present a high degree of parallelism, high locality and mostly-coalesced memory access patterns. Therefore, GPUs are good candidates to speed up their computation. However, the development of stencil programs that can work with huge grids in distributed systems with multiple GPUs is not straightforward, since it requires solving problems related to the partition of the grid across nodes and devices, and the synchronization and data movement across remote GPUs. In this work, we present EPSILOD, a high-productivity parallel programming skeleton for iterative stencil computations on distributed multi-GPUs, of the same or different vendors that supports any type of n-dimensional geometric stencils of any order. It uses an abstract specification of the stencil pattern (neighbors and weights) to internally derive the data partition, synchronizations and communications. Computation is split to better overlap with communications. This paper describes the underlying architecture of EPSILOD, its main components, and presents an experimental evaluation to show the benefits of our approach, including a comparison with another state-of-the-art solution. The experimental results show that EPSILOD is faster and shows good strong and weak scalability for platforms with both homogeneous and heterogeneous types of GPU.

KPZ equation with a small noise, deep upper tail and limit shape

In this paper, we consider the KPZ equation under the weak noise scaling. That is, we introduce a small parameter $$\sqrt{\varepsilon }$$ in front of the noise and let $$\varepsilon \rightarrow 0$$ . We prove that the one-point large deviation rate function has a $$\frac{3}{2}$$ power law in the deep upper tail. Furthermore, by forcing the value of the KPZ equation at a point to be very large, we prove a limit shape of the solution of the KPZ equation as $$\varepsilon \rightarrow 0$$ . This confirms the physics prediction in Hartmann et al. (Phys Rev Res 1(3):032043, 2019), Kolokolov and Korshunov (Phys Rev B 75(14):140201, 2007, Phys Rev E 80(3):031107, 2009), Kamenev et al. (Phys Rev E 94(3):032108, 2016), Le Doussal et al. (Phys Rev Lett 117(7):070403, 2016) and Meerson et al. (Phys Rev Lett 116(7):070601, 2016).

Scalable computational algorithms for geospatial COVID-19 spread using high performance computing.

A nonlinear partial differential equation (PDE) based compartmental model of COVID-19 provides a continuous trace of infection over space and time. Finer resolutions in the spatial discretization, the inclusion of additional model compartments and model stratifications based on clinically relevant categories contribute to an increase in the number of unknowns to the order of millions. We adopt a parallel scalable solver that permits faster solutions for these high fidelity models. The solver combines domain decomposition and algebraic multigrid preconditioners at multiple levels to achieve the desired strong and weak scalabilities. As a numerical illustration of this general methodology, a five-compartment susceptible-exposed-infected-recovered-deceased (SEIRD) model of COVID-19 is used to demonstrate the scalability and effectiveness of the proposed solver for a large geographical domain (Southern Ontario). It is possible to predict the infections for a period of three months for a system size of 186 million (using 3200 processes) within 12 hours saving months of computational effort needed for the conventional solvers.

Characterization of Transmission Lines in Microelectronic Circuits Using the ARTEMIS Solver

Modeling and characterization of electromagnetic wave interactions with microelectronic devices to derive network parameters has been a widely used practice in the electronic industry. However, as these devices become increasingly miniaturized with finer-scale geometric features, computational tools must make use of manycore/GPU architectures to efficiently resolve length and time scales of interest. This has been the focus of our open-source solver, ARTEMIS (Adaptive mesh Refinement Time-domain ElectrodynaMIcs Solver), which is performant on modern GPU-based supercomputing architectures while being amenable to additional physics coupling. This work demonstrates its use for characterizing network parameters of transmission lines using established techniques. A rigorous verification and validation of the workflow is carried out, followed by its application for analyzing a transmission line on a CMOS chip designed for a photon-detector application. Simulations are performed for millions of timesteps on state-of-the-art GPU resources to resolve nanoscale features at gigahertz frequencies. The network parameters are used to obtain phase delay and characteristic impedance that serve as inputs to SPICE models. The code is demonstrated to exhibit ideal weak scaling efficiency up to 1024 GPUs and 84% efficiency for 2048 GPUs, which underscores its use for network analysis of larger, more complex circuit devices in the future.

Parthenon—a performance portable block-structured adaptive mesh refinement framework

On the path to exascale the landscape of computer device architectures and corresponding programming models has become much more diverse. While various low-level performance portable programming models are available, support at the application level lacks behind. To address this issue, we present the performance portable block-structured adaptive mesh refinement (AMR) framework Parthenon, derived from the well-tested and widely used Athena++ astrophysical magnetohydrodynamics code, but generalized to serve as the foundation for a variety of downstream multi-physics codes. Parthenon adopts the Kokkos programming model, and provides various levels of abstractions from multidimensional variables, to packages defining and separating components, to launching of parallel compute kernels. Parthenon allocates all data in device memory to reduce data movement, supports the logical packing of variables and mesh blocks to reduce kernel launch overhead, and employs one-sided, asynchronous MPI calls to reduce communication overhead in multi-node simulations. Using a hydrodynamics miniapp, we demonstrate weak and strong scaling on various architectures including AMD and NVIDIA GPUs, Intel and AMD x86 CPUs, IBM Power9 CPUs, as well as Fujitsu A64FX CPUs. At the largest scale on Frontier (the first TOP500 exascale machine), the miniapp reaches a total of 1.7 × 1013 zone-cycles/s on 9216 nodes (73,728 logical GPUs) at [Formula: see text] weak scaling parallel efficiency (starting from a single node). In combination with being an open, collaborative project, this makes Parthenon an ideal framework to target exascale simulations in which the downstream developers can focus on their specific application rather than on the complexity of handling massively-parallel, device-accelerated AMR.

Direct numerical simulation of compressible turbulence accelerated by graphics processing unit: An open-source high accuracy accelerated computational fluid dynamic software

This paper introduces open-source computational fluid dynamics software named open computational fluid dynamic code for scientific computation with graphics processing unit (GPU) system (OpenCFD-SCU), developed by the authors for direct numerical simulation of compressible wall-bounded turbulence. This software is based on the finite difference method and is accelerated by the use of a GPU, which provides an acceleration by a factor of more than 200 compared with central processing unit (CPU) software based on the same algorithm and a number of Message Passing Interface processes, and the running speed of OpenCFD-SCU with just 512 GPUs exceeds that of CPU software with 130 000 CPUs. GPU-Stream technology is used to implement overlap of computing and communication, achieving 98.7% parallel weak scalability with 24 576 GPUs. The software includes a variety of high-precision finite difference schemes and supports a hybrid finite difference scheme, enabling it to provide both robustness and high precision when simulating complex supersonic and hypersonic flows. When used with the wide range of supercomputers currently available, the software should be able to improve the performance of large-scale simulations by up to two orders on the computational scale. Then, OpenCFD-SCU is applied to a validation and verification case of a Mach 2.9 compression ramp with mesh numbers up to 31.2 × 109.

An efficient communication strategy for massively parallel computation in CFD

With the development of high-performance computers, it is necessary to develop efficient parallel algorithms in the field of computational fluid dynamics (CFD). In this study, a novel parallel communication strategy based on asynchronous and packaged communication is proposed. The strategy implements an aggregated communication process, which requires only one communication in each iteration step, significantly reducing the number of communications. The correctness and convergence of the novel strategy are demonstrated from both theoretical and experimental perspectives. And based on the real vehicle CHN-T model with 140 million meshes, a detailed performance comparison and analysis is performed for the novel strategy and the traditional strategy, showing that the novel strategy has significant advantages in terms of scalability. Finally, the strong scalability and weak scalability tests are carried out separately for the CHN-T model. The strong scaling efficiency can reach 74% with 10.5 billion meshes and 256,000 cores. The weak scaling parallel efficiency can reach 90% with 10 billion meshes and 179,000 cores. This research work has laid an important foundation for the development of the fast design of aircraft and cutting-edge numerical methods.

A high order finite difference solver for simulations of turbidity currents with high parallel efficiency

We present a high order finite difference solver, ParaTC , for the direct numerical simulations of turbidity currents with canonical turbulent channel configuration (periodic boundary conditions in the horizontal directions and non-periodic in vertical direction). Uniform meshes are adopted in streamwise and spanwise directions, while stretched grids can be used in wall-normal direction. In order to improve the parallel efficiency, we propose a new 2D pencil-like parallel configuration with totally 6 different pencil arrangements. A parallel Thomas algorithm is also included to further reduce the communication overhead when solving tridiagonal equations. In addition, we perform an optimal search method in the initializing stage to find the fastest Poisson solver scheme among four alternatives for a specific mesh configuration. The runtime ratio between traditional pencil-like Poisson solver and present solver is about 1.5. An approximate linear strong scaling performance is achieved, and the weak scaling performance is also improved. Three benchmark simulations are preformed, and the statistics are compared with those extracted from the simulations by the spectral method, and good agreements are achieved. The source code is freely available at https://github.com/GongZheng-Justin/ParaTC .

Catch-me-if-you-can: the overshoot problem and the weak/inflation hierarchy

We study the overshoot problem in the context of post-inflationary string cosmology (in particular LVS). LVS cosmology features a long kination epoch as the volume modulus rolls down the exponential slope towards the final minimum, with an energy density that scales as {m}_s^4 . It is a known fact that such a roll admits attractor tracker solutions, and if these are located the overshoot problem is solved. We show that, provided a sufficiently large hierarchy exists between the inflationary scale and the weak scale, this will always occur in LVS as initial seed radiation grows into the tracker solution. The consistency requirement of ending in a stable vacuum containing the weak hierarchy therefore gives a preference for high inflationary scales — an anthropic argument, if one likes, for a large inflation/weak hierarchy. We discuss various origins, both universal and model-dependent, of the initial seed radiation (or matter). One particularly interesting case is that of a fundamental string network arising from brane inflation — this may lead to an early epoch in which the universe energy density principally consists of gravitational waves, while an LVS fundamental string network survives into the present universe.

GPU-accelerated DNS of compressible turbulent flows

This paper explores strategies to transform an existing CPU-based high-performance computational fluid dynamics solver, HyPar, for compressible flow simulations on emerging exascale heterogeneous (CPU+GPU) computing platforms. The scientific motivation for developing a GPU-enhanced version of HyPar is to simulate canonical turbulent flows at the highest resolution possible on such platforms. We show that optimizing memory operations and thread blocks results in 200x speedup of computationally intensive kernels compared with a CPU core. Using multiple GPUs and CUDA-aware MPI communication, we demonstrate both strong and weak scaling of our GPU-based HyPar implementation on the NVIDIA Volta V100 GPUs. We simulate the decay of homogeneous isotropic turbulence in a triply periodic box on grids with up to 10243 points (5.3 billion degrees of freedom) and on up to 1,024 GPUs. We compare the wall times for CPU-only and CPU+GPU simulations. The results presented in the paper are obtained on the Summit and Lassen supercomputers at Oak Ridge and Lawrence Livermore National Laboratories, respectively.

An OpenMP GPU-offload implementation of a non-equilibrium solidification cellular automata model for additive manufacturing

In this paper, performance strategies on GPU-based HPC platforms of a cellular automata (CA) simulation code for non-equilibrium solidification, including nucleation, grain growth, solute partitioning and transport for the metal additive manufacturing (AM) process are investigated using OpenMP 4.5. To accurately report the speed-up for multicore CPUs and GPUs, a rigorous performance analysis employed optimizations appropriate for both CPU-only code (baseline) and GPU offload codes for an isothermal test problem. The performance results on Summit at the Oak Ridge Leadership Computing Facility indicate that using a precomputed list of interface cells significantly decreased the wall-clock time on GPUs. The speedup due to GPU acceleration was evaluated for a full Summit node and measured to be 1.8X when comparing a 6 MPI tasks run with 6 GPUs versus 36 MPI tasks on the CPU only. That speed-up was found to be 7.9X when comparing 6 MPI tasks with 6 GPUs versus the 6 MPI tasks running on the CPU only. Performance measurements showed that system total time is almost constant for runs with more than 96 MPI tasks (or GPUs), indicating that the GPU-accelerated code showed an excellent weak scaling performance.Finally, a rapid directional solidification problem was considered to demonstrate the CA code capability on Summit. It was found that a mesh size of at least 0.05 μm is recommended for the AM-like simulations in order to obtain accurate elongated grain microstructure and elongated subgrain features, which are in qualitative good agreement with experimental data. The results presented in this study indicate that the performance strategies on GPU-based HPC platforms for the CA code are appropriate for novel HPC exascale platforms.

A simple one-step index algorithm for implementation of lattice Boltzmann method on GPU

We proposed a simple one-step index (OSI) algorithm for solving the lattice Boltzmann equation, particularly the streaming of particle distribution functions (PDFs) on a single grid system. The OSI algorithm is derived from the conventional A-B pattern. The memory addresses of the PDFs are fixed in this algorithm and consistent with collision principles. The streaming process is implicitly computed by reassigning their indexes corresponding to the time steps, spatial coordinates, and directions of the PDFs. The algorithm is simple to program because it reads and writes the PDFs only once per time step and does not require the synchronization of odd and even time steps. In this implementation, the data layout of the PDFs is the structure of arrays (SoA), suitable for the memory access pattern of graphics processing units (GPUs). The accuracy and single-precision performance of the proposed algorithm for the three-dimensional lid-driven cavity flow simulation with the D3Q19 model were validated and tested on an NVIDIA A100 having a 40 GB PCIe using CUDA and OpenACC. Performances of 8.4 and 8.1 giga lattice updates per second were obtained for CUDA and OpenACC, respectively. OpenACC can outperform CUDA by up to 95% with significantly less programming work. The bandwidth usage rates on a single GPU were 96% and 94% for CUDA and OpenACC, respectively, close to the theoretical values. Lattice Boltzmann method parallelism is implemented using CUDA and MPI for multi-GPU usage. Finally, computation and communication overlaps were implemented to optimize the parallel efficiency, where the weak scaling parallel efficiency exceeded 0.98 on up to 512 GPUs.

A second-order distributed memory parallel fast sweeping method for the Eikonal equation

The Eikonal equation is used to calculate wave propagation and distance fields, and due to its complexity requires numerical treatment for its solution. We present a second-order distributed memory parallel fast sweeping method. The second-order solution switches on a two-point stencil when two upwind points are available, and reverts to first-order otherwise. In all examples, the second-order method improves the solution over the first-order, allowing for significant savings in memory while achieving the same accuracy. Parallelization over distributed memory saw good weak scaling with optimal convergence. The computational time for second-order was approximately 2.5 times slower than first-order, where the largest amount of mesh points ran on 144 cores (512 GB) was ≈20 billion. The savings in memory from the second-order method combined with the distributed memory algorithm result in the ability to solve problems much larger than are possible with the serial first-order method.