Sunway TaihuLight Supercomputer Research Articles

Direct simulation of flow field around SUBOFF in grid-generated turbulence with SWLBM

SWLBM is a Lattice Boltzmann Method (LBM) based software designed running on Sunway TaihuLight Supercomputer. In this paper, we introduce the simulation of flow over DARPA SUBOFF model in grid-generated turbulence with SWLBM. The simulation setting follows the Particle Image Velocimetry (PIV) measurement experiment done in the water tunnel of Beijing University of Aeronautics and Astronautics (BUAA) with a low speed of 0.25 m/s. The turbulent flow generated from a grid formed by 36 mm square mesh interlocking with 6 mm square bars was simulated directly, and the evolution of turbulence transition over the forebody surface of a scaled-down SUBOFF model with diameter of 200 mm was captured. The simulation results including velocity field and statistical variables were compared relatively well with experimental data. The simulation case has lattice size of 7.8billions and could finish 200,000 iterations within 24 h running on 2000 core groups of Sunway Taihulight supercomputer. This shows SWLBM is efficient tool for high resolution simulation for turbulent study.

Large-Scale Simulation of Full Three-Dimensional Flow and Combustion of an Aero-Turbofan Engine on Sunway TaihuLight Supercomputer

Computational fluid dynamics- (CFD-) based component-level numerical simulation technology has been widely used in the design of aeroengines. However, due to the strong coupling effects between components, the numerical simulation of the whole engine considering the full three-dimensional flow and multi-component chemical reaction is still very difficult at present. Aimed at this problem, an efficient implicit solver, 'sprayDyMFoam' for an unstructured mesh, is developed in this paper based on the Sunway TaihuLight supercomputer. This sprayDyMFoam solver improves the PIMPLE algorithm in the solution of aerodynamic force and adjusts the existing droplet atomization model in the solution of the combustion process so as to meet the matching situation between components and the combustion chamber in the solution process. Meanwhile, the parallel communication mechanism of AMI boundary processing is optimized based on the hardware environment of the Sunway supercomputer. The sprayDyMFoam solver is used to simulate a typical double-rotor turbofan engine: the calculation capacity and efficiency meet the use requirements, and the obtained compressor performance can form a good match with the test. The research proposed in this paper has strong application value in high-confidence computing, complex phenomenon capturing, and time and cost reduction for aeroengine development.

AGCM-3DLF: Accelerating Atmospheric General Circulation Model via 3-D Parallelization and Leap-Format

The atmospheric general circulation model (AGCM) has been an important research tool in the study of climate change for decades. As the demand for high-resolution simulation is becoming urgent, the scalability and simulation efficiency is faced with great challenges, especially for the latitude-longitude mesh-based models. In this paper, we propose a highly scalable 3-D atmospheric general circulation model based on leap-format, namely AGCM-3DLF. First, it utilizes a 3-D decomposition method allowing for parallelism release in all three physical dimensions. Then the leap-format difference computation scheme is adopted to maintain computational stability in grid updating and avoid additional filtering at the high latitudes. A novel shifting window communication algorithm is designed for parallelization of the unified model. Furthermore, a series of optimizations are conducted to improve the effectiveness of large-scale simulations. Experiment results in different platforms demonstrate good efficiency and scalability of the model. AGCM-3DLF scales up to the entire CAS-Xiandao1 supercomputer (196,608 CPU cores), attaining the speed of 11.1 simulation-year-per-day (SYPD) at a high resolution of 25KM. In addition, simulations conducted on the Sunway TaihuLight supercomputer exhibit a 1.06 million cores scalability with 36.1% parallel efficiency.

SwPHoToNs: Toward trillion‐body‐scale cosmologicalN‐body simulations on SunwayTaihuLightsupercomputer

AbstractCosmologicalN‐body simulations have been essential for astronomers to study the formation of nonlinear structures and hypotheses of dark matter, dark energy, etc. The scale of the problem naturally leads to extreme scenarios with billions or even trillions of particles, thus demanding massive computational power and highly efficient algorithms. In this paper, we present swPHoToNs, a Particle‐Mesh (PM) and Fast Multipole Method‐ (FMM) based code that can perform cosmological simulations with trillions of particles efficiently on the Sunway TaihuLight supercomputer. Our design includes three novel optimizations: (1) a multilevel domain decomposition and dynamic load‐balancing scheme; (2) a pipeline strategy for tree traversal and gravity calculation; (3) optimizations for both computation and MPI kernels with consideration of the hardware features. We manage to conduct cosmological simulations which contain up to 1.6 trillion particles, obtaining a sustained performance of 56.3 PFlops with a weak‐scaling parallel efficiency of 80.9% and a computational efficiency of 44.9%.

End-to-end I/O Monitoring on Leading Supercomputers

This paper offers a solution to overcome the complexities of production system I/O performance monitoring. We present Beacon, an end-to-end I/O resource monitoring and diagnosis system for the 40960-node Sunway TaihuLight supercomputer, currently the fourth-ranked supercomputer in the world. Beacon simultaneously collects and correlates I/O tracing/profiling data from all the compute nodes, forwarding nodes, storage nodes, and metadata servers. With mechanisms such as aggressive online and offline trace compression and distributed caching/storage, it delivers scalable, low-overhead, and sustainable I/O diagnosis under production use. With Beacon’s deployment on TaihuLight for more than three years, we demonstrate Beacon’s effectiveness with real-world use cases for I/O performance issue identification and diagnosis. It has already successfully helped center administrators identify obscure design or configuration flaws, system anomaly occurrences, I/O performance interference, and resource under- or over-provisioning problems. Several of the exposed problems have already been fixed, with others being currently addressed. Encouraged by Beacon’s success in I/O monitoring, we extend it to monitor interconnection networks, which is another contention point on supercomputers. In addition, we demonstrate Beacon’s generality by extending it to other supercomputers. Both Beacon codes and part of collected monitoring data are released. 1

Redesigning and Optimizing UCSF DOCK3.7 on Sunway TaihuLight

Molecular docking is the process of posing, scoring, and ranking small molecules at the binding sites of proteins to prioritize compounds for experimental testing. It is a widely-used computational method in the drug discovery process. However, it is a highly time-consuming procedure since a receptor may need to find favorable ligand orientations in billions of ligands. UCSF DOCK3.7 is one of the most widely used molecular docking applications. In this paper, we port and optimize UCSF DOCK3.7 on the Sunway TaihuLight supercomputer. To avoid the impact of load imbalance, we employ a producer-consumer strategy that can overlap I/O and computation in order to achieve high performance. Furthermore, we present a new binary file format to replace the mol2db2 file format for ligand storage and adopt xzip rather than gzip to compress ligand files. We show that our file format can reduce I/O time significantly while xzip saves significant storage. For the routines which determine the orientation of a ligand relative to the receptor, we present an improved algorithm to discard geometrically similar orientations. Furthermore, we fuse loops and compress memory usage to store data in fast Local Device Memory (LDM) in order to score ligand orientations with high efficiency. In addition, we propose a number of architecture-specific optimizations. Asynchronous data transfer and vectorization of computation are implemented to take full advantage of the SW26010 processor. Our experiments show that a speedup of 167 can be achieved by using the proposed strategies. Compared to a core of an Intel(R) Core(TM) i9-10900K CPU, our approach achieves speedups of 15 on a SW26010 core group. Furthermore, our implementation achieves strong scalability to hundreds of thousands of heterogeneous cores on the next-generation Sunway supercomputer.

Benchmarking 50-Photon Gaussian Boson Sampling on the Sunway TaihuLight

Boson sampling is expected to be an important milestone that will demonstrate quantum computational advantage (or quantum supremacy). This work establishes the benchmarking of Gaussian boson sampling (GBS) with threshold detection based on the Sunway TaihuLight supercomputer. To achieve the best performance and provide a competitive scenario for future quantum computing studies, the selected simulation algorithm is fully optimized based on a set of innovative approaches, including a parallel framework with almost perfect load balance and an instruction-level optimizing scheme based on a shortest-path-based instruction scheduling. In addition, data precision is carefully processed by an integer-instruction-based and multiple-precision fixed-point implementation, including 128- and 256-bit precison mode, which can be appropriately selected based on an adaptive precision optimizing scheme. Based on these methods, a highly efficient parallel quantum sampling algorithm is designed. The largest run enables us to obtain one Torontonian function of a <inline-formula><tex-math notation="LaTeX">$100\times 100$</tex-math></inline-formula> submatrix from 50-photon GBS within 20 hours in 128-bit precision and 2 days in 256-bit precision. To our knowledge, this was the largest quantum computing simulation based on Boson Sampling by using modern supercomputers.

FgSpMSpV: A Fine-grained Parallel SpMSpV Framework on HPC Platforms

Sparse matrix-sparse vector (SpMSpV) multiplication is one of the fundamental and important operations in many high-performance scientific and engineering applications. The inherent irregularity and poor data locality lead to two main challenges to scaling SpMSpV over high-performance computing (HPC) systems: (i) a large amount of redundant data limits the utilization of bandwidth and parallel resources; (ii) the irregular access pattern limits the exploitation of computing resources. This paper proposes a fine-grained parallel SpMSpV (fgSpMSpV) framework on Sunway TaihuLight supercomputer to alleviate the challenges for large-scale real-world applications. First,fgSpMSpVadopts an MPI\( + \)OpenMP\( +X \)parallelization model to exploit the multi-stage and hybrid parallelism of heterogeneous HPC architectures and accelerate both pre-/post-processing and main SpMSpV computation. Second,fgSpMSpVutilizes an adaptive parallel execution to reduce the pre-processing, adapt to the parallelism and memory hierarchy of the Sunway system, while still tame redundant and random memory accesses in SpMSpV, including a set of techniques like the fine-grained partitioner, re-collection method, and Compressed Sparse Column Vector (CSCV) matrix format. Third,fgSpMSpVuses several optimization techniques to further utilize the computing resources.fgSpMSpVon the Sunway TaihuLight gains a noticeable performance improvement from the key optimization techniques with various sparsity of the input. Additionally,fgSpMSpVis implemented on an NVIDIA Tesal P100 GPU and applied to the breath-first-search (BFS) application.fgSpMSpVon a P100 GPU obtains the speedup of up to\( 134.38\times \)over the state-of-the-art SpMSpV algorithms, and the BFS application usingfgSpMSpVachieves the speedup of up to\( 21.68\times \)over the state-of-the-arts.

Optimization of Reactive Force Field Simulation: Refactor, Parallelization, and Vectorization for Interactions

Molecular dynamics (MD) simulations are playing an increasingly important role in many areas ranging from chemical materials to biological molecules. With the continuing development of MD models, the potentials are getting larger and more complex. In this article, we focus on the reactive force field (ReaxFF) potential from LAMMPS to optimize the computation of interactions. We present our efforts on refactoring for neighbor list building, bond order computation, as well as valence angles and torsion angles computation. After redesigning these kernels, we develop a vectorized implementation for non-bonded interactions, which is nearly 100 × faster than the management processing element (MPE) on the Sunway TaihuLight supercomputer. Furthermore, we have implemented the three-body-list free torsion angles computation, and propose a line-locked software cache method to eliminate write conflicts in the torsion angle and valence angle interactions resulting in an order-of-magnitude speedup on a single Sunway TaihuLight node. In addition, we achieve a speedup of up to 3.5 compared to the KOKKOS package on an Intel Xeon Gold 6148 core. When executed on 1,024 processes, our implementation enables the simulation of 21,233,664 atoms on 66,560 cores with a performance of 0.032 ns/day and a weak scaling efficiency of 95.71 percent.

What Factors Affect the Performance of Software after Migration: A Case Study on Sunway TaihuLight Supercomputer

What Factors Affect the Performance of Software after Migration: A Case Study on Sunway TaihuLight Supercomputer

Parallel finite volume simulation of the spherical shell dynamo with pseudo-vacuum magnetic boundary conditions

In this paper, we study the parallel simulation of the magnetohydrodynamic (MHD) dynamo in a rapidly rotating spherical shell with pseudo-vacuum magnetic boundary conditions. A second-order finite volume scheme based on a collocated quasi-uniform cubed-sphere grid is applied to the spatial discretization of the MHD dynamo equations. To ensure the solenoidal condition of the magnetic field, we adopt a widely-used approach whereby a pseudo-pressure is introduced into the induction equation. The temporal integration is split by a second-order approximate factorization approach, resulting in two linear algebraic systems both solved by a preconditioned Krylov subspace iterative method. A multi-level restricted additive Schwarz preconditioner based on domain decomposition and multigrid method is then designed to improve the efficiency and scalability. Accurate numerical solutions of two benchmark cases are obtained with our code, comparable to the existing local method results. Several large-scale tests performed on the Sunway TaihuLight supercomputer show good strong and weak scalabilities and a noticeable improvement from the multi-level preconditioner with up to 10368 processor cores.

Implementation and optimization of ChaCha20 stream cipher on sunway taihuLight supercomputer

Implementation and optimization of ChaCha20 stream cipher on sunway taihuLight supercomputer

MD simulation of hundred-billion-metal-atom cascade collision on Sunway Taihulight

Radiation damage to the steel material of reactor pressure vessels is a major threat to the nuclear reactor safety. It is caused by the metal atom cascade collision, initialized when the atoms are struck by a high-energy neutron. The paper presents MISA-MD, a new implementation of molecular dynamics, to simulate such cascade collision with EAM potential. MISA-MD realizes (1) a hash-based data structure to efficiently store an atom and find its neighbors, and (2) several acceleration and optimization strategies based on SW26010 processor of Sunway Taihulight supercomputer, including an efficient potential table storage and interpolation method, a coloring method to avoid write conflicts, and double-buffer and data reuse strategies. The experimental results demonstrated that MISA-MD has good accuracy and scalability, and obtains a parallel efficiency of over 79% in an 655-billion-atom system. Compared with a state-of-the-art MD program LAMMPS, MISA-MD requires less memory usage and achieves better computational performance. Program summaryProgram Title: MISA-MDCPC Library link to program files:https://doi.org/10.17632/x2bmx6whcd.1Code Ocean capsule:https://doi.org/10.24433/CO.4041607.v1Licensing provisions: BSD 3-clauseProgramming language: C and C++Nature of problem: Molecular dynamics (MD) is a significant method to simulate the cascade collision progress of the key material in nuclear reactors. However, there are many difficulties for existing MD programs to perform large scale cascade collision simulations. Thus, it is especially essential to develop a new MD software to extend cascade collision simulations to larger spatial scale and longer temporal scale.Solution method: To achieve accuracy and effective MD cascade collision simulation, the EAM potential is selected to calculate interactional force between atoms in the simulation system. To extend MD simulation to larger scale, we proposed a hash-based data structure/algorithm to efficiently store an atom and find its neighbors, and several acceleration and optimization strategies based on SW26010 processor of Sunway Taihulight supercomputer.

Loop Selection for Multilevel Nested Loops Using a Genetic Algorithm

Loop selection for multilevel nested loops is a very difficult problem, for which solutions through the underlying hardware-based loop selection techniques and the traditional software-based static compilation techniques are ineffective. A genetic algorithm- (GA-) based method is proposed in this study to solve this problem. First, the formal specification and mathematical model of the loop selection problem are presented; then, the overall framework for the GA to solve the problem is designed based on the mathematical model; finally, we provide the chromosome representation method and fitness function calculation method, the initial population generation algorithm and chromosome improvement methods, the specific implementation methods of genetic operators (crossover, mutation, and selection), the offspring population generation method, and the GA stopping criterion during the GA operation process. Experimental tests with the SPEC2006 and NPB3.3.1 standard test sets were performed on the Sunway TaihuLight supercomputer. The test results indicated that the proposed method can achieve a speedup improvement that is superior to that by the current mainstream methods, which confirm the effectiveness of the proposed method. Solving the loop selection problem of multilevel nested loops is of great practical significance for exploiting the parallelism of general scientific computing programs and for giving full play to the performance of multicore processors.

Communication Coroutines For Parallel Program Using DW26010 Many Core Processor

Communication between parallel programs is an indispensable part of parallel computing. SW26010 is a heterogeneous many-core processor used to build the Sunway Taihu Light supercomputer, which is well suited for parallel computing. There is the designing and implementing a coroutine scheduling system on the SW26010 processor to improve its concurrency, it is very important and necessary to achieve communication between coroutines for the coroutine scheduling system in advance. Therefore, this paper proposes a communication system for data and information exchange between coroutines on SW26010 processor, which contains the following parts. The designing and implementation a producer-consumer mode channel communication based on ring buffer, and it designs synchronization mechanism for condition of multi-producer and multi-consumer based on the different atomic operation on MPE (management processing element) and CPE (computing processing element) of SW26010. There is also the designing of a wake-up mechanism between the producer and the consumer, which reduces the waiting of the program for communication. The testing and analysis of the performance of channel in different numbers of producers and consumers, draw the conclusion that when the number of producers and consumers increases, the channel performance will decrease.

SwFLOW: A large-scale distributed framework for deep learning on Sunway TaihuLight supercomputer

Deep learning technology is widely used in many modern fields and a number of models and software frameworks have been proposed. However, it is still very difficult to process deep learning tasks efficiently on traditional high performance computing (HPC) systems. In this paper, we propose swFLOW: a large-scale distributed framework for deep learning on Sunway TaihuLight. Based on the performance analysis results of convolutional neural network (CNN), we optimize the convolutional layer , and get 10.42× speedup compared to the original version. As for distributed training, we use elastic averaging stochastic gradient descent (EASGD) algorithm to reduce communication. On 512 processes, we get a parallel efficiency of 81.01% with communication period τ = 8 . Particularly, a decentralized implementation of distributed swFLOW system is presented to alleviate bottleneck of the central server. By using distributed swFLOW system, we can scale the batch size up to 4096 among 1024 concurrent processes for cancerous region detection algorithm . The successful application on swFLOW reveals the great opportunity for joint combination of deep learning and HPC system.

A quantum circuit simulator and its applications on Sunway TaihuLight supercomputer

Classical simulation of quantum computation is vital for verifying quantum devices and assessing quantum algorithms. We present a new quantum circuit simulator developed on the Sunway TaihuLight supercomputer. Compared with other simulators, the present one is distinguished in two aspects. First, our simulator is more versatile. The simulator consists of three mutually independent parts to compute the full, partial and single amplitudes of a quantum state with different methods. It has the function of emulating the effect of noise and support more kinds of quantum operations. Second, our simulator is of high efficiency. The simulator is designed in a two-level parallel structure to be implemented efficiently on the distributed many-core Sunway TaihuLight supercomputer. Random quantum circuits can be simulated with 40, 75 and 200 qubits on the full, partial and single amplitude, respectively. As illustrative applications of the simulator, we present a quantum fast Poisson solver and an algorithm for quantum arithmetic of evaluating transcendental functions. Our simulator is expected to have broader applications in developing quantum algorithms in various fields.

Millimeter-Scale and Billion-Atom Reactive Force Field Simulation on Sunway Taihulight

Large-scale molecular dynamics (MD) simulations on supercomputers play an increasingly important role in many research areas. With the capability of simulating charge equilibration (QEq), bonds and so on, Reactive force field (ReaxFF) enables the precise simulation of chemical reactions. Compared to the first principle molecular dynamics (FPMD), ReaxFF has far lower requirements on computational resources so that it can achieve higher efficiencies for large-scale simulations. In this article, we present our efforts on scaling ReaxFF on the Sunway TaihuLight Supercomputer (TaihuLight). We have carefully redesigned the force analysis and neighbor list building steps. By applying fine-grained optimizations we gain better single process performance. For the many-body interactions, we propose an isolated computation and update strategy and implement inverse trigonometric functions. For QEq, we implement a pipelined conjugate gradient (CG) approach to achieving better scalability. Furthermore, we reorganize the data layout and implement the update operation based on data locality in ReaxFF. Our experiments show that this approach can simulate chemical reactions with 1,358,954,496 atoms using 4,259,840 cores with a performance of 0.015 ns/day. To our best knowledge, this is the first realization of chemical reaction simulation with a millimeter-scale force field.

AeSpTV: An Adaptive and Efficient Framework for Sparse Tensor-Vector Product Kernel on a High-Performance Computing Platform

Multi-dimensional, large-scale, and sparse data, which can be neatly represented by sparse tensors, are increasingly used in various applications such as data analysis and machine learning. A high-performance sparse tensor-vector product (SpTV), one of the most fundamental operations of processing sparse tensors, is necessary for improving efficiency of related applications. In this article, we propose aeSpTV, an adaptive and efficient SpTV framework on Sunway TaihuLight supercomputer, to solve several challenges of optimizing SpTVon high-performance computing platforms. First, to map SpTV to Sunway architecture and tame expensive memory access latency and parallel writing conflict due to the intrinsic irregularity of SpTV, we introduce an adaptive SpTV parallelization. Second, to co-execute with the parallelization design while still ensuring high efficiency, we design a sparse tensor data structure named CSSoCR. Third, based on the adaptive SpTV parallelization with the novel tensor data structure, we present an autotuner that chooses the most befitting tensor partitioning method for aeSpTV using the variance analysis theory of mathematical statistics to achieve load balance. Fourth, to further leverage the computing power of Sunway, we propose customized optimizations for aeSpTV. Experimental results show that aeSpTV yields good sacalability on both thread-level and process-level parallelism of Sunway. It achieves a maximum GFLOPS of 195.69 on 128 processes. Additionally, it is proved that optimization effects of the partitioning autotuner and optimization techniques are remarkable.

Quantum circuits design for evaluating transcendental functions based on a function-value binary expansion method

Quantum arithmetic in the computational basis constitutes the fundamental component of many circuit-based quantum algorithms. There exist a lot of studies about reversible implementations of algebraic functions, while research on the higher-level transcendental functions is scant. We propose to evaluate the transcendental functions based on a novel methodology, which is called qFBE (quantum Function-value Binary Expansion) method. This method transforms the evaluation of transcendental functions to the computation of algebraic functions in a simple recursive way. We present the quantum circuits for solving the logarithmic, exponential, trigonometric and inverse trigonometric functions based on the qFBE method. The efficiency of the circuits is demonstrated on a quantum virtual computing system installed on the Sunway TaihuLight supercomputer. The qFBE method provides a unified and programmed solution for the evaluation of transcendental functions, and it will be an important building block for many quantum algorithms.