Sunway TaihuLight Research Articles

Implementation and optimisation of the cdugksFoam solver on the Sunway TaihuLight supercomputer

In this study, the cdugksFoam solver was successfully implemented on the Sunway TaihuLight system using the MPI + Athread programming model. To utilise the heterogeneous SW26010 many-core processor fully, we implemented three levels of parallelisation: MPI process-level hybrid parallelisation in physical space and velocity space, thread-level parallelisation to further partition physical space, and single-instruction multiple-data (SIMD) vectorisation. To address the performance bottleneck caused by the low memory bandwidth of the SW26010 processor, a series of optimisation methods, including kernel fusion, transcendental function optimisation, and soft cache, were designed and implemented to reduce discrete memory access and improve the computational efficiency of CPEs. The accuracy of the optimised program was validated through simulations of the 3D lid-driven cavity flow and rarefied supersonic flow past a sphere. Based on the experimental results from multiple grid scales, the overall performance achieved an acceleration of over 5 times compared to running on the management processing elements. In both strong and weak scalability tests, a parallel efficiency exceeding 90% was achieved.

Massively parallel simulations of multi-stage compressors on Sunway TaihuLight

Massively parallel simulations of multi-stage compressors on Sunway TaihuLight

ESA: An efficient sequence alignment algorithm for biological database search on Sunway TaihuLight

In computational biology, biological database search has been playing a very important role. Since the COVID-19 outbreak, it has provided significant help in identifying common characteristics of viruses and developing vaccines and drugs. Sequence alignment, a method finding similarity, homology and other information between gene/protein sequences, is the usual tool in the database search. With the explosive growth of biological databases, the search process has become extremely time-consuming. However, existing parallel sequence alignment algorithms cannot deliver efficient database search due to low utilization of the resources such as cache memory and performance issues such as load imbalance and high communication overhead. In this paper, we propose an efficient sequence alignment algorithm on Sunway TaihuLight, called ESA, for biological database search. ESA adopts a novel hybrid alignment algorithm combining local and global alignments, which has higher accuracy than other sequence alignment algorithms. Further, ESA has several optimizations including cache-aware sequence alignment, capacity-aware load balancing and bandwidth-aware data transfer. They are implemented in a heterogeneous processor SW26010 adopted in the world’s 6th fastest supercomputer, Sunway TaihuLight. The implementation of ESA is evaluated with the Swiss-Prot database on Sunway TaihuLight and other platforms. Our experimental results show that ESA has a speedup of 34.5 on a single core group (with 65 cores) of Sunway TaihuLight. The strong and weak scalabilities of ESA are tested with 1 to 1024 core groups of Sunway TaihuLight. The results show that ESA has linear weak scalability and very impressive strong scalability. For strong scalability, ESA achieves a speedup of 338.04 with 1024 core groups compared with a single core group. We also show that our proposed optimizations are also applicable to GPU, Intel multicore processors, and heterogeneous computing platforms.

Direct simulation of flow field around SUBOFF in grid-generated turbulence with SWLBM

SWLBM is a Lattice Boltzmann Method (LBM) based software designed running on Sunway TaihuLight Supercomputer. In this paper, we introduce the simulation of flow over DARPA SUBOFF model in grid-generated turbulence with SWLBM. The simulation setting follows the Particle Image Velocimetry (PIV) measurement experiment done in the water tunnel of Beijing University of Aeronautics and Astronautics (BUAA) with a low speed of 0.25 m/s. The turbulent flow generated from a grid formed by 36 mm square mesh interlocking with 6 mm square bars was simulated directly, and the evolution of turbulence transition over the forebody surface of a scaled-down SUBOFF model with diameter of 200 mm was captured. The simulation results including velocity field and statistical variables were compared relatively well with experimental data. The simulation case has lattice size of 7.8billions and could finish 200,000 iterations within 24 h running on 2000 core groups of Sunway Taihulight supercomputer. This shows SWLBM is efficient tool for high resolution simulation for turbulent study.

Bio-ESMD: A Data Centric Implementation for Large-Scale Biological System Simulation on Sunway TaihuLight Supercomputer

Molecular dynamics (MD) simulations of biological systems are playing an increasingly important role in the research of pathogens and drugs. Most MD methods for biological simulations rely on the listed bonds which interact among specific groups of atoms identified by atom tags (unique atom tags regardless the storage location). However, efficient mapping of tags to atom locations is often challenging on modern many-core processors because data locality can not always be guaranteed for large-scale systems. In this paper, we present Bio-ESMD, a new MD implementation supporting listed bonds. Bio-ESMD is designed and developed based on our previously designed ESMD framework for many-core processors. In Bio-ESMD, we have introduced a data-centric approach for refactoring MD algorithms by reorganizing the cell list data structure to adopt bond lists with guaranteed data locality. Our implementation achieves speedups of over two compared to SW_GROMACS on Sunway TaihuLight. Furthermore, Bio-ESMD can simulate a system of 308.8 million atoms at 1.33 ns/day or 14.44 million atoms at 17.28 ns/day with linear weak scaling efficiency.

Large-Scale Simulation of Full Three-Dimensional Flow and Combustion of an Aero-Turbofan Engine on Sunway TaihuLight Supercomputer

Computational fluid dynamics- (CFD-) based component-level numerical simulation technology has been widely used in the design of aeroengines. However, due to the strong coupling effects between components, the numerical simulation of the whole engine considering the full three-dimensional flow and multi-component chemical reaction is still very difficult at present. Aimed at this problem, an efficient implicit solver, 'sprayDyMFoam' for an unstructured mesh, is developed in this paper based on the Sunway TaihuLight supercomputer. This sprayDyMFoam solver improves the PIMPLE algorithm in the solution of aerodynamic force and adjusts the existing droplet atomization model in the solution of the combustion process so as to meet the matching situation between components and the combustion chamber in the solution process. Meanwhile, the parallel communication mechanism of AMI boundary processing is optimized based on the hardware environment of the Sunway supercomputer. The sprayDyMFoam solver is used to simulate a typical double-rotor turbofan engine: the calculation capacity and efficiency meet the use requirements, and the obtained compressor performance can form a good match with the test. The research proposed in this paper has strong application value in high-confidence computing, complex phenomenon capturing, and time and cost reduction for aeroengine development.

AGCM-3DLF: Accelerating Atmospheric General Circulation Model via 3-D Parallelization and Leap-Format

The atmospheric general circulation model (AGCM) has been an important research tool in the study of climate change for decades. As the demand for high-resolution simulation is becoming urgent, the scalability and simulation efficiency is faced with great challenges, especially for the latitude-longitude mesh-based models. In this paper, we propose a highly scalable 3-D atmospheric general circulation model based on leap-format, namely AGCM-3DLF. First, it utilizes a 3-D decomposition method allowing for parallelism release in all three physical dimensions. Then the leap-format difference computation scheme is adopted to maintain computational stability in grid updating and avoid additional filtering at the high latitudes. A novel shifting window communication algorithm is designed for parallelization of the unified model. Furthermore, a series of optimizations are conducted to improve the effectiveness of large-scale simulations. Experiment results in different platforms demonstrate good efficiency and scalability of the model. AGCM-3DLF scales up to the entire CAS-Xiandao1 supercomputer (196,608 CPU cores), attaining the speed of 11.1 simulation-year-per-day (SYPD) at a high resolution of 25KM. In addition, simulations conducted on the Sunway TaihuLight supercomputer exhibit a 1.06 million cores scalability with 36.1% parallel efficiency.

SwPHoToNs: Toward trillion‐body‐scale cosmologicalN‐body simulations on SunwayTaihuLightsupercomputer

AbstractCosmologicalN‐body simulations have been essential for astronomers to study the formation of nonlinear structures and hypotheses of dark matter, dark energy, etc. The scale of the problem naturally leads to extreme scenarios with billions or even trillions of particles, thus demanding massive computational power and highly efficient algorithms. In this paper, we present swPHoToNs, a Particle‐Mesh (PM) and Fast Multipole Method‐ (FMM) based code that can perform cosmological simulations with trillions of particles efficiently on the Sunway TaihuLight supercomputer. Our design includes three novel optimizations: (1) a multilevel domain decomposition and dynamic load‐balancing scheme; (2) a pipeline strategy for tree traversal and gravity calculation; (3) optimizations for both computation and MPI kernels with consideration of the hardware features. We manage to conduct cosmological simulations which contain up to 1.6 trillion particles, obtaining a sustained performance of 56.3 PFlops with a weak‐scaling parallel efficiency of 80.9% and a computational efficiency of 44.9%.

End-to-end I/O Monitoring on Leading Supercomputers

This paper offers a solution to overcome the complexities of production system I/O performance monitoring. We present Beacon, an end-to-end I/O resource monitoring and diagnosis system for the 40960-node Sunway TaihuLight supercomputer, currently the fourth-ranked supercomputer in the world. Beacon simultaneously collects and correlates I/O tracing/profiling data from all the compute nodes, forwarding nodes, storage nodes, and metadata servers. With mechanisms such as aggressive online and offline trace compression and distributed caching/storage, it delivers scalable, low-overhead, and sustainable I/O diagnosis under production use. With Beacon’s deployment on TaihuLight for more than three years, we demonstrate Beacon’s effectiveness with real-world use cases for I/O performance issue identification and diagnosis. It has already successfully helped center administrators identify obscure design or configuration flaws, system anomaly occurrences, I/O performance interference, and resource under- or over-provisioning problems. Several of the exposed problems have already been fixed, with others being currently addressed. Encouraged by Beacon’s success in I/O monitoring, we extend it to monitor interconnection networks, which is another contention point on supercomputers. In addition, we demonstrate Beacon’s generality by extending it to other supercomputers. Both Beacon codes and part of collected monitoring data are released. 1

Redesigning and Optimizing UCSF DOCK3.7 on Sunway TaihuLight

Molecular docking is the process of posing, scoring, and ranking small molecules at the binding sites of proteins to prioritize compounds for experimental testing. It is a widely-used computational method in the drug discovery process. However, it is a highly time-consuming procedure since a receptor may need to find favorable ligand orientations in billions of ligands. UCSF DOCK3.7 is one of the most widely used molecular docking applications. In this paper, we port and optimize UCSF DOCK3.7 on the Sunway TaihuLight supercomputer. To avoid the impact of load imbalance, we employ a producer-consumer strategy that can overlap I/O and computation in order to achieve high performance. Furthermore, we present a new binary file format to replace the mol2db2 file format for ligand storage and adopt xzip rather than gzip to compress ligand files. We show that our file format can reduce I/O time significantly while xzip saves significant storage. For the routines which determine the orientation of a ligand relative to the receptor, we present an improved algorithm to discard geometrically similar orientations. Furthermore, we fuse loops and compress memory usage to store data in fast Local Device Memory (LDM) in order to score ligand orientations with high efficiency. In addition, we propose a number of architecture-specific optimizations. Asynchronous data transfer and vectorization of computation are implemented to take full advantage of the SW26010 processor. Our experiments show that a speedup of 167 can be achieved by using the proposed strategies. Compared to a core of an Intel(R) Core(TM) i9-10900K CPU, our approach achieves speedups of 15 on a SW26010 core group. Furthermore, our implementation achieves strong scalability to hundreds of thousands of heterogeneous cores on the next-generation Sunway supercomputer.

SwSpAMM: optimizing large-scale sparse approximate matrix multiplication on Sunway Taihulight

swSpAMM: optimizing large-scale sparse approximate matrix multiplication on Sunway Taihulight

Benchmarking 50-Photon Gaussian Boson Sampling on the Sunway TaihuLight

Boson sampling is expected to be an important milestone that will demonstrate quantum computational advantage (or quantum supremacy). This work establishes the benchmarking of Gaussian boson sampling (GBS) with threshold detection based on the Sunway TaihuLight supercomputer. To achieve the best performance and provide a competitive scenario for future quantum computing studies, the selected simulation algorithm is fully optimized based on a set of innovative approaches, including a parallel framework with almost perfect load balance and an instruction-level optimizing scheme based on a shortest-path-based instruction scheduling. In addition, data precision is carefully processed by an integer-instruction-based and multiple-precision fixed-point implementation, including 128- and 256-bit precison mode, which can be appropriately selected based on an adaptive precision optimizing scheme. Based on these methods, a highly efficient parallel quantum sampling algorithm is designed. The largest run enables us to obtain one Torontonian function of a <inline-formula><tex-math notation="LaTeX">$100\times 100$</tex-math></inline-formula> submatrix from 50-photon GBS within 20 hours in 128-bit precision and 2 days in 256-bit precision. To our knowledge, this was the largest quantum computing simulation based on Boson Sampling by using modern supercomputers.

FgSpMSpV: A Fine-grained Parallel SpMSpV Framework on HPC Platforms

Sparse matrix-sparse vector (SpMSpV) multiplication is one of the fundamental and important operations in many high-performance scientific and engineering applications. The inherent irregularity and poor data locality lead to two main challenges to scaling SpMSpV over high-performance computing (HPC) systems: (i) a large amount of redundant data limits the utilization of bandwidth and parallel resources; (ii) the irregular access pattern limits the exploitation of computing resources. This paper proposes a fine-grained parallel SpMSpV (fgSpMSpV) framework on Sunway TaihuLight supercomputer to alleviate the challenges for large-scale real-world applications. First,fgSpMSpVadopts an MPI\( + \)OpenMP\( +X \)parallelization model to exploit the multi-stage and hybrid parallelism of heterogeneous HPC architectures and accelerate both pre-/post-processing and main SpMSpV computation. Second,fgSpMSpVutilizes an adaptive parallel execution to reduce the pre-processing, adapt to the parallelism and memory hierarchy of the Sunway system, while still tame redundant and random memory accesses in SpMSpV, including a set of techniques like the fine-grained partitioner, re-collection method, and Compressed Sparse Column Vector (CSCV) matrix format. Third,fgSpMSpVuses several optimization techniques to further utilize the computing resources.fgSpMSpVon the Sunway TaihuLight gains a noticeable performance improvement from the key optimization techniques with various sparsity of the input. Additionally,fgSpMSpVis implemented on an NVIDIA Tesal P100 GPU and applied to the breath-first-search (BFS) application.fgSpMSpVon a P100 GPU obtains the speedup of up to\( 134.38\times \)over the state-of-the-art SpMSpV algorithms, and the BFS application usingfgSpMSpVachieves the speedup of up to\( 21.68\times \)over the state-of-the-arts.

Optimization of Reactive Force Field Simulation: Refactor, Parallelization, and Vectorization for Interactions

Molecular dynamics (MD) simulations are playing an increasingly important role in many areas ranging from chemical materials to biological molecules. With the continuing development of MD models, the potentials are getting larger and more complex. In this article, we focus on the reactive force field (ReaxFF) potential from LAMMPS to optimize the computation of interactions. We present our efforts on refactoring for neighbor list building, bond order computation, as well as valence angles and torsion angles computation. After redesigning these kernels, we develop a vectorized implementation for non-bonded interactions, which is nearly 100 × faster than the management processing element (MPE) on the Sunway TaihuLight supercomputer. Furthermore, we have implemented the three-body-list free torsion angles computation, and propose a line-locked software cache method to eliminate write conflicts in the torsion angle and valence angle interactions resulting in an order-of-magnitude speedup on a single Sunway TaihuLight node. In addition, we achieve a speedup of up to 3.5 compared to the KOKKOS package on an Intel Xeon Gold 6148 core. When executed on 1,024 processes, our implementation enables the simulation of 21,233,664 atoms on 66,560 cores with a performance of 0.032 ns/day and a weak scaling efficiency of 95.71 percent.

OpenACC + Athread collaborative optimization of Silicon-Crystal application on Sunway TaihuLight

The Silicon-Crystal application based on molecular dynamics (MD) is used to simulate the thermal conductivity of the crystal, which adopts the Tersoff potential to simulate the trajectory of the silicon crystal. Based on the OpenACC version, to better solve the problem of discrete memory access and write dependency, task pipeline optimization and the interval graph coloring scheduling method are proposed. Also, the part of codes on CPEs is vectorized by the SIMD command to further improve the computational performance. After the collaborative development of OpenACC+Athread, the performance has been improved by 16.68 times and achieves 2.34X speedup compared with the OpenACC version. Moreover, the application is expanded to 66,560 cores and can simulate reactions of 268,435,456 silicon atoms.

The Exascale Era is Upon Us: The Frontier supercomputer may be the first to reach 1,000,000,000,000,000,000 operations per second

In 2018, a new supercomputer called Summit was installed at Oak Ridge National Laboratory, in Tennessee. Its theoretical peak capacity was nearly 200 peta-flops-that's 200 thousand trillion floating-point operations per second. At the time, it was the most powerful supercomputer in the world, beating out the previous record holder, China's Sunway TaihuLight, by a comfortable margin, according to the well-known Top500 ranking of supercomputers. (Summit is currently No.2, a Japanese supercomputer called Fugaku having since overtaken it.)

What Factors Affect the Performance of Software after Migration: A Case Study on Sunway TaihuLight Supercomputer

What Factors Affect the Performance of Software after Migration: A Case Study on Sunway TaihuLight Supercomputer

Parallel finite volume simulation of the spherical shell dynamo with pseudo-vacuum magnetic boundary conditions

In this paper, we study the parallel simulation of the magnetohydrodynamic (MHD) dynamo in a rapidly rotating spherical shell with pseudo-vacuum magnetic boundary conditions. A second-order finite volume scheme based on a collocated quasi-uniform cubed-sphere grid is applied to the spatial discretization of the MHD dynamo equations. To ensure the solenoidal condition of the magnetic field, we adopt a widely-used approach whereby a pseudo-pressure is introduced into the induction equation. The temporal integration is split by a second-order approximate factorization approach, resulting in two linear algebraic systems both solved by a preconditioned Krylov subspace iterative method. A multi-level restricted additive Schwarz preconditioner based on domain decomposition and multigrid method is then designed to improve the efficiency and scalability. Accurate numerical solutions of two benchmark cases are obtained with our code, comparable to the existing local method results. Several large-scale tests performed on the Sunway TaihuLight supercomputer show good strong and weak scalabilities and a noticeable improvement from the multi-level preconditioner with up to 10368 processor cores.

FMapper: Scalable read mapper based on succinct hash index on SunWay TaihuLight

One of the most important application in bioinformatics is read mapping. With the rapidly increasing number of reads produced by next-generation sequencing (NGS) technology, there is a need for fast and efficient high-throughput read mappers. In this paper, we present FMapper – a highly scalable read mapper on the TaihuLight supercomputer optimized for its fourth-generation ShenWei many-core architecture (SW26010). In order to fully exploit the computational power of the SW26010, we employ dynamic scheduling of tasks, asynchronous I/O and data transfers and implement a vectorized version of the banded Myers algorithm tailored to the 256 bit vector registers of the SW26010. Our performance evaluation demonstrates that FMapper using all 4 compute groups of a single SW26010 processor outperforms S-Aligner on the same hardware as well as RazerS3, Hobbes3, Minimap2 and BWA running on a 4-core Xeon W-2123v3 CPU and achieves speedups of 4.7, 24.8, 2.4, 4.6 and 14.7 respectively. Using several optimizations, we achieve a speedup of 6 compared to the naïve implementation on one compute group of an SW26010 processor and a strong scaling efficiency of 65% on 512 compute groups.

Design and Optimization of Parallel Algorithm for Kalman Filter on SW26010 Many-Core Processors

Kalman filter algorithm, an effective data processing algorithm, has been widely used in space monitoring, wireless communications, tracking systems, financial industry, big data and so on. On Sunway TaihuLight platform, we present an optimized Kalman filter parallel algorithm which is according to new architecture of the SW26010 many-core processors (260 cores) and new programming mode (master and slave heterogeneous collaboration mode). Furthermore, we propose a pipelined parallel mode for Kalman filter algorithm based on seven-level pipeline of SW26010 processor. The vector optimization strategy and double buffering mechanisms are provided to improve parallel efficiency of Kalman filter parallel algorithm on SW26010 processors. The vector optimization strategy can improve data concurrency in parallel computing. In addition, the communication time can be hidden by double buffering mechanisms of SW26010 processors. The experimental results show that the performance and scalability of the parallel Kalman filter algorithm based on SW26010 are greatly improved compared with the CPU algorithm for five data sets, and is also improved compared to the algorithm on GPU.